Digging Into The Top500

Who cares about the FLOPS. The real story is about who and how.

The Top500 is the favorite punching bag of many people in HPC. My beef is not with the Top500, it is with all those who make it out to be something it is not. In my opinion, it is good historical record of those machines that can run a single benchmark. There is of course certain bragging rights for landing on the Top500 and it may help people justify an expensive pile of hardware. I suppose getting your system listed on a web page is good thing™, but there are those who have real work to get done and can brag about minor things like application throughput, optimization, and utilization.

I have followed the list for the past several years. I think I can sum it as follows, more nodes, more cores, more InfiniBand, Blue Gene, bigger HPL number. Yawn. This year, however, I find the changes in the Top500 quite interesting. I’m not going to focus on top performance, but rather some trends in the list that seem to be changing.

There is a new champ. In and of itself this is often not very interesting, however, this time it was different in several ways. First, it was from China. The Tianhe-1A system at the National Supercomputer Center in Tianjin, achieved a performance level of 2.57 PFLOPS (i.e. it crunched more numbers than anyone else). China has been marching up in the Top500 ranking. This fall they had 41 total systems in the list. That is rather impressive considering two years earlier they had just 14 systems that made the cut. From what I hear, this is just the beginning.

The other notable difference is how they got to the top. Tianhe-1A is the first number one machine to use GPUs (NVidia). It also used a custom interconnect. Moreover, three of the the top systems used NVidia GPUs while a total of 28 systems on the list used GPU technology (NVidia, AMD, Cell). One other note about GPUs. This years Green500 had five GPU based systems (4 NVidia, 1 AMD). The Green500 ranks systems in MFLOPS/Watt.

I believe we will see more and more systems using GPUs — at least to get on the list. The GPU trend took a firm hold in 2010 and I expect it to continue in many verticals where “array processors” make sense. As I noted previously, I predict the GPU will migrate into the processor just like the co-processors of the past. GPUs for HPC computing are here to stay.

Moving on to other trends. Quad-core processors were used in 73% (365) of the systems, while 19% (95 systems) are already using processors with six or more cores. It is probably safe to say that the typical HPC node has at least 8 cores (2P motherboard with quad processors) and new systems will have at least twelve cores. Intel dominates the high-end processor market, with 79.6% (398) of all systems using Intel processors, although this is slightly down from six months ago (406 systems, 81.2%). AMD Opterons found their way into 57 systems (11.4%), up from 47 on the June list. AMD notes that 24 of the top 50 systems used their processors. IBM Power processor use is slowly declining and is now at 40 systems (8.0%), down from 42 previously.

Gigabit Ethernet (GigE) is still the most-used system interconnect technology (227 systems, down from 244 systems), due to its widespread use by industrial customers. The fact that GigE is “free” on the motherboard probably has something to do with this number. There were only seven systems sporting 10-GigE. As 10-GigE costs come down, it will be interesting to see if it gains in the list against InfiniBand (IB). Speaking of IB, 214 systems used InfiniBand on this list, up from 205 systems in the previous list. Interestingly, InfiniBand-based systems account for two and a half times as much performance (20.4 Pflop/s) than Gigabit Ethernet ones (8.7 Pflop/s).

In terms of operating systems, Linux continues to dominate. By my count, it was used on 449 systems. That is close to 90% of the list and probably the most dominate trend. I don’t expect this change in the near future. Windows HPC server 2008 was reported on five systems. Most notably, the Magic Cube Cluster at the Shanghai Supercomputer Center in China runs Windows. That system uses at least 2K nodes (possibly using using 4P motherboards). I think that is the largest Windows cluster I have seen to date. I’m not sure what advantage Windows brings to the party. Once you see a disk-less node boot in 20 seconds, you get the power of “open plumbing.”

In terms of application area, the leading sectors were “Not Specified” (34%) Research (16.4%), Finance (8.6%), Information Service (7.0%), and Geophysics (3.8%). Remember not all systems are on the Top500 and many are not specified in any case. There were even four clusters in the WWW category.

Finally, a question for the audience. There were 291 systems that had between 4K and 8K processors. That is a lot of processors. I am always curious how many of those are used at the same time by a single job. Surveys indicate that many applications don’t scale to more than 32 cores. Even the “big codes” may not exceed a thousand processors. The heroic codes can use thousands, but these applications are not very common. There were 9 systems with over 128K processors.

The question I always like to ask is; “Who uses 128K processors at once?” I usually don’t see that many hands go up in the air for that question. “How about 1000, or 100, or 50?” More hands start to go up. “Thank you. And now can you explain to me why you care about running Linpack on 128K processors?”

I never hear a good answer to that question.

Comments on "Digging Into The Top500"


here is an answer your question about who uses a large number of processors for one job:

Stochastic methods are very well adapted to using thousands of processors since an unlimited number tasks can be run independently and asynchronously. In addition, fault-tolerance is easy to achieve: if one task doesn’t transmit its output it affects only the error bar on the total result and not the average. These methods are quite frequent (namely in physics, chemistry, finance…) and will probably spread even more in the future due to its massively parallelizable property.

For example, this page shows an application of Quantum Monte Carlo
on the EGEE grid : http://qmcchem.ups-tlse.fr/index.php?title=Large-scale_quantum_Monte_Carlo_electronic_structure_calculations_on_the_EGEE_grid
In addition to grids, this code runs routinely on 512 cores (for a single calculation) with a nearly perfect speedup. It could run on much more if we could have access to more cores!



Nice summary about on the direction HPC clusters are taking.



I think Doug is referring to a single application running on 128,000 cores using MPI.



Boinc uses more then 128K cores. Their cluster just has some latency issues. :)

Im thankful for the article post.Thanks Again. Really Cool.

This is one awesome article.Really looking forward to read more. Really Cool.

My partner and I absolutely love your blog and find a lot of your post’s to be just what I’m looking for. Does one offer guest writers to write content for you personally? I wouldn’t mind writing a post or elaborating on a lot of the subjects you write regarding here. Again, awesome site!

The next time I read a blog, I hope that it doesnt disappoint me as a lot as this one. I imply, I know it was my choice to learn, however I really thought youd have something attention-grabbing to say. All I hear is a bunch of whining about one thing that you might fix for those who werent too busy on the lookout for attention.

The truth just shines thogruh your post

Your articles are for when it abyelutols, positively, needs to be understood overnight.

Hi there! This post couldn’t be written any better! Reading through this post reminds me of my previous room mate! He always kept talking about this. I will forward this article to him. Pretty sure he will have a good read. Thank you for sharing!

A big thank you for your post.Thanks Again.

http://www.healthyingredients.net Health workshops, corporate gym membership and counseling for the management of stress are a few practical methods that are identified by have produced pretty decent results for organizations. The cost of the implementation of such activities is comparatively less than the one incurred by the extended leave of a hardworking distress employee.

http://www.connectfinance.net One particular company is private equity firm, Huntsman Gay Global Capital, who announced on the 30th of September they have made a majority investment in iQor Holdings Inc. The investment through Huntsman Gay in iQor, the second largest accounts receivable services provider on the planet, will allow iQor to increase its share in the accounts receivable (A/R) freelancing market, while introducing Huntsman Gay into the A/R outsourcing market.
http://www.homesmarthome.org While Everest has predicted the finance and accounting outsourcing (FAO) market accurately to date, it will be interesting to see whether the market will still grow as predicted as the year nears its conclusion.
http://www.mypetshop.org When people choose building their home, first thing comes to our mind is cost. Definitely you can save a lot of money if you construct yourself. But that process takes a lot of time, energy through you. If you can afford these two generously then nothing like building your own house! It is just ‘out of the world’ feeling. Building your home enables you to much attached and valued since you have built it brick through brick.
http://www.sportzup.org Home building and home designs go hand in hand. It also demands absolute time and energy from you. You end up being firm on these. Could you afford time and energy? If you can actually, just go for it and make your dream home come the case! Usually you expect at least 5% more on the price of your home when considering your possessions. Because builders usually showcase the competent price to triumph you people. So be very calculative in buying house. Display family homes, mostly will be costlier by 5%than what they have quoted previously.
http://www.travellersbay.net Submission: StrategyThe distribution strategy t2india is focused on the target market each nationallyand internationally. Secondarily, t2india seeks to establish its presence in thecommunity with the media.
http://www.thefinancecompany.net The distribution: StrategyThe distribution strategy t2india is focused on the target market simultaneously nationallyand internationally. Secondarily, t2india seeks to establish its presence in thecommunity on the media.
http://www.businesser.net Distribution: StrategyThe distribution strategy t2india is focused on the target market both nationallyand internationally. Secondarily, t2india seeks to establish its presence in thecommunity through the media.
http://www.remodelinghomeimprovements.net Travel and leisure Perks: Obviously, one of the best things about being a home-based travel agency business owner is that you reap the discounts exclusively on your own travel. The travel perks vary from organization to organization though.

I loved your blog.Much thanks again. Fantastic.

This is really interesting, You are a very
skilled blogger. I have joined your feed and look
forward to seeking more of your great post.
Also, I have shared your web site in my social networks!

I am really loving the theme/design of your blog.
Do you ever encounter any browser compatibility issues?
A couple of my blog audience have complained about my website not working correctly in Explorer but looks great
in Safari. Have you got any advice to aid fix this problem?

my weblog :: BertICoderre

Hi there, I believe your internet site could possibly
be having internet browser compatibility problems. Once I review your internet
site in Safari, it seems fine however, when opening in Internet Explorer,
it’s got some overlapping issues. I simply planned to
offer you a quick heads up! Besides that, wonderful

my weblog: BarrettSAhlf

[APP][2.2+] INSTWOGRAM – Two Instagram Accou… | …

xaDPY3 odlczdatsynx, [url=http://oreweupupjfj.com/]oreweupupjfj[/url], [link=http://fnjgsbyroxig.com/]fnjgsbyroxig[/link], http://bqdljkfrhpzu.com/

Leave a Reply