# Roadrunner (supercomputer)

> Mediated Wiki article. Canonical URL: https://mediated.wiki/source/Roadrunner_(supercomputer)
> Markdown URL: https://mediated.wiki/source/Roadrunner_(supercomputer).md
> Source: https://en.wikipedia.org/wiki/Roadrunner_(supercomputer)
> Source revision: 1353174948
> License: Creative Commons Attribution-ShareAlike 4.0 International (https://creativecommons.org/licenses/by-sa/4.0/)

Former supercomputer built by IBM

For other uses, see [Roadrunner (disambiguation)](/source/Roadrunner_(disambiguation)).

Roadrunner Roadrunner components Active Operational in 2008 Final completion in 2009 Sponsors IBM Operators National Nuclear Security Administration Location Los Alamos National Laboratory Architecture 12,960 IBM PowerXCell 8i CPUs, 6,480 AMD Opteron dual-core processors, InfiniBand Power 2.35 MW Operating system Red Hat Enterprise Linux Space 296 racks, 560 m2 (6,000 sq ft) Memory 103.6 TiB Storage 1,000,000 TiB Speed 1.042 petaFLOPS Cost US$100 million[1] (equivalent to $150 million in 2025) Ranking TOP500: 10, June 2011 Purpose Modeling the decay of the U.S. nuclear arsenal Legacy First TOP500 Linpack sustained 1.0 petaflops, May 25, 2008 Website www.lanl.gov/roadrunner/[dead link]

**Roadrunner** was a [supercomputer](/source/Supercomputer) built by [IBM](/source/IBM) for the [Los Alamos National Laboratory](/source/Los_Alamos_National_Laboratory) in New Mexico, USA. The US$100-million Roadrunner was designed for a peak performance of 1.7 [petaflops](/source/FLOPS). It achieved 1.026 petaflops on May 25, 2008, to become the world's first [TOP500](/source/TOP500) [LINPACK](/source/LINPACK_benchmarks) sustained 1.0 petaflops system.[2][3]

In November 2008, it reached a top performance of 1.456 [petaFLOPS](/source/PetaFLOPS), retaining its top spot in the [TOP500](/source/TOP500) list.[4] It was also the fourth-most energy-efficient supercomputer in the world on the [Supermicro](/source/Supermicro) Green500 list, with an operational rate of 444.94 megaflops per watt of power used. The hybrid Roadrunner design was then reused for several other energy efficient supercomputers.[5] Roadrunner was decommissioned by Los Alamos on March 31, 2013.[6] In its place, Los Alamos commissioned a supercomputer called [Cielo](/source/Cielo_(supercomputer)), which was installed in 2010.

## Overview

IBM built the computer for the U.S. Department of Energy's (DOE) [National Nuclear Security Administration](/source/National_Nuclear_Security_Administration) (NNSA).[7][8] It was a hybrid design with 12,960 IBM [PowerXCell 8i](/source/Cell_(microprocessor))[9] and 6,480 AMD [Opteron](/source/Opteron) dual-core processors[10] in specially designed [blade servers](/source/Blade_server) connected by [InfiniBand](/source/InfiniBand). The Roadrunner used [Red Hat Enterprise Linux](/source/Red_Hat_Enterprise_Linux) along with [Fedora](/source/Fedora_(operating_system))[11] as its operating systems, and was managed with [xCAT](/source/XCAT) [distributed computing](/source/Distributed_computing) software. It also used the [Open MPI](/source/Open_MPI) [Message Passing Interface](/source/Message_Passing_Interface) implementation.[12]

Roadrunner occupied approximately 296 server racks[13] which covered 560 square metres (6,000 sq ft)[14] and became operational in 2008. It was decommissioned March 31, 2013.[13] The DOE used the computer for simulating how nuclear materials age in order to predict whether the USA's aging arsenal of nuclear weapons are both safe and reliable. Other uses for the Roadrunner included the science, financial, automotive, and aerospace industries.

## Hybrid design

Roadrunner differed from other contemporary supercomputers because it continued the hybrid approach[13] to supercomputer design introduced by [Seymour Cray](/source/Seymour_Cray) in 1964 with the [Control Data Corporation](/source/Control_Data_Corporation) [CDC 6600](/source/CDC_6600) and continued with the order of magnitude faster [CDC 7600](/source/CDC_7600) in 1969. However, in this architecture the peripheral processors were used only for operating system functions and all applications ran in the one central processor. Most previous supercomputers had only used one processor architecture, since it was thought to be easier to design and program for. To realize the full potential of Roadrunner, all software had to be written specially for this hybrid architecture. The hybrid design consisted of [dual-core](/source/Dual-core) [Opteron](/source/Opteron) server processors manufactured by [AMD](/source/Advanced_Micro_Devices) using the standard [AMD64](/source/AMD64) [architecture](/source/Instruction_set). Attached to each Opteron core is an IBM-designed and -fabricated [PowerXCell 8i](/source/PowerXCell_8i) processor. As a supercomputer, the Roadrunner was considered an Opteron cluster with Cell accelerators, as each node consists of a Cell attached to an Opteron core and the Opterons to each other.[15]

IBM Engineers and Analysts pose in front of a small portion of the Roadrunner Super Computer

## Development

Roadrunner was in development from 2002 and went online in 2006. Due to its novel design and complexity it was constructed in three phases and became fully operational in 2008. Its predecessor was a machine also developed at Los Alamos named Dark Horse.[16] This machine was one of the earliest hybrid architecture systems originally based on ARM and then moved to the Cell processor. It was entirely a 3D design, its design integrated 3D memory, networking, processors and a number of other technologies.

### Phase 1

The first phase of the Roadrunner was building a standard Opteron based cluster, while evaluating the feasibility to further construct and program the future hybrid version. This Phase 1 Roadrunner reached 71 teraflops and was in full operation at Los Alamos National Laboratory in 2006.

### Phase 2

Phase 2 known as AAIS (Advanced Architecture Initial System) included building a small hybrid version of the finished system using an older version of the Cell processor. This phase was used to build prototype applications for the hybrid architecture. It went online in January 2007.

### Phase 3

Roadrunner undergoing system test in Poughkeepsie in June 2008

The goal of Phase 3 was to reach sustained performance in excess of 1 petaflops. Additional Opteron nodes and new PowerXCell processors were added to the design. These PowerXCell processors are five times as powerful as the Cell processors used in Phase 2. It was built to full scale at IBM’s [Poughkeepsie, New York](/source/Poughkeepsie%2C_New_York) facility,[1] where it broke the 1 petaflops barrier during its fourth attempt on May 25, 2008. The complete system was moved to its permanent location in New Mexico in the summer of 2008.[1]

## Technical specifications

### Processors

Roadrunner used two different models of processors. The first is the **AMD [Opteron 2210](/source/Opteron)**, running at 1.8 GHz. Opterons are used both in the computational nodes feeding the Cells with useful data and in the system operations and communication nodes passing data between computing nodes and helping the operators running the system. Roadrunner has a total of 6,912 Opteron processors with 6,480 used for computation and 432 for operation. The Opterons are connected together by [HyperTransport](/source/HyperTransport) links. Each Opteron has two cores for a total 13,824 cores.

The second processor is the **IBM [PowerXCell 8i](/source/Cell_(microprocessor))**, running at 3.2 GHz. These processors have one general purpose core (PPE), and eight special performance cores (SPE) for [floating point](/source/Floating_point) operations. Roadrunner has a total of 12,960 PowerXCell processors, with 12,960 PPE cores and 103,680 SPE cores, for a total of 116,640 cores.

### TriBlade

A schematic description of the TriBlade module.

Logically, a **TriBlade** consists of two dual-core Opterons with 16 [GB](/source/Gigabyte) [RAM](/source/RAM) and four PowerXCell 8i CPUs with 16 GB Cell RAM.[10]

Physically, a TriBlade consists of one [LS21](/source/IBM_BladeCenter#LS21) [Opteron](/source/Opteron) [blade](/source/Blade_server), an expansion blade, and two [QS22](/source/IBM_BladeCenter#QS22) [Cell](/source/Cell_(microprocessor)) blades. The LS21 has two 1.8 GHz dual-core Opterons with 16 GB memory for the whole blade, providing 8GB for each CPU. Each QS22 has two PowerXCell 8i CPUs, running at 3.2 GHz and 8 GB memory, which makes 4 GB for each CPU. The expansion blade connects the two QS22 via four [PCIe](/source/PCIe) x8 links to the LS21, two links for each QS22. It also provides outside connectivity via an [InfiniBand](/source/InfiniBand) 4x DDR adapter. This makes a total width of four slots for a single TriBlade. Three TriBlades fit into one [BladeCenter H](/source/IBM_BladeCenter#IBM_BladeCenter_H) chassis. The expansion blade is connected to the Opteron blade via HyperTransport.

### Connected Unit (CU)

A **Connected Unit** is 60 BladeCenter H full of TriBlades, that is 180 TriBlades. All TriBlades are connected to a 288-port Voltaire ISR2012 Infiniband switch. Each CU also has access to the [Panasas](/source/Panasas) [file system](/source/File_system) through twelve [System x3755](/source/Lenovo_System_x) servers.[10]

CU system information:[10]

- 360 dual-core Opterons with 2.88 [TiB](/source/Tebibyte) [RAM](/source/RAM).

- 720 PowerXCell 8i cores with 2.88 TiB RAM.

- 12 System x3755 with dual 10-GBit Ethernet each.

- 288-port Voltaire ISR2012 switch with 192 Infiniband 4x DDR links (180 TriBlades and twelve I/O nodes).

### Roadrunner cluster

A schematic overview of the tiered composition of the Roadrunner supercomputer cluster.

The final [cluster](/source/Computer_cluster) is made up of 18 connected units, which are connected via eight additional (second-stage) Infiniband ISR2012 switches. Each CU is connected through twelve uplinks for each second-stage switch, which makes a total of 96 uplink connections.[10]

Overall system information:[10]

- 6,480 Opteron processors with 51.8 TiB RAM (in 3,240 LS21 blades)

- 12,960 Cell processors with 51.8 TiB RAM (in 6,480 QS22 blades)

- 216 System x3755 I/O nodes

- 26 288-port ISR2012 Infiniband 4x DDR switches

- 296 racks

- 2.345 [MW](/source/Megawatt) power[13]

## Shutdown

IBM Roadrunner was shut down on March 31, 2013.[13] While the supercomputer was one of the fastest in the world, its energy efficiency was relatively low. Roadrunner delivered 444 megaflops per watt vs the 886 megaflops per watt of a comparable supercomputer.[17] Before the supercomputer is dismantled, researchers will spend one month performing memory and data routing experiments that will aid in designing future supercomputers.[13]

After IBM Roadrunner is dismantled, the electronics will be shredded.[18] Los Alamos will perform the majority of the supercomputer's destruction, citing the classified nature of its calculations. Some of its parts will be retained for historical purposes.[18]

## See also

Wikimedia Commons has media related to [Roadrunner (supercomputer)](https://commons.wikimedia.org/wiki/Category:Roadrunner_(supercomputer)).

- [Central processing unit](/source/Central_processing_unit)

- [Computer architecture](/source/Computer_architecture)

- [Computer science](/source/Computer_science)

- [Computing](/source/Computing)

- [Multi-core processor](/source/Multi-core_processor)

## References

1. ^ [***a***](#cite_ref-ibmpress_1-0) [***b***](#cite_ref-ibmpress_1-1) [***c***](#cite_ref-ibmpress_1-2) ["Fact Sheet & Background: Roadrunner Smashes the Petaflop Barrier"](http://www-03.ibm.com/press/us/en/pressrelease/24405.wss). IBM. 9 June 2008. Retrieved April 1, 2013.{{[cite web](https://en.wikipedia.org/wiki/Template:Cite_web)}}: CS1 maint: deprecated archival service ([link](https://en.wikipedia.org/wiki/Category:CS1_maint:_deprecated_archival_service))

1. **[^](#cite_ref-2)** Gaudin, Sharon (2008-06-09). ["IBM's Roadrunner smashes 4-minute mile of supercomputing"](https://web.archive.org/web/20081224001155/http://www.computerworld.com/action/article.do?command=viewArticleBasic&taxonomyName=hardware&articleId=9095318&taxonomyId=12&intsrc=kc_top). *Computerworld*. Archived from [the original](http://www.computerworld.com/action/article.do?command=viewArticleBasic&taxonomyName=hardware&articleId=9095318&taxonomyId=12&intsrc=kc_top) on 2008-12-24. Retrieved 2008-06-10.

1. **[^](#cite_ref-3)** Fildes, Jonathan (2008-06-09). ["Supercomputer sets petaflop pace"](http://news.bbc.co.uk/2/hi/technology/7443557.stm). *BBC News*. [Archived](https://web.archive.org/web/20090104004433/http://news.bbc.co.uk/2/hi/technology/7443557.stm) from the original on 2009-01-04. Retrieved 2008-06-09.

1. **[^](#cite_ref-4)** ["TOP500 Supercomputer Sites"](http://www.top500.org/lists/2008/11). *top500.org*. 11 November 2008. [Archived](https://web.archive.org/web/20180906051921/https://www.top500.org/lists/2008/11/) from the original on 6 September 2018. Retrieved 22 June 2009.

1. **[^](#cite_ref-5)** ["The Green500 List — June 2009"](https://web.archive.org/web/20130512030333/http://www.green500.org/lists/green200906). *The Green500*. Archived from [the original](http://www.green500.org/lists/green200906) on 2013-05-12. Retrieved 2013-04-02.

1. **[^](#cite_ref-AP_6-0)** Montoya, Susan (30 March 2013). ["End of the Line for Roadrunner Supercomputer"](https://web.archive.org/web/20150402160104/http://bigstory.ap.org/article/end-line-roadrunner-supercomputer). The Associated Press. Archived from [the original](http://bigstory.ap.org/article/end-line-roadrunner-supercomputer) on 2 April 2015.

1. **[^](#cite_ref-7)** ["IBM to Build World's First Cell Broadband Engine Based Supercomputer"](https://web.archive.org/web/20070224122420/http://www-03.ibm.com/press/us/en/pressrelease/20210.wss) (Press release). IBM. 2006-09-06. Archived from [the original](http://www-03.ibm.com/press/us/en/pressrelease/20210.wss) on February 24, 2007. Retrieved 2008-05-31.

1. **[^](#cite_ref-8)** ["IBM Selected to Build New DOE Supercomputer"](https://web.archive.org/web/20080618162000/http://nnsa.energy.gov/news/1015.htm) (Press release). NNSA. 2006-09-06. Archived from [the original](http://nnsa.energy.gov/news/1015.htm) on 2008-06-18. Retrieved 2008-05-31.

1. **[^](#cite_ref-9)** ["International Supercomputing Conference to Host First Panel Discussion on Breaking the Petaflops Barrier"](https://web.archive.org/web/20081011055756/http://www.top500.org/blog/2008/06/09/international_supercomputing_conference_host_first_panel_discussion_breaking_petaflop_s_barrier). *TOP500 Supercomputing Sites*. 9 June 2008. Archived from [the original](http://www.top500.org/blog/2008/06/09/international_supercomputing_conference_host_first_panel_discussion_breaking_petaflop_s_barrier) on 11 October 2008. Retrieved 11 October 2015.

1. ^ [***a***](#cite_ref-OVERVIEW_10-0) [***b***](#cite_ref-OVERVIEW_10-1) [***c***](#cite_ref-OVERVIEW_10-2) [***d***](#cite_ref-OVERVIEW_10-3) [***e***](#cite_ref-OVERVIEW_10-4) [***f***](#cite_ref-OVERVIEW_10-5) Koch, Ken (2008-03-13). ["Roadrunner Platform Overview"](http://www.lanl.gov/orgs/hpc/roadrunner/pdfs/Koch%20-%20Roadrunner%20Overview/RR%20Seminar%20-%20System%20Overview.pdf) (PDF). Los Alamos National Laboratory. [Archived](https://web.archive.org/web/20131023065617/http://www.lanl.gov/orgs/hpc/roadrunner/pdfs/Koch%20-%20Roadrunner%20Overview/RR%20Seminar%20-%20System%20Overview.pdf) (PDF) from the original on 2013-10-23. Retrieved 2008-05-31.

1. **[^](#cite_ref-11)** Borrett, Ann (2007). ["Roadrunner - Integrated Hybrid Node"](https://web.archive.org/web/20081010180011/http://sti.cc.gatech.edu/SC07-BOF/06-Borrett.pdf) (PDF). Archived from [the original](http://sti.cc.gatech.edu/SC07-BOF/06-Borrett.pdf) (PDF) on 2008-10-10. Retrieved 2009-06-30.

1. **[^](#cite_ref-12)** Squyres, Jeff. ["Open MPI: 10^15 Flops Can't Be Wrong"](http://www.open-mpi.org/papers/sc-2008/jsquyres-cisco-booth-talk-1up.pdf) (PDF). Open MPI. [Archived](https://web.archive.org/web/20210223014051/https://www.open-mpi.org/papers/sc-2008/jsquyres-cisco-booth-talk-1up.pdf) (PDF) from the original on 2021-02-23. Retrieved 2008-11-22.

1. ^ [***a***](#cite_ref-Ars1_13-0) [***b***](#cite_ref-Ars1_13-1) [***c***](#cite_ref-Ars1_13-2) [***d***](#cite_ref-Ars1_13-3) [***e***](#cite_ref-Ars1_13-4) [***f***](#cite_ref-Ars1_13-5) Brodkin, Jon (31 March 2013). ["World's top supercomputer from '09 is now obsolete, will be dismantled"](https://arstechnica.com/information-technology/2013/03/worlds-fastest-supercomputer-from-09-is-now-obsolete-will-be-dismantled/). Ars Technica. [Archived](https://web.archive.org/web/20130331203506/http://arstechnica.com/information-technology/2013/03/worlds-fastest-supercomputer-from-09-is-now-obsolete-will-be-dismantled/) from the original on 31 March 2013. Retrieved March 31, 2013.

1. **[^](#cite_ref-14)** ["Los Alamos computer breaks petaflop barrier"](http://www.ibm.com/news/us/en/2008/06/09/d074803y96972a84.html). IBM. 2008-06-09. [Archived](https://web.archive.org/web/20110728190006/http://www.ibm.com/news/us/en/2008/06/09/d074803y96972a84.html) from the original on 2011-07-28. Retrieved 2008-06-12.

1. **[^](#cite_ref-BarkerDavis2008_15-0)** Barker, Kevin J.; Davis, Kei; Hoisie, Adolfy; Kerbyson, Darren J.; Lang, Mike; Pakin, Scott; Sancho, Jose C. (2008). ["Entering the petaflop era: The architecture and performance of Roadrunner"](https://web.archive.org/web/20110813133941/http://www.cse.psu.edu/~raghavan/cse598C/RoadrunnerSC08-CR-final.doc.pdf) (PDF). *2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis*. pp. 1–11. [doi](/source/Doi_(identifier)):[10.1109/SC.2008.5217926](https://doi.org/10.1109%2FSC.2008.5217926). [ISBN](/source/ISBN_(identifier)) [978-1-4244-2834-2](https://en.wikipedia.org/wiki/Special:BookSources/978-1-4244-2834-2). [S2CID](/source/S2CID_(identifier)) [7844349](https://api.semanticscholar.org/CorpusID:7844349). Archived from [the original](http://www.cse.psu.edu/~raghavan/cse598C/RoadrunnerSC08-CR-final.doc.pdf) (PDF) on 2011-08-13. Retrieved 2013-04-02.

1. **[^](#cite_ref-16)** Poole, Steve (September 2006). ["DarkHorse: a Proposed PetaScale Architecture"](http://archive.ll.mit.edu/HPEC/agendas/proc06/Day2/27_Poole_Panel.pdf) (PDF). *Los Alamos National Laboratory*. [Archived](https://web.archive.org/web/20211022141121/https://archive.ll.mit.edu/HPEC/agendas/proc06/Day2/27_Poole_Panel.pdf) (PDF) from the original on 22 October 2021. Retrieved 11 October 2015.

1. **[^](#cite_ref-2012top500_17-0)** ["Top500 List - November 2012"](https://web.archive.org/web/20130826135532/http://www.top500.org/list/2012/11/). [TOP500](/source/TOP500). Archived from [the original](http://top500.org/list/2012/11/) on August 26, 2013. Retrieved April 2, 2013.

1. ^ [***a***](#cite_ref-ars2_18-0) [***b***](#cite_ref-ars2_18-1) ["World's first petascale supercomputer will be shredded to bits"](https://arstechnica.com/information-technology/2013/04/worlds-first-petascale-supercomputer-will-be-shredded-to-bits/). Ars Technica. April 2013. [Archived](https://web.archive.org/web/20130403015810/http://arstechnica.com/information-technology/2013/04/worlds-first-petascale-supercomputer-will-be-shredded-to-bits/) from the original on April 3, 2013. Retrieved April 1, 2013.

## External links

- ["Los Alamos National Laboratory Roadrunner Home Page"](https://www.lanl.gov/about/mission/advanced-simulation-and-computing/platforms). Los Alamos National Laboratory. 2007-03-30. Retrieved 2008-05-31.

- ["In Pictures: A look inside what may be the world's fastest supercomputer"](https://web.archive.org/web/20120612020928/http://www.computerworld.com/s/article/9085021/In_Pictures_A_look_inside_what_may_be_the_world_s_fastest_supercomputer). Computerworld. 2008-05-13. Archived from [the original](http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9085021) on 2012-06-12. Retrieved 2008-05-31.

- [World's Fastest Computer](https://web.archive.org/web/20090123050510/http://www.worldsmostunique.com/worldsfastestcomputer.php)

Records Preceded by Blue Gene/L 478.20 teraflops World's most powerful supercomputer June 2008 – November 2009 Succeeded by Jaguar 1.75 petaflops

---
Adapted from the Wikipedia article [Roadrunner (supercomputer)](https://en.wikipedia.org/wiki/Roadrunner_(supercomputer)) by Wikipedia contributors ([contributor history](https://en.wikipedia.org/wiki/Roadrunner_(supercomputer)?action=history)). Available under [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/). Changes may have been made.