An innovative collaboration between computer and computational scientists from the Data Science and Learning division (DSL) and Advanced Photon Source (APS) at Argonne and the Electrical and Computer Engineering department at the University of Illinois at Urbana-Champaign (UIUC) garnered the Best Paper Award at SC20.

The research team — led by Tekin Bicer of DSL and PhD candidate Mert Hidayetoglu (UIUC) and comprising Simon Garcia de Gonzalo (Barcelona Supercomputing Center), Bin Ren (College of William & Mary), Vincent De Andrade (APS), Doga Gursoy (APS), Rajkumar Kettimuthu (DSL), Ian Foster (DSL) and Wen-mei Hwu (UIUC) — is focused on using a high-performance, iterative reconstruction system for noninvasive imaging at synchrotron facilities. The new design, described in the team’s winning paper ​“Petascale XCT: 3D Image Reconstruction with Hierarchical Communications on Multi-GPU Nodes,” can be used for terabyte(s)-scale 3D volumes and is not constrained by computational requirements.

The work involves the following three novel optimizations: (1) extends the 2D memory-centric approach to 3D; (2) includes hierarchical communications by exploiting ​“fat-node” architecture with many GPUs; and (3) employs mixed-precision types while preserving convergence rate and quality.

The team extensively evaluated the proposed optimizations and scaling on the Summit supercomputer at Oak Ridge National Laboratory. Their largest reconstruction is a mouse brain volume with 9K×11K×11K voxels, where the total reconstruction time is under 3 minutes using 24,576 GPUs and reaching 65 PFLOPS, which is 34% of Summit’s peak performance.