Lessons from Post-Processing Climate Data on Modern Flash-Based HPC Systems
Student: Adnan Haider (Illinois Institute of Technology)
Supervisor: Sheri Mickelson (National Center for Atmospheric Research)
Abstract: Flash devices are a plausible solution to accelerate I/O bound applications. However, the tradeoffs associated with different flash architectures is unclear. We quantitatively assess two modern flash architectures, Gordon (local) and Wrangler (pooled), to facilitate correct matching between I/O workloads and flash storage architectures. We analyze performance using post-processing climate data applications, which have a diverse set of I/O workloads and are integral for climate science discovery. We learned three main concepts. First, we found that an incorrect matching between storage architecture and I/O workload can hide the benefits of flash by increasing runtime by 2x. Second, after tuning Gordon's architecture, we found that a local flash architecture could be a cost-effective alternative to a pooled architecture if scalability and interconnect bottlenecks are alleviated. Third, the benefits of running post-processing applications on the latest data-intensive systems which lack flash devices can provide significant speedups, lessening the need for flash.
Two-page extended abstract: pdf