Exciting news! We're transitioning to the Statewide California Earthquake Center. Our new website is under construction, but we'll continue using this website for SCEC business in the meantime. We're also archiving the Southern Center site to preserve its rich history. A new and improved platform is coming soon!

Using Open-Science Workflow Tools to Produce SCEC CyberShake Physics-Based Probabilistic Seismic Hazard Models

Scott Callaghan, Philip J. Maechling, Fabio Silva, Mei-Hui Su, Kevin R. Milner, Robert W. Graves, Kim B. Olsen, Yifeng Cui, Karan Vahi, Ewa Deelman, Christine A. Goulet, Thomas H. Jordan, & Yehuda Ben-Zion

In Preparation December 20, 2023, SCEC Contribution #13382

The Statewide (formerly Southern) California Earthquake Center (SCEC) conducts multidisciplinary earthquake system science research that aims to develop predictive models of earthquake processes, and to produce accurate seismic hazard information that can improve societal preparedness and resiliency to earthquake hazards. As part of this program, SCEC has developed the CyberShake platform, which calculates physics-based probabilistic seismic hazard analysis (PSHA) models for regions with high-quality seismic velocity and fault models. PSHA hazard models provide estimates of possible future peak ground motions for sites of interest that are useful for building engineering, insurance rates, and disaster planning. CyberShake is a sophisticated computational pipeline that includes over 15 individual codes written by 6 developers. These codes are heterogeneous, ranging from short-running high-throughput serial CPU codes to large parallel GPU codes. Additionally, CyberShake simulation campaigns are computationally extensive, typically producing tens of terabytes of meaningful scientific data and metadata over several months of around-the-clock execution on leadership class supercomputers. We present the workflow software stack required to support CyberShake campaigns, including open-source workflow tools and custom solutions. We identify how the CyberShake platform and supporting tools enable us to meet a variety of challenges that come with large-scale simulations, such as automated remote job submission, data management, and verification and validation. This platform enabled us to perform our most recent campaign, CyberShake Study 22.12, from December 2022 to April 2023. During this time, our workflow tools executed approximately 28,000 jobs, and used about 770,000 node-hours on the Summit system at Oak Ridge Leadership Computing Facility. At peak, we utilized 73% of the system. Our workflow tools managed about 2.5 PB of total data, and automatically staged 19 million output files totaling 74 TB back to archival storage on the University of Southern California Center for Advanced Research Computing systems. The simulation data generated by the CyberShake platform is being used by seismological, engineering, and governmental communities.

Key Words
Keywords: scientific workflows, probabilistic seismic hazard analysis, high performance computing, seismic simulations, distributed computing

Citation
Callaghan, S., Maechling, P. J., Silva, F., Su, M., Milner, K. R., Graves, R. W., Olsen, K. B., Cui, Y., Vahi, K., Deelman, E., Goulet, C. A., Jordan, T. H., & Ben-Zion, Y. (2023). Using Open-Science Workflow Tools to Produce SCEC CyberShake Physics-Based Probabilistic Seismic Hazard Models. Frontiers in High Performance Computing, (in preparation).


Related Projects & Working Groups
CyberShake, Community Models, Seismology, Ground motion, Community models, Engineering Interface