SCEC Award Number 21051 View PDF
Proposal Category Individual Proposal (Integration and Theory)
Proposal Title Statistical evaluation of CSEP Forecasts
Investigator(s)
Name Organization
Frederic Schoenberg University of California, Los Angeles
Other Participants
SCEC Priorities 5c, 5d, 5a SCEC Groups EFP, CS
Report Due Date 03/15/2022 Date Report Submitted 03/15/2022
Project Abstract
 This research focused on the assessment of models for forecasting seismicity as part of the Collaboratory for the Study of Earthquake Predictability (CSEP). We used modern methods and recent advances in residual analysis and model assessment for space-time point process models to compare the fit of competing models, including notably ETAS and STEP models that are used for earthquake hazard assessment. We used superthinned residuals and Voronoi deviance residuals in order to ensure that model components are correctly specified and properly incorporated into these seismicity models. While previous research suggested the superior fit of ETAS, our results found surprisingly good fit of the STEP forecasts, which slightly outperformed ETAS from 2013-2017 according to most metrics. We also augmented our publicly available R packages stppResid and nphawkes to enable fitting, simulating, and assessing models for large point process data sets, including the fitting of triggering functions nonparametrically and the efficent automatic assessment of such models and forecasts using Voronoi deviance residuals and superthinning. These model fitting and residual analysis tools will be useful in assessing models for Southern California and for global seismicity, in conjunction with the CSEP and WGCEP programs, and will also be useful for assessing the fit of earthquake simulation models.
Intellectual Merit This research directly assists SCEC's mission toward probabilistic seismic hazard analysis (item 3 in SCEC5 Thematic Areas), as the methods here are directly useful for assessing, comparing and improving probabilistic models for earthquake occurrences, especially ETAS and STEP.

In particular, our implementation of Voronoi deviance residuals and super-thinned residuals and their application to the assessment of ETAS and STEP as implemented in CSEP has suggested important areas for improving these state-of-the-art models for earthquake forecasting. As a statistician rather than a seismologist, I have been been able to give a fair assessment of these types of models without bias toward one model or the other. I have been working closely with Maximilian Werner, Bill Savran, Danijel Schorlemmer, David Jackson and others involved with CSEP and WGCEP, and will continue to do so in order to explain how the residual methods explored here may readily be used to assess models in these earthquake forecasting projects. The improved assessment techniques and more accurate forecasts of seismicity resulting from this research will also further our understanding of seismicity and the mechanisms for its generation.

In addition, addressing SCEC5's research priorities P5.c, P5.d, and P5.a, the tools described in this research, especially Voronoi and superthinned residuals, can be used to help calibrate and detect departures from fit for earthquake simulation models such as Stanford's QCN model and data and to suggest areas where such models can be improved, and in particular the R code produced in this research can be used to assist other SCEC researchers in performing this analysis.
Broader Impacts The research in this project augments methods for assessing seismicity forecasts and also contributes to our understanding of which seismicity models currently perform best. These improvements will lead to improved forecasts of seismicity, which may save lives, prevent catastrophic financial losses, and aid in safe urban planning, and in addition will increase our understanding of earthquakes and their causes and clustering features. I have been working closely with members of CSEP and will continue to do so in order to help incorporate the residual methods explored here for their earthquake forecasting projects. In addition, this project aided the research toward the Masters degrees in Statistics for 2 UCLA graduate students (Joshua Ward and Zhe Zhang).
Exemplary Figure Figure 1 from the project report.
Top row: histogram of raw Voronoi Residuals for STEP (left) and ETAS (right). Bottom row: Spatial plot of Voronoi residuals for STEP (left) and ETAS (right). Blue indicates better fit.