Liemohn M. W., J. P. McCollough, V. K. Jordanova, C. M. Ngwira, S. K. Morley, C. Cid, W. K. Tobiska, P. Wintoft, N. Y. Ganushkina, D. T. Welling, S. Bingham, M. A. Balikhin, H. J. Opgenoorth, M. A. Engel, R. S. Weigel, H. J. Singer, D. Buresova, S. Bruinsma, I. S. Zhelavskaya, Y. Y. Shprits, R. Vasile, (2018), Model Evaluation Guidelines for Geomagnetic Index Predictions, Space Weather, 16, 2079-2102, doi:10.1029/2018SW002067

Abstract

Abstract Geomagnetic indices are convenient quantities that distill the complicated physics of some region or aspect of near-Earth space into a single parameter. Most of the best-known indices are calculated from ground-based magnetometer data sets, such as Dst, SYM-H, Kp, AE, AL, and PC. Many models have been created that predict the values of these indices, often using solar wind measurements upstream from Earth as the input variables to the calculation. This document reviews the current state of models that predict geomagnetic indices and the methods used to assess their ability to reproduce the target index time series. These existing methods are synthesized into a baseline collection of metrics for benchmarking a new or updated geomagnetic index prediction model. These methods fall into two categories: (1) fit performance metrics such as root-mean-square error and mean absolute error that are applied to a time series comparison of model output and observations and (2) event detection performance metrics such as Heidke Skill Score and probability of detection that are derived from a contingency table that compares model and observation values exceeding (or not) a threshold value. A few examples of codes being used with this set of metrics are presented, and other aspects of metrics assessment best practices, limitations, and uncertainties are discussed, including several caveats to consider when using geomagnetic indices.

Authors (sorted by name)

Balikhin Bingham Bruinsma Buresova Cid Engel Ganushkina Jordanova Liemohn McCollough Morley Ngwira Opgenoorth Shprits Singer Tobiska Vasile Weigel Welling Wintoft Zhelavskaya

Journal / Conference

Space Weather

Acknowledgments

This paper is the product of the Geomagnetic Indices Working Group of the International CCMC‐LWS Working Meeting on Space Weather Metrics. The authors would like to thank the organizers of the workshop for their time and effort to rally the community into action on devising assessment standards for space weather models. We would also like to thank others that contributed but declined authorship, specifically Lutz Rastätter and Leila Mays at NASA and Joshua Rigler at USGS. The projects leading to these results have received funding from the European Union Seventh Framework Programme (FP7/2007‐2013) under grant agreement 606716 SPACESTORM and from the European Union's Horizon 2020 research and innovation program under grant agreement 637302 PROGRESS. Work in the United States was conducted under Work at the University of Michigan and was supported by NASA grants NNX14AF34G, NNX17AI48G, NNX17AB87G, 80NSSC17K0015, and NNX14AC02G, and NSF grant 1663770. The Catholic University of America effort was performed under the CUA‐NASA Cooperative Agreement supported by NASA grant NNG11PL10A 670.135. Funding at the University of Sheffield was provided by STFC UK grant ST/R000697/1. The work done at the University of Alcala was supported by grant from MINECO AYA2016‐80881‐P. S. K. M. acknowledges support from the U.S. Department of Energy's Laboratory Directed Research and Development program (grant 20170047DR). The work at GFZ Potsdam was supported by Geo. X; the Research Network for Geosciences in Berlin and Potsdam, under grant SO_087_GeoX; and by the European Union's Horizon 2020 research and innovation program under grant agreement 776287 SWAMI. Work at Los Alamos was supported through the Laboratory Directed Research and Development program by the U.S. Department of Energy under contract DE‐AC52‐06NA25396. Work at the Institute of Atmospheric Physics was supported by the H2020 COMPET‐2017 TechTIDE Project (776011). Work at IRF‐Lund was supported by ESA contract SSA‐SWE‐P2‐1.5. Data used in the metrics assessments in this paper were obtained from the Space Physics Data Facility at http://cdaweb.gsfc.nasa.gov/, Supermag at http://supermag.jhuapl.edu/, and WDC‐Kyoto at http://wdc.kugi.kyoto‐u.ac.jp/. The model output and the code used to create the figures and calculate the metrics are available at the University of Michigan Deep Blue Data repository (https://deepblue.lib.umich.edu/data/?locale=en). Specifically, the data are available at https://doi.org/10.7302/Z25T3HQC.

Grants

1663770 637302 NNX17AI48G

Bibtex

@article{doi:10.1029/2018SW002067,
author = {Liemohn, Michael W. and McCollough, James P. and Jordanova, Vania K. and Ngwira, Chigomezyo M. and Morley, Steven K. and Cid, Consuelo and Tobiska, W. Kent and Wintoft, Peter and Ganushkina, Natalia Yu. and Welling, Daniel T. and Bingham, Suzy and Balikhin, Michael A. and Opgenoorth, Hermann J. and Engel, Miles A. and Weigel, Robert S. and Singer, Howard J. and Buresova, Dalia and Bruinsma, Sean and Zhelavskaya, Irina S. and Shprits, Yuri Y. and Vasile, Ruggero},
title = {Model Evaluation Guidelines for Geomagnetic Index Predictions},
journal = {Space Weather},
volume = {16},
number = {12},
pages = {2079-2102},
keywords = {space weather, geomagnetic indices, metrics, statistical analysis, forecasting, ROC curve},
doi = {10.1029/2018SW002067},
url = {https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2018SW002067},
eprint = {https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2018SW002067},
abstract = {Abstract Geomagnetic indices are convenient quantities that distill the complicated physics of some region or aspect of near-Earth space into a single parameter. Most of the best-known indices are calculated from ground-based magnetometer data sets, such as Dst, SYM-H, Kp, AE, AL, and PC. Many models have been created that predict the values of these indices, often using solar wind measurements upstream from Earth as the input variables to the calculation. This document reviews the current state of models that predict geomagnetic indices and the methods used to assess their ability to reproduce the target index time series. These existing methods are synthesized into a baseline collection of metrics for benchmarking a new or updated geomagnetic index prediction model. These methods fall into two categories: (1) fit performance metrics such as root-mean-square error and mean absolute error that are applied to a time series comparison of model output and observations and (2) event detection performance metrics such as Heidke Skill Score and probability of detection that are derived from a contingency table that compares model and observation values exceeding (or not) a threshold value. A few examples of codes being used with this set of metrics are presented, and other aspects of metrics assessment best practices, limitations, and uncertainties are discussed, including several caveats to consider when using geomagnetic indices.},
year = {2018}
}