“Good” forecasts of hail can be determined in multiple ways and must depend on both the performance of the guidance and the perspective of the end-user. This work, led by AER, looks at different verification strategies to capture the performance of the joint AER- and NSSL-developed CAM-HAILCAST hail forecasting model across three years of the NOAA Hazardous Weather Testbed Spring Forecasting Experiment (SFE) in different parent models. As part of this evaluation, NOAA Hazardous Weather Testbed Spring Forecasting Experiment (HWT SFE) participants were polled about their definition of a “good” hail forecast. Participants were presented with two different verification methods conducted over three different spatiotemporal scales, and they were then asked to subjectively evaluate the hail forecast as well as the different verification methods themselves. Results recommended use of multiple verification methods tailored to the type of forecast expected by the end-user interpreting and applying the forecast. The hail forecasts evaluated during this period included an implementation of CAM-HAILCAST in the Limited Area Model of the Unified Forecast System with the Finite Volume 3 (FV3) dynamical core. Evaluation of FV3-HAILCAST over both 1- and 24-h periods found continued improvement from 2019 to 2021. The improvement was largely a result of wide intervariability among FV3 ensemble members with different microphysics parameterizations in 2019 lessening significantly during 2020 and 2021. Overprediction throughout the diurnal cycle also lessened by 2021. A combination of both upscaling neighborhood verification and an object-based technique that only retained matched convective objects was necessary to understand the improvement, agreeing with the HWT SFE participants’ recommendations for multiple verification methods.
Figure 1: Performance diagrams showing the skill of several different CAM-HAILCAST versions at the (a) 2019, (b) 2020, and (c) 2021 NOAA HWT SFE at forecasting the occurrence of 1.5-in hail over (stars) a 24-h period or (circles) 1-h period. Skill was determined using an object-based technique that only retains matched simulated and radar-estimated hail swath objects. A perfect score would be in the upper right-hand corner.
Citation: Just What Is “Good”? Musings on Hail Forecast Verification through Evaluation of FV3-HAILCAST Hail Forecasts
R. D. Adams-Selin, C. Kalb, T. Jensen, J. Henderson, T. Supinie, L. Harris, Y. Wang
Weather and Forecasting, 38, 371-387, 2023.
https://doi.org/10.1175/WAF-D-22-0087.1