Forecast quality assessment
Forecast quality assessment is one of the steps in data post-processing, involving the systematic comparison of past predictions with observations. This assessment aims to inform the users on the quality of the predictions.
What is forecast quality assessment?
Forecast quality assessment is a necessary step in post-processing of climate data, in order to assess the quality of predictions. It is carried out by comparing forecasts of past periods with observations for that period to determine how well the predictions match the observed conditions.
Informing the user of the forecast quality is important for determining the added value of these forecasts.
How do we do this?
Our team conducts a systematic comparison of past predictions and observations for long periods of time, for example 20-30 years, and then uses these to evaluate the quality of the predictions or products provided to the user. Reanalysis data are typically used to perform this comparison.
The forecast quality assessment is conducted in a similar manner for all temporal scales (e.g. sub-seasonal, seasonal and decadal forecasts), with small differences in how the calculations are performed.
By conducting this assessment, we aim to provide users information on the forecast skill, which is obtained based on different verification metrics, and the forecast reliability.
- Verification metrics (also known as skill scores) are scores used to indicate the quality (skill) of a specific forecast product that is provided to users. Depending on the aspect of the predictions that we want to evaluate, deterministic or probabilistic skill scores can be selected. Some examples include the ensemble mean correlation, ranked probability skill score, continuous ranked probability skill score and Brier skill score, among others.
- A reliability diagram is used to show how reliable the forecast is and if the user can rely on the forecast to make decisions. This type of diagram can show the users how often the conditions were below normal / normal / above normal at a specific location in the past, and how often the forecast could have predicted this, indicating if the prediction can be trusted.
Our post-processing techniques aim to increase the reliability of the predictions over specific regions, which can be affected by poor availability or uncertainty of observations, and ultimately improve the forecast quality.