Contributed by Marie-Amélie Boucher, Maria-Helena Ramos and Ioanna Zalachori
It is often assumed that probabilistic forecasts should lead to better water and risk management through increased benefits (economic or not) to users in their decision-making processes.
Most often, this assumption arises from studies based on evaluations of forecast quality, which propose comparisons of performance between deterministic and probabilistic (or, for example, ensemble) forecasts using metrics such as the CRPS and the MAE to support their conclusions.
But, really, does quality (a ‘good’ average statistics like the CRPS) automatically translates into value? Assuming that from the water manager’s point of view, the question of value really is the most important one, after all, if no gain ($$$) is to be anticipated, why should one change a deterministic forecasting system for a probabilistic one?
A famous quote from Murphy in his 1993 paper, ‘What is a good forecast? An essay on the nature of goodness in weather forecasting’, reads that ‘… forecasts possess no intrinsic value. They acquire value through their ability to inﬂuence the decisions made by users of the forecasts’. It seems the question then is to measure ‘this ability’, which basically means that first we have to be able to ‘evaluate decisions’.
We once asked a decision-maker responsible for deciding on whether or not to open a control gate of a dam, and of how many meters it should be opened in case the decision was to open it, how he knew afterwards that his decision was the ‘best decision’. The answer we got was (not ipsis verbis): ‘It is the best decision: we take the best decision given the forecasts we receive and other complementary information we have on the situation. If the result is not good, the problem is not in the decision, but in the forecasts, which were not good’.
Concretely, we understood that the evaluation of ‘good’ or ‘bad’ decisions is not as straightforward as it appears even for decision-makers (although most probably the consequences of ‘bad decisions’ may come back somehow to decision-makers or to the forecasters if human or high economic losses are observed).
Still, investigating the forecast quality/value relationship can be useful to at least throw lights on the expected benefits of research and operational studies seeking to improve the quality of hydrometeorological forecasts: if improvements in quality translate into improvements in value or, generally speaking, in forecast utility, one may justify to continue seeking for improved forecast quality.
Now, we come back to the questions: how to measure forecast value in hydrology? How to link it to forecast quality?
There have been some interesting experiments regarding the value of hydrological forecasts. Typically, they use concurrent forecasting systems successively as inputs in a decision making model and evaluate the respective gains.
The most popular decision-making model involved in forecasts value comparisons is the one based on the ‘cost-loss ratio’ (C/L). This ratio is strongly user-subjective and depends on the user’s risk tolerance. In hydrologic forecasting, for instance, the study by Roulin (2007) used a cost-loss decision making model to evaluate the value of 1 to 10-days streamflow forecasts, using various C/L ratios and probability thresholds for decision-making. They demonstrated, for two Belgian catchments, that using ensemble rather than deterministic forecasts translates into monetary gains. McCollor and Stull (2008), Van de Bergh and Roulin (2010), Muluye (2011) and Verkade and Werner (2011) also used a cost-loss ratio to compare ensemble and deterministic forecasts value.
A number of studies also investigated the potential added value of probabilistic forecasts using numerical optimization techniques for reservoir management decisions. Kim et al. (2007) used stochastic dynamic programming to compare the value of ensemble and deterministic forecasts for a Korean catchment (the ensemble won the comparison). Weijs (2011), using a similar type of approach, showed that the gain is greater for short lead times and high reservoir levels. Boucher et al. (2011), in a case study on the Gatineau catchment in Quebec, found no significant added value in raw streamflow ensemble forecasts (compared to deterministic forecasts), although the situation was reversed with the use of statistical post-processing of ensemble forecasts (Boucher et al., 2012).
These studies and approaches share a common goal: assess the value of hydrological forecasts obtained from various meteorological forecasts (multi-model or not). The end-users’ interests, however, are quite diverse: hydropower, urban water supply and flood prevention.
In general, what is particularly interesting with hydrologic forecast value studies is that, while ensemble forecasts are often considered to be useful for the decision-making process, it is not always the ones associated with the highest forecast value. After numerous demonstrations of how the quality of ensemble forecasts is superior to the quality of deterministic forecasts, it can be shocking (and deceiving!) to realize that quality improvements do not always correspond so straightforwardly to higher monetary value.
Certainly, conclusions on the economic value of hydrological ensemble forecasts are strongly dependent on several issues (e.g., the user’s decision model and objectives, the forecast lead time of interest), which include the methodological approach itself used to evaluate it. So, even though ensemble forecasts show to be better forecasts, sometimes the most important factor for them to be of higher value is the decision-making model itself! Its limits can largely affect the assessment of the added value of some forecasting system over another.
Maybe to convince potential users of ensemble forecasts that changing their practices is worth the trouble, we must demonstrate that doing so can make them richer! … Or maybe ‘less poor’, if viewed from a flood protection point of view.