Contributed by Massimiliano Zappa and Kaethi Liechti
The wisdom of the crowd is “the process of taking into account the collective opinion of a group of individuals rather than a single expert to answer a question.
A large group’s aggregated answers to questions involving quantity estimation, general world knowledge, and spatial reasoning has generally been found to be as good as, and often better than, the answer given by any of the individuals within the group.” (Source: wikipedia).
The concept of the wisdom of the crowds was first established by Francis Galton, who published his findings about an ox and the crowd in Nature (Galton F. 1907. Vox populi. Nature 75: 450-451):
His paper was recently revisited by K. Wallis in a study where the author states that Galton’s forecasting competition was a precursors of two developments in statistical forecasting: ‘forecast combination’ and ‘two-piece distribution for symmetry description’.
Another nice example of the wisdom of the crowds is the audience assistance in the “who wants to be a millionaire” show.
The questions are from all domains and ranges of difficulties, and the poll is drawn from a random group of people who happen to spend a weekday afternoon in a TV studio. Even if only a few persons know the right answer, this will stick out, because the answers of the persons who have no idea will distribute about normally over all the answers. In about 91 % the audience gets the right answer. Although not always:
In hydrological ensemble prediction
The same concept can be translated into the world of hydrological ensemble prediction: an ensemble forecast can be seen as a crowd of members. Does a crowd of ensemble members have more wisdom than a single valued forecast?The Peak Box Game
The Peak-Box Game was played at the session “Ensemble hydro-meteorological forecasting” during the EGU Assembly in Vienna, on May 1st 2014 (see abstract here) and at the 10-year HEPEX workshop in Maryland last June.
It was conceived to offer an opportunity to play with the “Peak Box” approach for supporting interpretation and verification of operational ensemble peak-flow forecasts, proposed by Zappa and colleagues, and to discuss the use of ensemble predictions in operational hydrology.
The Peak-Box defines the “best estimate” of a flood event’s timing and magnitude by framing the discharge peaks of all members of an ensemble forecast and taking their median in timing and magnitude.
The game is simple: when looking at the evolution in time of an ensemble prediction of streamflows with N members, one has to ‘guess’ how big the observed peak discharge will be and at what time it will occur. With the help of a worksheet, every person writes down the coordinates of her/his estimated peak, following a re-established coordinate system:
At the end, the observed flows are given as well as the estimates made by the Peak-Box approach (Zappa et al., 2013). Everybody then compares their guesses with the location of the observed peak and the Peak-Box estimate.
Peak Box Game – The results
The four following figures show the results of the four forecast days that were played at the HEPEX workshop, together with the results from other applications of the game (at EGU, AWEL (end user), and ETH). The observed highest discharge is also indicated (red circle). The intersections of the blue rectangles are used to capture the Peak Box center.
- the observed maximum peak (Observation),
- the median of the experts’ guess (Median out of 162 Experts), and
- the Peak Box Center (Median of HEPS with 16 members).
The yellow columns give the total error (Manhatten distance) of the experts’ guess and the Peak Box guess (against the observed coordinates). We can see that the medium errors of the Peak Box guess are lower than or equal to the medium errors of the experts in all forecast days. The most important lesson, however, is, that the collective guess of the crowds of experts and HEPS members is very good compared to most “deterministic” guesses of single experts and HEPS members. So, to put it short: trust in your ensembles (or hire 200 experts…)!
This is only one example, but it shows that the Peak Box estimate can be useful as additional information extracted from the ensemble prediction to help in the analysis of forecast events. It is also an interesting tool to train forecasters and better assess the value of ensemble forecasts in flood prediction.
If you are interested in using the Peak Box game, it is available for download in the Resources Page of the Hepex website.