Consider the following situation. You are the head of a large online fashion retailer. You observe that a larger percentage of your sales happen through your mobile app rather than your desktop website. It is fairly expensive to maintain both. Should you go app-only? What will your annual revenues be if you take the plunge?

This is a fairly complex decision to make. Crores of rupees are possibly at stake. You hope that your decision will not reduce your sales revenues or volumes in any way. But nobody has made such a decision before, and data about such an action is not available. So you talk to your experts – statisticians, economists, consultants, designers, etc. They each have some educated guesses, which are very different from one another. How do you go forward? Is it possible to arrive at a collective forecast, and hence a decision based on it, without stepping on egos?

Unsurprisingly, the US Army faced such problems throughout its history, especially during the Cold War. Imagine a contentious question like “How many soldiers could we lose if we invade Cuba?” at the height of the Cold War, and managing a group of experts from the Army, Air Force, Navy, CIA, NSA, etc. Each of these experts is likely to have a very different take on the situation, and fragile egos could lead to a complete breakdown of discussions. Furthermore, timid people or junior experts, whose inputs may otherwise be valuable, could be dissuaded from speaking up, for fear of displeasing their bosses. Keeping such considerations in mind, the RAND Corporation (a military and public policy think tank) developed a crowdsourcing technique called the Delphi method.

Named after the Oracle of Delphi, the Delphi method involves a group of experts and a moderator who asks a question. The experts anonymously answer these questions based on whatever rationale they may have, including intuition. The moderator then collates the responses and presents summary statistics of these collective estimates. Experts are then invited to revise their estimate based on collective statistics presented in the previous round. Such an exercise is repeated till the moderator is satisfied that there is some sort of group consensus. The idea is that even experts are bound to revise their opinions, given sufficient inputs from other experts, and anonymity.

Norman Dalkey and Olaf Helmer, co-creators of the Delphi method (Theodore Gordon is credited as the third co-creator), provide details of an experiment at the RAND Corporation. They assembled a team of consisting of four economists, one physical vulnerability specialist, one systems analyst, and one electronics engineer, to determine the answer to a very pertinent question – How many 20 kiloton atom bombs will the USSR require to reduce US ammunitions output to a quarter of its current level? Sufficient data was provided to each expert, along with production budgets, current US production numbers, locations of targets etc. (a Delphi method is most successful when the question is unambiguously framed, and sufficient data is provided – the actual question in the research paper is much more detailed than the one presented here).

While initial estimates of the experts varied wildly between 50 and 5,000 bombs, five iterations of this method considerably reduced this range between 167 and 360, with the median going up from 200 in round 1 to 276 in round 5. Of course, such a method is not perfect, discussions between experts could have compromised anonymity (there was a gap of a week between each round) and experts could have possibly agreed on a completely wrong estimate.

A further study by marketing researchers Marvin Jolson and Gerald Rossow used Delphi forecasting for a problem whose answer was already known, to test its efficacy. They asked an 11-member panel of the Pace Computing Corporation in Arlington, Virginia, to estimate demand for Pace’s courses in New York. Rather than ask for an estimate, experts estimated their subjective probabilities for different levels of demand. Their estimates narrowed sufficiently close to the real known answers within three rounds.

Today, the Delphi method is used for forecasting scenarios in many situations. For example, the audience poll option in Kaun Banega Crorepati is a one-round Delphi exercise. Companies and policy agencies across the world use it as an aid for high level decision making. At worst, it encourages structured thinking and group consensus.

So the next time you are faced with a contentious and ambiguous decision like going app-only, banning a cultural tradition, or invalidating currency notes, remember. A Delphi forecast with an expert panel may just be more helpful than your individual hunch.

 

---