Imagine being a policymaker at the beginning of the COVID-19 pandemic. You have to decide which actions to recommend, how much risk to tolerate and what sacrifices to ask your citizens to bear.
Who would you turn to for an accurate prediction about how people would react? Many would recommend going to the experts — social scientists. But we are here to tell you this would be bad advice.
As psychological scientists with decades of combined experience studying decision-making, wisdom, expert judgment and societal change, we hoped social scientists’ predictions would be accurate and useful. But we also had our doubts.
Our discipline has been undergoing a crisis due to failed study replications and questionable research practices. If basic findings can’t be reproduced in controlled experiments, how confident can we be that our theories can explain complex real-world outcomes?
To find out how well social scientists could predict societal change, we ran the largest forecasting initiative in our field’s history using predictions about change in the first year of the COVID-19 pandemic as a test case.
To do this, we tested how well social scientists could predict societal change in two ways. First, we asked social scientists for quick guesses about how things would change over the next two years of the pandemic.
Second, we ran a competition where over 100 teams of social scientists with access to historical data made month-by-month forecasts. We formally assessed their predictions for a range of social sciences phenomena, including changes in prejudice, subjective well-being, violence, individualism and political polarisation between May 2020 and May 2021.
Our findings, detailed in peer-reviewed papers in Nature Human Behaviour and in American Psychologist, paint a sobering picture. Despite the causal nature of most theories in the social sciences, and the fields’ emphasis on prediction in controlled settings, social scientists’ forecasts were generally not very good.
In both papers, we found that experts’ predictions were generally no more accurate than those made by samples of the general public. Further, their predictions were often worse than predictions generated by simple statistical models.
Our studies did still give us reasons to be optimistic. First, forecasts were more accurate when teams had specific expertise in the domain they were making predictions in. If someone was an expert in depression, for example, they were better at predicting societal trends in depression.
Second, when teams were made up of scientists from different fields working together, they tended to do better at forecasting. Finally, teams that used simpler models to generate their predictions and made use of past data generally outperformed those that didn’t.
These findings suggest that, despite the poor performance of the social scientists in our studies, there are steps scientists can take to improve their accuracy at this type of forecasting.
Our research also found that, compared to lay people, social scientists were more aware of the herculean nature of the task at hand. In our studies, they expressed uncertainty and less confidence than lay people when making forecasts.
…despite the poor performance of the social scientists in our studies, there are steps scientists can take to improve their accuracy at this type of forecasting.
Similarly, social scientists expressed uncertainty in their open-ended predictions for the World after COVID project, a video series we conducted with eminent scholars in the first year of the pandemic.
Thus, social scientists still have some wisdom to offer, reminding us of the uncertainty and the need for humility when forecasting the future.
Our work highlights the importance of developing reliable sources of data and suggests strategies that can improve the accuracy of such forecasts.
These results are a call to action for the scientific community to continue developing better methods for predicting societal change so the public can rely on scientists in times of crisis.
Our projects show that expert prediction of societal change during the COVID-19 pandemic was far from perfect. But they also suggest ways such predictions can be improved. By drawing on specific expertise, collaborating across disciplines and making data-driven models, social scientists can produce more accurate and useful forecasts for policymakers and the public.
The scientific community should strive to develop better methods for predicting societal change, while acknowledging the uncertainty and complexity involved. Policymakers should appreciate the value of expert insight, but also be aware of its limitations and potential biases. If we want to predict the future, or shape it for that matter, then a bit of humility would likely help.
This article was co-authored by Igor Grossmann, Professor of Psychology, University of Waterloo; and Michael Varnum, Associate Professor of Psychology, Arizona State University.
This articled was originally published on The Conversation.
Photo by Gabriella Clare Marino on Unsplash.
