If you have not heard yet, a really important election will be held on Tuesday for the presidency of the United States. Americans are evenly divided this year between Kamala Harris and Donald Trump and the polls are so close that people are looking to the so called political experts – the “gurus” and “pundits” – for their insights into what will happen. However, there is new research that shows that these “experts” may not really know any more than the rest of us.
Big-money political campaigns rely on consultants to craft winning messages. But are these experts really worth the cost? Yale’s Joshua L. Kalla and colleagues found that political professionals are no better than ordinary people at predicting which messages will sway voters. In their study, they tested the accuracy of both groups in assessing the effectiveness of 172 campaign messages on issues like marijuana legalization, student debt, and border security.
They found that both groups performed barely better than chance and that the practitioners were no more perceptive than laypeople in identifying messaging that resonates with people.
“We found that neither political practitioners nor the mass public are particularly accurate in predicting which persuasive messages are more effective than others,” said Kalla, associate professor of political science in Yale’s Faculty of Arts and Sciences. “This suggests that political practitioners who craft language intended to persuade have fairly poor intuitions about which messages people will find persuasive.”
The study, published in the journal Proceedings of the National Academy of Sciences, was coauthored by David E. Brookman, Christian Caballero, and Matthew Easton, all of the University of California, Berkeley.
For the study, the researchers gathered 172 text-based political messages that political practitioners have used to support or oppose 21 distinct issues. They pulled the messages from sources such as voter guides published by various advocacy organizations or the social media accounts of prominent politicians.
For example, the Marijuana Policy Project used the following message to support cannabis legalization: “Polls show that a strong and growing majority of Americans agree it is time to end cannabis prohibition. Nationwide, a recent Gallup poll found that 66% support making marijuana use legal for adults.”
To evaluate the effectiveness of such messages, the researchers conducted a large-scale survey experiment. They randomly assigned 23,167 participants to either a treatment group or a control group. The treatment groups were exposed to messages on three specific issues, while the control group received no messages. Participants in both groups were then surveyed about their opinions on these issues, resulting in 67,215 observations. The researchers used this data to estimate the actual impact of each message.
Next, 1,524 political professionals and 21,247 laypeople were asked to predict the effectiveness of these messages. Despite their experience, both groups performed only slightly better than random guessing in predicting the messages’ persuasive power.
The findings suggest that, rather than relying on their intuition, political practitioners should consider incorporating data-science techniques into their evaluations of potential messages, said Kalla, a faculty fellow at Yale’s Institution for Social and Policy Studies.
“The main takeaway here is that political practitioners have tools available to help them identify effective messages without having to rely on their gut feeling,” he said. “They could use survey experiments similar to what we did in this study. We see political campaigns already doing that, and I suspect more will adopt such techniques moving forward.”
