Second, groups with diverse viewpoints are the best protection against confirmatory thought. Peer review, the gold standard that epitomizes the open-mindedness and hypothesis testing of the scientific method, “offers much less protection against error when the community of peers is politically homogeneous.” In other words, the opinions of group members aren’t much help if it is a group of clones. Experimental studies cited in the BBS paper found that confirmation bias led reviewers “to work extra hard to find flaws with papers whose conclusions they dislike, and to be more permissive about methodological issues when they endorse the conclusions.” The authors of the BBS paper concluded that “[n]obody has found a way to eradicate confirmation bias in individuals, but we can diversify the field to the point to where individual viewpoint biases begin to cancel out each other.”
The BBS paper, and the continuing work of Heterodox Academy, includes specific recommendations geared toward encouraging diversity and dissenting opinions. I encourage you to read the specific recommendations, which include things like a stated antidiscrimination policy (against opposing viewpoints), developing ways to encourage people with contrary viewpoints to join the group and engage in the process, and surveying to gauge the actual heterogeneity or homogeneity of opinion in the group. These are exactly the kinds of things we would do well to adopt (and, where necessary, adapt) for groups in our personal lives and in the workplace.
Even among those who are committed to truthseeking, judges and academics, we can see how strong the tendency is to seek out confirmation of our beliefs. If you have any doubt this is true for all of us, put this book down for a moment and check your Twitter feed for whom you follow. It’s a pretty safe bet that the bulk of them are ideologically aligned with you. If that’s the case, start following some people from the other side of the aisle.
Wanna bet (on science)?
If thinking in bets helps us de-bias, couldn’t we apply it to help solve the Heterodox Academy problem? One might guess that scientists would be
more accurate if they had to bet on the likelihood that results would replicate as compared to traditional peer review, which can be vulnerable to viewpoint bias. Especially in an anonymous betting market, confirming the strength of your pre-existing ideology or betting solely on the basis that replication of a study confirms your own work or beliefs counts for nothing.
The way a scientist would be “right” in such a betting market is by using their skill in a superior way to make the most objective bets on whether results would or would not replicate. Researchers who knew in advance their work would be subject to a market test would also face an additional form of accountability that would likely modulate their reporting of results.
At least one study has found that, yes, a betting market where scientists wager on the likelihood of experimental results replicating was more accurate than expert opinion alone. In psychology, there has been a controversy over the last decade about a potentially large number of published studies with results subsequent researchers could not replicate.
The Reproducibility Project: Psychology has been working on replicating studies from top psychology journals. Anna Dreber, a behavioral economist at the Stockholm School of Economics, with several colleagues set up a betting market based on these replication attempts. They recruited a bunch of experts in the relevant fields and asked their opinions on the likelihood the Reproducibility Project would replicate the results of forty-four studies.
They then gave those experts money to bet on each study’s replication in a prediction market.
Experts engaging in traditional peer review, providing their opinion on whether an experimental result would replicate, were right 58% of the time.
A betting market in which the traders were the exact same experts and those experts had money on the line predicted correctly 71% of the time.
A lot of people were surprised to learn that the expert opinion expressed as a bet was more accurate than expert opinion expressed through peer review, since peer review is considered a rock-solid foundation of the scientific method. Of course, this result shouldn’t be surprising to readers of this book. We know that scientists are dedicated to truthseeking and take peer review seriously. Arguably, there is already an implied betting element in the scientific process, in that researchers and peer reviewers have a reputational stake in the quality of their review. But we know that scientists, like judges—and like us—are human and subject to these patterns of
confirmatory thought. Making the risk explicit rather than implicit refocuses us all to be more objective.
A growing number of businesses are, in fact, implementing betting markets to solve for the difficulties in getting and encouraging contrary opinions. Companies implementing prediction markets to test decisions include Google, Microsoft, General Electric, Eli Lilly, Pfizer, and Siemens.
People are more willing to offer their opinion when the goal is to win a bet rather than get along with people in a room.
Accuracy, accountability, and diversity wrapped into a group’s charter all contribute to better decision-making, especially if the group promotes thinking in bets. Now that we understand the elements of a good charter, we move on to the rules of engagement for a productive decision group, how to most effectively communicate with one another. A pioneering sociologist actually designed a set of truthseeking norms for a group (scientists) that form a pretty good blueprint for engagement. I don’t know if he was a bettor, but he was influenced by something very relevant to thinking about bias, rationality, and the potential gulf between perception and reality: he was a magician.