Can Reflection Dislodge a Faulty Intuition?

Steve Jobs once said, “Intuition is more powerful than intellect.” But can it be too powerful? In research recently published in the journal Cognition, Yale SOM’s Shane Frederick and Andrew Meyer of the Chinese University of Hong Kong not only show situations in which widely shared intuitions are certifiably wrong, they also illustrate how these erroneous intuitions become calcified and prevent discovery of the truth, even among those who are told to reflect.

Their work centers on the “bat and ball problem”—the best known of Frederick’s three-item “Cognitive Reflection Test.”

A bat and a ball cost $110 in total.
The bat costs $100 more than the ball.
How much does the ball cost? $____

Most respondents answer $10—a conclusion that cannot be reconciled with the two sentences they’ve just read. If the ball is $10 and the bat is $100 more, the bat itself would cost $110 and the two prices would sum to $120. (Alternatively, if two prices sum to $110 and the ball costs $10, the bat must cost $100, which is only $90 more.) The correct answer, of course, is $5.

This problem is frequently invoked to illustrate what’s known in behavioral economics as “dual system theory,” which posits two distinct modes of thinking: fast intuition and slower reflection. The problem evokes an incorrect heuristic operation—subtraction—and one must reflect upon the output of that operation to conclude that something has gone wrong. But if respondents were encouraged to reflect, they would spot their error…right?

Sometimes, but not as often as you might expect—or hope. In their latest paper (involving 59 new studies and over 70,000 respondents) Meyer and Frederick found that the erroneous intuition is surprisingly difficult to unseat—despite manipulations intended to help the subjects catch their error.

In one of these studies, respondents were warned that “the problem is trickier than it looks.” In another, they were given a second chance after receiving the feedback that the answer is not $10. In a third, they were prompted to “consider whether the answer could be $5.” One study even told respondents: “The answer is $5. Please write the number 5 in the blank below.” Though all of these manipulations increased solution rates, only the last was sufficient to get most respondents to answer correctly—and even there 18% failed to comply with the instruction, insisting that the answer was $10.

The “careless” and the “hopeless”

Meyer and Frederick’s studies undermine a long-held assumption among dual-system advocates: that errors in judgment can be fixed simply by getting people to slow down and think harder. Some of their studies effectively partitioned respondents into four distinct types: the reflective, who get the answer right initially; the careless, who answer incorrectly, but who can solve the problem when told that $10 is not the answer; the hopeless, who are unable or unwilling to solve the problem; and the stubborn, who repeat their $10 answer, even after being told it is incorrect. The intuitive response is tough to dislodge in part because of the high confidence with which it is held: most respondents who answered $10 were 100% confident in their answer.

Meyer and Frederick showed that supplying the $110 price from which participants might subtract $100 to arrive at a $10 ball (and $100 bat) was enough to stop many of them from noticing that those two prices differ by less than $100.

A bat costs $100 more than a ball.
If you said the bat cost $100 and the ball costs $10, would your prices be correct? Yes/No

versus

A bat and a ball cost $110 in total. The bat costs $100 more than the ball.
If you said the bat cost $100 and the ball costs $10, would your prices be correct? Yes/No

Both conditions ask the same question and both clearly stipulate that the bat costs $100 more, yet the proportion answering incorrectly (“Yes”) jumped from 18% in the top condition to 53% in the bottom condition.

Although the bat and ball problem was the star of the paper, Meyer and Frederick also used other items to illustrate how the structure of a problem can elicit a correct or incorrect intuition. Consider two variants of their “smokers” question:

If 1 in 10 men smoke and 1 in 30 women smoke, then 1 in ___ people smoke.

versus

If 3 in 30 men smoke and 1 in 30 women smoke, then 1 in ___ people smoke.

Though these are nearly equivalent questions with identical answers, only 7% solved the first problem, whereas 49% solved the second. In both problems, people are inclined to average the two numbers which differ—10 and 30, in the top version, or 3 and 1, in the bottom version. (Your middle school math teacher would like to remind you that it’s OK to average numerators, but not denominators.) If you, like most people, answered 1 in 20 to the top problem, ask yourself what the average rate would be if no women smoked. The answer would then be 1 in 20. But since some women do smoke, that can’t also be the answer to the question asked. (That, says Frederick, is what he means by cognitive reflection.)

The authors say that both the formation and revision of intuitions have implications beyond mathematical puzzles. Marketers and politicians selling selling products, services, positions, and policies should be interested in how to frame problems that yield a clear intuition, if that intuition favors their objectives, and how to re-frame them when they don’t. Their research also reveals the difficulty of dislodging strongly held intuitions—or relinquishing them in the face of new evidence and arguments.

“When we began studying the bat and ball problem,” the authors write, “we assumed respondents missed it because they didn’t bother to check…. We discovered instead that many respondents maintain the erroneous response in the face of facts that plainly falsify it, even after their attention has been directed to those facts.” They conclude, “The remarkable durability of that error paints a more pessimistic picture of human reasoning than we were initially inclined to accept; those whose thoughts most require additional deliberation benefit little from whatever additional deliberation can be induced.”