Does the Cochrane Review show that masks aren’t effective at preventing COVID-19?
The vast majority of the studies assessed by the Cochrane Review ask, “If we give people masks and information about masking, do they get healthier?” Most of these studies find that the answer is, “Not much healthier.”
But there is a problem: giving people masks is not generally enough to get them to wear masks! In piloting in Bangladesh, we found that mask distribution plus information plus involving village leaders increased mask use by less than 10% (we later added other elements that were more impactful). In other scale-ups, masks and information alone did even less. One study in Uganda found that giving people masks and information increased mask use by one percentage point—that is, by 1 in 100 people.
I wouldn’t say the results of our study contradict the studies in the Cochrane Review. I would say our study is designed to answer the question, ‘Does masking reduce COVID?’ while the Cochrane Review is answering the question, ‘Do mask distribution and information reduce COVID?’
Summarizing all of these studies, the Cochrane Review finds that the average effect is a 5% reduction in illness (COVID or influenza), insignificantly different from zero. But if fewer than one tenth of people were induced to wear masks by the studies in question, this would mean that the effect among mask-wearers was a 50% reduction or more. Now, my point is not that the Cochrane Review shows that the effect is at least 50%. My point is, there is very little information about the impacts of actually wearing masks in the Cochrane Review.
Now, you might say, “Wait a minute, if it’s impossible to get people to wear masks, doesn’t that mean community mask-wearing is futile?” But of course, plenty of people do wear masks! In some locations in the U.S., city- and state-level mandates in 2020 achieved over 90% compliance. Airlines had consistently high use while mandates were in place. And of course, mask use was nearly universal in East Asia. The issue isn’t that no one wore masks, but that the studies in the Cochrane Review compare control groups with intervention groups where people were likely not much more likely to wear masks.
Do the results of your study in Bangladesh contradict the Cochrane Review?
I wouldn’t say the results of our study contradict the studies in the Cochrane Review. I would say our study is designed to answer the question, “Does masking reduce COVID?” while the Cochrane Review is answering the question, “Do mask distribution and information reduce COVID?” At one point, the Cochrane Review states, “Wearing masks in the community probably makes little or no difference to the outcome of influenza‐like illness.” This interpretation is flat wrong given the point above about compliance. The studies they summarize simply do not answer this question.
In Bangladesh, through several rounds of piloting, we arrived at an intervention that considerably increases mask use, achieving a 30-percentage-point increase. A key element of this was having “mask reinforcement”—people in crowded public areas asked people not wearing masks to please put on masks. We found a 10% reduction in COVID. Extrapolating, the total impact of masking relative to no masking (that is, a 100-percentage-point increase) might be three times as large.
Also relevant here are non-random studies of actual mask mandates in the U.S. (here and here) and Germany. These studies ask whether there is a relative decline in infections in regions that implement mask mandates compared to those that do not. They typically find that mask mandates were impactful.
One complication in comparing these studies to our study in Bangladesh is that we don’t know how much mandates increased mask-wearing: one correlational study suggests the increase in mask-wearing from mandates is in line with what we see in Bangladesh, but this may be overstating their impact (probably, mask-wearing is higher in places with mandates for other reasons aside from the mandate).
Why is there still confusion on this question?
There are several reasons for persistent confusion:
Firstly, science is very difficult and this is a highly politicized issue. This means that there is tons of noise in the public debate generated by people trying to score political points rather than trying to figure out what is true. A non-scientist would be exposed mostly to this noise, rather than the underlying scientific debate.
There is a crucial distinction between ‘masks effectively reduce COVID’ and ‘we should have mask mandates.’ The benefits today are probably 40 times lower in the U.S. than their peak in 2021, so the case for mask mandates is much weaker.
Second, it is generally the case that medical researchers attend too little to questions of compliance. There is a longstanding distinction in medicine between “pragmatic” and “explanatory” estimates (in economics, this distinction roughly coincides with “reduced form” versus “instrumental variable” estimates). The “pragmatic” effect asked: if we did this policy in practice, what might happen? The “explanatory” effect is, what would happen under ideal circumstances—e.g., if everyone did it? Medical researchers will often say, “What matters is the pragmatic effect. The trials tell us the pragmatic effect. If the thing doesn’t work because no one complies, then it doesn’t work.” But as the masking example illustrates, things are not so simple. If information and distribution don’t get people to wear masks, it is simply wrong to conclude from studies of information and distribution that masking is ineffective because other policies might have higher compliance. The solution is to collect better data on who actually complies by direct observation (not self-reports, which tremendously overstate compliance).
Third, there is a crucial distinction between “masks effectively reduce COVID” and “we should have mask mandates.” Determining where mask mandates are called for requires cost-benefit analysis. The benefits today are probably 40 times lower in the U.S. than their peak in 2021, so the case for mask mandates is much weaker. We still don’t know much about the very long-term benefits of masking, and we also don’t know much about the costs of masking (discomfort, communication, development for younger children).
Four, it would be nice if we had more than one large trial of the relevant question. There is a lot we don’t know: Would N95 masks or similar quality masks work better? Do masks actually prevent transmission or do they reduce the severity of infections so there are fewer symptomatic cases? The latter would still be valuable, just with very different long-run implications. I think the fundamental issue is that public health agencies do not fund large trials of this nature on short notice (our study was funded by Givewell, a private organization). Organizations like the CDC and NIA need to change their procedures to be more nimble and flexible in funding large-scale trials of urgent and timely questions.