Experiment to Save an Endangered Fish Holds Lessons for Policymakers

By Ted O’Callahan

It’s no accident that Indiana Jones was an archeologist, not an economist. Economists are better known for digging into data sets than digging up clues. But in recent years, a number of economists, particularly development economists, have led a revolution in the field—by going into the field.

Many of these new adventurers are motivated to better understand which new policies, philanthropic programs, or other interventions have the greatest positive impact for people in developing economies. The randomized controlled trial has become a key tool for them to compare the effects of an intervention with what would happen in the absence of such an action.

“There are often uncertainties about which strategy would be most effective for the problem at hand,” says Mushfiq Mobarak, an economist at Yale. “A randomized controlled trial is a great way to collect very rigorous data on what works and what doesn’t.”

In one recent example, Mobarak and his collaborator Andres Gonzalez-Lira ended up designing an experiment in which they sent incognito investigators into fish markets in Chile to determine whether a banned species was being offered for sale. The big question they wanted to study was how merchants and shoppers would respond to different enforcement efforts—would they ignore them, evade them, or obey them?—and therefore how to make environmental regulations actually work.

Read the study: “Enforcing Regulation under Illicit Adaptation”

The project started in 2015 when Sernapesca, the government agency in charge of Chile’s fisheries, reached out to propose collaborating with Mobarak and Gonzalez-Lira, a PhD candidate at the University of California Berkeley, who had trained agency representatives while working at the Abdul Latif Jameel Poverty Action Lab. Mobarak and Gonzalez-Lira saw an opportunity to do an innovative experiment that could have broad impact.

Chile is among the top 15 countries in fish production. With 4,000 miles of coastline and much of the fishing done by artisan fishermen working from small boats, enforcement of fishing regulations has had limited success. As a result, the government estimates 72% of fish species are overexploited or collapsed despite decades of management efforts that included restrictive quotas and ban periods.

Globally, the state of marine fisheries continues to decline. According to the UN’s Food and Agriculture Administration, a third of marine fish stocks are fished at biologically unsustainable levels. Better policy tools could help preserve critical fisheries around the world.

“Most developing economies face these kinds of issues,” Gonzalez-Lira says. “Overexploitation of natural resources, including illegal fishing, are closely tied to the quality of institutions and the wealth of the country.”

So how do countries best protect natural resources and other types of public goods? Mobarak notes that when governments create a new rule to address a problem, there’s typically a short-term shift in behavior that makes it look like the rule is having its intended impact—but reverses over time. “People react to new rules in unanticipated ways which can make enforcement much more difficult or complicated and may undermine the intent of the law.”

Gonzalez-Lira adds, “We wanted to dig in on what’s failing with enforcement.” Understanding why regulations aren’t having the desired effect can help create more sophisticated tools that allow actual policy outcomes to be closer to the intended outcome, even after unanticipated adaptations.

Sernapesca asked Mobarak and Gonzalez-Lira to look at its seasonal ban on the critically endangered Pacific hake (merluza común), an important source of protein for low- and middle-income Chileans. Because this particular variety of hake is caught and sold almost entirely within central Chile, an experiment could capture a very complete picture. The fish reproduces in September, so banning catching or selling the fish for that month, each year, is intended to help the species rebuild its numbers.

Typically, Sernapesca, like regulators worldwide, focuses on monitoring the catch at the docks. But in this case, Sernapesca wanted to understand the impact of targeting the two other pieces of the supply chain—vendors and consumers.

Gonzalez-Lira and Mobarak designed an experiment to understand how effective the existing ban was at reducing hake harvest and sale, as well as comparing the impacts of various enforcement mechanisms and the effect of an information campaign publicizing the existence and reasons for the ban.

The experiment would also show the comparative cost effectiveness of each strategy. “It’s not common for a research paper to be able to let the government design enforcement to be more cost effective,” Gonzalez-Lira says. “That was super-cool. We were able to do that because we designed the program in close cooperation with Sernapesca.”

Part of the reason economists must be in the field to run a randomized control trial is that designing the experiment requires a nitty-gritty understanding of a market failure and the potential fixes. “I spent two or three months just visiting markets, thinking about how to design everything in order to collect data that reflected what was actually happening,” Gonzalez-Lira says.

He and a research assistant visited dozens of municipal markets throughout central Chile, where 98% of the hake harvest occurs. They gathered basic information about the neighborhoods served by each market and confirmed that, even in the smaller towns, vendors don’t know every single customer, so an unfamiliar face won’t necessarily raise suspicions. But an economist sneaking around a fish market didn’t always manage to stay under the radar. Simple curiosity led Gonzalez-Lira to ask a vendor how much fish he sold each day. “The immediate response from him was, ‘Why are you asking that?’”

Even that quick pushback offered useful information. The level of reluctance to share information about everyday legal business underscored how challenging it would be to get accurate information on illegal sales. “Nobody likes to tell you they cheat,” Gonzalez-Lira says, “They’re going to hide that they do it. You can’t ask them directly.” And you can’t just penalize the problem away. “Imagine a uniformed government inspector steps into the market. The moment he or she appears, the illegal fish will disappear. Vendors will hide it and do all kinds of other things to avoid a fine.”

Having realized that stepping outside the typical back-and-forth between customer and vendor would create wariness, Gonzalez-Lira sought a way to get the data he needed within that exchange. Through trial-and-error conversations with vendors, Gonzalez-Lira wrote, piloted, and polished a script that elicited the information he needed.

Mobarak noticed during his visits to markets that many vendors use the colloquial term casera to attract the attention of typical shoppers —women in their 40s and 50s. The research team recruited and trained 29 such women as “mystery shoppers” to visit stalls and surreptitiously collect data on the fish being sold. Each market was visited an average of three times in the course of the month. Each time, the mystery shopper would talk with all the fish vendors in that market. Once the shopper knew whether hake was available, how much it cost, and the availability and cost of a comparable alternative, the shopper would sometimes make a purchase, as long as it wasn’t hake. “We realized it would be fishy to go there, ask questions, and not buy something,” Gonzalez-Lira says. (The mystery shopper got to keep the purchase. “They ate a lot of fish that month,” Gonzalez-Lira adds.)

Separately, there were surveys of consumers, vendors, and fishermen in order to gather other information such as the ban’s impact on consumer choices and the hake supply chain.

The results showed that many vendors complied with the annual September ban on hake sales. But some continued to sell hake openly. As time went by, techniques for evading the ban developed. Some vendors hid the fish but had it available for those who asked. Still others claimed the hake they had for sale had been caught during the legal season, then frozen. Tellingly, the workarounds developed most quickly in the markets where inspections were the most frequent.

The researchers found that the information campaign and enforcement visits each reduced sales of hake. Outside of the September ban, hake is typically available at 90% of fish vendor stalls. During the September ban, availability dropped by 40 percentage points. Enforcement done on a predictable schedule didn’t further reduce hake sales significantly. But enforcement inspections on an unpredictable schedule reduced availability an additional 19 percentage points. The information campaign reduced vendors likelihood of selling hake by 13 percentage points. Enforcement visits combined with an information campaign did not make a significant difference over enforcement alone.

The study also provided Sernapesca with valuable information about the cost-effectiveness of each approach. “At the end of the experiment we were able to tell the government, ‘If you spend $1 doing this, you will reduce illegal fish getting to market by x percent. If you invest $1 in something else, that will reduce illegal fish getting to market by y percent,” Gonzalez-Lira says.

As a result of the research, since the information campaign was effective and significantly less expensive than enforcement efforts, the government decided to adapt and scale it, as well as extending it with similar consumer awareness campaigns about three other species. In addition, the experiment also helped Sernapesca redesign its primary activity, enforcing fishing regulation, based on the evidence of the value of unpredictable patterns of enforcement.

As the authors note, their paper provides “empirical evidence on the real-world challenges to implementing policies that are designed to address negative externalities and collective action failures.” Demonstrating that policymaking is hard likely won’t surprise many policymakers—but offering data on the longer-term impacts of government interventions provides a baseline to work to develop policies that are better able to reach their intended goals.

Mobarak is already looking to that next phase of evidence-based policy making, in his role as faculty director of the Yale Research Initiative on Innovation and Scale (Y-RISE). Y-Rise focuses on the challenges and implications of scaling promising policy interventions developed through randomized controlled trials.

He says, “If you design randomized control trials well, taking economic or political theory into account, then you might be able to generate much better and more rigorous answers to fundamental questions about human behavior.”