How Innovations in Understanding Everyday Data Can Power More Effective Aid

Hundreds of billions of dollars are spent each year on social protection programs and humanitarian aid. How do governments determine who gets the support?

Providing social protection transfers is one important role played by developing country governments. Even in developed, rich countries like the U.S., governments target support to subpopulations that are particularly needy. That requires solving a really important problem of identifying who should be targeted.

The stakes are high because you don’t want to mistarget—that leads to something called inclusion and exclusion errors. Exclusion errors are people who are needy who don’t get identified. Inclusion errors are people who are not as needy, but who receive transfers anyway.

Previous research found that nearly 25% of poverty-targeting programs are actually regressive—the relatively less poor are receiving the transfers, while the extreme poor are not in the system. That happens when the poverty targeting approach being used isn’t right for the circumstances.

During COVID, it became extremely important to solve this targeting problem. Our government was asking people to social distance and stay at home, so it was important for us to provide those transfers to people who could not work, to make up for the shortfall in income. In the United States, the government solved the problem by just simply looking at the previous year’s tax returns. If anybody earned less than $75,000 a year, they received a check in the mail from the government.

Now, the question is, how do you solve that kind of a problem in a country where less than 5% of people submit tax returns, where there’s a lot of informal sector work and people are not really part of the tax system? So then you can’t just look at tax returns and identify the poor. You have to figure out some other way. And during pandemics, you can’t run surveys to identify the poor.

What are the leading approaches to targeting aid, historically?

There were two categories of approaches that were very popular. One is called proxy means tests. You’re trying to create proxies for people’s means, their ability to support themselves. Data collection is expensive, so you want to use easily observable proxies. What does their house look like? What is the roof material? That’s a proxy for poverty. People who are richer will have more concrete structures; people who are poor may have tin structures or roofs made with thatch. That’s an example of something that’s relatively quick and easy to observe. And you don’t have to worry about whether people will be misreporting; obviously that’s a concern if they know that their responses will determine whether they receive a transfer.

A second category of solutions, what we call community-based targeting, is asking the community to nominate the neediest members of their society. That requires you to get everybody from a small community together in the same room and implement a consultative process like putting people in an income ranking and then choosing who is the poorest within that group.

Between proxy means testing and community-based targeting, which was most accurate?

When rigorous research has been done, we tend to find that proxy means tests outperform community-based targeting. To measure accuracy, though, we would need to know the truth, which is another complicated exercise. To set the true benchmark, we need to ask very detailed questions about people’s consumption, how much they’re eating, what they’re spending money on, and that is a very data-intensive and resource-intensive task. A program might do that in a small subset of a population to verify that a proxy means test is identifying people who are “consumption poor.”

But one nice thing about community-based targeting is that even if it’s not as good at identifying the consumption poor, sometimes community members have some specialized knowledge about who actually is very needy for some other unanticipated reason. Maybe they’re a widow or disabled and it might be hard for them to access government services. The community might have a good idea about need that doesn’t even show up in standard consumption surveys.

A project in Bangladesh that led to the paper you co-authored, “Poverty Targeting at Scale: Algorithmic vs. Traditional Approaches,” used phone data for evaluating need. How did that come about?

In a lot of data-poor environments, nonstandard kinds of data already exist because people are constantly creating information just by virtue of using mobile phones, or satellites constantly passing over their communities, capturing images. Our approach builds on the intuition that if rich people and poor people use their phones differently and there are some such systematic differences, then we can extract those systematic patterns in the data to make an indirect inference about who might be poor, without doing a separate survey. So we explored that possibility.

And we found that rich and poor people indeed use their phones differently. As one example, in most countries, rather than getting on monthly plans people buy phone credit intermittently—and these are called top-ups. Poor people tend to top up their phone more frequently in smaller increments due to cash constraints, whereas richer people might top up more infrequently but in larger increments, because it’s inconvenient to keep topping up your phone.

The use of text messages is also more common among relatively richer people. Especially in an environment where literacy is not universal, you can see why phone calls would be the communication method of choice for poor people.

A third example would be the type of networks each person has. Every time you call somebody, that allows us to indirectly infer what your network looks like, because every time the phone pings into a particular cell tower, we know something about your location. So we can characterize the spatial extent of the networks for each person. Richer people have networks that are more spatially diffused; they’re connected to people farther away. Poor people are more often calling others who are very close to them.

Q: How did you identify those differences in phone usage?

We conducted the expensive survey to actually figure out the ground truth of poverty, but only for about 5,000 households in the Cox’s Bazar administrative district in southern Bangladesh. We asked them detailed questions about their consumption patterns, what they’re spending money on, etc. That gave us a ground truth: who’s actually poor? That’s the left-hand side variable. On the right-hand side, we had features that come out of the phone data—the nature of your network and how spatially dispersed it is, the pattern of your top-up behavior, who you call, who do you text, how often do you text, how often do you call, and hundreds of other such phone features.

Then we had a machine-learning model determine which of 1,578 features were the most accurate proxies for poverty. We did that by letting the algorithm train on a portion of data from the ground-truthed population, leaving the rest as a hold-out sample to test the model. Then we ran that trained algorithm on the remaining ground-truthed data to evaluate its accuracy.

Once you have the algorithm that you’re comfortable with, you hand it to the mobile network operators and have them run that algorithm for their entire customer base of a hundred million to generate a quick prediction for the larger population.

This research happened in connection with programs that were actually distributing aid in Bangladesh?

Yes. When we started during the pandemic, using phone data was a novel method and therefore it took us a while to get it going. I was working with the Bangladesh government at that time to develop pandemic response policies, and I used those connections to set up partnerships with the cell phone service providers and the government. Of course, there are ethical issues involved with data privacy, so we set up a system where no one entity would have access to all the survey data as well as the call detail records. By the time we got everything done and all the data access permissions sorted, it was just too late for pandemic transfers. The novelty of the method meant that it took some time for government agencies to figure out the relevant data privacy protections to institute.

So instead we refocused our attention on providing support in Cox’s Bazar district in southern Bangladesh, which hosts over a million Rohingya refugees from Myanmar. The long-running persecution of the Rohingya minority by the Myanmar military was exacerbated from 2017 onwards, which led to a big refugee influx into Bangladesh in response to the uptick of violence against Rohingya civilians in Myanmar. And poor communities in Cox’s Bazar are now bearing a disproportionate share of the burden of hosting the Rohingya.

So we used this method to identify who within several hundred host communities are most in need of transfers, and we sent transfers to the poorest 21% of people.

In the process of delivering that aid in Cox’s Bazar, you also gathered data on multiple methods for identifying the neediest population?

Exactly. The actual transfers were made on the basis of the phone data. But for the purpose of research, we got a separate grant from the Global Innovation Fund, which is interested in using research to improve development policymaking. And even though we were going to use the phone-based transfers, we also conducted community-based targeting and proxy means tests in the exact same context, and sent additional transfers on the basis of community meetings that identified the poor among them.

So the research output, separate from the money we actually transferred, was to answer the question: how well does the phone algorithm-based method do relative to community-based targeting and relative to proxy means testing in terms of accuracy? Because we have the ground truth, we can measure the accuracy of all three methods and compare them to each other.

At what point did you realize, oh, we should be layering a study with a practical thing we’re going to do?

From day one. The way I think about this work is that while it’s important for us to accurately target and send transfers to the host communities who are most needy, it’s also an important learning opportunity. The Bangladesh government needs to deliver transfers on a regular basis. Other governments around the world need to do it on a regular basis. If you design research right, in the process of sending the transfers, we can also learn a lot about both accuracy and cost-effectiveness of these competing methods. And then we publish those results, so others in the future, when they are thinking about designing transfer programs, don’t waste money choosing the wrong method.

What did you find? What was most accurate, and how does cost factor in?

The phone algorithm-based targeting is less accurate than proxy means tests, but it’s slightly more accurate than the community-based targeting. But a second important insight from the research is that deploying phone-based targeting is the cheapest, especially at large scale when large numbers of potential beneficiaries need to be screened.

It made sense for us to carefully compare the phone-based targeting against proxy means tests, because there was a cost-accuracy tradeoff; the phone-based targeting is cheaper, but it’s less accurate. We dropped community-based targeting from that part of the analysis because it was least accurate and it’s more expensive than phones.

Then we use a very simple model to think about how we generate the greatest possible welfare gains for the community in trading off cost versus accuracy. In the model, we assume that you generate larger gains in welfare by giving a dollar to a much poorer person than by giving it to a richer person. Then we have to make an assumption about what the shape of that welfare gain is—how much better it is for us to target a poorer person than a slightly richer person. The second assumption we make is, imagine you could save money by not having to identify the poor so accurately, and when you save that money, you just transfer it over to the poor. That tells us the dollar value of the tradeoff.

The bottom line is that if you’re making relatively small transfers to lots and lots of people—think about national-scale programs where you have to screen a population of 175 million—the phone-based targeting does best because you save so much on the cost of screening so many people. Whereas if you’re going to have to screen relatively few people and make larger transfers, that’s where proxy means testing is going to win out.

What are the broader implications of this project? What are the opportunities?

We can’t observe everything we need to about people to run programs effectively. So we need to think creatively about, if we don’t know enough, and it’s very expensive to go and collect the data, what types of data already exist based on people’s daily behaviors. Let’s extract that data and see how best to make creative use of it for the social good.

There are administrative records—for example, a schooling system is generating data. Satellite data has a lot of potential. Satellites can take pictures or observe in the infrared band that can be processed into sophisticated measurements. For example, we can learn something from outer space about people’s economic status or their agricultural productivity.

And there’s more we can do with phone data. Even people in remote places who are difficult to survey, and who we don’t really know much about from direct interactions, they’re producing data every day; we can make creative uses of those data. In some countries, there are mobile money systems creating data based on the transactions. Let’s make use of that.

Under the current U.S. administration, USAID no longer exists, and therefore a lot of data-collection activities that the U.S. government was supporting that were immensely valuable are not happening. For example, demographic health surveys, which have been done in many countries around the world, have been incredibly valuable for research. People have used those for a variety of purposes and not just in one field—food and nutrition, medical sciences, health sciences, social sciences. It’s going to be a big loss in the world if that data collection is just not done as regularly. We’re not going to be able to replace that value. So it has become urgent for us to identify creative uses of nonstandard data because the standard sources of high-quality data are disappearing.

Your work is advancing an academic discipline. It is also engaging with existing aid programs to help them understand their impacts and improve. How do you think about innovation?

There are many, many important problems in the world that are valuable to work on. My personal interest is to dedicate my time and attention and effort to working on the problems that are most relevant for the poorest people on earth.

One part of innovation is identifying what’s important to work on. What are the important questions? What are the important gaps? A second part of innovation is asking where we can have the largest marginal effect. Is this research or this implementation going to make a big difference to that problem? That depends on how big the problem is, how important it is, and whether our idea is just tinkering in a small way or really making a transformational change.

And then, implementation is challenging and complex. You could have a brilliant idea that goes nowhere just because politically there’s no appetite for it. Or it could be that you help someone with your brilliant, innovative idea, but then it has unintended consequences on some other domains that you’re not tracking. I founded Y-Rise, the Yale Research Initiative on Innovation & Scale, precisely for that purpose: for thinking through developing the science, developing the research methods, and for thinking about all these complexities of scaling.

There are lots of creative ideas that we get excited about that only affect people’s behavior on the margin. It might read well in an interview. It might lead to more clicks. But we should always keep in the back of our mind, not just whether it’s creative and cool, but also whether it’s profound and important.