GiveDirectly? Not So Fast.
We are mistaking an important experiment for a proven solution.
Related Posts
Online articles that discuss the GiveDirectly model.
GiveDirectly is the current flavor of the month, and every couple of days someone asks us what we think of it. For those of you behind on your news feed, GiveDirectly does unconditional cash transfers—it sends money via mobile payment straight to the poorest people in Kenyan villages. It’s a brilliant approach to cutting out corruption and waste: 93 cents of every donated dollar goes to recipients as cash, and those recipients reliably spend it on food, housing, health care, education, and business investment. GiveWell lists GiveDirectly as one of its three “Top Charities” and recently gave the organization $5 million to do a lot more of the same.
The GiveDirectly team is really impressive, and we like approaches to development that give poor people the power of choice. We also like the cheekiness of simply handing out cash, and it was fun to hear the GiveDirectly crew talking about their work on NPR’s “This American Life” (equally fun to hear Heifer International pinned to wall for its lack of—almost contempt for—impact data on the same show). But is GiveDirectly’s model, as Slate put it: “the best and simplest way to fight poverty”?
No. It’s an experiment—an important one, but an experiment nonetheless. We hope we’re wrong, but our hunch is that it is more of a 1-year reprieve from deprivation than a cost-effective, lasting “solution to poverty.”
Poor people are poor because they don’t have money, and so we think unconditional cash transfers should be judged primarily by how much money recipients are making a few years out from the windfall. GiveDirectly’s work in Kenya is too new to know that, but cash-transfer enthusiasts like GiveWell point to other studies of other cash-transfer programs and predict a slam-dunk.
We looked at those studies, and we’re puzzled as to why GiveWell’s analysts, who rightly prize impact and cost-effectiveness, chose GiveDirectly as one of their “Top Charity” trinity. In the most relevant, longer-term study that GiveWell cited—a program working with unemployed youth in Uganda—recipients of an initial $382 grant had a 41 percent greater monthly income 4 years out. This sounds like a big return on the donor’s dollar.
It’s not. Working out the cost-effectiveness of income-generating funding can be confusing, and we at Mulago find it useful to benchmark grants by calculating the amount of additional income over 3 years, divided by the amount of grant money it took to generate it: the income bang for the donor buck. Of the baseline income of those youth, 41 percent turns out to be $11 per month. By that calculation ($11 per month x 36 months ÷ $382), the unconditional cash grant produced $1.03 of additional income over 3 years per donor dollar, essentially a wash.
By contrast, One Acre Fund, which works with the same population in Western Kenya, spends about $120 for roughly the same $380 in additional 3-year income. KickStart and Proximity Designs, which distribute subsidized irrigation equipment, generate substantially more than $10 over 3 years per donor dollar. VisonSpring, which helps people recover lost livelihoods by supplying reading glasses, turns a donor dollar into $60 of additional income. It can be hard to compare the results of academic studies to data generated by growing, evolving field organizations, but all of these business-minded NGOs spend substantial amounts on monitoring and evaluation, and have credible methodologies.
With these results in mind, the 1-year results of a randomized controlled trial (RCT) on GiveDirectly’s work are illuminating. On the basis of that study, GiveDirectly reports a 28 percent income return on a $500 grant. By our benchmark analysis, if this return is consistent over 3 years (0.28 a year x $500 x 3 years = $420 total), then the income return per donor dollar on a $500 grant is less than $1. Ouch. Here’s the kicker, though: That’s gross income, not profit. When you factor in additional expenses, well, as the study’s authors said, “We do not observe a significant increase in estimated profits from self-employment.” Profits may materialize in the next few years, but really, who knows?
The other findings of the Kenya RCT left us underwhelmed, including:
- Food security: A the start of the study, 64 percent of people didn’t have enough food for the next day; at the end of the year, 57 percent still didn’t. Kids were still hungry: The average kid went from 4 days a year without food to 2.5 days. This isn’t food security; it’s a little less food insecurity. And since the net income gains are minimal, it’s hard to be sanguine about lasting effects.
- Education: No apparent effect on outcomes, at least not yet.
- Health: Ditto.
- Assets: Some people—23 percent—use the money to replace their thatch roof with a metal one. Enthusiasts are citing the replacement as a significant savings of money, since people must replace thatch every 1-2 years at a cost of $55 (an average of about $37 a year). However, a metal roof costs $400 (All of these cost figures come from the RCT report). That means it takes a decade for people realize any savings, and by that time they’re looking to replace the damn thing anyway! It’s no wonder that 77 percent of folks chose to do something else with the money.
It was interesting to see that $1 put into livestock yielded $2 in revenue—maybe Heifer is on to something. In contrast, $10 invested in non-agricultural enterprise yielded only $11 in revenue. It may be that the average rural Kenyan isn’t aware of or doesn’t have access to great investments.
We like the evidence that people are happier and less stressed, and it’s intriguing to speculate that improved decision-making enabled by lower stress levels may yield perceptible long-term benefits. But we worry that stress levels may revert to the status quo ante after a year or two. In the Uganda study, initial psychosocial benefits didn’t last.
The excitement about unconditional cash transfers is perhaps more a reflection of the sorry state of development aid overall than of the real impact of the cash itself. And while the transfers are posed as a replacement of other subsidies, it is still the case that for people to make the most of the cash, they must be able to purchase high-quality, affordable products and services—such as water purification, education, health care, fortified foods, and farm extension services—that are typically not available in rural areas unless someone else subsidizes them.
Choice is a good thing; the notion that poor people must simply take whatever we feel like giving them is odious and wrong. However, a blind belief that “people always know best” ignores the reality that when you don’t have access to high-quality education, information, and products, it can be hard to make optimal decisions and take effective action. The poor don’t spend the cash on stupid things; they just may not have access to great things.
We’ll be delighted if the work in Kenya eventually shows cost-effective and lasting health, education, and income benefits. GiveDirectly—whose own claims are more modest than what we see in the media—should continue its investigation of how, when, and for whom unconditional cash transfers will have the most impact. What we’ve seen so far, though, leaves us skeptical, and the hype remains well ahead of the impact.
Kevin Starr (@mulagostarr) directs the Mulago Foundation and the Rainer Arnhold Fellows Program.
Laura Hattendorf joined the Mulago Foundation in 2007, having survived start-ups in both the for-profit and non-profit sectors. As the portfolio director, she is responsible for developing Mulago’s investment framework and building a portfolio of exemplary social investments.
These groups' perspectives are closer than most think—and it’s good news for philanthropy.
Are traditional assumptions about how we “do” philanthropy preventing us from finding new and better ways of working?
Three ways to create an environment where interdependent stakeholders can perform their individual roles optimally and collaborate with each other effectively.
If we want the nonprofit sector to innovate, we need to acknowledge the gender gap between nonprofits and the private sector.
Money doesn't make you happy, but giving it away does
The nonprofit funding process lacks transparency and fosters insecurity—and only funders can fix it.
what can be done to help money-soliciting callers become more enthused and successful?
No-strings-attached prizes incentivize innovation, but private foundations need to structure them carefully to avoid prohibitive penalty taxes.
By embracing a more-inclusive outreach approach, effective philanthropy advocates can attract more funders.
Participation in a network allows foundations to leverage their individual investment by surfacing multiple, ongoing opportunities for collaborative grantmaking.
COMMENTS
BY brad
ON March 11, 2014 12:40 PM
You wrote: “Enthusiasts are citing the [roof] replacement as a significant savings of money, since people must replace thatch every 1-2 years at a cost of $55 (an average of about $37 a year). However, a metal roof costs $400 (All of these cost figures come from the RCT report). That means it takes a decade for people realize any savings, and by that time they’re looking to replace the damn thing anyway!”
I’m having a hard time understanding this argument, because when you look at it from the recipient’s perspective, GiveWell just handed them $1,000 (pennies from heaven) and if they spend $400 of that money on a new metal roof it seems to me that (from the recipient’s perspective) they would realize savings in the first year. From the donor’s perspective I can see the point in terms of ROI. But the recipient’s perspective is different: it’s not like they’re taking $400 out of their own pockets and investing it in a metal roof.
There are probably other more profitable things they could do with that $400, but I do think that recipients would see benefits in the first year because they won’t have to spend any of thei own money repairing a thatched roof.
BY adam thomson
ON March 12, 2014 02:12 PM
Great article Kevin and Laura. Your perspective of impact is very helpful for those of us trying to generate social impact in the developing world. What advice do you have for organizations attempting to generate a more holistic impact that can’t be measured in one or two economic indicators?
I’ve been following Mulago for a long time and am familiar with your desire for organizations to ‘pick one indicator’ to demonstrate impact. I’m challenged by this because projects that have a mission to care for various parts of the individual cost more and seem less cost-effective when measured by only one of the many and often diverse outcomes of the project. Any resources or thoughts?
Thanks,
Adam
BY Eric Stowe
ON March 12, 2014 03:43 PM
I would echo your sentiment that Give Directly is more modest than many of the media stories about them seem to be. I found them to be quite pragmatic about the unknowns and quite unassuming about the immediate term impacts of their work.
The intensity in which many donors have championed this cause may not be commensurate with the real numbers re: future income generated and tangible impact which sadly calls to, what I think is, one of the most powerful statements above: “The excitement about unconditional cash transfers is perhaps more a reflection of the sorry state of development aid overall than of the real impact of the cash itself.”
I would flip my comment too, though, that the naysayers of Give Directly have mostly rallied against them with a fairly venomous tone against this as an option within the int’l development community- often with little corroborating data for that particular stance.
To date, and from my limited aperture, there hasn’t been much middle ground analysis so it is nice to see a level-headed take on where things stand now.
Thanks.
BY gwendoline
ON March 13, 2014 09:21 AM
Brad, I think the point was that for those few rural Kenyans who receive the GiveDirectly grant, the savings come at only about $37/year. For all other Kenyans purchasing metal roofs to replace thatch (a superior material by some standards, but that’s another story), the savings only come into play ten years down the line, if indeed the metal roof has not needed repairs before then. In any case, the metal roofs need to be replaced every 10-15 years, so the savings from such a purchase are not so long-lasting, whether “pennies from heaven” or not.
BY brad
ON March 13, 2014 10:19 AM
@gwendoline—thanks, that makes sense. But I wouldn’t say “only” $37/year, because according to the GiveDirectly site the recipients chosen to receive aid live on an average of 65 cents per day. So saving $37 year is like having nearly two months’ worth of money available.
From the recipient’s point of view post-handout versus pre-handout, over a 10 year lifetime the savings from a metal roof be close to the equivalent of two years of earnings. That’s not insignificant. Plus I remember reading that metal roofs reduce malaria risk compared with thatch roofs, since mosquitoes can easily get in through thatch roofs. I imagine in housing that poor, though, mosquitoes have plenty of other entry points available.
BY Gayle Gifford
ON March 13, 2014 01:19 PM
Surely there must be some data somewhere from the many years when the child sponsorship organizations gave families cash, sustained over a number of years? Some of those sponsored children also had their sponsoring families pay some or all of school fees and even college tuition.
Here is a study from 2005, which is later than the period I was thinking about: http://www.ids.ac.uk/files/MakingCashCountfinal.pdf There must be others?
What I do remember from the 1980s was that the international development community was ruthless in its critique of handing money directly to individuals, which led to the demise of direct cash transfers. It will be very interesting to me to follow how this plays out today.
In all of the back and forth around cash transfers, however, I am reminded that many countries including the USA have cash transfers to their poorest citizens - through government social welfare programs. In most cases those programs are not one time payments with expectations that the poorest among us will suddenly build their own way out of poverty without significant other government and community supports - like decent jobs and quality education.
BY Chris Blattman
ON March 14, 2014 06:41 AM
These are fair points. I’ve blogged a comment here
BY Lorna Van
ON March 14, 2014 10:20 AM
@Brad @ Gwendolyn Something else in the roofing line…. $37 to replace a thatch roof keeps money in the local and sustainable economy rather than sending out money for a metal roof, likely made in India not Kenya. Something to think about. @Brad, you are right, mosquitos have lots of access!
BY John Anner
ON March 16, 2014 12:55 PM
Kevin and Laura are to be congratulated for bringing some intellectual rigor to the analysis of the performance of NGOs. But unfortunately across our sector (I am president of an NGO working in Asia and Africa) we are all too often comparing guavas and mangos, and this undermines the critique. NGOs that conduct scientifically-valid research are few and far between, and self-reporting on the basis of an M&E exercise, no matter how honest and thorough, is a pale substitute. It’s not the fault of the authors, but comparing the results of an RCT to the data generated by an NGO itself is just not fair.
Even an independent case study of a single organization, or a one-group pre-test/post-test analysis, falls far short of the minimum requirements of good social science research. There are too many serious threats to validity. We are fortunate to have a PhD biostatistician on our staff, and he constantly has to reign me in from claiming results for our programs until they can be scientifically verified.
In any event, I’m a bit surprised at all the fuss over Give Directly, since the positive results of direct cash transfers (conditional or otherwise) seems to be well established. Organizations like the World Bank and International Poverty Centre and journals like the Lancet have published peer-reviewed analyses on CCTs for at least the past decade.
BY Steve Wright
ON March 17, 2014 10:01 AM
In an effort to complicate this a bit more I think it is essential to look at levels of poverty relative to this analysis. At Grameen Foundation we create the Progress out of Poverty Index (http://www.progressoutofpoverty.org) which measures the likelihood that a household is below a specific poverty line. Through this lens you can identify the families that are the most likely to be extremely poor because these are the families that are most likely to benefit the greatest amount from the direct gift of cash. This this paper
http://graduation.cgap.org/library/a-double-bottom-line-business-case-for-serving-very-poor-households/
we worked with Fonkoze (MFI in Haiti) to look at the cast/benefit of developing different products for different needs and different “cost structures” at different levels of poverty.
In this paper
http://www.microlinks.org/library/pathways-out-poverty-case-study-livelihood-pathways-sequencing-sustainable-market-integratio
we looked at sequencing interventions by poverty level. Because, as stated in this post, extreme poverty is debilitating and cash transfers can provide a reprieve. It is critical that during that reprieve the next strategic intervention is applied. At Grameen, we believe financial services are essential and micro-savings in particular. It is also essential to make linkages at this stage to any government or other “services” that are available. Then, after some of these interventions are in place, livelihood development.
I say all this not only as an advertisement for our work but also to note that there is no panacea for poverty and also to say, that we know more than we realize but we must be willing to around to find what we have already learned. Our learning in microfinance is actually very nuanced but we have spent a lot of energy painting microfinance is a corner (not unlike cash transfers) so we think of it as binary (good or bad).
BY GiveDirectly
ON March 17, 2014 10:16 AM
We appreciate the opportunity to engage in a robust discussion about impact, and have acknowledged and responded to these points on our blog: http://www.givedirectly.org/blog_post.php?id=7431955057156519484.
BY Delia Floue
ON March 17, 2014 05:27 PM
This sounds like a personal attack on an organization that has both set a high bar for its work and also has PROVEN its work. Would be great to hear Kevin Starr’s response to GiveDirectly’s response. I feel embarrassed for Kevin Starr, Laura Hattendorf, and their portfolio nonprofits that such gross misinterpretation of key stats were made.
BY Kevin Starr
ON March 17, 2014 10:27 PM
Hey, GiveDirectly guys - thanks so much for the response you posted. Important points to raise, and it is useful to look at it from your point of view. We’ll see if we can put together something intelligent in response. This is great.
And Delia - relax. This is a healthy debate and we have enormous respect for the GiveDirectly team. You seem to feel quite strongly about all this, so perhaps you would enjoy reading the policy paper yourself. Here it is: http://web.mit.edu/joha/www/publications/Haushofer_Shapiro_Policy_Brief_UCT_2013.10.22.pdf
BY Adam Pickering
ON March 18, 2014 05:03 AM
Excellent article.
I do have some concerns about the promotion of unrestricted cash transfers as a panacea, particularly in an era where donors are increasingly being told by the media that overheads are wasteful. Having said that, I do think that they offer a wake-up-call for any organisation who has yet to find the will to properly engage in impact measurement. Give Directly co-founder Paul Niehaus told This American Life:
“We would like to see organisations make the case that they think they could do more for the poor with a dollar, than the poor could do for themselves.”
If organisations rise to this challenge then the impact of organisations like Give Directly could be very positive. If they fail to do so then our sector, and its beneficiaries will suffer from a paucity of strategic and nuanced help and support because handouts can only ever be a small part of the solution.
I explored this in much more detail on the Future World Giving website http://bit.ly/1j00pu5
BY Anna
ON October 23, 2014 09:22 PM
Interesting perspective. I do think that the Givedirectly model’s ROI should not just be measured on how much money people are making down the road. Giving people choice and dignity is something in of itself. On a personal level, giving up a Starbucks concoction so that a desperately poor person can have five actual dollars in their pocket to feed themselves for a week is empowering. I do agree that calling it a cure for poverty would be an overstatement.
BY FBOB
ON November 1, 2014 09:58 PM
Well, aside from the previous pious comments from the big NGO reps I’m quite happy to see 99% of the donated $ in the actual hands of people who could use the money. On the other side, big $$$ NGOs? why is only about 1% of whats left goes to those who could really benefit.
I know that answer, we need statistical models, governance, auditing, accounting, fraud detection, on and on and on. Eventually, near nothing is left for those who you donated to.
Apparently, there is some rather large brain trusts that think or better yet bet against the people of impoverished nations to not spend the money wisely. When money is scare, spending it is taken with great care. Remember that, o foolish living on the donations of others (yes I mean you CEO, CFO, CTO, COO and every other donation dollar sucker at the top), we get it.. Many of us won’t give to your organizations just because you waste of our effort =$$$.
givedirectly will continue receive our donations for the foreseeable future and will get more recommendations for donations…..
thanks.