Randomization for Causality, Ethnography for Mechanisms: Illiquid Savings for Liquor in an Autarkic Society

What should researchers do when confronted with surprising results? Financial access innovations usually leave “temptation” spending unaffected or reduced. However, the authors found that promotion of savings lockboxes in a largely autarkic society increased alcohol consumption and blood pressure, despite no one reporting intentions to save for alcohol. To probe mechanisms that could explain this pattern, they then used ethnographic methods, including direct observations of drinking (“scans”) and debriefing interviews to discuss the earlier trial results. The researchers learn that sponsoring drinks confers prestige, but the stigma attached to drinking by outsiders likely discouraged reporting intentions to save for it. Research was funded by a grant from the World Bank Research Committee (RFP12503-RESEBBRSB) and the Bill and Melinda Gates Foundation. The authors thank Harry Patrinos (World Bank) for his support and patience, Dylan Ramshaw and Innovations for Poverty Action for managing fieldwork, Tomás Huanca and Esther Conde of CBIDSI and CBIDSI surveyors and translators for exceptional fieldwork, and Mike Feng, Zachary Groff, Michael Ioannou, Javier Madrazo, Diego Santa Maria, Glynis Startz, and Jonathan Hy Vayness for excellent research assistance. Institutional Review Board approval from Yale University (0805003895), Northwestern University (STU00208598) and Innovations for Poverty Action (SP201.2010.01.01). This trial is registered in the American Economic Association’s registry for randomized controlled trials (ID AEARCTR-0008670). Human behavior is complex and often confounds researchers. What is a researcher to do when confronted with a surprising result? Economists tend to search for alternative theories and probe for evidence of faulty data or identification assumptions. Increasingly, we may use Bayesian estimation and/or attempt to replicate the study. We take a complementary approach of using ethnographic methods to probe mechanisms. Our ethnography comes after a standard randomized controlled trial (RCT) setup: an intervention sandwiched between a baseline and an endline survey. Ethnographic and related qualitative methods are well established in other social sciences, particularly cultural anthropology. Whether such methods are commonly used in economics as we do here is difficult to quantify, but we can quantify whether such methods are discussed in publications. They are not, if the universe of 96 articles using RCTs published in the “top five” economics journals from January 2016 through August 2021 is a good indication (Table 1). We do not find any paper taking the approach we do here, of using qualitative methods to probe mechanisms, with Carneiro et al. (2021) coming closest in their use of "a parallel stream of qualitative analysis" to corroborate quantitative findings and develop hypotheses to test in future work (p. 2545).1 We started this project focused on identifying whether and how the introduction of a savings technology would affect economic outcomes and well-being in a relatively autarkic setting. We find some expected results, namely evidence that financial and physical assets increase. We also find some unexpected results: temptation spending and particularly hard alcohol consumption increase. This last result withstands additional statistical scrutiny; most compellingly, we find biometric evidence that blood pressure increases, and specifically on the sub-group (men) in which we observe the largest increase in alcohol consumption. The increase in alcohol consumption pushed against the theory and prior empirical evidence that motivated our initial study design. 1 Beyond the 96 papers, the closest economics paper we know of is Bergman et al. (2020), which uses indepth qualitative interviews to explore mechanisms underlying treatment effects that were not unexpected. See also Fryer (2011) and Allen et al. (2014) in economics. Even in public health, where researchers have used qualitative methods extensively to inform survey design, study recruitment, and outcome measurement, using ethnographic methods to help interpret results, as we do here, is “virtually unheard of” (Mannell et al. 2021, p. 19). Mannell et al. is an exception: they use debriefing interviews (but not scans) to probe mechanisms following a diabetes intervention in Bangladesh. Davis et al. (2019) also advocates using a subset of the follow-up methods we use here to probe mechanisms as part of a broader framework for more systematically integrating qualitative methods into public health RCT designs.