Episode 2.4: Randomizing Together (Part 2), with Tara Slough and Graeme Blair

Today’s episode is Part 2 of our conversation about metaketas with Dr. Tara Slough, an Assistant Professor of Politics at NYU, who co-led with Daniel Rubenson a metaketa on the governance of natural resources that was published this year in PNAS; and Dr. Graeme Blair, an Assistant Professor of Political Science at UCLA, who co-led a metaketa with Fotini Christia and Jeremy Weinstein testing the effects of community policing. The main paper from that project was just published last month in Science.

In Part 1, we learned what a metaketa is, how it’s typically organized, and what the benefits are of having coordinated teams experimentally test the same (or very similar) interventions across multiple contexts. We also talked about each of the two EGAP metaketas that Tara and Graeme co-led – the first on natural resource governance and the second on community policing. 

In today’s episode, we talk with Tara and Graeme about deeper conceptual issues, practical constraints, and equity considerations around metaketas. It’s fairly simple to interpret the results if we find the same effect across settings -- but what do we conclude if we see different treatment effects across the different sites? We also ask how far metaketas can get us toward generalizability -- it’s one thing to compare results across 6 test sites, but can we extrapolate to other contexts outside of the metaketa?

And while metaketas are a powerful tool, we also learn from Tara and Graeme about their challenges and limitations. What was it like coordinating across six research teams, all with their own local constraints, timelines, and publication incentives? What are the equity concerns that come up when so many resources are allocated to a single question? And we talk about the professional considerations that scholars, particularly junior scholars, should keep in mind when signing up to participate in a metaketa.

Now, here’s Part 2 of our conversation with Tara Slough and Graeme Blair. As a reminder, we left off in Part 1 discussing how to pool estimates across study sites to get an average treatment effect. This is where we pick up the conversation. 

Works discussed in the episode:

Blair, Graeme and Gwyneth McClendon. “Experiments in Multiple Contexts.” In Advances in Experimental Political Science, James N. Druckman and Donald P. Green eds., 2021. Cambridge: Cambridge University Press.

Slough, Tara and Scott A. Tyson. 2021. "External Validity and Meta-Analysis". Working Paper. Available at http://taraslough.com/assets/pdf/ev_ma.pdf