Survey Experiments

On this page: 
       Course description
       Target audience
       Course objectives
       Prerequisites
       Course content
       Preparation for the course
       References
       Instructor’s Bio

Duration: 12 hours/3 days                                                                        

An hour-long lunch break from 13:00-14:00 with RECSM Winter School participants and members

Format

Hybrid (in-person and online)

Lecturer

Jordi Muñoz
 

Course Description

This is a short, hands-on course on the logic, types and practicalities of survey experimental research. The goal is that participants become familiar with the method, so that they can critically assess published survey-experimental work and design good survey experiments on their own research topics.
First, we discuss the notion of causality in social research and how randomized experiments allow us to identify causal effects, using the potential outcomes framework. We specify the conditions under which survey experiments estimate causal effects, and what randomization does and does not guarantee. Then, we introduce the history and the main types of survey experiments (question wording and framing, vignettes, priming and cue experiments, endorsement and list experiments for sensitive topics, factorial designs, unexpected events during surveys and conjoint experiments), mostly but not only from the political science literature. Through the review of multiple published examples, we learn to match each design to the research question it answers best. Finally, we cover the practical elements that we need to consider when designing, fielding and analyzing our own survey experiments: treatment design, outcome measurement, manipulation checks, pre-registration, power analysis, estimation of treatment effects and ethics.
The course combines short lectures with three in-class activities: dissecting a pre-registered survey experiment, drafting a design for a survey experiment in small groups, and analyzing survey-experimental data in R. Detailed and commented R code for randomization, power analysis and the estimation of various types of treatment effects will be shared. After the course, participants can submit a proposal for a survey experiment linked to their own research and receive individual feedback from the instructor.

Target Audience

The course is aimed at researchers who want to use survey experiments to answer causal questions, or who need to evaluate survey-experimental evidence. It is particularly suited for PhD students, postdoctoral and early-career researchers in political science, sociology, economics, communication, social psychology and public policy, but more senior researchers who are new to experimental methods will also benefit. It is also relevant for practitioners working in public opinion institutes, polling and market research firms, government agencies and NGOs who design or commission surveys and want to add experimental components to them.

Course objectives

The course aims to provide participants with the conceptual and practical tools needed to design, implement and analyze survey experiments, and to critically assess survey-experimental research.
By the end of the course participants will be able to:

• Explain why and when randomization identifies causal effects.
• Choose the survey-experimental design that best fits a given research question.
• Write valid, effective and precise treatments, and measure outcomes well.
• Compute statistical power and justify the sample size before fieldwork.
• Pre-register a design, including correctly used manipulation checks.
• Estimate and interpret ATEs, ITTs, CATEs, and AMCEs and marginal means in conjoint experiments.
• Recognize the ethical dimensions of experimenting on survey respondents.

Prerequisites

No prior experience with experiments is required. Participants should have a basic knowledge of statistics (descriptive statistics, hypothesis testing and linear regression).
Recommended: some familiarity with R (loading data, running a regression), since the hands-on analysis activity uses R, and a basic knowledge of survey research.

Course Content

The course combines lectures with in-class activities. Participants work in small groups that remain stable throughout the course. There is time every day for collective discussion of participants' research topics and proposals: on Day 1 groups choose and discuss a registered survey experiment related to their substantive interests; on Day 2 they draft a design for their own survey experiment; and on Day 3 the proposals are discussed in class, with feedback from the instructor and the other participants. These activities are the building blocks of an individual research proposal that participants can submit after the course to receive written feedback.

Preparation for the course

Participants must bring their own laptop with a recent version of R (4.3 or later) and RStudio installed. Both are free and open source, so no licences are needed. Stata users are welcome, but the course materials and code are in R.
Please install the following R packages in advance: tidyverse, pwr, estimatr, marginaleffects, cregg and DeclareDesign. Datasets, code and activity templates will be made available on the course platform before the sessions.

References

Suggested reading before the course (not compulsory):

• Mutz, Diana C. 2011. Population-Based Survey Experiments. Princeton University Press.
• EGAP Methods Guides. “10 Things to Know About Survey Experiments.” https://egap.org/resource/10-things-to-know-about-survey-experiments/

Further references (background and after-course reading; specific papers are also linked in the slides):

• Bansak, Kirk, Jens Hainmueller, Daniel J. Hopkins and Teppei Yamamoto. 2018. “The Number of Choice Tasks and Survey Satisficing in Conjoint Experiments.” Political Analysis 26(1):112–119.
• Bansak, Kirk, Jens Hainmueller, Daniel J. Hopkins and Teppei Yamamoto. 2021. “Beyond the Breaking Point? Survey Satisficing in Conjoint Experiments.” Political Science Research and Methods 9(1):53–71.
• Bansak, Kirk, Jens Hainmueller, Daniel J. Hopkins and Teppei Yamamoto. 2021. “Conjoint Survey Experiments.” In James N. Druckman and Donald P. Green (eds.), Advances in Experimental Political Science. Cambridge University Press.
• Barabas, Jason and Jennifer Jerit. 2010. “Are Survey Experiments Externally Valid?” American Political Science Review 104(2):226–242.
• Blair, Graeme, Alexander Coppock and Macartan Humphreys. 2023. Research Design in the Social Sciences: Declaration, Diagnosis, and Redesign. Princeton University Press.
• Blair, Graeme and Kosuke Imai. 2012. “Statistical Analysis of List Experiments.” Political Analysis 20(1):47–77.
• Clifford, Scott, Geoffrey Sheagley and Spencer Piston. 2021. “Increasing Precision without Altering Treatment Effects: Repeated Measures Designs in Survey Experiments.” American Political Science Review 115(3):1048–1065.
• Druckman, James N. and Donald P. Green (eds.). 2021. Advances in Experimental Political Science. Cambridge University Press.
• Franco, Annie, Neil Malhotra and Gabor Simonovits. 2014. “Publication Bias in the Social Sciences: Unlocking the File Drawer.” Science 345(6203):1502–1505.
• Gaines, Brian J., James H. Kuklinski and Paul J. Quirk. 2007. “The Logic of the Survey Experiment Reexamined.” Political Analysis 15(1):1–20.
• Gerber, Alan S. and Donald P. Green. 2012. Field Experiments: Design, Analysis, and Interpretation. W.W. Norton.
• Hainmueller, Jens, Dominik Hangartner and Teppei Yamamoto. 2015. “Validating Vignette and Conjoint Survey Experiments against Real-World Behavior.” Proceedings of the National Academy of Sciences 112(8):2395–2400.
• Imai, Kosuke, Luke Keele, Dustin Tingley and Teppei Yamamoto. 2011. “Unpacking the Black Box of Causality: Learning about Causal Mechanisms from Experimental and Observational Studies.” American Political Science Review 105(4):765–789.
• Kuhn, Patrick M. and Nick Vivyan. 2022. “The Misreporting Trade-Off between List Experiments and Direct Questions in Practice: Partition Validation Evidence from Two Countries.” Political Analysis 30(3):381–402.
• Leeper, Thomas J., Sara B. Hobolt and James Tilley. 2020. “Measuring Subgroup Preferences in Conjoint Experiments.” Political Analysis 28(2):207–221.
• Montgomery, Jacob M., Brendan Nyhan and Michelle Torres. 2018. “How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It.” American Journal of Political Science 62(3):760–775.
• Muñoz, Jordi, Albert Falcó-Gimeno and Enrique Hernández. 2020. “Unexpected Event during Survey Design: Promise and Pitfalls for Causal Inference.” Political Analysis 28(2):186–206.
• Riambau, Guillem and Kai Ostwald. 2021. “Placebo Statements in List Experiments: Evidence from a Face-to-Face Survey in Singapore.” Political Science Research and Methods 9(1):172–179.
• Sniderman, Paul M. 2011. “The Logic and Design of the Survey Experiment: An Autobiography of a Methodological Innovation.” In James N. Druckman et al. (eds.), Cambridge Handbook of Experimental Political Science. Cambridge University Press.

Instructor’s Bio

​​​​​​​​​​​​​​​

Jordi Muñoz is Associate Professor of Political Science at the University of Barcelona and a member of the Institutions and Political Economy Research Group (IPERG). Between 2021 and 2024 he was the director of the Centre d'Estudis d'Opinió (CEO), the public opinion institute of the Government of Catalonia. He teaches quantitative methods, comparative politics and political behavior. His research focuses on political behavior and public opinion, using surveys, survey experiments and historical data. His work using survey experiments, on topics such as corruption and voting, immigration and welfare, coalition signals, political violence, terrorism, COVID-19 and support for independence, has been published in journals such as the Journal of Politics, Political Analysis, Comparative Political Studies, Political Science Research and Methods, the Journal of Peace Research, the European Political Science Review and Party Politics. More information: www.jordimunoz.cat.