(Syllabus last updated: 2019-November-21)
Class meetings: Tuesdays, 12.30-3.30pm Demography Seminar Room
Office hours: by appointment (please send me an email and we can find a time)
Email: feehan [at] berkeley.edu
Web: www.dennisfeehan.org/teaching/2019fa_demog260.html
Piazza: https://bit.ly/2lsHm8p
This class will cover topics in the design and implementation of field research for quantitative social science. We will consider studies whose aim is measurement (including surveys), causal inference (including experiments), and prediction (including machine learning). Our goal is to focus on issues of study design and execution, rather than the details of specific tools or methodologies. We will consider examples from a diverse range of different ‘fields’, both digital and physical. Our approach will be to conceptually understand key design and implementation issues for several different types of study design, but it will often be necessary to sacrifice depth for breadth. Thus, our discussion of each design will provide interested students with a useful starting point for deeper study. The class is designed for graduate students who have had at least some exposure to statistics, and who are starting to develop field research projects of their own.
Please re-check the syllabus before you start each week’s reading; it will be updated as the semester progresses
| Week | Date | Theme | Topic |
|---|---|---|---|
| 1 | 2019-09-03 | Course overview and background | Overview; Introduction to sampling |
| 2 | 2019-09-10 | Designing measurement with surveys | Fielding a survey in the real world: modes, interviewer training, questionnaire design, cost structure, etc |
| 3 | 2019-09-17 | Complex sampling: stratification, clustering, and unequal probabilities of selection | |
| 4 | 2019-09-24 | Complex sampling: multi-stage designs | |
| 5 | 2019-10-01 | Sample size estimation; First set of presentations | |
| 6 | 2019-10-08 | Survey ethics and the IRB; Field operations; Weighting a survey: non-response, post-stratification, calibration, MRP | |
| 7 | 2019-10-15 | (No class) | |
| 8 | 2019-10-22 | Guest: Daniel Schneider | |
| 9 | 2019-10-29 | Digital data collection | Second set of presentations |
| 10 | 2019-11-05 | Linking different data sources | Record linkage framework; errors and inference |
| 11 | 2019-11-12 | Record linkage lab | |
| 12 | 2019-11-19 | Third set of presentations | |
| 13 | 2019-11-26 | Guest; Designing experiments and prediction | Power; manipulation checks, pre-registration; prediction and generalization vs parameter estimates |
| 14 | 2019-12-03 | Meet about projects |
The requirements of the class are designed to achieve two goals: the first goal is to become familiar with some classic and contemporary issues in the design and implementation of quantitative social science research projects; and the second goal is to write a proposal for a study design. You should think of the proposed design as the first draft of a project that you might be able to continue working on beyond this class.
Reading and class participation
Each week, you should read the assigned materials and show up to class prepared to discuss them. Some of the reading will be listed as background reading, which generally means that I will lecture on it. It’s up to you whether you want to read background reading before the class or not. There will not be lots of reading in the class, but it’s important to read what is assigned.
Papers to present
For 3 or 4 of the classes, there will be a list of papers that students can choose to present from. Each student will pick one of the papers and present it to the class. These presentations should be around 10 minutes each, and they should try to explain the methods discussed in the papers in a way that is intelligble to the rest of the class. You should plan to make slides or to go through details on the board to help explain the proposed approach. The goals of these presentations are (1) to give you some practice taking a deep dive into a technical paper; (2) to give you some practice distilling technical results to an audience; (3) to give you and the class some exposure to cutting edge ideas in research design.
Homework/labs
For each of the main segments of the class, I’ll release a homework/lab. You will be asked to submit these, but I will not grade them (other than noting whether or not you attempted everything). I’ll release solutions that you can refer to after they are submitted. These assignments will be aimed at helping build intuition and practically applying what we’ve discussed in class. I hope that they may be a helpful reference later on in your research careers, when you make use of these ideas.
Final paper
You will write a short (~10-15 pages) research proposal to conclude the class. The proposal will focus on the methodological aspects of a study that you wish to conduct. It should have a lit review, a description of the data or proposed data collection, and a discussion of how the data will be analyzed. If you have preliminary results, that is great (but it is not necessary). Your final paper should identify an important problem to be studied, briefly review the related literature, describe your proposed research design, and either present some preliminary empirical findings or describe how analysis will proceed once data have been collected.
The purpose of this paper is to connect the topics of this class to your actual research, so my hope is that this will be an opportunity to get some feedback on an idea you care about, and that you might continue to pursue beyond class.
The paper is due on Wednesday, December 18th.
NB: Please read each week’s articles in the order they are listed on the syllabus
Tue, Sep 3
Background for lecture:
Tue, Sep 10 - Fielding a survey
Background for lecture:
Tue, Sep 17 - Complex sampling designs
Background for lecture:
Topics for lecture:
Possible papers to present on Tue, Oct 1:
NB: In order to claim the specific paper you want to present, please post to the Piazza thread
STM for open-ended survey responses
- Margaret E. Roberts et al., “Structural Topic Models for Open-Ended Survey Responses,” American Journal of Political Science 58, no. 4 (2014): 1064–1082, https://onlinelibrary.wiley.com/doi/abs/10.1111/ajps.12103.
List experiments
- Graeme Blair and Kosuke Imai, “Statistical Analysis of List Experiments,” Political Analysis 20, no. 1 (2012): 47–77, https://www.cambridge.org/core/journals/political-analysis/article/statistical-analysis-of-list-experiments/6AEE6C9D3AB6DA410D602CB035D5959A.
- Graeme Blair, Kosuke Imai, and Jason Lyall, “Comparing and Combining List and Endorsement Experiments: Evidence from Afghanistan,” American Journal of Political Science 58, no. 4 (2014): 1043–1063, https://onlinelibrary.wiley.com/doi/abs/10.1111/ajps.12086.
- Adam N. Glynn, “What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment,” Public Opinion Quarterly 77, no. S1 (2013): 159–172.
- Jeffrey R. Lax, Justin H. Phillips, and Alissa F. Stollwerk, “Are Survey Respondents Lying About Their Support for Same-Sex Marriage? Lessons from a List Experiment,” Public Opinion Quarterly 80, no. 2 (January 2016): 510–533, https://academic.oup.com/poq/article/80/2/510/2588811. (Combine with methodological paper)
Survey weighting and regression models
- Gary Solon, Steven J. Haider, and Jeffrey M. Wooldridge, “What Are We Weighting for?” Journal of Human Resources 50, no. 2 (March 2015): 301–316, https://www.nber.org/papers/w18859.pdf.
- Andrew Gelman, “Struggles with Survey Weighting and Regression Modeling,” Statistical Science 22, no. 2 (May 2007): 153–164, https://projecteuclid.org/euclid.ss/1190905511. NB: the Gelman paper is part of a special issue of Statistical Science that has many comments and a rejoinder.
Time-Location Sampling
- John M. Karon and Cyprian Wejnert, “Statistical Methods for the Analysis of TimeLocation Sampling Data,” Journal of Urban Health 89, no. 3 (June 2012): 565–586, https://doi.org/10.1007/s11524-012-9676-8.
- Lucie Leon, Marie Jauffret-Roustide, and Yann Le Strat, “Design-Based Inference in Time-Location Sampling,” Biostatistics 16, no. 3 (July 2015): 565–579, https://academic.oup.com/biostatistics/article/16/3/565/269802.
Tue, Sep 24 - Complex sampling designs, cont
Background for lecture:
Tue, Oct 1 - Designing a survey and survey paper presentations
We’ll have our first round of paper presentations today.
Background for lecture on basic sample size calculations:
Tue, Oct 8 - Weighting a survey, non-response, calibration and post-stratification
Ethics and IRB reading for discussion:
Background for lecture on weighting and calibration:
Background for response rates:
Tue, Oct 15 - Class cancelled
Prof. Schneider will visit and discuss the design of the SHIFT project, an online survey that he and Prof. Kristen Harknett (UCSF) have been developed and implemented.
Possible papers to present on Tue, Oct 29:
NB: In order to claim the specific paper you want to present, please post to the Piazza thread
Anchoring Vignettes
- Daniel J. Hopkins and Gary King, “Improving Anchoring Vignettes: Designing Surveys to Correct Interpersonal Incomparability,” Public Opinion Quarterly 74, no. 2 (January 2010): 201–222, https://academic.oup.com/poq/article/74/2/201/1936649.
- Gary King and Jonathan Wand, “Comparing Incomparable Survey Responses: Evaluating and Selecting Anchoring Vignettes,” Political Analysis 15, no. 1 (n.d.): 46–66, https://www.cambridge.org/core/journals/political-analysis/article/comparing-incomparable-survey-responses-evaluating-and-selecting-anchoring-vignettes/AA12FD1F8DD8BF9A58BA10EC89C5F3B1.
Survey experiments
- Luke W. Miratrix et al., “Worth Weighting? How to Think About and Use Weights in Survey Experiments,” Political Analysis 26, no. 3 (July 2018): 275–291, https://www.cambridge.org/core/journals/political-analysis/article/worth-weighting-how-to-think-about-and-use-weights-in-survey-experiments/C22485D7BA0F76400B75E02559F110E2.
Wiki surveys
- Matthew J. Salganik and Karen EC Levy, “Wiki Surveys: Open and Quantifiable Social Data Collection,” PloS One 10, no. 5 (2015): e0123483, http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0123483.
Non-probability sampling
- Reg Baker et al., “Summary Report of the AAPOR Task Force on Non-Probability Sampling,” Journal of Survey Statistics and Methodology (2013): smt008, http://jssam.oxfordjournals.org/content/early/2013/09/26/jssam.smt008.
- Courtney Kennedy et al., Evaluating Online Nonprobability Surveys, May 2016, https://www.pewresearch.org/methods/2016/05/02/evaluating-online-nonprobability-surveys/.
- Andrew Mercer, Arnold Lau, and Courney Kennedy, For Weighting Online Opt-in Samples, What Matters Most?, January 2018, https://www.pewresearch.org/methods/2018/01/26/for-weighting-online-opt-in-samples-what-matters-most/.
- David Dutwin and Trent D. Buskirk, “Apples to Oranges or Gala Versus Golden Delicious?Comparing Data Quality of Nonprobability Internet Samples to Low Response Rate Probability Samples,” Public Opinion Quarterly 81, no. S1 (April 2017): 213–239, https://academic.oup.com/poq/article/81/S1/213/3749202.
Multilevel Regression and Poststratification
- Yair Ghitza and Andrew Gelman, “Deep Interactions with MRP: Election Turnout and Voting Patterns Among Small Electoral Subgroups,” American Journal of Political Science 57, no. 3 (2013): 762–776, https://onlinelibrary.wiley.com/doi/abs/10.1111/ajps.12004.
- Matthew K. Buttice and Benjamin Highton, “How Does Multilevel Regression and Poststratification Perform with Conventional National Surveys?” Political Analysis 21, no. 4 (2013): 449–467, https://www.jstor.org/stable/24572674.
Differential privacy
- John M. Abowd and Ian M. Schmutte, “An Economic Analysis of Privacy Protection and Statistical Accuracy as Social Choices,” American Economic Review 109, no. 1 (January 2019): 171–202, https://www.aeaweb.org/articles?id=10.1257/aer.20170627.
- Steven Ruggles et al., “Differential Privacy and Census Data: Implications for Social and Economic Research,” AEA Papers and Proceedings 109 (May 2019): 403–408, https://www.aeaweb.org/articles?id=10.1257/pandp.20191107.
- Steven Ruggles and Task Force on Differential Privacy for Census Data, “Implications of Differential Privacy for Census Bureau Data and Scientific Research” (December 2018), https://assets.ipums.org/_files/mpc/MPC-Working-Paper-2018-6.pdf.
Miscellaneous
- Graeme Blair et al., “Declaring and Diagnosing Research Designs,” American Political Science Review 113, no. 3 (August 2019): 838–859, https://www.cambridge.org/core/journals/american-political-science-review/article/declaring-and-diagnosing-research-designs/3CB0C0BB0810AEF8FF65446B3E2E4926.
Tue, Nov 5 Lecture on Fellegi-Sunter framework
Background for lecture:
Ivan P. Fellegi and Alan B. Sunter, “A Theory for Record Linkage,” Journal of the American Statistical Association 64, no. 328 (December 1969): 1183–1210, https://www.tandfonline.com/doi/abs/10.1080/01621459.1969.10501049.
Tue, Nov 12 In-class lab on record linkage
Please be sure to bring your laptops; we’ll be using RStudio for the lab.
Materials TBA
Tue, Nov 19 Third round of paper presentations
Possible papers to present on Tue, Nov 19:
NB: In order to claim the specific paper you want to present, please post to the Piazza thread
Applications of record linkage
- Kristian Lum, Megan Emily Price, and David Banks, “Applications of Multiple Systems Estimation in Human Rights Research,” The American Statistician 67, no. 4 (November 2013): 191–200, https://amstat.tandfonline.com/doi/full/10.1080/00031305.2013.821093.
- Nir Grinberg et al., “Fake News on Twitter During the 2016 U.S. Presidential Election,” Science 363, no. 6425 (January 2019): 374–378, https://science.sciencemag.org/content/363/6425/374.
Custom online data collection
- Gary King, Jennifer Pan, and Margaret E. Roberts, “How Censorship in China Allows Government Criticism but Silences Collective Expression,” American Political Science Review 107, no. 2 (May 2013): 326–343, https://www.cambridge.org/core/journals/american-political-science-review/article/how-censorship-in-china-allows-government-criticism-but-silences-collective-expression/C7EF4A9C9D59425C2D09D83742C1FE00.
Using search query data
- Rediet Abebe et al., “Using Search Queries to Understand Health Information Needs in Africa,” arXiv:1806.05740 [Cs] (June 2018), http://arxiv.org/abs/1806.05740.
Inference
- Andrew Gelman and John Carlin, “Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors,” Perspectives on Psychological Science 9, no. 6 (2014): 641–651, https://journals.sagepub.com/doi/full/10.1177/1745691614551642.
- Andrew Gelman and David Weakliem, “Of Beauty, Sex and Power: Too Little Attention Has Been Paid to the Statistical Challenges in Estimating Small Effects,” American Scientist 97, no. 4 (2009): 310–316, https://www.jstor.org/stable/27859361?casa_token=TOTDYnte9l0AAAAA:ev_c58LGBDS2Q6tmLy4Ww7FfrIczgB-Krv77a--aV_huVfpF9nr65tYH6t2IO0cTYcIqhcGiUdDhG9J8Tfe8Cw6c_aZ-ppTAQm1vjUiPHS2efXM48wwO\&seq=1\#metadata_info_tab_contents.
Background for lecture:
Useful resources on experimental design from JPAL:
Ugur Yildirim will visit and discuss the design and implementation of two online experiments.
Religious Accommodations
Requests to accommodate a student’s religious creed by scheduling tests or examinations at alternative times should be submitted directly to the instructor. Reasonable common sense, judgment and the pursuit of mutual goodwill should result in the positive resolution of scheduling conflicts. The regular campus appeals process applies if a mutually satisfactory arrangement cannot be achieved.
Statement on Academic Freedom
Both students and instructors have rights to academic freedom. Please respect the rights of others to express their points of view in the classroom.
DSP Accommodations
Please see the instructor to discuss accommodations for physical disabilities, medical disabilities and learning disabilities.
Student Resources
The Student Learning Center provides a wide range of resources to promote learning and academic success for students. For information regarding these services, please consult the Student Learning Center Website: https://slc.berkeley.edu/
Academic Integrity
The high academic standard at the University of California, Berkeley, is reflected in each degree that is awarded. As a result, every student is expected to maintain this high standard by ensuring that all academic work reflects unique ideas or properly attributes the ideas to the original sources.
These are some basic expectations of students with regards to academic integrity: