We investigate how good survey questions about different topics are, across different countries and languages. We particularly focus on estimating the measurement quality of survey questions (defined as the product of reliability and validity) by means of Multitrait-Multimethod (MTMM) experiments. Thus, we design and implement such MTMM experiments in several surveys, such as the European Social Survey, the Gesis Panel, the Norwegian Citizen Panel and many more. Data from these experiments are analysed using advanced structural equation models. Results are used to provide guidelines about how to best formulate survey questions in different contexts, but also to correct for the remaining measurement errors. Furthermore, we also investigate how to best design, estimate and test the fit of MTMM experiments.

 

Research projects:

European Social Survey (ESS) ERIC - Core Scientific Team (CST) (2008-ongoing). Additional information

 

Selected Publications:

Measurement quality

Schwarz, H., Weber, W., Minderop, I., Weiß, B. (2021) In Search of the Best Response Scale in a Mixed-mode Survey (Web and Mail). Evidence from MTMM Experiments in the GESIS Panel. methods, data, analyses, 15(2), 30. https://doi.org/10.12758/mda.2021.05

Bosch, O.J., Revilla, M., DeCastellarnau, A. and W. Weber (2018). Measurement reliability, validity and quality of slider versus radio button scales in an online probability-based panel in Norway. Social Science Computer Review. https://doi.org/10.1177/0894439317750089

DeCastellarnau, A. and M. Revilla. (2017). Two approaches to evaluate measurement quality in online surveys: An application using the Norwegian Citizen Panel. Survey Research Methods, 11(4), 415-433. https://doi.org/10.18148/srm/2017.v11i4.7226

 

Design and Estimation of MTMM Experiments

Revilla, M., Bosch, O.J., and W. Weber (2018). Unbalanced 3-group Split-Ballot Multitrait-Multimethod design? Structural Equation Modeling: A Multidisciplinary Journal. Published online first November, 5, 2018. https://doi.org/10.1080/10705511.2018.1536860

Saris, Willem, and Albert Satorra. (2018). “The Pooled Data Approach for the Estimation of Split-Ballot Multitrait–Multimethod Experiments.” Structural Equation Modeling: A Multidisciplinary Journal 25(5), 659–72. https://doi.org/10.1080/10705511.2018.1431543

Zavala-Rojas, D., Tormos, R., Weber, W., & Revilla, M. (2018). Designing response scales with multi-trait-multi-method experiments. Mathematical Population Studies, 25(2), 66–81. https://doi.org/10.1080/08898480.2018.1439241

Helm, J. L., Castro-Schilo, L., Zavala-Rojas, D., DeCastellarnau, A. & Oravecz, Z. (2017). Bayesian Estimation of the True Score Multitrait–Multimethod Model with a Split-Ballot Design. Structural Equation Modeling: A Multidisciplinary Journal. Access to the article

DeCastellarnau, A. (2017). A classification of response scale characteristics that affect data quality: a literature review. Quality & Quantity, 1–37. https://doi.org/10.1007/s11135-017-0533-4 

DeCastellarnau, A., & Revilla, M. (2017). Two Approaches to Evaluate Measurement Quality in Online Surveys: An Application Using the Norwegian Citizen Panel. Survey Research Methods, 11(4), 415–433. Access to the article

Saris, W.E. and Revilla, M. (2016) Correction for measurement errors in survey research: necessary and possible. Social Indicators Research, 127 (3), 1005–1020. https://doi.org/10.1007/s11205-015-1002-x