We investigate how good survey questions about different topics are, across different countries and languages. We particularly focus on estimating the measurement quality of survey questions (defined as the product of reliability and validity) by means of Multitrait-Multimethod (MTMM) experiments. Thus, we design and implement such MTMM experiments in several surveys, such as the European Social Survey, the Gesis Panel, the Norwegian Citizen Panel and many more. Data from these experiments are analysed using advanced structural equation models. Results are used to provide guidelines about how to best formulate survey questions in different contexts, but also to correct for the remaining measurement errors. Furthermore, we also investigate how to best design, estimate and test the fit of MTMM experiments.

Moreover, we do research on how to predict the measurement quality of survey question based on the question characteristics. This research has led to the Survey Quality Predictor (SQP) software, a free license online software that can predict the measurement quality of questions in more than 20 languages. We maintain and improve SQP constantly, by developing new features of the software, redefining the characteristics that are used for predictions, testing different methods of meta-analyses, etc. SQP is an ongoing research project in which anyone interested can collaborate. 

Research projects:

European Social Survey (ESS) ERIC - Core Scientific Team (CST) (2008-ongoing). Additional information

Survey Quality Predictor (SQP) (2008-2022). Additional information

Selected Publications:

Measurement quality

Schwarz, H., Weber, W., Minderop, I., Weiß, B. (2021) In Search of the Best Response Scale in a Mixed-mode Survey (Web and Mail). Evidence from MTMM Experiments in the GESIS Panel. methods, data, analyses, 15(2), 30. https://doi.org/10.12758/mda.2021.05

Bosch, O.J., Revilla, M., DeCastellarnau, A. and W. Weber (2018). Measurement reliability, validity and quality of slider versus radio button scales in an online probability-based panel in Norway. Social Science Computer Review. https://doi.org/10.1177/0894439317750089

DeCastellarnau, A. and M. Revilla. (2017). Two approaches to evaluate measurement quality in online surveys: An application using the Norwegian Citizen Panel. Survey Research Methods, 11(4), 415-433. https://doi.org/10.18148/srm/2017.v11i4.7226

Design and Estimation of MTMM Experiments

Revilla, M., Bosch, O.J., and W. Weber (2018). Unbalanced 3-group Split-Ballot Multitrait-Multimethod design? Structural Equation Modeling: A Multidisciplinary Journal. Published online first November, 5, 2018. https://doi.org/10.1080/10705511.2018.1536860

Saris, Willem, and Albert Satorra. (2018). “The Pooled Data Approach for the Estimation of Split-Ballot Multitrait–Multimethod Experiments.” Structural Equation Modeling: A Multidisciplinary Journal 25(5), 659–72. https://doi.org/10.1080/10705511.2018.1431543

Zavala-Rojas, D., Tormos, R., Weber, W., & Revilla, M. (2018). Designing response scales with multi-trait-multi-method experiments. Mathematical Population Studies, 25(2), 66–81. https://doi.org/10.1080/08898480.2018.1439241

Helm, J. L., Castro-Schilo, L., Zavala-Rojas, D., DeCastellarnau, A. & Oravecz, Z. (2017). Bayesian Estimation of the True Score Multitrait–Multimethod Model with a Split-Ballot Design. Structural Equation Modeling: A Multidisciplinary Journal. Access to the article

DeCastellarnau, A. (2017). A classification of response scale characteristics that affect data quality: a literature review. Quality & Quantity, 1–37. https://doi.org/10.1007/s11135-017-0533-4 

DeCastellarnau, A., & Revilla, M. (2017). Two Approaches to Evaluate Measurement Quality in Online Surveys: An Application Using the Norwegian Citizen Panel. Survey Research Methods, 11(4), 415–433. Access to the article

Saris, W.E. and Revilla, M. (2016) Correction for measurement errors in survey research: necessary and possible. Social Indicators Research, 127 (3), 1005–1020. https://doi.org/10.1007/s11205-015-1002-x