Back Correcting for measurement error in surveys comparing data between countries is essential for reliability

Correcting for measurement error in surveys comparing data between countries is essential for reliability

Research conducted by two researchers from the RECSM-UPF has used an innovative tool to correct measurement error, the Survey Quality Predictor (SQP, sqp.upf.edu), and they applied it to the 2006 European Social Survey (ESS). The study, published in the journal PLOS ONE, seeks to determine whether attitudes towards immigration on the concept of “perceived ethnic threat” as measured in this edition of the ESS can be compared between the countries involved in the survey. According to the findings, Denmark, Estonia and France would not be able to be compared with other participating countries without having corrected for measurement error, an essential aspect for consideration to improve quality standards in public opinion surveys and studies.

28.10.2020

Imatge inicial

The availability of international data to facilitate comparison between countries is on the increase, and in today’s society this may help, through surveys, to measure such issues as citizens’ attitudes towards immigration, political trust or universal human values. However, although international comparisons provide many advantages, they are not without enormous challenges: before cross-national comparisons can be made, measurement equivalence between countries has to be established, as not doing so can lead to biased comparisons between different groups or countries and to erroneous conclusions.

Before cross-national comparisons can be made, measurement equivalence between countries has to be established.

In this regard, much emphasis has been placed on the importance of testing for “measurement invariance”, i.e. checking whether the concept of interest is measured in the same way across different groups. But little attention has been paid to how measurement error can affect substantial conclusions and measurement invariance, which is a fundamental aspect for the reliability and quality of a comparative survey.

In a study published recently in the journal PLOS ONE, André Pirralha and Wiebke Weber, members of the Research and Expertise Centre for Survey Methodology (RECSM) at the UPF Department of Political and Social Sciences, tested for measurement equivalence using an innovative program, the Survey Quality Predictor (SQP), to correct also for measurement error. They used real survey data, the third round (2006) of the European Social Survey (ESS), that measures the concept of “perceived ethnic threat” and made a comparison between the participating countries.

Correcting measurement error may change the results of the European Social Survey

The aim of the research is twofold: first, to determine whether attitudes towards immigration arising from the concept of “perceived ethnic threat” analysed in the ESS are invariantly measured between these countries (i.e., if in order to measure this parameter, reliable bases have been established that allow comparability between countries), and secondly, to show the extent to which correction for measurement error affects measurement invariance and the comparative results.

“In our research, we compared the measurement invariance results in this 2006 round of the ESS before and after correction for measurement error using our SQP tool. We concluded that there are reasons to believe that measurement error correction can change some of the conclusions of this European survey”, the RECSM-UPF researchers state.

The researchers showed that after analysing the ESS without correcting the measurement error, Denmark, Estonia and France could not be compared with the other countries.

Specifically, the researchers showed that after analysing the ESS without correcting the measurement error, Denmark, Estonia and France could not be compared with the other countries (i.e., they were non-invariant). However, after correcting for measurement error, by disentangling the cognitive part related to the concept of “perceived ethnic threat” from the measurement process, they demonstrated that these groups were partially invariant.

Therefore, as the survey proposes, the attitudes of these three countries could not be compared to other European countries in terms of perceived ethnic threat. “This is highly significant because we use data from the ESS, probably one of the transnational surveys to devote most resources to ensuring intercultural comparability, and yet the measurement error still occurs”, the authors assure.

According to the researchers, it is essential to ensure that when two groups really cannot be compared in a survey (non-invariance), and the differences they have on the same construct cannot be corrected, it is due to the fact that the groups have a different understanding of this concept, and not because there are methodological differences caused by the measurement instruments: “Correction for measurement error should be increasingly regarded as a valid avenue to improve both survey and substantial research standards”, the authors conclude.

SQP, an innovative tool developed by RECSM-UPF to correct measurement error

The researchers, as they state in the article published in PLOS ONE, are aware that the Survey Quality Predictor (SQP) program has some limitations, and that it still does not perfectly predict measurement quality. For now, it is only available for European countries and languages.

The SQP program is based on an extensive open access database of survey questions with quality predictions created through user collaboration, on many different issues, in a variety of formats and languages. It is a coding system of formal and linguistic characteristics of the survey questions which yields a prediction of their reliability, validity and quality.

It is an ongoing research project, which is updated continuously and this affects quality predictions, whose accuracy increases as more information is used. However, “even though SQP has in itself limitations, in the absence of any other available source of measurement quality, it allows correcting for measurement error in survey data”, they assert.

Reference article: “Pirralha, A., Weber, W (October 2020). "Correction for measurement error in invariance testing: An illustration using SQP”. PLOS ONE

https://doi.org/10.1371/journal.pone.0239421

Multimedia

Categories:

SDG - Sustainable Development Goals:

Els ODS a la UPF

Contact

For more information

News published by:

Communication Office