ESS Quality control
From the start the Central Coordinating Team (CCT) of the European Social Survey (ESS) was concerned with respect to the comparability of the results of the obtained data across countries. For this reason there was a group that controlled the comparability of the data collection procedures in the different countries. In fact the research in each country could not start before this group had signed off the specified procedure of that country. Also during the data collection process information about all steps had to be registered to control the quality of the process in order to determine the weighing procedure to obtain optimal comparable data across countries.
Comparability of questions
Another, at least as important process was the design of the English source questionnaire and translation of this questionnaire in the different languages. For the design of the source questionnaire a special group was created but for the translation process only a procedure, TRAP, developed by Janet Harkness and colleagues, was prescribed to be used in all countries. There was no instrument to control the comparability of the translations of the questionnaires in the different languages. The problem is of course that nobody is familiar with all possible languages that were used in Europe. Therefore the questionnaires could not be controlled on comparability by members of the CCT .
Quality of questions
On the other hand the quality of the source questionnaire was controlled using the program Survey Quality Prediction (SQP). This program predicts the quality of the formulated questions on the basis of the characteristics of the question of interest. To obtain the prediction of the quality the user only had to code more than 50 prespecified characteristics of the question. The predictions were based on the observed strong relationships between question characteristics and the quality of the questions evaluated in Mutitrait-Multimethod (MTMM) experiments developed by several researchers in different countries. Such MTMM experiments were also done in the ESS to get information about the quality of the ESS questions in the different countries. Below we see for question B7 in Round 1 of the ESS the formulated question about trust in the parliament. The quality of this question determined in the MTMM experiment, the predicted quality on the basis of an authorized coding of the characteristics of this question and an unauthorized prediction of another person. An authorized coding means that the researchers who developed SQP decided that this is the proper coding of the caracteristics of this question. An example of the coding procedure is presented below. The figure shows that the predicted quality of the question based on the authorized coding and the quality based on the MTMM experiment differ only minimally while the unauthorized prediction is a bit more different, possibly due to an error in the coding of the question..
Comparability of questions
Another, at least as important process was the design of the English source questionnaire and translation of this questionnaire in the different languages. For the design of the source questionnaire a special group was created but for the translation process only a procedure, TRAP, developed by Janet Harkness and colleagues, was prescribed to be used in all countries. There was no instrument to control the comparability of the translations of the questionnaires in the different languages. The problem is of course that nobody is familiar with all possible languages that were used in Europe. Therefore the questionnaires could not be controlled on comparability by members of the CCT .
Quality of questions
On the other hand the quality of the source questionnaire was controlled using the program Survey Quality Prediction (SQP). This program predicts the quality of the formulated questions on the basis of the characteristics of the question of interest. To obtain the prediction of the quality the user only had to code more than 50 prespecified characteristics of the question. The predictions were based on the observed strong relationships between question characteristics and the quality of the questions evaluated in Mutitrait-Multimethod (MTMM) experiments developed by several researchers in different countries. Such MTMM experiments were also done in the ESS to get information about the quality of the ESS questions in the different countries. Below we see for question B7 in Round 1 of the ESS the formulated question about trust in the parliament. The quality of this question determined in the MTMM experiment, the predicted quality on the basis of an authorized coding of the characteristics of this question and an unauthorized prediction of another person. An authorized coding means that the researchers who developed SQP decided that this is the proper coding of the caracteristics of this question. An example of the coding procedure is presented below. The figure shows that the predicted quality of the question based on the authorized coding and the quality based on the MTMM experiment differ only minimally while the unauthorized prediction is a bit more different, possibly due to an error in the coding of the question..
Comparing translations using SQP codes
At some moment, we realized that comparing the codes of the characteristics of the same question in the different countries with the codes obtained for the source questionnaire in English would provide an indication of the comparability of the formulation of the questions in the different languages. The coding has to be done by persons who speak the specific language but the codes could be used to determine if the formulations of the questions in the different countries differ from the formulations of the source questions on the 50 characteristics used in SQP. These 50 characteristics are important because they were found to determine the quality of the questions.
Irmtraud was asked to organize the coding of the questions involved in the MTMM experiments in the first three rounds.
Irmtraud´s evaluation study
First of all I coded the 54 questions of the source questionnaire once more and compared the results with the earlier coding by Willem to be sure that we agreed about the codes of the characteristics of the source questionnaire. Then I coded the questions in languages I was sufficiently familiar with to be able to code the questions.
This was done for the following languages: English, German, Dutch, French, Spanish, Portuguese and Italian. For the Nordic languages and the Eastern European countries and for Greek, Hebrew and Turkish I had to hire native speakers. In Barcelona enough native speakers in all these languages could be found at the Universities. I hired the students and gave them an instruction in coding using the SQP program. Then they started to code and I asked them to provide me with their codes of their first question to check whether they were doing a proper job. I checked that by comparing our codes of the first question of the source questionnaire with the codes of the same question in their language. If differences occurred these differences could be due to errors in the coding or to differences in the formulation of the question in their language. This had to be found out by discussing the differences. Students who did not code carefully enough had to stop, the others could start coding the other questions. For all questions we compared the codes. If we agreed on the reason for the difference their code was fixed in the program as the authorized code and so we obtained authorized codes for all questions that were evaluated in all languages that were used in the first three rounds of the ESS for MTMM questions.
Some results
It turned out that this coding was indeed a helpful tool to control the comparability of the translations of the source questionnaire because in all translations at least some deviations were detected. The two most striking cases were the following.
In one country all introductions and instructions were omitted in the questionnaires.
This was of course a serious deviation which was absolutely inacceptable.
In the source questionnaire the word "immigrant" was avoided by using the term
“ people who are coming to live here” .
This was done to avoid the possible negative connotation of “immigrant”.
In one country the translaters understood that with this long expression
“immigrants” were meant and used this term in their translation .
These errors were not detected without the coding. The coding procedure detected these errors and many others.
Signing off the questionnaires
The Central Coordinating Team of the ESS realized on the basis of this study that the questionnaires also had to be signed off before the countries could start with their data collection. A problem is that it is a very big task if all questions had to be coded in this way. Therefore the following procedure was developed.
In all countries the questions that were involved in the MTMM experiments had to be coded using SQP by the translator in the country. These codes were compared with the codes of the same questions of the source questionnaire. The differences were discussed with the National coordinator of the research. Only if an agreement was reached about the way to translate the questions between the national coordinator and the person in charge for this task the questionnaire was signed off and the data collection was allowed.
It will be clear that this procedure did not guarantee complete agreement between the translations and the source questionnaire but in this way we thought that the national coordinator would understand how the translation had to be done and would instruct the translator about this and check the full questionnaire before the data collection started.
This work for the CCT was done by Diana Zavalla-Rojas of RECSM, substituting me. I thought that I had done my job to design this procedure. Somebody else could perform this complex task in the next rounds.
It turned out that this coding was indeed a helpful tool to control the comparability of the translations of the source questionnaire because in all translations at least some deviations were detected. The two most striking cases were the following.
In one country all introductions and instructions were omitted in the questionnaires.
This was of course a serious deviation which was absolutely inacceptable.
In the source questionnaire the word "immigrant" was avoided by using the term
“ people who are coming to live here” .
This was done to avoid the possible negative connotation of “immigrant”.
In one country the translaters understood that with this long expression
“immigrants” were meant and used this term in their translation .
These errors were not detected without the coding. The coding procedure detected these errors and many others.
Signing off the questionnaires
The Central Coordinating Team of the ESS realized on the basis of this study that the questionnaires also had to be signed off before the countries could start with their data collection. A problem is that it is a very big task if all questions had to be coded in this way. Therefore the following procedure was developed.
In all countries the questions that were involved in the MTMM experiments had to be coded using SQP by the translator in the country. These codes were compared with the codes of the same questions of the source questionnaire. The differences were discussed with the National coordinator of the research. Only if an agreement was reached about the way to translate the questions between the national coordinator and the person in charge for this task the questionnaire was signed off and the data collection was allowed.
It will be clear that this procedure did not guarantee complete agreement between the translations and the source questionnaire but in this way we thought that the national coordinator would understand how the translation had to be done and would instruct the translator about this and check the full questionnaire before the data collection started.
This work for the CCT was done by Diana Zavalla-Rojas of RECSM, substituting me. I thought that I had done my job to design this procedure. Somebody else could perform this complex task in the next rounds.