A scientific crisis changed my life
Willem´s research group had clearly shown that people express themselves in different ways in survey research. We called this “variation in response function”. We had also shown that this leads to observation of large supposed “errors” in survey responses and to relative large effects on the correlations between the variables which are the basis for the study or relationships between these variables. Especially the last problems made me think how we ever could trust correlations anymore and therefore our estimates of relations between variables in substantive research. Our successful research on this topic led in my mind to a real scientific crisis. First of all I did not see a solution to this problem. We had seen that we could reduce the effect of variation of response functions but not completely prevent it. All the work on structural equation models to estimate relations between variables was based on the analysis of correlations and covariances. However, if this information can´t be trusted what sense does it make to use these models to estimate relationships between the variables of interest. This problem kept me busy for some time in 1988.
Had Frank Andrews suggested the solution?
In an earlier story I have told that Frank Andrews from the University of Michigan had asked me in 1979 for help with the analysis of a correlation matrix that was generated in a survey where people were asked to give their opinions about three issues measured in three different ways. At that time I just looked at his problem as a statistical problem of model testing and did not see the relevance of his research. Also when he published his research in 1984 I was not alert enough to realize how relevant this approach was for social science research.
This approach was known in social science research as Multi-Trait Multi-Method or MTMM approach. So far this approach was used in psychology for complex concepts (traits) and complex measurement procedures or for evaluation of people by different judges.
New was in the research of Frank Andrews that he measured different opinions with different question forms. In this way one gets the correlation between 9 observed variables (each a combination of an opinion and a question form).
For each question form normally the observed correlations are different. These differences in correlations are due to differences in the measurement procedures used. The MTMM model allowed to estimate the correlations between the variables of interest corrected for the measurement errors caused by the methods used.
In 1988, during my scientific crisis situation, the memory of this approach came up in my mind again. Therefore I read his paper again. I was impressed by his work because he not only did a single experiment but several experiments. In each experiment he got an estimate of the qualities of the question forms used and using that information he could derive the correlation between the variables of interest corrected for measurement errors (lack of quality) of the different question forms. He even could suggest which question forms were better that others.
In 1988, during my scientific crisis situation, the memory of this approach came up in my mind again. Therefore I read his paper again. I was impressed by his work because he not only did a single experiment but several experiments. In each experiment he got an estimate of the qualities of the question forms used and using that information he could derive the correlation between the variables of interest corrected for measurement errors (lack of quality) of the different question forms. He even could suggest which question forms were better that others.
I saw a light at the horizon
I was impressed by his research but wanted to apply this approach also myself. So I looked in my research data files for a similar experiment and I found one which had the same structure as his experiments. During my stay at the Essex summer school where I was teaching again two courses I spent during the whole month July the afternoons and the many evenings on the analysis of the correlation matrices of the MTMM experiments I had detected in my computer. It turned out that this was not enough. I had serious problems with the analysis of the correlation matrices I had. At home I continued my analyses but I continued to have problems with the analysis: the estimation did not converse or I got negative variances what is impossible. This made me wonder whether this was indeed the best approach.
I was impressed by his research but wanted to apply this approach also myself. So I looked in my research data files for a similar experiment and I found one which had the same structure as his experiments. During my stay at the Essex summer school where I was teaching again two courses I spent during the whole month July the afternoons and the many evenings on the analysis of the correlation matrices of the MTMM experiments I had detected in my computer. It turned out that this was not enough. I had serious problems with the analysis of the correlation matrices I had. At home I continued my analyses but I continued to have problems with the analysis: the estimation did not converse or I got negative variances what is impossible. This made me wonder whether this was indeed the best approach.
What to do?
With the knowledge I have now of this approach I would have said that it made no sense to analyze these correlation matrices because something unusual must have happened in the data collection otherwise the correlations obtained were not possible. At that moment I didn´t know that and I wanted to give up the idea of the MTMM approach but I needed communication with other specialists, especially Frank Andrews, about the problems I had observed. I contacted him and asked him if he liked to come to the Netherlands in the spring of 1989 to have a meeting with us about the MTMM approach and possible alternative approaches to correct observed correlation matrices for measurement errors. He was very interested to come, therefor I decided to organize a conference on this topic.
With the knowledge I have now of this approach I would have said that it made no sense to analyze these correlation matrices because something unusual must have happened in the data collection otherwise the correlations obtained were not possible. At that moment I didn´t know that and I wanted to give up the idea of the MTMM approach but I needed communication with other specialists, especially Frank Andrews, about the problems I had observed. I contacted him and asked him if he liked to come to the Netherlands in the spring of 1989 to have a meeting with us about the MTMM approach and possible alternative approaches to correct observed correlation matrices for measurement errors. He was very interested to come, therefor I decided to organize a conference on this topic.
The first MTMM conference
I asked subsidy for the meeting from the Dutch Science Foundation which took place in the building of the Royal Dutch Academy of Arts and Sciences. The participants were people who I expected would contribute to our discussions about the design, analysis and meta-analysis of MTMM experiments. There were besides Frank Andrews (USA), Albert Satorra (Spain) , Richard Költringer (Austria) and from the Netherlands Don Mellenbergh, Nico Molenaar and Willem Saris.
I asked subsidy for the meeting from the Dutch Science Foundation which took place in the building of the Royal Dutch Academy of Arts and Sciences. The participants were people who I expected would contribute to our discussions about the design, analysis and meta-analysis of MTMM experiments. There were besides Frank Andrews (USA), Albert Satorra (Spain) , Richard Költringer (Austria) and from the Netherlands Don Mellenbergh, Nico Molenaar and Willem Saris.
It was a very successful meeting. Frank Andrews clarified how he analyzed the MTMM data therefore my proposal for a more complex design was not necessary anymore. Nico Molenaar made suggestions how to formulate the questions and Richard Költringer , Albert Satorra and Don Mellenbergh discussed with us some important statistical aspects that had to be taken into account.
Th next steps
About this conference a book has been published but before the book was ready Lex van Meurs and I did an important study about memory effects in MTMM experiments. This was relevant because approximately the same questions had to be repeated and memory effects could disturb the results. Therefore we did an experiment to determine how long the time should be between a repetition of a question to avoid memory effects. It turned out that in 25 minutes for must survey questions the memory of a previous answer is gone which means the repetition of approximately the same questions could be done within interviews which were longer than 25 minutes. Together with Lex van Meurs I edited the book about the conference plus this last contributed paper.
This very informative meeting gave me sufficient trust in this approach to continue using this approach and to start an International research group on MTMM experiments which had meetings to discuss results and problems of MTMM experiments and more general measurement issues during more than 10 years.
Th next steps
About this conference a book has been published but before the book was ready Lex van Meurs and I did an important study about memory effects in MTMM experiments. This was relevant because approximately the same questions had to be repeated and memory effects could disturb the results. Therefore we did an experiment to determine how long the time should be between a repetition of a question to avoid memory effects. It turned out that in 25 minutes for must survey questions the memory of a previous answer is gone which means the repetition of approximately the same questions could be done within interviews which were longer than 25 minutes. Together with Lex van Meurs I edited the book about the conference plus this last contributed paper.
This very informative meeting gave me sufficient trust in this approach to continue using this approach and to start an International research group on MTMM experiments which had meetings to discuss results and problems of MTMM experiments and more general measurement issues during more than 10 years.