Our work got recognition
Around 1980 Irmtraud had realized a big success by detecting rules for argumentation which were not mentioned by the decision makers. These rules could in nearly 100% of the cases correctly predict the choice they made based on the description of the decision problem. This test was done on the basis of a sample of decisions made by the Dutch government in the 20th century. When this work was published she wanted to study whether in other countries the arguments for decisions were formulated in the same way. For that she started to collect data concerning three very influential decisions during the last century: the first and second World War and the Cuba missiles crisis. First, she had to collect the documents and then she had to analyze them. This takes of course some time.
Willem had obtained some recognition for the work he did by introducing the LISREL approach for data analysis in the social science. Especially the work he did on detecting misspecifications in models seemed to be received well because he got several requests for help in analyses of data. Some interesting examples which led to long term cooperation will be mentioned here. Besides that he developed some new activities with respect to measurement.
Willem had obtained some recognition for the work he did by introducing the LISREL approach for data analysis in the social science. Especially the work he did on detecting misspecifications in models seemed to be received well because he got several requests for help in analyses of data. Some interesting examples which led to long term cooperation will be mentioned here. Besides that he developed some new activities with respect to measurement.
School career research
In the Netherlands there was a lot of discussion about the effect of the background of the family on the school career of children. Many projects were started to support children from lower class families to prepare them better for their school career. In order to show how serious this problem was several studies were done. One of these studies was done by Henk Blok. He had collected the necessary data and had made an explanatory model of the transition process from the primary school to the secondary school. The idea was that especially the choice of the secundary school created a difference between people of different social backgrounds. The model he developed for this study is presented below. He computed by hand on the basis of the correlations between the variables all effects suggested in the model and then he tested the model by using these coefficients to compute the correlations again. If the model was right these two sets of correlations should be very similar, however that was not the case.
In the Netherlands there was a lot of discussion about the effect of the background of the family on the school career of children. Many projects were started to support children from lower class families to prepare them better for their school career. In order to show how serious this problem was several studies were done. One of these studies was done by Henk Blok. He had collected the necessary data and had made an explanatory model of the transition process from the primary school to the secondary school. The idea was that especially the choice of the secundary school created a difference between people of different social backgrounds. The model he developed for this study is presented below. He computed by hand on the basis of the correlations between the variables all effects suggested in the model and then he tested the model by using these coefficients to compute the correlations again. If the model was right these two sets of correlations should be very similar, however that was not the case.
He thought the model was correct, so he checked his computations and did not detect an error. The difference between the observed correlation and the reproduced correlation for the relationship of the quality of the school and the test score was much too high. He did not know why. Therefore he asked me for help. I first tested his model using the LISREL program which gave in 5 minutes exactly the same result as he obtained in a week of computing by hand. The conclusion was that there was indeed an error in the model.
I immediately had a suspicion of what was wrong in the model: The variable Scholastic achievement was measured by the opinion of the teacher who may have a wrong opinion of a child. This means that this variable may contain errors. Henk Blok agreed with me with respect to this possibility. I specified the alternative model and the model fitted now very well to the data. In an hour this problem was solved using the LISREL approach. The paper was published and later we got a prize for the best paper of that year in the journal. As a consequence I have been asked regularly to help researchers with such models.
The results
An important result of this research was that the studies showed that the family background had only a very small effect in this process. Most of the effect came from the scholastic achievement which was used by the parents to choose the secondary school for their children. This choice was only minimally affected by the background of the family. If you might think that after this study less money was spent on extra education for lower class people you are mistaken. The opinion that the family background had a strong effect on the choice of the secundary school remained unchanged.
The MTMM model
Frank Andrews of the University of Michigan in An Arbor did so called Multi-trait Multi-method or MTMM experiments in order to determine the quality of questions and the effect of the method used on the responses to the different questions, which were measures for the different traits. He sent me a data set and the model he used and asked me if I could detect why the model did not fit the data well. This evaluation was based on the test statistic that was provided by the LISREL program. I studied his model and estimated it and could not see that there was anything wrong in the model but the test statistic suggested otherwise. The program we had developed to detect misspecifications also did not suggest any possible improvements. This raised the question whether something was wrong with the test statistic of the program. Frank Andrews used rather large samples of more than 1000 respondents in his studies. Was the test statistic too sensitive to the sample size? This was an issue I was worried about together with other researchers. This led to research I will mention below.
Why didn´t I see the importance of his research?
Here I like to mention a surprising observation. At that time I was very much concerned about the quality of measurement in the social science but in this case I concentrated only on the statistical problem. I did not see the relevance of this research for the evaluation of the quality of questions. Even after Frank Andrews had published his results in 1984 I did not immediately see the relevance of his approach. It was only in 1988 that I realized it. I organized a conference to discuss this approach and worked on it the rest of my academic life. In fact I continued the research of Frank Andrews, who suddenly died in 1992. Other people still continue this research which shows how important the work of Frank Andrews was and how preoccupied I was when I saw it for the first time.
Here I like to mention a surprising observation. At that time I was very much concerned about the quality of measurement in the social science but in this case I concentrated only on the statistical problem. I did not see the relevance of this research for the evaluation of the quality of questions. Even after Frank Andrews had published his results in 1984 I did not immediately see the relevance of his approach. It was only in 1988 that I realized it. I organized a conference to discuss this approach and worked on it the rest of my academic life. In fact I continued the research of Frank Andrews, who suddenly died in 1992. Other people still continue this research which shows how important the work of Frank Andrews was and how preoccupied I was when I saw it for the first time.
A book on second generation multivariate analysis
Around the same time I received a request for a contribution for a book on second generation multivariate analysis from Claes Fornell of the Business school of the University of MIchigan in Ann Arbor. I provided a paper on another measurement issue. People used and use all kinds of different questions to measure the same concept. My opinion was that these different questions also measured different concepts and combining them in a score one does not know what this variable represents. In order to be sure that one measures with different questions the same concepts one has to show that the questions after correction for random measurement error measure the same concept which means that these corrected variables should perfectly correlate. In my paper I showed that this correlation is quite different from perfect if different questions are used, but not if one uses the same question but with different response procedures for the two measures. I asked people to evaluate the status of occupations in numbers and lengths of lines: the higher the status of the occupations the higher the number and the longer the lines.
Around the same time I received a request for a contribution for a book on second generation multivariate analysis from Claes Fornell of the Business school of the University of MIchigan in Ann Arbor. I provided a paper on another measurement issue. People used and use all kinds of different questions to measure the same concept. My opinion was that these different questions also measured different concepts and combining them in a score one does not know what this variable represents. In order to be sure that one measures with different questions the same concepts one has to show that the questions after correction for random measurement error measure the same concept which means that these corrected variables should perfectly correlate. In my paper I showed that this correlation is quite different from perfect if different questions are used, but not if one uses the same question but with different response procedures for the two measures. I asked people to evaluate the status of occupations in numbers and lengths of lines: the higher the status of the occupations the higher the number and the longer the lines.
What to do?
This point of view is still not accepted by most people because they think that asking the same question twice in the same survey leads to memory effects. Recently I have written a paper to suggest how different questions can be formulated which measure the same concept while the memory effect can be reduced.
A more serious problem for me was that the test statistic led to rejections of the models if the sample was large and not if the sample was small.
This bothered me a lot and therefore I started to generate data with the same correlation to see when the model was accepted and when it was rejected. It turned out that the result of the test depended on the sample sizes but other factors were at least as influential, such as the number of indicators and the size of the random errors. So this topic asked more research. That is indeed what I did, as we will see below.
Albert Satorra contacted me
Albert Satorra is a Spanish mathematical statistician. He participated in my course in Essex on Structural Equation models and LISREL. He liked the topic and later sent me a letter asking me if he could study with me on this topic for his dissertation. Because I was not that sure how serious a researcher he was, I firstly suggested that we meet and discuss the issue. That year we wanted to see the processions during the “Semana Santa” in Spain, therefore I suggested that we would meet in that week and speak about it between the processions.
Albert Satorra is a Spanish mathematical statistician. He participated in my course in Essex on Structural Equation models and LISREL. He liked the topic and later sent me a letter asking me if he could study with me on this topic for his dissertation. Because I was not that sure how serious a researcher he was, I firstly suggested that we meet and discuss the issue. That year we wanted to see the processions during the “Semana Santa” in Spain, therefore I suggested that we would meet in that week and speak about it between the processions.
Semana Santa in Segovia
We went together with Irmtraud and Montse, his wife, to Toledo and Segovia to see the processions. The processions are impressive and archaic as you will see in the video. In between the processions we discussed the problem of testing models mentioned above. He told me that he was interested in this issue and we made an appointment to start research the next academic year. |
|
Albert starts his research
In September he came to our university. I had done many experiments that showed the problem and I had collected a lot of literature about testing. So the fundamental problem was clear to me. The problem was that one has to take into account the power of the test: under some conditions the test will not detect the error in the model and under other condition, like very large samples, minimal irrelevant errors lead to the rejection of the model. So the task for him was to study how we can determine the power of the test. A lot of the literature I had already collected but some texts were statistically too complex for me. So there was a nice task for a mathematical statistician. Albert liked the topic a lot, he was studying very seriously and had a very nice time with the students in Amsterdam. Within a year he delivered his first results: the estimation of the power of the test.
Albert´s diploma of proper behavior
Before Albert went back to Spain we organized a meeting in the department in which we pretended to give him a document that indicated the quality of the work he had done. However the document was made by Irmtraud on parchment with a seal but the text was a Spanish erotic poem which he frequently had cited to make us familiar with the Spanish language, we supposed. The people of the department thought that this was a serious document but Albert, Montse, Irmtraud and me knew better and laughed a lot. One of the versions of this famous Spanish poem is presented below.
In September he came to our university. I had done many experiments that showed the problem and I had collected a lot of literature about testing. So the fundamental problem was clear to me. The problem was that one has to take into account the power of the test: under some conditions the test will not detect the error in the model and under other condition, like very large samples, minimal irrelevant errors lead to the rejection of the model. So the task for him was to study how we can determine the power of the test. A lot of the literature I had already collected but some texts were statistically too complex for me. So there was a nice task for a mathematical statistician. Albert liked the topic a lot, he was studying very seriously and had a very nice time with the students in Amsterdam. Within a year he delivered his first results: the estimation of the power of the test.
Albert´s diploma of proper behavior
Before Albert went back to Spain we organized a meeting in the department in which we pretended to give him a document that indicated the quality of the work he had done. However the document was made by Irmtraud on parchment with a seal but the text was a Spanish erotic poem which he frequently had cited to make us familiar with the Spanish language, we supposed. The people of the department thought that this was a serious document but Albert, Montse, Irmtraud and me knew better and laughed a lot. One of the versions of this famous Spanish poem is presented below.
Que pereza dijo la marquesa poniendo las tetas sobre la mesa
oh, y ahora con quien culeo si solo maricas veo... no os
preocupéis, exclamó un tipo, yo soy Juan de la cosa, el que tiene
la verga más hermosa... pero otro dijo: yo soy Juan Tenorio
el que tiene la verga más grande de en todo este territorio...
entonces la marquesa exclamó ¡¡¡dejad la recocha que para
todos hay chocha!!!
oh, y ahora con quien culeo si solo maricas veo... no os
preocupéis, exclamó un tipo, yo soy Juan de la cosa, el que tiene
la verga más hermosa... pero otro dijo: yo soy Juan Tenorio
el que tiene la verga más grande de en todo este territorio...
entonces la marquesa exclamó ¡¡¡dejad la recocha que para
todos hay chocha!!!
Since this first cooperation we are friends and have worked together on different topics. In the year that I write this, 2019, we celebrated our 40 years of friendship and cooperation together with Montse and Irmtraud in Barcelona.