Scientific result of the NIMMO-Telepanel
One of the purposes of the grant of the Dutch Science Foundation for the telepanel was that it would allow for a lot of scientific research. I had applied for the grant especially in order to do a number of MTMM experiments to evaluate the quality of questions. Because the telepanel was a permanent data collection instrument it was possible to do this methodological research next to commercial research .
The MTMM studies 1990-1995
For this research I looked for a PhD student with interest in measurement problems and a sufficient basis of knowledge of experimental research to perform this study and write a dissertation about this topic. I selected for this research a psychology student, Annette Scherpenzeel, who had some experience with magnitude estimation which was one of the topics I liked to test. Two months after she started with us she presented in an excellent way at a conference in Madrid the work she was planning to do. So I was sure that I could rely on her capabilities to do this research which I thought was very relevant.
After getting familiar with the telepanel and the computer assisted data collection she started with the experiments. In total 50 of these experiments with different question formulations have been performed. 24 in the first year, 18 in the second and 8 in the third year. The MTMM experiments provide estimates of the reliability and validity of the questions involved. If one codes the characteristics of these questions one can also study the effect of these characteristics of the questions on the reliability and validity of these questions. On the basis of this analysis she wrote her dissertation in 1995. This was the first set of experiments where computer assisted data collection was involved. This study has played an important role in the further development of this approach and, finally, to the computer program SQP for prediction of the quality of survey questions.
Annette Scherpenzeel worked later in different functions in and outside the Netherlands in applied and scientific research and is at the moment the international coordinator of the European research project SHARE which stands for Survey of Health, Aging and Retirement in Europe.
For this research I looked for a PhD student with interest in measurement problems and a sufficient basis of knowledge of experimental research to perform this study and write a dissertation about this topic. I selected for this research a psychology student, Annette Scherpenzeel, who had some experience with magnitude estimation which was one of the topics I liked to test. Two months after she started with us she presented in an excellent way at a conference in Madrid the work she was planning to do. So I was sure that I could rely on her capabilities to do this research which I thought was very relevant.
After getting familiar with the telepanel and the computer assisted data collection she started with the experiments. In total 50 of these experiments with different question formulations have been performed. 24 in the first year, 18 in the second and 8 in the third year. The MTMM experiments provide estimates of the reliability and validity of the questions involved. If one codes the characteristics of these questions one can also study the effect of these characteristics of the questions on the reliability and validity of these questions. On the basis of this analysis she wrote her dissertation in 1995. This was the first set of experiments where computer assisted data collection was involved. This study has played an important role in the further development of this approach and, finally, to the computer program SQP for prediction of the quality of survey questions.
Annette Scherpenzeel worked later in different functions in and outside the Netherlands in applied and scientific research and is at the moment the international coordinator of the European research project SHARE which stands for Survey of Health, Aging and Retirement in Europe.
Time budget research 1990-1993
A large scale research that we thought would be interesting also for commercial research was time budget research. This research relevant to determine the time spent on the media, on transport, on working, domestic work etc.. Normally time budget research was done using paper and pencil diaries where the respondent had to indicate per hour what (s)he had done.
We thought that it would make sense to compare this form of a diary with two computer assisted procedures, one done by an interviewer by telephone and the other using the telepanel where people registered their own responses. We thought that these computer assisted procedures would be more efficient and could be used for commercial purposes as well. Therefore I looked for a student who was interested in this study and would like to do this methodological study and eventually later also the commercial research. The student who was selected for this research was Nelly Kalfs, a former student of me.
Her task was first of all to develop the different forms of time budget research and then perform a study to find out the differences between the various procedures.
The computer assisted diaries, she developed, were different from the paper and pencil diary because the emphasis was on changes in activities. If the respondent indicated an activity, the next questions followed: “ Till what time was this done?” and “How much did you like this activity”. Then followed the same three questions for the next activity. Nelly also organized the data collections with the paper and pencil study and the telephone interviews but the self completion version was done on the telepanel of the University of Amsterdam.
Significant differences were observed between the different approaches but no general effect of the different methods across all activities. A big advantage of the computer assisted approach is that one can make programs that more or less immediately generate reports about the time spent on the different activities and how much these activities were liked. In the paper and pencil version one had first to code all the answers. That was a very expensive and time consuming activity. An advantage of the computer assited approach was also that one can build in checks to prevent errors. For example if people say that they went “Shopping” they may have forgotten the transport before. Many of such tests can be introduced.
This was a very informative study and she reported about this study in her dissertation which was published in 1993.
Unfortunately, the organizations that normally were doing this kind of research did not have the budget for such a large scale study at that time. This meant that this research was mainly of scientific value and did not contribute to the exploitation of the telepanel. After the fact I think that we have underestimated the potential commercial possibilities of this research. Given the possible separated applications I have mentioned above one could have tried to interest different clients for different parts of this big study.
It is at least interesting that she found with all three data collection procedures that "sleeping" was the most positive evaluation activity, while the second best was “walking with the dog”. This is a clear illustration how important this research is.
Nelly Kalfs continued working on different aspects of time use, especially media use and travel research. At some point in time she made a switch to a management function and became 2016 chief engineer director of Eastern Netherlands at the governmental organization for transport and waterworks “Rijkswaterstaat”.
A large scale research that we thought would be interesting also for commercial research was time budget research. This research relevant to determine the time spent on the media, on transport, on working, domestic work etc.. Normally time budget research was done using paper and pencil diaries where the respondent had to indicate per hour what (s)he had done.
We thought that it would make sense to compare this form of a diary with two computer assisted procedures, one done by an interviewer by telephone and the other using the telepanel where people registered their own responses. We thought that these computer assisted procedures would be more efficient and could be used for commercial purposes as well. Therefore I looked for a student who was interested in this study and would like to do this methodological study and eventually later also the commercial research. The student who was selected for this research was Nelly Kalfs, a former student of me.
Her task was first of all to develop the different forms of time budget research and then perform a study to find out the differences between the various procedures.
The computer assisted diaries, she developed, were different from the paper and pencil diary because the emphasis was on changes in activities. If the respondent indicated an activity, the next questions followed: “ Till what time was this done?” and “How much did you like this activity”. Then followed the same three questions for the next activity. Nelly also organized the data collections with the paper and pencil study and the telephone interviews but the self completion version was done on the telepanel of the University of Amsterdam.
Significant differences were observed between the different approaches but no general effect of the different methods across all activities. A big advantage of the computer assisted approach is that one can make programs that more or less immediately generate reports about the time spent on the different activities and how much these activities were liked. In the paper and pencil version one had first to code all the answers. That was a very expensive and time consuming activity. An advantage of the computer assited approach was also that one can build in checks to prevent errors. For example if people say that they went “Shopping” they may have forgotten the transport before. Many of such tests can be introduced.
This was a very informative study and she reported about this study in her dissertation which was published in 1993.
Unfortunately, the organizations that normally were doing this kind of research did not have the budget for such a large scale study at that time. This meant that this research was mainly of scientific value and did not contribute to the exploitation of the telepanel. After the fact I think that we have underestimated the potential commercial possibilities of this research. Given the possible separated applications I have mentioned above one could have tried to interest different clients for different parts of this big study.
It is at least interesting that she found with all three data collection procedures that "sleeping" was the most positive evaluation activity, while the second best was “walking with the dog”. This is a clear illustration how important this research is.
Nelly Kalfs continued working on different aspects of time use, especially media use and travel research. At some point in time she made a switch to a management function and became 2016 chief engineer director of Eastern Netherlands at the governmental organization for transport and waterworks “Rijkswaterstaat”.
,Four more dissertations
Annette Scherpenzeel and Nelly Kalfs worked full time at their PhD research while working at the NIMMO. They were however not the only ones who got their PhD thanks to their work at NIMMO. Also people working on the management of the telepanel or for commercial research were able to write their dissertation at least partially based on the work they did there.
Robert Voogt was manager of the telepanel, especially, with respect to participation, dropout and incentives to cooperate. He wrote his dissertation on this topic and got his PhD in 1997. After that he started to work at the ministry of Social affairs and employment. Since 2012 he is research specialist of the department of Inspection of the same ministry.
Bas van de Putte worked on the research for the trade union FNV at the telepanel. Partially based on this research he wrote his dissertation and got his PhD degree in 1993. He was associate professor at the University of Amsterdam. Now he is full professor of Health Communication.
Coen van Rij was the successor of Bas van de Putte in het FNV research. He also made his PhD. based on this research, especially, on the length of time of participation in organizations. He worked in different research organizations. Since 2012 he is director of CEBEON, centrum for policy advice research.
Adriaan Hoogendoorn worked at the NIMMO on sampling problems and representativity of the panel, He wrote his dissertation in 1999 on statistical models for estimation of product penetration in panels with missing data. He is now senior researcher in the research department of the Mental health organization In Geest.
Edwin van den Oord came to work at NIMMO on consumer research after he had obtained his PhD degree. When he stopped at the NIMMO he got an offer from the US to become the director of a laboratory for research on traits of twins, the research he had done for his dissertation. He is still working in the US.
Only two people who worked at the NIMMO did not continue their studies to get a PhD degree.
Corrie Vis came to the NIMMO for work not for a PhD. She was responsible for controlling the questionnaires before they went into the field. When the telepanel moved to Tilburg, she continued the same work there till she stopped with working after a car accident. A year latter she applied with some other people for a grant to start an internet panel in Curaçao to support people there with respect to health and financial literacy.
Harm Hartman was supposed to continue his studies with a dissertation. I will explain in a next story why he did not realize that aim. He has been working for different marketing research organizations and is now insights director of the marketing research company Ipsos in Paris, France.
Marius the Pijper who came as the substitute for Harm Hartman to the telepanel, has solved several software problems which Harm Hartman had left unfinished. When the telepanel moved to Tilburg he continued working there till he went with pension. His programming skills were highly appreciated and he enjoyed working on these problems.
Articles and books
As scientific director I have been able to contribute alone and with other people to the output of NIMMO over the years 1991- 1995 in total 18 articles and 4 books. In this overview I have only counted the scientific publications not the reports made for clients.
Annette Scherpenzeel and Nelly Kalfs published in the same period besides their dissertations also 13 scientific papers
This overview shows clearly that NIMMO with the telepanel has been a big scientific success. So many dissertations and publications would never have been possible without a facility like the telepanel.
Annette Scherpenzeel and Nelly Kalfs worked full time at their PhD research while working at the NIMMO. They were however not the only ones who got their PhD thanks to their work at NIMMO. Also people working on the management of the telepanel or for commercial research were able to write their dissertation at least partially based on the work they did there.
Robert Voogt was manager of the telepanel, especially, with respect to participation, dropout and incentives to cooperate. He wrote his dissertation on this topic and got his PhD in 1997. After that he started to work at the ministry of Social affairs and employment. Since 2012 he is research specialist of the department of Inspection of the same ministry.
Bas van de Putte worked on the research for the trade union FNV at the telepanel. Partially based on this research he wrote his dissertation and got his PhD degree in 1993. He was associate professor at the University of Amsterdam. Now he is full professor of Health Communication.
Coen van Rij was the successor of Bas van de Putte in het FNV research. He also made his PhD. based on this research, especially, on the length of time of participation in organizations. He worked in different research organizations. Since 2012 he is director of CEBEON, centrum for policy advice research.
Adriaan Hoogendoorn worked at the NIMMO on sampling problems and representativity of the panel, He wrote his dissertation in 1999 on statistical models for estimation of product penetration in panels with missing data. He is now senior researcher in the research department of the Mental health organization In Geest.
Edwin van den Oord came to work at NIMMO on consumer research after he had obtained his PhD degree. When he stopped at the NIMMO he got an offer from the US to become the director of a laboratory for research on traits of twins, the research he had done for his dissertation. He is still working in the US.
Only two people who worked at the NIMMO did not continue their studies to get a PhD degree.
Corrie Vis came to the NIMMO for work not for a PhD. She was responsible for controlling the questionnaires before they went into the field. When the telepanel moved to Tilburg, she continued the same work there till she stopped with working after a car accident. A year latter she applied with some other people for a grant to start an internet panel in Curaçao to support people there with respect to health and financial literacy.
Harm Hartman was supposed to continue his studies with a dissertation. I will explain in a next story why he did not realize that aim. He has been working for different marketing research organizations and is now insights director of the marketing research company Ipsos in Paris, France.
Marius the Pijper who came as the substitute for Harm Hartman to the telepanel, has solved several software problems which Harm Hartman had left unfinished. When the telepanel moved to Tilburg he continued working there till he went with pension. His programming skills were highly appreciated and he enjoyed working on these problems.
Articles and books
As scientific director I have been able to contribute alone and with other people to the output of NIMMO over the years 1991- 1995 in total 18 articles and 4 books. In this overview I have only counted the scientific publications not the reports made for clients.
Annette Scherpenzeel and Nelly Kalfs published in the same period besides their dissertations also 13 scientific papers
This overview shows clearly that NIMMO with the telepanel has been a big scientific success. So many dissertations and publications would never have been possible without a facility like the telepanel.