Finally a research grant
Although I did not get a grant for my research I decided to go on with it because I liked the topic and I was hoping to get a grant when I would publish more papers. Since the study of content analysis procedures showed that there did not exist an approach for the study of the argumentation of decision makers, I had to develop my own instrument to investigate the decisions of politicians. It turned out to be an undertaking of three years with a lot of problems, because many documents were very complicated and produced little agreement among coders, although a high agreement is a prerequisite for a serious investigation.
To give some idea of the study of decision making, I will illustrate our approach on the basis of a short note of the governor , Van Mook, in Indonesia to the Dutch government about what to do against independence movement in this colony after WO II.
He wrote to the Dutch government:
“If we take military action the chance is very high that we will restore law and order. But there is also a very small chance that we would lose our colony. If we do nothing, it is almost certain that we will lose our colony. I therefore suggest to start the military action.”
To give some idea of the study of decision making, I will illustrate our approach on the basis of a short note of the governor , Van Mook, in Indonesia to the Dutch government about what to do against independence movement in this colony after WO II.
He wrote to the Dutch government:
“If we take military action the chance is very high that we will restore law and order. But there is also a very small chance that we would lose our colony. If we do nothing, it is almost certain that we will lose our colony. I therefore suggest to start the military action.”
The coding of the text
In this text it is easy to detect the concepts. The coders had to indicate in brackets the symbols for the aspects detected. In this case the result was as follows:
“(S1: If we take military action) (P11: the chance is very high) (O11: that we will restore law and order). (P12: But there is also a very small chance) (O12: that we would lose our colony). (S2: If we do nothing), (P21: it is almost certain) (O21: that we will lose our colony). (Choice: I therefore suggest to start the military action.)”
Based on this analysis a decision tree can be constructed which gives an overview of the argumentation.
In this text it is easy to detect the concepts. The coders had to indicate in brackets the symbols for the aspects detected. In this case the result was as follows:
“(S1: If we take military action) (P11: the chance is very high) (O11: that we will restore law and order). (P12: But there is also a very small chance) (O12: that we would lose our colony). (S2: If we do nothing), (P21: it is almost certain) (O21: that we will lose our colony). (Choice: I therefore suggest to start the military action.)”
Based on this analysis a decision tree can be constructed which gives an overview of the argumentation.
Comparing this ideal decision tree with the analysis of the text one can see that utilities (evaluations) of the outcomes (U(Oij) are not mentioned. The theory would say that the utilities are essential to compare the Expected utility of the different strategies. This was a first result of our analyses that surprised us: expected components of the decision trees were missing. That raised the question how people could make choices if parts of the information was missing. This finding needed further research.
Coding a text like the one mentioned above, our students had no problems. Also the construction of the decision trees was not a problem. The agreement between their coding results was very high. We measured it for the concepts and the trees.
But unfortunately most of the documents in which decision makers motivate their choice were much more complicated. They overlooked more possible actions or more consequences due to actions of the opposing group. Besides they spoke a lot of past experiences, general remarks about the present choice and the motivations and options of the enemy etc.
Coding a text like the one mentioned above, our students had no problems. Also the construction of the decision trees was not a problem. The agreement between their coding results was very high. We measured it for the concepts and the trees.
But unfortunately most of the documents in which decision makers motivate their choice were much more complicated. They overlooked more possible actions or more consequences due to actions of the opposing group. Besides they spoke a lot of past experiences, general remarks about the present choice and the motivations and options of the enemy etc.
Reverse the coding procedure
In our first coding instruction the students coded the complicated documents in the same way as above looking for texts which indicate expected concepts. But the result of the coder agreement was devastating. The students who were very engaged in the project sometimes insulted each other because of little agreement. When I heard them shouting: “You imbecile, you don´t understand decision making! You produce fantasy structures!”, I had to intervene and restore peace.
Finally, Willem and I decided to change the coding approach. First the students had to gain insight in the structure of the argument by reading the text carefully and give a brief summary of each paragraph: like introduction, discussion about the aim etc. and indicate the relevant paragraphs for the actual decision. Based on the later paragraphs they constructed a first decision tree. After that they had to search the concepts in the text. Sometimes they had to correct parts of the decision tree but in the end they constructed their final decision tree, based on the parts they had indicated.
In this way, starting with the broad picture instead with the details, we obtained a high agreement among coders even for such a complex decision tree as shown. Finally the coders discussed the differences to determine a common solution. These final decision trees were a solid basis for further substantial research of the arguments of decision makers. This was a big relief after all the hardship!
In our first coding instruction the students coded the complicated documents in the same way as above looking for texts which indicate expected concepts. But the result of the coder agreement was devastating. The students who were very engaged in the project sometimes insulted each other because of little agreement. When I heard them shouting: “You imbecile, you don´t understand decision making! You produce fantasy structures!”, I had to intervene and restore peace.
Finally, Willem and I decided to change the coding approach. First the students had to gain insight in the structure of the argument by reading the text carefully and give a brief summary of each paragraph: like introduction, discussion about the aim etc. and indicate the relevant paragraphs for the actual decision. Based on the later paragraphs they constructed a first decision tree. After that they had to search the concepts in the text. Sometimes they had to correct parts of the decision tree but in the end they constructed their final decision tree, based on the parts they had indicated.
In this way, starting with the broad picture instead with the details, we obtained a high agreement among coders even for such a complex decision tree as shown. Finally the coders discussed the differences to determine a common solution. These final decision trees were a solid basis for further substantial research of the arguments of decision makers. This was a big relief after all the hardship!
The struggle to get a research grant
Now Willem and I thought that we were ready to apply at the National Science Foundation for a grant for a detailed study of decision making. The last time the application was rejected because of a lack of publications. But now we had fulfilled this requirement: we had developed a reliable content analysis instrument and showed by some case studies that the approach worked. In total we had 13 publications in recognized journals, including some international ones. To our great disillusion our application was rejected again. The committee did not even indicate the reason for the rejection. I was completely devastated. Should it be true what my Suisse professor in French said that they only award the grants to their friends?
Willem was not willing to accept this rejection. So he wrote a polite protest letter against this decision indicating the number of publications made. In the meantime he also did something else. Since the names of the member of the committee which made the decision was public, he studied how many grants the members of the committees had received for the projects of their research groups during the last years. And indeed they had assigned to themselves plenty of grants. With this information in mind Willem politely phoned each social scientist of the committee to convince them to revise the decision because we now had sufficient publications made to deserve a grant. Most of them agreed with him. Only one professor of our own university was not willing to co-operate. Then Willem told him that if we did not get a revision of the decision, he would start research about the relationship between membership of the committee and the received grants.
I don´t know if Willem´s action forced the happy end but some months later we got the grant for the decision making project which meant that I had a paid research job for the coming years.
Now Willem and I thought that we were ready to apply at the National Science Foundation for a grant for a detailed study of decision making. The last time the application was rejected because of a lack of publications. But now we had fulfilled this requirement: we had developed a reliable content analysis instrument and showed by some case studies that the approach worked. In total we had 13 publications in recognized journals, including some international ones. To our great disillusion our application was rejected again. The committee did not even indicate the reason for the rejection. I was completely devastated. Should it be true what my Suisse professor in French said that they only award the grants to their friends?
Willem was not willing to accept this rejection. So he wrote a polite protest letter against this decision indicating the number of publications made. In the meantime he also did something else. Since the names of the member of the committee which made the decision was public, he studied how many grants the members of the committees had received for the projects of their research groups during the last years. And indeed they had assigned to themselves plenty of grants. With this information in mind Willem politely phoned each social scientist of the committee to convince them to revise the decision because we now had sufficient publications made to deserve a grant. Most of them agreed with him. Only one professor of our own university was not willing to co-operate. Then Willem told him that if we did not get a revision of the decision, he would start research about the relationship between membership of the committee and the received grants.
I don´t know if Willem´s action forced the happy end but some months later we got the grant for the decision making project which meant that I had a paid research job for the coming years.