Policy-related research is big business. But is all the academic advisory work that goes on really scientific enough? Jarl Kampen (Research Methodology Group) developed a bullshit detector in order to separate the chaff from the grain. The article he and his colleague Peter Tamas published in Quality and Quantity is entitled Should I take this seriously? A simple checklist for calling bullshit on policy supporting research. A provocative title for a scientific article.
Is there a lot of frustration underlying this?
‘Frustration is not the right word. Concern, commitment and hope come closer. It is a checklist for both producers and users of policy-related research, intended to help them set feasible and realistic objectives.’
It sounds as though there is a lot of bullshit research. Why else would such a checklist be needed?
‘This is not a verdict on the work of colleagues. I don’t have enough of an overview for that. But there is concern in various scientific institutions in the Netherlands (and Flanders too) about certain practices. The royal academy of sciences KNAW has just published a report about the quality of social science research. But criteria such as those we have laid down for the quality of the research are not included.’
Who makes the most mistakes?
‘It goes wrong as early as the formulation of the research question. People confuse the social objective they want to reach with the objective of the research itself. Then you get questions like: how can we ensure that a particular exotic mushroom becomes the food of the future? That is not a scientific question. You can research consumer attitudes to the mushroom. You can research its nutritional value etc. Research should provide policymakers and politicians with information but it is not part of the researcher’s mandate to get mixed up in politics. In a sense, a researcher should not be interested in the results of the research.’
Is Quality and Quantity the right platform? Policymakers do not read scientific journals.
‘Of course the checklist is primarily intended for clients. But the hidden agenda is that the producers of research, our colleagues, see this too. They are all simple rules: Does your research question make sense? What is the quality of your data, the method you use, and your analysis? Does your conclusion figure and does it answer the question you asked? This all needs to be right, otherwise the research is worthless and, to put it bluntly, bullshit.’
Isn’t the checklist really just an appeal for honesty?
‘Scientists need to be honest about what they can deliver and what they cannot. In fact, the checklist helps them to be humble. The scientist should be more modest about making sweeping statements. And the client about what this sort of research can achieve. In effect, badly implemented research is bad for all those involved.’