To help in the fight against sloppy and fraudulent science, researchers should replicate the experiments of others much more frequently. Even if they do not want to.
© Marumari Wikimedia Commons
This is written in a new advisory document of the Royal Netherlands Academy of Arts and Sciences (KNAW). The Dutch social psychology field was in shock when the heavy fraud committed by the Tilburg professor Diederik Stapel was uncovered in 2011: for years, he just made up data from experiments. How was it possible that he did this unimpeded for so many years? Results of experiments in other scientific fields are also regularly questioned, if only because some scientists embroider their conclusions, as their publications might otherwise pass by unnoticed.
Reproduce and replicate
How can we fight this type of ‘science’? According to John Mackenbach, professor of Public Health and chair of a committee that was established for this reason by the KNAW, there is only one thing to do. ‘Scientists should replicate each other’s experiments more often. This should be the normal approach.’
This committee presented their results today. They say that research which cannot be replicated impedes scientific progress. Furthermore, scientific mistakes that are overlooked could even cause irreparable damage. An example that is not so far-fetched is applying the wrong treatment to patients. Mackenbach’s advice is clear: replicate research more often and make additional funds available to do so. To make replication easier, scientists should also disclose additional details about their research method. ‘Now that journals are being published online, space is not a limitation anymore.’
Study replication is not innovative by definition. Does it require a different kind of scientist?
‘No, we should all be doing it. It is still high-quality research. It requires the same competencies and equipment as the original research. It should therefore be carried out by the best researchers. The sole difference is indeed that it will yield little innovation.’
Should it be a sort of compulsory scientific service?
‘That is not the way we phrased it in our advisory report. But perhaps all starting PhD candidates could replicate earlier studies in the frame of their education: they will thus not only learn the trade, but also contribute to the reliability of their discipline.’
Speaking of replication, this has been pleaded before. Even Nobel Prize laureates have been advocates for this approach. Why is this such a difficult message to convey?
‘Scientists are interested in new things, and what is reproduced isn’t new, of course. There is a difference in the way it is experienced. But there are also more objective obstacles. Journals are less interested in replications, and so they yield less status. Researchers are appreciated more for innovative research, which will increase their chances for new grants, etc. We should try getting rid of these obstacles.’
How do you suggest we do that?
‘There are various large international initiatives in which entire scientific fields are analysing the state of reproducibility of their results. Journals are also joining in. Some journals have promised to publish replication studies of their earlier articles – regardless of their results. Changes still need to take place in the Netherlands in that respect. Research financier NWO has already started a programme for replication studies, but the budget is limited. It is far from enough to tackle the problem.’
Which scientific fields are faced with the most difficulties?
‘We understand that fields such as psychology run a greater risk, as it involves much more work with people. In fields such a physics, most observations are performed using equipment, and some physicists say that “this problem does not apply to them”. But how can you be so certain if you don’t investigate? One of the primary recommendations in our report is that scientists should investigate replicability and reproducibility within entire fields of research or subdisciplines.’
Do scientists perceive replication studies as a form of motion of no confidence?
‘I cannot exclude that this might be at play for some, but the scientists I have spoken to have not said anything about it. They are simply interested in the possibility.’
What costs would be reasonable? One percent of the research budget, or perhaps rather ten percent?
‘Something in between, I think. But we really need to know more about the magnitude of the problem.’