As ever, John Naughton had an interesting piece in the Observer yesterday about fake news. He pointed to some pretty concerning findings about academic research that I wasn’t aware of.
To illustrate how difficult getting at the “truth” can be, consider science, which, after all, represents the most serious attempt our culture has made to achieve accuracy and dependable knowledge. Peer review is one of the central pillars of this enterprise, but it turns out that that has weaknesses. “Statistical mistakes are widespread”, says one survey, and peer reviewers who evaluate papers before journals commit to publishing them are much worse at spotting mistakes than they or others appreciate.
This gloomy verdict is confirmed by Statcheck, a program written by a Dutch researcher and employed to conduct examine statistical inferences drawn in scientific papers. In 2015, the program took less than two hours to read through more than 30,000 papers published in eight respected journals between 1985 and 2013. Its conclusion: about half of those papers contained a statistical error.
What this suggests is that even in an area of human activity that is professionally committed to getting things right, accuracy can be hard to achieve and truth even more elusive.
I’d imagine that a high proportion of the errors are pretty trivial. However, when you add in problems of publication bias and the behaviour that it encourages among academics, it makes me feel even more self-righteous about academic referees who reject a paper stating that something is “already known” because one paper using one model with one sample in one location has found one supporting finding. Could be sour grapes on my part too.