You expect scientists to be very careful when referring to past research

Everything that is said in scientific publications is always perfectly correct. Although such an article is sometimes less readable due to all the neat references and strong justifications. Or as Paulien Cornelisse wrote last Thursday: ‘[Ik] I was surprised how boring and endless I found it.

Or wait, Cornelisse was talking about BFG by Roald Dahl and not on a scientific article. And the first sentence of my column turns out to be incorrect on closer inspection. Let me start over.

Scientific research often builds on previous research – it’s the infamous “standing on the shoulders of giants”. You expect scientists to be very careful when referring to the results of past research.

Last week, the results of a study in which psychologist Cory Cobb and four colleagues checked whether scientific publications correctly cite previous work. This was a total of 3,347 references to previous research in 89 scientific articles on psychology. The good news is that 2,718 of these quotes clearly describe what the original study found. Less good news is that 311 references were a little less accurate. For example, findings for Chinese immigrants were generalized to other Asian populations.

Worse still, 318 references were completely wrong. These were mostly assertions that were completely absent from the original study – as I did with Paulien Cornelisse’s column. Worse still, about a third of the fake references claimed something that completely contradicted the quoted article. For example, a study that found no evidence for a particular phenomenon was casually cited as “this study shows that this phenomenon occurs.”

In short: almost 10% of the references checked were incorrect in terms of content. This study focused on psychology, but previous studies have shown that similar misery occurs in fields ranging from medicine to marine biology and earth sciences to education (although the scientific article does not did not use the term “poverty”).

Cobb and his colleagues offer a number of tips for improving citations, in particular the importance of better citation checking in the future. The question, of course, is who should do it. Scientific articles are reviewed by other researchers (often in their time) and there are easily dozens of references in a publication. It is unrealistic to expect these reviewers to examine all of these underlying articles.

Perhaps – as so often – we should start at the other end: with the young researchers we train. This week, I reviewed two student research proposals. Several times I wrote in the margin of a statement: “Do you have a reference for this?” Suddenly, I realized that this question could prompt students to quickly find an article that seemed more or less relevant to their request. So from now on I’ll sideline that they should see if they can back up their claim. What does previous research say about this? And maybe the right conclusion is that their claim is wrong and they need to start over. “Boring and tedious”, as Paulien Cornelisse would say.