Journalistic deficit disorder

What newspapers don’t say matters as much as what they do

IF ALL the stories in the newspapers claiming that a cure for cancer is just around the corner were true, the dread disease would have been history long ago. Sadly, it isn’t. But though some publications do have a well-deserved reputation for exaggeration in this area, many of these reports are at least based on respectable research published in peer-reviewed journals. So what is going on?

That, more or less, is the question addressed by François Gonon of the University of Bordeaux and his colleagues in a paper in the Public Library of Science. Dr Gonon, a neurobiologist, looked not at cancer but at another disease that gets a lot of coverage in the press, attention-deficit hyperactivity disorder (ADHD)—a fashionable diagnosis for children who struggle to pay attention at school.

Dr Gonon’s group scoured PubMed and Factiva, two databases containing biomedical papers and newspaper articles respectively, for reports on ADHD. Comparing the two turned up 47 papers on ADHD that were covered in 347 articles written in English-language newspapers during the 1990s. From these, the researchers chose to focus on the ten papers that had received the most press coverage.

First, they studied subsequent scientific literature to see what had become of the claims reported in the top ten papers. Seven of these had reported research designed to test novel hypotheses. Though each concluded at the time that the hypothesis in question might be correct (ie, the data collected did not refute it), the conclusions of six were either completely refuted or substantially weakened by the subsequent investigations unearthed by Dr Gonon. The seventh has neither been confirmed nor rejected, but he and his colleagues, citing two independent experts on ADHD, say its hypothesis “appears unlikely”.

The other three papers in the top ten were following up existing hypotheses rather than presenting novel ideas. Two of them were confirmed by the subsequent work Dr Gonon tracked down, whereas one was weakened.

All this is to be expected. Research on research—particularly on medical research, where sample sizes are often small—shows that lots of conclusions do not stand the test of time. The refutation of plausible hypotheses is the way that science progresses. The problem was in the way the work was reported in the press.

Absence of evidence

In total, the original top ten papers received 223 write-ups in the news. But then the newspapers lost interest. Dr Gonon and his team found 67 further studies examining the conclusions of the original ten, but these subsequent investigations earned just 57 newspaper articles between them. Moreover, the bulk of this coverage concerned just two of the ten. Follow-ups to the other eight got almost no attention. Dr Gonon’s team do not pull their punches. There is, they say, an “almost complete amnesia in the newspaper coverage of biomedical findings”.

If the hypotheses reported in the original studies had stood the test of time, then such short journalistic attention spans would not be a problem. But as Dr Gonon has shown, 80% of the papers in the study turned out to be either wrong or questionable. Failure to follow that up risks leaving even attentive readers with a false impression of progress in the field.