Wednesday, December 17, 2014

The BMJ: The association between exaggeration in health related science news and academic press releases: retrospective observational study

An early Christmas gift. Image by Me (CC BY-NC-ND)

Every December The British Medical Journal releases its holiday edition where fluffy light-hearted research that scientists do just for fun or personal curiosity gets to see the light of day. Today's article was something of a surprise gift from the Red Man himself to my blog. I must have been a good blogger this year!

The goal of this research was to identify where distortions, exaggerations, and changes to research conclusions as presented in media come from: research --> press release or press release --> news story? For example how did research showing that when a female Telostylinus angusticllis fly mates with two males, most of her offspring will inherit genes from the second male, but may inherit adult body size from the first male (suggesting some non-genetic mechanism of inheritance: possibly hormones in the semen) turn into this news story? Which heavily implies that your baby--your human baby--might look like the guy you lost your virginity to instead of your husband/baby-daddy. It's only several paragraphs down that they admit this research was done on flies and might does not have anything to do with humans. Jezebel has a nice not-too-technical explanation of the research if you're interested. But again, why does this happen? And whose fault is it? This is what researchers wanted to know.

When a group of researchers completes a study and their research is published the University they work at is often excited. After all, productive researchers means more grants and more grants means more overhead money for the university (here's a decent definition of overhead, basically it's the costs of maintaining a research university that can't be pegged to any one researcher: buildings, electricity, office staff, etc.).

To celebrate the release of some new research the university's media relations department will often issue a press release. These press releases, which are often not written by the scientists themselves, will be picked up by major media outlets, fleshed out, potentially embellished, hopefully given context, and then published as news stories. Previous research has suggested that these press releases can contain misinformation: a study of 200 random research press releases from 2005 found exaggerations in nearly a third and a lack of appropriate caveats in more than half. In another study some press releases were compared not only to the actual study, but also to the text of the study's own abstract; in this study they found the exaggerations were often found in the abstract and the press release. This suggests that the media office was not trying to spin things and just copying what they saw in the abstract. Today's article in The BMJ attempts to further understand the source of exaggerations in science journalism.


What did the authors of The BMJ paper find out about the source of exaggerations in health and science news?


The researchers identified all the press releases put out by the top 20 UK research universities in fields with potential relevance to human health. For this collection of 462 press releases, they identified the research paper and any/all written news stories about the research.

The researchers then started with the assumption that all information in the peer reviewed research paper was true (not always an accurate assumption. Remember that paper about glyphosate?), and compared the information in the press releases and news stories. Specifically they looked for cases involving advice to readers to change behavior, causal statements drawn from correlative results (cross sectional and longitudinal observational data), and inference to humans from animal research (like the fly thing).

In the researchers analysis they found that 40% of the press releases contained advice that was not present in research paper, 33% stated that a correlation was a direct causal relationship, and 38% incorrectly implied human implications for findings. They then looked at the odds ratio that a news story would be exaggerated if the press release was or wasn't. I think the easiest way for you to see how this shakes out is for me to copy their image.

Fig 2 Proportions of news with exaggerated advice, causal statements
from correlational research, or inference to humans from non-human studies
were higher when the associated press releases contained such exaggeration.
Error bars are 95% confidence intervals. From Sumner et al.

What we can see here is that the news stories are almost always following the lead of the press releases. If the press release does not mix up correlation and causation then the news stories don't (Fig 2B), but if the press release does mix them up then so do the news stories. Oh, and those error bars were made by boot-strapping.

The researchers suspected that the reason press releases are exaggerated is to encourage the media buzz. So they compared the exaggerations in the press releases to the rate of news story generation. Surprisingly they did not find a statistically significant difference in news stories between exaggerated and non-exaggerated press releases. However, this finding is limited in that they can't compare an exaggerated and accurate version of the same press release. It's possible that there is bias in which research papers get exaggerated press releases.

Based on these finding the researchers find that, while it is popular to blame the media for exaggerating researchers findings, it seems the source of the exaggeration is most often the press release and that the news outlets merely follow that lead. The researchers acknowledge that their observational approach does not allow them to determine causality. It is possible that journalists and the media office both look at the research and independently decide to exaggerate the findings. They also wonder about the biasing of which articles get exaggerated by the media office: maybe the less interesting/applicable to humans articles are more likely to get exaggerated. This bias would confound the analysis of how many news stories each press release generated.  (For example...a good press release of an interesting article could result in 5 news stories. And an exaggerated press release of a good article could still result in 5 stories. A good press release of a boring article could result in 0. But an exaggerated press release of a boring article could result in 4 stories. If this is true (and the media office knows it), they would exaggerate only the boring stories). One final potential weakness of this study is that the researches had no way to control the level of exaggeration and over-statement present in the research papers. Peer-review should remove the most egregious stuff, but research does sometimes get over hyped in the research journals by authors hoping to land in a higher impact journal (and journals hoping to attract more readers).


So what are my conclusions?


This makes it sound like I should cut the journalists some slack, huh? NOPE. If I can look up the papers then so can the journalists. It's journalism, it's their job to get to the truth. They seem to have forgotten. My crusade will continue. (Don't worry, all dismemberment will be metaphorical in my crusade).

This does, however, make an important point about where all these exaggerations may be starting: at the universities. Especially if the media offices are working with the scientists, the scientists need to take a more principled stand against this sort of bubble shoeing that undermines the credibility of the whole institution in the eyes of the public.

Scientists, if we want to be taken seriously we need to be careful with press releases so that this doesn't turn into this. And journalists, do your job--which, incidentally, is not just to generate click bait. Or don't and maybe one day this little blog will be popular and make me fabulously wealthy. (But really, no, I'd give up all the glitter for a better informed public.)

And P.S. If you're wondering why there's a trash bag behind my Christmas tree, every year Mr. Dr. Kris and I fulfill a few wish lists for the Salvation Army Angel Tree. Those trash bags are full of Christmas gifts for some area kids.

References


Crean, Angela J., Anna M. Kopps, and Russell Bonduriansky. "Revisiting telegony: offspring inherit an acquired characteristic of their mother's previous mate." Ecology letters 17.12 (2014): 1545-1552.
 
Sumner, Petroc, et al. "The association between exaggeration in health related science news and academic press releases: retrospective observational study." BMJ 349 (2014): g7015.
 
Goldacre, Ben. "Preventing bad reporting on health research." BMJ 349 (2014): g7465.
 
 


No comments:

Post a Comment