Last week the ballroom at the San Antonio Hyatt was filled with a group that relies on medical evidence in their daily work. The American Medical Writer’s Association was gathered to hear Ivan Oransky, MD. He is the cofounder of Retraction Watch and VP of MedPage Today. His speech focused on the multiple ways scientific evidence can be faulty, unreliable and fraudulent.
It was disconcerting, but we weren’t surprised.
In his book, The Half-Life of Facts, Samuel Arbesman explains that knowledge, like radioactive material, has a half-life. “As scientific knowledge grows rapidly, it leads to a certain overturning of old truth, a churning of knowledge.” We expect some facts to change. In school, you probably learned that the population of the earth was some number less than the 7 billion it is today. But you aren’t surprised that fact has changed.
It’s a bit more jolting when other facts lose their fact-ness. Last year whole eggs weren’t part of a healthy diet. This year they are. These changes can be troubling.
But Arbesman explains, “Sometimes we get things wrong, or not as accurate as we would like. But on the whole, the aggregate collection of scientific knowledge is progressing toward a better understanding of the world around us.”
And that is what makes Dr. Oransky’s talk so exciting. When evidence is found to be faulty, unreliable or even fraudulent, we inch closer to the truth. He outlined several ways this is happening, thanks to the technology provided by the Internet.
Reproducibility – We Can Use Some Stinking Badges
One of the foundations of science is the idea that research should be reproducible. That is, we should be able to conduct the same research and get the same results. Unknown factors can confound research, so we hope to see results confirmed in a number of studies conducted by different researchers.
Now, if research is not found to be reproducible, that doesn’t mean the original researchers were involved in anything nefarious. It simply means that something skewed the results in a way that they can’t be replicated. The problem is that the original research stands. After all it was research done in good faith and the data is the data.
But it would be helpful to know this little fact when you are looking at the research, no?
Queue the Reproducibility Initiative. This initiative offers researchers the chance to have their work reproduced by independent labs. If it is indeed reproducible, their research can then sport a shiny badge of reproducibility. It will also be listed in a variety of online directories with the other independently validated research.
Peer Review – If You Must Choose a Peer, Choose Yourself
This can be a sticky issue in academic research. Reputable journals conduct peer reviews on research before they publish it. They look at the research process, the statistical methods, potential conflicts of interest, and other factors. Anything that might raise concerns about the trustworthiness of the articles they consider. Without peer review, we look at research a little sideways.
I won’t go into the particularly devious machinations that can go into scamming the peer review process. I will just say that there are loopholes that researchers have used to evade the peer review process.
Enter PubPeer. This is an online forum for scientists to comment on research after publication. It fosters scientific discussion, with the pleasant side effect of as acting as an innate peer review. This is particularly useful for research that found its way around legitimate peer review prior to publication.
One word of caution: The publish or perish world of academic research is not always friendly to negative feedback. For this reason, comments on PubPeer are anonymous. This opens up its own obvious can of worms. It’s important to consider the entire body of evidence, and not just a subset of anonymous comments.
Retractions – You Can’t Put Toothpaste Back in the Tube
Finally, there is the issue of retractions. According to Dr. Oransky, there has been a ten-fold increase in the number of retractions but publications have only increased 40 percent in the same time period. Worse, two-thirds of retractions are due to misconduct.
The news gets worse before it gets better. About one-third of publications do not indicate that a study has been retracted. This means the wonderful research you are relying on that you found in the journal itself may have been retracted because of poor methodology, plagiarism, or even fraud.
In the end you must be diligent and search for further information on any research you plan to rely on. Right now your best bet is a Google search. However Retraction Watch plans to publish a database of retractions next year. In the meantime you can check their leaderboard of the most retracted authors.
We are making healthcare, lifestyle and political choices based on this information! But it’s not such a gloomy scenario. In fact, science hasn’t changed – people have always manipulated the system. But now we have the miracle of the Internet to illuminate their actions. It’s not even a reason we can’t trust science as a whole. Science has always been a process of making observations, proposing hypotheses, and testing them. While there is room for dishonesty and fraud in science, the science world has become even more interconnected via the web. We now have better tools than ever to help us move toward a more complete picture of scientific truth.
Amy Rogers MD is not a practicing physician and nothing written here should be taken as medical advice from either Amy or AssetBuilder. Medical decisions should be made with care in consultation with your health care provider.