Reproducing Scientific Results: Challenges & Opportunities in Biology

Scientific Results Under New Scrutiny?

After its initial foray into the public dialogue about reproducibility, Nature published a piece that called for an “authoritative forum to hash out collective problems” on the matter of the ease of reproducibility in science. They also put out a reference and primary sources collection so that lay media and others could make sense of what was being discussed. Still, there is confusion about what reproducibility means for professionals and savvy consumers alike. It must be understood that this is more than just whistle-blowing—it is the core of scientific research.

“Reproducibility is an integral component of science,” says Anna Crist. “If you cannot reproduce results, there is a flaw in the research. It could be a wide variety of factors. An overlooked part of the method, an unreliable material (contamination, bad batch of a chemical, etc.), a low replicate number, a misleading statistical analysis, and a great many more.”

Claims of accessible reproducibility methodologies are being challenged outside the realm of biological science, too. The fields of psychology and sociology have recently opened up about the rising difficulty in the verification and validation of research results. There is a coming watershed moment for biological research and its effects can be seen in many other disciplines at this precise moment.

“If the reproducibility crisis continues, I think we risk incurring more costs, losing credibility as scientists, and reducing public scientific funding,” warns Crist. “When a lab publishes unreliable results, basing future studies on those results is equivalent to not only building a house on a bad foundation but also putting in a bad foundation for your neighbors’ houses.”

As a professional with the utmost dedication to research flow and the publication of verifiable results—whether positive or negative—Crist’s career has been made as an experiment engineer with an eye for precision.

“This concept of a reproducibility crisis is the phenomenon that, too often, researchers in one lab are unable to obtain the same results of a published study from another lab and sometimes even from their own studies,” says Crist. One of her premiere concerns is that scientists can waste time and funding by basing future studies on these published yet flawed results. At present, her time spent as a researcher shows that, in general, there is uncompromising loyalty to experimental rigor in scientific research.

“I was involved in a large study at the University of Oregon trying to identify compounds that would increase the health- and lifespan of C. elegans and other nematode species,” says Crist. “This was part of an NIH-funded collaboration with Rutgers University and The Buck Institute for Research on Aging.”

Perhaps the most salient fact about this collaboration was that if Crist’s teams couldn’t obtain reproducible results, their grants would not be renewed. This shows a shift in focus in the wider research community from ‘results’ to ‘reproducible results.’ This is a distinction that might be lost on the lay individual, but for Crist, it means that the stakes are always high.

“We hoped to find compounds that would have a positive effect across diverse genetic backgrounds (multiple species and strains within those species)—the worm-equivalent of general instead of personalized medicine,” she says. To adhere to her department’s conditions of continued research, it was critical that Crist’s team establish standard operating procedures (SOPs) for three laboratories that were geographically separated.

“We went to extreme measures to ensure everything was as identical as possible. Each lab bought the same brand and model of incubators just for these experiments,” explains Crist. “We had individual temperature monitors in each box of Petri-dishes that the nematodes grew on to track environmental changes over their entire lives. We ordered the same reagents that were from the same brand and lot number.”

It was this attention to rigor and control over the experiments’ environments that made it possible for the team to bolster their reputation and secure funding for the coming year. They managed to almost entirely eliminate variation in results between the three labs. In fact, there was more variation from experiment to experiment within a given lab than there was between the three labs.

A study conducted by Freedman, Cockburn, and Simcoe called “The Economics of Reproducibility in Preclinical Research” has estimated the cost of flawed or irreproducible research at $28 billion per year in the United States alone, with 65 percent of the cost incurred by the pharmaceutical industry. This doesn’t mean that only the pharmaceutical industry should improve its standards, however. All scientific disciplines should heed these figures.

She identifies competition as the main obstacle when it comes to increasing research result transparency. “High competition is perpetuated by low funding,” explains Crist. “There is an enormous amount of pressure to publish amazing findings in a high-profile journal and if you don’t, the grant money soon dries up.”

It is this combination of pressure and a lack of resources that can cause scientists to consciously—or subconsciously—perform less rigorous experiments. It can even mean the purposeful adjustment of experiment or study results to support a more favorable or profitable outcome. “This could also mean rushing through assays, taking inaccurate measurements, using fewer replicates, censoring data points here and there to tailor results, or performing a different statistical analysis that gives you that precious low p-value,” says Crist. “These all add up to irreproducible results.”