The director of the National Institutes of Health, Dr. Francis Collins, is worried about the lack of reproducibility and 'secret sauce' in a large number of studies funded by their $30 billion government agency. 

Fraud happens everywhere, as does cherry-picking of results, but the more scientific the field, the less it happens. It's hard to get 2,000 people to be fraudulent about an experimental physics result while a lone psychologist writing about surveys of college students is difficult to catch.

Carnegie-Mellon and Stanford scholars say they have a solution for Dr. Collins, at least - use 
massive online laboratories and videogames to engage large numbers of non-professional investigators and prevent scientists from manually testing their own hypotheses.

CMU's Adrien Treuille and Stanford's Rhiju Das argue that an online, game-like approach actually is more scientifically rigorous than the standard practice of scientists proposing an explanation for some phenomenon and then testing that hypothesis through experimentation. They believe that massive online labs could be a model for the entirety of science.

Earlier this year, Treuille and Das reported in the Proceedings of the National Academy of Sciences on the success of their own online lab, an RNA-design project called EteRNA, which has produced unprecedented design insights that have advanced knowledge of RNA (ribonucleic acid).

"If you strip away the game part, projects such as EteRNA present a fundamentally new model of remote science that can prevent many common forms of scientific fraud," said Treuille, an assistant professor of computer science and robotics at Carnegie Mellon. "We registered more than 150,000 participants who contributed in excess of 2 million human-hours to EteRNA. That means there were a lot of eyes, a lot of people looking over each other's shoulders as hypotheses were developed and experimental results evaluated. Everything is out in the open."

Online participants use computer design tools to propose RNA designs that meet certain criteria. The designs are then synthesized in the Stanford lab of Das, an assistant professor of biochemistry. The results are made available to the entire EteRNA community for analysis and use in future design challenges.

Treuille said that transparency makes it difficult for any individual to retrospectively adjust scientific hypotheses to match experimental results, or to cherry-pick data to reflect a scientist's biases. "It's not typically acknowledged, but having the same team both develop a hypothesis and test it in the lab creates a conflict of interest — something that may be contributing to a plague of irreproducible results in many research studies," Treuille said. "EteRNA separates these two important processes without inhibiting the science."

The sheer number of people involved in the research also addresses a problem associated with a hallmark of modern biology — the use of high-throughput experimental techniques.

"A single DNA or RNA sequencing run can generate billions of data points," Das noted. "Following up those experiments with cycles of hypothesis generation and testing is critical to establishing scientific truth. But it can be expensive and time-consuming. It's tempting to skip the extra work and cherry-pick the data to produce a publishable manuscript.

"If you have thousands of people helping you, however, the data deluge isn't so bad," Das added.

Though people attracted to EteRNA by its game-like elements were not necessarily trained scientists, many developed deep understandings of RNA design as they used computerized tools to design molecules and then saw which designs were successful when synthesized in Das' lab.

Treuille and Das said setting up a massive online lab isn't cheap, but should be within the budget of a conventional life sciences grant.

"The major current barrier may instead be the career risks that these projects pose for their creators," they write in the new position paper. "In particular, videogames, which appear critical for recruiting scientifically engaged citizens, are generally viewed as incompatible with 'serious,' or rigorous research."