Testing a new therapeutic intervention such as a drug or surgical procedure on human subjects is not an option so the vast majority are first tested on animals and only when they have been established in those trials can human trials be considered.

But in recent years cultural campaigns against animal testing have increased, making researchers increasingly leery of them. That means size constraints and limited statistical power, and as a result the scientific literature contains many studies that are either uncertain in their outcomes or even contradictory.

A way around this limitations in the ability to conduct animal trials has been to conduct a "meta-analysis" - where scientists collect data from a large number of published studies on the same intervention, combine them using statistical methods, and then end up with a basis on which to decide whether to proceed with human clinical trials. 

A new claim in the journal PLoS Biology is that substantial bias in the reporting of animal studies may be giving a misleading picture of the chances that potential treatments could work in humans.

Professor John Ioannidis specializes in evidence-based medicine and clinical research methodology and has long contended that most published research findings are false. With Konstantinos Tsilidis and colleagues, Ioannidis examined 160 previously published meta-analyses of animal studies looking at potential treatments for a range of serious human neurological disorders, such as multiple sclerosis, stroke, Parkinson's disease, Alzheimer's disease and spinal cord injury.

These meta-analyses covered 1,000 original published animal studies comparing more than 4,000 sets of animals. The authors contend their "meta-analysis of meta-analyses" is showing the problem because it used the most precise method in each meta-analysis as an estimate of the true effect size of a particular treatment. It then asked whether the expected number of studies had statistically significant conclusions. 

The authors found that more than twice as many studies as expected appeared to reach statistical significance.

Observed and expected number of “positive” studies by type of neurological disease. Link: doi:10.1371/journal.pbio.1001609

The authors are not suggesting fraud or misconduct, but that the "excess significance bias" comes from two main sources. One is that scientists conducting an animal study tend to choose the method of data analysis that appears to give them the "better" result. The second arises because scientists usually want to publish in higher profile journals; such journals tend to strongly prefer studies with positive, rather than negative, results. Many studies with negative results are not even submitted for publication or, if submitted, either cannot get published or are published belatedly in low-visibility journals, reducing their chances of inclusion in a meta-analysis.

They contend it is likely that the types of bias reported in their paper have been responsible for the inappropriate promotion of treatments from animal studies into human clinical trials. It also seems unlikely that this phenomenon is confined to studies of neurological disorders; rather this is probably a general feature of the reporting of animal studies.

The authors suggest several remedies for the bias that they have observed. First, animal studies should adhere to strict guidelines (such as the ARRIVE guidelines) for study design and analysis. Second, animal studies (like human clinical trials) should be pre-registered so that publication of the outcome, however negative, is ensured. Third, availability of methodological details and raw data would make it easier for other scientists to verify published studies.

They left out a fourth, obvious one: fewer animals studies means less statistical power. And that means obstructionists need to stop framing science as an ethical issue.

Citation:Konstantinos K. Tsilidis, Orestis A. Panagiotou, Evangelos Evangelou, Emily S. Sena, Rustam Al-Shahi Salman, Malcolm R. Macleod, David W. Howells, Eleni Aretouli, John P. A. Ioannidis, 'Evaluation of Excess Significance Bias in Animal Studies of Neurological Diseases', PLoS Biol 11(7): e1001609. doi:10.1371/journal.pbio.1001609