Image from Seralini et al. 2012

That was the whole basis of the sensational 2012 paper that actually resulted in entire African countries rejecting all GMO imports. Reread that previous sentence because it shows the power of this one, poorly-designed study with 120 rats.

So livers were being harvested from these 20 females – several of which were compromised and euthanized “early” (2 from the control group, and 5 from the “treatment” group) at different ages due to the tumor load. Is it not obvious that these additional factors of tumor load and different ages would confound any data collected from their livers?

The 2015 paper goes on to show electronic microscope analysis of liver sections from females. But it turns out the photograph of the control female was actually the same photo at a different magnification as that shown for the control male hepatocyte image in the 2012 paper. The authors have since stated that was an honest mistake and have submitted a corrigendum, but go on to suggest that there are differences in the hepatocytes from Roundup-treated rats, specifically showing “a disruption of glycogen dispersion”, a disruption of nucleolar function and an overall decreased level of transcription. How transcription can be determined based on an electron micrograph is unclear. No mention is made of the “fatty liver disease” promised in the 2017 paper’s title.

So let me sum this up for those of you who may be lost. The original, highly-controversial 2012 study was done on 120 rats. The most recent study was performed on the livers of a subset of 10 female control rats and 10 female rats from that same 2012 study that were in “Roundup group (A)” which received 50 ng/L glyphosate equivalent dilution in their water. We do not know their water intake so have no idea of actual dosage of “Roundup”; we have little histological data on female liver samples - let alone a diagnosis of fatty liver disease; we know that the control and “treated” rats were euthanized at a variety of differing ages, and that the majority of these female rats had huge tumors that required several of the rats in both the control and Roundup groups to be euthanized before two years of age. And the livers from these 20 rats were the basis of the most recent “omics” paper. There is a saying in science (and perhaps other disciplines): “garbage in – garbage out”.

So let’s plough on – and read the 2017 paper which concludes that the metabolome and proteome analyses of the livers from the “Roundup-drinking” rats versus the controls “showed a substantial overlap with biomarkers of non-alcoholic fatty liver disease and its progression to steatohepatosis”. Hooray – now THERE is a testable hypothesis – so what ARE the biomarkers of non-alcoholic fatty liver disease? In other words, what proteins and metabolites might you expect to see upregulated (or downregulated) if in fact animals had non-alcoholic fatty liver disease? I have read the paper several times now and seen no reference to a paper that answers that question. So in the absence of  knowledge of fatty liver biomarkers, and given the fact no pathology diagnosed “fatty liver disease”, to conclude that “Multiomics reveal non-alcoholic fatty liver disease in rats following chronic exposure to an ultra-low dose of Roundup herbicide” is – to put it kindly – overstating the results of the research and making conclusions beyond that supported by the data.

Interestingly the bioinformatics analysis in this 2017 paper appears to be an improvement on previous works by this group in that the p-values were adjusted to account for the fact there were a high number of metabolites measured (1906 proteins and 673 metabolites), and there was therefore a need to do corrections for multiple comparisons to try to minimize the number of false positives. The authors even include a discussion of the need for corrections for multiple comparisons on page 9, and correctly state that there is a need to do this when measuring hundreds or thousands of observations to reduce the chance of making a type I error (false positive). However, they then lament the fact that there was a lack of statistical significance following the multiple comparison correction for all but three metabolites, due to the small sample size. That is the point! That is why these studies need to have sample size determinations based on the hypothesis being tested.

This study, which was based on the experimental design of a 90 day subchronic toxicity study (OECD, 1998) such that 10 animals were assigned to each group, was critiqued by the German Federal Institute for Risk Assessment (BfR) for small sample size for that very reason.

“subchronic studies show a substantially lower variation of age-related pathological changes between animals within a group while those changes are inevitable in long-term studies. As the published study has confirmed, the two-year duration of the study is of the order of the expected life span in rats including the Sprague Dawley strain that was used in the study. This strain, provided by the breeder Harlan, is known to develop spontaneous tumors, particularly mammary and pituitary tumors, at relatively high rates compared to other strains (Brix et al., 2005; Dinse et al., 2010). Therefore, it can be expected that a significant number of animals develop age-related illnesses or die for diverse reasons already during conduct of the study. The distribution of the cases of death between groups can be random, and a number of 10 animals per sex and group is too low to confirm a trend or an effect. Furthermore, no statements on statistically significant dose-response-relationships can be made. Larger sample sizes, as recommended for carcinogenicity studies in OECD Test Guidelines No. 451 or No. 453, would be required in order to allow precise statements with respect to the findings."

In other words you need to have bigger sample sizes to perform long term studies because many changes are associated with old age – especially when working with a rat strain that is known to develop spontaneous tumors, particularly female mammary and pituitary tumors!

Frustratingly, when the multiple comparisons removed all but three of the 673 metabolites as being statistically significant due to multiple comparison correction in the 2017 paper, the authors just went ahead and included the 55 that had a significant uncorrected p value(!), because “the non-adjusted statistically significant levels” fit a narrative, and so were revived from the statistical trash can on the basis that “they were found to be non-random and thus biologically meaningful”. This is the very definition of confirmation bias which is what multiple comparison correction and correct experimental design is trying to weed out because scientists are people too, and they are not without their own preconceived notions of how the world works.

More concerning, this 2017 paper is yet another in a string of papers from this group that was accepted in a peer-reviewed journal, in this case Scientific Reports, an online journal from the publishers of Nature. The problems in experimental design, lack of supporting pathology data on the test subjects, and wildly subjective overinterpretation of the results should have been grounds for soundly rejecting this manuscript. We live in an age of the willful neglect of scientific evidence, and the emergence of “alternative facts” and realities.  As a scientist it worries me that papers like this are published in apparently respected journals. I remember once hearing a member of the activist industry say that “peer-reviewed journals are the tool of the enemy” suggesting they were the gold standard communication tool for scientists to report inconvenient facts. At the time I did not appreciate the importance of that statement, and concerningly it appears that this in no longer the case. If we can’t trust the peer-review process to ensure the integrity of papers published in scientific journals, what can we trust? This is a problem that should worry the entire scientific community, not only those concerned with the topic of this particular paper.