Dr. Francis Collins, former head of the Human Genome Project
and current Director of the National Institutes of Health, has a $30 billion
problem. Taxpayers are spending that much money each year, primarily with
scientists at very prominent universities like Johns Hopkins, and he isn't sure
how much of it is actually resulting in science.
To be sure, it seems like science. It involves experiments and published
papers and lab equipment. The people doing the real work are generally outstanding scientists, but in January
he reported in Nature that up to two-thirds of the science they fund is not reproducible.
Reproducibility is, as we all know, one of the pillars of the scientific
method.
One of the other pillars of modern science is peer review, and it may be on
just as shaky ground. Peer review involves an anonymous
panel of experts critiquing a paper. Ethically, it
should not be someone who agrees in advance to give it special attention.
Yet a variety of journals have had just that problem. A recent paper concluded that if a hurricane had a female name, it
resulted in more deaths than male-named hurricanes - because implicit sexism
made people take them less seriously. It initially passed the guffaw test with several national publications, probably because it appeared in Proceedings of the National Academy of Sciences,
the publication of one of the most prestigious clubs in the world, the National
Academy of Sciences. The National Academy is so elite even Carl Sagan, he of Cosmos fame, was
excluded from its ranks.
Membership has its privileges. If you are a member, you can basically get
anything published. And if
you edit it yourself, you just have to answer a few questions to an editorial
board. Their system is delightfully archaic, and it has
led to some head shaking in the science community, like in this case, where a psychologist who studies gender bias fast-tracks a paper about gender bias in hurricane names that was written by marketing people.
Apart from the paper's obvious methodological flaws, which were soon well-documented, the bigger concern is the suspect nature of peer review itself. In his examination of the larger problem, Collins points to "poor
training of researchers in experimental design," an "emphasis on making
provocative statements," publications that don't "report basic elements of experimental design"
and that "some scientists reputedly use a ‘secret sauce’ to make
their experiments work — and withhold details from publication or describe them
only vaguely to retain a competitive edge."
That's a peer review issue, it is the very thing peer review is supposed to
catch. PNAS has stumbled this way before.
In 2002, a PNAS paper claimed that pesticides were
leading to hermaphroditic frogs. The hand-picked, prearranged editor for the
paper was Professor David Wake of Berkeley, who just happened to be close friends
with Professor Tyrone Hayes, the lead author of the study. In 2010, PNAS
published another paper by the same lead author which said the same
herbicide, called atrazine, caused complete feminization of frogs. The
hand-picked, prearranged editor for that second paper? Once again, Professor
David Wake, Hayes' friend at Berkeley.
In neither case did the studies include the data. No one has ever been able to
reproduce the results of either paper and the EPA never used those studies
because they didn't meet the criteria for legitimate work. Most non-scientists would be surprised to learn that researchers are not required to make their data public, and many, such as Dr. Hayes, never do.
PNAS is not the only publication that gets tripped up by
flawed studies, they are simply the only one that
lets members edit their own work or fast-track friends for publication. A
site called Retraction Watch even exists to talk about such flawed studies. It
is a favorite of scientists because most scientists do good work and they want
to weed out the the bad studies.
But peer review clearly needs to modernize. Professor
Michael Eisen, a biologist at U.C. Berkeley, has a special status because not
only is he a world-class researcher, he is also co-founder of the Public Library of
Science (PLOS), one of the largest non-profit science publishers in the world.
He wrote in an email, "We need to get away from the notion, proven wrong
on a daily basis, that peer review of any kind at any journal means that a work
of science is correct. What it means is that a few (1-4) people read it over
and didn't see any major problems. That's a very low bar in even the best of circumstances."
Since traditional peer review is clearly overwhelmed by the 25,000 journals that exist today, that means post-publication peer review will need to become
much stronger. But for that to happen, the data needs to be available so the
work can be reproduced, which is exactly what Collins would like to see also.
Because they would rather light a candle than curse our
science darkness, PLOS is taking steps to improve such post-publication peer review: beginning in March, they began requiring that data be publicly available. That means anyone with the
ability should be able to reproduce, validate and understand the findings in a
paper.
That will mean much greater scrutiny for flawed claims about sexist weather
events and hermaphroditic frogs, hopefully before they appear on every news station in America.
