Peer review in science, in which independent scientists who are experts on the subject assess papers, but this system frequently receives harsh criticism about its effectiveness and transparency.  It came to light again in a humanities study which had a conclusion that everyone desiring social engineering from academia - that if people just talked to opponents of gay marriage they would change their minds - was found to have no data. It happened last year in a study claiming men did not take hurricane names seriously, it has happened for decades - but it happens in real science also.

A new paper on peer review discusses the weaknesses we all see - it is easy to hijack peer review when it is a volunteer effort that can drive out anyone who does not meet the political or cultural litmus test. Wikipedia is dominated by angry white men and climate science is dominated by different angry white men, but in both cases they were caught conspiring to block out anyone who dissented from their beliefs.  Then there is the fluctuating nature of guidelines. Some peer review is lax if you are a member, like at the National Academy of Sciences, while the most prominent open access journal is really editorial review, where they check off four boxes and it may never go to peer review or require any data, especially if it matches the aesthetic self-identification of the editor or they don't want to be yelled at on Twitter.

Those issues make peer review slow and inconsistent, says Catarina Ferreira, researcher at Trent University in Canada, the Institute of Research in Game Resources and CIBIO-UP in Portugal. The problem will only get worse, since there are now over 35,000 open access journals alone, all claiming to be legitimate.  

For the new paper, the authors focused on ecology and evolution. From the hundreds of scientific journals that are published in these fields, they chose to contact the 38 considered to be the top journals, as classified by Google Scholar. 

"We did it in this manner simply because we assumed that the journals that have a higher 'traffic' of articles received also have more pressure to improve the peer review system," argued Ferreira.

Peer review was defined as "not consistent" in all of the 38 journals selected on ecology and evolution. 

"The definition of an 'ideal' peer review is somewhat complex and, currently, even the journals that have the most organized systems struggle with weak reviews, which demonstrates that standardizing review criteria and guidelines does not at all resolve all of the weaknesses in the system," the researcher added.  

Some measures have been suggested as potential solutions. Science 2.0 has advocated getting rid of all fees, including subscription and open access companies making tens of millions of dollars each year at taxpayer expense, paying reviewers, and making them a mandatory part of academia, since 2007. 

"Applying correction factors to the h-index -the highest number of articles that an author has published and been cited at least the same number of times, paying fees to the reviewers or offering them royalties (such as discounts on subscription fees or acknowledgement notes) are some of the proposals. Moreover, some editors are making efforts to homogenize review criteria between them, such as the British Ecological Society, and some review guidelines exist, although there is no agreed criteria on which is the most relevant," says Ferreira. 

In the 38 publications analyzed, the disparity of methods used by the journals to instruct their reviewers on peer reviews was evident, from the complete absence of guidelines and unclear criteria, to more formal systems with forms and defined criteria. 

"None of the measures proposed to date has the potential to resolve the problems in the long term, because they are partial and not holistic. In our opinion, a contemporary peer review process, in which the current needs of the scientific community are addressed, should be centralized in a platform independent of journals, whose interests are above all financial - with clear review criteria and guidelines, adjusted according to the scientific field." 

They also propose that this centralization be led by scientists, since this would facilitate the standardization of the process, as well as increasing its transparency and reliability.

Citation: Catarina Ferreira et al. "The evolution of peer review as a basis for scientific publication: directional selection towards a robust discipline?" Biological Reviews 2015, doi: 10.1111/brv.12185.