Peer-Review Problems, add sexism to the list

Fiona Ingleby recently submitted a manuscript to a PLoS journal and received one terrible and sexist review rejecting it (interestingly the manuscript investigated the progression of PhD students to post docs in biology and found evidence for gender bias in their data). She and her co-author appealed three weeks ago with no response from the journal at which point she posted excerpts from the review to twitter, unleashing a storm of outrage towards the reviewer and PLoS. I'll let everyone else comment on the problems of sexism that this raises (here, here, here, and here) and focus on the problems of the current peer-review system that this situation highlights.

Objectivity
One of the first problems was that the reviewer was not blind to the authors which allowed them to bring their sexist attitudes to the review. Even more inappropriately the reviewer took time to research the authors and made other comments about them including that they were too young and inexperienced. Introducing double-blind reviews may help prevent or slow down such easy efforts to undermine particular researchers. However, in practice, especially when publishing in specialized journals and when you are firmly planted into your research line, it is fairly easy to figure out who the senior author is on a manuscript, blinded or not. Generally, more than one person serves as a gatekeeper to publication with at least two, sometimes three and other times even more. By having more than one reviewer you can get different perspectives and improve the reach and impact of the paper. It can also be good for editors where consensus among reviewers indicates the potential high-impact quality of publications. But by having more than one reviewer, a pragmatic issue may be to help the editor screen out any bad reviews. Two reviewers with completely opposite opinions may indicate that the paper needs clarification or that one of the reviewers is trying to tank a paper (oh no scientists are people with emotions and egos too!) or is not qualified to review the paper (oh no sometimes reviews are passed off to under qualified trainees). This issue of objectivity and double-blind review leads into the next point of pre-registration.

Pre-registration
The review had no critiques that could be used to improve the manuscript. The closest that the review came to actually commenting on the science was that the manuscript was “methodologically weak” and “has fundamental flaws and weaknesses that cannot be adequately addressed by mere revision of the manuscript, however extensive." Anytime that a review attempts to go after the methodology, it irks me a little. If we think back to the origin of any particular study, it has to go through at least one if not more reviews before you can even carry out the research. Every study gets a once over from IRB and other institutions to make sure that the study crosses some bar of ethics and usually includes that it is being carried out with good science. Next, many, if not most studies are carried out with funding. Funding success at federal levels is hovering around 10% and only slightly better in other realms meaning that any research that has funding has potential implications and is being carried out in a way that may achieve those implications that it beats 9 other solid research proposals. So it always seems odd that its not until after the study is complete that apparent methodological flaws make a study unworthy of publishing as if every published paper is perfect in its formulation and the way that it was carried out. Pre-registration, in which a study's methodology is submitted and if it is accepted is published regardless of the results, allows us to avoid these issues and provides early feedback in the publishing process.

Slow
Three weeks! Three weeks of sitting on an appeal that justifiably points out that the review did not actually comment on the science of the article and was sexist seems like a poor move for PLoS. Now the outrage from the science community is reiterating that sentiment. The circumstances of this review raises so many questions, what was the editor thinking sending that review out, why was there only one reviewer, why would you appeal after a terrible review process and not go somewhere else? Post-hoc review is slow with quick decisions on the order of a month and slow decisions that more than a year (not including shopping to multiple journals or going down the prestige ladder). Before even submitting a manuscript for review, researchers are faced with questions that often pit time to publication against ease of publication. Many researchers always go down the prestige ladder just because the turn around (i.e. rejection) can be fast and if something hits, its worth the time. Other researchers would rather publish fast and will choose specialized journals with reputations for turning around manuscripts quickly.

Could we improve the quality and speed of peer-review by giving more credit or reward (some people think this option removes objectivity) for peer-review? Perhaps peer-review could improve if we shift how we approach it. Instead of thinking of ourselves as the gate-keepers of the sanctity of science we could see ourselves as collaborators in trying to improve science and disseminate science, messy though it may be.

Reviewer Etiquette: Just because you're anonymous doesn't mean you need to troll

scienceblogs.com

Peer-review may be the cornerstone of the scientific process as it is the gate through which both forms of scientific currency (publications and grants) must pass. Peer-review serves as the quality control system in which all scientific discoveries and ideas are scrutinized and and vetted by other experts. Peer-review is meant to make scientific communication trusted, but peer-reviewed work isn't necessarily correct or conclusive, even if it has met some standard of science. After the peer-review process is over and a paper/grant has passed through the gate, it doesn't mean that the process is over, science must deal with it somehow either through responding with commentary, other related studies or replication attempts. However, it often seems as though peer-review is treated as the final say, that flaws are magnified and if the paper doesn't meet some arbitrary standard or give the anonymous reviewer enough citations then it is the end and that idea or that study is stopped dead in its tracks right then and there.

In general the responsibilities of reviewers are to:

• Comment on the validity of the science, identifying scientific errors and evaluating the design and methodology used
• Judge the significance by evaluating the importance of the findings and how/where the findings fit in the literature (including identify missing or inaccurate references)
• Determine the originality of the work based on how much it advances the field
• Recommend that the paper be published or rejected. Editors don't have to heed this recommendation, but most do (this also varies by the number of reviewers rounded up - sometimes the decision needs to be unanimous, sometimes if the reviews are split another is brought in to break the tie and sometimes the editor makes the decision regardless of the reviewers)

In single-blind (where the authors don't know the reviewer identities) and double-blind (where neither reviewers nor authors know each other's identities) post-hoc (as opposed to pre-registration) review tend to have a few problems. 

1) Focusing on methodology when the study has already been completed

Pre-registration would seem to solve some of the most frequent reviewer requests, methodological issues, including adding new conditions, new controls or becoming obsessed with the fact that an experiment was run a particular way that they don't quite like. This change would take care of the first responsibility of reviewers, correcting errors in the design and methodology before the study even takes place. 

2) Lack of responsibility/accountability for reviewers

The removal of reviewers being blinded to the authors and readers may help change the level of constructiveness of the comments made in reviews. A tumblr aggregates some of the reviews that don't appear to fall into any of the responsibilities I mentioned above. Removing the veil of anonymity should help to improve the dialog and impact of the review process (here and here). We may even give more significance and value to reviewing. Review is a service and valuable contribution, but given little value in the eyes of the determination of your impact as a scientist. Often it seems as though review is passed off to trainees whose only experience evaluating the literature comes from lab meetings and reading groups where group think causes everyone to dump on the papers and nitpick the papers into oblivion. By naming reviewers (as the Frontiers family does) we give reviewers acknowledgment and accountability. We could go even further and note what they contributed to the study and figure out ways to evaluate the impact that individuals make as a reviewer similar to how we use authorship on papers to evaluate impact.

In the end, peer-review is not the end of the process but just a part of the process that extends far beyond a paper being published. By giving more responsibility and reward for review we should improve the process and remove the "me against the world" feelings shared by both authors and reviewers.