Reviewer Etiquette: Just because you're anonymous doesn't mean you need to troll

scienceblogs.com

Peer-review may be the cornerstone of the scientific process as it is the gate through which both forms of scientific currency (publications and grants) must pass. Peer-review serves as the quality control system in which all scientific discoveries and ideas are scrutinized and and vetted by other experts. Peer-review is meant to make scientific communication trusted, but peer-reviewed work isn't necessarily correct or conclusive, even if it has met some standard of science. After the peer-review process is over and a paper/grant has passed through the gate, it doesn't mean that the process is over, science must deal with it somehow either through responding with commentary, other related studies or replication attempts. However, it often seems as though peer-review is treated as the final say, that flaws are magnified and if the paper doesn't meet some arbitrary standard or give the anonymous reviewer enough citations then it is the end and that idea or that study is stopped dead in its tracks right then and there.

In general the responsibilities of reviewers are to:

• Comment on the validity of the science, identifying scientific errors and evaluating the design and methodology used
• Judge the significance by evaluating the importance of the findings and how/where the findings fit in the literature (including identify missing or inaccurate references)
• Determine the originality of the work based on how much it advances the field
• Recommend that the paper be published or rejected. Editors don't have to heed this recommendation, but most do (this also varies by the number of reviewers rounded up - sometimes the decision needs to be unanimous, sometimes if the reviews are split another is brought in to break the tie and sometimes the editor makes the decision regardless of the reviewers)

In single-blind (where the authors don't know the reviewer identities) and double-blind (where neither reviewers nor authors know each other's identities) post-hoc (as opposed to pre-registration) review tend to have a few problems. 

1) Focusing on methodology when the study has already been completed

Pre-registration would seem to solve some of the most frequent reviewer requests, methodological issues, including adding new conditions, new controls or becoming obsessed with the fact that an experiment was run a particular way that they don't quite like. This change would take care of the first responsibility of reviewers, correcting errors in the design and methodology before the study even takes place. 

2) Lack of responsibility/accountability for reviewers

The removal of reviewers being blinded to the authors and readers may help change the level of constructiveness of the comments made in reviews. A tumblr aggregates some of the reviews that don't appear to fall into any of the responsibilities I mentioned above. Removing the veil of anonymity should help to improve the dialog and impact of the review process (here and here). We may even give more significance and value to reviewing. Review is a service and valuable contribution, but given little value in the eyes of the determination of your impact as a scientist. Often it seems as though review is passed off to trainees whose only experience evaluating the literature comes from lab meetings and reading groups where group think causes everyone to dump on the papers and nitpick the papers into oblivion. By naming reviewers (as the Frontiers family does) we give reviewers acknowledgment and accountability. We could go even further and note what they contributed to the study and figure out ways to evaluate the impact that individuals make as a reviewer similar to how we use authorship on papers to evaluate impact.

In the end, peer-review is not the end of the process but just a part of the process that extends far beyond a paper being published. By giving more responsibility and reward for review we should improve the process and remove the "me against the world" feelings shared by both authors and reviewers.

Improving the Scientific Journal?

I've been trying to wrap-up and submit my dissertation research and have been dealing with a number of different journals. I recently looked up how expensive it would be to start your own journal and found that the physical costs would be under a few hundred dollars a year, while the costs of people (e.g., Editors, Copywriters, etc.) would cost a few hundred dollars per paper which does't seem all that expensive when profit margins are at least 20% for publishers. But in starting your own journal, what things would you do to improve the process? Below I offer a few possibilities, but am interested in other ways (and am not as radical as others have suggested 1, 2).

1. Pre-registration
Pre-registration, where the design of a study and its subsequent analysis protocol is submitted to a journal before competing the study, is an emerging, but controversial topic, with some in support, some against, and some unsure.  Pre-registration appears to be a fundamental starting point for many of the other suggestions below, with preregistration encouraging open access, data sharing and encouraging attempts at replication. One of the main concerns about pre-registration is that it will constrain scientists from the fiddling and post-hoc data-mining that currently runs science. However, I don't see why pre-registration with your original design and analysis plan can't also report new analyses that come after the fact. The difference with pre-registration is that your original and possible non-significant results are reported as well as the finagled and later thought of results. Especially since we are over-reporting positive results, it will be nice to see why experiments fail as well as see non-replications of high-profile studies that are currently suspect but not challenged in the literature.

2. Open Commenting/Changing Peer-Review
Some journal have adapted a two-step review process in which papers that pass an initial review are posted for open commenting as a discussion paper and then later reviewed in the traditional ad hoc review system. As the review process often takes a minimum of a few months and could potentially gone on for years, the open commenting period helps alleviate the strain of finding reviewers. This also opens up scientific dialog spurring and collaboration.

3. Open Access
Without open access most of scientific articles are only available to those who are willing to pay for the articles or work in a place who buys publisher access. Open access asks researchers to pay for their research to be made available to anyone once accepted. The Right to Research Coalition provides a list of benefits across various agents. Though Nature and other publishers caution that Open Access are not a panacea that some think.

4. Data Sharing
Data sharing provides a number of benefits including: scientific integrity, increasing the impact of research, preserving data for future use and teaching purposes. Some fields seem to be more interested in open data, but there are a some issues, like confidentiality and vulnerable subjects, that need to be kept in mind.

5. Removing Impact Factor
The impact factor (IF) - the number of citations a journal receives in a year divided by the number of articles published in the journal, is meant to be a measure of the impact and quality of scientific output. Over time, IF has become a shorthand prestige and has lead to both journals and scientists chasing every higher numbers. However, this chase has perverted whether any particular article is making an impact and how science is crafted and published.

Edit 1/15/2015 - Michael Eisen who co-founded PLoS did an excellent AMA on Reddit yesterday about academic publishing addressing many of these issues.