 |
| scienceblogs.com |
Peer-review may be the cornerstone of the scientific process as it is the gate through which both forms of scientific currency (publications and grants) must pass. Peer-review serves as the quality control system in which all scientific discoveries and ideas are scrutinized and and vetted by other experts. Peer-review is meant to make scientific communication trusted, but peer-reviewed work isn't necessarily correct or conclusive, even if it has met some standard of science. After the peer-review process is over and a paper/grant has passed through the gate, it doesn't mean that the process is over, science must deal with it somehow either through responding with commentary, other related studies or replication attempts. However, it often seems as though peer-review is treated as the final say, that flaws are magnified and if the paper doesn't meet some arbitrary standard or give the anonymous reviewer enough citations then it is the end and that idea or that study is stopped dead in its tracks right then and there.
In general the responsibilities of reviewers are to:
- Comment on the validity of the science, identifying scientific errors and evaluating the design and methodology used
- Judge the significance by evaluating the importance of the findings and how/where the findings fit in the literature (including identify missing or inaccurate references)
- Determine the originality of the work based on how much it advances the field
- Recommend that the paper be published or rejected. Editors don't have to heed this recommendation, but most do (this also varies by the number of reviewers rounded up - sometimes the decision needs to be unanimous, sometimes if the reviews are split another is brought in to break the tie and sometimes the editor makes the decision regardless of the reviewers)
In single-blind (where the authors don't know the reviewer identities) and double-blind (where neither reviewers nor authors know each other's identities) post-hoc (as opposed to pre-registration) review tend to have a few problems.
1) Focusing on methodology when the study has already been completed
Pre-registration would seem to solve some of the most frequent reviewer requests, methodological issues, including adding new conditions, new controls or becoming obsessed with the fact that an experiment was run a particular way that they don't quite like. This change would take care of the first responsibility of reviewers, correcting errors in the design and methodology before the study even takes place.
2) Lack of responsibility/accountability for reviewers
The removal of reviewers being blinded to the authors and readers may help change the level of constructiveness of the comments made in reviews. A tumblr aggregates some of the reviews that don't appear to fall into any of the responsibilities I mentioned above. Removing the veil of anonymity should help to improve the dialog and impact of the review process (here and here). We may even give more significance and value to reviewing. Review is a service and valuable contribution, but given little value in the eyes of the determination of your impact as a scientist. Often it seems as though review is passed off to trainees whose only experience evaluating the literature comes from lab meetings and reading groups where group think causes everyone to dump on the papers and nitpick the papers into oblivion. By naming reviewers (as the Frontiers family does) we give reviewers acknowledgment and accountability. We could go even further and note what they contributed to the study and figure out ways to evaluate the impact that individuals make as a reviewer similar to how we use authorship on papers to evaluate impact.
In the end, peer-review is not the end of the process but just a part of the process that extends far beyond a paper being published. By giving more responsibility and reward for review we should improve the process and remove the "me against the world" feelings shared by both authors and reviewers.
No comments:
Post a Comment