Revising peer review

Marc Bellemare with a post on adding a crowd-sourcing element to peer review. Marc highlights an editorial in Review of Financial Studies that asserts that what is often viewed as the last step–the publication of a paper in a peer review journal–should be seen as the first step, with the real value of a paper determined by the give and take that comes after publication.

I have blogged a bit about peer review the past few months ( open science, alternative metrics to track value).

It worries me to move away from the peer review system we have without having something better to replace it, especially the statement in the editorial that we shouldn’t worry about whether a paper is correct or not, the crowd sourcing will sort that out. Evidence is very important to policy discussions (or at least it should be….) and one of the reasons I blog is to try and lengthen the ‘shelf life’ of good, peer reviewed research, so we need to strive for identifying ‘correct’ research, with the crowd sourcing perhaps helping to identify how important a study may be. I stand by my ideas of how to change peer review (along with the uncertainties) that were published in January, 2012, and which are summarized below.

• The identity of reviewer and reviewee should be known to one another
• The title of manuscripts under review should be public, along with the authors of the manuscript and the identity of the reviewers
• How long the reviewers have been reviewing the manuscript should be public
• How long authors have had a request for revision should be public
• Upon publication, the correspondence between reviewers/editors/authors should be public (this is important because often people say “why didn’t you do this subanalysis”; often it was done, but cut from a published paper due to length restrictions)
• The use of online early publication is a good thing; I wonder if it will eventually become the only modality? (I only take one journal in hard copy now, Health Affairs, and otherwise utilize Duke University’s global subscription service)
• Gated papers hinder academic investigation and discourse, but I am unsure of how to fund journals without subscriptions

This is an important conversation for researchers interested in policy, and one that seems to produce more questions than answers.

A few suggestions on peer review

This post was also published in the LSE impact blog today.

A few weeks back, the LSE Impact blog had a piece by Jason Priem on the use of twitter by academics that suggested peer review journals might become a thing of the past. Austin Frakt and I wrote a brief post noting that as much as we love twitter, the role of peer review journals cannot be replaced by twitter, blogs or anything else (and we really believe in blogs!). We need the slow deliberative process that emphasizes trying to get it right, as opposed to doing it quickly. We concluded:

We absolutely need the slow, peer review system as the foundation of thoughtful, careful scholarship. Twitter and other social media are important additions that can give scholarly content “reach” and “relevancy”. However, it’s a both/and, not an either/or proposition. Traditional peer review journals should remain the bedrock of the research evidence that can be brought to bear on health policy.

However, I think the peer review process often is too slow and could be sped up without losing precision. In addition, I think there is too much secrecy in the process and a bit more disclosure would likely be good (though there are likely pluses and minuses). Following are a few personal thoughts about changes I would like to see in the peer review process used by journals that are based on my personal experience and preferences (I have published ~70 peer review papers and reviewed dozens of manuscripts for journals). Others will likely have different thoughts, and I would be interested to know them. This is not meant to be a definitive word, just my personal thoughts.

• The identity of reviewer and reviewee should be known to one another
• The title of manuscripts under review should be public, along with the authors of the manuscript and the identity of the reviewers
• How long the reviewers have been reviewing the manuscript should be public
• How long authors have had a request for revision should be public
• Upon publication, the correspondence between reviewers/editors/authors should be public (this is important because often people say “why didn’t you do this sub-analysis”; often it was done, but cut from a published paper due to length restrictions)
• The use of online early publication is a good thing; I wonder if it will eventually become the only modality? (I only take one journal in hard copy now, Health Affairs, and otherwise utilize Duke’s global subscription service)
• Gated papers hinder academic investigation and discourse, but I am unsure of how to fund journals without subscriptions

Making the identify of all parties public and how long they have had to review a manuscript or complete revisions should provide some “speed” to the process. More information about the give and take leading up to the publication would provide a fuller context for the paper. And a big issue going forward is the financial model by which journals survive. Who should pay for them and how much?

DT

update: a few tweaks for clarity