A few suggestions on peer review–ctd.

Continuing the conversation on how to revamp peer review…yesterday’s issue of Inside Higher Ed has a piece by Scott Jaschik making the case that peer review is “dead, but just doesn’t know it” at least in the Humanities. I don’t think that is true, at least not in health policy/medicine/economics, nor should it be.

In the comments and in a few emails, folks were asking for more on the plusses and minuses of moving away from authors and reviewers being blinded to the identity of the other. In thinking this through I ran across  Bradley Voytek’s interesting post in Scientific American blog on peer review back in November, 2011. The post is titled “What is Peer Review For?” and this answer is given:

The scientific peer-review process increases the probability that the scientific literature converges–at long time scales–upon scientific truth via distributed fact-checking, replication, and validation by other scientists. Peer review publication gives the scientific process “memory”.

This post has links to many interesting pieces on the purpose of peer review and its shortcomings, and I can’t do the rich dialogue justice, so just go read through the links if you are interested in this topic. I don’t have a neatly wrapped up bottom line, but the below points may amplify a few of my thoughts:

  • At its best peer review can improve the quality of published research. This should improve our quest for scientific truth. That should be the goal
  • I think (unnecessary) delay harms the process of identifying scientific truth (too fast also harms it if it causes error)
  • I believe the primary positive effects of making identities of reviewers and authors known is to speed the review process, out of the belief that reviewers and authors will be embarrassed to delay too long in doing what they have said they will do (review the manuscript/complete revisions)
  • In seeking scientific truth, I think that replicability is particularly important, and more openness of the negotiation between author/editor/reviewer could aid by providing more information
  • The biggest positive of maintaining blind review is to prevent bias of reviewers either for or against publication based on reputation of authors or institutions
  • Interestingly, in review of grants by NIH and many other funders, track record/ability of the team/institutional capability is an explicit review criteria. Is this at odds with goal of blind review in journals? There are many issues here including degree of risk taken with public research money….those decisions of course effect the pipeline of publishable research
  • When I review papers I often think that I know who wrote the paper. Is that better or worse than actually knowing? How about being wrong?

In collecting more information, I found this interesting paper by Richard Smith in the Journal of the Royal Medical Society that reviews these issues (and studies) more elegantly than have I. The most important studies he discusses are related to randomized trials done that show that impact of blinded review on review quality; unfortunately, they provide mixed results with the initial trial showing an improvement in review quality due to blinding with follow up trials not doing so. Further studies that assessed open review also found no impact on review quality.* They raise many questions about reviewer training that I won’t review.

This is an important topic. The further into it I go, the number of questions I have increases, while answers decrease.


*Key studies reviewed in the Smith paper (I have not reviewed all of the studies independently)

McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. A randomized trial. JAMA. 1990 Mar 9;263(10):1371-6.
Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D. Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. JAMA. 1998 Jul 15;280(3):240-2.
van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review: a randomized trial.JAMA. 1998 Jul 15;280(3):234-7.
van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ. 1999 Jan 2;318(7175):23-7.

A few suggestions on peer review

This post was also published in the LSE impact blog today.

A few weeks back, the LSE Impact blog had a piece by Jason Priem on the use of twitter by academics that suggested peer review journals might become a thing of the past. Austin Frakt and I wrote a brief post noting that as much as we love twitter, the role of peer review journals cannot be replaced by twitter, blogs or anything else (and we really believe in blogs!). We need the slow deliberative process that emphasizes trying to get it right, as opposed to doing it quickly. We concluded:

We absolutely need the slow, peer review system as the foundation of thoughtful, careful scholarship. Twitter and other social media are important additions that can give scholarly content “reach” and “relevancy”. However, it’s a both/and, not an either/or proposition. Traditional peer review journals should remain the bedrock of the research evidence that can be brought to bear on health policy.

However, I think the peer review process often is too slow and could be sped up without losing precision. In addition, I think there is too much secrecy in the process and a bit more disclosure would likely be good (though there are likely pluses and minuses). Following are a few personal thoughts about changes I would like to see in the peer review process used by journals that are based on my personal experience and preferences (I have published ~70 peer review papers and reviewed dozens of manuscripts for journals). Others will likely have different thoughts, and I would be interested to know them. This is not meant to be a definitive word, just my personal thoughts.

  • The identity of reviewer and reviewee should be known to one another
  • The title of manuscripts under review should be public, along with the authors of the manuscript and the identity of the reviewers
  • How long the reviewers have been reviewing the manuscript should be public
  • How long authors have had a request for revision should be public
  • Upon publication, the correspondence between reviewers/editors/authors should be public (this is important because often people say “why didn’t you do this sub-analysis”; often it was done, but cut from a published paper due to length restrictions)
  • The use of online early publication is a good thing; I wonder if it will eventually become the only modality? (I only take one journal in hard copy now, Health Affairs, and otherwise utilize Duke’s global subscription service)
  • Gated papers hinder academic investigation and discourse, but I am unsure of how to fund journals without subscriptions

Making the identify of all parties public and how long they have had to review a manuscript or complete revisions should provide some “speed” to the process. More information about the give and take leading up to the publication would provide a fuller context for the paper. And a big issue going forward is the financial model by which journals survive. Who should pay for them and how much?


update: a few tweaks for clarity

On The Record (with daily recap)

  • RWJF: Federally-Facilitated Exchanges and the continuum of state options

Today in TIE: Reflex, Austin looks at pricing CABG, Don updates the Center for Health Care Research and Transformation’s ACA flow chart, Aaron provides more context for why the Plan B decision matters, and TIE wishes readers Happy Hanukkah.


On The Record (with daily recap)

  • CBO: Report on federal budget deficit during first months of FY 2012
  • Mathematica: MPR launches new working paper series
  • RWJF: Do students who are more active do better in school?

Today in TIE: Reflex, Austin intros a series on premium support, and notes that Gov. Romney is keeping his options open on that issue, Don updates on teaching about the individual mandate, Aaron looks for Plan C on the FDA over-the-counter-morning-after pill controversy, and he is having trouble finding the words a lot this week, this time per Sen. Santorum’s take on the individual mandate, while Kevin rocks it out in the NEJM on the First Amendment and smoking


On The Record

  • Kaiser: Webcast Noon 12/7 EST CMS estimates of state-by-state health spending
  • CBO: Spending patterns for prescription drugs under Medicare part D

Today in TIE: Reflex, Austin parses moral hazard, Dr. Rorschach brings a chart and Don says Medicare is simple while Austin says it is complicated, Don notes that health care never has a recession (good news/bad news), while Harold praises George W. Bush on AIDS day, and Kevin looks at monopoly litigation, Pittsburgh style

On The Record (with daily recap)

  • NIH: Discontinution of Tenofovir gel in VOICE HIV Clinical Trial

Today in TIE: Reflex, Austin defends reduced form equations, and he and Don love twitter but want to keep peer review, Don glances at short v. long term debt, while Kevin imagines how we could improve the training of doctors

On The Record (with daily recap)

  • CBO: Estimate of ARRA impact in third quarter, 2011

Today in TIE: Reflex, Austin on why the ACA is not enough, and he analyzes Medicare’s decision to cover Provenge, Aaron takes a look at potential Avastin savings in Medicare, while Kevin grades the media in covering that story, while Don notes the post super committee reality is a lot like the pre committee reality.

On The Record wishes you a Happy and Joyful Thanksgiving….it will be back on Monday.

On The Record (with daily recap)

Today in TIE: Reflex, TIE is thankful that Brad Flansbaum is ok (wear your seat belt!), Austin considers retail health clinics and passes on a budget calculator, Don recommends a great book, while Aaron looks at health reform, Texas style, and compares tomato sauce to fruit and is glad he skipped last week’s freak out.

On The Record (with daily recap)

Today in TIE: Reflex, Don on bundled payments and nursing, Social Security and LTC, and prices! are really important, Austin answers what happens to losing bids under competitive bidding? and the reading list is reinstated by popular demand, Aaron shows us that premiums and deductibles have gone up, up, up, Kevin considers the Pharma angle of global health, and brings his expected clarity on the FDA’s Avastin decision.

On The Record (with daily recap)

  • GAO: VA Community Living Centers: Actions Needed to Better Manage Risks to Vets Quality of Life and Care
  • Commonwealth of Massachusetts: Recs of Special Commission on Provider Price Reform

Today in TIE: Reflex, Austin on the spreading of the competitive bidding bug (TIE ground zero?), different types of competition, and why traditional Medicare might thrive under competition, Don is planning to blog the flu, while Aaron puzzles over differential antibiotic use rates, and how hard it is to change behavior.