Positive outcome bias in peer review of journal submissions

January 01, 0001

Positive outcome bias in peer review of journal submissions

If positive-outcome bias exists, it threatens the integrity of evidence-based medicine. These US authors sought to determine whether positive- outcome bias is present during peer review by testing whether peer reviewers would (1) recommend publication of a "positive" version of a fabricated manuscript over an otherwise identical "no-difference" version, (2) identify more purposefully placed errors in the no-difference version, and (3) rate the "Methods" section in the positive version more highly than the identical "Methods" section in the no-difference version. Two versions of a well- designed randomized controlled trial that differed only in the direction of the finding of the principal study end point were submitted for peer review to 2 journals in 2008-2009. Of 238 reviewers for The Journal of Bone and Joint Surgery and Clinical Orthopaedics and Related Research randomly allocated to review either a positive or a no-difference version of the manuscript, 210 returned reviews.

They found: "Reviewers were more likely to recommend the positive version of the test manuscript for publication than the no-difference version (97.3% vs 80.0). Reviewers detected more errors in the no-difference version than in the positive version (0.85 vs 0.41). Reviewers awarded higher methods scores to the positive manuscript than to the no-difference manuscript (8.24 vs 7.53), although the ‘Methods’ sections in the 2 versions were identical."

The authors concluded: "Positive-outcome bias was present during peer review. A fabricated manuscript with a positive outcome was more likely to be recommended for publication than was an otherwise identical no- difference manuscript."

These are eye-opening and concerning findings.

For the full abstract, click here.

Arch Intern Med 170(21):1934-1939, 22 November 2010 © 2010 to the American Medical Association
Testing for the Presence of Positive-Outcome Bias in Peer Review-A Randomized Controlled Trial. Gwendolyn B. Emerson, Winston J. Warme, Fredric M. Wolf, James D. Heckman, Richard A. Brand, Seth S. Leopold. Correspondence to Dr.Leopold: [email protected]

Category: HSR. Health Services Research. Keywords: publication bias, peer review, evidence-based medicine, randomized controlled trial, journal watch.
Synopsis edited by Dr Linda French, Toledo, Ohio. Posted on Global Family Doctor 14 December 2010

Pearls are an independent product of the Cochrane primary care group and are meant for educational use and not to guide clinical care.