Author + information
- ↵*Reprint requests and correspondence:
Anthony N. DeMaria, MD, MACC, Editor-in-Chief, Journal of the American College of Cardiology, 3655 Nobel Drive, Suite 400, San Diego, California 92122, USA.
Although I don’t have any objective data, I would bet that peer review of medical scientific publications must rank near the bottom of the list of topics of interest to the general public. It would be hard to imagine that editorial peer review would ever be the focus of a Robin Cook thriller or be the cover story of one of the weekly news magazines. Nevertheless, the June 4 New York Times devoted an entire feature article to the topic by lead medical writer, Lawrence Altman, MD. Although the piece was accurate, from my admittedly sensitive position, it seemed to be a harsh assessment. Quoting the then Editor of the British Medical Journal, Dr. Steven Lock, Altman admonished that “Editors, the arbiters of rigor, quality and innovativeness in publishing scientific work, do not apply to their own work the standards they apply to judging the work of others.”
The stimulus for the article in the New York Times was the publication of the Proceedings of the Fourth International Congress of Peer Review in Biomedical Publication in the June 5 issue of the Journal of the American Medical Association (JAMA). Scheduled as it was in the days immediately following September 11, the meeting was absent some participants (primarily American). Nevertheless, 40 of 43 scheduled presentations were delivered, and 58 of 65 posters were presented. Organized under the sponsorship of JAMA and the direction of Dr. Drummond Rennie, Deputy Editor, the conference presented data resulting from systematic studies of the editorial peer-review process. Topics discussed included Authorship and Contributorship, Editorial Peer Review, Bias, Ethical and Legal Issues, and E-Journals and On-Line Information. Not surprisingly, most of the studies were carried out by editors or those involved with current medical journals. In aggregate, the Congress and Proceedings represent a serious attempt by those responsible to address problems in, and improve the system of, editorial peer review.
Dr. Altman utilized the occasion of the JAMA publication to lament the shortcomings of peer review as practiced by current medical journals. He indicated that, despite the time and expense required for peer review, data are sometimes published which are erroneous, plagiarized, or even fraudulent. Moreover, the process occasionally fails to identify work of exceptional importance, as evidenced by rejections of Jenner’s report of the first development of smallpox vaccine as well as two other papers whose authors were subsequently awarded the Nobel Prize. These failings were at least partly attributed to the fact that reviewers did not examine the original raw data from the experiments. Errors in statistical analysis were pointed to as a frequent occurrence in scientific papers. Finally, Dr. Altman observed that authors sometimes overinterpret the implications of their findings and fail to acknowledge their limitations, both in their papers and in communications with the lay press.
That the above missteps of editorial peer review exist at all is problem enough. However, Dr. Altman observes that a journal’s decision to publish a manuscript has evolved over the years into a sort of “Good Housekeeping Seal of Approval,” conveying legitimacy upon a manuscript that may not be deserved. He regrets the fact that journals restrict communication of research findings to the lay press until after publication, a process that may slow the flow of information. Also pointed out, as has been recognized for many years, was the fact that most manuscripts that are rejected by one journal ultimately appear in another. Thus, peer review seems often to determine where papers are published, rather than which papers.
It is not my purpose here to provide a point-by-point response to Dr. Altman’s critique of editorial peer review. However, a few brief observations do seem in order. It is true that actual raw data are not evaluated in peer review, but representative illustrations of raw data are submitted for most papers. Considering the large number of submissions, it would clearly be logistically impossible for any journal to review the actual raw data for all papers received. Even were this feasible, it would often still be impossible to verify that the experiments had actually been performed in the manner described. However, we can take consolation in the fact that all truly important findings are subjected to the crucible of reproduction, and those that fail are ultimately discarded. It should also be recognized that rejection of a manuscript certainly does not mean that it is without merit. JACC is currently able to publish less than 20% of the submitted papers, and many that fail to achieve adequate priority are of considerable value. Nevertheless, by prioritizing papers, peer review directs manuscripts to journals with more appropriate audiences and provides readers with the advice of experts as to which communications are apt to be of greatest importance. With the explosion of medical literature, even specialists cannot read everything written in their field. With regard to embargo of articles to the press, it seems apparent that the lay public is often not sufficiently knowledgeable to interpret the significance of new research data. Reason would therefore dictate that physicians have the opportunity to read new reports first, so that they can interpret them for patients, tempering both the unrealistic fears and unrealistic expectations of their patients. In fact, it was surprising that Dr. Altman would want the press and patients to have immediate access to new research reports, many of which he fears exaggerate their importance and neglect their limitations. In terms of the charge that landmark advances such as Jenner’s development of vaccination are occasionally rejected, every editor I have met acknowledged that the peer-review process is fallible. Sometimes editorial decisions lead to embarrassing mistakes, putting this endeavor in company with many other aspects of daily life. However, even this imperfection has a silver lining; it ensures the humility of editors and provides reassurance to authors that rejection of any of their papers was likely one of the erroneous decisions.
As stated earlier, it is neither necessary for me, nor my responsibility, to defend peer review. There is no doubt in my mind that the editorial system has definite limitations and that Dr. Altman does a service in reminding us of this. As an editor, reviewer, author, and reader I have experienced many of these limitations firsthand. Nevertheless, peer review has survived some 200 years as the cornerstone of medical literature. I believe the reason for this is a characteristic it shares with democracy. Specifically, just as democracy is the “worst form of government… except for all others,” so editorial peer review is the worst system for the scientific literature … except for the alternatives. Although modifications such as unblinded reviews, publication of reviews with manuscripts, and even publication of virtually all submitted manuscripts (at least on-line) have been proposed, and may yet be incorporated, they have thus far either failed to demonstrate benefit or have not been embraced by authors and readers. Despite its imperfections, even critics admit that the peer-review process often results in removal of errors and significant improvement in published papers, and it occasionally salvages manuscripts that may otherwise have been lost.
My initial Editor’s Page indicated my belief that medical literature and journals will undergo dramatic change over the next decade. Nevertheless, the fundamental functions fulfilled by peer review are likely to persist. I believe that readers will continue to seek guidance as to the most original, accurate, and relevant literature to read and that authors will continue to compete for this audience and for the recognition that this designation conveys. Moreover, I continue to think that arbitration by an objective editor of the critiques of expert reviewers will still be the method by which new research is prioritized for publication and improved in presentation. While physicians who read journals must remain cognizant of the imperfections of peer review, I believe that this process will continue to be the cornerstone of the medical literature. With all its flaws, editorial peer review is still superior to all the alternatives.
↵1 Editor-in-Chief, Journal of the American College of Cardiology
- American College of Cardiology Foundation