Author + information
- ∗University of Minnesota School of Medicine and VA Medical Center, Minneapolis, Minnesota
- †Icahn School of Medicine at Mount Sinai, New York, New York
- ↵∗Reprint requests and correspondence:
Dr. Jagat Narula, Icahn School of Medicine at Mount Sinai, One Gustave L. Levy Place, New York, New York 10029.
“Throughout the whole history of science most of the really great discoveries which had ultimately proved to be beneficial to mankind had been made by men and women who were driven not by the desire to be useful but merely the desire to satisfy their curiosity.”
—Abraham Flexner (1)
A previous Editor’s Page in the Journal focused on the challenges faced by authors, reviewers, editors, and readers during a time when science and medical publishing are exploding with new information (2). Nearly 35 trillion gigabytes of published research data are expected to be available within a decade (3), and a related question is how quality should be assessed in this vast expanse. Traditionally, the prestige of the journal in which research was published provided the imprimatur of quality. Currently, despite clear limitations and almost universal criticism, the impact factor (IF), a measure of the number of times papers are cited in other publications, seems to persist as the de facto measure of quality (4). An exchange in the Guardian (5) between Randy Schekman, the 2013 Nobel laureate in medicine and physiology, and the editors of 3 of the highest-rated journals underscores the difficulty of this issue. Philip Campbell, the Editor-in-Chief of Nature, although concerned with the undue influence of IFs, suggests that “it is up to the scientific community to decide how much importance they want to place on papers that appear in the journal” (6). On the other hand, a survey of more than 20,000 scientists showed that the reputation of the journal and its IF were among the top 3 criteria for choosing a journal in which to publish. Dr. Schekman takes the opposite view, suggesting that defining the prestige of a journal through its IF makes for luxury journals that seek exclusivity by making waves to increase subscriptions rather than publishing the highest-quality research.
The IF of a journal depends on many factors and does not indicate the quality of any individual piece of research, especially because many papers, even in high-quality journals, may be rarely cited (7). The determinants of citation include both perceived scientific value and a host of nonscientific factors. Thus, citation count is not always linked to journal quality, and the definition of quality still remains elusive.
Consequences of a “Quality Rating”
Perceived journal quality has implications beyond where to publish, bragging rights after publication, or the satisfaction that comes from enlarging the envelope of science. Promotions, salaries, incentives, and research grants depend on publishing in elite journals, and these factors, in turn, can critically affect an investigator’s ability to publish his or her work in these very journals. Subject areas that are encouraged become trendy, and there is relentless gravitation to fields fostered by such journals, which can distort scientific progress and misrepresent its value to society (8). More recently, nations and research enterprises have incorporated this into their calculations of the rate of return for scientific output of universities, research institutions, and investigators. Multiple experiments in this domain, especially in the social sciences, have yielded mixed results. The sensitivity of this topic is evident in the controversies about criteria envisaged for productivity, and it has fomented revolt from journal editors and researchers and, in some cases, even political intervention (9).
What Is Quality in a Journal?
In the early “bibliometric period,” journal quality depended on subjective impressions and quasi-objective criteria, like the prestige and reputation of the learned body represented by a journal, the gravitas of its editorial board, low manuscript acceptance rates, robust readership, or subscription volume. These were acceptable in more genteel times but are insufficient for the high-stakes “publish or perish” environment of modern academia and the fiercely competitive economic landscape of the publishing industry. Complex scientific metrics are developing but remain inadequate (10). Despite this evolution, top-tier journals are too often still identified by difficult-to-quantify terms to reflect prestige, impact, or influence. Some would define rigorous peer review as the cornerstone of quality, but several eminent scientists, like the 2002 Nobel laureate Sidney Brenner, have argued that this index hinders the quality of science (11). Although nearly every editor is uncomfortable with it (4), IF somehow survives and even thrives as the imprimatur of quality.
Another Look at IF as a Surrogate of Journal Quality?
The use of IF as the currency of quality can introduce subtle bias favoring activities that enhance IF. IF correlates with prestige, but the relationship may be stronger in the minds of clinicians (12) than investigators (13). Although a miniscule number of journals focus primarily on IF, the race toward quality may have subconsciously changed the publishing landscape, for example, reducing the number of elements in the “denominator”—like the number of papers published—or increasing the number of review articles (Figure 1). Some have suggested that this metric makes top journals restrictive and channels papers into winner and loser pigeonholes to the detriment of science.
Drs. Schekman and Brenner (5,11), among others, lament that a paper can be often cited because it describes good science or because it is eye-catching, provocative, or even wrong. A few frequently cited papers can cast disproportionate prestige upon uncited papers published in a high-impact journal. In fact, a substantial proportion of papers in the most highly regarded journals have low citation rates, and some are never cited at all (7,14). Citations generally improve with time, but approximately 20% to 35% of papers in the top 3 cardiology journals were not cited between 2007 and 2013 (Figure 1). The situation may be even worse in other specialties (7), leading to the famous quip (14) that most academic papers are read by about 3 people—the author, the reviewer, and the editor. Publishing a low-impact paper in a journal with a high IF can, therefore, lead to inaccurate assessment of a paper’s quality, inflating the prestige of the researcher, and overestimating the impact of his or her current or future work. Various types of papers affect IF differently, and it is often unclear whether a journal’s high IF indicates that many papers are cited many times or a few papers are cited overwhelmingly. This is not captured by the overall IF and has not been well investigated.
In this issue of the Journal, Nuti et al. (15) used the Gini coefficient (a parameter of interest to economists for assessing income inequality) to determine whether there is an association between a journal’s citation distribution (variation) and its IF. Among 129 cardiology journals, those with a higher IF had a lower Gini coefficient (i.e., higher-quality journals exhibited more balanced distribution of citations, and the quality of papers [as defined by IF] was uniformly high, rather than limited to just a few papers that were frequently cited). This relationship was less robust among the 4 highest-impact journals (Gini coefficient 0.45 to 0.68, whereas the median value for the whole group was 0.54), suggesting that the types of manuscripts published (e.g., reviews, guidelines, and position papers) were a powerful determinant of IF. As a corollary, high-impact papers published in some journals contributed disproportionately to IF. The editors routinely have to make strategic decisions, and some decisions may inadvertently favor some types of manuscripts over others (indirectly influencing the direction of science as Schekman  and others have suggested). The editors of other journals may not have access to certain types of papers, because specialty societies generally publish guidelines in their own vehicles. During the period from 2010 to 2012, guidelines represented 6.4% of total articles in Circulation, 2.9% in JACC, and 2.3% in the European Heart Journal (EHJ), but accounted for a high portion of citations: 22.4% for Circulation, 17.7% for the EHJ, and 8.3% in JACC (H. Krumholz et al., personal communication, March 2015). Gini coefficients tightened (less inequality in IF between the top 4 journals) when these types of papers were removed: changing by −8% for Circulation, −4% for EHJ, and −2% for JACC. Nuti et al. (15) did not parse the effects of “massively cited” paper types (e.g., guidelines, consensus statements, or white papers vs. original articles), so firm conclusions cannot be drawn. Unlike data concerning reviews from the published pathology data (16), guidelines or papers of similar format are the most frequently cited papers in cardiology and rank among the 10 most-often cited papers in each of the top 3 cardiology journals: Circulation: 7 of 10; JACC: 7 of 10; and EHJ: 9 of 10 (2011 to 2015, from a Scopus database search performed February 21, 2015). Updated versions of these landmark papers continue to have sustained high impact, as exemplified by the heart disease and stroke statistics paper, which appears 5 times among the 10 most-cited papers during this period.
What Is the Future of Assessing Journal Quality?
As discussed in a previous Editor’s Page (2), despite the lack of agreed-upon criteria for quality, publishing only high-quality papers is the hallmark of a high-quality journal and the dream of every editor. Many other ranking tools are available, such as SCImago Journal Rank, or Google metrics. Like IF, these deal with mean values and are not a good index of the quality of any individual paper. It is difficult to anticipate how current metrics will endure when newer journals use “soundness” rather than “soundness + significance” as their criteria for publishing. Will future paradigms (17) like letting the “marketplace or end user review” rather than peer review comport with current concepts of “quality?” Newer measures, such as altmetrics, which provide dynamic, real-time analyses, are novel ways to assess publication data. These have not yet deeply penetrated the “quality” fabric, but several journals display these measures as reflections of quality. The web-savvy reader might respond better to popularity on social media, reference managers, or Twitter feeds, whereas a scientific investigator might wonder if downloads, page views, or dwell times on papers are appropriate indicators for the prestige of their research.
The future of assessing journal quality is uncharted. As publishing is commercialized, a truly free market may see journals bidding for valuable papers the way brokers trade in goods and services; in an extreme scenario, seminal papers may even become “Veblen goods.” On the other hand, national programs like the research excellence framework may spawn tiers of quality that might make where a paper is published less important than publishing in any highly rated journal tier (9), making quality a more amorphous concept. Given that the differences in quality between the top journals in a field like cardiology are smaller than the IF differences between them suggest, this may be a reasonable by-product of evolution. People will publish and read what suits their interests or what fits with their area of expertise without worrying about a specific number. Journals may wish to publish more in the areas they want to showcase without having to worry about what is “hot” in other areas or in competing journals.
The current creative disruption in publishing could generate a new pecking order for journals based on newer indexes of quality. As new metrics abound, it will be increasingly difficult to identify 1 journal as an undisputed leader. In both general science and clinical medicine, the proportion of top (1% to 5%) cited papers published in elite journals has already fallen, as highly cited papers appear in other journals with lesser reputations (18). In business and in politics, the leader is the one people choose to follow, and this will probably continue to apply to journals. As consumers become more influential, editors will probably shepherd science with the end user in mind, investigators will become more mindful about explaining why their work is important to society, and society will more vigorously seek a tangible return on its investment in science. Having said that, it is ironic that despite its flaws and forceful repudiation, the use of IF as a surrogate of quality refuses to go away. In a recent survey (19), a majority of researchers disdained the use of these metrics for promotion and tenure decisions. However, when asked what they consider the best criteria for evaluating researchers, their top answer was “publication in high-impact journals”—in other words, high IF journals. And, although we see the value of measurement, we cannot help but be fearful that the coming tsunami of scientometrics may cloud the very rationale for research. As Richard Feynman would say, “the pleasure of finding a thing out, the kick in the discovery, the observation that other people use it” (20). Caveat lector and caveat emptor.
- American College of Cardiology Foundation
- Flexner A.
- Chandrashekhar Y.,
- Narula J.
- ↵Mark Ware Consulting. Unlocking the value of research data. Available at: http://www.markwareconsulting.com/science-2-0/unlocking-the-value-of-research-data. Accessed March 11, 2014.
- ↵San Francisco DORA. San Francisco Declaration on Research Assessment. Available at: http://am.ascb.org/dora/. Accessed March 11, 2014.
- ↵Schekman R. How journals like Nature, Cell and Science are damaging science. The Guardian. December 9, 2013. Available at: http://www.theguardian.com/commentisfree/2013/dec/09/how-journals-nature-science-cell-damage-science. Accessed March 11, 2015.
- ↵Nature. A statement from Dr. Philip Campbell, Editor-in-Chief of Nature. Available at: http://www.nature.com/press_releases/editor-statement.html. Accessed March 11, 2014.
- Lariviere V.,
- Gingras Y.,
- Archambault E.
- Bishop D.
- Pontille D.,
- Torny D.
- ↵Dzheng L. How academia and publishing are destroying scientific innovation: a conversation with Sydney Brenner. Available at: http://kingsreview.co.uk/magazine/blog/2014/02/24/how-academia-and-publishing-are-destroying-scientific-innovation-a-conversation-with-sydney-brenner/. Accessed March 11, 2015.
- ↵Eveleth R. Academics write papers arguing over how many people read (and cite) their papers. Available at: http://www.smithsonianmag.com/smart-news/half-academic-studies-are-never-read-more-three-people-180950222/#kP32tmkiQpdjvUZ0.99. Accessed March 25, 2014.
- Nuti S.V.,
- Ranasinghe I.,
- Murugiah K.,
- et al.
- Derrick G.E.,
- Pavone V.
- Larivière V.,
- Lozano G.A.,
- Gingras Y.
- Feynman R.P.