"Show Me the Data" ISI's Impact Factors

Mike Rossner, Executive Director, The Rockefeller University Press, Heather Van Epps, Executive Editor, The Journal of Experimental Medicine, and Emma Hill, Executive Editor, The Journal of Cell Biology, wrote the editorial, “Show Me the Data,” in the latest issue (published online 17 December 2007) of The Journal of Cell Biology (Vol. 179, No. 6, 1091-1092). In it they contend that they cannot verify journal impact factors published by Thomson Scientific (formerly ISI). Considering the significance many faculty, academic departments and funding agencies assign to these impact factors in making tenure, promotion, funding and other decisions, the allegation is a very serious one.

Extracts:

The integrity of data, and transparency about their acquisition, are vital to science. The impact factor data that are gathered and sold by Thomson Scientific (formerly the Institute of Scientific Information, or ISI) have a strong influence on the scientific community, affecting decisions on where to publish, whom to promote or hire, the success of grant applications, and even salary bonuses . Yet, members of the community seem to have little understanding of how impact factors are determined, and, to our knowledge, no one has independently audited the underlying data to validate their reliability. . . .

When we examined the data in the Thomson Scientific database, two things quickly became evident: first, there were numerous incorrect article-type designations. Many articles that we consider "front matter" were included in the denominator. This was true for all the journals we examined. Second, the numbers did not add up. The total number of citations for each journal was substantially fewer than the number published on the Thomson Scientific, Journal Citation Reports (JCR) website (http://portal.isiknowledge.com, subscription required). The difference in citation numbers was as high as 19% for a given journal, and the impact factor rankings of several journals were affected when the calculation was done using the purchased data (data not shown due to restrictions of the license agreement with Thomson Scientific). . . .

It became clear that Thomson Scientific could not or (for some as yet unexplained reason) would not sell us the data used to calculate their published impact factor. If an author is unable to produce original data to verify a figure in one of our papers, we revoke the acceptance of the paper. We hope this account will convince some scientists and funding organizations to revoke their acceptance of impact factors as an accurate representation of the quality—or impact—of a paper published in a given journal.

Just as scientists would not accept the findings in a scientific paper without seeing the primary data, so should they not rely on Thomson Scientific's impact factor, which is based on hidden data. As more publication and citation data become available to the public through services like PubMed, PubMed Central, and Google Scholar®, we hope that people will begin to develop their own metrics for assessing scientific quality rather than rely on an ill-defined and manifestly unscientific number.

Full editorial