See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/276361881

Understanding Bibliometric Parameters and Analysis

Article  in  Radiographics · May 2015

DOI: 10.1148/rg.2015140036 · Source: PubMed

CITATIONS READS
119 739

4 authors:

Asim F Choudhri Adeel Siddiqui

The University of Tennessee Health Science Center Le Bonheur Children's Hospital
213 PUBLICATIONS   4,417 CITATIONS    12 PUBLICATIONS   236 CITATIONS   

SEE PROFILE SEE PROFILE

Nickalus R Khan Harris L Cohen

University of Tennessee The University of Tennessee Health Science Center
91 PUBLICATIONS   1,566 CITATIONS    210 PUBLICATIONS   2,745 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Elastography in Pediatric Liver View project

ACR Appropriateness Criteria ® Plexopathy View project

All content following this page was uploaded by Harris L Cohen on 20 April 2016.

The user has requested enhancement of the downloaded file.

 Note:  This copy is for your personal non-commercial use only. To order presentation-ready
INFORMATICS copies for distribution to your colleagues or clients, contact us at www.rsna.org/rsnarights.

736

Understanding Bibliometric
Parameters and Analysis1
Asim F. Choudhri, MD
Adeel Siddiqui, MD Bibliometric parameters have become an important part of modern
Nickalus R. Khan, MD assessment of academic productivity. These parameters exist for the
Harris L. Cohen, MD purpose of evaluating authors (publication count, citation count,
h-index, m-quotient, hc-index, e-index, g-index, i-10 [i-n] index)
Abbreviations: AIS = article influence score, and journals (impact factor, Eigenfactor, article influence score,
PoP = Publish or Perish, SJR = SCImago jour- SC­Imago journal rank, source-normalized impact per paper). Al-
nal rank, SNIP = source-normalized impact per
paper, WOS = Web of Science
though in recent years there has been a proliferation of bibliometric
parameters, the true meaning and appropriate use of these param-
RadioGraphics 2015; 35:736–746
eters is generally not well understood. Effective use of existing and
Published online 10.1148/rg.2015140036 emerging bibliometric tools can aid in assessment of academic
Content Codes: productivity, including readiness for promotions and other awards.
1
From the Department of Radiology, Le Bon- However, if not properly understood, the data can be misinter-
heur Children’s Hospital, 848 Adams Ave, preted and may be subject to manipulation. Familiarity with biblio-
G216, Memphis, TN 38103 (A.F.C., A.S.,
H.L.C.); and Departments of Radiology (A.F.C.,
metric parameters will aid in their effective implementation in the
A.S., H.L.C.), Neurosurgery (A.F.C., N.R.K.), review of authors—whether individuals or groups—and journals, as
Ophthalmology (A.F.C.), Pediatrics (H.L.C.), well as their possible use in the promotions review process, maxi-
and Obstetrics (H.L.C.), University of Ten-
nessee Health Science Center, Memphis, Tenn. mizing the effectiveness of bibliometric analysis.
Presented as an education exhibit at the 2013
©
RSNA Annual Meeting. Received February 15, RSNA, 2015 • radiographics.rsna.org
2014; revision requested May 23 and received
June 12; accepted July 1. All authors have dis-
closed no financial relationships. Address cor-
respondence to A.F.C. (e-mail: achoudhri@
uthsc.edu). Introduction
Bibliometrics is a field that uses quantitative means to evaluate aca-
demic productivity. This quantitative analysis of scientific literature
is rapidly changing with the creation of new evaluation tools, pa-
rameters, and normative data. The use of bibliometrics in academic
medicine is in a relative state of infancy.
RG  •  Volume 35  Number 3 Choudhri et al  737

The potential utility of bibliometrics may be

TEACHING POINTS apparent to those in academic medicine, whether
■■ The h-index indicates that a given author has had h articles (a) individuals seeking promotion or grants or
published, each of which has h or more citations. Additional
publications will not increase the h-index until and unless they
(b) departments or institutions looking to assess
are cited an appropriate number of times. the performance of their current and potential
■■ In multiple fields of medicine, a progressive increase in the faculty. However, bibliometric parameters are
h-index is seen for academic physicians as they move from also relevant for physicians outside academic
instructor to assistant professor to associate professor to full medicine, as well as nonphysicians. With the cur-
professor, although the absolute values vary by specialty. rent push for evidence-based medicine, evaluat-
■■ The impact factor takes into account all indexed citations ing the medical literature will become increas-
received by a given journal (target window) divided by the ingly important for physicians in private practice.
number of “citable” articles published by a journal (census
period). By convention, the impact factor usually refers to the
Bibliometrics can aid in determining which
previous 2 years of publication, although a 5-year impact fac- articles, journals, and authors will be the most
tor is also used. helpful in assessing the quality of a given prod-
■■ One widely cited article in a journal can artificially inflate the uct; however, an accurate understanding of these
impact factor for that journal, even if it is a controversial article tools is required for their effective use.
that is cited in criticism. Case reports are often cited only infre- In this article, we discuss bibliometric param-
quently, contributing to many journals opting to discontinue
eters in terms of author evaluation (publication
publishing this type of article for fear of negatively affecting
the impact factor. count, citation count, h-index, m-quotient, hc-
■■ The journal impact factor has been widely criticized as being
index, e-index, g-index, and i-10 [i-n] index) and
imprecise and subject to manipulation, yet it continues to be journal evaluation (impact factor, Eigenfactor,
the most widely used metric. AIS, SJR, and SNIP), as well as databases and
tools for calculating these parameters.

Evaluation of Authors
The most widely known bibliometrics param-
eter for evaluation of individual authors is publi- Publication Count
cation count, followed by citation count. Fueled Calculating the publication count is the simplest
by the imperfections of these data, as well as the bibliometric parameter. Typically, only peer-re-
recent availability of digital resources for cal- viewed articles in journals indexed in a database
culating more complex parameters, a new gen- such as PubMed or Index Medicus are consid-
eration of bibliometric tools has arisen (Table ered. Book chapters are not often considered.
1). Perhaps the h-index (1), which has recently Case reports are included, whereas editorials and
undergone numerous modifications (2–5), can opinion pieces are not typically included. This
be considered the prototype of advanced biblio- parameter, although easy to calculate, does not
metric parameters. take into consideration the position of authorship
There are several quantitative parameters that of an author or the quality of the journal. Articles
can help measure the academic strength of sci- published in “throwaway” journals are counted
entific journals (Table 2), the most widely known equally with those in more academically rigorous
of which is the 2-year journal impact factor (6). journals. With this metric, no distinction is made
Recently developed tools such as the Eigenfactor between original groundbreaking research and
provide more complex analysis of journals (7). less impactful articles such as case reports.
Each of these parameters has its own strengths
and weaknesses, from both theoretic and practi- Citation Count
cal standpoints. The citation count for an article is a method of
Bibliometric parameters are playing an in- giving weight to articles that have influenced
creasing role in the evaluation of academic pro- subsequent publications. The limitation of this
ductivity, readiness for promotion and/or tenure technique is that a widely read educational or
(8), and scores on grant applications (9), with informative article may not be cited, despite
journals with higher impact factors being more having a positive impact on the dissemination of
likely to receive submissions of high-quality information. A case report that provides guid-
manuscripts. A variety of databases exist for ance to others who encounter the same rare
obtaining the information required to calculate entity may be cited much less frequently than
bibliometric parameters (Table 3). These data- other types of articles. In addition, citation
bases have sparked recent interest; however, bib- count does not differentiate between positive
liometrics is not a new concept, and important and negative citations. Thus, an article cited in a
analyses have been performed in the past in the critical fashion could still influence the citation
field of radiology (10). count positively. Self-citations by an author are
738  May-June 2015 radiographics.rsna.org

Table 1: Comparison of Bibliometric Parameters for Evaluating Researchers

Calculated Calculated by Calculated

Parameter by Scopus Google Scholar by WOS Strengths Weaknesses
Publication Yes Yes Yes Provides an overall Does not differentiate
count estimate of quan- between articles such as
titative impact on case reports and higher-
literature impact articles (eg, meta-
analyses and randomized
controlled trials)
Citation Yes Yes Yes Provides an overall Accumulates over time and
count estimate of quali- can never decrease, may
tative impact on be compromised by self-
literature citations, is susceptible to
manipulation
h-Index Yes Yes Yes Combines publica- Can never decrease, com-
tion count and parison of authors across
citation count disciplines is difficult, can
to form a single vary by database, does
index not address frequently
cited articles
hc-Index No Yes (PoP) No Time-weights cita- Does not give credit to fre-
tion to remove quently cited articles, can
temporal bias vary across disciplines, is
from h-index not readily provided by
most databases
m-Quotient Yes No Yes Time-weights cita- Does not give credit to fre-
tion to remove quently cited articles, can
temporal bias vary across disciplines, is
from h-index not readily provided by
most databases
e-Index No Yes (PoP) No Gives credit to Complements but does not
frequently cited replace information ob-
articles tained from h-index
g-Index No No No Includes citations Does not distinguish between
that are beyond old and new articles; is
h-index; provides highly susceptible to arti-
summation of e- ficial elevation by frequent
index and h-index citations of one article
i-n Index No No No Simple method of Does not distinguish between
calculating number old and new publications, is
of significant publi- susceptible to self-citations,
cations does not account for fre-
quently cited publications
Note.—PoP = Publish or Perish, WOS = Web of Science.

also counted. Self-citations may be appropriate an attempt to describe the quantity and quality
when an author is building on the research per- or impact of a given author’s academic publica-
formed by his or her research group; however, tions and is based on a set of the author’s most
self-citations of uncertain relevance to the newly frequently cited articles. The h-index indicates
published work may be considered dishonest. that a given author has had h articles published,
each of which has h or more citations (Fig 1a)
h-Index (1). Additional publications will not increase
Recognition of limitations in the utility of publi- the h-index until and unless they are cited an
cation count and citation count led to develop- appropriate number of times. For example, a
ment of a metric that attempts to quantify both radiologist with seven publications with three
publication count and citation count in a single citations each would have an h-index of 3, and
metric that is less prone to manipulation. The h- the h-index would remain at 3 until there are four
index is a dimensionless number that represents publications with four or more citations each.
RG  •  Volume 35  Number 3 Choudhri et al  739

Table 2: Comparison of Bibliometric Parameters for Evaluating Journals

Parameter Strengths Weaknesses

Impact factor Widely used to determine importance Susceptible to manipulation, not as robust as
(2 y) of a journal over a 2-year period other metrics
Impact factor Widely used to determine importance Susceptible to manipulation, not as robust as
(5 y) of a journal over a 5-year period other metrics
Eigenfactor Gives increased weight to citations from Difficult to compare numbers across disciplines,
more widely read journals may artificially lower score for a journal that
receives citations from less widely read (eg,
subspecialty) journals
AIS Suggests percentage of all scientific articles Difficult to compare numbers across disciplines,
published by a specific journal derived from Eigenfactor and susceptible to
same weaknesses
SJR Gives greater weight to journals that often Difficult to compare numbers across disciplines,
cite each other (thematically related artificially lowers score for a journal that re-
journals) ceives citations from less widely read journals
SNIP Gives greater weight to citations from the Assumes that the weighting between different
same field disciplines is “fair”; a journal that is cited in a
widely read journal may receive an artificially
lower SNIP score because its thematic “scope”
has increased
Note.—AIS = article influence score, SJR = SCImago journal rank, SNIP = source-normalized impact per paper.

Table 3: Comparison of Online Databases for Publication Tracking

No.of Tracks Calculates Calculates Ancillary Update

Database Journals Indexed Citations h-Index Bibliometric Indexes* Availability Interval
PubMed 25,000 No No No Free Daily
Embase 8300 No No No Subscription NA
Scopus 20,500 Yes Yes No Subscription Weekly
Google NA Yes Yes Yes Free Monthly
Scholar
WOS 11,000 Yes Yes No Subscription Weekly
Note.—NA = not available.
*Including hc-index, e-index, and g-index.

Conversely, a radiologist with seven publications In recent years, extensive work has been per-
with seven or more citations each would have an formed to determine, on a specialty-by-specialty
h-index of 7. Thus, articles that have been cited basis, normative data for h-index values at
less frequently (implying little impact on the sci- different levels of academic rank. In multiple
entific community) are given less weight in the h- fields of medicine, a progressive increase in the
index. Furthermore, incremental increases in the h-index is seen for academic physicians as they
h-index become progressively more difficult. It is move from instructor to assistant professor to
more difficult for an author to progress from 19 associate professor to full professor, although
articles with 19 citations each (h-index = 19) to the absolute values vary by specialty (8,11–21).
20 articles with 20 citations each (h-index = 20) Existing work has identified a progressive in-
than for an author to progress from two articles crease in h-index between the academic ranks of
with two citations each (h-index = 2) to three ar- clinical instructor (mean, 1.1 ± 2.7), assistant
ticles with three citations each (h-index = 3). professor (mean, 2.3 ± 4.1), associate profes-
Citations within the h-index do include self- sor (mean, 6.2 ± 7.2), and full professor (mean,
citations; however, owing to the nature of the 12.5 ± 10.9) in academic radiology departments
metric, it is difficult for most authors to bring in the United States (8). Thus, an h-index of
about a significant change in their h-index simply 14 is more than one standard deviation above
through self-citation. the mean for associate professors, which could
740  May-June 2015 radiographics.rsna.org

Figure 1.  (a) Graph shows the citation count for selected publications (in order from highest
to lowest number of citations). The curve crosses the identity line at publication number h, so
that the gray square has an area of h2. The area in the curve above this square (e2) represents
excess citations in the top h articles that do not contribute to the h-index. (b) Graph shows
that the g-index uses all citations up to publication number g; therefore, there are no excess
citations. By definition, g ≥ h. (Fig 1 adapted, with permission, from reference 5.)

be used to help justify early promotion of a ra- be multiplied by four, whereas that for an article
diologist from assistant to associate professor. published 4 years ago would be multiplied by
The stratification of academic rank appears to one and that for an article published 6 years
be more reproducible on the basis of h-index ago would be multiplied by four and divided by
than on the basis of publication count or cita- six. Thus, older published articles are given less
tion count; consequently, the h-index is used weight, and emphasis is placed on articles that
as a factor in academic promotions at some are more recent (4).
institutions.
e-Index
m-Quotient Given that incremental increases in the h-index
The m-quotient (or m-index) is a variant of the become progressively more difficult, additional
h-index and is defined as an individual’s h-index citations of articles that constitute the h-index do
divided by the number of years since his or her not count toward an individual’s h-index. A re-
first publication (3). This value represents the av- cently created adjunct parameter called the e-index
erage amount the author’s h-index has increased indicates the excess citations of the top h articles
per year over his or her publishing career and can that do not count toward the h-index (5). For in-
help differentiate between two authors with simi- stance, for an author with an h-index of 10, the e-
lar h-indexes but different career lengths. An h- index is the average number of citations beyond 10
index of 12 for an individual 10 years into his or for the 10 most frequently cited articles. In other
her career (m-quotient of 1.2) may be considered words, e is the square root of all excess citations for
as more substantial than an h-index of 12 for an articles that constitute the h-index. Alternatively, if
individual 24 years into his or her career (m-quo- one identifies the set of articles that constitute the
tient of 0.5). However, use of the m-quotient can h-index and adds the number of citations, h2 cita-
penalize individuals who demonstrate research tions are used for the h-index, and the remainder
productivity early in their career (eg, during are considered excess citations (e2) (Fig 1a). Thus,
undergraduate school) followed by years during e = (total citations − h2)0.5. It is probably easier
which their focus is on education (ie, medical to understand the equation after learning the
school and residency training). concept than the other way around. The e-index
does not have to represent a whole number and
hc-Index will typically decrease when the h-index increases;
The contemporary index (hc) is a variant of the however, the e-index will increase during the time
h-index that time-weights citations. It is derived when the h-index is stable.
by multiplying the citation count for an article
by four, then dividing by the number of years g-Index
since publication. The citation count for an ar- The g-index is defined as a number such that
ticle published in the current year would simply the top g articles are cited an average of g times
RG  •  Volume 35  Number 3 Choudhri et al  741

(or are cited g2 or more times). Whereas the e- tor differs from the impact factor in two impor-
index attempts to complement the h-index by tant ways. First, citations from more widely read
addressing excess citations beyond h (which are journals, as determined by the citing journal’s Ei-
ignored by the h-index), the g-index includes all genfactor score, are given greater weight, thereby
citations for the top g articles (2). That is, the limiting the ability to use articles in low-impact
top g articles average g citations each (Fig 1b). journals as a means of garnering excess citations.
Second, although there is a 1-year census period
i-10 (i-n) Index (as with the impact factor), the target window is 5
The i-10 index is the number of publications years. The mathematic algorithm used to calculate
that have been cited 10 or more times and rep- the Eigenfactor is much more robust and less sub-
resents an attempt to sift through unsubstantial ject to rapid fluctuations or manipulation.
work (ie, throwaway articles). An i-n index could The Eigenfactor gives increased weight to cita-
accordingly be calculated for any n, such as i-5 tions from more widely read journals by using the
(which may be more helpful in evaluating more concept of eigenvector centrality, which measures
junior authors) or i-100 (which could be applied the importance of a specific “node” to a network.
to compare entire departments). On the Internet, a more highly trafficked web-
site would receive a higher eigenvector centrality
Evaluation of Journals score than a less highly trafficked website. Simi-
larly, the Eigenfactor calculates which journals
Impact Factor (nodes) receive more citations (higher eigenvec-
The impact factor was developed by Eugene tor centrality score). Citations from more active
Garfield and the Institute for Scientific Infor- journals are given more weight than citations
mation (acquired by Thomson Scientific and from less active journals.
Healthcare in 1992). The impact factor takes At its core, this is what gives the Eigenfactor
into account all indexed citations received by a little more credibility than the impact factor. It
a given journal (target window) divided by the is more difficult to “game” where citations come
number of “citable” articles published by a jour- from and to do so consistently. It is also very diffi-
nal (census period). By convention, the impact cult for a journal to artificially increase the number
factor usually refers to the previous 2 years of of citations it receives from a more popular and
publication, although a 5-year impact factor is widely read journal. Journals with a higher Eigen-
also used (6). factor than impact factor are those that have gar-
Because different sciences can have vastly dif- nered the attention of more established journals
ferent publication and citation rates, it is incor- and more researchers within their community.
rect to use the impact factor to compare journals
in different fields. It is also very difficult to com- Article Influence Score
pare individual researchers. Instead, the impact The AIS is derived from the Eigenfactor score
factor is best used to compare different journals (23). The first step in deriving the AIS is to de-
from the same scientific discipline. termine the number of articles published by a
The impact factor has been published since journal over a 5-year period and divide by the
1972 and has been widely used to determine the total number of articles published by all journals
importance of a journal. Unfortunately, this has during the same period. This gives an idea of
led to many criticisms regarding impact factor and what percentage of the total number of scientific
to certain changes in editorial policy. For example, articles were published in a given journal. The
the definition of what constitutes a citable article Eigenfactor score is then divided by this percent-
in a journal can be manipulated to decrease the age, and the number is normalized to 1. An AIS
denominator and increase the impact factor. One greater than 1 means that each article in that
widely cited article in a journal can artificially in- journal has above-average influence, whereas a
flate the impact factor for that journal, even if it score below 1 means that each article has below-
is a controversial article that is cited in criticism. average influence. For instance, the 2012 AIS for
Case reports are often cited only infrequently, RadioGraphics was 1.087, suggesting that articles
which contributes to many journals opting to dis- in RadioGraphics have a greater influence than the
continue publishing this type of article for fear of average article in the scientific literature.
negatively affecting the impact factor.
SCImago Journal Rank
Eigenfactor The SJR was developed by the SCImago Re-
The Eigenfactor was developed by researchers at search Group at the University of Extremadura
the University of Washington and the University of in Spain (24). Like the Eigenfactor, the SJR
California at Santa Barbara (7,22). The Eigenfac- uses a page-rank algorithm to determine which
742  May-June 2015 radiographics.rsna.org

citations are from more widely read journals, print” articles, which have previously not been
with these citations being given more weight. available on other databases. The major limita-
The main difference between the Eigenfactor tions of PubMed are that (a) articles are con-
and the SJR is that the former relies on the In- fined to the biomedical and life sciences journal
stitute for Scientific Information WOS database literature and (b) there is very little citation
(25), whereas the SJR relies on the Scopus da- analysis. MEDLINE is the most common source
tabase (26). of material sought by physicians using PubMed,
In 2012, an updated version of the SJR although PubMed does include other databases.
known as the SJR2 was introduced (27). The
SJR2 differs from the SJR in that it measures Scopus
the cosine of the citing and cited journals to Launched in 2004 by Elsevier, Scopus (26) is
determine the thematic relationship of the jour- the largest online bibliometric database and in-
nals. Journals that often cite each other are con- cludes journal articles from all major disciplines
sidered to be thematically close and are given published from 1966 onward, including articles
greater weight. In addition, unlike any other bib- from the social and physical sciences that are not
liometric parameters, the SJR2 divides the pres- included in PubMed. Citation analysis is more
tige gained by a journal by the number of citable robust with Scopus than with PubMed and is
documents. The more often that related journals available for articles published after 1996; how-
cite a specific journal, the more prominence that ever, there are stated plans to extend archiving
journal is given in its respective discipline. This back to 1970. A unique advantage of the Scopus
computation was included to address a fun- database is individual author identification, which
damental issue that affects many other biblio- groups articles by author on the basis of affilia-
metric measures: As more journals and articles tion and coauthors. This allows separation of re-
are added to research databases, bibliometric sults for authors with similar names, and authors
parameters are “diluted,” and comparison of can report errors or omissions to maintain the
numbers over time becomes limited. accuracy of their listings. In contrast, PubMed,
Google Scholar, and WOS all search for specific
Source-normalized Impact per Paper strings of text to group authors, so that authors
SNIP was created by Professor Henk Moed at with similar names are not separated. More re-
the University of Leiden, the Netherlands (28). cently, Scopus has added partial indexing of ar-
Similar to the SJR, SNIP gives greater weight ticles published online ahead of print. However,
to citations from the same scientific discipline. with Scopus, each author has a unique identifier,
Citations in fields that have fewer overall cita- which provides accurate author identification.
tions are given more weight. In essence, SNIP Unlike PubMed, Scopus is not free to users. Al-
divides a journal’s citation count per paper by though it is owned by Elsevier, Scopus is run by
the “citation potential” in a given discipline. A a separate administrative group to limit any con-
major factor is the number of citations included flict of interest.
in a given article, a number that varies by dis-
cipline. For instance, a citation from an article Web of Science
with 200 references will count for less than a Although the Thomson Reuters WOS database
citation from an article with only 20 references contains fewer articles than Scopus, it has articles
(29). Because SNIP takes this citation potential from 1900 onward (25). Like Scopus, WOS in-
into account, it can be used to compare journals cludes journal articles from all major disciplines,
from different disciplines, and even to compare although the total number of disciplines covered
different disciplines with one another. Like the is slightly less than that of Scopus. WOS does have
SJR, SNIP makes use of the Scopus database robust citation analysis, although a recent study
(discussed later). found 20% more articles in a citation analysis per-
formed with Scopus compared with WOS (30).
Databases WOS is perhaps most robust when evaluating
research conducted prior to 1996. Like Scopus,
PubMed WOS is not free to users.
Created by the National Library of Medicine
and launched in 1997 as a freely available in- Google Scholar
terface to the MEDLINE database, PubMed Google Scholar is the newest entry into the sci-
has become one of the most popular and widely entific database community. It is free to users,
used search engines for use with the medical offers robust search capacity, and may be the
literature. A unique advantage of PubMed is best way to access obscure information, such as
the incorporation of “published online ahead of articles published in journals that have yet to
RG  •  Volume 35  Number 3 Choudhri et al  743

Table 4: Comparison of Individuals with Different Publication Histories

Years in Publication Citation Current

Physician Practice Count Count Academic Rank* h-Index m-Quotient
A 2 4 2 Instructor 1 0.5
B 4 28 14 Assistant 3 0.75
C 5 16 742 Assistant 13 2.6
D 6 12 1 Assistant 1 0.17
E 12 8 48 Associate 5 0.42
F 20 15 155 Associate 8 0.4
Note.—Doctors A−F are six colleagues in an academic department with 2−20 years of experi-
ence and ranks that vary from instructor to associate professor. Although Dr B has published the
most articles, this output is rarely cited (average of 0.5 citations per article), and these articles
may be of low quality. However, Dr B is in the early stages of his/her career, and the h-index
has risen by approximately 0.75 per year (m-quotient). Dr C has written articles that have been
widely cited, with an h-index and m-quotient that likely indicate a promising academic career. If
other aspects of his/her clinical, educational, and academic productivity are appropriate, Dr C is
perhaps a reasonable candidate for promotion to associate professor in the first year of eligibil-
ity. Dr D has written articles that are almost never cited and does not have the academic output
typically expected of an associate professor.
*Assistant = assistant professor, Associate = associate professor.

be indexed in other databases. Google Scholar Conclusion

includes citations from books, online sources, Bibliometric parameters are increasingly being
and conference proceedings. The major limita- used to summarize the academic output of re-
tion of Google Scholar is that it is updated only searchers and journals alike. The limitations of
monthly (as opposed to weekly for Scopus and these tools have led to the development of new
WOS and daily for PubMed), and there is very metrics, which must be understood in terms of
little in the way of citation analysis or author their underlying theory and their theoretic and
identification. Although Google Scholar will practical strengths and weaknesses before be-
create a profile for a given author and provide ing adopted. Academic productivity, whether of
a list of articles the author may have written, an individual, department, or journal, cannot
the author must select the articles that should accurately be reduced to a single number on
be assigned to his or her profile. For a given a linear scale. Any imperfect metric is subject
author profile, Google Scholar will calculate the to manipulation, and an understanding of the
h-index and i-10 index for that author’s publish- strengths and weaknesses of different bibliomet-
ing career, as well as citations earned during the ric parameters is required to detect attempts at
past 5 years. manipulation. The possible role of bibliomet-
rics in comparing (a) individuals with different
Tools publication histories and (b) individuals with
similar publication totals but different biblio-
Publish or Perish metric parameters is illustrated in Tables 4 and
PoP is a software program that can be used to 5, respectively.
retrieve and analyze academic citations (31). PoP Currently, normative data exist primarily
can calculate an author’s h-index, g-index, and for the h-index, publication count, and citation
e-index, as well as many other bibliometric pa- count; further work will be required to determine
rameters. It is used by individual researchers to normative values for other parameters. In ad-
determine the impact of their research. PoP pri- dition, defining normative values according to
marily uses the Google Scholar database and is existing levels of academic rank assumes that in-
free of charge for personal nonprofit use. Authors dividuals deserve their rank based on publication
can search for their articles by author name, just history but without regard to other measures of
as they would with Google Scholar; however, they academic productivity and service.
must manually confirm which articles should be It is worth nothing that, for many researchers,
included in their calculation. Selecting articles to the citation count and h-index calculated by Sco-
be used in a PoP calculation can be time consum- pus will be lower than those calculated by Google
ing for authors with the same name as another Scholar (Fig 2) (32–34), primarily for two rea-
author and/or a large number of articles. sons: For researchers who started their careers
744  May-June 2015 radiographics.rsna.org

Table 5: Comparison of Individuals with Similar Publication

Totals but Different Bibliometric Parameters

No. of Citations and

Index Value by Physician

Publication or Index W X Y Z
Publication
  Paper 1 25 6 10 18
  Paper 2 1 6 8 7
  Paper 3 1 6 5 2
  Paper 4 1 6 3 1
  Paper 5 1 6 2 1
  Paper 6 1 0 2 1
  Total 30 30 30 30
Index
  h-Index 1 5 3 2
  e-Index 5.4 2.2 4.6 5.1
  g-Index 5 5 5 5
Note.—Table 5 shows the citation profile of four physicians, each
with six publications and a total of 30 citations. Dr X has the
highest possible h-index of 5; accordingly, the e-index is lower. Dr
W has the lowest possible h-index of 1 and therefore the highest
possible e-index of 5.4. Thus, it is not valid to compare e-indexes
between researchers who do not have comparable h-indexes.

before 1996, Scopus will not include their early

career (articles published before 1996) in cita-
tion analysis or h-index calculation. In addition,
Google Scholar includes more sources for cita-
tions, including books and some online sources.
Even within a database, Scopus will tend to
underestimate true bibliometric calculations for
authors whose publication history started before
1996 but will be fully inclusive for those whose
publication history began in 1996 or thereafter.
Thus, comparing parameters between individu-
als or to an established normative table requires
using similar databases and understanding the
relative strengths and weaknesses of each param-
eter. Future changes to these databases, such
as the attempt by Scopus to include all articles
published since 1970, may mitigate some of these
issues, although they may simultaneously alter
the accuracy of benchmarks created using prior Figure 2.  Graph shows the citation count (y-axis), number
of publications (x-axis), and identity line (x = y; angled black
versions of the database. line) as calculated by Scopus (blue) and Google Scholar (red)
Although the h-index seems to be a rather for a single author. Although the curved lines follow one an-
crude tool for measuring academic performance, other closely, Google Scholar is higher at all levels, with a
it has stood the test of time and has correlated well Google Scholar h-index of 17 and a Scopus h-index of 14
(shown by the vertical lines drawn to where the curved lines
with more mathematically elaborate techniques. cross the identity line).
In a study by Silagadze (35), the h-index showed
a strong correlation with a much more complex
S-index. The explanation for this correlation is less of the author or scientific community being
that citations follow Zipf’s law (36), meaning that evaluated. Of course, exceptions do exist, such
citations within the scientific community follow a as articles with an extremely high citation count.
logical pattern of mathematic progression regard- Overall, the h-index correlates strongly with more
RG  •  Volume 35  Number 3 Choudhri et al  745

elaborate measures because of the “zipfian” nature 7. Bergstrom CT, West JD, Wiseman MA. The Eigenfactor
metrics. J Neurosci 2008;28(45):11433–11434.
of academic citations. 8. Rad AE, Brinjikji W, Cloft HJ, Kallmes DF. The H-index
The same cannot be said of the various bib- in academic radiology. Acad Radiol 2010;17(7):817–821.
liometric parameters used for assessing journals. 9. Svider PF, Mauro KM, Sanghvi S, Setzen M, Baredes
S, Eloy JA. Is NIH funding predictive of greater research
The journal impact factor has been widely criti- productivity and impact among academic otolaryngologists?
cized as being imprecise and subject to manipula- Laryngoscope 2013;123(1):118–122.
tion, yet it continues to be the most widely used 10. Chew FS. The scientific literature in diagnostic radiology for
American readers: a survey and analysis of journals, papers,
metric. Other metrics, such as the Eigenfactor, and authors. AJR Am J Roentgenol 1986;147(5):1055–1061.
AIS, and SJR, offer more robust analysis and 11. Pagel PS, Hudetz JA. H-index is a sensitive indicator of
are starting to gain traction within the scientific academic activity in highly productive anaesthesiologists:
results of a bibliometric analysis. Acta Anaesthesiol Scand
community. SNIP allows comparison of different 2011;55(9):1085–1089.
journals across disciplines, although any exact 12. Poynard T, Thabut D, Jabre P, et al. Ranking hepatologists:
correlation will always be difficult. which Hirsch’s h-index to prevent the “e-crise de foi-e”? Clin
Res Hepatol Gastroenterol 2011;35(5):375–386.
In recent years, there have been multiple 13. Svider PF, Choudhry ZA, Choudhry OJ, Baredes S, Liu JK,
studies investigating the most frequently cited Eloy JA. The use of the h-index in academic otolaryngology.
radiology articles, either for a specific journal Laryngoscope 2013;123(1):103–106.
14. Quigley MR, Holliday EB, Fuller CD, Choi M, Thomas CR
(37,38) or for the field of radiology (39–41). Jr. Distribution of the h-index in radiation oncology conforms
Similar work has been performed for other med- to a variation of power law: implications for assessing academic
ical subspecialties (42–57). The use of different productivity. J Cancer Educ 2012;27(3):463–466.
15. Turaga KK, Gamblin TC. Measuring the surgical academic
methodologies (eg, different databases, years output of an institution: the “institutional” H-index. J Surg
of inclusion, or exclusion criteria) yields article Educ 2012;69(4):499–503.
lists that largely overlap but elucidate different 16. Benway BM, Kalidas P, Cabello JM, Bhayani SB. Does cita-
tion analysis reveal association between h-index and academic
trends. For instance, three articles that evaluated rank in urology? Urology 2009;74(1):30–33.
the 100 most frequently cited articles in the field 17. Lee J, Kraus KL, Couldwell WT. Use of the h index in
of radiology were published in 2013 and early neurosurgery. J Neurosurg 2009;111(2):387–392.
18. Khan NR, Thompson CJ, Taylor DR, et al. Should the h-index
2014 (39–41). Each of these articles used dif- be modified? An analysis of the m-quotient, contemporary h-
ferent databases and different journal and time index, authorship value, and impact factor. World Neurosurg
selection criteria and extracted different infor- 2013;80(6):766–774.
19. Khan N, Thompson CJ, Choudhri AF, Boop FA, Klimo P
mation from each article, resulting in different Jr. The application of the h-index to groups of individuals
lists of the “top 100” articles (39–41). Compari- and departments in academic neurosurgery. World Neurosurg
son of two of these lists demonstrates an overlap 2013;80(6):759, e3.
20. Ponce FA, Lozano AM. Academic impact and rankings of
of approximately 70%, indicating that each list American and Canadian neurosurgical departments as as-
identified 30 frequently cited articles unique to sessed using the h index. J Neurosurg 2010;113(3):447–457.
the corresponding search criteria (39,41). 21. Spearman CM, Quigley MJ, Quigley MR, Wilberger JE.
Survey of the h index for all of academic neurosurgery: an-
Bibliometric analysis adds a quantitative as- other power-law phenomenon? J Neurosurg 2010;113(5):
pect to an otherwise somewhat qualitative pro- 929–933.
cess. Moving beyond simple tallies of publication 22. Bergstrom CT. Eigenfactor: measuring the value and prestige
of scholarly journals. Coll Res Libr News 2007;68(5):314–
totals and impact factors, modern analytic tools 316. http://crln.acrl.org/content/68/5/314.full.pdf+html.
have emerged to improve on prior methods. Al- 23. West JD, Bergstrom TC, Bergstrom CT. The Eigenfactor
though there is no one ideal tool, an accurate metrics: a network approach to assessing scholarly journals.
Coll Res Libr 2010;71(3):236–244.
understanding of bibliometric parameters can aid 24. González-Pereira B, Guerrero-Bote VP, Moya-Anegón F. A
in effectively evaluating individual authors, de- new approach to the metric of journals’ scientific prestige:
partments, and institutions, as well as individual the SJR indicator. J Informetrics 2010;4(3):379–391.
25. ISI Web of Knowledge. Thomson Reuters Web site. http:
articles and journals. //www.isiwebofknowledge.com. Published 2013. Accessed
March 1, 2013.
26. Scopus. Elsevier Web site. http://www.scopus.com. Accessed
References November 20, 2013.
1. Hirsch JE. An index to quantify an individual’s scientific 27. Guerrero-Bote VP, Moya-Anegón F. A further step forward
research output. Proc Natl Acad Sci U S A 2005;102(46): in measuring journals’ scientific prestige: the SJR2 indicator.
16569–16572. J Informetrics 2012;6(4):674–688.
2. Egghe L. Theory and practice of the g-index. Scientometrics 28. Moed HF. The source normalized impact per paper is a valid
2006;69(1):131–152. and sophisticated indicator of journal citation impact. J Am
3. Hirsch JE. Does the h index have predictive power? Proc Soc Inf Sci Technol 2011;62(1):211–213.
Natl Acad Sci U S A 2007;104(49):19193–19198. 29. Waltman L, Jan van Eck N, van Leeuwen T, Visser MS.
4. Sidiropoulos A, Katsaros D, Manolopoulos Y. Generalized Some modifications to the SNIP journal impact indicator. J
Hirsch h-index for disclosing latent facts in citation networks. Informetrics 2013;7(2):272–285.
Scientometrics 2007;72(2):253–280. 30. Vieira ES, Gomes JA. A comparison of Scopus and
5. Zhang CT. The e-index, complementing the h-index for Web of Science for a typical university. Scientometrics
excess citations. PLoS ONE 2009;4(5):e5429. 2009;81(2):587–600.
6. Garfield E. Citation analysis as a tool in journal evaluation. 31. Harzing AW. Publish or Perish. http://www.harzing.com
Science 1972;178(4060):471–479. /pop.htm. Published 2007. Accessed June 10, 2014.
 746  May-June 2015 radiographics.rsna.org

32. Bar-Ilan J. Which h-index? A comparison of WoS, Scopus 45. Ohba N, Nakao K. The 101 most frequently cited articles
and Google Scholar. Scientometrics 2008;74(2):257–271. in ophthalmology journals from 1850 to 1949. Arch Oph-
33. Alonso S, Cabrerizo FJ, Herrera-Viedma E, Herrera F. thalmol 2010;128(12):1610–1617.
h-Index: a review focused in its variants, computation and 46. Lefaivre KA, Shadgan B, O’Brien PJ. 100 most cited
standardization for different scientific fields. J Informetrics articles in orthopaedic surgery. Clin Orthop Relat Res
2009;3(4):273–289. 2011;469(5):1487–1497.
34. Kulkarni AV, Aziz B, Shams I, Busse JW. Comparisons of 47. Namdari S, Baldwin K, Kovatch K, Huffman GR, Glaser
citations in Web of Science, Scopus, and Google Scholar D. Fifty most cited articles in orthopedic shoulder surgery.
for articles published in general medical journals. JAMA J Shoulder Elbow Surg 2012;21(12):1796–1802.
2009;302(10):1092–1096. 48. Baldwin KD, Kovatch K, Namdari S, Sankar W, Flynn
35. Silagadze ZK. Citation entropy and research impact estima- JM, Dormans JP. The 50 most cited articles in pediatric
tion. Acta Phys Polon B 2010;41:2325–2333. http://arxiv orthopedic surgery. J Pediatr Orthop B 2012;21(5):
.org/abs/0905.1039. 463–468.
36. Silagadze ZK. Citations and the Zipf-Mandelbrot’s law. 49. Fenton JE, Roy D, Hughes JP, Jones AS. A century of
Complex Syst 2010;11:487–499. http://arxiv.org/abs citation classics in otolaryngology–head and neck surgery
/physics/9901035. journals. J Laryngol Otol 2002;116(7):494–498.
37. Bui-Mansfield LT. Top 100 cited AJR articles at the AJR’s 50. Ponce FA, Lozano AM. Highly cited works in neurosurgery.
centennial. AJR Am J Roentgenol 2006;186(1):3–6. I. The 100 top-cited papers in neurosurgical journals. J
38. Bui-Mansfield LT. Whatever happened to the 50 most Neurosurg 2010;112(2):223–232.
frequently cited articles published in AJR? AJR Am J Roent- 51. Ponce FA, Lozano AM. Highly cited works in neurosur-
genol 2005;185(3):597–601. gery. II. The citation classics. J Neurosurg 2010;112(2):
39. Yoon DY, Yun EJ, Ku YJ, et al. Citation classics in radiol- 233–246.
ogy journals: the 100 top-cited articles, 1945−2012. AJR 52. Tripathi RS, Blum JM, Papadimos TJ, Rosenberg AL. A
Am J Roentgenol 2013;201(3):471–481. bibliometric search of citation classics in anesthesiology.
40. Brinjikji W, Klunder A, Kallmes DF. The 100 most-cited BMC Anesthesiol 2011;11:24.
articles in the imaging literature. Radiology 2013;269 53. Zhang WJ, Li YF, Zhang JL, Xu M, Yan RL, Jiang H.
(1):272–276. Classic citations in main plastic and reconstructive surgery
41. Pagni M, Khan NR, Cohen HL, Choudhri AF. Highly cited journals. Ann Plast Surg 2013;71(1):103–108.
works in radiology: the top 100 cited articles in radiologic 54. Loonen MP, Hage JJ, Kon M. Plastic surgery classics: charac-
journals. Acad Radiol 2014;21(8):1056–1066. teristics of 50 top-cited articles in four plastic surgery journals
42. Rosenberg AL, Tripathi RS, Blum J. The most influential since 1946. Plast Reconstr Surg 2008;121(5):320e–327e.
articles in critical care medicine. J Crit Care 2010;25 55. Nason GJ, Tareen F, Mortell A. The top 100 cited articles
(1):157–170. in urology: an update. Can Urol Assoc J 2013;7(1-2):
43. Gehanno JF, Takahashi K, Darmoni S, Weber J. Citation E16–E24.
classics in occupational medicine journals. Scand J Work 56. Heldwein FL, Rhoden EL, Morgentaler A. Classics of
Environ Health 2007;33(4):245–251. urology: a half century history of the most frequently cited
44. Ohba N, Nakao K, Isashiki Y, Ohba A. The 100 most articles (1955−2009). Urology 2010;75(6):1261–1268.
frequently cited articles in ophthalmology journals. Arch 57. Thomas K, Moore CM, Gerharz EW, O’Brien T, Emberton
Ophthalmol 2007;125(7):952–960. M. Classic papers in urology. Eur Urol 2003;43(6):591–595.

View publication stats