Sunday, 19 February 2017

Using citation analysis to measure research impact | Editage Insights


Using citation analysis to measure research impact

Measuring research impact

landscape of science and research is rapidly evolving. Gone are the
days when all members of a university department would celebrate the
successful publication of a colleague’s paper.1 Earlier,
scientists would simply consider the number of papers they had published
as a measure of their academic standing. Today, the focus is
increasingly shifting from whether a researcher has published a paper to
where he/she has published it and the impact that piece of research has
on the scientific community and the world at large.2 

can you measure the quality of a research paper? More importantly, how
can you determine whether your research is making an impact and is
considered important? An objective way is through citation analysis. 

Citation analysis

count citations in the first place? The list of references directing
readers to prior relevant research is considered a fundamental part of
any research paper.
3 A
reference or citation is a form of acknowledgment that one research
paper gives to another. Research is additive—scientists build on past
work to discover new knowledge. To identify gaps in existing research
and choose a research topic, researchers read the relevant published
research and use this existing material as a foundation for arguments
made in their own research papers.

11 reasons to cite previous work

  1. To direct readers to an authentic source of relevant information
  2. To help other researchers trace the genealogy of your ideas
  3. To acknowledge pioneers and peers
  4. To direct readers to previously used methods, and equipment
  5. To criticize or correct previous work
  6. To substantiate your claims and arguments with evidence
  7. To show that you have considered various opinions in framing your arguments
  8. To highlight the originality of your work in the context of previous work
  9. To guide other researchers in their work
  10. To build your credibility as an author
  11. Finally, because not citing sources can amount to plagiarism4
What are the various citation-based metrics?

Citation analyses can be grouped according to some broad types based on who/what is being evaluated.

  1. Ranking journals:
    Journals are ranked by counting the number of times their papers are
    cited in other journals. Journal-level metrics are generally meant to
    serve as an indicator of journal prestige. The most well known of these
    is the journal impact factor, from Journal Citation Reports
    product of Thomson Reuters). The journal impact factor is calculated as
    the average number of citations all articles in a journal receive over a
    specific period of time.
  2. Ranking researchers:
    Various citation metrics are now used for this purpose. Researchers are
    ranked by counting the number of times their individual papers are
    cited in other published studies. These metrics are also used to
    evaluate researchers for hiring, tenure, and grant decisions. A
    researcher-level metric that is gaining popularity is the h index,
    6 which
    is calculated by considering a combination of the number of papers
    published by a researcher and the number of citations these papers have
  3. Ranking articles:
    Article-level citation counts may provide an accurate evaluation of the
    quality and impact of a specific piece of work, regardless of the
    author. Unfortunately though, such metrics are rarely considered because
    obtaining these data is tedious and time-consuming.
  4. Ranking universities and countries:
    There are databases that rank universities and countries by considering
    their overall research output through criteria such as citable
    documents, citations per document, and total citations. These metrics
    help determine which universities and countries have the most and/or
    best scientific output. For example, Scimago Research Group ( ) releases annual reports of institution- and country-wise rankings.
How can citation analysis help you?

today are faced with increasing pressure to get published. Academic
departments are expected to meet specific levels of publication output.
Clearly, there is a lot at stake in the assessment of research quality
for both individuals and institutions. Given this, governments, funding
agencies, and tenure and promotion committees are looking toward simple
and objective methods to assess increasing research volumes in the least
possible time. To this end, they are turning more and more to citation
analysis for objective parameters of impact assessment. 

Pitfalls of citation analysis

When using citation analysis, it is important to bear in mind some of its limitations3,7

  • It
    overlooks the disparity in discipline-wise citation rates, that is, the
    fact that citation patterns differ among disciplines and over time.
  • It
    ignores the fact that certain manuscript types such as letters and case
    reports offer inadequate scope for citation and typically have short
    reference lists. 
    sentiment of the citation is not considered; that is a negative
    citation (one used to refute a prior claim) is given as much merit as a
    positive citation (one used to further the claim being made). So even a
    paper that has been cited simply to discredit it can work to the
    author’s advantage in citation analysis.
  • It
    does not account for author contribution on papers with multiple
    authors: such citations are as meritorious as those to single-author
    papers. Citation analysis attributes equal importance to all authors of a
    paper, regardless of their individual contribution.
sole reliance on citation data provides an incomplete understanding of
research. Although citation analysis may be simple to apply, it should
be used with caution to avoid it coming under disrepute through
uncritical use.
3 Ideally,
citation analysis should be performed to supplement, not replace, a
robust system of expert review to determine the actual quality and
impact of published research.

Future of citation analysis

the shift to online interfaces by more and more journals and
repositories, digital information is now available at a few clicks. With
the advent of linking tools and digital archives of research papers,
scientific literature is more easily retrievable than ever before.
Therefore, it is only to be expected that the population of researchers
turning to citation data will continue to grow. In such a scenario,
researchers cannot afford to undermine the importance of citation

next time you are preparing for a promotion or applying for a new
position, consider using citation analysis as a means to bolster your
eligibility. Use the citation count feature offered by online databases
like Web of Science to compile your citation data and employ multiple
citation metrics to highlight your research output.

  • Dodson MV (2008). Research paper citation record keeping: It is not for wimps. Journal of Animal Science, 86: 2795-2796.
  • Thomson Reuters. History of citation indexing. Essay in Free Scientific Resources. [] 
  • Smith L (1981). Citation analysis. Library Trends, 30: 83-106.
  • Garfield E. Citation indexing-Its Theory and Application in Science, Technology, and Humanities. New York: Wiley, 1979.
  • Garfield
    E (2006). The history and meaning of the journal impact factor. The
    Journal of the American Medical Association, 295: 90-93.
  • Hirsch
    JE (2005). An index to quantify an individual’s scientific research
    output. Proceedings of the National Academy of Sciences USA, 102:
  • Neylon C and Wu S (2009). Article-level metrics and the evolution of scientific impact. PLoS Biology, 7: 1-6.
  • Moed
    HF (2007) The future of research evaluation rests with an intelligent
    combination of advanced metrics and transparent peer review. Science and
    Public Policy, 34: 575-583.

Using citation analysis to measure research impact | Editage Insights

No comments:

Post a Comment