Monday, 5 June 2017

Research metrics


Research metrics




Performance metrics
Research is assessed on a number of criteria already and with the Web
now providing the opportunity for the development of new tools and
techniques for measuring 'things to do with research' the list of
possible assessment criteria is growing. Of course, assessment of an
individual's performance for, say, tenure will take into account things
such as grants awarded, prizes and medals, patents, student mentoring,
teaching duties, offices held, and other measures of contribution
towards institutional life. Qualitative measures may include
collaborations, some form of peer review, r esponsibilities and nowadays
some of the so-called Web 2.0-enabled activities (social network

As well as these, there will be attention on bibliographic measures
that reflect that individual's publications record - the evidence of his
or her research output trail.

The Journal Impact factor (JIF)
Until recently, there was only really one such measure used
ubiquitously and that is the Journal Impact Factor. Just the name of
that measure serves to signal how absurd an application it is as a
metric to measure the performance of an individual: it measures the
impact of journals, not people, yet its significance in shaping
publishing practices, directions of research, funding and academic
careers cannot be overestimated. Its use has been widespread and it has
been employed in the most misguided of ways, even at national research
policy level.

The JIF is a metric developed by the Institute for Scientific
Information (ISI, now part of Thomson Reuters) calculated through an
algorithm based on the total number of citations that have been accrued
by articles in a journal over a two-year period after publication. The
total citations for that journal over the period of a given year are
divided by the number of articles published in that year and the result
is the JIF. This is published for around 9000 journals every year in the
Journal Citation Index, an index eagerly awaited by publishers and
journal editors and editorial boards because a culture of competing on
JIFs has grown up in this community. To have a good JIF is considered a
measure of success, despite all the flaws of the system and the
opportunities for manipulation that exist.

And to have a good JIF - and there is no such thing as a
straightforwardly good JIF, only a relatively good JIF since the measure
is a measure or relativity - may be a fine thing for a journal and its
publishers and editors. It is not a good thing for authors, since
authors cannot have a JIF: it is a journal-based metric. Yet authors are
measured, commonly and seriously, on the JIF of the journals in which
they publish their work. The absurdity of this can be likened to
awarding a candidate a university place on the basis of how many other
students from his high school have been awarded places. It ignores the
performance of the individual and rewards him or her on the basis of a
collective measure.

New bibliometrics
The JIF was a metric developed in the era when journal existed only
in print form and there was only one database large and comprehensive
enough to use for such a calculation - the one assembled by ISI. ISI
still produces the Journal Citation Index each year but other, new,
bibliometrics are also emerging now that there are substantial bodies of
literature held in collections elsewhere. The growing Open Access
literature provides huge opportunities in this respect. If all research
outputs are open to analysis, useful new measures can be developed
encompassing not only research papers, but datasets and other types of
output from research activity. This is an exciting area for future
development. But even with a focus just on research articles there are
many things that can be done to assess impact now that the Open Access
literature is growing.

There are two main bases for developing measures for the research literature:

  • usage-based metrics (data generated through measuring user activity)
  • citation-based metrics (data generated by measuring author activity: the JIF is one metric in this category)
Usage metrics
Some examples of usage metrics are:

  • COUNTER statistics: COUNTER is a publishing industry standard
    developed to ensure that the usage statistics provided by publishers to
    libraries for each journal to which the library subscribes are
    consistent and in a form where comparisons and analyses can be
    performed. COUNTER statistics measure at journal-level only at present,
    though article-level usage data may be available in future
  • The usage-based measures developed by the MESUR Project
    carried out by the LANL laboratory. MESUR has defined and validated a
    range of usage metrics and produced guidelines and recommendations for
    their application
  • Repository usage measures. These systems record usage of materials
    in Open Access repositories so that authors can see how much (and in
    some cases where) their articles are being downloaded. Examples include LogEc (which measures usage from the RePec economics Open Access collection, AWstats, and Google Analytics (which are general website-usage analysers) and Interoperable Repository Statistics
    (an open source program from the EPrints team at Southampton University
    specifically designed for measuring usage of Open Access repositories)
Citation metrics
Some examples of citation-analysis systems are:

  • h-index: developed by Hirsch, this is a simple measure that relates
    an individual's published outputs to the number of citations they gather
    and computes them into a single metric
  • g-index: developed by Egghe. Other modifications of the h-index also exist
  • eigenfactor, now used by Thomson Reuters as part of their author service offering
  • Y-factor (developed by the LANL laboratory)
  • Harzing's Publish-or-Perish service for measuring an individual's citation impact
  • CiteSeer, a program analysing citations to the Open Access computer science literature
  • CitEc, a program analysing citations to the Open Access economic literature
  • Citebase, a
    program that works on any body of Open Access literature but currently
    operating across the physics and cognitive science Open Access

The Open Access literature provides opportunities for the development
of a much richer array of bibliometrics, too. Things such as indices of
citation latency (how long citations continue to be made to articles),
immediacy (how soon citations occur), decay index (the pattern of
citations to an article), cited-by and co-citation measures and so forth
will be tools that enable bibliometricians to explore the literature in
new ways and gain greater understanding of how research is
communicated, especially when coupled with semantic analysis
technologies. This understanding will help to improve research
communication in the future.

Research indicators in development
A number of large-scale projects are underway to study the potential
for the development of new indicators in different areas of research.
Some examples include:

The Humanities Indicators Project run by the National Humanities Alliance in the US

The European Educational Research Quality Indicators Project, funded by the European Union

The European Reference Index for the Humanities, funded by the European Science Foundation

See also:

Citation impact

Research metrics

No comments:

Post a Comment