team-stacyStacy
Konkiel is a Research Metrics Consultant at Altmetric, a data science
company that helps researchers discover the attention their work
receives online. Since 2008, she has worked at the intersection of Open
Science, research impact metrics, and academic library services with
teams at Impactstory, Indiana University & PLOS.


When all you have is a hammer, everything looks like a nail.
When all you have is a hammer, everything looks like a nail.
Many institutions in the US and worldwide are changing the way that
they assess academics for tenure. They’re making a move towards using a
more holistic set of impact metrics (a “baskets of metrics”,
if you will) and, in some cases, doing away with metrics like the
journal impact factor (IF) as a sole means of evaluating their faculty’s
work. In so doing, they are righting wrongs that have plagued academia
for too long, particularly the misuse of citation-based metrics for
evaluation.


In this post, I’ll discuss which metrics should be abandoned, which
we should put into context, and what data we can add to tenure dossiers
to make it easier to interpret the “real world” impacts of scientific
research.


Citation-based metrics are not inherently faulty…


Using the journal IF as a means of evaluation, for example, can help
librarians understand the importance of academic journals within the
scholarly community (the purpose for which it was originally invented). And citations can help authors understand the scholarly attention that their journal article or book has received over time.


However, citation-based metrics offer an incomplete picture of the
total impact of research. Citations can only shed light on the attention
that articles and books alone receive, and only upon the attention of
other scholars. These limitations can be problematic in an era when an
increasing number of scholars are moving beyond the journal article to
share their work (often in the form of data, software, and other
scholarly outputs) and many universities and funders are asking scholars
to consider the broader impacts of their research (on public policy,
technology commercialization, and so on).


Moreover, some citation-based metrics are being used incorrectly. The
journal IF, in particular, is a victim of this: it is often used to
measure the quality of a journal article, when it (a) is a journal-level measure and therefore inappropriate for measuring article-level impact, and (b) cannot measure quality per se, but instead can help evaluators understand the far more nebulous concept of “scholarly impact”.


Luckily, a growing number of universities are changing the way
research is evaluated for tenure and promotion. They are reducing their
dependence on journal-level measures to understand article-level
impacts, and taking into consideration the effects that research has had
on the public, policymakers, and more.


These institutions are using metrics much more holistically, and I
think they serve as an excellent roadmap for changes that all
universities should make.


The current state of “impact” and tenure in the United States


At many universities in the US, researchers who are being assessed
for tenure are required to prepare a dossier that captures their value
as a researcher, instructor, and as a member of the larger disciplinary
community. And at some universities with medical schools, a researcher’s
clinical practice may also be considered.


Dossiers are designed to be able to easily communicate impact to both
a jury of disciplinary colleagues (other department members and
external colleagues within the same field), as well as university
committees of faculty and administrators, some of whom may know little
about the candidate’s specific research area.


There are many ways to showcase research impact and universities
typically offer dossier preparation guidelines to help faculty navigate
their options. And more often than not,
these guidelines primarily recommend listing raw citation counts and
journal IFs as a means of helping others understand the significance of
their work.


But there are better and more inclusive recommendations for the use
of research impact metrics in a dossier. Here are three in particular
that I think form a good starting point for anyone interested in
reviewing their own university’s tenure and promotion preparation
guidelines with an eye towards improvement.


  • Improvement 1: Remove journals from the equation
The journal that an article is published in cannot predict the quality of a single article. Moreover, journal titles may unfairly prejudice reviewers. (“I don’t recognize this journal’s name, so it must be rubbish.”)


So, why don’t more institutions disallow any mention of journal titles in tenure dossiers? Or make it clear that journal IFs should not be (miss)used as evidence of quality?


At the very least, instructions should require reviewers to read and
evaluate the quality of an article on its own merits, rather than rely
upon shortcuts like journal IFs or journal title prestige. And
contextualized, article-level citation counts (which I’ll talk about
more below) should only be used to inform such reviews, not replace
them.


  • Improvement 2: Require context for all metrics used in documentation
Tenure dossier preparation guidelines should make it clear that raw citation counts aren’t of much use for evaluation unless they’re put into context.
Such context should include how the article’s citations compare to
other articles published in the same discipline and year, and possibly
even the same journal. Here’s an example of how such context might work,
taken from an Impactstory profile:


impactstory1


Context can also include who has cited a paper, and in what manner
(to acknowledge prior work, commend a study that’s advanced a field, and
so on).


Some researchers include contextualized citation counts in their dossier’s narrative section, for example:


In 2012, I published my landmark study on causes for
Acute Respiratory Distress Syndrome. The article has since been cited
392 times. (In contrast, an average epidemiology journal article
published in 2012 has been cited only 68 times, according to
Impactstory.org.)
How researchers might best document contextualized impact may differ
from discipline to discipline and institution to institution, but
context is vital if academic contributions are to be evaluated fairly.
So, those creating tenure and promotion preparation guidelines should
take care to include instructions for how to provide context for any and
all metrics included in a dossier.


  • Improvement 3: Expand your university’s definition of impact
Citations are good for understanding the scholarly attention a
journal article or book has received, but they can’t help others
understand the impacts research has had on public policy, if it has
introduced a new technique that’s advanced research in a field, and so
on. However, altmetrics can.


Altmetrics are
loosely defined as data sourced from the social web that can help you
understand how often research has been discussed, read, shared, saved,
and reused. Some examples of altmetrics include:


  • Downloads and views on PubMed Central,
  • Mentions in policy documents,
  • Bookmarks on Mendeley, and
  • Software forks (adaptations) on GitHub.
Altmetrics are an excellent source of supplemental impact data to
include alongside citation counts. They can help evaluators understand
the effects that research has had on members of the public and other
non-scholarly audiences (in addition to the non-traditional effects that
research has had among other scholars, like whether they’re reading,
bookmarking, and adapting others’ scholarship). They can also help
evaluators understand the influence of scholarly outputs other than
books and journal articles (for example, if a piece of bioinformatics
data analysis software is being used by other researchers to run their
own analyses). Altmetrics also often include qualitative data, so you
can to discover who’s saying what about an article (providing some of
the all-important context that I touched upon above).


Altmetric, the company I work for, offers a bookmarklet
that makes it easy to find altmetrics from a variety of sources,
including an important non-scholarly source: public policy documents.
The report generated includes not only a raw count of the mentions an
article has received in public policy documents, but also links out to
the documents that mention it.


Here’s an example of an Altmetric report for an article that’s been mentioned in several policy documents:


AM_skonkiel


Using the report data, researchers can more easily document the effect their research has had on public health policy, like so:


My 2012 landmark article on causes for Acute Respiratory
Distress Syndrome has been cited in at least three public policy
documents, including the World Health Organization’s report on
recommendations for reducing mortality rates for children living in
poverty.
Mentions in policy documents are just one of the ways scientists can showcase the many flavors of impact their work has had. Others include: has their article been recommended by experts on Faculty of 1000? Has their research software been widely adopted by others in their discipline? Are their peers discussing their recent articles on their research blogs? The possibilities for understanding the broader dissemination and applications of research are many.


An increasing number of universities and departments (including the University of Colorado Denver Medical School and IUPUI)
are beginning to offer guidance on using (and interpreting) altmetrics
data in tenure dossiers. Any instructions for using altmetrics should be
careful to recommend including context (both “who’s saying what” about
research, as well as how much attention a work has received in
comparison to other research in a discipline) and steer candidates away
from using numerical indicators like raw counts or the Altmetric score
(which we created as a means of helping people identify where there is
altmetrics data to explore, not to rate the quality of articles).


And of course researchers themselves need not wait for guidelines to
include supplementary indicators like altmetrics in their tenure
dossiers, to help paint a more complete picture of the impact their work
has had. Researcher-oriented tools like Impactstory and the Altmetric
bookmarklet, described above, can help them discover where their work is
making a difference, and provide the contextualized data to help them
share those impacts with others.


It’s time for the tenure and promotion process to get smarter and
more nuanced. Let’s start by using relevant data to understand real
world impacts, and manage how we use traditional data like citation
counts to put scholarly impact into a better context.


Do you have ideas for how we could improve the use of metrics in
tenure and promotion? Leave them in the comments below or share them
with me (@skonkiel) via Twitter.