Scholarly behaviour and evaluation criteria: Uncovering the superficial characteristics that lead to higher citations
Do scholars adjust their publication behaviour depending on the criteria used in their evaluation? Maarten van Weselpresents findings showing how the publishing behaviour of scholars
changed when evaluation switched from emphasising ‘publish-or-perish’ to
impact factors. Whilst this may suggest a shift from quantity to
quality, the number of citations a paper receives not only depends on
its scholarly value, but also on seemingly superficial characteristics
of a paper.
Evaluation criteria for scholars and universities have recently come under critique in the Netherlands by groups such as Science in Transition, various Nieuwe Universiteit movements, and Platform Hervorming Nederlandse Universiteiten. Elsewhere, movements such as those behind the San Francisco Declaration on Research Assessment
have focused on the improper use of the Impact Factor. There are many
reasons for these critiques, but it is at least in part a reaction to
cases of scholarly misconduct, such as the Stapel affaire,
and ensuing debates about whether publication pressures are a cause of
such misconduct. But how do evaluation criteria influence publication
behaviour?
My present study
first looks at the literature about the influence of publish-or-perish
attitudes on publication behaviour. This literature is quite extensive
and focuses partly on dubious behaviour, such as Salami Slicing
and gift authorship, which are both linked to the pressure to publish
as much as possible. But the focus is shifting, to the number of
citations a paper receives and to the Impact Factor of the journals in
which they are published. This shift in focus suggests a move from
quantity to quality, as surely researchers and their managers would want
to emphasise the scholarly quality and value of a paper!
However, from the field of Scientometrics we learn that the number of
citations a paper receives not only depends on its scholarly value, but
also on seemingly superficial characteristics of a paper (e.g. van Wesel, Wyatt & Haaf 2014).
Factors such as total number of words, the number of authors, author
prominence, and the number of references all influence subsequent
citation. The authors who, deliberately or not, produce papers with the
correct mix of these characteristics become more influential than others
in the same field, and will secure more prestige and ultimately perhaps
more resources. Because of this virtuous circle, the publication style
may be imitated. This is not necessarily unethical, but when these
superficial factors are purposely tweaked in order to get more citations
it becomes trickier. Adding authors who contributed nothing to the
text, lengthening a paper without adding any extra, relevant
information, adding unnecessary references etc. can all be seen as
misconduct.
Image credit: Crossing Paths by Bala Sivakumar (CC BY)
In the sample used for this study, the rise in the number of authors has been especially spectacular for The Lancet and Nature articles (see table). Whilst the mean number of authors does not rise above ten, the extremes are spectacular: a Nature paper authored by 241 individuals, almost three times as high in The Lancet 697 individuals.These extremes represent the results of extensive collaborations, but
does this justify listing all of those individuals as authors? These
extremes, most likely, do not represent an attempt to beat the
publishing game, but might represent an attempt to acknowledge the
contributions which were too important to merely mention them in the
acknowledgements. This, however, greatly devalues what it means to be
an author. Such practices could be problematic if something is wrong
(e.g. plagiarism, data fabrication) with the paper. Are all ‘authors’
willing to accept the responsibility which comes with authorship? In
relation to this research, these extremes provide clear evidence for the
growing practice of listing non-authors as authors, and the acceptance
of this practice by leading journals.
This study also observed an increase in both title length and total
number of pages for almost all sets in the sample. The abstracts contain
more sentences and become harder to read. All of these characteristics
have been associated with a greater number of subsequent citations in
previous studies. Of course, all these changes could be due to
adjustments of editorial policies of the journals involved. It would,
however, be a huge coincidence if this happened in multiple journals
from different publishers across diverse fields for various
characteristics in the same time period.
Table 1: Author Count Across Journals 1960 to 1974 and 1990 to 2004.
Adapted from Wesel, 2015
There are still other characteristics of papers that are known to 
influence the number of citations a paper receives, which are untested
in this research, most notably the number of references and prominence
of the authors. It is likely that further scientometric research will
reveal further characteristics which influence the scholarly impact of
papers but which are not related to their actual content.
Evaluation criteria continue to shift, due to pressure from scholars
and from formal government-sponsored evaluations, such as the REF in the UK. The social impact of papers is on the rise, mapped among others by Altmetrics,
but also by scholars self-reporting of news appearances and
publications in non-scholarly outlets. Knowledge valorisation is also on
the rise as a criterion, relating to how the knowledge gained from
research be put in practice. And, maybe more important, how can the
results be made valuable in economic terms?
These changing evaluation criteria will lead to changes in the
behaviour of scholars, as scholars are highly reflexive and responsive
to their environments. For instance, the rising importance of Societal
Impact will have a positive effect on the willingness of scholars to
write blogposts about their research.
For more on this study and analysis, the full Science and
Engineering Ethics article can be accessed, online first, here: Wesel,
Maarten van (2015) Evaluation by Citation: Trends in Publication Behavior, Evaluation Criteria, and the Strive for High Impact Publications. Science and Engineering Ethics doi:10.1007/s11948-015-9638-0
Note: This article gives the views of the author, and not the
position of the Impact of Social Science blog, nor of the London School
of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
About the Author
Maarten van Wesel works at the University
Library and the Department of Family Medicine at Maastricht University.
He is an external PhD candidate in the field of Sociology of Science and
Technology at the Faculty of Arts and Social Sciences where he is
especially interested in the influence of Word Processing on Social
Scientists.
Impact of Social Sciences – Scholarly behaviour and evaluation criteria: Uncovering the superficial characteristics that lead to higher citations
No comments:
Post a Comment