Introduction
Researchers
in Europe are occupied with a wide range of tasks outside of research,
such as teaching, managing projects, curating data, and writing funding
applications to name a few, which limits their time to perform research
and write publications. Ideally, when it comes to scholarly
communication, one’s focus should be on the dissemination of knowledge.
One should aim to publish only the most definitive work, of highest
quality, in the most reputable journals in the field. Yet, when
researchers are judged by the absolute value of their publication
metrics, conflicting incentives come into play [1]. Research decisions may be influenced by the effect on a researcher’s publication metrics, rather than by scientific merit [2].
A researcher could be working in a trending field due to personal
interest, ease of funding, or because they believe their publications
are more likely to get into high impact journals [3].
Similarly, a researcher could be writing guidelines and reviews, citing
their own work several times in each publication because the science
merits it, or simply to increase their number of citations [4–7].
One could share new publications on social media to reach new readers,
or simply to boost their Altmetric scores. One could seek out
international collaborators to garner fresh insights and generate new
questions, or simply to add their name to a growing list of co-authors
to improve access to “better” journals and a broader network.
Researchers
are aware that their publication metrics are being used in tenure track
assessments, grant applications and job interviews and target them
accordingly [8–14].
Any metric that can be measured, can be gamed and naturally, there are
many methods in use whose principle aim is to boost the h-index, h, [15,16] number of publications, NP, [17,18] and number of citations, NC [19,20]. All the while every year there are new proposals for how a researcher’s impact should be measured [21–27].
We
asked successful researchers how they have planned their publications
and deal with these conflicting incentives in academic publishing, when
the measurement methods are constantly changing. Researchers were asked
about their publication strategies, a recognized term in Scientometrics,
which Joubert & Rogers (2015) [28] describe as: “a road map delineating what, when, and how research will be published. It should include the type of article and journal, based on the target audience”.
In addition, if they do focus on particular metrics, they were asked to
report them. We define a successful researcher as someone who is an
established researcher in their field with a permanent position at their
institute, and an effective publication strategy as one that allowed
them to achieve such a position. We focus on the h-index, h, as it is widely used to rank academics [29–32] and group researchers by the degree of efficiency, a, where a = NC/h2 as taken from Hirsch 2005 [29],
and support a qualitative investigation of the publishing practices of a
group of clinical researchers, based on semi-structured interviews,
with a quantitative investigation of their publication metrics. Our main
aim is to examine the behavior of high a and low a
researchers in relation to what role the importance of a high h-index,
number of publications, number of citations, international or local
collaborations, field specific journal publishing or high journal impact
factor (JIF) journal publishing plays in their career. Researchers in
clinical science are in a special position, as their discipline offers
the potential to publish in journals of the highest JIF of any field. We
therefore assume that if optimizing behavior occurs, we will find it in
this field.
In the following sections
we explain how we sampled researchers to interview and present the
analysis of their interview responses. For validation purposes, we then
compare the researcher’s stated publication strategies with their
publication metrics from the last 6 years and discuss the implications.
Methods
Selection of interviewees
We
invited 18 researchers to take part in a recorded interview regarding
their h-index and publication metrics based on the degree of efficiency,
a, where a = NC/h2, of their publications in Scopus. The a
was chosen over the h-index, as the h-index is a cumulative metric,
where higher values likely indicate older researchers, while the a could provide more immediate insight into their publication strategies. At first glance, the degree of efficiency, a, would appear to have no correlation with the h-index (r = 0.07, S1 File). Yet, the a is indicating a flaw in the h-index, where highly cited articles cease to impact the metric. Researchers with a high value for a
could have only a few extremely well-cited publications or have well
cited publications overall. While researchers with a low value for a
could be in a field where citations are not so freely given, they could
have well cited citations overall, or they could simply be poorly
cited. To fully understand the data, input from the actual researchers
is needed. At 71%, the bulk of the researchers have a degree of
efficiency, a, clustered between 3 < a ≤ 5, with an average h of 33±16. This is similar to the spread reported by Hirsch 2005 [22]
for physicists. To uncover differences in the researcher’s publication
strategies, we selected researchers to interview from the outlying
groups with a degree of efficiency, a, below 3 (n = 10) and
above 5 (n = 8). All invited researchers are senior faculty, have PhD
students under their supervision and have an extensive network.
In total, we conducted 9 interviews (5 with a ≤ 3 and 4 with a
> 5). The remaining 9 invitees did not respond or declined our
invitation. The interviews were semi-structured, allowing room for
openness and broader conversation around the main interview framework
which can be found in the S1 File.
We preferred this method over an online survey or a questionnaire in
order to elicit more spontaneous, and perhaps more truthful, responses
as the interviewee is allowed to talk freely. Each interview lasted
approximately 10 minutes and was conducted in the office of the
interviewees. Interviewees gave verbal consent for being recorded and
participated freely and voluntarily in our study. To minimize the time
investment for the interviewees, verbal consent was recorded at the
start of each interview in lieu of written consent.
Interview analysis and publication metrics
Transcriptions
of the audio recordings were separately scanned for meaning by three of
the authors. To perform a statistical analysis and compare stated
publication strategies to actual publication outcomes the responses of
the researchers were independently distilled into simplified yes or no
answers by three of the authors. Elaborations on their answers and
quotes are included in the discussion. Responses from the interviewees
were analyzed using multiple correspondence analysis (MCA), commonly
used to analyze categorical survey data [33]. We used the FactoMineR [34] and factoextra [35]
packages in the R Statistical Computing software to run the analysis.
The responses were compared to their actual behavior by tracking their
publication metrics from the previous 6 years (2013–2018) in SciVal. In
addition, we generated a list of all the Scopus sources (journals
indexed in Scopus) that the authors published in and compiled two lists
for the two types of interviewees. We then counted how many times each
group published in the top 30 JIF journals, ranked by Journal Citation
Reports (JCR), for medicine on Feb 14th 2019 from 2013 to 2018. We complied with the terms of services for collecting data from Scopus, SciVal and JCR.
Results and discussion
Interview analysis
The interviewed researchers were asked about what they took into consideration when planning their publications (see the S1 File
for the interview guide). What did they consider before submitting an
article to a journal or conference, for instance did they look up the
journal impact factor (JIF)? Did they generally work with international
collaborators or local ones? To get a sense of their awareness of their
publication metrics, they were asked about the importance of their
number of publications, number of citations, and h-index to their work.
Their responses are summarized in ,
where each researcher’s responses are simplified to indicate whether
they find each publication metric (or variable) important to their work
(yes) or not (no).
Table 1
Responses of interviewed researchers to what publication metrics they find important to their work.The degree of efficiency, a,
separates the researchers into “low a” and “high a”. The remaining 7
variables categorize the researchers into those who place importance to a
variable (yes) and those who do not (no). The 7 variables are: NP (Number of publications) importance, NC
(Number of citations) importance, h-index importance, high JIF
importance—importance of publishing in high impact factor journals,
field specific importance–importance of publishing in field specific
journals, Local collaboration importance, and International
collaboration importance.
| Researcher | a | NP importance | NC importance | h-index importance | high JIF importance | field specific importance | local collaboration importance | international collaboration importance |
|---|
| 1 | Low | Yes | Yes | Yes | No | Yes | Yes | No |
| 2 | Low | Yes | No | Yes | No | Yes | Yes | Yes |
| 3 | Low | Yes | No | Yes | No | Yes | Yes | Yes |
| 4 | Low | Yes | Yes | Yes | No | Yes | Yes | Yes |
| 5 | Low | Yes | No | No | No | Yes | Yes | Yes |
| 6 | High | No | Yes | Yes | Yes | No | Yes | Yes |
| 7 | High | Yes | Yes | No | Yes | No | Yes | Yes |
| 8 | High | No | Yes | No | Yes | No | No | Yes |
| 9 | High | No | Yes | No | No | Yes | No | Yes |
We
used multiple correspondence analysis (MCA) to identify potential
associations between the importance of each publication metric and to
identify groups of individuals with similar profiles in their answers. is used as input for the MCA. The correlation of each of the 7 variables from to the two dimensions of the MCA is shown in ,
where the two axes represent the best linear combination of the
variables so that the variance along the new dimension is maximum. The
two dimensions are sufficient to retain 71% of the total inertia
(variation) contained in the data. Not all the variables are equally
well displayed in the two dimensions. The variables a, NP,
high JIF, and FieldSpecific are more correlated with Dimension 1, while
the variables h-index, international collaboration and local
collaboration are more correlated with Dimension 2. It is clearly shown
that the variable a is contributing heavily on Dimension 1 and is highly correlated with it.
MCA of publication metric variables.Correlation of each variable from
to the best linear combination of the variables so that the variance
along the new dimension is maximum, given here by Dimension 1 (Dim1) and
Dimension 2 (Dim2). The square of the correlations between the
variables and the dimensions are used as the coordinates of each
variable.
To identify global patterns
within the data, the correlation of each researcher (represented by the
blue circles) and each variable (represented by the red triangles) to
the two dimensions is shown in the MCA biplot ().
The distance between each researcher and variable shows how similar or
different they are to one another, where similar points are closer
together and dissimilar points are farther apart. Two broad clusters,
grouping researchers with either a High or Low a, can be distinguished based on their positive or negative correlation with Dimension 1. Overall, researchers with High a
(numbered 6–9) place importance in high JIF journals, the number of
citations, and international collaborations while they do not place
importance in field specific journals, the h-index, local collaborations
or the number of publications. On the other hand, researchers with Low a
(numbered 1–5) place importance in field specific journals, the
h-index, international and local collaborations, and the number of
publications, while they do not place importance in high JIF or the
number of citations. From the centrality of the variable in the factor
map it can be inferred that international collaborations are important
to all researchers.
MCA positioning of individual researchers around the publication metric variables.(a)
MCA biplot positioning the researchers and variables with respect to
the two dimensions. The blue circles indicate the researchers. The red
triangles indicate the variables, where the 7 original variables are
split into the position for a “yes” or “no” answer by the sub-indices
“_y” and “_n” respectively. (b) MCA factor map grouping the interviewed
researchers per publication metric variable, shown in a different color
for each variable. The ellipses indicate potential groupings of the
researchers by their response of whether each publication metric was of
importance to them or not.
A more detailed view of this clustering of researchers is shown in .
Here we see that for each publication metric but international
collaborations, there is a clear divide between researchers with a High a or Low a
based on their positive or negative correlation with Dimension 1. The
greater axis of each ellipse skews towards Dimension 1, indicating that
it is the most influential dimension.
Actual publication metrics
To
examine whether the publication metrics the interviewed researchers
indicated as important to their publication strategies are influenced at
all by their actual actions and practices, a quantitative analysis of
their publications from 2013 to 2018 was performed. To indicate focus on
NP, NC,
h-index, high JIF, field specific journals, local collaborations and
international ones, we examined the publication metrics given in .
Table 2
Actual
publication metrics used to examine preference for each of the initial
publication metric variables in the researcher’s publication strategies.
| Initial variable | Symbol | Actual publication metric |
|---|
| NP importance | NP6 | Total number of publications published from 2013–2018 |
| NC importance | NC6 | Field-Weighted Citation Impact of NP6 |
| h-index importance | h5-index | the h-index only taking into account the previous 5 years |
| high JIF importance | JIF6 | Percentage of NP6 in the top 1% of journals by their Scimago Journal Ranking or SJR |
| field specific importance | Field6 | Percentage of NP6 in the bottom 90% of journals by their SJR |
| local collaboration importance | Local6 | Percentage of NP6 with national collaborators, and without international collaborators |
| international collaboration importance | International6 | Percentage of NP6 with international collaborators |
In
the value of these actual publication metrics is listed, for each
researcher. The variables are used to broadly support each researcher’s
stated focus on NP, NC,
h-index, high JIF journals, field specific journals, local
collaboration and international collaboration, respectively. Within this
time period, researchers with High a have an overall higher average NC6, h5-index, and international collaboration rate, while researchers with Low a have published a larger percentage of their publications in lower ranked journals (Field6). The average number of publications and percentage of local collaborations is almost the same for High and Low a researchers, while the percentage of publications in the top 1% of journals (JIF6) is highly researcher dependent.
Table 3
Actual
publication metrics of interviewed researchers based on publications
from 2013–3018 used as supplementary variables for the MCA.The degree of efficiency, a, separates the researchers into “Low a” and “High a”. The remaining 7 variables, as explained in , can indicate each researcher’s focus on NP, NC, h-index, high JIF journals, field specific journals, local collaboration and international collaboration, respectively.
| Researcher | a | NP6
[–] | NC6
[–] | h5-index [–] | JIF6
[%] | Field6
[%] | Local6
[%] | International6
[%] |
|---|
| 1 | Low | 28 | 1.5 | 9 | 0 | 79 | 18 | 18 |
| 2 | Low | 96 | 1.5 | 15 | 3 | 50 | 35 | 36 |
| 3 | Low | 17 | 2.1 | 6 | 0 | 47 | 24 | 35 |
| 4 | Low | 39 | 0.8 | 8 | 0 | 56 | 33 | 13 |
| 5 | Low | 47 | 1.3 | 6 | 4 | 57 | 21 | 17 |
| 6 | High | 32 | 9.5 | 13 | 44 | 25 | 31 | 50 |
| 7 | High | 62 | 1.7 | 11 | 0 | 39 | 6 | 58 |
| 8 | High | 39 | 7.1 | 13 | 0 | 21 | 15 | 54 |
| 9 | High | 63 | 2.0 | 11 | 8 | 54 | 33 | 59 |
For this supplementary set of quantitative variables from , we proceed with a similar MCA. We assume that the higher the value of the supplementary quantitative variable from , the closer this will correspond to the variable being of importance to the researcher from .
For instance, if a researcher indicates an importance for the number of
publications during the interview, that will correspond to an actual
high number of publications over the last 6 years.
The results of the MCA based on both the researcher’s responses, and their publication metrics are shown in .
Overall, the actual publication metrics of the researchers (indicated
by the blue triangles and “_val” sub-indices) are spread in the same
global pattern as their responses (indicated by the red triangles). The
actual publication metrics are less correlated with Dimension 2 than the
researcher responses, as they appear lower in . In Dimension 1 we can see that the position of the NC,
h-index, and perhaps the field specific journal variables is almost
aligned with their respective supplementary variables, and in Dimension 2
the same applies to the high JIF variable.
Multiple Correspondence Analysis (MCA) of researcher responses compared to their actual publication metrics.(a)
Correlation of the reviewer responses, indicated by the red triangles,
and actual publication metrics from the last 6 years, indicated by the
blue triangles and “_val” sub-indices, to the two dimensions. (b)
Correlation circle representation of the actual publication metrics in
the two dimensions, where each variable is represented by its
correlation coefficience with the dimension.
As
the actual publication metrics are quantitative, to compare the
positioning of these supplementary variables to the researcher responses
we show the correlation circle representation in
instead of a biplot. The variables for local collaborations and field
specific journals are negatively correlated with Dimension 1, positioned
in the Low a plane. High JIF journals, international collaborations, Nc and the h-index are positively correlated with Dimension 1, positioned in the High a plane, and only the number of publications is not correlated with Dimension 1.
Discussion of researcher preference, publication metrics and interview analysis
Our
results indicate that there are two publication strategies in use by
researchers, which can be distinguished based on their degree of
efficiency, a. Researchers with a Low a prefer to
publish in field specific journals, often resulting in a journal with a
lower journal impact factor (JIF) and SCImago journal rank (SJR). In
their responses they indicated they prioritize the number of their
publications, NP, focusing on the topic and readership. Yet, this focus did not translate to a higher NP6 () in the 6 years we examined. They “want the message out where it is understood” as one of our interviewees formulated it, and often appear in the public arena, such as national media.
Researchers with a High a
indicated they prefer to publish in journals with the highest JIF, and
this is largely corroborated by their lower overall publication rate in
the bottom 90% of journals ranked by SJR. They prioritize their number
of citations, NC, and did indeed receive a
higher rate of citations than would be expected for the average
publication in their field, as signified by the NC6.
Even though they claim not to focus on their number of publications,
there is no measurable difference between the publication rate of the
two groups. Indeed, from the MCA it is clear NP is not dependent on Dimension 1, and therefore probably not dependent on a.
Furthermore, they do indeed focus on establishing international
collaborations, publishing 55% of their articles with international
collaborators as opposed to the 24% of the researchers with Low a. As one of the researchers put it, they believe that “collaborating is how you learn to do good research” and select international collaborators based on their potential.
Interestingly, researchers with High a had a higher h5-index () than those with Low a
even though they indicated that the h-index was of no importance to
their publication strategies. This inversion between publication metric
preference and actual publication outcome is supported by the results
from the MCA, where the h-index is negatively correlated with Dimension 1
in and positively correlated in . In their interviews, researchers with Low a were specifically concerned with the influence the h-index has on garnering funding from national and international sources.
Both
groups indicated the importance of performing research that is relevant
for clinicians, focusing on high quality randomized controlled trials
and methodological papers to generate interest. The impact metrics and “number of publications don’t count if they are not transformed into clinical practice”,
one said. The belief that publishing clinical guidelines,
methodological work and reviews has a positive effect on individual
metrics was also prevalent in both groups.
The near equivalence of the publication rate in the two groups is surprising, yet as the Low a
researchers tend to have a lower total number of co-authors on their
papers, it is likely they take on a larger bulk of the work in bringing
each article from concept to publication.
For further clarity, in
we summarize the outcome of the comparison between the publication
strategies and the actual publication outcomes of High and Low a researchers.
Table 4
Confirmation of publication variable preference in the examined high a and low a research groups.
| Researcher group | Variable | Preference confirmed by actual publication metrics |
|---|
| Low a | field specific journal importance | Yes |
| local collaboration importance | Yes |
| h-index, h, importance | No–preference inverted |
| number of publications, NP, importance | No–preference inconclusive |
| High a | number of citations, NC, importance | Yes |
| high JIF importance | Yes |
| international collaboration importance | Yes |
Due to the open form of our interview process, when asked about their publication strategies several Low a
researchers elected to speak about news and social media dissemination
of their work. Research impact can also be measured by changes in the
real world, by affecting policies, services, and health guidelines …
Preferring to disseminate knowledge via local media is then a logical,
albeit less quantifiable, path. The mindset of wanting to publish
research and disseminate it rapidly within the research community pairs
well with the turnaround time of lower impact journals, as a general
practice, and may be a secondary factor in choosing the appropriate
place to publish research. Additionally, research in specialist fields
may not have broad enough appeal to fit within the scope of high JIF
journals.
All things being equal, strategically
choosing a co-author can be a boon. If there are two potential partners
to work with, researchers with High a tend to choose the one
which will help get their work into a journal with a higher JIF. In one
example, by including a co-author who is a journal editor or
well-renowned researcher experienced at writing for high JIF. Preferring
to add a researcher from another institute or country as a co-author
can generate broader appeal and potentially more citations. Researchers
with a history in publishing in high JIF are probably also engaged in
interesting and unique projects. Our work suggests that researchers with
High a tend to wisely select their collaborators and research
projects, and this eventually pays off leading to a higher citation
count overall.
The degree of efficiency, a, is
by no means a target for researchers or analysts to judge the impact of
a researchers’ output. It could be used as a tool to help researcher
support services better identify which researcher can use help in what
area.
We note that a limitation of
this work is that we have only interviewed 9 individuals, within a
single research unit of clinical research. We are also aware that the
MCA method becomes more valid with a larger pool of subjects compared to
the investigated variables. As all the participants in our study are
senior faculty, it is impossible to separate the impact their
established position may have on their publication strategies. The
choices described above may not be open to every researcher, let alone
junior researchers. Additionally, the motivation for the identified
publication strategies remains unclear. Whether these researchers choose
certain publication strategies due to personal preference, trendiness
of their topic, funding opportunities, departmental support or prestige
we can not say. Further research needs to be done to uncover the
motivations for the identified publication strategies. We are therefore
fully aware that our results may not be generalizable to the research
community as a whole, yet we have shown indications of different
publication strategies employed by researchers who have achieved at
least some degree of academic success.
