Tuesday, 27 September 2016
Monday, 26 September 2016
Visualizing Citation Cartels | The Scholarly Kitchen
Source: https://scholarlykitchen.sspnet.org/2016/09/26/visualizing-citation-cartels/
By their very nature, citation cartels are difficult to detect.
Unlike self-citation, which can be spotted when there are high levels of
references to other papers published in the same journal, cartels work
by influencing incoming citations from other journals.
In 2012, I reported on the first case of a citation cartel involving four biomedical journals. Later that year, Thomson Reuters suspended three of the four titles from receiving an Impact Factor. In 2014, they suspended six business journals for similar behavior.
This year, Thomson Reuters suspended Applied Clinical Informatics (ACI) for its role in distorting the citation performance of Methods of Information in Medicine (MIM). Both journals are published by Schattauer Publishers in Germany. According to the notice, 39% of 2015 citations to MIM came from ACI.
More importantly, 86% of these citations were directed to the previous
two years of publication — the years that count toward the journal’s
Impact Factor.
Thomson Reuters purposefully avoids using the term “citation cartel,”
which implies a willful attempt to game the system, and uses the more
ambiguous term “citation stacking” to describe the pattern itself.
Ultimately, we never know the intent of the authors who created the
citation pattern in the first place, only that it can distort the
ranking of a journal within its field. This is what Thomson Reuters
wants to avoid.
Schattauer Publishers appealed the suspension,
offering to exclude the offending papers from their Impact Factor
calculation as a concession. Their appeal was denied. Offering some
consolation to its readers, the publisher made all 2015 ACI papers freely available. It has also offered all ACI authors one free open access publication in 2016.
To better understand the citation pattern that resulted in ACI being suspended, I created, using VOS Viewer, a visualization of the citation network of papers published in ACI (blue) and MIM (red) from 2013 through 2015. Each paper lists its first author, year of publication and links to the papers it cites.
From the graph, there appear to be four papers that strongly influence the flow of citations in this network, two MIM papers by Lehmann (red) and two ACI
papers by Haux (blue). Each of these papers cites a large number of
papers published in the other journal within the previous two
years. Does this alone imply an intent to distort one’s Impact Factors?
We need more information.
Both Lehmann and Haux are on the editorial boards of both journals. Lehman is the Editor-in-Chief of ACI and also sits on the editorial board of MIM. Haux is the Senior Consulting Editor of MIM and also sits on the ACI editorial board.
This illustrates that there is a close relationship among the two
editors, but still this is not enough to imply intent. We need to look
at the four offending papers:
editors is that they are analyzing papers published in their own
journals within the time frame that affects the calculation of their
Impact Factors. Again, this alone does not imply an intent to game their
Impact Factor. Indeed, the publisher explained that
citation stacking was “an unintentional consequence of efforts to
analyze the effects of bridging between theory and practice.”
I can’t dispute what the editors and publisher state was their
intent. However, what is uniformly odd about these papers is that they
cite their dataset as if each datapoint (paper) required a reference.
Why is this odd? If I conducted a brief analysis and summary of
all papers published in a journal, would I need to cite each paper
individually, or merely state in the methods section that my dataset
consists of all 70 research papers published in Journal A in years X
and Y? While ACI and MIM are relatively small journals, if this approach were used to analyze papers published in, say, PNAS, their reference section would top 8000+ citations. Similarly, a meta-analysis of publication in PLOS ONE
would require citing nearly 60K papers. Clearly, there is something
about the context of paper-as-datapoint that distinguishes it from
paper-as-reference.
One could play devil’s advocate by assuming that it is normal
referencing behavior in the field of medical informatics to cite one’s
data points, even if they are papers, and unfortunately we’ve seen this
pattern before. In 2012, I took the editor of another medical
informatics journal to task for a similar self-referencing study. The editor conceded by removing all data points from his reference list, acknowledging that this was a “minor error” in a correction statement.
Citing papers-as-datapoints, in the cases of Lehmann and Haux is not
standard citation practice. The editors should have known this.
If it was not the intention of the editors to influence their
citation performance, there were other options open to them at the time
of authorship:
blow to the ability of a journal to attract future manuscripts. The
editors apologized for their actions in an editorial published soon after ACI suspension. In the future, they will refrain from publishing these kinds of papers or put their references in an appendix.
Thanks to Ludo Waltman for his assistance with VOS Viewer.
Visualizing Citation Cartels | The Scholarly Kitchen
Visualizing Citation Cartels
A citation cartel or valid study?
Unlike self-citation, which can be spotted when there are high levels of
references to other papers published in the same journal, cartels work
by influencing incoming citations from other journals.
In 2012, I reported on the first case of a citation cartel involving four biomedical journals. Later that year, Thomson Reuters suspended three of the four titles from receiving an Impact Factor. In 2014, they suspended six business journals for similar behavior.
This year, Thomson Reuters suspended Applied Clinical Informatics (ACI) for its role in distorting the citation performance of Methods of Information in Medicine (MIM). Both journals are published by Schattauer Publishers in Germany. According to the notice, 39% of 2015 citations to MIM came from ACI.
More importantly, 86% of these citations were directed to the previous
two years of publication — the years that count toward the journal’s
Impact Factor.
Thomson Reuters purposefully avoids using the term “citation cartel,”
which implies a willful attempt to game the system, and uses the more
ambiguous term “citation stacking” to describe the pattern itself.
Ultimately, we never know the intent of the authors who created the
citation pattern in the first place, only that it can distort the
ranking of a journal within its field. This is what Thomson Reuters
wants to avoid.
Schattauer Publishers appealed the suspension,
offering to exclude the offending papers from their Impact Factor
calculation as a concession. Their appeal was denied. Offering some
consolation to its readers, the publisher made all 2015 ACI papers freely available. It has also offered all ACI authors one free open access publication in 2016.
To better understand the citation pattern that resulted in ACI being suspended, I created, using VOS Viewer, a visualization of the citation network of papers published in ACI (blue) and MIM (red) from 2013 through 2015. Each paper lists its first author, year of publication and links to the papers it cites.
Citation
network of papers published in Applied Clinical Informatics (blue) and
Methods of Information in Medicine (red), 2013–2015.
network of papers published in Applied Clinical Informatics (blue) and
Methods of Information in Medicine (red), 2013–2015.
papers by Haux (blue). Each of these papers cites a large number of
papers published in the other journal within the previous two
years. Does this alone imply an intent to distort one’s Impact Factors?
We need more information.
Both Lehmann and Haux are on the editorial boards of both journals. Lehman is the Editor-in-Chief of ACI and also sits on the editorial board of MIM. Haux is the Senior Consulting Editor of MIM and also sits on the ACI editorial board.
This illustrates that there is a close relationship among the two
editors, but still this is not enough to imply intent. We need to look
at the four offending papers:
- The 2014 Lehmann paper (coauthored
by Haux) includes the following methods statement in its abstract:
“Retrospective, prolective observational study on recent publications of
ACI and MIM. All publications of the years 2012 and 2013 from these journals were indexed and analysed.” - Similarly, the 2014 Haux paper (coauthored
by Lehmann) includes this methods statement: “Retrospective, prolective
observational study on recent publications of ACI and MIM. All publications of the years 2012 and 2013 were indexed and analyzed.” - The 2015 Lehman paper states: “We conducted a retrospective observational study and reviewed all articles published in ACI during the calendar year 2014 (Volume 5)…”, and lastly,
- The 2015 Haux paper states: “We conducted a retrospective, observational study reviewing MIM articles published during 2014 (N=61) and analyzing reference lists of ACI articles from 2014 (N=70).
editors is that they are analyzing papers published in their own
journals within the time frame that affects the calculation of their
Impact Factors. Again, this alone does not imply an intent to game their
Impact Factor. Indeed, the publisher explained that
citation stacking was “an unintentional consequence of efforts to
analyze the effects of bridging between theory and practice.”
I can’t dispute what the editors and publisher state was their
intent. However, what is uniformly odd about these papers is that they
cite their dataset as if each datapoint (paper) required a reference.
Why is this odd? If I conducted a brief analysis and summary of
all papers published in a journal, would I need to cite each paper
individually, or merely state in the methods section that my dataset
consists of all 70 research papers published in Journal A in years X
and Y? While ACI and MIM are relatively small journals, if this approach were used to analyze papers published in, say, PNAS, their reference section would top 8000+ citations. Similarly, a meta-analysis of publication in PLOS ONE
would require citing nearly 60K papers. Clearly, there is something
about the context of paper-as-datapoint that distinguishes it from
paper-as-reference.
One could play devil’s advocate by assuming that it is normal
referencing behavior in the field of medical informatics to cite one’s
data points, even if they are papers, and unfortunately we’ve seen this
pattern before. In 2012, I took the editor of another medical
informatics journal to task for a similar self-referencing study. The editor conceded by removing all data points from his reference list, acknowledging that this was a “minor error” in a correction statement.
Citing papers-as-datapoints, in the cases of Lehmann and Haux is not
standard citation practice. The editors should have known this.
If it was not the intention of the editors to influence their
citation performance, there were other options open to them at the time
of authorship:
- They could have simply described their dataset without citing each paper.
- If citing each paper was important to the context of their paper,
they could have worked from a group of papers published outside the
Impact Factor window. Or, - They could have listed their papers in a footnote, appendix, or provided simple online links instead of formal references.
blow to the ability of a journal to attract future manuscripts. The
editors apologized for their actions in an editorial published soon after ACI suspension. In the future, they will refrain from publishing these kinds of papers or put their references in an appendix.
Thanks to Ludo Waltman for his assistance with VOS Viewer.
Visualizing Citation Cartels | The Scholarly Kitchen
Measuring Scientific Impact Beyond Citation Counts
Source: http://www.dlib.org/dlib/september16/patton/09patton.html
"Measuring Scientific Impact Beyond Citation Counts"
Robert M. Patton, Christopher G. Stahl and Jack C. Wells have published "Measuring Scientific Impact Beyond Citation Counts" in D-Lib Magazine.Here's an excerpt:
The measurement of scientific progress remains a significantDigital Curation and Digital Preservation Works | Open Access Works | Digital Scholarship | Digital Scholarship Sitemap
challenge exasperated by the use of multiple different types of metrics
that are often incorrectly used, overused, or even explicitly abused.
Several metrics such as h-index or journal impact factor (JIF) are often
used as a means to assess whether an author, article, or journal
creates an "impact" on science. Unfortunately, external forces can be
used to manipulate these metrics thereby diluting the value of their
intended, original purpose. This work highlights these issues and the
need to more clearly define "impact" as well as emphasize the need for
better metrics that leverage full content analysis of publications.
Measuring Scientific Impact Beyond Citation Counts
Subscribe to:
Posts (Atom)