GLOBAL
How to win citations and rise up the university rankings
Richard Holmes 07 November 2014 Issue No:342
A large part of the academic world has either been congratulating itself on performing well in the latest Times Higher Education, or THE, World University Rankings,
the data for which is provided by Thomson Reuters, or complaining that
only large injections of public money will keep their universities from
falling into the great pit of the unranked.
Some, however, have been baffled by some of the placings reported by THE
this year. Federico Santa Maria Technical University in Chile is
allegedly the fourth best university in Latin America, Scuola Normale
Superiore di Pisa the best in Italy and Turkish universities are
apparently the rising stars of the academic world.
When there is a university that appears to be punching above its weight the cause often turns out to be the citations indicator.
So what is going on?
Methodology
The problem lies with various aspects of Thomson Reuters' methodology.
First, they use field normalisation. That means that they do not simply
count the number of citations but compare the number of citations in 250
fields with the world average in each field. Not only that, but they
compare each year in which the paper is cited with the world average of
citations for that year.
The rationale for this is that the number of citations and the rapidity with which papers are cited vary from field to field.
A paper reporting a cure for cancer or the discovery of a new particle
will be cited hundreds of times within weeks. A paper in philosophy,
economics or history may languish for years before anyone takes notice.
John Muth's work on rational expectations was hardly noticed or cited for years before eventually starting a revolution in economic theory.
So universities should be compared to the average for fields and years.
Otherwise, those that are strong in the humanities and social sciences
will be penalised.
Up to a point this is not a bad idea. But it does assume that all disciplines are equally valuable and demanding.
But if the world has decided that it will fund medical research or
astrophysics and support journals and pay researchers to read and cite
other researchers' papers rather than media studies or education, then
this is perhaps something rankers and data collectors should take
account of.
In any case, by normalising for so many fields and then throwing
normalisation by year into the mix, Thomson Reuters increases the
likelihood of statistical anomalies.
If someone can get a few dozen citations within a couple of years after
publication in a field where citations, especially early ones, average
below one a year, then this could give an enormous boost to a
university's citation score.
That is precisely what happened with Egypt’s Alexandria University in
2010. Methodological tweaking has mitigated the risk to some extent but
not completely. A university could also get a big boost by getting
credit, no matter how undeserved, for a breakthrough paper or a review
that is widely cited.
How citations work
So let's take a look at some of the influential universities in the 2014 THE rankings.
Scuola Normale Superiore di Pisa, or SNSP, is a small research intensive
institution that might not even meet the criteria to be ranked by
Thomson Reuters. Its output is modest – 2,407 publications in the Web of
Science core collection between 2009 and 2013 – although for a small
institution that is quite good.
One of those publications is “Observation of a new boson...” in Physics Letters B in September 2012, which has been cited 1,631 times.
The paper has 2,896 ‘authors’, whom I counted by looking for semicolons
in the ‘find’ box, affiliated to 228 institutions. Five of them are from
SNSP.
To put it crudely, SNSP is making an ‘authorship’ contribution of 0.17 %
to the paper, but getting 100% of the citation credit, as does every
other contributor. Perhaps its researchers are playing a leading role in
the Large Hadron Collider project or perhaps it has made a
disproportionate financial contribution, but Thomson Reuters provides no
reason to think so.
The University of the Andes, supposedly the second best university in
Latin America, is also a contributor to this publication, as is Panjab
University, supposedly the second best institution on the Indian
subcontinent.
Meanwhile, Royal Holloway, University of London, has contributed to
“Observation of a new particle...” in the same journal and issue. This
has received 1,734 citations and involved 2,932 authors from 267
institutions, along with Tokyo Metropolitan University, Federico Santa
Maria Technical University, Middle East Technical University and
Bogazici University.
The University of California, Santa Cruz, is one of 119 institutions
that contributed to the “Review of particle physics...” 2010, which has
been cited 3,739 times to date. Like all the other contributors it gets
full credit for all those citations.
It is not just the number of citations that boosts citation impact
scores, but also their occurrence within a year or two of publication so
that the number of citations is much greater than the average for that
field and those years.
The proliferation of papers with hundreds of authors is not confined to
physics. There are several examples from medicine and genetics as well.
At this point, the question arises why not divide the citations for each
paper among the authors of the paper? This is an option available in
the Leiden Ranking so it should not be beyond Thomson Reuters’ technical
capabilities.
Or why not stop counting multi-authored publications when they exceed a
certain quota of authors? This is exactly what Thomson Reuters did earlier this year
when collecting data for its new highly cited researchers' lists.
Physics papers with more than 30 institutional affiliations were
omitted, a very sensible procedure that should have been applied across
the board.
So basically, one route to success in the rankings is to get into a multi-collaborator mega-cited project.
Keep it small
But that is not enough in itself. There are hundreds of universities
contributing to these publications. But not all of them reap such
disproportionate benefits. It is important not to publish too much.
A dozen LHC papers will do wonders if you publish 400 or 500 papers a
year. Four thousand a year and it will make little difference. One
reason for the success of otherwise obscure institutions is that the
number of papers by which the citations are divided is small.
So why on earth is Thomson Reuters using a method that produces such
laughable results? Let’s face it, if any other ranker put SNS Pisa,
Federico Santa Maria or Bogazici at the top of its flagship indicator we
would go deaf from the chorus of academic tut-tutting.
Thomson Reuters, I suspect, is doing this because this method is
identical or nearly identical to that used for their InCites system for
evaluating individual academics within institutions, which appears very
lucrative, and they do not want the expense and inconvenience of
recalculating data.
Also, perhaps, Thomson Reuters has become so enamoured of the complexity
and sophistication of its operations that it really does think it has
actually discovered pockets of excellence in unlikely places that nobody
else has the skill or the resources to even notice.
But we have not finished. There is one more element in Thomson Reuters'
distinctive methodology and that is its regional modification,
introduced by Thomson Reuters in 2011.
This means that the normalised citation impact score of the university
is divided by the square root of the impact score of the country in
which it is located. A university located in a low-scoring country will
get a bonus that will be greater the lower the country's impact score
is. This would clearly be an advantage to countries like Chile, India
and Turkey.
Every year there are more multi-authored, multi-cited papers. It would
not be surprising if university presidents started scanning the author
lists of publications like the Review of Particle Physics, send out recruitment letters and get ready for ranking stardom.
* Richard Holmes is writer of the University Ranking Watch blog on which this article was first published.
the data for which is provided by Thomson Reuters, or complaining that
only large injections of public money will keep their universities from
falling into the great pit of the unranked.
Some, however, have been baffled by some of the placings reported by THE
this year. Federico Santa Maria Technical University in Chile is
allegedly the fourth best university in Latin America, Scuola Normale
Superiore di Pisa the best in Italy and Turkish universities are
apparently the rising stars of the academic world.
When there is a university that appears to be punching above its weight the cause often turns out to be the citations indicator.
- Scuola Normale Superiore di Pisa is 63rd in the world with an overall score of 61.9 but a citations score of 96.4.
- Royal Holloway, University of London is 118th in the world with an overall score of 53 but a citations score of 98.9.
- The University of California, Santa Cruz, is top of the world
for citations with an overall score of 53.7 and 100 for citations.
- Bogazici University in Turkey is 139th in the world with an overall score of 51.1 and a citations score of 96.8.
- Federico Santa Maria Technical University in Valparaiso is in
the 251-175 band so the total score is not given, although it would be
easy enough to work out. It has a score of 99.7 for citations.
So what is going on?
Methodology
The problem lies with various aspects of Thomson Reuters' methodology.
First, they use field normalisation. That means that they do not simply
count the number of citations but compare the number of citations in 250
fields with the world average in each field. Not only that, but they
compare each year in which the paper is cited with the world average of
citations for that year.
The rationale for this is that the number of citations and the rapidity with which papers are cited vary from field to field.
A paper reporting a cure for cancer or the discovery of a new particle
will be cited hundreds of times within weeks. A paper in philosophy,
economics or history may languish for years before anyone takes notice.
John Muth's work on rational expectations was hardly noticed or cited for years before eventually starting a revolution in economic theory.
So universities should be compared to the average for fields and years.
Otherwise, those that are strong in the humanities and social sciences
will be penalised.
Up to a point this is not a bad idea. But it does assume that all disciplines are equally valuable and demanding.
But if the world has decided that it will fund medical research or
astrophysics and support journals and pay researchers to read and cite
other researchers' papers rather than media studies or education, then
this is perhaps something rankers and data collectors should take
account of.
In any case, by normalising for so many fields and then throwing
normalisation by year into the mix, Thomson Reuters increases the
likelihood of statistical anomalies.
If someone can get a few dozen citations within a couple of years after
publication in a field where citations, especially early ones, average
below one a year, then this could give an enormous boost to a
university's citation score.
That is precisely what happened with Egypt’s Alexandria University in
2010. Methodological tweaking has mitigated the risk to some extent but
not completely. A university could also get a big boost by getting
credit, no matter how undeserved, for a breakthrough paper or a review
that is widely cited.
How citations work
So let's take a look at some of the influential universities in the 2014 THE rankings.
Scuola Normale Superiore di Pisa, or SNSP, is a small research intensive
institution that might not even meet the criteria to be ranked by
Thomson Reuters. Its output is modest – 2,407 publications in the Web of
Science core collection between 2009 and 2013 – although for a small
institution that is quite good.
One of those publications is “Observation of a new boson...” in Physics Letters B in September 2012, which has been cited 1,631 times.
The paper has 2,896 ‘authors’, whom I counted by looking for semicolons
in the ‘find’ box, affiliated to 228 institutions. Five of them are from
SNSP.
To put it crudely, SNSP is making an ‘authorship’ contribution of 0.17 %
to the paper, but getting 100% of the citation credit, as does every
other contributor. Perhaps its researchers are playing a leading role in
the Large Hadron Collider project or perhaps it has made a
disproportionate financial contribution, but Thomson Reuters provides no
reason to think so.
The University of the Andes, supposedly the second best university in
Latin America, is also a contributor to this publication, as is Panjab
University, supposedly the second best institution on the Indian
subcontinent.
Meanwhile, Royal Holloway, University of London, has contributed to
“Observation of a new particle...” in the same journal and issue. This
has received 1,734 citations and involved 2,932 authors from 267
institutions, along with Tokyo Metropolitan University, Federico Santa
Maria Technical University, Middle East Technical University and
Bogazici University.
The University of California, Santa Cruz, is one of 119 institutions
that contributed to the “Review of particle physics...” 2010, which has
been cited 3,739 times to date. Like all the other contributors it gets
full credit for all those citations.
It is not just the number of citations that boosts citation impact
scores, but also their occurrence within a year or two of publication so
that the number of citations is much greater than the average for that
field and those years.
The proliferation of papers with hundreds of authors is not confined to
physics. There are several examples from medicine and genetics as well.
At this point, the question arises why not divide the citations for each
paper among the authors of the paper? This is an option available in
the Leiden Ranking so it should not be beyond Thomson Reuters’ technical
capabilities.
Or why not stop counting multi-authored publications when they exceed a
certain quota of authors? This is exactly what Thomson Reuters did earlier this year
when collecting data for its new highly cited researchers' lists.
Physics papers with more than 30 institutional affiliations were
omitted, a very sensible procedure that should have been applied across
the board.
So basically, one route to success in the rankings is to get into a multi-collaborator mega-cited project.
Keep it small
But that is not enough in itself. There are hundreds of universities
contributing to these publications. But not all of them reap such
disproportionate benefits. It is important not to publish too much.
A dozen LHC papers will do wonders if you publish 400 or 500 papers a
year. Four thousand a year and it will make little difference. One
reason for the success of otherwise obscure institutions is that the
number of papers by which the citations are divided is small.
So why on earth is Thomson Reuters using a method that produces such
laughable results? Let’s face it, if any other ranker put SNS Pisa,
Federico Santa Maria or Bogazici at the top of its flagship indicator we
would go deaf from the chorus of academic tut-tutting.
Thomson Reuters, I suspect, is doing this because this method is
identical or nearly identical to that used for their InCites system for
evaluating individual academics within institutions, which appears very
lucrative, and they do not want the expense and inconvenience of
recalculating data.
Also, perhaps, Thomson Reuters has become so enamoured of the complexity
and sophistication of its operations that it really does think it has
actually discovered pockets of excellence in unlikely places that nobody
else has the skill or the resources to even notice.
But we have not finished. There is one more element in Thomson Reuters'
distinctive methodology and that is its regional modification,
introduced by Thomson Reuters in 2011.
This means that the normalised citation impact score of the university
is divided by the square root of the impact score of the country in
which it is located. A university located in a low-scoring country will
get a bonus that will be greater the lower the country's impact score
is. This would clearly be an advantage to countries like Chile, India
and Turkey.
Every year there are more multi-authored, multi-cited papers. It would
not be surprising if university presidents started scanning the author
lists of publications like the Review of Particle Physics, send out recruitment letters and get ready for ranking stardom.
* Richard Holmes is writer of the University Ranking Watch blog on which this article was first published.
Disclaimer
All reader responses posted on this site are those of the reader ONLY and NOTthose of University World News or Higher Education Web Publishing, their
associated trademarks, websites and services. University World News or
Higher Education Web Publishing does not necessarily endorse, support,
sanction, encourage, verify or agree with any comments, opinions or
statements or other content provided by readers.
How to win citations and rise up the university rankings - University World News
No comments:
Post a Comment