Most researchers are aware of the increasing emphasis
placed on citations as an indicator of quality by their colleagues, as
well as by administrators and librarians. But is it true that high
quality papers are cited more often? In what we believe to be the only
previous publication to examine this question, West and McIlwaine (2002)
selected 79 papers that had been published in the journal Addiction
in 1997. These papers were rated by an assistant editor of the journal
on a five-level quality scale (ranging from 'among the top 10%' to
'should not have been published') as well as a second 'blind' rater. The
authors found that the number of citations to each paper from 1997 to
2000 did not correlate with the mean score of the two reviewers, the
score assigned by the editors, or the score assigned by the blind
referees.

The peer-review system used by Physics in Medicine and Biology
(PMB) facilitates a larger, prospective study with much less effort. In
general, two reviewers are asked to assign each paper a score of 1 to
10 in each of three categories: originality, soundness and significance.
The total scores (out of 30) are averaged and on this basis manuscripts
are given a quality rating from Q1 (highest quality) to Q5. In a paper
recently published in Scientometrics (Patterson and Harris
2009), we compared the number of citations to papers published in PMB in
2003, 2004 and 2005 to the mean quality rating assigned by the
reviewers. In total, data were analyzed for 1095 published papers. For
each year we found that there was a low but statistically robust
correlation between citations and quality rating: for 2003, -0.227 (p < 0.001); for 2004, -0.238 (p < 0.001), and for 2005, -0.154 (p
< 0.01). The low correlation means that it is not possible to
predict the citation frequency of an individual paper with accuracy but,
as a group, the highest ranked papers were cited about twice as often
as the average for all published papers. We also examined the data
retrospectively by dividing the papers published in each year into five
citation quintiles. A paper of the highest quality (Q1) was about ten
times more likely to be found in the most-cited quintile than in the
least-cited.

While it is reassuring to find that the best
papers in PMB are, indeed, cited more often on average, we cannot
discount the hypothesis that both measures might be influenced by an
extrinsic factor, such as the reputation of the authors. We suggest that
a study similar to ours be performed for a journal that utilizes a
system of double-blind peer review.

Michael S Patterson

Editorial Board Member

Simon Harris

Publisher

References West
R and McIlwaine A 2002 What do citation counts count for in the field
of addiction? An empirical evaluation of citation counts and their link
with peer ratings of quality Addiction 97 501–4

Patterson
M S and Harris S 2009 The relationship between reviewers'
quality-scores and number of citations for papers published in the
journal Physics in Medicine and Biology from 2003–2005 Scientometrics 80 343–9