Source:
http://ecco.vub.ac.be/?q=node/68
Scientific impact
How can I increase the scientific impact of my work?
Publish or perish
Science is a collective, collaborative enterprise, which means that
researchers build on the results of other researchers (cf. "standing on
the shoulders of giants", the slogan adopted by
Google Scholar).
That means that you would like any research you do to inform others, so
that they may refer to it, use it, and hopefully develop and improve
it. This will depend of course on the
quality of your work: incorrect or vague ideas or unreliable data are unlikely to inspire anyone to make something good out of it.
However, it is not sufficient to produce good quality research, you
must also ensure that others are informed about your results, i.e. you
should "publish" it in the broadest sense of the term. This explains the
"publish or perish" motto that characterizes present-day research: an
unpublished document could as well not exist since nobody but you can
use it. And scientists who do not produce useable results, are
considered to be non-existent within the academic community. Therefore,
they are very unlikely to get or keep an academic position ("perish"),
or collect additional funding. Publication has become
the primary criterion by which research activitity is evaluated.
Still, publishing in the sense of making publically available is far from sufficient: we are presently
drowned in an ocean of available information,
and it is very difficult for any particular publication to stand out
sufficiently so as to be guaranteed that it will be noticed by the
people that matter, i.e. your
peers: the scientific colleagues working in the same domain as you.
Before the advent of the net, publication was a slow, difficult and
expensive process, and therefore only few documents would reach that
stage. This provided some kind of indication that these documents were
really worthwhile. Therefore being published, especially being published
internationally, was already enough of a indication of quality or
impact for a scientific work.
Peer review
Nowadays, with publication technologies becoming ever easier and
cheaper to use, another "gold standard" of research quality has emerged:
being
peer-reviewed,
i.e. having passed a strict selection procedure where anonymous experts
in the domain have judged that your document is good enough to be
published in the particular journal, book or website to which you have
submitted it.
The principle of using "peers", i.e. people with a similar expertise
as you, is motivated by the fact that in science there are no ultimate
authorities who can judge what is right and what is wrong. The only
people who have enough expertise to judge are the people who are doing
research in the same domain as you. This means that one time researcher A
may evaluate the work of B, but another time B may be invited to judge
the work of A. The anonymity of the referees (and more rarely of the
authors being refereed) allows them to express themselves more honestly,
without fearing to offend a colleague, friend, or even potential
employer.
Peer review, while being the best existing method for research evaluation, does have problems of its own.
- One difficulty is to find competent experts: working in the same
domain does not necessarily mean being skilled in the same methodologies
or theories.
- Another one is that even for a good expert, novel research is by
definition difficult to evaluate, as there are no accepted standards to
judge the worth of an idea or approach, and as scientific work by its
nature tends to be very complex and abstract.
Therefore, it is unavoidable that referee reports are to some degree
subjective, and that different experts often disagree about the value of
the same work. To counter this, journal editors normally consult at
least three referees, so that there can be a clear majority for or
against publication of a submitted paper.
But even then, good papers can be rejected, because they do not
match the referees' or editor's view of how problems in that domains
should be approached. Resubmitting to another journal is often
sufficient to turn a rejected paper into an accepted one. But usually
referees provide detailed arguments why they reject a paper, and you are
strongly advised to take them into account when resubmitting. Even if
you disagree with the criticisms made, it is worth addressing them in a
revised version, so as to avoid similar misunderstandings.
Journal impact factors
With the proliferation of publication media, the standards of
acceptability can vary widely from journal to journal, depending on the
number of referees they use, the critical attitude of the referees, and
the fact that journals with many submission only can publish a fraction
of submitted papers. This leads to great variation, with some accepting
more than 50% of all submissions, others less than 5%.This created the
need for some kind of "quality score" to rank journals.
The most used standard is the
impact factor
which measures how often on average papers in that journal have been
referred to in other journals. This gives an indication of the "impact"
of the journal on the scientific community, i.e. the visibility and
authority that typicals papers in that journal have. As could be
expected, more people submit papers to high impact journals and
therefore by necessity their acceptance rate is much lower. This means
they are more selective in the papers they publish, and therefore can be
expected to offer higher quality on average.
The impact factor, which is computed each year by the
Institute for Scientific Information (ISI)
for those journals which it tracks, has become about the most used
indicator of the quality of a scientist's publications. Therefore, all
researchers should be motivated to publish their work in journals with
the highest impact factor possible. About the highest scoring journals
overall are
Nature and
Science. Within the cognitive sciences,
Behavioral and Brain Sciences, and
Trends in Cognitive Sciences, have the highest impact, so it is worth submitting a paper there. Impact factors for journals can be found in the
Journal Citation Reports
However, the impact factor has a number of intrinsic shortcomings:
- only a fraction of all journals are tracked by the ISI, and therefore many journals don't have an impact factor.
- moreover, citations in journals not tracked by the ISI are not
counted. Therefore even an ISI journal's impact factor may be
underestimated
- an impact factor is only a snapshot over a particular time period (typically 2 years)
- depending on the discipline, there are large differences in the
average amount of citations (e.g. medical researchers cite much more
than mathematicians), and in the number of journals that have a high
impact factor–or any impact factor at all. New domains, such as
complexity science or memetics, will typically have few or no journals
with a high impact. Therefore, impact factors can in general not be used
to compare work across disciplines.
- an impact factor only measures the average citation rate for all
papers in a journal. Some papers may score much better, becoming
"citation classics" that everyone in the field knows, while others don't
get any citation at all.
Personal citation scores
For these reason, in the long term it seems better to aim at high
"personal citation scores" rather than high journal citation scores,
i.e. make sure that your paper/book/document is referred to by many
people, independently of the place where it has been published. For
example, books are typically cited more often than papers, but don't
have impact factors. Below, we'll discuss some methods to achieve that.
For the short-term, though, for young researchers who cannot afford to
wait several years before their works becomes widely known because they
need to renew their funding, the best strategy is to submit first to
high impact journals, and if this doesn't work, gradually go down in the
"pecking order" until you find a journal willing to accept your paper.
Personal citation scores are also tracked by the ISI, in their
Web of Science,
including the Science Citation Index, Social Science Citation Index and
Arts and Humanities Citation Index. However, since the ISI database
dates from long before the explosion in cheap computers and networks,
they tend to keep minimal data: merely author's last name and initials,
abbreviated title, journal name or abbreviation, and volume, issue and
page number.
This makes it very difficult to find all your citations, especially
if you have a common name shared with many other authors: you'll have to
go one by one through the list of cited papers to eliminate all those
written by someone with the same family name and initials, and keep
track manually of all the times papers of yours have been cited. Also,
the same name can sometimes be spelled differently, e.g. a paper by "Van
Overwalle, Frank" can be listed under VANOVERWALLE F, VAN OVERWALLE F,
VAN OVERWALLE F P or sometimes even OVERWALLE F V. It is worth checking
all possible alternative spellings, or your citation rate can be
strongly underestimated. Still, it may be worth to go through the
exercise to convince potential sponsors of your research that your work
is well-recognized internationally.
An advantage of this database is that it also includes citation of
books or documents that were not published in one of the journals that
ISI tracks, so you are not limited to traditional publication outlets to
collect citation scores. However, the citations themselves come
exclusively from tracked journals, and thus may underestimate
non-traditional domains for which there are no established journals in
the database.
PageRank: impact on the web
The newest generation of publication outlets makes fully use of the
ease and computability of the web, providing "impact" scores that more
flexibly reflect the true influence of your work on others. The most
popular and probably most effective method is the
PageRank algorithm that the
Google search engine
uses to calculate the importance of a web page. This algorithm does not
calculate importance or impact only by the number of links (equivalent
to citations or references), but by importance of the pages that link to
you. Thus, importance is calculated recursively, and a document cited
only once, but by a very well known other webpage may still get a high
PageRank score, while a paper cited by many other papers that are
themselves hardly referred to by others may have a quite low PageRank.
This alleviates the problem of the time it takes for novel work to
become widely known: it can be sufficient to convince one respected
authority in the field to link to you in order to immediately become
much more visible (i.e. easy to find through the Google Search engine).
With the recent advent of
Google Scholar,
a search engine only for the scientific literature, Google seems to
move towards an integation of (web-based) PageRanks, and (journal-based)
citation scores, though it is not clear how the algorithm weighs the
different contributions when finding the most "important" scientific
paper on a given topic. An advantage of Google Scholar is that it is not
limited to ISI-tracked journals as it also counts citations in working
papers published only on the web. But it remains unclear which
journal/papers are in Scholar's database, and which are not...
In the longer term, it seems likely that such recursive, web-based
methods will become increasingly important, as more literature becomes
available on the web, and the algorithms are further refined. Therefore,
it seems like a pretty safe bet that having a high (Scholar) PageRank
will be the most effective way to get your scientific work recognized.
Moreover, you don't lose anything by focusing on increasing this web
visibility, as people publishing papers in ISI journals also
increasingly use the web to find relevant literature. Therefore they are
more likely to cite a paper that is visible on the web, even if it was
never published in a journal, or even underwent peer review. But is it
not clear when funding authorities will start using web-based methods to
directly evaluate the worth/impact of your research...
Improving your impact
As has become clear, there are two major methods to make your research more authoritative:
- publishing in high-impact journals
- persuading other scientists to refer to your work
The first is the most traditional, but has the disadvantage that for
novel, interdisciplinary research it will be difficuld to find a journal
willing to accept a paper that does not fulfil its standard criteria of
subject, methodology, reliability, etc. Still, improving the quality of
your research and writing will definitely increase your chances of
success.
For the second too, quality is primary, but here you have more
leeway in getting recognition for unusual approaches. The most
straighforward method is to look around in order to spot who is
interested in similar approaches. You can then introduce your ideas to
these peers, e.g. by:
- presenting your work at conferences and seminars on the subject
- participating in email lists that discuss related topics, and referring to your papers when relevant for the on-going discussion
- directly contacting peers, and pointing them to your papers, e.g. as
available on a website, preprint archive, or journal, or sending them a
copy of your paper.
- having your work linked to by websites that attract a large number
of like-minded people. If the website is really well-known (e.g. Principia Cybernetica Web
for ECCO people) it will have a high PageRank, which will in part
"propagate" to your webpage, and thus increase your visibility in
Google.
- making sure that your paper is well-structured, containing all the
appropriate keywords, title, abstract, etc., so that people looking for
such papers are likely to effectively find them
- submitting your papers to "preprint archives" on related subjects (e.g ArXiv or CogPrints), which are typically scanned by many researchers looking for something that falls in their domain of interest
- making sure your paper is listed in the publications or working papers
page of your research group or university: usually the visibility of
the institution is much larger than that of a single member
- making your work known outside the academic community, e.g. by
contacting science journalists, who typically reach a much wider, albeit
less specialized audience. This will only work for non-technical work
with broad implications.
None of these methods implies that you should broadcast your work as
far and wide as possible ("spamming"). People are more likely to get
irritated by "off-topic" announcements with little relevance to their
work, and are more likely to ignore your announcements later, even if
now they are relevant. The point is to be selective, and find the best
academic environments, where your ideas have most chance to take root...
For that, you'll need a keen eye for what's going in your field, and
frequently investigate who is using similar keywords, or referrring to
the same authors as you.
Scientific impact | ecco.vub.ac.be
