Sunday, 18 December 2016

Ten Simple Rules for Writing a Literature Review


PLoS Comput Biol. 2013 Jul; 9(7): e1003149.
Published online 2013 Jul 18. doi:  10.1371/journal.pcbi.1003149
PMCID: PMC3715443

Ten Simple Rules for Writing a Literature Review

Philip E. Bourne, Editor
reviews are in great demand in most scientific fields. Their need stems
from the ever-increasing output of scientific publications [1].
For example, compared to 1991, in 2008 three, eight, and forty times
more papers were indexed in Web of Science on malaria, obesity, and
biodiversity, respectively [2].
Given such mountains of papers, scientists cannot be expected to
examine in detail every single new paper relevant to their interests [3].
Thus, it is both advantageous and necessary to rely on regular
summaries of the recent literature. Although recognition for scientists
mainly comes from primary research, timely literature reviews can lead
to new synthetic insights and are often widely read [4]. For such summaries to be useful, however, they need to be compiled in a professional way [5].
starting from scratch, reviewing the literature can require a titanic
amount of work. That is why researchers who have spent their career
working on a certain research issue are in a perfect position to review
that literature. Some graduate schools are now offering courses in
reviewing the literature, given that most research students start their
project by producing an overview of what has already been done on their
research issue [6]. However, it is likely that most scientists have not thought in detail about how to approach and carry out a literature review.
the literature requires the ability to juggle multiple tasks, from
finding and evaluating relevant material to synthesising information
from various sources, from critical thinking to paraphrasing,
evaluating, and citation skills [7].
In this contribution, I share ten simple rules I learned working on
about 25 literature reviews as a PhD and postdoctoral student. Ideas and
insights also come from discussions with coauthors and colleagues, as
well as feedback from reviewers and editors.

Rule 1: Define a Topic and Audience

to choose which topic to review? There are so many issues in
contemporary science that you could spend a lifetime of attending
conferences and reading the literature just pondering what to review. On
the one hand, if you take several years to choose, several other people
may have had the same idea in the meantime. On the other hand, only a
well-considered topic is likely to lead to a brilliant literature review
[8]. The topic must at least be:
  1. interesting
    to you (ideally, you should have come across a series of recent papers
    related to your line of work that call for a critical summary),
  2. an
    important aspect of the field (so that many readers will be interested
    in the review and there will be enough material to write it), and
  3. a well-defined issue (otherwise you could potentially include thousands of publications, which would make the review unhelpful).
Ideas for potential reviews may come from papers providing lists of key research questions to be answered [9],
but also from serendipitous moments during desultory reading and
discussions. In addition to choosing your topic, you should also select a
target audience. In many cases, the topic (e.g., web services in
computational biology) will automatically define an audience (e.g.,
computational biologists), but that same topic may also be of interest
to neighbouring fields (e.g., computer science, biology, etc.).

Rule 2: Search and Re-search the Literature

having chosen your topic and audience, start by checking the literature
and downloading relevant papers. Five pieces of advice here:
  1. keep track of the search items you use (so that your search can be replicated [10]),
  2. keep a list of papers whose pdfs you cannot access immediately (so as to retrieve them later with alternative strategies),
  3. use a paper management system (e.g., Mendeley, Papers, Qiqqa, Sente),
  4. define
    early in the process some criteria for exclusion of irrelevant papers
    (these criteria can then be described in the review to help define its
    scope), and
  5. do not just look for research papers in the area you wish to review, but also seek previous reviews.
The chances are high that someone will already have published a literature review (Figure 1),
if not exactly on the issue you are planning to tackle, at least on a
related topic. If there are already a few or several reviews of the
literature on your issue, my advice is not to give up, but to carry on
with your own literature review,
Figure 1
conceptual diagram of the need for different types of literature
reviews depending on the amount of published research papers and
literature reviews.
  1. discussing in your review the approaches, limitations, and conclusions of past reviews,
  2. trying to find a new angle that has not been covered adequately in the previous reviews, and
  3. incorporating new material that has inevitably accumulated since their appearance.
When searching the literature for pertinent papers and reviews, the usual rules apply:
  1. be thorough,
  2. use
    different keywords and database sources (e.g., DBLP, Google Scholar,
    ISI Proceedings, JSTOR Search, Medline, Scopus, Web of Science), and
  3. look at who has cited past relevant papers and book chapters.

Rule 3: Take Notes While Reading

you read the papers first, and only afterwards start writing the
review, you will need a very good memory to remember who wrote what, and
what your impressions and associations were while reading each single
paper. My advice is, while reading, to start writing down interesting
pieces of information, insights about how to organize the review, and
thoughts on what to write. This way, by the time you have read the
literature you selected, you will already have a rough draft of the
Of course, this draft will still need much rewriting, restructuring, and rethinking to obtain a text with a coherent argument [11],
but you will have avoided the danger posed by staring at a blank
document. Be careful when taking notes to use quotation marks if you are
provisionally copying verbatim from the literature. It is advisable
then to reformulate such quotes with your own words in the final draft.
It is important to be careful in noting the references already at this
stage, so as to avoid misattributions. Using referencing software from
the very beginning of your endeavour will save you time.

Rule 4: Choose the Type of Review You Wish to Write

having taken notes while reading the literature, you will have a rough
idea of the amount of material available for the review. This is
probably a good time to decide whether to go for a mini- or a full
review. Some journals are now favouring the publication of rather short
reviews focusing on the last few years, with a limit on the number of
words and citations. A mini-review is not necessarily a minor review: it
may well attract more attention from busy readers, although it will
inevitably simplify some issues and leave out some relevant material due
to space limitations. A full review will have the advantage of more
freedom to cover in detail the complexities of a particular scientific
development, but may then be left in the pile of the very important
papers “to be read” by readers with little time to spare for major
There is probably a
continuum between mini- and full reviews. The same point applies to the
dichotomy of descriptive vs. integrative reviews. While descriptive
reviews focus on the methodology, findings, and interpretation of each
reviewed study, integrative reviews attempt to find common ideas and
concepts from the reviewed material [12].
A similar distinction exists between narrative and systematic reviews:
while narrative reviews are qualitative, systematic reviews attempt to
test a hypothesis based on the published evidence, which is gathered
using a predefined protocol to reduce bias [13], [14].
When systematic reviews analyse quantitative results in a quantitative
way, they become meta-analyses. The choice between different review
types will have to be made on a case-by-case basis, depending not just
on the nature of the material found and the preferences of the target
journal(s), but also on the time available to write the review and the
number of coauthors [15].

Rule 5: Keep the Review Focused, but Make It of Broad Interest

Whether your plan is to write a mini- or a full review, it is good advice to keep it focused 16,17.
Including material just for the sake of it can easily lead to reviews
that are trying to do too many things at once. The need to keep a review
focused can be problematic for interdisciplinary reviews, where the aim
is to bridge the gap between fields [18].
If you are writing a review on, for example, how epidemiological
approaches are used in modelling the spread of ideas, you may be
inclined to include material from both parent fields, epidemiology and
the study of cultural diffusion. This may be necessary to some extent,
but in this case a focused review would only deal in detail with those
studies at the interface between epidemiology and the spread of ideas.
focus is an important feature of a successful review, this requirement
has to be balanced with the need to make the review relevant to a broad
audience. This square may be circled by discussing the wider
implications of the reviewed topic for other disciplines.

Rule 6: Be Critical and Consistent

the literature is not stamp collecting. A good review does not just
summarize the literature, but discusses it critically, identifies
methodological problems, and points out research gaps [19]. After having read a review of the literature, a reader should have a rough idea of:
  1. the major achievements in the reviewed field,
  2. the main areas of debate, and
  3. the outstanding research questions.
is challenging to achieve a successful review on all these fronts. A
solution can be to involve a set of complementary coauthors: some people
are excellent at mapping what has been achieved, some others are very
good at identifying dark clouds on the horizon, and some have instead a
knack at predicting where solutions are going to come from. If your
journal club has exactly this sort of team, then you should definitely
write a review of the literature! In addition to critical thinking, a
literature review needs consistency, for example in the choice of
passive vs. active voice and present vs. past tense.

Rule 7: Find a Logical Structure

a well-baked cake, a good review has a number of telling features: it
is worth the reader's time, timely, systematic, well written, focused,
and critical. It also needs a good structure. With reviews, the usual
subdivision of research papers into introduction, methods, results, and
discussion does not work or is rarely used. However, a general
introduction of the context and, toward the end, a recapitulation of the
main points covered and take-home messages make sense also in the case
of reviews. For systematic reviews, there is a trend towards including
information about how the literature was searched (database, keywords,
time limits) [20].
can you organize the flow of the main body of the review so that the
reader will be drawn into and guided through it? It is generally helpful
to draw a conceptual scheme of the review, e.g., with mind-mapping
techniques. Such diagrams can help recognize a logical way to order and
link the various sections of a review [21].
This is the case not just at the writing stage, but also for readers if
the diagram is included in the review as a figure. A careful selection
of diagrams and figures relevant to the reviewed topic can be very
helpful to structure the text too [22].

Rule 8: Make Use of Feedback

Reviews of the literature are normally peer-reviewed in the same way as research papers, and rightly so [23].
As a rule, incorporating feedback from reviewers greatly helps improve a
review draft. Having read the review with a fresh mind, reviewers may
spot inaccuracies, inconsistencies, and ambiguities that had not been
noticed by the writers due to rereading the typescript too many times.
It is however advisable to reread the draft one more time before
submission, as a last-minute correction of typos, leaps, and muddled
sentences may enable the reviewers to focus on providing advice on the
content rather than the form.
is vital to writing a good review, and should be sought from a variety
of colleagues, so as to obtain a diversity of views on the draft. This
may lead in some cases to conflicting views on the merits of the paper,
and on how to improve it, but such a situation is better than the
absence of feedback. A diversity of feedback perspectives on a
literature review can help identify where the consensus view stands in
the landscape of the current scientific understanding of an issue [24].

Rule 9: Include Your Own Relevant Research, but Be Objective

many cases, reviewers of the literature will have published studies
relevant to the review they are writing. This could create a conflict of
interest: how can reviewers report objectively on their own work [25]?
Some scientists may be overly enthusiastic about what they have
published, and thus risk giving too much importance to their own
findings in the review. However, bias could also occur in the other
direction: some scientists may be unduly dismissive of their own
achievements, so that they will tend to downplay their contribution (if
any) to a field when reviewing it.
general, a review of the literature should neither be a public relations
brochure nor an exercise in competitive self-denial. If a reviewer is
up to the job of producing a well-organized and methodical review, which
flows well and provides a service to the readership, then it should be
possible to be objective in reviewing one's own relevant findings. In
reviews written by multiple authors, this may be achieved by assigning
the review of the results of a coauthor to different coauthors.

Rule 10: Be Up-to-Date, but Do Not Forget Older Studies

the progressive acceleration in the publication of scientific papers,
today's reviews of the literature need awareness not just of the overall
direction and achievements of a field of inquiry, but also of the
latest studies, so as not to become out-of-date before they have been
published. Ideally, a literature review should not identify as a major
research gap an issue that has just been addressed in a series of papers
in press (the same applies, of course, to older, overlooked studies
(“sleeping beauties” [26])).
This implies that literature reviewers would do well to keep an eye on
electronic lists of papers in press, given that it can take months
before these appear in scientific databases. Some reviews declare that
they have scanned the literature up to a certain point in time, but
given that peer review can be a rather lengthy process, a full search
for newly appeared literature at the revision stage may be worthwhile.
Assessing the contribution of papers that have just appeared is
particularly challenging, because there is little perspective with which
to gauge their significance and impact on further research and society.
new papers on the reviewed topic (including independently written
literature reviews) will appear from all quarters after the review has
been published, so that there may soon be the need for an updated
review. But this is the nature of science [27][32]. I wish everybody good luck with writing a review of the literature.


thanks to M. Barbosa, K. Dehnen-Schmutz, T. Döring, D. Fontaneto, M.
Garbelotto, O. Holdenrieder, M. Jeger, D. Lonsdale, A. MacLeod, P.
Mills, M. Moslonka-Lefebvre, G. Stancanelli, P. Weisberg, and X. Xu for
insights and discussions, and to P. Bourne, T. Matoni, and D. Smith for
helpful comments on a previous draft.

Funding Statement

work was funded by the French Foundation for Research on Biodiversity
(FRB) through its Centre for Synthesis and Analysis of Biodiversity data
(CESAB), as part of the NETSEED research project. The funders had no
role in the preparation of the manuscript.


1. Rapple
C (2011) The role of the critical review article in alleviating
information overload. Annual Reviews White Paper. Available: Accessed May 2013.
2. Pautasso M (2010) Worsening file-drawer problem in the abstracts of natural, medical and social science databases. Scientometrics 85: 193–202doi:10.1007/s11192-010-0233-5
3. Erren TC, Cullen P, Erren M (2009) How to surf today's information tsunami: on the craft of effective reading. Med Hypotheses 73: 278–279doi:10.1016/j.mehy.2009.05.002 [PubMed]
4. Hampton SE, Parker JN (2011) Collaboration and productivity in scientific synthesis. Bioscience 61: 900–910doi:10.1525/bio.2011.61.11.9
5. Ketcham CM, Crawford JM (2007) The impact of review articles. Lab Invest 87: 1174–1185doi:10.1038/labinvest.3700688 [PubMed]
6. Boote DN, Beile P (2005) Scholars before researchers: on the centrality of the dissertation literature review in research preparation. Educ Res 34: 3–15doi:10.3102/0013189X034006003
7. Budgen
D, Brereton P (2006) Performing systematic literature reviews in
software engineering. Proc 28th Int Conf Software Engineering, ACM New
York, NY, USA, pp. 1051–1052.10.1145/1134285.1134500
8. Maier HR (2013) What constitutes a good literature review and why does its quality matter? Environ Model Softw 43: 3–4doi:10.1016/j.envsoft.2013.02.004
9. Sutherland WJ, Fleishman E, Mascia MB, Pretty J, Rudd MA (2011) Methods for collaboratively identifying research priorities and emerging issues in science and policy. Methods Ecol Evol 2: 238–247doi:10.1111/j.2041-210X.2010.00083.x
10. Maggio LA, Tannery NH, Kanter SL (2011) Reproducibility of literature search reporting in medical education reviews. Acad Med 86: 1049–1054doi:10.1097/ACM.0b013e31822221e7 [PubMed]
11. Torraco RJ (2005) Writing integrative literature reviews: guidelines and examples. Human Res Develop Rev 4: 356–367doi:10.1177/1534484305278283
12. Khoo CSG, Na JC, Jaidka K (2011) Analysis of the macro-level discourse structure of literature reviews. Online Info Rev 35: 255–271doi:10.1108/14684521111128032
13. Rosenfeld RM (1996) How to systematically review the medical literature. Otolaryngol Head Neck Surg 115: 53–63doi:10.1016/S0194-5998(96)70137-7 [PubMed]
14. Cook DA, West CP (2012) Conducting systematic reviews in medical education: a stepwise approach. Med Educ 46: 943–952doi:10.1111/j.1365-2923.2012.04328.x [PubMed]
15. Dijkers M (2009) The Task Force on Systematic Reviews and Guidelines (2009) The value of “traditional” reviews in the era of systematic reviewing. Am J Phys Med Rehabil 88: 423–430doi:10.1097/PHM.0b013e31819c59c6 [PubMed]
16. Eco U (1977) Come si fa una tesi di laurea. Milan: Bompiani.
17. Hart C (1998) Doing a literature review: releasing the social science research imagination. London: SAGE.
18. Wagner CS, Roessner JD, Bobb K, Klein JT, Boyack KW, et al. (2011) Approaches to understanding and measuring interdisciplinary scientific research (IDR): a review of the literature. J Informetr 5: 14–26doi:10.1016/j.joi.2010.06.004
19. Carnwell R, Daly W (2001) Strategies for the construction of a critical review of the literature. Nurse Educ Pract 1: 57–63doi:10.1054/nepr.2001.0008 [PubMed]
20. Roberts PD, Stewart GB, Pullin AS (2006) Are
review articles a reliable source of evidence to support conservation
and environmental management? A comparison with medicine
. Biol Conserv 132: 409–423doi:10.1016/j.biocon.2006.04.034
21. Ridley D (2008) The literature review: a step-by-step guide for students. London: SAGE.
22. Kelleher C, Wagener T (2011) Ten guidelines for effective data visualization in scientific publications. Environ Model Softw 26: 822–827doi:10.1016/j.envsoft.2010.12.006
23. Oxman AD, Guyatt GH (1988) Guidelines for reading literature reviews. CMAJ 138: 697–703 [PMC free article] [PubMed]
24. May RM (2011) Science as organized scepticism. Philos Trans A Math Phys Eng Sci 369: 4685–4689doi:10.1098/rsta.2011.0177 [PubMed]
25. Logan DW, Sandal M, Gardner PP, Manske M, Bateman A (2010) Ten simple rules for editing Wikipedia. PLoS Comput Biol 6: e1000941.doi:10.1371/journal.pcbi.1000941 [PMC free article] [PubMed]
26. van Raan AFJ (2004) Sleeping beauties in science. Scientometrics 59: 467–472doi:10.1023/B:SCIE.0000018543.82441.f1
27. Rosenberg D (2003) Early modern information overload. J Hist Ideas 64: 1–9doi:10.1353/jhi.2003.0017
28. Bastian H, Glasziou P, Chalmers I (2010) Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med 7: e1000326.doi:10.1371/journal.pmed.1000326 [PMC free article] [PubMed]
29. Bertamini M, Munafò MR (2012) Bite-size science and its undesired side effects. Perspect Psychol Sci 7: 67–71doi:10.1177/1745691611429353 [PubMed]
30. Pautasso M (2012) Publication growth in biological sub-fields: patterns, predictability and sustainability. Sustainability 4: 3234–3247doi:10.3390/su4123234
31. Michels C, Schmoch U (2013) Impact of bibliometric studies on the publication behaviour of authors. Scientometrics doi:10.1007/s11192-013-1015-7. In press.
32. Tsafnat G, Dunn A, Glasziou P, Coiera E (2013) The automation of systematic reviews. BMJ 346: f139.doi:10.1136/bmj.f139 [PubMed]
33. Pautasso M, Döring TF, Garbelotto M, Pellis L, Jeger MJ (2012) Impacts of climate change on plant diseases - opinions and trends. Eur J Plant Pathol 133: 295–313doi:10.1007/s10658-012-9936-1

Articles from PLoS Computational Biology are provided here courtesy of Public Library of Science

Ten Simple Rules for Writing a Literature Review

No comments:

Post a Comment