Conducting systematic reviews of intervention questions I: Writing the review protocol, formulating the question and searching the literature.

Zoonoses and Public Health

REVIEW ARTICLE

Conducting Systematic Reviews of Intervention Questions I: Writing the Review Protocol, Formulating the Question and Searching the Literature A. M. O’Connor1, K. M. Anderson2, C. K. Goodell1,5 and J. M. Sargeant3,4 1 2 3 4 5

Department of Veterinary Diagnostic and Production Animal Medicine, Iowa State University College of Veterinary Medicine, Ames, IA, USA Zalk Veterinary Medical Library, University of Missouri, Columbia, MO, USA Centre for Public Health and Zoonoses, University of Guelph, Guelph, ON, Canada Department of Population Medicine, Ontario Veterinary College, University of Guelph, Guelph, ON, Canada IDEXX Livestock, Poultry and Dairy, Westbrook, ME, USA

Impacts

• An extensive search to identify the literature relevant to the review question • •

is a key component of a review. Designing an extensive search often requires expertise of information scientists. Inclusion of sources other than peer reviewed journals can be an important source of information.

Keywords: Bibliographic databases; CAB Abstracts; information retrieval; literature searching; MEDLINE Correspondence: A. M. O’Connor. Building 4 Veterinary Medical Research Institute, College of Veterinary Medicine, Iowa State University, Ames, Iowa 50010, USA. Tel.: (515) 294 5012; Fax: (515) 294-1072; E-mail: [email protected] Received for publication September 7, 2013 doi: 10.1111/zph.12125

Summary This article is the fourth of six articles addressing systematic reviews in animal agriculture and veterinary medicine. Previous articles in the series have introduced systematic reviews, discussed study designs and hierarchies of evidence, and provided details on conducting randomized controlled trials, a common design for use in systematic reviews. This article describes development of a review protocol and the first two steps in a systematic review: formulating a review question, and searching the literature for relevant research. The emphasis is on systematic reviews of questions related to interventions. The review protocol is developed prior to conducting the review and specifies the plan for the conduct of the review, identifies the roles and responsibilities of the review team and provides structured definitions related to the review question. For intervention questions, the review question should be defined by the PICO components: population, intervention, comparison and outcome(s). The literature search is designed to identify all potentially relevant original research that may address the question. Search terms related to some or all of the PICO components are entered into literature databases, and searches for unpublished literature also are conducted. All steps of the literature search are documented to provide transparent reporting of the process.

Introduction Systematic reviews follow a prescribed process in clearly delineated steps (European Food Safety Authority (EFSA), 2010). Systematic reviews should always have a protocol. The protocol outlines the approach to each of the steps of the review. At the outset, a review team is assembled to help with the development of the review. The review team plays an active role in all aspects of the review process and writes 28

the detailed protocol. The steps in a systematic review that should be addressed in the protocol are: Step 1: Formulate the review question. Define the question that explicitly describes the scope of the review, including eligibility criteria. Step 2: Search the literature. Search for studies that will be relevant to the scope of the review. Step 3: Relevance Screening. Screen the title and abstract of the research studies identified by the literature search

© 2014 Blackwell Verlag GmbH Zoonoses and Public Health 61 (suppl. 1) (2014) 28–38

A. M. O’Connor et al.

and exclude studies that are not relevant to the review question. Step 4: Data Extraction. Identify and extract key data from relevant studies into paper or electronic data collection forms to allow the comparison of variables of interest across studies. Step 5: Assessment of Bias. Identify methodological features of the relevant studies and categorize them based on methodological quality criteria for consideration of risk of bias in the analysis and interpretation of results. Step 6: Analysis. Step 7: Written Reports. Prepare the review for presentation. Step 8: Knowledge Translation. Interpret the results of the review. This step does not necessarily include making recommendations about adopting an intervention as that step requires identification of values, preferences etc. However, it will include a summation about what the evidence tells about the efficacy of the intervention. This article describes in detail the development of a systematic review protocol, formulation of the review question and search of the literature. The remaining steps are described in subsequent articles in this series. Developing a Systematic Review Protocol Before starting a systematic review, the review team should verify that a prior review that would address the question of interest is not available. If systematic reviews are found that answer the same or a very similar question to that posed by the current review, further review may not be needed. The length of time before a systematic review requires updating will vary based in part of the amount of new data becoming available in the literature. Although no empirical work has been carried out in veterinary medicine, the median ‘survival time’ of systematic reviews of human medical interventions (i.e., the time before new conflicting evidence is published) has been estimated at 5.5 years (Shojania et al., 2007). Assembling the Review Team A critical task at the start of the systematic review process is for the review manager(s) to assemble a review team. Review team size will vary based on the complexity of the review but, at a minimum, a systematic review team must have two members. The review team should include expertise in the topic area being reviewed and expertise in systematic review methodology including information retrieval and knowledge about study design, sources of bias and approaches to analysis (Dudden and Protzko, 2011). Policy makers and funders may also participate and can be vital to ensuring that the focus remains a product of value to the end-user, but agree-

First Steps in a Systematic Review of Intervention Questions

ments about publication of the systematic review, regardless of the results, should be in place a priori. Although the review team should be identified early in the process, the review team should be open to inviting a new reviewer with particular expertise later in the review process, if it becomes apparent based on the research identified that a new area of expertise is needed to assess some aspect of the literature. The roles of each review panel member should be identified early. The review manager’s role is one of the first identified because this individual keeps the review on schedule and manages the steps in a review, including the large amount of data that might be generated. Roles of other members of the review team will depend on their expertise. For instance, a team member with content expertise may be assigned to assist another team member with library expertise in conducting the search. Team members with expertise in research methods might be assigned to conduct the risk of bias assessments. Writing the Systematic Review Protocol As with any research endeavour, good quality research requires the development of, and adherence to, a protocol. A systematic review protocol specifies the plan for the conduct of the review. The protocol is developed with input from all members of the review team. As with any research plan, the protocol is an iterative process and should be completed prior to starting the literature search. As reviewers work through the steps of the protocol, revisions may be necessary. All versions of the protocol should be numbered and dated. Often review authors outline the changes or additions since the last version of the protocol in a foreword or a separate protocol development log. A protocol development log is also useful to keep track of review team comments or suggestions that are not included in the working version of the protocol (Higgins and Green, 2011). The systematic review software Rev-Manâ (RevMan, 2012) provides a template for developing a protocol that ensures the protocol includes all the components of the review and allows for logging or protocol changes. Rev-Manâ is the software used for Cochrane reviews and is free for academic users. Systematic review protocols typically include the following sections: 1. Review team information: Roles and responsibilities for each review panel member. 2. Background and rationale: The justification for the review as well as a preliminary assessment of the relevant literature to address the review question. 3. Review question(s). The review question(s) in the PICO (S) format: population, intervention, comparison, outcome and study design when applicable. It is expected


29


4.

5.

6.

7. 8. 9.


that the final question for the review will be included in the protocol and little or no modification of this component should be made after the review has begun. Changing the review question would be akin to changing the hypothesis after a study has begun. Eligibility criteria. The checklist or series of questions that will be used to remove papers outside the scope of the review through evaluation of abstracts and titles of studies retrieved by the search. It is often worthwhile to include a few examples of abstracts that will be included and excluded to clarify the impact of the proposed screening approach. Literature search strategy. Details of all sources to be searched and detailed search strategies for one or two of the literature databases. Strategies used for web-based search engines should be as explicit as possible. Similarly, this section should explicitly state how any handsearching will be carried out. For example, instead of saying ‘we will search grey literature and conference proceedings’, the protocol should include the exact conference and years to be searched. Include date of search and any language or date restriction applied. Proposed data extraction, bias assessment and analysis strategy. A description of the data that are likely to be collected (e.g. qualitative or quantitative), a description of the outcomes that will be used in the systematic review and the proposed data synthesis and presentation strategy. Details of the criteria used to assess the quality of included studies should be described. Plans for knowledge translation. The strategy for reporting and disseminating the findings to relevant audiences. Project timetable. The proposed timelines for each stage of the review process. Information about funding sources for the systematic review and the role(s) in the systematic review, if any, of the funders. Unfortunately, peer-review funding for systematic reviews is limited and reviewers are often left to search for other sources of funding. The situation is especially challenging in veterinary medicine. If industry funding is secured, this support must be clearly specified in all documents and reports.

Step 1: Formulating the Systematic Review Question For reviews on interventions, the PICO(S) question format should be used to create the question (European Food Safety Authority (EFSA), 2010). PICO(S) stands for population(s), intervention(s), comparison (s), outcome(s) and possibly study design (S). Other approaches, such as PIT (population, index test and target population) or PO (population, outcome) may be used for review questions related to diagnostic test accuracy and estimating descriptive 30

parameters (such as incidence or prevalence), respectively (EFSA, 2010). However, this article will focus on systematic reviews of intervention questions. The question clarifies the scope of the review and outlines the inclusion criteria for the review. Population Population (P) characteristics that vary between studies in veterinary research may include species, age of animals or stage of production for food animals, niche of animal (research, companion, livestock, wildlife) and production system. Defining the population characteristics will aid the reader of the review in determining the scope and therefore the relevance of the review results to the inferring population. For instance, studies evaluating an intervention in pre-weaned animals may not be relevant to the efficacy of that intervention in adult animals. Likewise, differences in agriculture intensity, husbandry practices and species genetics with regard to food animals necessitates acknowledgement of country as an important population variable. Intervention Interventions (I) in veterinary medicine include therapies or preventive practices for a pathogen or disease, performance characteristic and food safety or animal welfare outcomes. Interventions may be administered to individual animals or to groups of animals: for example, at the pen, room, flock or herd level. The defining feature of most interventions is that a specific activity is undertaken with the aim of improving or preventing an outcome. In companion animals, many intervention trials are designed to evaluate the effect of an intervention on a health outcome. In food animals, some interventions are tested solely to evaluate their effect on production or their effect on foodborne pathogens, which may not be associated with adverse health in the food animal species. In performance animals, such as racehorses, trials may solely evaluate the effect of the intervention on outcomes related to performance. The definition of the intervention must be carefully crafted to avoid a review question that is difficult to assess. For example, a review question may describe the intervention of interest as ‘commercially available vaccines’. This may seem appropriate if the aim of the review was only to assess products that producers and veterinarians could access. However, closer examination of the phrase ‘commercially available vaccines’ identifies two concerns: (i) different regulatory processes may result in different vaccines being ‘commercially available’ in different countries, and (ii) it is not clear whether this definition excludes studies published about a vaccine before its commercial release.




Comparison For a review of interventions, it is necessary to clarify the appropriate comparison (C) of interest. For instance, consider a systematic review to evaluate the evidence for the effectiveness of a new antimicrobial product for treatment of a specific disease of interest. If there already are one or more antimicrobial products available to treat the disease of interest, one comparison group for a randomized controlled trial of the new product may be the existing standard of care; an alternative comparison group may be a negative control (placebo). The measures of effect and inferences from either comparison group will be different, and combining data from both comparison groups would not be sensible. Therefore, the review team must decide a priori the comparison(s) of interest and ensure this comparison group is explicitly stated in the systematic review question and recorded in the protocol. Outcome The outcome (O) component of the PICO question should be described as a quantifiable characteristic. The review should address outcomes that are meaningful to individuals making decisions. Initial literature searches and the subject experts on the panel will help identify the outcomes used in primary studies and what is important to end-users. Outcomes can be measured objectively (e.g. mortality, number of behaviour events per min, average daily gain) or subjectively (e.g. lung lesion scores). Outcomes also may be direct measures of adverse health effects, such as lesion size, morbidity or survival time, or may be surrogate measures of the disease of interest, such as titre change following vaccination or blood pressure. Subjectively measured outcomes often have clinical relevance, but steps should be taken to

ensure that the outcomes are comparable across studies. Surrogate measures are often used because they tend to be objective and easy to measure. However, if surrogate measures are used, there should be a clearly established relationship between the surrogate measure and a clinically meaningful health outcome. To make comparisons across studies in a systematic review, selected outcome measures are often converted to common summary measures some of which are described in Table 1 (Deeks et al., 2011). In animal populations, these measures may represent measurements in individual animals, group averages of measures taken on individual animals or measurements taken at the group level. Some trials measure the outcome by part within an animal (e.g. comparing results in one limb to another or one teat in the udder to another). The review should clarify which outcome measuring approach is of interest. For example, consider a trial in swine where the outcome of interest is weight gain. Weight gain could be measured at the individual level by weighing each pig at the start and again at the end of the study. At the group level, the mean weight of the individual pigs at the beginning and end of the trial could be calculated for all of the animals within a group (group average measurement). Alternatively, weight gain for a group could be calculated by weighing all of the pigs on a group scale at the beginning and end of the trial (group level measurement). For each of these situations, the outcome measure differs. For the group level measurement, the weight gain of any given individual pig is not known, but the group weight gain can be measured. In this situation, the number of pigs included in the group may be different due to mortality. If the review protocol does not clarify if all or just one outcome is relevant, issues will arise during the screening and data extraction.

Table 1. Common outcome types and related summary measures often used to compare groups Outcome type

Related summary measure

Continuous outcome for example, weight, area, blood concentrations Binary outcomes for example, alive or dead, clinical illness or no clinical illness Ordinal outcome for example, mild, moderate, severe disease

Mean difference, standardized mean difference

Counts and rates for example, number of disease events during a specified period of observation Time-to-event (survival) outcomes for example, time to death from onset of disease

Risk difference, risk ratio, odds ratio, number needed to treat

Proportional odds ratios, (other options include creating a dichotomized outcome from the categories or, treat as a continuous variable if a large number of categories) [Higgins and Green, 2011] Rate ratio, rate difference

Hazard ratio


31



Step 2: Conducting a Comprehensive Literature Search Having refined the review question with PICO components in Step 1, the next step is to conduct a search to identify studies that can inform the focused review question. The aim of literature searching in a systematic review is to find as many potentially relevant studies as possible to minimize bias (Counsell, 1997). There is always a trade-off between specificity (narrow searching that may miss relevant studies) and sensitivity (broader searching that may retrieve too much irrelevant information). As the aim of the literature search is to identify all potentially relevant research, high sensitivity is emphasized at this stage. Not all relevant research is published in peer-reviewed journals, so searches of other sources, often referred to as grey literature, are important (McManus et al., 1998; Higgins and Green, 2011). All searching should be reported transparently. As with other types of primary research, enabling replication and validation of the results of the search is seen as a means of facilitating a thorough critique and peer review. Furthermore, a transparent search allows end-users to understand potential sources of bias in the search that could be manifest in the review outcomes. Additionally, transparent reporting of searches expedites updating of the review. Successful search strategy design involves knowledge of databases, indexing and database structures. Hence, successful search strategies typically involve experienced information specialists (McGowan and Sampson, 2005). Many academic libraries (such as those affiliated with veterinary colleges) have librarians with specialized training in systematic review search strategies. The search strategy should be developed as an iterative process. Initially, a search should be performed and the results discussed with the review team and content experts to ensure that all potentially relevant search terms are included. An advantage of developing a focused review question is that PICO(S) components can be used to develop appropriate search strategies; however, using all the PICO components may be unnecessary if the body of literature is small or if adding terms risks excluding too many studies of interest. Constructing a Search Strategy Once a draft search strategy has been devised based on all or some of the PICO components, the search should be trialled on a single electronic database with a restricted publication date interval to assess the fitness of the search strategy to identify relevant articles. This enables potential refinements to be made at an early stage and may save time 32

later. These refinement processes should be briefly documented in the protocol to prevent repetition of search approaches that have previously been explored. During the refinement process, additional terms from the PICO(S) may be added to increase the specificity of the search if the initial set of citations is too large, or expand the sensitivity, if the initial set is too small. For example, a review team may decide to add O or S terms to the search if the search currently only included the P and I components. In human medicine, most systematic reviews for established pharmacologic interventions only consider evidence from randomized controlled trials (RCTs), and the use of different study design filters for electronic literature searches to find RCTs has been studied (McKibbon et al., 2009). However, clinical queries designed for the human literature may not be effective for finding veterinary literature (Murphy, 2002, 2003). Electronic Bibliographic Databases The most efficient means of identifying potentially relevant studies is through the use of health-related or agricultural electronic bibliographic databases. The following databases should be considered for systematic reviews in veterinary medicine: MEDLINE/PubMed, AGRICOLA, CAB Abstracts, Web of Science, and Scopus. It is not intended that all reviews will search all databases. Publications are available that describe the journal coverage of each database and coverage for various veterinary topics (Grindlay et al., 2012). Given the coverage of topics and journals indexed in the literature databases, a combination of MEDLINE/PubMed and CAB Abstracts are recommended as the minimum standard in livestock animal health-associated systematic reviews. Considerations When Searching Electronic Databases Although an information specialist should be involved in designing and conducting the search, the following tips for designing search strings are included. Table 2 and 3 provide examples of searches; Table 4 shows the importance of searching more than one literature database. Search strategies are devised by developing strings of terms, linked together with Boolean operators (AND/OR/ NOT). The Boolean operator ‘AND’ is used to combine components to ensure that all concepts (e.g. PICO for an intervention question) appear in the record. The Boolean operator ‘OR’ is used within a component to ensure that all records with at least one of the specified terms are identified. The Boolean operator ‘NOT’ can also be used to exclude records from a search. For example, a search to find studies of interventions in swine may include the following population terms: (pig OR piglet OR hog OR swine OR




Table 2. Search terms and number of citations retrieved from PubMed: Date of Search 31 May 2013 Set

Terms

Results

1 2

‘ruminants’[MeSH] OR ‘ruminants’[All Fields] OR ‘cattle’[All Fields] OR ‘sheep’[All Fields] OR ‘goat’[All Fields] OR ‘goats’[All Fields] ‘probiotics’[MeSH] OR ‘vaccines’[MeSH] OR ‘anti-bacterial agents’[MeSH] OR ‘bacteriophages’[MeSH] OR ‘food additives’[MeSH] OR ‘probiotic*’[All Fields] OR ‘vaccin*’[All Fields] OR ‘antibiotic*’[All Fields] OR ‘bacteriophage*’[All Fields] OR ‘additive*’[All Fields] ‘Escherichia coli o157’[MeSH Terms] OR (‘Escherichia’[All Fields] AND ‘coli’[All Fields] AND ‘O157’[All Fields]) OR ‘E coli O157’[All Fields] OR ‘VTEC’[All Fields] 1 AND 2 AND 3

435 793 617 665

3 4

7891 391

Table 3. Search terms and number of citations retrieved from CAB Abstracts via the CAB Direct platform: Date of Search 31 May 2013 Set

Terms

Results

Keyword search 1 ruminants OR cow OR cattle OR sheep OR goat 2 probiotic OR lactobacillus OR streptococcus OR enterococcus OR bifidobacterium OR saccharomyces OR vaccin* OR antibiotic OR antibacterial OR bacteriophage OR (food additives) OR (sodium chlorate) 3 Escherichia coli O157 OR E coli O157 4 1 AND 2 AND 3 Combined keyword and controlled vocabulary search 1 od:(ruminants) OR (ruminant OR cow OR cattle OR sheep OR goat) 2 subject:(probiotics OR antibiotics OR vaccines OR food additives) OR (lactobacillus OR streptococcus OR enterococcus OR bifidobacterium OR saccharomyces OR vaccin* OR antibacter* OR bacteriophage OR sodium chlorate) 3 od:(Escherichia coli O157) OR (E coli O157 OR VTEC) 4 1 AND 2 AND 3

Table 4. Duplication between PubMed and CAB searches Database(s)

Results

PubMed and CAB (before removing duplicates) PubMed (Table 2) CAB (Table 3) PubMed and CAB (after removing duplicates) Citations in both PubMed and CAB Citations in PubMed only Citations in CAB only

819 391 428 622 197 200 225

porcine OR sow OR boar OR gilt) NOT (guinea pig OR wart hog). The ‘NOT’ Boolean operator should be used with caution to avoid excluding relevant citations that include the excluded word. For instance, if you were interested in citations that described an event on-farm, but used ‘NOT’ abattoir’ in your search string, you could exclude citations that evaluated the event ‘on-farm and in abattoirs’. Tables 2, 3 and 4 illustrate the impact of combining population search strings, intervention search strings and outcome search strings. To create a well-structured search, it is necessary to understand the indexing used by the database. The US National Library of Medicine (NLM) controlled vocabulary used for indexing articles for MEDLINE/PubMed is called Medical Subject Headings (MeSH). MeSH terminology provides a consistent way to retrieve information that may

899 041 474 781 6868 408 899 324 449 686 7296 428

use different terminology for the same concepts. For example, an article using the term ‘bovine’ and one using the term ‘cow’ will both be indexed with the MeSH term ‘Cattle’. Indeed, using the term ‘bovine’ in PubMed results in an automatic search of the following terms – ‘cattle’[MeSH Terms] OR ‘cattle’[All Fields] OR ‘bovine’[All Fields] with over 350 000 hits. Also, PubMed automatically explodes search terms according to the MeSH hierarchy of broader and narrower terms. A search of ‘ruminants’[MeSH] includes all narrower terms beneath ruminants, such as antelopes, buffaloes, camels, cattle and goats. A database of MeSH terms and tutorials about using MeSH terms for PubMed are available at the NLM website (http://www. ncbi.nlm.nih.gov/mesh). CAB Abstracts, a comprehensive veterinary literature database produced by the Centre for Agricultural Bioscience International (CABI), has an extensive thesaurus and uses descriptors (DS), organism descriptors (OD) and topic searches (TS) in a similar but not exact way as MEDLINE/PubMed uses MeSH terms. For comprehensive searching, a combination of keywords and controlled vocabulary terms is recommended (see Table 3). When defining search terms, consider the use of the singular and/or plural versions of the search terms as well as different spellings and acronyms for the search terms: for example, mice and mouse, faecal and fecal, verotoxigenic Escherichia coli and VTEC. Truncation (the word stem following by a symbol, often the *) can also be useful: for


33



example, ‘pig*’ will identify pigs and piglet. However, ‘pig*’ will also identify ‘pigment’ and ‘pigeon’. Additionally, the use of truncation in PubMed turns off the automatic term mapping. A search of ruminant* would not map to ‘ruminant’ [MeSH] and would, therefore, not retrieve additional terms from the MeSH hierarchy such as cattle and goats. More and more often, databases automatically perform a single/plural search, such that a search of ‘pig’ will automatically capture ‘pigs’. It is easy to verify whether this is true for the database being used by conducting different searches for singular and plural terms or consulting an information specialist. It is also possible to limit searches to specific years. For example, E. coli O157 emerged as an important human pathogen in the early 1980s. Thus, the use of interventions in cattle to prevent the shedding of E. coli O157 is a fairly recent procedure. Therefore, the literature search for interventions with this outcome could be restricted to papers published after 1980. Similarly, if a follow-up search is performed after completion of a systematic review, the date since last search can be included to identify only articles published since the date of the original search. A language filter may be used to exclude non-English references, which can be time-consuming and costly to translate. However, it is more common to use no language screen and later decide which articles to translate when the volume of literature in different languages is determined. Any language restrictions should be outlined in the protocol. Restriction of language may introduce bias and decrease the precision of the systematic review although a retrospective analysis from the human literature found that excluding trials published in languages other than English generally had little effect on summary treatment effect estimates (Juni et al., 2002). The importance of non-English language trials is difficult to reliably predict for individual systematic reviews. For questions related to livestock, there may be significant differences among countries in production systems, intervention approaches and, in some cases, animal genetics. Therefore, if the review question is related to a population, intervention or outcome primarily of relevance in English-speaking countries, language bias may not be as much of an issue. Additional Methods to Identify Studies Electronic database searches may not identify all of the relevant literature. Therefore, additional methods are recommended, including handsearching of journals and conference proceedings, checking reference lists and identifying unpublished data. Recent methodological work shows the importance of author tracing, using personal contacts and other methods to identify studies (Greenhalgh and Peacock, 2005). However, these methods can be time-con34

suming and may or may not result in the identification of new useable information. Therefore, decisions as to which additional sources will be used, and timelines for collecting this information should be considered a priori as part of the protocol development. 1. Handsearching of journals: Some journals may not be included in the electronic databases, and some issues, particularly older issues, might not be fully indexed. In such situations, handsearching of the journal can be utilized. Handsearching involves an examination of the contents of a journal issue to identify all eligible reports whether they appear in articles, news columns, editorials, letters or other text. Handsearching should be documented using the full title of the journal and the first and latest years searched. Issues not searched because of missing journal issues should be recorded. 2. Handsearching of conference proceedings: Although some conference proceedings are indexed, many are not, and therefore conference proceedings that may contain relevant information should be identified a priori. The indexes of these conferences should then be handsearched for relevant abstracts. In many cases, the abstracts may contain sufficient detail to enable relevance screening. However, not all conference proceedings provide sufficient detail to allow comprehensive data extraction and assessment of biases. In such cases, it may be necessary to contact the authors for additional details of the study which can be costly, time-consuming and sometimes fruitless. In some instances, authors may be willing to provide more detailed information about the study design and results. However, issues such as pending publication or patents may preclude investigators from sharing this information. 3. Web search engines: Terms used in the literature searches can be entered into web-based search engines such as Google, Google Scholar and Yahoo. Because this approach may lead to an overwhelming number of ‘hits’, a more refined search strategy may be required. It is important to know the limitations of web-based search engines, such as difficulties reproducing results and difficulties exporting results to reference management software (Boeker et al., 2013). 4. Investigators: Some completed studies are never published, and even when they are published, there is often a lag time from completion of a study to publication. The association between significant results and publication has been well documented and it is therefore important to search for unpublished studies to minimize bias in the systematic review (Dwan et al., 2008). Unfortunately, it is difficult to obtain information from research that has been completed but never published. Informal channels of communication, such as contacting researchers in the area of the review, can sometimes be the only means of



identifying unpublished data. However, they may also introduce a bias. To maintain the reproducibility of the search strategy, specific criteria for identifying and contacting investigators should be established a priori and documented in the final report with a list of all investigators contacted by the review team. 5. Currently funded studies: For some questions, the review team may have good reason to believe relevant studies include studies in progress. Databases that may contain information on this source of information include study registries. Databases recommended for this purpose are described in Appendix 1. 6. Theses and dissertations: Another way to find unpublished literature and ongoing research is by searching databases that contain information on theses and dissertations. Databases recommended for this purpose are described in Appendix 1. 7. Previously conducted systematic and narrative reviews: During the search process, systematic and narrative reviews on related topics should be identified. The referenced studies from those reviews should be assessed for eligibility for inclusion in the current review. Assessing the Search Strategy The reference lists of selected relevant articles and/or reviews should be examined to ensure that the search strategy has identified all potentially relevant studies, providing a means of validating the literature search. Following up references from one article is another means of identifying studies for possible inclusion in a review. Finding the ten most recent articles published on the topic and, checking the references back against articles identified by other methods, and finding the ten oldest papers on the topic and using a citation search from that date forward is method of checking the search strategy that we have found to be useful. Managing the Results of the Literature Search It is recommended that reviewers use a reference management system (e.g. Endnote, Reference Manager, RefWorks) to manage the retrieval of studies (Higgins and Green, 2011). Import filters differ between databases and reference management software, and care should be taken to read the user guide when importing citations from electronic databases. Failure to use the correct filter may result in loss of information that is needed for the review, such as the language information. During the process of conducting the search for information, it is not uncommon to identify several reports from a single research study. Due to the overlap in literature databases, the same journal citation may be retrieved a number


of times (see Table 4). Most reference management software programs have built-in features to identify and remove duplicate records. Note that in some cases, only identical records will be identified, and records from the same study with different citation formats will not always be captured as duplicates. Duplicate reports also occur when the same material is published in different forms. This may occur when study reports are published in conference proceedings and subsequently as a journal article; when results are reported on an Internet site or by the agencies that funded the research; or when different aspects of a study are published as separate manuscripts. Initially, all reports that meet the inclusion criteria should be included and cross-referenced. However, results from the same study should only be included once in the systematic review analysis. Generally, the most comprehensive report of the results is used in the systematic review analysis. Step 3: Recording the Search Results The search strategy developed should be included as an appendix to the systematic review protocol. Depending on journal requirements, systematic review publications should include details of the search string or the search string should be made available on request from the authors (Liberati et al., 2009). In addition to specifying the terms used, the name of the database, the specific interface used, the date of the search, the years covered by the database and limitations applied in the search should be clearly documented. There are evidence-based guidelines for reviewing electronic search strategies (Sampson et al., 2009). The PRISMA document also describes approaches to reporting the literature search in a manner that is repeatable (Moher et al., 2009). An Example of Question Development for a Systematic Review In this example, we illustrate the process of turning a general review concept into a PICO question. A systematic review team was interested in the impact of pre-harvest interventions on E. coli O157 in the faeces of ruminants (Sargeant et al., 2007). However, as worded, this question is not constructed in a PICO format, and it would be difficult to construct specific search terms to identify the relevant literature. Thus, the review panel revised the broad question into a focused one, ‘Do pre-harvest interventions (probiotics, vaccines, antibiotics, sodium chlorate, bacteriophages) reduce faecal shedding of E. coli O157 in weaned domestic ruminants compared to faecal shedding of E. coli O157 in weaned domestic ruminants receiving no pre-harvest intervention?’


35



In this example, the population (P) was defined as weaned domestic ruminants. This allowed the inclusion of multiple domestic species that may shed E. coli O157 in their faeces, including: dairy cattle, beef cattle, sheep and goats. In addition, the review panel believed pre-weaned ruminants were not relevant to contamination in animals marketed as a food source. By further focusing the population to ‘weaned’ animals, the review excluded studies in pre-weaned ruminants. There are several interventions (I) described in the review question: probiotics, vaccines, antibiotics, sodium chlorate and bacteriophages. This approach clarifies the vague term ‘pre-harvest control’. This clarification was necessary for identifying appropriate primary research studies because the search term ‘pre-harvest control’ is not likely to be an indexed term. The focused question includes the comparator of no intervention, as this was the current norm when the review was conducted. In other reviews, this may be an inappropriate comparator, if there is a current standard of practice in terms of interventions. The outcome for the focused question is clarified as ‘reduced faecal shedding’ which is more specific than original ‘impact’ included in the broad question. Further, inclusion of the wording ‘faecal shedding’ in the systematic review question excluded an evaluation of other indicators of E. coli O157, such as serological titres, as well as nonbacterial outcomes such as cattle performance. While this systematic review question considered multiple pre-harvest interventions, more specific questions could have been used. For example, ‘Does feeding of Lactobacillus strains [specific types of probiotic bacteria] within 1 month of marketing reduce faecal shedding of E. coli O157 in feedlot beef cattle compared to feedlot cattle receiving no intervention?’ While this specific question would have allowed for a focused search, it may have resulted in too few studies for meaningful interpretation. The review panel needs to balance focusing the question with the available information and the review goals. An Example of Developing the Search String In the systematic review of pre-harvest interventions to reduce the shedding of E. coli O157 in the faeces of weaned domestic ruminants described earlier, the search strategy was conducted in two phases: initially to identify studies pertaining to probiotic use, and then to identify studies related to the remaining interventions. The identification of potentially relevant original research began by compiling a comprehensive list of search terms. As the comparator was ‘no intervention’, the search terms related to only three components of the PICO question: population (live-post-weaned ruminants), intervention 36

(probiotics, vaccines, antibiotics, sodium chlorate, bacteriophages) and outcome (reduction of E. coli O157 in faeces). Search terms within each component were combined using ‘OR’ with components linked using ‘AND’. Due to the low percentage of abstracts that were relevant in the probiotic search, the search terms were modified for the remaining interventions. The main modification between the searches, other than different intervention terms, was to exclude ‘Escherichia coli’ as an independent search term. In the first search, inclusion of this term resulted in a large percentage of abstracts that were not relevant to the review question. There are many types of E. coli, and illness due to various non-O157 strains is common, particularly in young animals. By excluding the independent term ‘Escherichia coli’ and including ‘O157’, ‘VTEC’, and other terms related to the O157 strain, fewer non-relevant articles were identified. Examples of search strings related to this example are presented in Table 2 and 3. Summary Here, we have discussed development of a systematic review protocol and the first two steps of a systematic review of an intervention question. A systematic review protocol should be developed a priori and be used to guide the review. A focused review question, including the population, intervention, comparison and outcome (PICO), should be developed and used to identify the keywords used in the literature search. The aim of the literature search is to identify all potentially relevant research that may address the systematic review question. The next steps in a systematic review include relevance screening, data extraction, quality assessment, preparing for qualitative and quantitative analysis (if appropriate) and dissemination of results. These activities are reviewed in subsequent articles. Acknowledgements and Funding Support The authors thank Annette Wilkins for assistance with this manuscript. Partial funding was obtained from the Laboratory for Foodborne Zoonoses, Public Health Agency of Canada and the Canadian Institutes of Health Research (CIHR) Institute of Population and Public Health/Public Health Agency of Canada Applied Public Health Chair. References Boeker, M., W. Vach, and E. Motschall, 2013: Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC Med. Res. Methodol. 13, 131.



Counsell, C., 1997: Formulating questions and locating primary studies for inclusion in systematic reviews. Ann. Intern. Med. 127, 380–387. Deeks, J. J., J. P. T. Higgins, and D. G. Altman, 2011: Chapter 9: Analysing data and undertaking meta-analyses. In: Higgins, J. P. T., and S. Green (eds), Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration. Available at: www. cochrane-handbook.org. Dudden, R. F., and S. L. Protzko, 2011: The systematic review team: contributions of the health sciences librarian. Med. Ref. Serv. Q. 30, 301–315. Dwan, K., D. G. Altman, J. A. Arnaiz, J. Bloom, A. W. Chan, E. Cronin, E. Decullier, P. J. Easterbrook, E. Von Elm, C. Gamble, D. Ghersi, J. P. Ioannidis, J. Simes, and P. R. Williamson, 2008: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3, e3081. European Food Safety Authority (EFSA). 2010: Application of systematic review methodology to food and feed safety assessments to support decision making. EFSA J., 8, 1–84 Article 1637. Greenhalgh, T., and R. Peacock, 2005: Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 331, 1064– 1065. Grindlay, D. J., M. L. Brennan, and R. S. Dean, 2012: Searching the veterinary literature: a comparison of the coverage of veterinary journals by nine bibliographic databases. J. Vet. Med. Educ. 39, 404–412. Higgins, J. P. T., and S. Green (eds), 2011: Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration. Available at: www.cochrane-handbook.org. Juni, P., F. Holenstein, J. Sterne, C. Bartlett, and M. Egger, 2002: Direction and impact of language bias in meta-analyses of controlled trials: empirical study. Int. J. Epidemiol. 31, 115– 123. Liberati, A., D. G. Altman, J. Tetzlaff, C. Mulrow, P. C Gøtzsche, J. P. Ioannidis, M. Clarke, P. J. Devereaux, J. Kleijnen, and D. Moher, 2009: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 6, e1000100. McGowan, J., and M. Sampson, 2005: Systematic reviews need systematic searchers. J. Med. Libr. Assoc. 93, 74–80. McKibbon, K. A., N. L. Wilczynski, R. B. Haynes, and T. Hedges, 2009: Retrieving randomized controlled trials from Medline: a comparison of 38 published search filters. Health. Info. Libr. J. 26, 187–202. McManus, R. J., S. Wilson, B. C. Delaney, D. A. Fitzmaurice, C. J. Hyde, R. S. Tobias, S. Jowett, and F. D. R. Hobbs, 1998: Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. BMJ 317, 1562–1563.


Moher, D., A. Liberati, J. Tetzlaff, D. G. Altman, and P. Group, 2009: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J. Clin. Epidemiol., 62, 1006–1012. Murphy, S. A., 2002: Applying methodological search filters to CAB abstracts to identify research for evidence-based veterinary medicine. J. Med. Libr. Assoc. 90, 406–410. Murphy, S. A., 2003: Research methodology search filters: are they effective for locating research for evidence-based veterinary medicine in PubMed? J. Med. Libr. Assoc. 91, 484–489. RevMan, 2012: Review Manager (RevMan) [Computer program]. Version 5.2. The Cochrane Collaboration, The Nordic Cochrane Centre, Copenhagen. Sampson, M., J. McGowan, E. Cogo, J. Grimshaw, D. Moher, and C. Lefebvre, 2009: An evidence-based practice guideline for the peer review of electronic search strategies. J. Clin. Epidemiol. 62, 944–952. Sargeant, J. M., M. R. Amezcua, A. Rajic, and L. Waddell, 2007: Pre-harvest interventions to reduce the shedding of E. coli O157 in the faeces of weaned domestic ruminants: a systematic review. Zoonoses Public Health 54, 260–277. Shojania, K. G., M. Sampson, M. T. Ansari, J. Ji, S. Doucette, and D. Moher, 2007: How quickly do systematic reviews go out of date? A survival analysis. Ann. Intern. Med. 147, 224– 233.

Appendix 1 Commonly used and available sources of information that may be used in the literature search of a systematic review Electronic bibliographic databases MEDLINE, the premier citation database of the US National Library of Medicine, indexes over 5400 journals in 39 languages. MEDLINE is the largest component of NLMS’s PubMed, a freely available interface that includes over 22 million citations from MEDLINE and other life science journals. MEDLINE is also available through a number of commercial vendors. AGRICOLA (AGRICultural OnLine Access) is a bibliographic database created by the US National Agricultural Library. Freely available at http://agricola.nal.usda.gov/ or through several commercial vendors, AGRICOLA includes publications and resources related to agriculture and allied disciplines including animal and veterinary sciences. CAB Abstracts, produced by the Centre for Agricultural Bioscience International (CABI), is a comprehensive veterinary literature database with an extensive archive. CAB includes citations to journal articles, book chapters and conference proceedings. CAB Abstracts is available from a number of commercial vendors.


37



EMBASE (http://www.embase.com/search) is a biomedical and pharmacological bibliographic database produced by Elsevier, which provides access to up-to-date citations and abstracts from biomedical and drug literature. While the scope of EMBASE may be larger than PubMed, its strength is drug information. Access to EMBASE requires an academic or individual account. EMBASE records are included in the Scopus database. Scopus is available by subscription on Elsevier’s SciVerse platform. It contains 40 million citations from over 16 000 journals and includes MEDLINE and EMBASE records. Scopus covers all subjects and disciplines and provides times cited information. Web of Science is available by subscription on the Thomson Reuters (formerly ISI) Web of Knowledge platform. Web of Science indexes over 12 000 journals, covers all disciplines and includes the Science Citation Index. Cross-platform searching is provided by some vendors and offer multiple database searching via one platform interface. For example and depending on institutional subscriptions, the Web of Knowledge platform can be used to search Web of Science, MEDLINE and CAB Abstracts simultaneously. Elsevier’s SciVal allows searching across Scopus and ScienceDirect journals. Keep in mind that a specialized search feature of an individual database might not be available when searching across multiple databases. Full-text searching: The literature databases above are considered ‘abstracting and indexing’ resources; therefore, searches are based on the available abstracts, keywords and other fields in the record—and not the full text of the article. Full-text searching is available through journal packages such as ScienceDirect (Elsevier), IngentaConnect, Wiley Online Library and Highwire Press (Stanford University). Google Scholar will also search the full text of many articles, depending on publisher agreements. While full-text searching can retrieve many more citations than a search of a literature database like PubMed, the results can be overwhelming and off-target. Study registries: ClinicalTrials.gov is a registry of federally and privately supported clinical trials conducted in the United States and around the world. This website is a service of the US National Institutes of Health (http://www.clinicaltrials.gov). The Current ControlledTrials website (available at http://www.controlled-trials. com) includes a registry of international trials. This website is managed by the Science Navigation Group, a group of independent biomedical publishing companies. The International Clinical Trials Registry Platform (available at

38

http://www.who.int/ictrp/en/) is an international registry of all intervention trials managed by the World Health Organization. This platform provides data from several of the large registries of clinical trials and the database is updated regularly. The Inventory of Canadian Agri-Food Research (ICAR, available at http://publications.gc.ca/pub?id=275013&sl=0) contains detailed information on research projects in agriculture, food, human nutrition, aquaculture and related areas of biotechnology. The United States Department of Agriculture provides information on national and international projects involving the Agricultural Research Service (ARS) [TEKTRAN], the US Department of Agriculture’s chief scientific research agency (USDA)[CRIS]. TEKTRAN (available at http://www.ars.usda.gov/services/tektran.htm) contains published or soon-to-be-published articles of recent research results from the ARS. This database also contains scientists’ contact information and additional links to their respective research projects and other publications (USDA). Current Research Information System (CRIS, available at http://cris.csrees.usda.gov/) is the USDA’s documentation and reporting system for ongoing and recently completed research projects in agriculture, food and nutrition and forestry. Projects are conducted or sponsored by USDA research agencies, state agricultural experiment stations, the state land grant university system, other cooperating state institutions and participants in a number of USDA-administered grant programs (USDA). Theses and dissertations: The National Library of Canada provides a searchable Thesis Canada Portal for theses and dissertations from Canadian universities (http://www.phddata.org/). The website www.phdData.org is run by doctoral students and provides an index of doctoral dissertations in progress (phdData.org). Access to many European veterinary dissertations is available through ViFa Vet (http://elib.tiho-hannover.de/virtlib/dat1860.html). DART-Europe provides electronic access to European research theses (www.dart-Europe.eu). The ProQuest Dissertations and Theses A&I database has over 700 active academic institution publishing partners. This database is available through most academic libraries (http://www.proquest.com/en-US/products/dissertations/). The Networked Digital Library of Theses and Dissertations (NDLTD) provides open-access to electronic theses and dissertations (http://www.ndltd.org/).


Conducting systematic reviews of intervention questions III: Synthesizing data from intervention studies using meta-analysis.

Conducting a successful systematic review of the literature, part 2.

Conducting the literature review.

Conducting a successful systematic review of the literature, part 1.

Conducting systematic reviews of intervention questions II: Relevance screening, data extraction, assessing risk of bias, presenting the results and interpreting the findings.

Improving the uptake of systematic reviews: a systematic review of intervention effectiveness and relevance.

Literature reviews vs systematic reviews.

Guidance for conducting systematic scoping reviews.

Impacts of structuring the electronic health record: a systematic review protocol and results of previous reviews.

Question the questions on CPAF.

The effects of asking a fertility intention question in primary care settings: a systematic review protocol.

The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching.

Getting started in research: research questions, supervisors and literature reviews.

Increasing the quantity and quality of searching for current best evidence to answer clinical questions: protocol and intervention design of the MacPLUS FS Factorial Randomized Controlled Trials.

An overview of conducting systematic reviews with network meta-analysis.

Searching for qualitative research for inclusion in systematic reviews: a structured methodological review.

What value is the CINAHL database when searching for systematic reviews of qualitative studies?

Factors contributing to chronic ankle instability: a protocol for a systematic review of systematic reviews.

Tutorial for writing systematic reviews for the Brazilian Journal of Physical Therapy (BJPT).

Family-based interventions for substance misuse: a systematic review of systematic reviews--protocol.

The characteristics and effectiveness of Question Prompt List interventions in oncology: a systematic review of the literature.

The role of intervention mapping in designing disease prevention interventions: A systematic review of the literature.

Writing book reviews.

Does the Finnish intervention prevent obstetric anal sphincter injuries? A systematic review of the literature.