Journal of Evidence-Based Medicine ISSN 1756-5391
ORIGINAL ARTICLE
Systematic reviews: Work that needs to be done and not to be done Clive E Adams1 , Stefanie Polzmacher2 and Annabelle Wolff2 1 2
Cochrane Schizophrenia Group, Institute of Mental Health, University of Nottingham, UK University of Applied Sciences, Ulm, Germany
Keywords systematic reviews; medical Informatics; unified medical language system. Correspondence Clive E Adams, Cochrane Schizophrenia Group, Institute of Mental Health, University of Nottingham Jubilee Campus. Triumph Road, NG7 2TU, UK. Tel: +44(0)115 823 1274 Fax: +44(0)115 823 1274 Email: [email protected]
Abstract Systematic reviews are researches requiring great attention to detail. They may well necessitate considerable investment of effort to ensure relevant data are identified, extracted, synthesized, written up and disseminated. These tasks have already been greatly refined and, in some cases, simplified, by machines. The last two decades have seen remarkable progress in machine-assisted production of reviews – the next two should see much more.
Conflict of interest None. Received 9 September 2013; accepted for publication 20 October 2013. doi: 10.1111/jebm.12072
Introduction In the opening pages of Effective Care in Pregnancy and Childbirth nearly 25 years ago (1), we were reminded of the 19th century wisdom of Lord Rayleigh, the professor of experimental physics in the University of Cambridge. In 1885, in his address to the 54th Meeting of the British Association of the Advancement of Science, he said “If, as is sometimes supposed, science consisted of nothing but the laborious accumulation of facts, it would soon come to a standstill, crushed, as it were, under its own weight. The suggestion of a new idea, or the detection of a law, supersedes much that had previously been a burden upon the memory, and by introducing order and coherence facilitates the retention of the remainder in an available form.[ . . . . .] Two processes are thus at work side by side, the reception of new material and the digestion and assimilation of the old; and as both are essential, we may spare ourselves the discussion of their relative importance. One remark however, should be made. The work which deserves, but I am afraid does not
232
always receive, the most credit is that in which discovery and explanation go hand in hand, in which not only are new facts presented, but their relation to old ones is pointed out.” (2) This, and the clear fact that healthcare was getting crushed under the weight of disordered evidence, led to the production of systematic reviews and meta-analyses (3, 4) and development of methodology by which these were undertaken, eventually, industrially within organizations such as guideline technology appraisal groups and The Cochrane Collaboration (5). Truly remarkable advances have been made in the last 20 years for both production and maintenance of these reviews. In these days of personalized medicine, average, ‘ball-park’, figures on the effects of healthcare are, nevertheless, welcome – but it is important that these data should be as unbiased as possible, accessible and current. The laudable aim to maintain reviews in the light of new evidence or valid criticism (6) is difficult to achieve. It is not in most researchers’ tradition to update their work. Research is undertaken, written up, sent for publication, and by the time it is fully published, the author has moved on to new
C 2013 Chinese Cochrane Center, West China Hospital of Sichuan University and Wiley Publishing Asia Pty Ltd JEBM 6 (2013) 232–235
C. E. Adams et al.
research. The researcher may answer criticisms but is unlikely to re-undertake the original piece of work. Now, with electronic publishing, and especially that followed within The Cochrane Collaboration, maintenance is possible but often painful and not entirely achieved (7). The danger is that this remarkable and innovative Cochrane ‘project’ is seen as lumbering; in reality, it is attempting to undertake a process and fulfill objectives that are truly ambitious. It is easy to trip into criticism of what was seen as the hare and now can be perceived as the tortoise. However, never write off a tortoise. Those undertaking systematic reviews have greatly benefited from electronic innovations that have facilitated the process of systematic reviews. For example, the Cochrane software, RevMan remains a valuable writing tool, helping people to write their reviews, input extracted data, and analyze and present these data graphically (8). RevMan remains innovative software, but software of the late 20th century and there are many ways by which the process of systematic reviews could be facilitated by using additional electronic innovations.
Data Selection Currently systematic reviewers, whether working electronically or from the printed page, manually identify and then select relevant studies. These studies may have been published many times and so there is a risk of double counting that is only somewhat offset by intermittent use of unique study identifiers. Text mining techniques are advanced enough to facilitate this process of study identification. Searching of relevant databases can be undertaken automatically, as can the sifting through the considerable number of records identified by these searches. For example, should a review already exist of ‘Drug x for condition y’, text mining can analyze structure of that review, and the studies and references within the review, and the wording configuration can be ‘understood’ by the machine. These configurations can then be applied to the incoming data and the machine can suggest whether the new reports may be a further record of an already known study, a study of relevance to the review that has not been seen before, or a report likely not to be of importance to that review. Text mining techniques are no longer in their infancy and it would seem that in the near future they could be more routinely applied to the process of systematic review of studies of the effects of healthcare. (9)
Systematic reviews
often undertaken with pencil and paper by highly trained healthcare professionals or researchers. Then, the extracted data are used within the systematic review but the link – tracking extracted data back to the report – is lost. In future it seems likely that there will be repositories of extracted information appropriately marked up, and traceable back to the original reports, for use by both machines and researchers. For example, should the AllTrials initiative be fully successful (12), more participant-level data will be available. It would also seem conceivable that appropriately ‘marked up’ data would be a requirement of publication for any randomized trial. Currently, good journals require compliance with the CONSORT Statement (13, 14) and those data tend to be very structured. It is one short step to ensure that data regarding participants, interventions, outcomes and biases are also structured in ontological mark up to facilitate machine reading. This, of course, does not cope with the large backlog of randomized trials that need to have data extracted. Machines could also help this. As said above, there seems to be no avoidance of the person reading the report of the original study, but in these days of touch screen technology and voice recognition it would seem entirely possible to be able to touch the screen on which the pdf is displayed and voice command where the highlighted data should be extracted to, at the same time annotating the pdf. Early versions of this idea are already available, for example, in the Covidence programme where highlighted text within a PDF automatically extracts into a table (15). It also would not seem beyond imagination for the backlog of randomized trials in any one area to have all relevant data extracted and held centrally, perhaps in some Wikipediatype repository. These data could then be made accessible to researchers, recipients of care, or mechanized systems of review.
Data Assimilation The assimilation process has been semi-automated for a long time. Within many software programmes, simply inputting data leads to default analyses, that, although malleable, nevertheless happens without additional commands. Of course, full control of this process should be possible, but many reviewers have grown used to and are comfortable with automation to some degree, as provided by calculators within machines.
Data Extraction Although text mining techniques have advanced it remains problematic, currently, for machines to automatically extract data (10), although it has been tested and found feasible for some types of information (11). However, the detailed reading of a trial report is still necessary. This is time consuming,
Written up A further bottle neck in the process of systematic reviews is the final write up. Many parts of the report writing lend themselves to use of information technology. Already, some Cochrane groups add in generic methods sections to files for
C 2013 Chinese Cochrane Center, West China Hospital of Sichuan University and Wiley Publishing Asia Pty Ltd JEBM 6 (2013) 232–235
233
Systematic reviews
C. E. Adams et al.
use by review authors (16). There is always some degree of necessary repetition and some re-iteration of tabulated findings in the ‘Results’ sections of reviews. Some readers relate better to text than to graphs and vice versa. Once data are extracted, machines can draft entirely comprehensible text. For example, in the pilot programme (RevMan-HAL) already extracted data are used to generate readable, formatted, text results (17). This approach affords further opportunities to avoid translation – where one language is taken as baseline and translated with all the danger of loss of meaning. Rather than writing originally in, for example, English and then translating into French, German or Arabic, the machine can do the first draft in any language that is required of it. Further opportunities for automatic drafting lie in the structure of systematic reviews. Often PRISMA flow diagrams are obligatory. These structured tables lend themselves to automatic text writing. Furthermore, systematic reviews often list the scales or measures used within the trials for which data are presented. For example, systematic reviews in rheumatology may list the various pain, functioning and quality of life scales that have been used in the included trials. Simple repositories of these scales, do exist (18), and could be used to automatically relate the findings, in intelligible text; thus describing the nature of many of the outcomes recorded in the systematic reviews. With the advent of GRADE and Summary of Finding tables (19), this further structures data and allows the first foray into automatic writing, even of the ‘Discussion’ section of the review. Review authors have to have good understanding of the data, and their clinical and local implications. Nevertheless, machines can draft initial text for consideration by authors and save enormous amounts of time.
Dissemination The epidemiology of many illnesses is broadly known. Clinical pathways, although differing across care-cultures, are reasonably predictable. It seems entirely possible now to map the likely pathways of illness to allow users to information they need off at different points in that pathway (20). For example clinicians may be interested in the best available evidence for a treatment early on in the illness. Recipients of care may be curious as to the longer term adverse effects of a given drug or care package. Interacting with an intelligible and often graphic map of illness could substantially improve linking to evidence and, where carefully crafted systematic reviews are not yet available, simple first draft summaries could be machine-written and made available. As the same data means different things to different people, machine presentation of best available data may have advantages over the ‘fossilised’ written word, since the latter, out of necessity, often contains opinions of a group of authors. If data
234
are clearly presented then readers may well be able to make their own minds up as to their meaning for them. Only a little further down the line, readers will be able to add their own values into consideration of these data and the machine, weighting those values, should be able to produce a first draft personalized best evidence synthesis.
Conclusions In the last 20 years, we have come an extraordinary distance in the process of systematic reviews and the next 20 years should see many exciting and useful developments. It is entirely possible that out of the averages of randomized trials we can facilitate personalized medicine. Machines can help us do this, and do so very swiftly.
Acknowledgements Authors thank Prof Mike Clarke for his invitation to contribute.
References 1. Cochrane AL, Chalmers I, Enkin M, Keirse M. Effective care in pregnancy and childbirth. Oxford, UK: Oxford University Press, 1989. 2. Lord Rayleigh. Address by the Rt. Hon. Lord Rayleigh. Montreal, Canada: John Murray, 1885. p. 3–23. http://www. jameslindlibrary.org/illustrating/records/address-by-the-rt-honlord-rayleigh-in-report-of-the-fifty-f/title_pages (accessed 9 September 2013). 3. Davis JM. Overview: maintenance therapy in psychiatry: I. Schizophrenia. Am erican Jouranl of Psychiatry 1975;132(12): 1237–45. 4. Glass GV. Primary, Secondary, and Meta-Analysis of Research. Educ cation Res earch 1976; 5(10): 3–8. 5. Chalmers I. The Cochrane collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. Annals of the New York Academy of Sciences 1993; 703:156–163–165. 6. The Cochrane Collaboration. http://www.cochrane.org/(accessed 9 September 2013). 7. Brassey J. Liberating the literature – Some interesting, and not so interesting, issues relating to our work at TRIP Database Ltd. Crit. Cochrane Collab. http://blog.tripdatabase.com/2013/04/acritique-of-cochrane-collaboration.html(accessed 20 August 2013). 8. The Cochrane Collaboration. Review Manager. Copenhagen: The Nordic Cochrane Centre. http://ims.cochrane.org/revman(accessed 20 August 2013). 9. Thomas J, McNaught J, Ananiadou S. Applications of text mining within systematic reviews. Research Synthesis Methods 2011; 2(1): 1–14.
C 2013 Chinese Cochrane Center, West China Hospital of Sichuan University and Wiley Publishing Asia Pty Ltd JEBM 6 (2013) 232–235
C. E. Adams et al.
10. Ceci F, Pietrobon R, Gonc¸alves AL. Turning text into research networks: information retrieval and computational ontologies in the creation of scientific databases. PLOS One 2012; 7(1): e27499. 11. M¨uller H-M, Kenny EE, Sternberg PW. Textpresso: an ontology-based information retrieval and extraction system for biological literature. PLOS Biology 2004; 2(11): e309. 12. All Trials Registered, All Results Reported. http://www.alltrials.net/(accessed 4 March 2013). 13. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trial. B ritish Medical Journal 2010; 340: c869. 14. Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database of Systematic Reviews 2012; (11): MR000030.
Systematic reviews
15. Covidence http://www.covidence.org/(accessed 10 Octbertor 2013). 16. Welsh E. Re: The automation of systematic reviews. BMJ http://www.bmj.com/content/346/bmj.f139/rr/628302(accessed 14 Octbertor 2013). 17. Review Manager (RevMan) [Computer program]. Version 5.0. Nottingham, UK: Cochrane Schizophrenia Group. http://szg.cochrane.org/revman-hal (accessed 10 October 2013). 18. The University of Adelaide Library. Psychiatric Rating Scales and Diagnostic Aids. http://www.adelaide.edu.au/library/ guide/med/menthealth/scales.html (accessed 14 Octbertor 2013) 19. Guyatt G, Oxman AD, Akl E, Kunz R, Vist G, Brozek J, et al. GRADE guidelines 1. Introduction-GRADE evidence profiles and summary of findings tables. Jouranl of Clinical Epidemiology 2011; 64(4): 383–94. 20. Health Guidelines, Department of Health. The Map of Medicine – England & Wales. http://healthguides.mapofmedicine.com/ choices/map/schizophrenia2.html (accessed 6 Journary 2011).
C 2013 Chinese Cochrane Center, West China Hospital of Sichuan University and Wiley Publishing Asia Pty Ltd JEBM 6 (2013) 232–235
235
ORIGINAL ARTICLE
Systematic reviews: Work that needs to be done and not to be done Clive E Adams1 , Stefanie Polzmacher2 and Annabelle Wolff2 1 2
Cochrane Schizophrenia Group, Institute of Mental Health, University of Nottingham, UK University of Applied Sciences, Ulm, Germany
Keywords systematic reviews; medical Informatics; unified medical language system. Correspondence Clive E Adams, Cochrane Schizophrenia Group, Institute of Mental Health, University of Nottingham Jubilee Campus. Triumph Road, NG7 2TU, UK. Tel: +44(0)115 823 1274 Fax: +44(0)115 823 1274 Email: [email protected]
Abstract Systematic reviews are researches requiring great attention to detail. They may well necessitate considerable investment of effort to ensure relevant data are identified, extracted, synthesized, written up and disseminated. These tasks have already been greatly refined and, in some cases, simplified, by machines. The last two decades have seen remarkable progress in machine-assisted production of reviews – the next two should see much more.
Conflict of interest None. Received 9 September 2013; accepted for publication 20 October 2013. doi: 10.1111/jebm.12072
Introduction In the opening pages of Effective Care in Pregnancy and Childbirth nearly 25 years ago (1), we were reminded of the 19th century wisdom of Lord Rayleigh, the professor of experimental physics in the University of Cambridge. In 1885, in his address to the 54th Meeting of the British Association of the Advancement of Science, he said “If, as is sometimes supposed, science consisted of nothing but the laborious accumulation of facts, it would soon come to a standstill, crushed, as it were, under its own weight. The suggestion of a new idea, or the detection of a law, supersedes much that had previously been a burden upon the memory, and by introducing order and coherence facilitates the retention of the remainder in an available form.[ . . . . .] Two processes are thus at work side by side, the reception of new material and the digestion and assimilation of the old; and as both are essential, we may spare ourselves the discussion of their relative importance. One remark however, should be made. The work which deserves, but I am afraid does not
232
always receive, the most credit is that in which discovery and explanation go hand in hand, in which not only are new facts presented, but their relation to old ones is pointed out.” (2) This, and the clear fact that healthcare was getting crushed under the weight of disordered evidence, led to the production of systematic reviews and meta-analyses (3, 4) and development of methodology by which these were undertaken, eventually, industrially within organizations such as guideline technology appraisal groups and The Cochrane Collaboration (5). Truly remarkable advances have been made in the last 20 years for both production and maintenance of these reviews. In these days of personalized medicine, average, ‘ball-park’, figures on the effects of healthcare are, nevertheless, welcome – but it is important that these data should be as unbiased as possible, accessible and current. The laudable aim to maintain reviews in the light of new evidence or valid criticism (6) is difficult to achieve. It is not in most researchers’ tradition to update their work. Research is undertaken, written up, sent for publication, and by the time it is fully published, the author has moved on to new
C 2013 Chinese Cochrane Center, West China Hospital of Sichuan University and Wiley Publishing Asia Pty Ltd JEBM 6 (2013) 232–235
C. E. Adams et al.
research. The researcher may answer criticisms but is unlikely to re-undertake the original piece of work. Now, with electronic publishing, and especially that followed within The Cochrane Collaboration, maintenance is possible but often painful and not entirely achieved (7). The danger is that this remarkable and innovative Cochrane ‘project’ is seen as lumbering; in reality, it is attempting to undertake a process and fulfill objectives that are truly ambitious. It is easy to trip into criticism of what was seen as the hare and now can be perceived as the tortoise. However, never write off a tortoise. Those undertaking systematic reviews have greatly benefited from electronic innovations that have facilitated the process of systematic reviews. For example, the Cochrane software, RevMan remains a valuable writing tool, helping people to write their reviews, input extracted data, and analyze and present these data graphically (8). RevMan remains innovative software, but software of the late 20th century and there are many ways by which the process of systematic reviews could be facilitated by using additional electronic innovations.
Data Selection Currently systematic reviewers, whether working electronically or from the printed page, manually identify and then select relevant studies. These studies may have been published many times and so there is a risk of double counting that is only somewhat offset by intermittent use of unique study identifiers. Text mining techniques are advanced enough to facilitate this process of study identification. Searching of relevant databases can be undertaken automatically, as can the sifting through the considerable number of records identified by these searches. For example, should a review already exist of ‘Drug x for condition y’, text mining can analyze structure of that review, and the studies and references within the review, and the wording configuration can be ‘understood’ by the machine. These configurations can then be applied to the incoming data and the machine can suggest whether the new reports may be a further record of an already known study, a study of relevance to the review that has not been seen before, or a report likely not to be of importance to that review. Text mining techniques are no longer in their infancy and it would seem that in the near future they could be more routinely applied to the process of systematic review of studies of the effects of healthcare. (9)
Systematic reviews
often undertaken with pencil and paper by highly trained healthcare professionals or researchers. Then, the extracted data are used within the systematic review but the link – tracking extracted data back to the report – is lost. In future it seems likely that there will be repositories of extracted information appropriately marked up, and traceable back to the original reports, for use by both machines and researchers. For example, should the AllTrials initiative be fully successful (12), more participant-level data will be available. It would also seem conceivable that appropriately ‘marked up’ data would be a requirement of publication for any randomized trial. Currently, good journals require compliance with the CONSORT Statement (13, 14) and those data tend to be very structured. It is one short step to ensure that data regarding participants, interventions, outcomes and biases are also structured in ontological mark up to facilitate machine reading. This, of course, does not cope with the large backlog of randomized trials that need to have data extracted. Machines could also help this. As said above, there seems to be no avoidance of the person reading the report of the original study, but in these days of touch screen technology and voice recognition it would seem entirely possible to be able to touch the screen on which the pdf is displayed and voice command where the highlighted data should be extracted to, at the same time annotating the pdf. Early versions of this idea are already available, for example, in the Covidence programme where highlighted text within a PDF automatically extracts into a table (15). It also would not seem beyond imagination for the backlog of randomized trials in any one area to have all relevant data extracted and held centrally, perhaps in some Wikipediatype repository. These data could then be made accessible to researchers, recipients of care, or mechanized systems of review.
Data Assimilation The assimilation process has been semi-automated for a long time. Within many software programmes, simply inputting data leads to default analyses, that, although malleable, nevertheless happens without additional commands. Of course, full control of this process should be possible, but many reviewers have grown used to and are comfortable with automation to some degree, as provided by calculators within machines.
Data Extraction Although text mining techniques have advanced it remains problematic, currently, for machines to automatically extract data (10), although it has been tested and found feasible for some types of information (11). However, the detailed reading of a trial report is still necessary. This is time consuming,
Written up A further bottle neck in the process of systematic reviews is the final write up. Many parts of the report writing lend themselves to use of information technology. Already, some Cochrane groups add in generic methods sections to files for
C 2013 Chinese Cochrane Center, West China Hospital of Sichuan University and Wiley Publishing Asia Pty Ltd JEBM 6 (2013) 232–235
233
Systematic reviews
C. E. Adams et al.
use by review authors (16). There is always some degree of necessary repetition and some re-iteration of tabulated findings in the ‘Results’ sections of reviews. Some readers relate better to text than to graphs and vice versa. Once data are extracted, machines can draft entirely comprehensible text. For example, in the pilot programme (RevMan-HAL) already extracted data are used to generate readable, formatted, text results (17). This approach affords further opportunities to avoid translation – where one language is taken as baseline and translated with all the danger of loss of meaning. Rather than writing originally in, for example, English and then translating into French, German or Arabic, the machine can do the first draft in any language that is required of it. Further opportunities for automatic drafting lie in the structure of systematic reviews. Often PRISMA flow diagrams are obligatory. These structured tables lend themselves to automatic text writing. Furthermore, systematic reviews often list the scales or measures used within the trials for which data are presented. For example, systematic reviews in rheumatology may list the various pain, functioning and quality of life scales that have been used in the included trials. Simple repositories of these scales, do exist (18), and could be used to automatically relate the findings, in intelligible text; thus describing the nature of many of the outcomes recorded in the systematic reviews. With the advent of GRADE and Summary of Finding tables (19), this further structures data and allows the first foray into automatic writing, even of the ‘Discussion’ section of the review. Review authors have to have good understanding of the data, and their clinical and local implications. Nevertheless, machines can draft initial text for consideration by authors and save enormous amounts of time.
Dissemination The epidemiology of many illnesses is broadly known. Clinical pathways, although differing across care-cultures, are reasonably predictable. It seems entirely possible now to map the likely pathways of illness to allow users to information they need off at different points in that pathway (20). For example clinicians may be interested in the best available evidence for a treatment early on in the illness. Recipients of care may be curious as to the longer term adverse effects of a given drug or care package. Interacting with an intelligible and often graphic map of illness could substantially improve linking to evidence and, where carefully crafted systematic reviews are not yet available, simple first draft summaries could be machine-written and made available. As the same data means different things to different people, machine presentation of best available data may have advantages over the ‘fossilised’ written word, since the latter, out of necessity, often contains opinions of a group of authors. If data
234
are clearly presented then readers may well be able to make their own minds up as to their meaning for them. Only a little further down the line, readers will be able to add their own values into consideration of these data and the machine, weighting those values, should be able to produce a first draft personalized best evidence synthesis.
Conclusions In the last 20 years, we have come an extraordinary distance in the process of systematic reviews and the next 20 years should see many exciting and useful developments. It is entirely possible that out of the averages of randomized trials we can facilitate personalized medicine. Machines can help us do this, and do so very swiftly.
Acknowledgements Authors thank Prof Mike Clarke for his invitation to contribute.
References 1. Cochrane AL, Chalmers I, Enkin M, Keirse M. Effective care in pregnancy and childbirth. Oxford, UK: Oxford University Press, 1989. 2. Lord Rayleigh. Address by the Rt. Hon. Lord Rayleigh. Montreal, Canada: John Murray, 1885. p. 3–23. http://www. jameslindlibrary.org/illustrating/records/address-by-the-rt-honlord-rayleigh-in-report-of-the-fifty-f/title_pages (accessed 9 September 2013). 3. Davis JM. Overview: maintenance therapy in psychiatry: I. Schizophrenia. Am erican Jouranl of Psychiatry 1975;132(12): 1237–45. 4. Glass GV. Primary, Secondary, and Meta-Analysis of Research. Educ cation Res earch 1976; 5(10): 3–8. 5. Chalmers I. The Cochrane collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. Annals of the New York Academy of Sciences 1993; 703:156–163–165. 6. The Cochrane Collaboration. http://www.cochrane.org/(accessed 9 September 2013). 7. Brassey J. Liberating the literature – Some interesting, and not so interesting, issues relating to our work at TRIP Database Ltd. Crit. Cochrane Collab. http://blog.tripdatabase.com/2013/04/acritique-of-cochrane-collaboration.html(accessed 20 August 2013). 8. The Cochrane Collaboration. Review Manager. Copenhagen: The Nordic Cochrane Centre. http://ims.cochrane.org/revman(accessed 20 August 2013). 9. Thomas J, McNaught J, Ananiadou S. Applications of text mining within systematic reviews. Research Synthesis Methods 2011; 2(1): 1–14.
C 2013 Chinese Cochrane Center, West China Hospital of Sichuan University and Wiley Publishing Asia Pty Ltd JEBM 6 (2013) 232–235
C. E. Adams et al.
10. Ceci F, Pietrobon R, Gonc¸alves AL. Turning text into research networks: information retrieval and computational ontologies in the creation of scientific databases. PLOS One 2012; 7(1): e27499. 11. M¨uller H-M, Kenny EE, Sternberg PW. Textpresso: an ontology-based information retrieval and extraction system for biological literature. PLOS Biology 2004; 2(11): e309. 12. All Trials Registered, All Results Reported. http://www.alltrials.net/(accessed 4 March 2013). 13. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trial. B ritish Medical Journal 2010; 340: c869. 14. Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database of Systematic Reviews 2012; (11): MR000030.
Systematic reviews
15. Covidence http://www.covidence.org/(accessed 10 Octbertor 2013). 16. Welsh E. Re: The automation of systematic reviews. BMJ http://www.bmj.com/content/346/bmj.f139/rr/628302(accessed 14 Octbertor 2013). 17. Review Manager (RevMan) [Computer program]. Version 5.0. Nottingham, UK: Cochrane Schizophrenia Group. http://szg.cochrane.org/revman-hal (accessed 10 October 2013). 18. The University of Adelaide Library. Psychiatric Rating Scales and Diagnostic Aids. http://www.adelaide.edu.au/library/ guide/med/menthealth/scales.html (accessed 14 Octbertor 2013) 19. Guyatt G, Oxman AD, Akl E, Kunz R, Vist G, Brozek J, et al. GRADE guidelines 1. Introduction-GRADE evidence profiles and summary of findings tables. Jouranl of Clinical Epidemiology 2011; 64(4): 383–94. 20. Health Guidelines, Department of Health. The Map of Medicine – England & Wales. http://healthguides.mapofmedicine.com/ choices/map/schizophrenia2.html (accessed 6 Journary 2011).
C 2013 Chinese Cochrane Center, West China Hospital of Sichuan University and Wiley Publishing Asia Pty Ltd JEBM 6 (2013) 232–235
235
