ORIGINAL ARTICLE
Evaluation of primary open-angle glaucoma clinical practice guidelines Annie M. Wu, BA,*,†,‡ Connie M. Wu, BS,*,†,‡ Benjamin K. Young, BA, MS,*,†,‡ Dominic J. Wu, BS,*,†,‡ Allison Chen, BA,*,†,‡ Curtis E. Margo, MD, MPH,§ Paul B. Greenberg, MD*,†,‡ ABSTRACT ● RÉSUMÉ Objective: To evaluate the methodologic quality of 3 primary open-angle glaucoma (POAG) clinical practice guidelines (CPGs). Design: The CPGs were assessed with the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument. Participants: Four authors (A.M.W., C.M.W., B.K.Y., D.J.W.) performed independent assessments of POAG CPGs. Methods: POAG CPGs published by the American Academy of Ophthalmology (AAO), Canadian Ophthalmological Society (COS), and National Institute for Health and Care Excellence (NICE) were appraised using the AGREE II instrument’s 6 domains (Scope and Purpose, Stakeholder Involvement, Rigor of Development, Clarity of Presentation, Applicability, and Editorial Independence) and Overall Assessment score summarizing guideline quality across all domains. Results: Scores ranged from 28% to 85% for the AAO CPG, 51% to 96% for the COS CPG, and 55% to 97% for the NICE CPG. Intraclass correlation coefficients for the reliability of mean scores for the AAO, COS, and NICE CPGs were 0.89, 0.86, and 0.74; 95% CIs were 0.80 to 0.95, 0.74 to 0.93, and 0.51 to 0.87, respectively. The strongest domains were Scope and Purpose (AAO, COS, NICE) and Clarity of Presentation (COS, NICE). The weakest domains were Stakeholder Involvement (AAO, COS) and Editorial Independence (AAO, COS, NICE). Conclusions: Future POAG CPGs can be improved by addressing potential conflicts of interest within the development group, ensuring transparency of guideline development methodology, and involving all relevant stakeholders in guideline development and review. Objet : Évaluer la qualité méthodologique de trois guides de pratique clinique sur le glaucome à angle ouvert primaire (GAOP). Nature : Les guides ont été évalués à l’aide de la Grille d’évaluation de la qualité des recommandations pour la pratique clinique II (grille AGREE II). Participants : Quatre auteurs (AMW, CMW, BKY, DJW) ont réalisé des évaluations indépendantes des guides de pratique clinique. Méthodes : Les guides de pratique clinique sur le glaucome à angle ouvert primaire publiés par l’American Academy of Ophthalmology (AAO), la Société canadienne d’ophtalmologie (SCO) et le National Institute for Health and Care Excellence (NICE) ont été évalués d’après les six domaines de la grille AGREE II (portée et objet, participation des intervenants, rigueur de la mise en œuvre, clarté de la présentation, applicabilité et indépendance rédactionnelle). De plus, la cote de l’évaluation globale résume la qualité des guides dans tous les domaines. Résultats : Les cotes ont varié de 28 % à 85 % pour le guide de l’AAO; de 51 % à 96 % pour le guide de la SCO; et de 55 % à 97 % pour le guide du NICE. Les coefficients de corrélation interne pour la fiabilité des cotes moyennes pour les guides de l’AAO, de la SCO et du NICE étaient respectivement de 0,89, 0,86 et 0,74, avec IC de 95 % [0,80-0,95], [0,74-0,93] et [0,51-0,87]. Les domaines ayant obtenu les cotes les plus élevées sont « portée et objet » (AAO, SCO, NICE) et « clarté de la présentation » (SCO, NICE). Les domaines ayant obtenu les cotes les plus basses sont « participation des intervenants » (AAO, SCO) et « indépendance rédactionnelle » (AAO, SCO, NICE). Conclusions : Il serait possible d’améliorer les futurs guides de pratique clinique sur le glaucome à angle ouvert primaire en éliminant les conflits d’intérêts potentiels au sein du groupe de développement, en assurant la transparence de la méthodologie d’élaboration du guide et en faisant participer tous les intervenants pertinents à l’élaboration et à l’évaluation du guide.
Glaucoma affects an estimated 60.5 million people and is currently the second leading cause of blindness in the world.1 Hence it is important to ensure that clinical practice guidelines
(CPGs) for glaucoma adhere to rigorous standards of development. Recent studies, however, reveal significant discrepancies in the methodologic quality of CPGs, such as poor evidence
From the *Section of Ophthalmology, Providence VA Medical Center ; † Division of Ophthalmology, Warren Alpert Medical School of Brown University; ‡Division of Ophthalmology, Rhode Island Hospital, Providence, RI, and §Departments of Ophthalmology, Pathology, and Cell Biology, Morsani College of Medicine, Tampa, Fl.
(A.M.W., C.M.W., B.K.Y., D.J.W., C.E.M., A.C., P.B.G.); preparation of the manuscript (A.M.W., C.M.W., P.B.G.); review and approval of the manuscript (A.M.W., C.M.W., B.K.Y., D.J.W., C.E.M., A.C., P.B.G.).
Presented as a poster at the American Academy of Ophthalmology Annual Meeting in Chicago, Ill., October 18–21, 2014. The views expressed in this article are those of the authors’ and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government. Contributions of authors: design of the study (A.M.W., C.M.W., B.K.Y., C.E.M., P.B.G.); conduct of the study (A.M.W., C.M.W., B.K.Y., D.J.W.); collection, management, analysis, and interpretation of the data
192
CAN J OPHTHALMOL — VOL. 50, NO. 3, JUNE 2015
Originally received Aug. 30, 2014. Final revision Dec. 20, 2014. Accepted Mar. 16, 2015 Correspondence to Paul B. Greenberg, MD, Section of Ophthalmology, Providence VA Medical Center, 830 Chalkstone Avenue, Providence, RI 02908; [email protected] Can J Ophthalmol 2015;50:192–196 0008-4182/15/$-see front matter Published by Elsevier Inc on behalf of the Canadian Ophthalmological Society. http://dx.doi.org/10.1016/j.jcjo.2015.03.005
POAG clinical practice guidelines—Wu et al. quality, potential conflicts of interest (COIs) among guideline development group members, and a lack of transparency in reporting of guideline development methods.2–6 The Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument is a reliable and validated method of assessing the methodologic quality of CPGs.7–9 The original version of AGREE was an effective tool for evaluating selected primary open-angle glaucoma (POAG) guidelines published in 2003 and 2006.10 To improve construct validity and compliance with methodologic standards of health measurement design, the revised AGREE II uses a 7point assessment scale, reviews the body of evidence for each CPG, and expands the requirements for conflict of interest disclosures.8 We used the AGREE II instrument to evaluate the methodologic rigor of POAG CPGs developed by the American Academy of Ophthalmology (AAO), Canadian Ophthalmological Society (COS), and National Institute for Health and Care Excellence (NICE).
recommendations for screening, diagnosis, intraocular pressure measurement, central corneal thickness measurement, diagnostic tests, progression, and therapies.12 The NICE CPG outlines recommendations for POAG diagnosis, monitoring, treatment, service and information provision to patients, and research questions.13 Four of the authors (A.M.W., C.M.W., B.K.Y., D.J.W.) independently assessed each CPG, focusing on recommendations for POAG management in adult patients; included in the evaluation were any appendices and/or supplementary documents relevant to each guideline’s development and applicability. The scores were then averaged and summarized as scaled percentage scores using the following formula prescribed by the AGREE II: (Obtained score – minimum possible score)/(maximum possible score – minimum possible score).8 An intraclass correlation coefficient (ICC) was used to measure interrater agreement for each guideline’s scores.
METHODS RESULTS The AGREE II instrument is composed of 23 assessment items, which evaluators score on a scale of 1 (strongly disagree) to 7 (strongly agree). The 23 items are organized into 6 quality domains: Scope and Purpose, Stakeholder Involvement, Rigor of Development, Clarity of Presentation, Applicability, and Editorial Independence.8 In addition, an Overall Assessment score included as a final item in the AGREE II rubric enables appraisers to integrate various components of all domains into a comprehensive evaluation of each CPG. The AAO CPG outlines recommendations for diagnosis (Evaluation of Visual Function, Ophthalmic Evaluation, Supplemental Ophthalmic Testing), management (Target Intraocular Pressure, Medical Treatment, Incisional Glaucoma Surgery), and follow-up care for adult patients with POAG.11 The COS CPG provides recommendations for management of both open-angle and angle-closure glaucoma. Only those recommendations pertaining to management of POAG were considered for evaluation in this study; the COS CPG provides a total of 33 POAG
Domain scores are summarized in Table 1; mean scores per reviewer were averaged across all items in each domain. The AAO CPG scored lowest in domain 2 (Stakeholder Involvement) with a scaled score of 28%, and highest in domain 1 (Scope and Purpose) with 85%; the mean Overall Assessment score was 4.8 of 7. The COS CPG also scored lowest in domain 2 with 51%, and highest in domain 4 (Clarity of Presentation) with 96%; the mean Overall Assessment score was 5.5 of 7. The NICE CPG scored lowest in domain 6 (Editorial Independence) with 55%, and highest in domain 4 with 97%; the mean Overall Assessment score was 6.3 of 7. The ICCs for the AAO, COS, and NICE CPG scores were 0.89, 0.86, and 0.74, respectively; 95% CIs were 0.80 to 0.95, 0.74 to 0.93, and 0.51 to 0.87, respectively. Strengths and weaknesses of the guidelines are summarized in Table 2, as determined based on consensus among at least 3 of the 4 evaluators with respect to average scores and evaluator comments per each AGREE II item.
Table 1—Comparison of Appraisal of Guidelines for Research and Evaluation II Instrument mean scores* of individual reviewers (A, B, C, D) and scaled domain percentage scores for primary open-angle glaucoma clinical practice guidelines from the American Academy of Ophthalmology, the Canadian Ophthalmological Society, and the National Institute for Health and Care Excellence AAO Ratings AGREE II Domain 1. Scope and Purpose 2. Stakeholder Involvement 3. Rigor of Development 4. Clarity of Presentation 5. Applicability 6. Editorial Independence Overall Assessment
COS Ratings
NICE Ratings
A
B
C
D
Scaled Score
A
B
C
D
Scaled Score
A
B
C
D
Scaled Score
6.7 2.0 4.3 6.3 5.0 4.5 4
6.7 1.7 4.3 6.7 3.5 6.0 5
5.3 2.7 4.6 5.3 4.3 4.0 5
5.7 2.3 5.0 4.3 4.8 5.5 5
85% 28% 63% 78% 58% 64% 4.75
6.3 3.7 4.5 6.0 5.0 5.5 5
6.0 3.7 5.3 7.0 5.0 6.5 7
6.7 4.3 4.8 7.0 5.3 5.5 5
5.7 4.7 6.1 7.0 4.5 5.5 5
86% 51% 72% 96% 67% 77% 5.50
7.0 6.3 6.1 6.7 6.5 4.5 6
7.0 4.7 6.6 7.0 7.0 4.0 6
5.3 6.0 6.5 6.7 5.5 3.5 6
7.0 6.0 6.5 7.0 7.0 6.0 7
93% 79% 92% 97% 92% 55% 6.25
AAO, American Academy of Ophthalmology; COS, Canadian Ophthalmological Society; NICE, National Institute for Health and Care Excellence; AGREE II, Appraisal of Guidelines Research and Evaluation II. n
Scores were averaged across all items within each domain (maximum item score ¼ 7).
CAN J OPHTHALMOL — VOL. 50, NO. 3, JUNE 2015
193
POAG clinical practice guidelines—Wu et al. DISCUSSION This analysis with the AGREE II instrument outlined a number of areas where the methodologic rigor of the 3 POAG CPGs could be improved. Regarding Stakeholder Involvement (domain 2), all 3 CPGs failed to describe specific roles of panel members and/or delineate the involvement of patient representatives, if any, in the guideline development process. It was not possible to ascertain whether patient representatives were consulted with the exception of the NICE guideline, which cited the involvement of 2 patient representatives in the guideline development group. None of the guidelines detailed outcomes of public input or how the information gathered was used to inform the processes of guideline development or recommendation formulation. In addition, included within domain 2 is the requirement for full disclosure of panel members’ roles in CPG
development. Although each CPG listed panel members’ occupational degrees and areas of expertise, none described panel members’ specific roles concerning guideline development. Similar shortcomings regarding stakeholder involvement were noted in a study by Ou and colleagues10 assessing POAG CPGs from the AAO, European Glaucoma Society, and South East Asia Glaucoma Interest Group using the first version of the AGREE instrument. It is critical that the guideline development panel consist of a multidisciplinary group, including representatives of all stakeholders, to ensure the comprehensive inclusion of relevant scientific evidence and guard against recommendation biases toward a particular specialty or mode of treatment.14 The POAG CPGs had several gaps in Rigor of Development (domain 3), the largest AGREE II domain. First, external review processes were unclear in all 3 CPGs, because they lacked disclosure of external review methods, patient
Table 2—Summary of reviewers’ comments organized by Appraisal of Guidelines for Research and Evaluation II domains on primary open-angle glaucoma clinical practice guidelines from the American Academy of Ophthalmology, the Canadian Ophthalmological Society, and the National Institute for Health and Care Excellence* AGREE II Domain
Strengths
1. Scope and Purpose
Patient population clearly specified; objectives clearly stated; specific health questions clearly described
2. Stakeholder Involvement
CPG development group members’ names, disciplines, institutions, and geographic locations clearly listed and easy to locate (NICE) Guideline development group included individuals from relevant professional groups, including patient representatives (NICE)
Weaknesses
Lack specific descriptions of CPG development group members’ individual roles Lack statement of type of strategy used to capture views and preferences of the patients/public (AAO, COS) Lack outcomes gathered from patients/public and how information was used in guideline development process
3. Rigor of Development
Detailed evidence search terms provided (AAO, NICE) Recommendations describe health benefits, risks, and side effects of recommendations Recommendations preceded by a section detailing pertinent evidence Each recommendation cites a primary source (AAO, COS)
Lack evidence search terms, full search strategy (COS) Methods/outcomes of external review and/or identity of external reviewers not explicit
4. Clarity of Presentation
Specific and unambiguous wording of recommendations
Key recommendations relatively difficult to identify when embedded in main body of text (AAO)
Key recommendations easily identifiable via highlighted tables and/or boxed headings for each recommendation (COS, NICE) 5. Applicability
Tools and advice provided for recommendation implementation: diagrams, treatment algorithms, referral letters, medications lists, and/or summary documents Facilitators and barriers to application explicitly described in “Other Considerations” section of each recommendation table (NICE)
Lacks a clear summary document listing key recommendations (COS) Resource implications not addressed: no mention of health economist as part of guideline development group, no information on types of cost data considered and/or methods by which data were sought, no description of how information gathered was used to inform the guideline development process (AAO, COS)
Resource implications addressed in “Economic Considerations” section of each recommendation table and in cost-effectiveness analysis of appendix (NICE) 6. Editorial Independence
COIs named and categorized Independence of funding body clearly stated (AAO, COS) COIs listed for each meeting over the course of guideline development; methods of COI disclosure described and each COI divested when necessary (NICE)
A majority of committee members had potential COIs (AAO, COS) Independence of funding body not clearly stated (NICE) Lack methods by which potential COIs were sought; description of how competing interests influenced the guideline development process
AGREE II, Appraisal of Guidelines Research and Evaluation II; NICE, National Institute for Health and Care Excellence; CPG, clinical practice guideline; AAO, American Academy of Ophthalmology; COS, Canadian Ophthalmological Society; COI, conflicts of interest. n
Comments specific to certain CPGs are indicated by parentheses. Comments that lack denotations pertain to all 3 CPGs.
194
CAN J OPHTHALMOL — VOL. 50, NO. 3, JUNE 2015
POAG clinical practice guidelines—Wu et al. input, if any, information gathered from external review, and how the outcomes informed guideline development. The NICE CPG was the only guideline to explicitly disclose identities of external reviewers and indicate involvement of a “lay member” in the independent review panel.13 Second, although each guideline provided an evidence rating system, evidence search methods in some cases lacked detailed search terms and dates of search history (COS) or explicit inclusion criteria beyond specification of language (AAO, COS). Finally, the link between recommendations and specific evidence was unclear in the NICE CPG due to the absence of primary source citations following each recommendation. Disclosure of evidence search methods and consistent citation of evidence should be implemented in all POAG CPGs to ensure transparent reporting of evidence-based guideline development. In Clarity of Presentation (domain 4), there were several shortcomings. Although the COS CPG recommendations were readily identifiable throughout the text via clear numbers and bold headings, the guideline lacked a section summarizing all key recommendations for practical use. Relative to the COS and NICE CPGs, key recommendations in the main text of the AAO CPG were sometimes difficult to identify because many of them were embedded in paragraphs; Appendix 2 of the AAO CPG listed major recommendations for care under “diagnosis” and “followup,” but “management recommendations” were listed only within the main text of the guideline.11 Improving clarity of presentation is important for effective communication of guideline recommendations. A common area of weakness in Applicability (domain 5) was the lack of attention given to methods by which cost information was sought and how economic considerations informed the guideline development process. The NICE CPG was the only one to address resource implications by including a cost-effectiveness analysis and providing an “Economic Considerations” section within each recommendation table.13 Although the AAO and COS CPGs mentioned the importance of cost considerations as factors in treatment adherence, few provisions were given regarding specific ways to address economic barriers to
implementation of particular recommendations, and there was no mention of a health economist as part of the guideline development groups. The lack of consideration given to resource limitations was also noted by Ou et al.,10 underscoring the need for increased attention toward economic barriers to facilitate guideline implementation. Lastly, regarding Editorial Independence (domain 6), although all 3 CPGs named and categorized potential COIs, neither the AAO nor the COS CPG specified methods by which potential competing interests were sought, and none of the CPGs explicitly described how potential COIs may have influenced recommendation development. This is particularly important given that a number of panel members had commercial interests that could represent COIs. As a supplementary document on its website, the AAO provides a statement on its policy with regard to industry relationships within clinical guideline committees15; however, this document does not specifically address how industry relationships within the AAO POAG CPG development group might have affected guideline development. Although the NICE CPG did not explicitly describe how potential COIs may have influenced recommendation development, the guideline was the only one to provide a clear description of how COIs were disclosed.16,17 Finally, unlike the AAO and COS CPGs, the NICE guideline did not clearly state the independence of the funding body or provide a list of specific financial disclosures. It is critical to ensure editorial independence and complete disclosure of COIs whenever industry is involved in research, because potential conflicts may bias reporting.18 We acknowledge several limitations to this study. First, assessments within domains such as Editorial Independence and Applicability are unable to distinguish between guidelines that obtained data and did not disclose the information from guidelines that failed to obtain any information pertaining to those categories. Although each guideline development group may have internally investigated COIs and/or economic implications of guideline recommendations, a failure to disclose such information results in a lack of transparency of guideline methodology, undermining the
Table 3—Recommendations for improvement from an Appraisal of Guidelines for Research and Evaluation II assessment of primary open-angle glaucoma clinical practice guidelines from the American Academy of Ophthalmology, the Canadian Ophthalmological Society, and the National Institute for Health and Care Excellence Primary Recommendations
Model from Evaluated CPGs
Stakeholder involvement—clarify roles and input of patients and/or patient representatives in guideline development and review Transparency and rigor of guideline development process—clearly state roles of each member and delineate methods and results of guideline development and review Editorial independence—disclose methods by which competing interests were sought and how competing interests may have potentially influenced the recommendation formulation process
None None None
Additional Recommendations Clarify evidence search methods to include detailed search terms, dates, and specific inclusion criteria Improve formatting to clearly highlight recommendations throughout main text (e.g., with numbered headings) Address facilitators and barriers to application recommendations by detailing economic considerations of guideline implementation Ensure and disclose independence of the guideline development group from potential influences of the funding body
AAO, NICE COS, NICE NICE AAO, COS
CPG, clinical practice guideline; AAO, American Academy of Ophthalmology; NICE, National Institute for Health and Care Excellence; COS, Canadian Ophthalmological Society.
CAN J OPHTHALMOL — VOL. 50, NO. 3, JUNE 2015
195
POAG clinical practice guidelines—Wu et al. ability of users to reflect upon how CPG development could have influenced the recommendations presented. Second, any rating system may lack objectivity. However, the AGREE II instrument has been demonstrated to be valid and reliable, especially with 4 independent raters. In this study, the independent evaluators were highly reliable (ICCs of 0.89, 0.86, and 0.74), which is consistent with AGREE II guideline assessment in other medical fields.19–21 Third, the AGREE II weights equally each of the 6 domains. Because domains vary in number of items and evaluation criteria, we used the Overall Assessment score and evaluator comments to guide holistic appraisals of each guideline. In summary, the AGREE II assessment of 3 POAG CPGs highlighted methodologic shortcomings in the domains of Stakeholder Involvement, Rigor of Development, and Editorial Independence. We recommend several improvements for future guidelines, using whenever possible Z1 of the 3 CPGs as models (Table 3). This appraisal underscores the importance of regular assessment of CPGs to ensure a constructive and transparent dialogue between primary sources and recommendations for clinical practice.
Disclosure: The authors have no proprietary or commercial interest in any materials discussed in this article. Supported by: This work was supported by a U.S. Veterans Health Administration Health Services Research & Development (HSR&D) Veterans Integrated Service Network (VISN) 1 Career Development Award (to P.B.G.). REFERENCES 1. Quigley HA, Broman AT. The number of people with glaucoma worldwide in 2010 and 2020. Br J Ophthalmol. 2006;90:262-7. 2. Ransohoff DF, Pignone M, Sox HC. How to decide whether a clinical practice guideline is trustworthy. JAMA. 2013;309:139-40. 3. Ransohoff DF, Sox HC. Guidelines for guidelines: measuring trustworthiness. J Clin Oncol. 2013;31:2530-1. 4. Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet institute of medicine standards: two more decades of little, if any, progress. Arch Intern Med. 2012;172:1628-33. 5. Shaneyfelt TM, Centor RM. Reassessment of clinical practice guidelines: go gently into that good night. JAMA. 2009;301:868-9. 6. Laine C, Taichman DB, Mulrow C. Trustworthy clinical guidelines. Ann Intern Med. 2011;154:774-5.
196
CAN J OPHTHALMOL — VOL. 50, NO. 3, JUNE 2015
7. AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003;12:18-23. 8. Brouwers MC, Kho ME, Browman GP, et al. AGREE Next Steps Consortium. AGREE II: Advancing guideline development, reporting and evaluation in healthcare. CMAJ. 2010;182:839-42. 9. Brouwers MC, Kho ME, Browman GP, et al. Development of the AGREE II, part 1: performance, usefulness and areas for improvement. CMAJ. 2010;182:1045-52. 10. Ou Y, Goldberg I, Migdal C, Lee PP. A critical appraisal and comparison of the quality and recommendations of glaucoma clinical practice guidelines. Ophthalmology. 2011;118:1017-23. 11. American Academy of Ophthalmology Glaucoma Panel. Preferred Practice Pattern. Primary Open-Angle Glaucoma. San Francisco, CA: American Academy of Ophthalmology; 2010:1-35. Available at: one.aao.org/preferred-practice-pattern/primary-openangle-glauco ma-ppp–october-2010. Accessed June 4, 2014. 12. Canadian Ophthalmological Society evidence-based clinical practice guidelines for the management of glaucoma in the adult eye. Can J Ophthalmol. 2009;44(Suppl. 1):S7-93. Available at: www. ophtalmo.net/bv/Doc/2009-426-guide-glaucome.pdf. Accessed June 4, 2014. 13. National Collaborating Centre for Acute Care. Glaucoma: Diagnosis and Management of Chronic Open Angle Glaucoma and Ocular Hypertension. London, UK: National Institute for Health and Care Excellence; 2009. Available at: www.nice.org.uk/guidance/cg85/ resources/cg85-glaucoma-full-guideline2. Accessed June 4, 2014. 14. Institute of Medicine. Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press; 2011. . 15. American Academy of Ophthalmology Glaucoma Panel. Preferred Practice Patterns and Ophthalmic Technology Assessments: New Relationship with Industry Procedures. San Francisco, CA: American Academy of Ophthalmology; 2010:1-5. Available at: one.aao.org/ asset.axd?id=f00adda4-1a35-4aa3-9aa1-bcbe2fd9b16e&t=6349574 83963230000. Accessed June 4, 2014 . 16. National Collaborating Centre for Acute Care. Glaucoma: diagnosis and management of chronic open angle glaucoma and ocular hypertension: appendices A–G. London, UK: National Institute for Health and Care Excellence; 2009. Available at: www.nice.org.uk/guidance/ cg85/resources/glaucoma-appendices2. Accessed August 4, 2014 . 17. Yip J, Sparrow J. Glaucoma critical guidelines [letter]. Ophthalmology. 2012;119:427-8. author reply 428. Available at: www.ncbi.nlm. nih.gov/m/pubmed/22305314/. Accessed March 7, 2015. 18. Novack GD. The role of pharmaceutical companies in sponsored research. Ophthalmology. 2007;114:1037-8. 19. Marciano NJ, Merlin TL, Bessen T, Street JM. To what extent are current guidelines for cutaneous melanoma follow up based on scientific evidence? Int J Clin Pract. 2014;68:761-70. 20. Zeng L, Zhang L, Hu Z, et al. Systematic review of evidence-based guidelines on medication therapy for upper respiratory tract infection in children with AGREE instrument. PLoS One. 2014;9: e87711. 21. Sabharwal S, Patel NK, Gauher S, Holloway I, Athanasiou T. High methodologic quality but poor applicability: assessment of the AAOS guidelines using the AGREE II instrument Clin Orthop Relat Res. 2014;472:1982-8.
Evaluation of primary open-angle glaucoma clinical practice guidelines Annie M. Wu, BA,*,†,‡ Connie M. Wu, BS,*,†,‡ Benjamin K. Young, BA, MS,*,†,‡ Dominic J. Wu, BS,*,†,‡ Allison Chen, BA,*,†,‡ Curtis E. Margo, MD, MPH,§ Paul B. Greenberg, MD*,†,‡ ABSTRACT ● RÉSUMÉ Objective: To evaluate the methodologic quality of 3 primary open-angle glaucoma (POAG) clinical practice guidelines (CPGs). Design: The CPGs were assessed with the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument. Participants: Four authors (A.M.W., C.M.W., B.K.Y., D.J.W.) performed independent assessments of POAG CPGs. Methods: POAG CPGs published by the American Academy of Ophthalmology (AAO), Canadian Ophthalmological Society (COS), and National Institute for Health and Care Excellence (NICE) were appraised using the AGREE II instrument’s 6 domains (Scope and Purpose, Stakeholder Involvement, Rigor of Development, Clarity of Presentation, Applicability, and Editorial Independence) and Overall Assessment score summarizing guideline quality across all domains. Results: Scores ranged from 28% to 85% for the AAO CPG, 51% to 96% for the COS CPG, and 55% to 97% for the NICE CPG. Intraclass correlation coefficients for the reliability of mean scores for the AAO, COS, and NICE CPGs were 0.89, 0.86, and 0.74; 95% CIs were 0.80 to 0.95, 0.74 to 0.93, and 0.51 to 0.87, respectively. The strongest domains were Scope and Purpose (AAO, COS, NICE) and Clarity of Presentation (COS, NICE). The weakest domains were Stakeholder Involvement (AAO, COS) and Editorial Independence (AAO, COS, NICE). Conclusions: Future POAG CPGs can be improved by addressing potential conflicts of interest within the development group, ensuring transparency of guideline development methodology, and involving all relevant stakeholders in guideline development and review. Objet : Évaluer la qualité méthodologique de trois guides de pratique clinique sur le glaucome à angle ouvert primaire (GAOP). Nature : Les guides ont été évalués à l’aide de la Grille d’évaluation de la qualité des recommandations pour la pratique clinique II (grille AGREE II). Participants : Quatre auteurs (AMW, CMW, BKY, DJW) ont réalisé des évaluations indépendantes des guides de pratique clinique. Méthodes : Les guides de pratique clinique sur le glaucome à angle ouvert primaire publiés par l’American Academy of Ophthalmology (AAO), la Société canadienne d’ophtalmologie (SCO) et le National Institute for Health and Care Excellence (NICE) ont été évalués d’après les six domaines de la grille AGREE II (portée et objet, participation des intervenants, rigueur de la mise en œuvre, clarté de la présentation, applicabilité et indépendance rédactionnelle). De plus, la cote de l’évaluation globale résume la qualité des guides dans tous les domaines. Résultats : Les cotes ont varié de 28 % à 85 % pour le guide de l’AAO; de 51 % à 96 % pour le guide de la SCO; et de 55 % à 97 % pour le guide du NICE. Les coefficients de corrélation interne pour la fiabilité des cotes moyennes pour les guides de l’AAO, de la SCO et du NICE étaient respectivement de 0,89, 0,86 et 0,74, avec IC de 95 % [0,80-0,95], [0,74-0,93] et [0,51-0,87]. Les domaines ayant obtenu les cotes les plus élevées sont « portée et objet » (AAO, SCO, NICE) et « clarté de la présentation » (SCO, NICE). Les domaines ayant obtenu les cotes les plus basses sont « participation des intervenants » (AAO, SCO) et « indépendance rédactionnelle » (AAO, SCO, NICE). Conclusions : Il serait possible d’améliorer les futurs guides de pratique clinique sur le glaucome à angle ouvert primaire en éliminant les conflits d’intérêts potentiels au sein du groupe de développement, en assurant la transparence de la méthodologie d’élaboration du guide et en faisant participer tous les intervenants pertinents à l’élaboration et à l’évaluation du guide.
Glaucoma affects an estimated 60.5 million people and is currently the second leading cause of blindness in the world.1 Hence it is important to ensure that clinical practice guidelines
(CPGs) for glaucoma adhere to rigorous standards of development. Recent studies, however, reveal significant discrepancies in the methodologic quality of CPGs, such as poor evidence
From the *Section of Ophthalmology, Providence VA Medical Center ; † Division of Ophthalmology, Warren Alpert Medical School of Brown University; ‡Division of Ophthalmology, Rhode Island Hospital, Providence, RI, and §Departments of Ophthalmology, Pathology, and Cell Biology, Morsani College of Medicine, Tampa, Fl.
(A.M.W., C.M.W., B.K.Y., D.J.W., C.E.M., A.C., P.B.G.); preparation of the manuscript (A.M.W., C.M.W., P.B.G.); review and approval of the manuscript (A.M.W., C.M.W., B.K.Y., D.J.W., C.E.M., A.C., P.B.G.).
Presented as a poster at the American Academy of Ophthalmology Annual Meeting in Chicago, Ill., October 18–21, 2014. The views expressed in this article are those of the authors’ and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government. Contributions of authors: design of the study (A.M.W., C.M.W., B.K.Y., C.E.M., P.B.G.); conduct of the study (A.M.W., C.M.W., B.K.Y., D.J.W.); collection, management, analysis, and interpretation of the data
192
CAN J OPHTHALMOL — VOL. 50, NO. 3, JUNE 2015
Originally received Aug. 30, 2014. Final revision Dec. 20, 2014. Accepted Mar. 16, 2015 Correspondence to Paul B. Greenberg, MD, Section of Ophthalmology, Providence VA Medical Center, 830 Chalkstone Avenue, Providence, RI 02908; [email protected] Can J Ophthalmol 2015;50:192–196 0008-4182/15/$-see front matter Published by Elsevier Inc on behalf of the Canadian Ophthalmological Society. http://dx.doi.org/10.1016/j.jcjo.2015.03.005
POAG clinical practice guidelines—Wu et al. quality, potential conflicts of interest (COIs) among guideline development group members, and a lack of transparency in reporting of guideline development methods.2–6 The Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument is a reliable and validated method of assessing the methodologic quality of CPGs.7–9 The original version of AGREE was an effective tool for evaluating selected primary open-angle glaucoma (POAG) guidelines published in 2003 and 2006.10 To improve construct validity and compliance with methodologic standards of health measurement design, the revised AGREE II uses a 7point assessment scale, reviews the body of evidence for each CPG, and expands the requirements for conflict of interest disclosures.8 We used the AGREE II instrument to evaluate the methodologic rigor of POAG CPGs developed by the American Academy of Ophthalmology (AAO), Canadian Ophthalmological Society (COS), and National Institute for Health and Care Excellence (NICE).
recommendations for screening, diagnosis, intraocular pressure measurement, central corneal thickness measurement, diagnostic tests, progression, and therapies.12 The NICE CPG outlines recommendations for POAG diagnosis, monitoring, treatment, service and information provision to patients, and research questions.13 Four of the authors (A.M.W., C.M.W., B.K.Y., D.J.W.) independently assessed each CPG, focusing on recommendations for POAG management in adult patients; included in the evaluation were any appendices and/or supplementary documents relevant to each guideline’s development and applicability. The scores were then averaged and summarized as scaled percentage scores using the following formula prescribed by the AGREE II: (Obtained score – minimum possible score)/(maximum possible score – minimum possible score).8 An intraclass correlation coefficient (ICC) was used to measure interrater agreement for each guideline’s scores.
METHODS RESULTS The AGREE II instrument is composed of 23 assessment items, which evaluators score on a scale of 1 (strongly disagree) to 7 (strongly agree). The 23 items are organized into 6 quality domains: Scope and Purpose, Stakeholder Involvement, Rigor of Development, Clarity of Presentation, Applicability, and Editorial Independence.8 In addition, an Overall Assessment score included as a final item in the AGREE II rubric enables appraisers to integrate various components of all domains into a comprehensive evaluation of each CPG. The AAO CPG outlines recommendations for diagnosis (Evaluation of Visual Function, Ophthalmic Evaluation, Supplemental Ophthalmic Testing), management (Target Intraocular Pressure, Medical Treatment, Incisional Glaucoma Surgery), and follow-up care for adult patients with POAG.11 The COS CPG provides recommendations for management of both open-angle and angle-closure glaucoma. Only those recommendations pertaining to management of POAG were considered for evaluation in this study; the COS CPG provides a total of 33 POAG
Domain scores are summarized in Table 1; mean scores per reviewer were averaged across all items in each domain. The AAO CPG scored lowest in domain 2 (Stakeholder Involvement) with a scaled score of 28%, and highest in domain 1 (Scope and Purpose) with 85%; the mean Overall Assessment score was 4.8 of 7. The COS CPG also scored lowest in domain 2 with 51%, and highest in domain 4 (Clarity of Presentation) with 96%; the mean Overall Assessment score was 5.5 of 7. The NICE CPG scored lowest in domain 6 (Editorial Independence) with 55%, and highest in domain 4 with 97%; the mean Overall Assessment score was 6.3 of 7. The ICCs for the AAO, COS, and NICE CPG scores were 0.89, 0.86, and 0.74, respectively; 95% CIs were 0.80 to 0.95, 0.74 to 0.93, and 0.51 to 0.87, respectively. Strengths and weaknesses of the guidelines are summarized in Table 2, as determined based on consensus among at least 3 of the 4 evaluators with respect to average scores and evaluator comments per each AGREE II item.
Table 1—Comparison of Appraisal of Guidelines for Research and Evaluation II Instrument mean scores* of individual reviewers (A, B, C, D) and scaled domain percentage scores for primary open-angle glaucoma clinical practice guidelines from the American Academy of Ophthalmology, the Canadian Ophthalmological Society, and the National Institute for Health and Care Excellence AAO Ratings AGREE II Domain 1. Scope and Purpose 2. Stakeholder Involvement 3. Rigor of Development 4. Clarity of Presentation 5. Applicability 6. Editorial Independence Overall Assessment
COS Ratings
NICE Ratings
A
B
C
D
Scaled Score
A
B
C
D
Scaled Score
A
B
C
D
Scaled Score
6.7 2.0 4.3 6.3 5.0 4.5 4
6.7 1.7 4.3 6.7 3.5 6.0 5
5.3 2.7 4.6 5.3 4.3 4.0 5
5.7 2.3 5.0 4.3 4.8 5.5 5
85% 28% 63% 78% 58% 64% 4.75
6.3 3.7 4.5 6.0 5.0 5.5 5
6.0 3.7 5.3 7.0 5.0 6.5 7
6.7 4.3 4.8 7.0 5.3 5.5 5
5.7 4.7 6.1 7.0 4.5 5.5 5
86% 51% 72% 96% 67% 77% 5.50
7.0 6.3 6.1 6.7 6.5 4.5 6
7.0 4.7 6.6 7.0 7.0 4.0 6
5.3 6.0 6.5 6.7 5.5 3.5 6
7.0 6.0 6.5 7.0 7.0 6.0 7
93% 79% 92% 97% 92% 55% 6.25
AAO, American Academy of Ophthalmology; COS, Canadian Ophthalmological Society; NICE, National Institute for Health and Care Excellence; AGREE II, Appraisal of Guidelines Research and Evaluation II. n
Scores were averaged across all items within each domain (maximum item score ¼ 7).
CAN J OPHTHALMOL — VOL. 50, NO. 3, JUNE 2015
193
POAG clinical practice guidelines—Wu et al. DISCUSSION This analysis with the AGREE II instrument outlined a number of areas where the methodologic rigor of the 3 POAG CPGs could be improved. Regarding Stakeholder Involvement (domain 2), all 3 CPGs failed to describe specific roles of panel members and/or delineate the involvement of patient representatives, if any, in the guideline development process. It was not possible to ascertain whether patient representatives were consulted with the exception of the NICE guideline, which cited the involvement of 2 patient representatives in the guideline development group. None of the guidelines detailed outcomes of public input or how the information gathered was used to inform the processes of guideline development or recommendation formulation. In addition, included within domain 2 is the requirement for full disclosure of panel members’ roles in CPG
development. Although each CPG listed panel members’ occupational degrees and areas of expertise, none described panel members’ specific roles concerning guideline development. Similar shortcomings regarding stakeholder involvement were noted in a study by Ou and colleagues10 assessing POAG CPGs from the AAO, European Glaucoma Society, and South East Asia Glaucoma Interest Group using the first version of the AGREE instrument. It is critical that the guideline development panel consist of a multidisciplinary group, including representatives of all stakeholders, to ensure the comprehensive inclusion of relevant scientific evidence and guard against recommendation biases toward a particular specialty or mode of treatment.14 The POAG CPGs had several gaps in Rigor of Development (domain 3), the largest AGREE II domain. First, external review processes were unclear in all 3 CPGs, because they lacked disclosure of external review methods, patient
Table 2—Summary of reviewers’ comments organized by Appraisal of Guidelines for Research and Evaluation II domains on primary open-angle glaucoma clinical practice guidelines from the American Academy of Ophthalmology, the Canadian Ophthalmological Society, and the National Institute for Health and Care Excellence* AGREE II Domain
Strengths
1. Scope and Purpose
Patient population clearly specified; objectives clearly stated; specific health questions clearly described
2. Stakeholder Involvement
CPG development group members’ names, disciplines, institutions, and geographic locations clearly listed and easy to locate (NICE) Guideline development group included individuals from relevant professional groups, including patient representatives (NICE)
Weaknesses
Lack specific descriptions of CPG development group members’ individual roles Lack statement of type of strategy used to capture views and preferences of the patients/public (AAO, COS) Lack outcomes gathered from patients/public and how information was used in guideline development process
3. Rigor of Development
Detailed evidence search terms provided (AAO, NICE) Recommendations describe health benefits, risks, and side effects of recommendations Recommendations preceded by a section detailing pertinent evidence Each recommendation cites a primary source (AAO, COS)
Lack evidence search terms, full search strategy (COS) Methods/outcomes of external review and/or identity of external reviewers not explicit
4. Clarity of Presentation
Specific and unambiguous wording of recommendations
Key recommendations relatively difficult to identify when embedded in main body of text (AAO)
Key recommendations easily identifiable via highlighted tables and/or boxed headings for each recommendation (COS, NICE) 5. Applicability
Tools and advice provided for recommendation implementation: diagrams, treatment algorithms, referral letters, medications lists, and/or summary documents Facilitators and barriers to application explicitly described in “Other Considerations” section of each recommendation table (NICE)
Lacks a clear summary document listing key recommendations (COS) Resource implications not addressed: no mention of health economist as part of guideline development group, no information on types of cost data considered and/or methods by which data were sought, no description of how information gathered was used to inform the guideline development process (AAO, COS)
Resource implications addressed in “Economic Considerations” section of each recommendation table and in cost-effectiveness analysis of appendix (NICE) 6. Editorial Independence
COIs named and categorized Independence of funding body clearly stated (AAO, COS) COIs listed for each meeting over the course of guideline development; methods of COI disclosure described and each COI divested when necessary (NICE)
A majority of committee members had potential COIs (AAO, COS) Independence of funding body not clearly stated (NICE) Lack methods by which potential COIs were sought; description of how competing interests influenced the guideline development process
AGREE II, Appraisal of Guidelines Research and Evaluation II; NICE, National Institute for Health and Care Excellence; CPG, clinical practice guideline; AAO, American Academy of Ophthalmology; COS, Canadian Ophthalmological Society; COI, conflicts of interest. n
Comments specific to certain CPGs are indicated by parentheses. Comments that lack denotations pertain to all 3 CPGs.
194
CAN J OPHTHALMOL — VOL. 50, NO. 3, JUNE 2015
POAG clinical practice guidelines—Wu et al. input, if any, information gathered from external review, and how the outcomes informed guideline development. The NICE CPG was the only guideline to explicitly disclose identities of external reviewers and indicate involvement of a “lay member” in the independent review panel.13 Second, although each guideline provided an evidence rating system, evidence search methods in some cases lacked detailed search terms and dates of search history (COS) or explicit inclusion criteria beyond specification of language (AAO, COS). Finally, the link between recommendations and specific evidence was unclear in the NICE CPG due to the absence of primary source citations following each recommendation. Disclosure of evidence search methods and consistent citation of evidence should be implemented in all POAG CPGs to ensure transparent reporting of evidence-based guideline development. In Clarity of Presentation (domain 4), there were several shortcomings. Although the COS CPG recommendations were readily identifiable throughout the text via clear numbers and bold headings, the guideline lacked a section summarizing all key recommendations for practical use. Relative to the COS and NICE CPGs, key recommendations in the main text of the AAO CPG were sometimes difficult to identify because many of them were embedded in paragraphs; Appendix 2 of the AAO CPG listed major recommendations for care under “diagnosis” and “followup,” but “management recommendations” were listed only within the main text of the guideline.11 Improving clarity of presentation is important for effective communication of guideline recommendations. A common area of weakness in Applicability (domain 5) was the lack of attention given to methods by which cost information was sought and how economic considerations informed the guideline development process. The NICE CPG was the only one to address resource implications by including a cost-effectiveness analysis and providing an “Economic Considerations” section within each recommendation table.13 Although the AAO and COS CPGs mentioned the importance of cost considerations as factors in treatment adherence, few provisions were given regarding specific ways to address economic barriers to
implementation of particular recommendations, and there was no mention of a health economist as part of the guideline development groups. The lack of consideration given to resource limitations was also noted by Ou et al.,10 underscoring the need for increased attention toward economic barriers to facilitate guideline implementation. Lastly, regarding Editorial Independence (domain 6), although all 3 CPGs named and categorized potential COIs, neither the AAO nor the COS CPG specified methods by which potential competing interests were sought, and none of the CPGs explicitly described how potential COIs may have influenced recommendation development. This is particularly important given that a number of panel members had commercial interests that could represent COIs. As a supplementary document on its website, the AAO provides a statement on its policy with regard to industry relationships within clinical guideline committees15; however, this document does not specifically address how industry relationships within the AAO POAG CPG development group might have affected guideline development. Although the NICE CPG did not explicitly describe how potential COIs may have influenced recommendation development, the guideline was the only one to provide a clear description of how COIs were disclosed.16,17 Finally, unlike the AAO and COS CPGs, the NICE guideline did not clearly state the independence of the funding body or provide a list of specific financial disclosures. It is critical to ensure editorial independence and complete disclosure of COIs whenever industry is involved in research, because potential conflicts may bias reporting.18 We acknowledge several limitations to this study. First, assessments within domains such as Editorial Independence and Applicability are unable to distinguish between guidelines that obtained data and did not disclose the information from guidelines that failed to obtain any information pertaining to those categories. Although each guideline development group may have internally investigated COIs and/or economic implications of guideline recommendations, a failure to disclose such information results in a lack of transparency of guideline methodology, undermining the
Table 3—Recommendations for improvement from an Appraisal of Guidelines for Research and Evaluation II assessment of primary open-angle glaucoma clinical practice guidelines from the American Academy of Ophthalmology, the Canadian Ophthalmological Society, and the National Institute for Health and Care Excellence Primary Recommendations
Model from Evaluated CPGs
Stakeholder involvement—clarify roles and input of patients and/or patient representatives in guideline development and review Transparency and rigor of guideline development process—clearly state roles of each member and delineate methods and results of guideline development and review Editorial independence—disclose methods by which competing interests were sought and how competing interests may have potentially influenced the recommendation formulation process
None None None
Additional Recommendations Clarify evidence search methods to include detailed search terms, dates, and specific inclusion criteria Improve formatting to clearly highlight recommendations throughout main text (e.g., with numbered headings) Address facilitators and barriers to application recommendations by detailing economic considerations of guideline implementation Ensure and disclose independence of the guideline development group from potential influences of the funding body
AAO, NICE COS, NICE NICE AAO, COS
CPG, clinical practice guideline; AAO, American Academy of Ophthalmology; NICE, National Institute for Health and Care Excellence; COS, Canadian Ophthalmological Society.
CAN J OPHTHALMOL — VOL. 50, NO. 3, JUNE 2015
195
POAG clinical practice guidelines—Wu et al. ability of users to reflect upon how CPG development could have influenced the recommendations presented. Second, any rating system may lack objectivity. However, the AGREE II instrument has been demonstrated to be valid and reliable, especially with 4 independent raters. In this study, the independent evaluators were highly reliable (ICCs of 0.89, 0.86, and 0.74), which is consistent with AGREE II guideline assessment in other medical fields.19–21 Third, the AGREE II weights equally each of the 6 domains. Because domains vary in number of items and evaluation criteria, we used the Overall Assessment score and evaluator comments to guide holistic appraisals of each guideline. In summary, the AGREE II assessment of 3 POAG CPGs highlighted methodologic shortcomings in the domains of Stakeholder Involvement, Rigor of Development, and Editorial Independence. We recommend several improvements for future guidelines, using whenever possible Z1 of the 3 CPGs as models (Table 3). This appraisal underscores the importance of regular assessment of CPGs to ensure a constructive and transparent dialogue between primary sources and recommendations for clinical practice.
Disclosure: The authors have no proprietary or commercial interest in any materials discussed in this article. Supported by: This work was supported by a U.S. Veterans Health Administration Health Services Research & Development (HSR&D) Veterans Integrated Service Network (VISN) 1 Career Development Award (to P.B.G.). REFERENCES 1. Quigley HA, Broman AT. The number of people with glaucoma worldwide in 2010 and 2020. Br J Ophthalmol. 2006;90:262-7. 2. Ransohoff DF, Pignone M, Sox HC. How to decide whether a clinical practice guideline is trustworthy. JAMA. 2013;309:139-40. 3. Ransohoff DF, Sox HC. Guidelines for guidelines: measuring trustworthiness. J Clin Oncol. 2013;31:2530-1. 4. Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet institute of medicine standards: two more decades of little, if any, progress. Arch Intern Med. 2012;172:1628-33. 5. Shaneyfelt TM, Centor RM. Reassessment of clinical practice guidelines: go gently into that good night. JAMA. 2009;301:868-9. 6. Laine C, Taichman DB, Mulrow C. Trustworthy clinical guidelines. Ann Intern Med. 2011;154:774-5.
196
CAN J OPHTHALMOL — VOL. 50, NO. 3, JUNE 2015
7. AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003;12:18-23. 8. Brouwers MC, Kho ME, Browman GP, et al. AGREE Next Steps Consortium. AGREE II: Advancing guideline development, reporting and evaluation in healthcare. CMAJ. 2010;182:839-42. 9. Brouwers MC, Kho ME, Browman GP, et al. Development of the AGREE II, part 1: performance, usefulness and areas for improvement. CMAJ. 2010;182:1045-52. 10. Ou Y, Goldberg I, Migdal C, Lee PP. A critical appraisal and comparison of the quality and recommendations of glaucoma clinical practice guidelines. Ophthalmology. 2011;118:1017-23. 11. American Academy of Ophthalmology Glaucoma Panel. Preferred Practice Pattern. Primary Open-Angle Glaucoma. San Francisco, CA: American Academy of Ophthalmology; 2010:1-35. Available at: one.aao.org/preferred-practice-pattern/primary-openangle-glauco ma-ppp–october-2010. Accessed June 4, 2014. 12. Canadian Ophthalmological Society evidence-based clinical practice guidelines for the management of glaucoma in the adult eye. Can J Ophthalmol. 2009;44(Suppl. 1):S7-93. Available at: www. ophtalmo.net/bv/Doc/2009-426-guide-glaucome.pdf. Accessed June 4, 2014. 13. National Collaborating Centre for Acute Care. Glaucoma: Diagnosis and Management of Chronic Open Angle Glaucoma and Ocular Hypertension. London, UK: National Institute for Health and Care Excellence; 2009. Available at: www.nice.org.uk/guidance/cg85/ resources/cg85-glaucoma-full-guideline2. Accessed June 4, 2014. 14. Institute of Medicine. Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press; 2011. . 15. American Academy of Ophthalmology Glaucoma Panel. Preferred Practice Patterns and Ophthalmic Technology Assessments: New Relationship with Industry Procedures. San Francisco, CA: American Academy of Ophthalmology; 2010:1-5. Available at: one.aao.org/ asset.axd?id=f00adda4-1a35-4aa3-9aa1-bcbe2fd9b16e&t=6349574 83963230000. Accessed June 4, 2014 . 16. National Collaborating Centre for Acute Care. Glaucoma: diagnosis and management of chronic open angle glaucoma and ocular hypertension: appendices A–G. London, UK: National Institute for Health and Care Excellence; 2009. Available at: www.nice.org.uk/guidance/ cg85/resources/glaucoma-appendices2. Accessed August 4, 2014 . 17. Yip J, Sparrow J. Glaucoma critical guidelines [letter]. Ophthalmology. 2012;119:427-8. author reply 428. Available at: www.ncbi.nlm. nih.gov/m/pubmed/22305314/. Accessed March 7, 2015. 18. Novack GD. The role of pharmaceutical companies in sponsored research. Ophthalmology. 2007;114:1037-8. 19. Marciano NJ, Merlin TL, Bessen T, Street JM. To what extent are current guidelines for cutaneous melanoma follow up based on scientific evidence? Int J Clin Pract. 2014;68:761-70. 20. Zeng L, Zhang L, Hu Z, et al. Systematic review of evidence-based guidelines on medication therapy for upper respiratory tract infection in children with AGREE instrument. PLoS One. 2014;9: e87711. 21. Sabharwal S, Patel NK, Gauher S, Holloway I, Athanasiou T. High methodologic quality but poor applicability: assessment of the AAOS guidelines using the AGREE II instrument Clin Orthop Relat Res. 2014;472:1982-8.
