Downloaded from http://bjo.bmj.com/ on June 13, 2015 - Published by group.bmj.com
Clinical science
An evaluation of cataract surgery clinical practice guidelines Connie M Wu,1,2,3 Annie M Wu,1,2,3 Benjamin K Young,1,2,3 Dominic Wu,1,2,3 Allison Chen,1,2,3 Curtis E Margo,4 Paul B Greenberg1,2,3 1
Section of Ophthalmology, Providence VA Medical Center, Providence, Rhode Island, USA 2 Division of Ophthalmology, Warren Alpert Medical School of Brown University, Providence, Rhode Island, USA 3 Division of Ophthalmology, Rhode Island Hospital, Providence, Rhode Island, USA 4 Departments of Ophthalmology, Pathology and Cell Biology, Morsani College of Medicine, Tampa, Florida, USA Correspondence to Dr Paul B Greenberg, Section of Ophthalmology, Providence VA Medical Center, 830 Chalkstone Ave, Providence, Rhode Island 02908, USA; [email protected] Presentation Presented in part as a poster at the 2014 Association for Research in Vision and Ophthalmology Annual Meeting, 4–8 May, Orlando, FL. Received 31 May 2014 Revised 15 August 2014 Accepted 7 September 2014 Published Online First 24 September 2014
To cite: Wu CM, Wu AM, Young BK, et al. Br J Ophthalmol 2015;99: 401–404.
ABSTRACT Purpose This study used the Appraisal of Guidelines for Research and Evaluation (AGREE) II Instrument to evaluate the methodological quality of clinical practice guidelines (CPG) published by the American Academy of Ophthalmology (AAO), Canadian Ophthalmological Society (COS) and Royal College of Ophthalmologists (RCO) for the management of cataract in adults. Study design An evaluation of the AAO, COS and RCO CPGs using a reliable and validated instrument. Methods Four evaluators independently appraised the three CPGs using the AGREE II Instrument, which covers six domains (Scope and Purpose, Stakeholder Involvement, Rigour of Development, Clarity of Presentation, Applicability and Editorial Independence). The AGREE II includes an Overall Assessment summarising guideline methodological rigour across all domains, using a 7-point scale where perfect adherence equals a score of 7. Results Scores ranged from 36% to 75% for the AAO guideline; 45% to 94% for the COS guideline and 23% to 85% for the RCO guideline. Intraclass correlation coefficients for the reliability of mean scores for the AAO, COS, and RCO were 0.78, 0.74 and 0.80; 95% CIs (0.60 to 0.90), (0.45 to 0.88) and (0.53 to 0.91), respectively. The strongest domains were Scope and Purpose (COS, RCO), Clarity of Presentation (COS, RCO) and Editorial Independence (AAO, COS). The weakest were Stakeholder Involvement (AAO), Applicability (AAO, COS) and Editorial Independence (RCO). Conclusions Cataract surgery practice guidelines can be improved by targeting stakeholder involvement, applicability and editorial independence. Cataract surgery is the most commonly performed elective surgical procedure and is associated with significant improvement in quality of life.1–3 Therefore, it is imperative that clinical practice guidelines (CPG) for cataract management are developed with transparency and methodological rigour. However, recent studies evaluating the trustworthiness of CPGs have shown that guidelines contain conflicting recommendations, exhibit discrepancies in evidence quality, lack rigour and transparency in their development processes and have potential conflicts of interest (COI) among members of guideline development groups (GDG).4–7 These findings undermine the validity of CPGs, limiting their usefulness as evidence-based resources for clinicians.5 8 The Appraisal of Guidelines for Research and Evaluation (AGREE) II Instrument9 is a reliable and validated method for evaluating the methodological rigour of CPGs.10–12 With the goal of optimising
future CPGs, this study used the AGREE II Instrument to evaluate adult cataract CPGs published by the American Academy of Ophthalmology (AAO), Canadian Ophthalmological Society (COS) and Royal College of Ophthalmologists (RCO).
METHODS The AGREE II Instrument comprises 23 assessment items, which evaluators rate on a scale of 1 (strongly disagree) to 7 (strongly agree). The 23 items are organised into six quality domains: (1) Scope and Purpose; (2) Stakeholder Involvement; (3) Rigour of Development; (4) Clarity of Presentation; (5) Applicability; (6) Editorial Independence.13 Two final summary assessment items included in the AGREE II rubric enable the appraiser to integrate the various components in formulating a comprehensive evaluation of CPG methodological rigour. The AAO cataract guideline outlines 11 highlighted recommendations and an additional eight categories of recommendations for the care of adult patients with cataract.2 The COS cataract guideline outlines 65 recommendations organised into six categories.14 The RCO guideline for the management of cataract includes recommendations for both adult and paediatric patient groups, and is organised into 12 categories.3 Four of the authors (CMW, AMW, BKY, DW) independently assessed each guideline focusing on recommendations for cataract management in adult patients, including the AAO cataract guideline supplementary documents, Highlighted Recommendations and Summary Benchmarks. The scores were then averaged and summarised as a scaled percentage score using the following formula recommended by the AGREE II: (Obtained Score −Minimum Possible Score)/(Maximum Possible Score−Minimum Possible Score). An intraclass correlation coefficient (ICC) was used to measure interrater agreement for each guideline’s scores.
RESULTS Domain scores are summarised in table 1. The AAO guideline scored lowest in Domain 2 (Stakeholder Involvement) with a scaled score of 36% and highest in Domain 6 (Editorial Independence) with 75%. The COS guideline scored lowest in Domain 5 (Applicability) with 45% and highest in Domain 4 (Clarity of Presentation) with 94%. The RCO guideline scored lowest in Domain 6 with 23% and highest in Domains 1 (Scope and Purpose) and 4, with 83% and 85%, respectively. The ICCs for the AAO, COS and RCO were 0.78, 0.74 and 0.80; 95% CIs were (0.60 to 0.90),
Wu CM, et al. Br J Ophthalmol 2015;99:401–404. doi:10.1136/bjophthalmol-2014-305567
401
Downloaded from http://bjo.bmj.com/ on June 13, 2015 - Published by group.bmj.com
Clinical science Table 1 Comparison of Appraisal of Guidelines Research and Evaluation (AGREE II) Instrument scores* by each rater (A, B, C, D) and scaled domain percentage scores for the adult cataract clinical practice guidelines developed by the American Academy of Ophthalmology (AAO), the Canadian Ophthalmological Society (COS) and the Royal College of Ophthalmologists (RCO) AAO ratings
COS ratings
RCO ratings
AGREE II Domain
Item†
A
B
C
D
Scaled score
A
B
C
D
Scaled score
A
B
C
D
Scaled score
1. Scope and Purpose
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
4 2 7 2 2 6 6 1 2 2 6 4 3 7 4 5 6 2 3 4 3 4 4 3
3 1 6 3 1 2 6 2 3 1 6 5 3 6 2 3 6 5 4 5 2 7 4 3
7 3 5 2 1 6 7 1 2 2 5 6 4 7 2 5 7 5 6 6 3 7 5 4
6 5 3 4 3 6 6 5 3 3 6 6 7 5 6 7 7 2 2 5 6 7 6 5
56%
6 5 5 4 1 6 4 5 5 5 7 6 3 3 6 6 6 4 2 3 5 6 5 5
6 4 5 2 1 7 4 2 7 3 6 7 4 3 7 6 7 2 2 5 2 7 6 5
7 4 6 6 3 7 5 5 5 6 6 7 4 5 7 7 7 2 1 2 5 7 4 5
7 7 7 6 6 7 7 6 7 7 7 7 6 6 7 7 7 6 7 6 5 7 7 6
79%
6 5 5 3 4 5 3 6 3 1 6 4 2 2 4 6 5 5 5 4 4 2 2 4
7 5 7 5 5 7 4 7 7 2 6 5 3 5 6 7 7 6 6 5 4 2 2 6
7 5 6 3 4 4 5 5 2 3 7 5 1 4 6 7 6 5 4 4 5 1 2 4
7 6 6 6 6 6 7 7 5 6 6 6 5 3 6 7 6 6 7 6 7 4 4 6
83%
2. Stakeholder Involvement
3. Rigour of Development
4. Clarity of Presentation
5. Applicability
6. Editorial Independence Overall assessment
36%
55%
67%
49%
75% 4
61%
72%
94%
45%
85% 5
64%
58%
85%
70%
23% 5
*Maximum score=7. †Item descriptions are delineated in the AGREE II Instrument.9
(0.45 to 0.88) and (0.53 to 0.91), respectively. To determine the strengths and weaknesses of the guidelines summarised in table 2, we assessed the average scores of each domain for uniformity and examined comments for concurrence among at least three of the four evaluators.
DISCUSSION This AGREE II analysis highlighted several areas where future CPGs for the management of adult cataract could be improved. All three guidelines fell short in Stakeholder Involvement (Domain 2) by not delineating specific roles of GDG members or the involvement of patient representatives. Diversity among GDG members is integral to formulating less extreme recommendations, preventing the promotion of treatments biased toward a particular specialty, and ensuring the comprehensive inclusion of relevant scientific evidence.15 The AGREE II states that all stakeholders in the care process, including patients, should be involved in recommendation development. It was not possible to determine if patient groups were consulted, with the exception of the RCO guideline, which mentioned public involvement in the guideline development process from the ‘College Lay Advisory Group’. None of the guidelines specified the outcomes gathered from public input. Included within Domain 2 is the requirement for full disclosure of GDG members’ roles and areas of expertise. Although the guidelines addressed GDG members’ areas of expertise, none mentioned their individual roles pertaining to guideline development. Transparency of the guideline development process and strength of supporting evidence evaluated under Rigour of Development (Domain 3) presented areas for improvement across the board. First, the external review processes were unclear in all guidelines as they lacked descriptions of external review methods, external reviewers, patient input, information 402
gathered from external review and how the outcomes informed the guideline development. In the AAO guideline, the role of the Consumers Union, the only non-medical external reviewing body, was not clearly defined as representing patients or patient advocates.16 If the Consumers Union was intended to represent cataract patients, this objective should be explicitly stated in the guideline. The other guidelines provided no details of the external review process, so it was impossible to verify if public input through the external review played a role in shaping recommendation formulation. Second, the rigour of supporting evidence was variable, including the lack of a clear evidence rating system (AAO) and failure to disclose detailed search terms (COS, RCO). Consistent evidence rating systems as used by the COS and RCO should be implemented in all cataract CPGs because they illuminate the strength of supporting evidence and clarify which statements are recommendations. Similarly, detailed evidence search terms and methods were only available for the AAO guideline. Literature search methods should be explicitly described to enable independent reproduction of developers’ conclusions and to facilitate transparency of guideline development methods. On Clarity of Presentation (Domain 4), the guidelines employed different methods to ensure utility of recommendations, yet each displayed distinct needs for improvement. The RCO recommendations were flagged throughout the text using boxes, and the COS recommendations were easily identifiable with clear numbers and bold headings. However, neither document provided a concise summary of key recommendations. The AAO guideline provided a supplementary list of 11 major recommendations under ‘Highlighted Findings and Recommendations for Care’, but recommendations in the main text were harder to locate relative to those of the COS and RCO guidelines. Some of the recommendations were primarily summary statements (e.g.
Wu CM, et al. Br J Ophthalmol 2015;99:401–404. doi:10.1136/bjophthalmol-2014-305567
Downloaded from http://bjo.bmj.com/ on June 13, 2015 - Published by group.bmj.com
Clinical science Table 2 Summary of appraisers’ comments organised by Appraisal of Guidelines Research and Evaluation (AGREE II) domains on the adult cataract clinical practice guidelines (CPG) developed by the American Academy of Ophthalmology (AAO), the Canadian Ophthalmological Society (COS) and the Royal College of Ophthalmologists (RCO)* AGREE II domain
Strengths
Weaknesses
1. Scope and Purpose 2. Stakeholder Involvement
Target patient population is clearly specified Objectives are clearly stated Contains brief statement of Lay Advisory Group involvement (RCO) GDG members’ names, disciplines, institutions and geographical locations are clearly listed and easy to locate (RCO)
Specific health questions are not clearly stated or thoroughly described
3. Rigour of Development
Detailed search terms are provided online (AAO)
No statement of type of strategy used to capture views and preferences of the patients/public (AAO, COS) No description of feedback gathered from patient representatives and how the information shaped recommendation formulation Evidence search terms mention database and time periods searched, but lack search terms and full search strategy (COS, RCO) No description of recommendation formulation process or how development process influenced recommendations
The ‘Highlighted Findings’ detail health benefits, risks, side effects and costs for potential complications should the guidelines not be followed (AAO) Recommendations are preceded by a section detailing pertinent evidence; each recommendation cites a primary source (COS)
4. Clarity of Presentation
5. Applicability
6. Editorial Independence
Recommendations are graded according to NICE definitions; each recommendation is preceded by a box containing evidence grade (RCO) Eleven key recommendations are clearly indicated under ‘Highlighted Findings and Recommendations for Care’ (AAO) Recommendations are numbered, listed first in each subsection, proceeded directly by strength of evidence rating and citations (COS) Recommendations are flagged via boxes in the left margin featuring ratings for evidence strength (RCO) Contains section on socioeconomic considerations, and subsections on cost effectiveness and cost considerations (AAO) Includes thorough monitoring and auditing criteria (COS) Discussion on outsourcing cataract surgery includes consideration of problems for the provider, local eye unit and commissioner (RCO) Independence of funding body clearly stated (AAO, COS) Competing interests are named and categorised by type (AAO, COS) The PPP review panel was free of COI (AAO)
No details of external review methods, description of external reviewers, outcomes/info gathered from external review or how the info gathered was used to inform development No detailed methodology for updating the CPG (COS, RCO)
Long passages without clear formatting make it difficult to identify recommendations within the main text (AAO) Recommendation wording is poor: sometimes use language such as ‘techniques commonly used are’ instead of providing a concise, actionable recommendation (AAO) Implementation section lacks a summary document of the key recommendations, algorithms and surgical checklists Resource implications not covered: no identification of the methods by which cost information was sought, description of cost information, and how it informed the guideline development process (COS) At least half the PPP panel, PPP consulting work group, and PPP committee members had potential financial COI (AAO) No explicit financial disclosures for GDG members (RCO) No mention of funding body (RCO) No explanation of methods by which potential competing interests were sought, description of the competing interests, and how they influenced the recommendations
*Comments specific to certain CPGs are indicated in parentheses; comments lacking denotations pertain to all three CPGs. GDG, guideline development group; NICE, National Institute for Health and Care Excellence; COI, conflict of interest; PPP, preferred practice pattern.
‘cataract surgery is a procedure appropriately utilized in the United States’) rather than actionable recommendations. Regarding Applicability (Domain 5), the guidelines varied in their consideration for resource implications, auditing criteria and the provision of supplementary documents for guideline implementation. While the AAO guideline included sections on Socioeconomic Considerations, Cost Effectiveness and Cost Considerations, and the RCO guideline included a discussion of problems for the provider, local eye unit and commissioner, overall there was insufficient disclosure of methods by which cost information was sought, and the impact of such considerations on guideline development. The guidelines provided sparse auditing criteria except for the mention that ‘outcome measures should be varied and should match the indications for surgery’ (COS). The intangible influences of differing healthcare systems’ methods of reimbursement are difficult to assess, and it may not be possible to control for the different influences health insurance might have on assessing applicability. Nonetheless, among the CPGs, there was an overall paucity of supplementary algorithms, surgical checklists and other tools to facilitate implementation of recommendations. All CPGs should maintain Editorial Independence (Domain 6), yet all three guidelines failed to specify methods by which potential competing interests were sought and how the potential
COI influenced the formulation of recommendations. The AAO provides a blanket statement online indicating compliance for all its CPGs with the Council of Medical Specialty Societies Code for Interactions with Companies,17 but no specific information on potential COI influence on the development of the cataract CPG. Lack of COI transparency is of particular concern when chair members and a majority of GDG members have financial interests, as was the case for the AAO. Whereas the AAO and COS guidelines mentioned independence of the funding body and provided a list of specific financial disclosures, the RCO guideline provided no mention of the funding body and no comprehensive financial disclosures for GDG members, stating only that the ‘members of the working party have made the Chair aware of their commercial relationships.’ We acknowledge the limitations to this study. First, rating systems involving human evaluators will always limit objectivity, including potential biases associated with nationality. However, the AGREE II system has been demonstrated to be valid and reliable, especially with multiple independent raters, as this study used.10 11 All raters completed the AGREE II tutorial which provides clear benchmarks for scoring along the 7-point scale.13 The independent evaluators were highly reliable (based on ICCs of 0.78, 0.74 and 0.80), which is consistent with previous AGREE II studies.9 Second, AGREE II examines the
Wu CM, et al. Br J Ophthalmol 2015;99:401–404. doi:10.1136/bjophthalmol-2014-305567
403
Downloaded from http://bjo.bmj.com/ on June 13, 2015 - Published by group.bmj.com
Clinical science Table 3 Recommendations for improving adult cataract clinical practice guidelines from the Appraisal of Guidelines Research and Evaluation (AGREE II) assessment of guidelines developed by the American Academy of Ophthalmology (AAO), the Canadian Ophthalmological Society (COS) and the Royal College of Ophthalmologists (RCO) Models in AAO, COS or RCO cataract guidelines
Primary recommendations Stakeholder involvement—Clarify feedback gathered from patient representatives and how the information shaped recommendation formulation Transparency of guideline development process—Clearly state roles of each member pertaining to guideline development and delineate methods of guideline development and review Applicability—Include comprehensive summary documents of the key recommendations, algorithms and surgical checklists Editorial independence—Disclose methods by which competing interests were sought and how competing interests may have potentially influenced the recommendation formulation process Additional recommendations Clarify evidence search methods to include detailed search terms, dates and discussion of inclusion criteria Improve formatting to clearly highlight recommendations throughout main text (e.g. with numbered headings) Provide a summary list of major recommendations to facilitate their identification Address facilitators and barriers to application of recommendations by detailing economic considerations of guideline implementation Ensure and disclose independence of the guideline development group from potential influences of the funding body
methodological rigour and reporting transparency rather than recommendation validity or evidence quality.9 While rigorous guideline development processes do not necessarily guarantee quality guideline content, suggestions for methodological improvements can be universally applicable and transparency can facilitate evaluation of clinical applicability. Finally, the AGREE II gives equal weight to all six domains, whereas domains may vary in depth and importance. We reconciled this potential issue by using the Overall Assessment and evaluator comments to assess each guideline holistically and identify specific areas for improvement. In summary, the primary weaknesses of the adult cataract CPGs were in the areas of patient involvement, transparency of the guideline development process, applicability and editorial independence. We have formulated several recommendations for the targeted improvement of adult cataract CPGs, using models from the reviewed CPGs when possible (table 3). This assessment using the AGREE II instrument underscores the importance of regular evaluation of CPGs to optimise patient care. Contributors The seven authors meet authorship criteria. In detail: CW, AW, BY: study design, analysis and interpretation of data, drafting of the manuscript, final approval given; DW: acquisition of data, analysis and interpretation of data, final approval given; AC: statistical calculations, analysis and interpretation of data, critical review of the manuscript, final approval given; CM: analysis and interpretation of data, critical review of manuscript, final approval given; PBG—study design, analysis and interpretation of data, critical review of the manuscript, final approval given.
Provenance and peer review Not commissioned; externally peer reviewed.
404
None None None
AAO COS, RCO AAO AAO, RCO AAO, COS
REFERENCES 1 2
3
4
5 6 7 8 9 10 11
12 13 14
Disclaimer The views expressed in this article are those of the authors, and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government. Competing interests PBG has received a USA Veterans Health Administration Health Services Research & Development (HSR&D) Veterans Integrated Service Network (VISN) 1 Career Development Award. The funding organisation had no role in the design or conduct of this research.
None
15 16 17
Sparrow JM. Cataract surgical rates: is there overprovision in certain areas? Br J Ophthalmol 2007;91:852–3. Cataract in the Adult Eye Preferred Practice Protocol Guidelines. American Academy of Ophthalmology. 2011 Oct. http://one.aao.org/CE/PracticeGuidelines/PPP_Content. aspx?cid=a80a87ce-9042–4677-85d7-4b876deed276 (accessed 28 Mar 2013). Cataract Surgery Guidelines. Royal College of Ophthalmologists. 2010. http://www. rcophth.ac.uk/core/core_picker/download.asp?id=544&filetitle=Cataract+Surgery +Guidelines+2010 (accessed 28 Apr 2014). Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet Institute of Medicine standards: two more decades of little, if any, progress. Arch Intern Med 2012;172:1628–33. Ransohoff DF, Sox HC. Guidelines for guidelines: measuring trustworthiness. JCO 2013;31:2530–1. Shaneyfelt TM, Centor RM. Reassessment of clinical practice guidelines: go gently into that good night. JAMA 2009;301:868–9. Laine C, Taichman DB, Mulrow C. Trustworthy clinical guidelines. Ann Intern Med 2011;154:774–5. Ransohoff DF, Pignone M, Sox HC. How to decide whether a clinical practice guideline is trustworthy. JAMA 2013;309:139–40. Brouwers MC, Kho ME, Browman GP, et al. AGREE II: advancing guideline development, reporting and evaluation in healthcare. CMAJ 2010;182:E839–42. Brouwers MC, Kho ME, Browman GP, et al. Development of the AGREE II, part 1: performance, usefulness and areas for improvement. CMAJ 2010;182:1045–52. Brouwers MC, Kho ME, Browman GP, et al. Development of the AGREE II, part 2: assessment of validity of items and tools to support application. CMAJ 2010;182:E472–8. Holmer HK, Ogden LA, Burda BU, et al. Quality of clinical practice guidelines for glycemic control in type 2 diabetes mellitus. PLoS ONE 2013;8:e58625. AGREE II Tutorial+Practice Exercise. http://www.agreetrust.org/resource-centre/ agree-ii-training-tools/ (accessed 14 Oct 2013). Canadian Ophthalmological Society Cataract Surgery Clinical Practice Guideline Expert Committee. Canadian Ophthalmological Society evidence-based clinical practice guidelines for cataract surgery in the adult eye. Can J Ophthalmol 2008;43 (Suppl 1):S7–57. http://www.cos-sco.ca/cpgs/COS_CataractCPGs_Oct08.pdf (accessed 28 Apr 2014). Graham R, Mancher M, Wolman DM, et al. Clinical practice guidelines we can trust. Washington, DC: The National Academies Press, 2011:75–204. Consumers Union. http://consumersunion.org/ (accessed 25 Jul 2013). About Preferred Practice Patterns. http://one.aao.org/ about-preferred-practice-patterns (accessed 10 Oct 2014).
Wu CM, et al. Br J Ophthalmol 2015;99:401–404. doi:10.1136/bjophthalmol-2014-305567
Downloaded from http://bjo.bmj.com/ on June 13, 2015 - Published by group.bmj.com
An evaluation of cataract surgery clinical practice guidelines Connie M Wu, Annie M Wu, Benjamin K Young, Dominic Wu, Allison Chen, Curtis E Margo and Paul B Greenberg Br J Ophthalmol 2015 99: 401-404 originally published online September 24, 2014
doi: 10.1136/bjophthalmol-2014-305567 Updated information and services can be found at: http://bjo.bmj.com/content/99/3/401
These include:
References Email alerting service
Topic Collections
This article cites 11 articles, 5 of which you can access for free at: http://bjo.bmj.com/content/99/3/401#BIBL Receive free email alerts when new articles cite this article. Sign up in the box at the top right corner of the online article.
Articles on similar topics can be found in the following collections Lens and zonules (759)
Notes
To request permissions go to: http://group.bmj.com/group/rights-licensing/permissions To order reprints go to: http://journals.bmj.com/cgi/reprintform To subscribe to BMJ go to: http://group.bmj.com/subscribe/
Clinical science
An evaluation of cataract surgery clinical practice guidelines Connie M Wu,1,2,3 Annie M Wu,1,2,3 Benjamin K Young,1,2,3 Dominic Wu,1,2,3 Allison Chen,1,2,3 Curtis E Margo,4 Paul B Greenberg1,2,3 1
Section of Ophthalmology, Providence VA Medical Center, Providence, Rhode Island, USA 2 Division of Ophthalmology, Warren Alpert Medical School of Brown University, Providence, Rhode Island, USA 3 Division of Ophthalmology, Rhode Island Hospital, Providence, Rhode Island, USA 4 Departments of Ophthalmology, Pathology and Cell Biology, Morsani College of Medicine, Tampa, Florida, USA Correspondence to Dr Paul B Greenberg, Section of Ophthalmology, Providence VA Medical Center, 830 Chalkstone Ave, Providence, Rhode Island 02908, USA; [email protected] Presentation Presented in part as a poster at the 2014 Association for Research in Vision and Ophthalmology Annual Meeting, 4–8 May, Orlando, FL. Received 31 May 2014 Revised 15 August 2014 Accepted 7 September 2014 Published Online First 24 September 2014
To cite: Wu CM, Wu AM, Young BK, et al. Br J Ophthalmol 2015;99: 401–404.
ABSTRACT Purpose This study used the Appraisal of Guidelines for Research and Evaluation (AGREE) II Instrument to evaluate the methodological quality of clinical practice guidelines (CPG) published by the American Academy of Ophthalmology (AAO), Canadian Ophthalmological Society (COS) and Royal College of Ophthalmologists (RCO) for the management of cataract in adults. Study design An evaluation of the AAO, COS and RCO CPGs using a reliable and validated instrument. Methods Four evaluators independently appraised the three CPGs using the AGREE II Instrument, which covers six domains (Scope and Purpose, Stakeholder Involvement, Rigour of Development, Clarity of Presentation, Applicability and Editorial Independence). The AGREE II includes an Overall Assessment summarising guideline methodological rigour across all domains, using a 7-point scale where perfect adherence equals a score of 7. Results Scores ranged from 36% to 75% for the AAO guideline; 45% to 94% for the COS guideline and 23% to 85% for the RCO guideline. Intraclass correlation coefficients for the reliability of mean scores for the AAO, COS, and RCO were 0.78, 0.74 and 0.80; 95% CIs (0.60 to 0.90), (0.45 to 0.88) and (0.53 to 0.91), respectively. The strongest domains were Scope and Purpose (COS, RCO), Clarity of Presentation (COS, RCO) and Editorial Independence (AAO, COS). The weakest were Stakeholder Involvement (AAO), Applicability (AAO, COS) and Editorial Independence (RCO). Conclusions Cataract surgery practice guidelines can be improved by targeting stakeholder involvement, applicability and editorial independence. Cataract surgery is the most commonly performed elective surgical procedure and is associated with significant improvement in quality of life.1–3 Therefore, it is imperative that clinical practice guidelines (CPG) for cataract management are developed with transparency and methodological rigour. However, recent studies evaluating the trustworthiness of CPGs have shown that guidelines contain conflicting recommendations, exhibit discrepancies in evidence quality, lack rigour and transparency in their development processes and have potential conflicts of interest (COI) among members of guideline development groups (GDG).4–7 These findings undermine the validity of CPGs, limiting their usefulness as evidence-based resources for clinicians.5 8 The Appraisal of Guidelines for Research and Evaluation (AGREE) II Instrument9 is a reliable and validated method for evaluating the methodological rigour of CPGs.10–12 With the goal of optimising
future CPGs, this study used the AGREE II Instrument to evaluate adult cataract CPGs published by the American Academy of Ophthalmology (AAO), Canadian Ophthalmological Society (COS) and Royal College of Ophthalmologists (RCO).
METHODS The AGREE II Instrument comprises 23 assessment items, which evaluators rate on a scale of 1 (strongly disagree) to 7 (strongly agree). The 23 items are organised into six quality domains: (1) Scope and Purpose; (2) Stakeholder Involvement; (3) Rigour of Development; (4) Clarity of Presentation; (5) Applicability; (6) Editorial Independence.13 Two final summary assessment items included in the AGREE II rubric enable the appraiser to integrate the various components in formulating a comprehensive evaluation of CPG methodological rigour. The AAO cataract guideline outlines 11 highlighted recommendations and an additional eight categories of recommendations for the care of adult patients with cataract.2 The COS cataract guideline outlines 65 recommendations organised into six categories.14 The RCO guideline for the management of cataract includes recommendations for both adult and paediatric patient groups, and is organised into 12 categories.3 Four of the authors (CMW, AMW, BKY, DW) independently assessed each guideline focusing on recommendations for cataract management in adult patients, including the AAO cataract guideline supplementary documents, Highlighted Recommendations and Summary Benchmarks. The scores were then averaged and summarised as a scaled percentage score using the following formula recommended by the AGREE II: (Obtained Score −Minimum Possible Score)/(Maximum Possible Score−Minimum Possible Score). An intraclass correlation coefficient (ICC) was used to measure interrater agreement for each guideline’s scores.
RESULTS Domain scores are summarised in table 1. The AAO guideline scored lowest in Domain 2 (Stakeholder Involvement) with a scaled score of 36% and highest in Domain 6 (Editorial Independence) with 75%. The COS guideline scored lowest in Domain 5 (Applicability) with 45% and highest in Domain 4 (Clarity of Presentation) with 94%. The RCO guideline scored lowest in Domain 6 with 23% and highest in Domains 1 (Scope and Purpose) and 4, with 83% and 85%, respectively. The ICCs for the AAO, COS and RCO were 0.78, 0.74 and 0.80; 95% CIs were (0.60 to 0.90),
Wu CM, et al. Br J Ophthalmol 2015;99:401–404. doi:10.1136/bjophthalmol-2014-305567
401
Downloaded from http://bjo.bmj.com/ on June 13, 2015 - Published by group.bmj.com
Clinical science Table 1 Comparison of Appraisal of Guidelines Research and Evaluation (AGREE II) Instrument scores* by each rater (A, B, C, D) and scaled domain percentage scores for the adult cataract clinical practice guidelines developed by the American Academy of Ophthalmology (AAO), the Canadian Ophthalmological Society (COS) and the Royal College of Ophthalmologists (RCO) AAO ratings
COS ratings
RCO ratings
AGREE II Domain
Item†
A
B
C
D
Scaled score
A
B
C
D
Scaled score
A
B
C
D
Scaled score
1. Scope and Purpose
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
4 2 7 2 2 6 6 1 2 2 6 4 3 7 4 5 6 2 3 4 3 4 4 3
3 1 6 3 1 2 6 2 3 1 6 5 3 6 2 3 6 5 4 5 2 7 4 3
7 3 5 2 1 6 7 1 2 2 5 6 4 7 2 5 7 5 6 6 3 7 5 4
6 5 3 4 3 6 6 5 3 3 6 6 7 5 6 7 7 2 2 5 6 7 6 5
56%
6 5 5 4 1 6 4 5 5 5 7 6 3 3 6 6 6 4 2 3 5 6 5 5
6 4 5 2 1 7 4 2 7 3 6 7 4 3 7 6 7 2 2 5 2 7 6 5
7 4 6 6 3 7 5 5 5 6 6 7 4 5 7 7 7 2 1 2 5 7 4 5
7 7 7 6 6 7 7 6 7 7 7 7 6 6 7 7 7 6 7 6 5 7 7 6
79%
6 5 5 3 4 5 3 6 3 1 6 4 2 2 4 6 5 5 5 4 4 2 2 4
7 5 7 5 5 7 4 7 7 2 6 5 3 5 6 7 7 6 6 5 4 2 2 6
7 5 6 3 4 4 5 5 2 3 7 5 1 4 6 7 6 5 4 4 5 1 2 4
7 6 6 6 6 6 7 7 5 6 6 6 5 3 6 7 6 6 7 6 7 4 4 6
83%
2. Stakeholder Involvement
3. Rigour of Development
4. Clarity of Presentation
5. Applicability
6. Editorial Independence Overall assessment
36%
55%
67%
49%
75% 4
61%
72%
94%
45%
85% 5
64%
58%
85%
70%
23% 5
*Maximum score=7. †Item descriptions are delineated in the AGREE II Instrument.9
(0.45 to 0.88) and (0.53 to 0.91), respectively. To determine the strengths and weaknesses of the guidelines summarised in table 2, we assessed the average scores of each domain for uniformity and examined comments for concurrence among at least three of the four evaluators.
DISCUSSION This AGREE II analysis highlighted several areas where future CPGs for the management of adult cataract could be improved. All three guidelines fell short in Stakeholder Involvement (Domain 2) by not delineating specific roles of GDG members or the involvement of patient representatives. Diversity among GDG members is integral to formulating less extreme recommendations, preventing the promotion of treatments biased toward a particular specialty, and ensuring the comprehensive inclusion of relevant scientific evidence.15 The AGREE II states that all stakeholders in the care process, including patients, should be involved in recommendation development. It was not possible to determine if patient groups were consulted, with the exception of the RCO guideline, which mentioned public involvement in the guideline development process from the ‘College Lay Advisory Group’. None of the guidelines specified the outcomes gathered from public input. Included within Domain 2 is the requirement for full disclosure of GDG members’ roles and areas of expertise. Although the guidelines addressed GDG members’ areas of expertise, none mentioned their individual roles pertaining to guideline development. Transparency of the guideline development process and strength of supporting evidence evaluated under Rigour of Development (Domain 3) presented areas for improvement across the board. First, the external review processes were unclear in all guidelines as they lacked descriptions of external review methods, external reviewers, patient input, information 402
gathered from external review and how the outcomes informed the guideline development. In the AAO guideline, the role of the Consumers Union, the only non-medical external reviewing body, was not clearly defined as representing patients or patient advocates.16 If the Consumers Union was intended to represent cataract patients, this objective should be explicitly stated in the guideline. The other guidelines provided no details of the external review process, so it was impossible to verify if public input through the external review played a role in shaping recommendation formulation. Second, the rigour of supporting evidence was variable, including the lack of a clear evidence rating system (AAO) and failure to disclose detailed search terms (COS, RCO). Consistent evidence rating systems as used by the COS and RCO should be implemented in all cataract CPGs because they illuminate the strength of supporting evidence and clarify which statements are recommendations. Similarly, detailed evidence search terms and methods were only available for the AAO guideline. Literature search methods should be explicitly described to enable independent reproduction of developers’ conclusions and to facilitate transparency of guideline development methods. On Clarity of Presentation (Domain 4), the guidelines employed different methods to ensure utility of recommendations, yet each displayed distinct needs for improvement. The RCO recommendations were flagged throughout the text using boxes, and the COS recommendations were easily identifiable with clear numbers and bold headings. However, neither document provided a concise summary of key recommendations. The AAO guideline provided a supplementary list of 11 major recommendations under ‘Highlighted Findings and Recommendations for Care’, but recommendations in the main text were harder to locate relative to those of the COS and RCO guidelines. Some of the recommendations were primarily summary statements (e.g.
Wu CM, et al. Br J Ophthalmol 2015;99:401–404. doi:10.1136/bjophthalmol-2014-305567
Downloaded from http://bjo.bmj.com/ on June 13, 2015 - Published by group.bmj.com
Clinical science Table 2 Summary of appraisers’ comments organised by Appraisal of Guidelines Research and Evaluation (AGREE II) domains on the adult cataract clinical practice guidelines (CPG) developed by the American Academy of Ophthalmology (AAO), the Canadian Ophthalmological Society (COS) and the Royal College of Ophthalmologists (RCO)* AGREE II domain
Strengths
Weaknesses
1. Scope and Purpose 2. Stakeholder Involvement
Target patient population is clearly specified Objectives are clearly stated Contains brief statement of Lay Advisory Group involvement (RCO) GDG members’ names, disciplines, institutions and geographical locations are clearly listed and easy to locate (RCO)
Specific health questions are not clearly stated or thoroughly described
3. Rigour of Development
Detailed search terms are provided online (AAO)
No statement of type of strategy used to capture views and preferences of the patients/public (AAO, COS) No description of feedback gathered from patient representatives and how the information shaped recommendation formulation Evidence search terms mention database and time periods searched, but lack search terms and full search strategy (COS, RCO) No description of recommendation formulation process or how development process influenced recommendations
The ‘Highlighted Findings’ detail health benefits, risks, side effects and costs for potential complications should the guidelines not be followed (AAO) Recommendations are preceded by a section detailing pertinent evidence; each recommendation cites a primary source (COS)
4. Clarity of Presentation
5. Applicability
6. Editorial Independence
Recommendations are graded according to NICE definitions; each recommendation is preceded by a box containing evidence grade (RCO) Eleven key recommendations are clearly indicated under ‘Highlighted Findings and Recommendations for Care’ (AAO) Recommendations are numbered, listed first in each subsection, proceeded directly by strength of evidence rating and citations (COS) Recommendations are flagged via boxes in the left margin featuring ratings for evidence strength (RCO) Contains section on socioeconomic considerations, and subsections on cost effectiveness and cost considerations (AAO) Includes thorough monitoring and auditing criteria (COS) Discussion on outsourcing cataract surgery includes consideration of problems for the provider, local eye unit and commissioner (RCO) Independence of funding body clearly stated (AAO, COS) Competing interests are named and categorised by type (AAO, COS) The PPP review panel was free of COI (AAO)
No details of external review methods, description of external reviewers, outcomes/info gathered from external review or how the info gathered was used to inform development No detailed methodology for updating the CPG (COS, RCO)
Long passages without clear formatting make it difficult to identify recommendations within the main text (AAO) Recommendation wording is poor: sometimes use language such as ‘techniques commonly used are’ instead of providing a concise, actionable recommendation (AAO) Implementation section lacks a summary document of the key recommendations, algorithms and surgical checklists Resource implications not covered: no identification of the methods by which cost information was sought, description of cost information, and how it informed the guideline development process (COS) At least half the PPP panel, PPP consulting work group, and PPP committee members had potential financial COI (AAO) No explicit financial disclosures for GDG members (RCO) No mention of funding body (RCO) No explanation of methods by which potential competing interests were sought, description of the competing interests, and how they influenced the recommendations
*Comments specific to certain CPGs are indicated in parentheses; comments lacking denotations pertain to all three CPGs. GDG, guideline development group; NICE, National Institute for Health and Care Excellence; COI, conflict of interest; PPP, preferred practice pattern.
‘cataract surgery is a procedure appropriately utilized in the United States’) rather than actionable recommendations. Regarding Applicability (Domain 5), the guidelines varied in their consideration for resource implications, auditing criteria and the provision of supplementary documents for guideline implementation. While the AAO guideline included sections on Socioeconomic Considerations, Cost Effectiveness and Cost Considerations, and the RCO guideline included a discussion of problems for the provider, local eye unit and commissioner, overall there was insufficient disclosure of methods by which cost information was sought, and the impact of such considerations on guideline development. The guidelines provided sparse auditing criteria except for the mention that ‘outcome measures should be varied and should match the indications for surgery’ (COS). The intangible influences of differing healthcare systems’ methods of reimbursement are difficult to assess, and it may not be possible to control for the different influences health insurance might have on assessing applicability. Nonetheless, among the CPGs, there was an overall paucity of supplementary algorithms, surgical checklists and other tools to facilitate implementation of recommendations. All CPGs should maintain Editorial Independence (Domain 6), yet all three guidelines failed to specify methods by which potential competing interests were sought and how the potential
COI influenced the formulation of recommendations. The AAO provides a blanket statement online indicating compliance for all its CPGs with the Council of Medical Specialty Societies Code for Interactions with Companies,17 but no specific information on potential COI influence on the development of the cataract CPG. Lack of COI transparency is of particular concern when chair members and a majority of GDG members have financial interests, as was the case for the AAO. Whereas the AAO and COS guidelines mentioned independence of the funding body and provided a list of specific financial disclosures, the RCO guideline provided no mention of the funding body and no comprehensive financial disclosures for GDG members, stating only that the ‘members of the working party have made the Chair aware of their commercial relationships.’ We acknowledge the limitations to this study. First, rating systems involving human evaluators will always limit objectivity, including potential biases associated with nationality. However, the AGREE II system has been demonstrated to be valid and reliable, especially with multiple independent raters, as this study used.10 11 All raters completed the AGREE II tutorial which provides clear benchmarks for scoring along the 7-point scale.13 The independent evaluators were highly reliable (based on ICCs of 0.78, 0.74 and 0.80), which is consistent with previous AGREE II studies.9 Second, AGREE II examines the
Wu CM, et al. Br J Ophthalmol 2015;99:401–404. doi:10.1136/bjophthalmol-2014-305567
403
Downloaded from http://bjo.bmj.com/ on June 13, 2015 - Published by group.bmj.com
Clinical science Table 3 Recommendations for improving adult cataract clinical practice guidelines from the Appraisal of Guidelines Research and Evaluation (AGREE II) assessment of guidelines developed by the American Academy of Ophthalmology (AAO), the Canadian Ophthalmological Society (COS) and the Royal College of Ophthalmologists (RCO) Models in AAO, COS or RCO cataract guidelines
Primary recommendations Stakeholder involvement—Clarify feedback gathered from patient representatives and how the information shaped recommendation formulation Transparency of guideline development process—Clearly state roles of each member pertaining to guideline development and delineate methods of guideline development and review Applicability—Include comprehensive summary documents of the key recommendations, algorithms and surgical checklists Editorial independence—Disclose methods by which competing interests were sought and how competing interests may have potentially influenced the recommendation formulation process Additional recommendations Clarify evidence search methods to include detailed search terms, dates and discussion of inclusion criteria Improve formatting to clearly highlight recommendations throughout main text (e.g. with numbered headings) Provide a summary list of major recommendations to facilitate their identification Address facilitators and barriers to application of recommendations by detailing economic considerations of guideline implementation Ensure and disclose independence of the guideline development group from potential influences of the funding body
methodological rigour and reporting transparency rather than recommendation validity or evidence quality.9 While rigorous guideline development processes do not necessarily guarantee quality guideline content, suggestions for methodological improvements can be universally applicable and transparency can facilitate evaluation of clinical applicability. Finally, the AGREE II gives equal weight to all six domains, whereas domains may vary in depth and importance. We reconciled this potential issue by using the Overall Assessment and evaluator comments to assess each guideline holistically and identify specific areas for improvement. In summary, the primary weaknesses of the adult cataract CPGs were in the areas of patient involvement, transparency of the guideline development process, applicability and editorial independence. We have formulated several recommendations for the targeted improvement of adult cataract CPGs, using models from the reviewed CPGs when possible (table 3). This assessment using the AGREE II instrument underscores the importance of regular evaluation of CPGs to optimise patient care. Contributors The seven authors meet authorship criteria. In detail: CW, AW, BY: study design, analysis and interpretation of data, drafting of the manuscript, final approval given; DW: acquisition of data, analysis and interpretation of data, final approval given; AC: statistical calculations, analysis and interpretation of data, critical review of the manuscript, final approval given; CM: analysis and interpretation of data, critical review of manuscript, final approval given; PBG—study design, analysis and interpretation of data, critical review of the manuscript, final approval given.
Provenance and peer review Not commissioned; externally peer reviewed.
404
None None None
AAO COS, RCO AAO AAO, RCO AAO, COS
REFERENCES 1 2
3
4
5 6 7 8 9 10 11
12 13 14
Disclaimer The views expressed in this article are those of the authors, and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government. Competing interests PBG has received a USA Veterans Health Administration Health Services Research & Development (HSR&D) Veterans Integrated Service Network (VISN) 1 Career Development Award. The funding organisation had no role in the design or conduct of this research.
None
15 16 17
Sparrow JM. Cataract surgical rates: is there overprovision in certain areas? Br J Ophthalmol 2007;91:852–3. Cataract in the Adult Eye Preferred Practice Protocol Guidelines. American Academy of Ophthalmology. 2011 Oct. http://one.aao.org/CE/PracticeGuidelines/PPP_Content. aspx?cid=a80a87ce-9042–4677-85d7-4b876deed276 (accessed 28 Mar 2013). Cataract Surgery Guidelines. Royal College of Ophthalmologists. 2010. http://www. rcophth.ac.uk/core/core_picker/download.asp?id=544&filetitle=Cataract+Surgery +Guidelines+2010 (accessed 28 Apr 2014). Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet Institute of Medicine standards: two more decades of little, if any, progress. Arch Intern Med 2012;172:1628–33. Ransohoff DF, Sox HC. Guidelines for guidelines: measuring trustworthiness. JCO 2013;31:2530–1. Shaneyfelt TM, Centor RM. Reassessment of clinical practice guidelines: go gently into that good night. JAMA 2009;301:868–9. Laine C, Taichman DB, Mulrow C. Trustworthy clinical guidelines. Ann Intern Med 2011;154:774–5. Ransohoff DF, Pignone M, Sox HC. How to decide whether a clinical practice guideline is trustworthy. JAMA 2013;309:139–40. Brouwers MC, Kho ME, Browman GP, et al. AGREE II: advancing guideline development, reporting and evaluation in healthcare. CMAJ 2010;182:E839–42. Brouwers MC, Kho ME, Browman GP, et al. Development of the AGREE II, part 1: performance, usefulness and areas for improvement. CMAJ 2010;182:1045–52. Brouwers MC, Kho ME, Browman GP, et al. Development of the AGREE II, part 2: assessment of validity of items and tools to support application. CMAJ 2010;182:E472–8. Holmer HK, Ogden LA, Burda BU, et al. Quality of clinical practice guidelines for glycemic control in type 2 diabetes mellitus. PLoS ONE 2013;8:e58625. AGREE II Tutorial+Practice Exercise. http://www.agreetrust.org/resource-centre/ agree-ii-training-tools/ (accessed 14 Oct 2013). Canadian Ophthalmological Society Cataract Surgery Clinical Practice Guideline Expert Committee. Canadian Ophthalmological Society evidence-based clinical practice guidelines for cataract surgery in the adult eye. Can J Ophthalmol 2008;43 (Suppl 1):S7–57. http://www.cos-sco.ca/cpgs/COS_CataractCPGs_Oct08.pdf (accessed 28 Apr 2014). Graham R, Mancher M, Wolman DM, et al. Clinical practice guidelines we can trust. Washington, DC: The National Academies Press, 2011:75–204. Consumers Union. http://consumersunion.org/ (accessed 25 Jul 2013). About Preferred Practice Patterns. http://one.aao.org/ about-preferred-practice-patterns (accessed 10 Oct 2014).
Wu CM, et al. Br J Ophthalmol 2015;99:401–404. doi:10.1136/bjophthalmol-2014-305567
Downloaded from http://bjo.bmj.com/ on June 13, 2015 - Published by group.bmj.com
An evaluation of cataract surgery clinical practice guidelines Connie M Wu, Annie M Wu, Benjamin K Young, Dominic Wu, Allison Chen, Curtis E Margo and Paul B Greenberg Br J Ophthalmol 2015 99: 401-404 originally published online September 24, 2014
doi: 10.1136/bjophthalmol-2014-305567 Updated information and services can be found at: http://bjo.bmj.com/content/99/3/401
These include:
References Email alerting service
Topic Collections
This article cites 11 articles, 5 of which you can access for free at: http://bjo.bmj.com/content/99/3/401#BIBL Receive free email alerts when new articles cite this article. Sign up in the box at the top right corner of the online article.
Articles on similar topics can be found in the following collections Lens and zonules (759)
Notes
To request permissions go to: http://group.bmj.com/group/rights-licensing/permissions To order reprints go to: http://journals.bmj.com/cgi/reprintform To subscribe to BMJ go to: http://group.bmj.com/subscribe/
