EDUCATIONAL ADVANCES
Emergency Department Quality Improvement Activity: An Inventory From the American Board of Emergency Medicine Maintenance of Certification Program Terry Kowalenko, MD, Michael L. Carius, MD, Robert C. Korte, PhD, Michele C. Miller, and Earl J. Reisdorff, MD
Abstract Objectives: The American Board of Emergency Medicine (ABEM) Maintenance of Certification (MOC) program requires every ABEM-certified physician to attest to participating in a quality improvement (QI) activity every 5 years. Understanding the type and frequency of these QI activities could inform the emergency medicine community about the variety of QI activities in which emergency physicians (EPs) are involved. These QI activities could provide ideas for the development of additional quality measures. Methods: This was a retrospective descriptive study of self-reported QI activity attestations from the ABEM MOC program during 2013. Attestations were provided by ABEM-certified EPs using the ABEM MOC website. The type, number, and cumulative frequency of activities are reported. Results: ABEM received 9,380 attestations for QI activities in 91 different categories. The three most commonly reported activities were acute myocardial infarction-percutaneous coronary intervention within 90 minutes of arrival (includes door-to-balloon time), door-to-doctor times, and throughput time measures. These three activities comprised 36.4% of attestations. More than half (54.4%) of the attestations were captured by the five most frequently attested activities, 67.1% by the top seven categories, and 89.9% by the top 21 categories. Of these 21 categories, 10 involved clinical protocols, nine were time-centered measures, and two were patient-centered activities. Conclusions: This report demonstrates that diverse QI activities occur in emergency departments (EDs) across the United States. The majority of reported projects are nested in a few categories, following recognized areas of emphasis in emergency care, particularly in areas using time-sensitive metrics. ACADEMIC EMERGENCY MEDICINE 2015;22:367–372 © 2015 by the Society for Academic Emergency Medicine
A
ll medical specialty certifying boards that are members of the American Board of Medical Specialties (ABMS) are required to have Maintenance of Certification (MOC) programs. The four parts of MOC are: 1) professional standing, 2) lifelong learning and self-assessment, 3) cognitive expertise examination, and 4) assessment of practice performance (APP).1 To maintain certification, all four of these components must be successfully met within prescribed timelines. The American Board of Emergency Medicine
(ABEM) MOC program requires ABEM-certified physicians to attest to participating in practice-based quality improvement (QI) activities as a component of the APP improvement requirement.2 The APP improvement requirement is a clinically focused QI activity. ABEM certification is for 10 years, divided into two 5-year periods. The ABEM MOC program was introduced in 2004, and the APP reporting requirement was phased in starting in 2010. An APP QI activity must be completed during each 5-year period. For any given
From the Department of Emergency Medicine, Oakland University William Beaumont School of Medicine (TK), Royal Oak, MI; the Department of Emergency Medicine, Norwalk Hospital (MLC), Norwalk, CT; and the American Board of Emergency Medicine (RCK, MCM, EJR), East Lansing, MI. Received September 12, 2014; revision received October 16, 2014; accepted October 18, 2014. Drs. Kowalenko and Carius are directors of the American Board of Emergency Medicine (ABEM; no financial compensation is received for the reported activity). Dr. Korte, Dr. Reisdorff, and Ms. Miller are employed by ABEM. Supervising Editor: John Burton, MD. Address for correspondence and reprints: Terry Kowalenko, MD; e-mail: [email protected].
© 2015 by the Society for Academic Emergency Medicine doi: 10.1111/acem.12597
ISSN 1069-6563 PII ISSN 1069-6563583
367 367
368
year, about 20% of ABEM-certified physicians need to attest to completing practice improvement activities. Of this 20%, 10% need to complete the requirement during the first 5-year period to maintain the status of “meeting MOC requirements,” and 10% need to meet this requirement during the second 5-year period to prevent their certification from lapsing. As of December 31, 2013, there were 31,154 ABEM diplomates, of whom approximately 6,200 had APP QI activity requirements in 2013.3 Beginning in 2012, ABEM diplomates could apply for a financial incentive through the Physician Quality Reporting System (PQRS) MOC enhanced reimbursement program. This incentive was also available in 2013. To be eligible for this incentive, diplomates participate in MOC “more frequently.” To meet the more frequently requirement, an APP QI activity must occur in every year for which the diplomate applies for the bonus. Because of this annual requirement, the number of APP attestations could exceed the anticipated 20% annual reporting. The APP QI activity attestations provide an indication of the various types of QI activities in which emergency physicians (EPs) are involved. This information reflects the degree to which physicians are engaged in certain QI activities and could provide guidance to the development of future quality measures. This is particularly applicable for activities to which no PQRS or core measure is currently linked. Given the large number of physicians who report QI activities to meet ABEM MOC requirements, the goal of this study was to report the type and frequency of QI activities occurring in emergency departments (EDs) across the country. Understanding this information could inform the emergency medicine community about the variety of QI activities in which EPs are involved. These QI activities could provide ideas for the development of additional quality measures. METHODS Study Design This study was a retrospective descriptive review of all MOC APP QI activities reported during 2013. None were excluded. The study was approved as exempt research by the Beaumont Health System Research Institute Human Investigation Committee. Study Setting and Population All attestations were self-reported by ABEM-certified physicians. As part of the ABEM MOC program, every clinically active certified EP is required to report an APP QI activity at least once every 5 years. In addition, ABEM diplomates seeking to qualify for the PQRS MOC enhanced bonus would need to complete an attestation each year they apply. Study Protocol When reporting an APP QI activity, physicians were presented various options of reporting, including a drop-down menu of core measures, a drop-down menu of PQRS measures, preset common QI activities, and a free-text option for typing in a unique or custom-devel-
Kowalenko et al. • ABEM QUALITY IMPROVEMENT ATTESTATIONS
oped QI activity (Table 1). Details about the QI activity were not requested, nor were they part of the reporting requirements. As part of the attestation, the physician needed to confirm four steps: 1) that an initial measurement occurred, 2) that the sample was evaluated against a benchmark or standard (including possibly an internal baseline measurement), 3) that there was an improvement intervention, and 4) that a resampling had occurred. Improvement in process or patient outcomes was anticipated, but proof of improvement was not required. Details about the nature of the project or the processes involved were not solicited, nor were activity data submitted. Measures Data for this study consisted of attestations by individual ABEM-certified physicians about their APP QI activities. These attestations were self-entered by physicians. The number of attestations in each preset category was analyzed and reported by a single member of the ABEM data management team. The data are stored in a secure ABEM MOC website data repository. Data from preset-listed QI activities were reported in aggregate. The data from the “other” option were placed into preset categories when possible. For QI activities not listed in preset categories, activities were grouped together when possible. Any activity with two or more attestations was assigned a QI activity category. This was done by a single investigator (EJR) and then reviewed by the chair and vice chair of the ABEM MOC Committee. No adjustments were made to the original categories. Any attestation for which there were fewer than 25 attestations was placed into the “other” category (Table 2). The 21 categories, comprising approximately 90% of attestations, were then grouped into three divisions for further analysis. The divisions were time-centered activities, clinical protocols, and patient-centered activities. These divisions were identified post hoc after a qualitative review of the most commonly reported categories. Grouping activities into these divisions was done to determine if there were any characterizations that might assist in further understanding the results. Given the post hoc qualitative analysis, the divisions have a subjective element and potentially some degree of overlap. The PQRS and core measures were developed largely through the National Quality Foundation measurement development process, as well as the American Medical Association Physician Consortium for Performance Improvement development process. These measures underwent an arduous development, review, and approval process.4,5 The number of attestations that involved 2013 PQRS and core measures was determined. The type, number, and cumulative frequency of QI activities were reported. Data Analysis Descriptive statistics were used. Categories were assigned as described above. A simple count of the number of reports and cumulative frequency was determined. The ABEM MOC website, into which the attestations were entered, used an SQL 2008R2 database. The data queries were extracted using Microsoft Access
ACADEMIC EMERGENCY MEDICINE • March 2015, Vol. 22, No. 3 • www.aemj.org
369
Table 1 Part IV Attestation Menu Menu Options
Drop-down List Entries
Physician Quality Reporting System 12-lead ECG performed for nontraumatic chest pain 12-lead ECG performed for syncope Acute otitis externa: systemic antimicrobial therapy—avoidance of inappropriate use Acute otitis externa: topical therapy Aspirin at arrival for acute myocardial infarction Community-acquired pneumonia: empiric antibiotic Community-acquired pneumonia: vital signs Heart failure: left ventricular function testing Prevention of catheter-related bloodstream infections: central venous catheterization insertion protocol Preventive care and screening: screening for high blood pressure Rh immunoglobulin for Rh-negative pregnant women at risk of fetal blood exposure Stroke & stroke rehabilitation: screening for dysphagia Stroke & stroke rehabilitation: thrombolytic therapy Stroke & stroke rehabilitation: deep vein thrombosis prophylaxis for stroke or intracranial hemorrhage Ultrasound determination of pregnancy location—pregnant patients with abdominal pain Core measures Acute myocardial infarction: aspirin on arrival Acute myocardial infarction: ACE inhibitor or ARB given for left ventricular systolic dysfunction Acute myocardial infarction: beta-blocker within 24 hours of arrival Acute myocardial infarction: fibrinolytic within 30 minutes of arrival Acute myocardial infarction: percutaneous coronary intervention within 90 minutes of arrival Pneumonia: oxygenation assessment Pneumonia: blood cultures for intensive care unit Pneumonia: blood culture before first antibiotic Pneumonia: antibiotic timing (within 4 hours; within 8 hours) Sepsis pathways Asthma pathways Throughput time measures Door-to-balloon time Stroke protocol/pathways activation Door-to-doctor times Left without being seen Unscheduled return visits Patient call back program: assessment of clinical care given (not a satisfaction survey) Other (free text option) ACE = angiotensin-converting enzyme; ARB = angiotensin receptor blockers; ECG = electrocardiogram.
(Windows 7) and then transferred as deidentified aggregate data to Microsoft Excel for determining frequency data. RESULTS During 2013, ABEM received 9,380 attestations to APP QI activities. As of December 31, 2013, there were 31,154 ABEM diplomates; of these, about 20% (an estimated 6,200) would have had an APP reporting requirement. For the 9,380 attestations, 91 categories were defined (Table 1). The three most commonly reported quality activities were: 1) acute myocardial infarction-percutaneous coronary intervention within 90 minutes of arrival (includes door-to-balloon time activities), 2) door-to-doctor times, and 3) throughput time measures. These three activities comprised 36.4% of all attestations; all three are time-sensitive metrics. More than half of the attestations were captured by the five most frequently attested activities, roughly two-thirds (67.1%) of all attestations were captured by the seven most frequent categories, and about 90% (89.9%) of all attestations were captured by the 21 most common categories. Of these 21 categories, 10 (47.6%) involved clinical
protocols, nine (42.9%) were time-sensitive measures, and two (9.5%) were patient-centered activities (Figure 1). There were 33 total attestation categories, including the “other” category that combined all attestations for which there were fewer than 25 attestations per item. This category contained 609 attestations (6.5% of all attestations). There were 11 preset and drop-down items that were included in the “other” category (e.g., acute otitis externa: topical therapy). There were 5,435 attestations (57.9%) that involved PQRS and core measures. There were 8,097 (86.3%) reports that used preset and drop-down items and 1,283 (13.7%) that were free-text entered. There were some free-text attestations that were also in the preset and drop-down lists. When adding the additional freetext attestations, there were 8,176 attestations (87.2% of total attestations) that involved preset and drop-down items. DISCUSSION To the best of our knowledge, this review is the first report of its type and shows a broad diversity of quality activities by ABEM-certified EPs. The most common activities involved time-sensitive metrics. This is not
370
Kowalenko et al. • ABEM QUALITY IMPROVEMENT ATTESTATIONS
Table 2 Most Frequent Attestations
Activity
Attestations
Percent
Cumulative Percent
Acute myocardial infarction: percutaneous coronary intervention within 90 minutes of arrival (includes door to balloon time activities) Door-to-doctor times Throughput time measures Acute myocardial infarction: aspirin on arrival Sepsis pathways (includes goal-directed care) Stroke care: protocols for emergent assessment and care for stroke patients (excludes thrombolytic use) Pneumonia: blood culture before first antibiotic Pneumonia: antibiotic timing (administration within preset time) Left without being seen Patient call-back program: assessment of quality of care (not satisfaction) 12-lead ECG for nontraumatic chest pain Review of unscheduled return visits Pneumonia: empiric antibiotic selection Pneumonia: oxygen saturation assessment General QI activities (including routine chart reviews, morbidity and mortality reviews, Lean projects, Six Sigma projects) Safe sign-outs and handoffs Pneumonia: blood cultures for intensive care unit patients 12-lead ECG for syncope Risk management activities (includes chart reviews of high-risk conditions) Asthma treatment pathways (including appropriate discharge medications) Evaluation and risk stratification of chest pain (including TIMI risk assessment) Chart and record completion (timeliness and thoroughness) Pneumonia: vital signs Assessment of patients with abdominal pain (including approaches to imaging) Assessment of suicidal risk Patient experience of care QI activities ECG to interpretation times CT of nontraumatic headache CT for pulmonary embolus (including frequency of use, approach to decisionmaking, integration of D-dimer in decision-making). Acute myocardial infarction emergency care including catheterization lab activation (approach to care excluding time-based goals) Door to ECG time ED ultrasound use (including quality audits, accuracy of image interpretation, and credentialing) Other
1,201
12.8
12.8
1,172 1,045 877 810 750
12.5 11.1 9.3 8.6 8.0
25.3 36.4 45.8 54.4 62.4
439 344 299 281 248 218 159 88 85
4.7 3.7 3.2 3.0 2.6 2.3 1.7 0.9 0.9
67.1 70.8 74.0 77.0 79.6 81.9 83.6 84.6 85.5
84 73 70 70 61 60 49 40 39 31 30 29 28 26
0.9 0.8 0.7 0.7 0.7 0.6 0.5 0.4 0.4 0.3 0.3 0.3 0.3 0.3
86.4 87.1 87.9 88.6 89.3 89.9 90.4 90.0 91.3 91.6 91.9 92.2 92.5 92.8
25
0.3
93.1
25 25
0.3 0.3
93.3 93.6
609
6.5
100.0
Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18.5 18.5 20 21 22 23 24 25 26 27 28 29 31 31 31 32
ECG = electrocardiogram; QI = quality improvement; TIMI = Thrombolysis in Myocardial Infarction. 50% 45%
46%
45%
40% 35% 30% 25% 20% 15%
9%
10% 5% 0% Clinical
Time
Paent
Clinical = clinical protocols; Time = time-sensitive activities; Patient = patient-centered activities.
Figure 1. General category types for top 90% of attestations. Clinical = clinical protocols; Time = time-sensitive activities; Patient = patient-centered activities.
surprising given the temporal demands of clinical practice in the ED. The majority of attestations were included in the five most frequently attested activities
(Table 2). There were many other time-based measurements (e.g., time to analgesia for long bone fracture) or activities that had time-sensitive components (stroke and stroke rehabilitation: thrombolytic therapy). The frequency of activities follows that of a powerlaw curve. This is particularly noted in the tail, and calls attention to the broad but thin nature of departmentspecific QI activities. Data sets such as these can be used to help guide the development of new quality measures, particularly when reviewing QI activities for which CMS-approved measures do not exist. The ABEM MOC program requires that physicians be engaged in QI projects that are relevant to their clinical activity. At this time, only clinically active physicians can report on APP QI activities. The physician must follow a four-step improvement process of measuring current performance regarding a cohort of the physician’s patients, comparing performance to a benchmark, adjusting performance after an improvement intervention, and resampling his or her performance.1 Initially, ABEM required that the cohort should involve ten of the physician’s patients in the first sample and 10
ACADEMIC EMERGENCY MEDICINE • March 2015, Vol. 22, No. 3 • www.aemj.org
additional patients in the second sample. However, ABEM recognized the value in QI activities that might involve low-frequency, high-acuity medical conditions. In this instance, cohorts with fewer than 10 patients can be used, as long as the physician is involved in the care of patients in a group cohort that is being examined on a departmental level. Also, regarding the application of a performance benchmark, nationally developed, expert consensus benchmarks or peer-review published guidelines are acceptable. The APP attestation process is fairly simple; it involves entering a secure area of the ABEM website using an ABEM-specific user identification and password. Once signed in, diplomates navigate to a page that outlines the physician’s personal MOC requirements. If an APP attestation is required, the physician can readily opt to go to the APP attestation page and select the quality activity to which they wish to attest. The physician must also attest that the aforementioned four-step QI process has been followed. Finally, the physician must provide a name and contact information for a verifier. The types of activities that are performed tend to be commonly occurring QI activities for most, if not all, EDs. ABEM allows EDs and physicians to self-select an activity that will be relevant to the physician or the department. Thus, the relevance of an APP activity is solely determined by the reporting physician and his or her ED. All QI programs that follow processes akin to the plan-do-check-act cycle would likely meet the ABEM APP requirements.6 Moreover, any Lean or Six Sigma project would likely meet this requirement. ABEM’s philosophy is that EPs who are engaged in QI activities should receive credit for those activities without being encumbered by additional work. Given the ubiquitous interest by EPs to adhere to core measures and PQRS metrics, it is difficult to imagine that any hospital-based EP is not already participating in a department-based QI program. The frequent focus on PQRS and core measures is reflected by the majority of QI attestations involving these measure sets. The Physician Quality Reporting Initiative program was developed by the Centers for Medicaid and Medicare Services in 2006 and later renamed the PQRS. As a consequence of the Affordable Care Act, there is a greater emphasis on physician quality reporting and the adherence to quality measures. This focus will be yet greater with the implementation of the value-based modifier and the changing implications of PQRS for physician payment. It is important to have measures that are relevant to the clinical practice of EM. If a quality measure is rarely reported, it could be a proxy for a limited clinical relevance (e.g., measures about otitis externa). On the other hand, frequently reported QI activities (e.g., safe sign-outs and handoffs) that are not PQRS or core measures might signal an area where a relevant quality measure could be developed. Specifically, over 42% of QI activities were not anchored to either a PQRS or a core measure.
371
20% of all attestations were verified by contacting a designated verifier. This is often an ED director or director of QI. None of the verifications received by the ABEM offices for 2013 disputed any attestations. Self-reporting can also introduce social desirability bias. Social desirability bias can interfere with the interpretation of average tendencies as well as confound individual differences between respondents. However, this form of bias is less prevalent for this study, because all clinically active ABEM diplomates are required to report QI activities. Nonetheless, social desirability bias might alter the type of QI activity that was reported. The magnitude of the type of an activity should be cautiously interpreted. The number of attestations does not reflect the number of ED–based QI activities. There were instances when multiple physicians from the same ED attested to the same activity. Requiring two activities to create a specific category was an arbitrary threshold. A significant limitation is that only a single QI activity can be entered when using the preset and drop-down menus. Thus, this report gives only a sample of activities in which physicians are engaged, not the entire number or types of activities. EPs are often involved in multiple QI activities. Thus, the degree of participation was not captured in this report and likely represents a significant underestimation of the quantity of QI activities in which these EPs were involved. There was likely a response bias as a result of having predesignated measures and drop-down menus. Because it is easier to register by clicking on an existing field and measure, physicians might more readily attest to a dropdown or preset activity. Of course, it is also likely that an ED would be already engaged in PQRS or core measure reporting. Thus, projects to enhance adherence to PQRS and core measures might be more commonplace. Additional drop-down items beyond the PQRS and core measures were suggested by clinically active EPs. Using the “other” option required more effort involving free-text typing. However, 13.7% physicians opted to use the freetext option reporting using the “other” function. The attestation menu was developed in 2010 and not revised until March 2014. The gradual changes in PQRS and core measures over a 3-year span were not reflected in the attestation menu. This could have affected the types and frequency of reported quality activities. ABEM now revises the menu annually to capture changes in quality measures and commonly reported activities. This study only included clinically active, ABEM-certified physicians. The American Osteopathic Board of Emergency Medicine (AOBEM) has a similar program: Osteopathic Continuous Certification (OCC). The scope and magnitude of quality activities reported through OCC are uncertain. This report also does not include any quality activities of physicians working in EDs who are not certified by ABEM or AOBEM. CONCLUSIONS
LIMITATIONS The data were self-reported by physicians; there was no attempt to confirm the accuracy of every attestation. Nonetheless, as a part of the ABEM MOC program,
This report demonstrates that widespread and varied quality improvement activities occurred in emergency departments across the United States. The majority of reported projects are nested in a few categories,
372
following recognized areas of emphasis in emergency care, particularly in areas that use time-sensitive metrics. References 1. Nelson RN. Demystifying maintenance of certification. Ann Emerg Med 2014;63:467–70. 2. American Board of Emergency Medicine. Patient Care Practice Improvement. Available at: https://www. abem.org/public/abem-maintenance-of-certification(moc)/moc-assessment-of-practice-performance/patientcare-practice-improvement. Accessed Dec 3, 2014. 3. American Board of Emergency Medicine. 2013–2014 annual report. East Lansing, MI: American Board of Emergency Medicine, 2014.
Kowalenko et al. • ABEM QUALITY IMPROVEMENT ATTESTATIONS
4. National Quality Forum. Measure Evaluation Criteria. Available at: https://www.qualityforum.org/docs/ measure_evaluation_criteria.aspx. Accessed Dec 3, 2014. 5. American Medical Association Physician Consortium for Performance Improvement. About Our Measure Development: Process. Available at: http://www.amaassn.org/ama/pub/physician-resources/physician-con sortium-performance-improvement/pcpi-measures/ about-measure-development/process.page. Accessed Dec 3, 2014. 6. Liker JK. The Toyota Way: 14 Management Principles From the World’s Greatest Manufacturer. New York, NY: McGraw-Hill, 2004, pp 263–4.
Emergency Department Quality Improvement Activity: An Inventory From the American Board of Emergency Medicine Maintenance of Certification Program Terry Kowalenko, MD, Michael L. Carius, MD, Robert C. Korte, PhD, Michele C. Miller, and Earl J. Reisdorff, MD
Abstract Objectives: The American Board of Emergency Medicine (ABEM) Maintenance of Certification (MOC) program requires every ABEM-certified physician to attest to participating in a quality improvement (QI) activity every 5 years. Understanding the type and frequency of these QI activities could inform the emergency medicine community about the variety of QI activities in which emergency physicians (EPs) are involved. These QI activities could provide ideas for the development of additional quality measures. Methods: This was a retrospective descriptive study of self-reported QI activity attestations from the ABEM MOC program during 2013. Attestations were provided by ABEM-certified EPs using the ABEM MOC website. The type, number, and cumulative frequency of activities are reported. Results: ABEM received 9,380 attestations for QI activities in 91 different categories. The three most commonly reported activities were acute myocardial infarction-percutaneous coronary intervention within 90 minutes of arrival (includes door-to-balloon time), door-to-doctor times, and throughput time measures. These three activities comprised 36.4% of attestations. More than half (54.4%) of the attestations were captured by the five most frequently attested activities, 67.1% by the top seven categories, and 89.9% by the top 21 categories. Of these 21 categories, 10 involved clinical protocols, nine were time-centered measures, and two were patient-centered activities. Conclusions: This report demonstrates that diverse QI activities occur in emergency departments (EDs) across the United States. The majority of reported projects are nested in a few categories, following recognized areas of emphasis in emergency care, particularly in areas using time-sensitive metrics. ACADEMIC EMERGENCY MEDICINE 2015;22:367–372 © 2015 by the Society for Academic Emergency Medicine
A
ll medical specialty certifying boards that are members of the American Board of Medical Specialties (ABMS) are required to have Maintenance of Certification (MOC) programs. The four parts of MOC are: 1) professional standing, 2) lifelong learning and self-assessment, 3) cognitive expertise examination, and 4) assessment of practice performance (APP).1 To maintain certification, all four of these components must be successfully met within prescribed timelines. The American Board of Emergency Medicine
(ABEM) MOC program requires ABEM-certified physicians to attest to participating in practice-based quality improvement (QI) activities as a component of the APP improvement requirement.2 The APP improvement requirement is a clinically focused QI activity. ABEM certification is for 10 years, divided into two 5-year periods. The ABEM MOC program was introduced in 2004, and the APP reporting requirement was phased in starting in 2010. An APP QI activity must be completed during each 5-year period. For any given
From the Department of Emergency Medicine, Oakland University William Beaumont School of Medicine (TK), Royal Oak, MI; the Department of Emergency Medicine, Norwalk Hospital (MLC), Norwalk, CT; and the American Board of Emergency Medicine (RCK, MCM, EJR), East Lansing, MI. Received September 12, 2014; revision received October 16, 2014; accepted October 18, 2014. Drs. Kowalenko and Carius are directors of the American Board of Emergency Medicine (ABEM; no financial compensation is received for the reported activity). Dr. Korte, Dr. Reisdorff, and Ms. Miller are employed by ABEM. Supervising Editor: John Burton, MD. Address for correspondence and reprints: Terry Kowalenko, MD; e-mail: [email protected].
© 2015 by the Society for Academic Emergency Medicine doi: 10.1111/acem.12597
ISSN 1069-6563 PII ISSN 1069-6563583
367 367
368
year, about 20% of ABEM-certified physicians need to attest to completing practice improvement activities. Of this 20%, 10% need to complete the requirement during the first 5-year period to maintain the status of “meeting MOC requirements,” and 10% need to meet this requirement during the second 5-year period to prevent their certification from lapsing. As of December 31, 2013, there were 31,154 ABEM diplomates, of whom approximately 6,200 had APP QI activity requirements in 2013.3 Beginning in 2012, ABEM diplomates could apply for a financial incentive through the Physician Quality Reporting System (PQRS) MOC enhanced reimbursement program. This incentive was also available in 2013. To be eligible for this incentive, diplomates participate in MOC “more frequently.” To meet the more frequently requirement, an APP QI activity must occur in every year for which the diplomate applies for the bonus. Because of this annual requirement, the number of APP attestations could exceed the anticipated 20% annual reporting. The APP QI activity attestations provide an indication of the various types of QI activities in which emergency physicians (EPs) are involved. This information reflects the degree to which physicians are engaged in certain QI activities and could provide guidance to the development of future quality measures. This is particularly applicable for activities to which no PQRS or core measure is currently linked. Given the large number of physicians who report QI activities to meet ABEM MOC requirements, the goal of this study was to report the type and frequency of QI activities occurring in emergency departments (EDs) across the country. Understanding this information could inform the emergency medicine community about the variety of QI activities in which EPs are involved. These QI activities could provide ideas for the development of additional quality measures. METHODS Study Design This study was a retrospective descriptive review of all MOC APP QI activities reported during 2013. None were excluded. The study was approved as exempt research by the Beaumont Health System Research Institute Human Investigation Committee. Study Setting and Population All attestations were self-reported by ABEM-certified physicians. As part of the ABEM MOC program, every clinically active certified EP is required to report an APP QI activity at least once every 5 years. In addition, ABEM diplomates seeking to qualify for the PQRS MOC enhanced bonus would need to complete an attestation each year they apply. Study Protocol When reporting an APP QI activity, physicians were presented various options of reporting, including a drop-down menu of core measures, a drop-down menu of PQRS measures, preset common QI activities, and a free-text option for typing in a unique or custom-devel-
Kowalenko et al. • ABEM QUALITY IMPROVEMENT ATTESTATIONS
oped QI activity (Table 1). Details about the QI activity were not requested, nor were they part of the reporting requirements. As part of the attestation, the physician needed to confirm four steps: 1) that an initial measurement occurred, 2) that the sample was evaluated against a benchmark or standard (including possibly an internal baseline measurement), 3) that there was an improvement intervention, and 4) that a resampling had occurred. Improvement in process or patient outcomes was anticipated, but proof of improvement was not required. Details about the nature of the project or the processes involved were not solicited, nor were activity data submitted. Measures Data for this study consisted of attestations by individual ABEM-certified physicians about their APP QI activities. These attestations were self-entered by physicians. The number of attestations in each preset category was analyzed and reported by a single member of the ABEM data management team. The data are stored in a secure ABEM MOC website data repository. Data from preset-listed QI activities were reported in aggregate. The data from the “other” option were placed into preset categories when possible. For QI activities not listed in preset categories, activities were grouped together when possible. Any activity with two or more attestations was assigned a QI activity category. This was done by a single investigator (EJR) and then reviewed by the chair and vice chair of the ABEM MOC Committee. No adjustments were made to the original categories. Any attestation for which there were fewer than 25 attestations was placed into the “other” category (Table 2). The 21 categories, comprising approximately 90% of attestations, were then grouped into three divisions for further analysis. The divisions were time-centered activities, clinical protocols, and patient-centered activities. These divisions were identified post hoc after a qualitative review of the most commonly reported categories. Grouping activities into these divisions was done to determine if there were any characterizations that might assist in further understanding the results. Given the post hoc qualitative analysis, the divisions have a subjective element and potentially some degree of overlap. The PQRS and core measures were developed largely through the National Quality Foundation measurement development process, as well as the American Medical Association Physician Consortium for Performance Improvement development process. These measures underwent an arduous development, review, and approval process.4,5 The number of attestations that involved 2013 PQRS and core measures was determined. The type, number, and cumulative frequency of QI activities were reported. Data Analysis Descriptive statistics were used. Categories were assigned as described above. A simple count of the number of reports and cumulative frequency was determined. The ABEM MOC website, into which the attestations were entered, used an SQL 2008R2 database. The data queries were extracted using Microsoft Access
ACADEMIC EMERGENCY MEDICINE • March 2015, Vol. 22, No. 3 • www.aemj.org
369
Table 1 Part IV Attestation Menu Menu Options
Drop-down List Entries
Physician Quality Reporting System 12-lead ECG performed for nontraumatic chest pain 12-lead ECG performed for syncope Acute otitis externa: systemic antimicrobial therapy—avoidance of inappropriate use Acute otitis externa: topical therapy Aspirin at arrival for acute myocardial infarction Community-acquired pneumonia: empiric antibiotic Community-acquired pneumonia: vital signs Heart failure: left ventricular function testing Prevention of catheter-related bloodstream infections: central venous catheterization insertion protocol Preventive care and screening: screening for high blood pressure Rh immunoglobulin for Rh-negative pregnant women at risk of fetal blood exposure Stroke & stroke rehabilitation: screening for dysphagia Stroke & stroke rehabilitation: thrombolytic therapy Stroke & stroke rehabilitation: deep vein thrombosis prophylaxis for stroke or intracranial hemorrhage Ultrasound determination of pregnancy location—pregnant patients with abdominal pain Core measures Acute myocardial infarction: aspirin on arrival Acute myocardial infarction: ACE inhibitor or ARB given for left ventricular systolic dysfunction Acute myocardial infarction: beta-blocker within 24 hours of arrival Acute myocardial infarction: fibrinolytic within 30 minutes of arrival Acute myocardial infarction: percutaneous coronary intervention within 90 minutes of arrival Pneumonia: oxygenation assessment Pneumonia: blood cultures for intensive care unit Pneumonia: blood culture before first antibiotic Pneumonia: antibiotic timing (within 4 hours; within 8 hours) Sepsis pathways Asthma pathways Throughput time measures Door-to-balloon time Stroke protocol/pathways activation Door-to-doctor times Left without being seen Unscheduled return visits Patient call back program: assessment of clinical care given (not a satisfaction survey) Other (free text option) ACE = angiotensin-converting enzyme; ARB = angiotensin receptor blockers; ECG = electrocardiogram.
(Windows 7) and then transferred as deidentified aggregate data to Microsoft Excel for determining frequency data. RESULTS During 2013, ABEM received 9,380 attestations to APP QI activities. As of December 31, 2013, there were 31,154 ABEM diplomates; of these, about 20% (an estimated 6,200) would have had an APP reporting requirement. For the 9,380 attestations, 91 categories were defined (Table 1). The three most commonly reported quality activities were: 1) acute myocardial infarction-percutaneous coronary intervention within 90 minutes of arrival (includes door-to-balloon time activities), 2) door-to-doctor times, and 3) throughput time measures. These three activities comprised 36.4% of all attestations; all three are time-sensitive metrics. More than half of the attestations were captured by the five most frequently attested activities, roughly two-thirds (67.1%) of all attestations were captured by the seven most frequent categories, and about 90% (89.9%) of all attestations were captured by the 21 most common categories. Of these 21 categories, 10 (47.6%) involved clinical
protocols, nine (42.9%) were time-sensitive measures, and two (9.5%) were patient-centered activities (Figure 1). There were 33 total attestation categories, including the “other” category that combined all attestations for which there were fewer than 25 attestations per item. This category contained 609 attestations (6.5% of all attestations). There were 11 preset and drop-down items that were included in the “other” category (e.g., acute otitis externa: topical therapy). There were 5,435 attestations (57.9%) that involved PQRS and core measures. There were 8,097 (86.3%) reports that used preset and drop-down items and 1,283 (13.7%) that were free-text entered. There were some free-text attestations that were also in the preset and drop-down lists. When adding the additional freetext attestations, there were 8,176 attestations (87.2% of total attestations) that involved preset and drop-down items. DISCUSSION To the best of our knowledge, this review is the first report of its type and shows a broad diversity of quality activities by ABEM-certified EPs. The most common activities involved time-sensitive metrics. This is not
370
Kowalenko et al. • ABEM QUALITY IMPROVEMENT ATTESTATIONS
Table 2 Most Frequent Attestations
Activity
Attestations
Percent
Cumulative Percent
Acute myocardial infarction: percutaneous coronary intervention within 90 minutes of arrival (includes door to balloon time activities) Door-to-doctor times Throughput time measures Acute myocardial infarction: aspirin on arrival Sepsis pathways (includes goal-directed care) Stroke care: protocols for emergent assessment and care for stroke patients (excludes thrombolytic use) Pneumonia: blood culture before first antibiotic Pneumonia: antibiotic timing (administration within preset time) Left without being seen Patient call-back program: assessment of quality of care (not satisfaction) 12-lead ECG for nontraumatic chest pain Review of unscheduled return visits Pneumonia: empiric antibiotic selection Pneumonia: oxygen saturation assessment General QI activities (including routine chart reviews, morbidity and mortality reviews, Lean projects, Six Sigma projects) Safe sign-outs and handoffs Pneumonia: blood cultures for intensive care unit patients 12-lead ECG for syncope Risk management activities (includes chart reviews of high-risk conditions) Asthma treatment pathways (including appropriate discharge medications) Evaluation and risk stratification of chest pain (including TIMI risk assessment) Chart and record completion (timeliness and thoroughness) Pneumonia: vital signs Assessment of patients with abdominal pain (including approaches to imaging) Assessment of suicidal risk Patient experience of care QI activities ECG to interpretation times CT of nontraumatic headache CT for pulmonary embolus (including frequency of use, approach to decisionmaking, integration of D-dimer in decision-making). Acute myocardial infarction emergency care including catheterization lab activation (approach to care excluding time-based goals) Door to ECG time ED ultrasound use (including quality audits, accuracy of image interpretation, and credentialing) Other
1,201
12.8
12.8
1,172 1,045 877 810 750
12.5 11.1 9.3 8.6 8.0
25.3 36.4 45.8 54.4 62.4
439 344 299 281 248 218 159 88 85
4.7 3.7 3.2 3.0 2.6 2.3 1.7 0.9 0.9
67.1 70.8 74.0 77.0 79.6 81.9 83.6 84.6 85.5
84 73 70 70 61 60 49 40 39 31 30 29 28 26
0.9 0.8 0.7 0.7 0.7 0.6 0.5 0.4 0.4 0.3 0.3 0.3 0.3 0.3
86.4 87.1 87.9 88.6 89.3 89.9 90.4 90.0 91.3 91.6 91.9 92.2 92.5 92.8
25
0.3
93.1
25 25
0.3 0.3
93.3 93.6
609
6.5
100.0
Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18.5 18.5 20 21 22 23 24 25 26 27 28 29 31 31 31 32
ECG = electrocardiogram; QI = quality improvement; TIMI = Thrombolysis in Myocardial Infarction. 50% 45%
46%
45%
40% 35% 30% 25% 20% 15%
9%
10% 5% 0% Clinical
Time
Paent
Clinical = clinical protocols; Time = time-sensitive activities; Patient = patient-centered activities.
Figure 1. General category types for top 90% of attestations. Clinical = clinical protocols; Time = time-sensitive activities; Patient = patient-centered activities.
surprising given the temporal demands of clinical practice in the ED. The majority of attestations were included in the five most frequently attested activities
(Table 2). There were many other time-based measurements (e.g., time to analgesia for long bone fracture) or activities that had time-sensitive components (stroke and stroke rehabilitation: thrombolytic therapy). The frequency of activities follows that of a powerlaw curve. This is particularly noted in the tail, and calls attention to the broad but thin nature of departmentspecific QI activities. Data sets such as these can be used to help guide the development of new quality measures, particularly when reviewing QI activities for which CMS-approved measures do not exist. The ABEM MOC program requires that physicians be engaged in QI projects that are relevant to their clinical activity. At this time, only clinically active physicians can report on APP QI activities. The physician must follow a four-step improvement process of measuring current performance regarding a cohort of the physician’s patients, comparing performance to a benchmark, adjusting performance after an improvement intervention, and resampling his or her performance.1 Initially, ABEM required that the cohort should involve ten of the physician’s patients in the first sample and 10
ACADEMIC EMERGENCY MEDICINE • March 2015, Vol. 22, No. 3 • www.aemj.org
additional patients in the second sample. However, ABEM recognized the value in QI activities that might involve low-frequency, high-acuity medical conditions. In this instance, cohorts with fewer than 10 patients can be used, as long as the physician is involved in the care of patients in a group cohort that is being examined on a departmental level. Also, regarding the application of a performance benchmark, nationally developed, expert consensus benchmarks or peer-review published guidelines are acceptable. The APP attestation process is fairly simple; it involves entering a secure area of the ABEM website using an ABEM-specific user identification and password. Once signed in, diplomates navigate to a page that outlines the physician’s personal MOC requirements. If an APP attestation is required, the physician can readily opt to go to the APP attestation page and select the quality activity to which they wish to attest. The physician must also attest that the aforementioned four-step QI process has been followed. Finally, the physician must provide a name and contact information for a verifier. The types of activities that are performed tend to be commonly occurring QI activities for most, if not all, EDs. ABEM allows EDs and physicians to self-select an activity that will be relevant to the physician or the department. Thus, the relevance of an APP activity is solely determined by the reporting physician and his or her ED. All QI programs that follow processes akin to the plan-do-check-act cycle would likely meet the ABEM APP requirements.6 Moreover, any Lean or Six Sigma project would likely meet this requirement. ABEM’s philosophy is that EPs who are engaged in QI activities should receive credit for those activities without being encumbered by additional work. Given the ubiquitous interest by EPs to adhere to core measures and PQRS metrics, it is difficult to imagine that any hospital-based EP is not already participating in a department-based QI program. The frequent focus on PQRS and core measures is reflected by the majority of QI attestations involving these measure sets. The Physician Quality Reporting Initiative program was developed by the Centers for Medicaid and Medicare Services in 2006 and later renamed the PQRS. As a consequence of the Affordable Care Act, there is a greater emphasis on physician quality reporting and the adherence to quality measures. This focus will be yet greater with the implementation of the value-based modifier and the changing implications of PQRS for physician payment. It is important to have measures that are relevant to the clinical practice of EM. If a quality measure is rarely reported, it could be a proxy for a limited clinical relevance (e.g., measures about otitis externa). On the other hand, frequently reported QI activities (e.g., safe sign-outs and handoffs) that are not PQRS or core measures might signal an area where a relevant quality measure could be developed. Specifically, over 42% of QI activities were not anchored to either a PQRS or a core measure.
371
20% of all attestations were verified by contacting a designated verifier. This is often an ED director or director of QI. None of the verifications received by the ABEM offices for 2013 disputed any attestations. Self-reporting can also introduce social desirability bias. Social desirability bias can interfere with the interpretation of average tendencies as well as confound individual differences between respondents. However, this form of bias is less prevalent for this study, because all clinically active ABEM diplomates are required to report QI activities. Nonetheless, social desirability bias might alter the type of QI activity that was reported. The magnitude of the type of an activity should be cautiously interpreted. The number of attestations does not reflect the number of ED–based QI activities. There were instances when multiple physicians from the same ED attested to the same activity. Requiring two activities to create a specific category was an arbitrary threshold. A significant limitation is that only a single QI activity can be entered when using the preset and drop-down menus. Thus, this report gives only a sample of activities in which physicians are engaged, not the entire number or types of activities. EPs are often involved in multiple QI activities. Thus, the degree of participation was not captured in this report and likely represents a significant underestimation of the quantity of QI activities in which these EPs were involved. There was likely a response bias as a result of having predesignated measures and drop-down menus. Because it is easier to register by clicking on an existing field and measure, physicians might more readily attest to a dropdown or preset activity. Of course, it is also likely that an ED would be already engaged in PQRS or core measure reporting. Thus, projects to enhance adherence to PQRS and core measures might be more commonplace. Additional drop-down items beyond the PQRS and core measures were suggested by clinically active EPs. Using the “other” option required more effort involving free-text typing. However, 13.7% physicians opted to use the freetext option reporting using the “other” function. The attestation menu was developed in 2010 and not revised until March 2014. The gradual changes in PQRS and core measures over a 3-year span were not reflected in the attestation menu. This could have affected the types and frequency of reported quality activities. ABEM now revises the menu annually to capture changes in quality measures and commonly reported activities. This study only included clinically active, ABEM-certified physicians. The American Osteopathic Board of Emergency Medicine (AOBEM) has a similar program: Osteopathic Continuous Certification (OCC). The scope and magnitude of quality activities reported through OCC are uncertain. This report also does not include any quality activities of physicians working in EDs who are not certified by ABEM or AOBEM. CONCLUSIONS
LIMITATIONS The data were self-reported by physicians; there was no attempt to confirm the accuracy of every attestation. Nonetheless, as a part of the ABEM MOC program,
This report demonstrates that widespread and varied quality improvement activities occurred in emergency departments across the United States. The majority of reported projects are nested in a few categories,
372
following recognized areas of emphasis in emergency care, particularly in areas that use time-sensitive metrics. References 1. Nelson RN. Demystifying maintenance of certification. Ann Emerg Med 2014;63:467–70. 2. American Board of Emergency Medicine. Patient Care Practice Improvement. Available at: https://www. abem.org/public/abem-maintenance-of-certification(moc)/moc-assessment-of-practice-performance/patientcare-practice-improvement. Accessed Dec 3, 2014. 3. American Board of Emergency Medicine. 2013–2014 annual report. East Lansing, MI: American Board of Emergency Medicine, 2014.
Kowalenko et al. • ABEM QUALITY IMPROVEMENT ATTESTATIONS
4. National Quality Forum. Measure Evaluation Criteria. Available at: https://www.qualityforum.org/docs/ measure_evaluation_criteria.aspx. Accessed Dec 3, 2014. 5. American Medical Association Physician Consortium for Performance Improvement. About Our Measure Development: Process. Available at: http://www.amaassn.org/ama/pub/physician-resources/physician-con sortium-performance-improvement/pcpi-measures/ about-measure-development/process.page. Accessed Dec 3, 2014. 6. Liker JK. The Toyota Way: 14 Management Principles From the World’s Greatest Manufacturer. New York, NY: McGraw-Hill, 2004, pp 263–4.
