ANNALS OF EMERGENCY MEDICINE JOURNAL CLUB

Do More Rules Make Us Safer? Clinical Decision Rules, Patient Safety, and the Role of Emergency Physicians in Health Care Answers to the January 2014 Journal Club Questions Malkeet Gupta, MD, MS; Tyler W. Barrett, MD, MSCI; David L. Schriger, MD, MPH 0196-0644/$-see front matter Copyright © 2014 by the American College of Emergency Physicians. http://dx.doi.org/10.1016/j.annemergmed.2014.01.004

Editor’s Note: You are reading the 37th installment of Annals of Emergency Medicine Journal Club. This Journal Club refers the Perry et al1 article titled “Clinical Decision Rules to Rule Out Subarachnoid Hemorrhage for Acute Headache” that was published in JAMA. Information about Journal Club can be found at http://www.annemergmed.com/content/journalclub. Readers should recognize that these are suggested answers. We hope they are accurate; we know that they are not comprehensive. There are many other points that could be made about these questions or about the article in general. Questions are rated “novice” ( ), “intermediate” ( ), and “advanced ( ) so that individuals planning a journal club can assign the right question to the right student. The “novice” rating does not imply that a novice should be able to spontaneously answer the question. “Novice” means we expect that someone with little background should be able to do a bit of reading, formulate an answer, and teach the material to others. Intermediate and advanced questions also will likely require some reading and research, and that reading will be sufficiently difficult that some background in clinical epidemiology will be helpful in understanding the reading and concepts. We are interested in receiving feedback about this feature. Please e-mail [email protected] with your comments.

DISCUSSION POINTS 1. The authors state that “5.4% of confirmed subarachnoid hemorrhages were misdiagnosed during the patients’ initial emergency department [ED] assessment.”1 A. What source did the authors cite as the reference for this frequency of missed subarachnoid hemorrhage diagnoses? How does this source compare to other articles about the diagnosis of subarachnoid hemorrhage in terms of reported miss rate and study methodology? Does a subarachnoid hemorrhage misdiagnosis frequency of 1 in 20 seem reasonable in your ED? (For the purposes of this question, assume the same definition that the source authors use.) If not, what factors might decrease the number of missed subarachnoid hemorrhage diagnoses below that reported in the cited reference? B. How might different stakeholders (eg, patients, physicians, administrators, insurers) choose to define a “missed” diagnosis? How does, and how should, the risk of morbidity associated with a miss affect this definition?

774 Annals of Emergency Medicine

C. What are the implications for the practice of emergency medicine, depending on whose definition we choose to operationalize? How might resource availability in a particular health care setting affect how a “miss” is defined? 2. In this article, 26% of patients arrived by ambulance, and “arrival by ambulance” was one of the 4 variables in rule 2. The inclusion of this variable suggests that patients arriving by ambulance are at greater risk for subarachnoid hemorrhage. A. In some settings, patients arriving by ambulance are automatically triaged to higher-acuity beds. How might the patient’s location in the ED when treated by the clinician affect the evaluation that he receives? How might the “assignment bias” that occurs when low-acuity patients arriving by ambulance are preferentially placed into evaluation rooms rather than directed to the waiting room or a lower-acuity treatment area affect ED throughput and staffing models? Why might this be especially important for EDs that employ midlevel providers or resident moonlighters to staff the low-acuity areas? 3. The specificity of the Ottawa Subarachnoid Hemorrhage Rule is such that nearly 85% of patients with potentially concerning headache would require computed tomography (CT) and lumbar puncture. The authors acknowledge that “[t]he Ottawa Subarachnoid Hemorrhage Rule does not lead to a reduction of testing (ie, CT, lumbar puncture, or both) vs current practice; however, it may help to standardize which patients with acute headache require investigations, and its widespread use could help decrease missed subarachnoid hemorrhages.”1 A. The present accepted standard for a subarachnoid hemorrhage evaluation includes a nonenhanced CT study and, if the CT result is negative, a lumbar puncture. Using the data provided in the article, calculate the percentage of the entire cohort who underwent this complete evaluation. Do this using the data in Figure 1 and then repeat the exercise using the data in the tables. Are the estimates concordant? Can you reconcile them? What is your best estimate of how many patients received a

Volume 63, no. 6 : June 2014

Journal Club complete evaluation by the physicians, and how does this compare with the 85% who would receive an evaluation under the Ottawa clinical decision rule? B. What if the standard evaluation for subarachnoid hemorrhage was only a nonenhanced CT? How many patients had a brain CT in the ED? How does this percentage compare with the Ottawa rule’s 85% testing rate? What is the incremental benefit of the Ottawa Subarachnoid Hemorrhage Rule over CT imaging alone? C. Focus on Table 4, which reports the characteristics of the 11 patients with a subarachnoid hemorrhage not identified by 1 or more of the clinical decision rules. How many of the patients who ultimately required a surgical intervention were “missed” and discharged from the ED? Does Table 4 inform the reader how many of these patients underwent CT and lumbar puncture in the ED? If the data are not available in the table, can you find these numbers elsewhere in the article? D. The final rule has a specificity of 15.3%, meaning that 5 of 6 patients would receive a CT/lumbar puncture evaluation. Is the rule going to benefit patients or harm them? Did the authors compare the decision rules’ performance to the performance of the treating clinicians? If the investigation did not perform this comparison, opine why such a comparison was not conducted. Why is such a comparison critical to evaluating the impact of a decision rule? E. Consider which of the following outcomes, decreasing the frequency of missed subarachnoid hemorrhage diagnoses or reducing the use of tests (ie, CT, lumbar puncture, and angiography), might be more important to practicing clinicians. Which might be more important to hospital administrators adjusting to the reimbursement reductions that will accompany the full implementation of the Patient Protection and Affordable Care Act? What about public health administrators? 4. The initial study describes how classification and regression tree (CART) analysis was applied to a single data set to derive 3 4-element rules that seemed to have similar test characteristics.2 A. Why would the investigators derive 3 different rules from the same data set? What does the ability to derive 3 similarly performing rules from the data set suggest about the nature of the clinical question? The utility of rules for that question? The process by which the rules were developed? What special considerations arise when age is used as one of the criteria? B. The authors then further refine the rules by augmenting one after they had seen the results of the validation study. Do you think that the new rule is valid or, because it was developed post hoc, does it need to be validated in an external data set? If it fails to validate, what would that imply about the process by which it was created? Volume 63, no. 6 : June 2014

ANSWER 1 Q1. The authors state that “5.4% of confirmed subarachnoid hemorrhages were misdiagnosed during the patients’ initial emergency department [ED] assessment.”1 Q1.a What source did the authors cite as the reference for this frequency of missed subarachnoid hemorrhage diagnoses? How does this source compare to other articles about the diagnosis of subarachnoid hemorrhage in terms of reported miss rate and study methodology? Does a subarachnoid hemorrhage misdiagnosis frequency of 1 in 20 seem reasonable in your ED? (For the purposes of this question, assume the same definition that the source authors use.) If not, what factors might decrease the number of missed subarachnoid hemorrhage diagnoses below that reported in the cited reference? The authors’ reference is a 2007 retrospective review of patients admitted during a 2-year period to any hospital in Ontario, Canada, through an ED who were subsequently identified as having a subarachnoid hemorrhage through a search of discharge diagnoses through a province-wide administrative database.3 Patients were classified as having a missed diagnosis of subarachnoid hemorrhage if they had another ED visit within the previous 14 days and received an alternative ED main discharge diagnosis consistent with those previously described as misdiagnoses in other studies of missed subarachnoid hemorrhage patients. This study, which identified 81 of 1,603 patients (5.4%) with a missed subarachnoid hemorrhage, describes a substantially lower miss rate than previous studies that quote rates between 12% and 51%.4-16 However, many of these previous studies were single-site, retrospective chart reviews with study sizes ranging from 13 to 482 patients, which brings into question their external validity. In addition, many were conducted before 1990, when CT scanners were less ubiquitous and had lower resolution than current scanners. Because currentgeneration CT scanners are up to 92% sensitive for diagnosing subarachnoid hemorrhage, the age of these previous studies makes them less relevant to today’s practice.17 Finally, many of the subjects missed in the previous studies were initially treated in a variety of health care settings, including primary care offices with a lower-acuity patient population, making physician misdiagnosis more likely. In the 2 previous studies that subanalyzed the group of patients who initially presented to EDs, the miss rate was 5.2% and 10%, respectively.10,12 Given the heterogeneity and lack of external validity of previous studies, it is difficult to compare the quoted 5.4% miss rate to previous estimates. How the reader interprets an subarachnoid hemorrhage miss rate of 1 in 20 will likely depend on clinical and resource availability where they practice. In the United States, emergency medicine was first recognized as a board specialty in 1979, and now more than half of emergency physician demand is being met by those who are board certified.18 Emergency medicine residencies specifically train physicians to have a heightened awareness of high-morbidity, low-likelihood diagnoses such as subarachnoid hemorrhage. In other diagnoses, such as pulmonary

Annals of Emergency Medicine 775

Journal Club embolism, for example, data exist showing that this physician awareness of high-morbidity diseases may be leading to overtesting without an appreciable increase in patient benefit.19 Additionally, many emergency physicians are risk averse for a variety of reasons and will not want to miss an subarachnoid hemorrhage or any other serious cause of a headache. See answer 1a in the July 2013’s Journal Club answers for a more thorough discussion.20 One indicator of this heightened awareness is the increase in CT use for nontraumatic headaches in ED patients from 12% to 30% between 1998 and 2008.21 Given that the accepted miss rate for myocardial infarction in the United States is 3%,22 a miss rate of 5% for subarachnoid hemorrhage seems high. On the other hand, readers in less resource-intensive clinical settings and those without training or experience in emergency medicine may consider a 5% miss rate as perfectly acceptable. Q1.b How might different stakeholders (eg, patients, physicians, administrators, insurers) choose to define a “missed” diagnosis? How does, and how should, the risk of morbidity associated with a miss affect this definition? One way to think about defining a miss is in terms of risk thresholds. From a clinical standpoint, although near-zero adverse outcome rates may be possible for any individual patient (assuming the intensity of care does not itself produce adverse outcomes), doing so on a population level for any complaint or disease would be impossibly expensive and unrealistic. Thus, a certain level of risk, however small, is assumed by each physician who chooses not to order every possible test and admit every person for a particular clinical scenario. Inevitably, any strategy will produce misses. Whether a miss is considered an avoidable error or an unavoidable result of a necessarily imperfect system depends on the details of the case (did the physician ignore available information?), and the risk threshold of those making the judgment. Of course, hindsight bias and outcome bias influence this determination of error.23,24 In addition, misses may be attributed to physician error but may have as much to do with the structure and process of health care delivery.25,26 Because different stakeholders in medicine (physicians, patients, administrators, insurers, etc) also balance overlapping but distinct objectives, these objectives necessarily affect their perception of risk and misses. For high-morbidity diseases, such as a rupturing abdominal aortic aneurysm, stakeholders are likely to come close to consensus on a low level of acceptable miss rate. However, for low-morbidity diseases, risk tolerance will likely vary substantially. For example, is failing to find mesenteric adenitis (a self-limited, imaging-defined disease for which there is no cure) considered a true miss? The administrator who receives complaints from the patient for lack of testing may think that failing to order a CT scan to diagnose mesenteric adenitis is truly a miss. The same patient’s accountable care organization, charged with maximizing the health of its patients at reasonable cost, may be thrilled that no CT was ordered. A miss and an expected adverse event are the same event seen through the eyes of people with various risk thresholds. As the 776 Annals of Emergency Medicine

morbidity associated with the disease in question decreases, one can reasonably imagine room for less consensus and more room for other externalities (eg, cost, time, patient satisfaction) to factor into how a decision is evaluated. Regardless of disease severity, there is a tendency to assign a negative connotation to misses despite that they are an expected part of the care process. Q1.c What are the implications for the practice of emergency medicine, depending on whose definition we choose to operationalize? How might resource availability in a particular health care setting affect how a “miss” is defined? Stakeholders in medicine (patients, providers, insurers, hospitals, administrators, etc) implicitly and explicitly make tradeoffs between cost (financial and otherwise), time, and risk when evaluating clinical decisions, and these tradeoffs may be in direct conflict with those of other stakeholders. In the ED, for example, testing and treatment is more expensive but often more time efficient (compared with that of outpatient settings). Additionally, emergency physicians are more likely to test and treat liberally in situations of potential or real emergencies. However, emergency physicians also use the same expensive resources on patients with lower-acuity diseases for a variety of reasons, including patient expectation, lack of access to (or confidence in) other health care settings, and physician discomfort with diagnostic uncertainty.27-29 As the US federal government increasingly requires public reporting on resource use, timeliness of care, and patient satisfaction, emergency physicians will likely be forced into balancing the risk of a missed diagnosis with the previous 3 mandates.30 The balance may shift, depending on the power of the stakeholders involved. For example, it is not hard to imagine that administrators in wealthy hospitals will pressure emergency physicians to be resource intensive in the hopes of improving patient satisfaction. Here, mesenteric adenitis might well be considered a missed diagnosis. One rational suggestion would be to inversely tie ED resource use to outpatient health care access. In other words, physicians in resource-rich settings could triage nonemergency patients to outpatient settings for definitive testing and treatment if followup could be arranged in a timely manner. However, for patients who lack access to follow-up, more real-time ED testing would be appropriate. The irony in the United States is that the former set of patients, though they may not need it, are more likely to be able to afford the higher cost of intensive ED resource use, whereas for the latter set of patients, the opposite holds true.

ANSWER 2 Q2. In this article, 26% of patients arrived by ambulance, and “arrival by ambulance” was one of the 4 variables in rule 2. The inclusion of this variable suggests that patients arriving by ambulance are at greater risk for subarachnoid hemorrhage. Q2.a In some settings, patients arriving by ambulance are automatically triaged to higher-acuity beds. How might the patient’s location in the ED when treated by the clinician affect the evaluation Volume 63, no. 6 : June 2014

Journal Club that he receives? How might the “assignment bias” that occurs when low-acuity patients arriving by ambulance are preferentially placed into evaluation rooms rather than directed to the waiting room or a lower-acuity treatment area affect ED throughput and staffing models? Why might this be especially important for EDs that employ midlevel providers or resident moonlighters to staff the lowacuity areas? Because only 25% of patients in US EDs are treated within 15 minutes of arrival, the majority of them are triaged according to acuity.31 Depending on the resource availability of the ED, a proportion of these higher-acuity patients will go to monitored beds, whereas the lowest-acuity patients may be sent to the waiting room to be treated in an urgent care setting. Data exist demonstrating that the triage acuity level corresponds with the eventual number of resources used in the ED.32,33 Emergency physicians evaluating patients triaged to monitored beds naturally assume that some other health care professional decided that the patient needed increased resources compared with a similar patient who was assigned to fast track. A patient’s presence in a higher-acuity room is likely to prime the physician to increase the intensity of testing. Some of this increased intensity will be warranted, but some will simply be the result of “assignment bias.” In 2010, 16% of patients arrived in US EDs by ambulance, with the highest proportion being among those aged 65 years or older (38%).31 When patients arriving by ambulance are automatically placed in monitored beds, ED staffing will necessarily need to be increased compared with that of a system in which all patients are retriaged on ED arrival because even low-acuity patients arriving by ambulance receive more laboratory testing and imaging than nonambulance patients.34 Midlevel providers are increasingly being used in EDs, primarily in lower-acuity areas. Clearly, emergency medicine board certification is not necessary to take care of simple coughs and colds. However, subtle (and atypical) presentations of diseases are often the ones that are most difficult to diagnose, and it is specifically these patients who are both more likely to be triaged as nonurgent, where they will not be seen by the most highly trained providers in the ED (who are more likely to be aware of these subtle presentations). In 2 studies of subarachnoid hemorrhage, missed patients were more likely to be triaged as lower acuity and to have initial presentations of Hunt-Hess grade I or II.3,10 And in one study, the missed patients with initial Hunt-Hess grade I or II had higher morbidity and mortality compared with those with the same subarachnoid hemorrhage grade who were not missed.10 Ascertaining the appropriate staffing model for any ED requires consideration of a combination of patient census, resource availability, provider skills, and risk tolerance.

ANSWER 3 Q3. The specificity of the Ottawa subarachnoid hemorrhage rule is such that nearly 85% of patients with potentially concerning headache would require computed tomography (CT) and lumbar Volume 63, no. 6 : June 2014

puncture. The authors acknowledge that “[t]he Ottawa subarachnoid hemorrhage rule does not lead to a reduction of testing (ie, CT, lumbar puncture, or both) vs current practice; however, it may help to standardize which patients with acute headache require investigations, and its widespread use could help decrease missed subarachnoid hemorrhages.”1 Q3.a The present accepted standard for a subarachnoid hemorrhage evaluation includes a nonenhanced CT study and, if the CT result is negative, a lumbar puncture. Using the data provided in the article, calculate the percentage of the entire cohort who underwent this complete evaluation. Do this using the data in Figure 1 and then repeat the exercise using the data in the tables. Are the estimates concordant? Can you reconcile them? What is your best estimate of how many patients received a complete evaluation by the physicians, and how does this compare with the 85% who would receive an evaluation under the Ottawa clinical decision rule? The JAMA article’s figure implies that 25% of patients (539 of 2,131) had an “actual” evaluation (CT positive or CT negative followed by lumbar puncture performed). Table 1 suggests that 38% of patients (807 of 2,131) had both CT and lumbar puncture and there will be additional patients who had a positive CT result and therefore did not require lumbar puncture who will drive that percentage upward. It is unclear why the number in the figure is so much lower than that in the table. The maximum complete evaluation rate in Table 1 would be 807þ110¼917, or 43% (917/2,131), assuming that, from Table 2, 110 patients with subarachnoid hemorrhage (132–22) received a diagnosis by CT alone. Regardless of which numbers are used, all values are significantly lower than the 85% projected to require a complete evaluation if the Ottawa rule were universally used on this population. There are 2 other minor discrepancies in the article as well. Table 1 reports 833 lumbar punctures performed, whereas Table 2 reports 832 (810þ22). Additionally, Table 3 reports that the Ottawa rule has a specificity of 15.3%, which means the rule’s investigation rate would be 84.7%. However, in the last sentence of the results, the authors state that the investigation rate would be 85.7%. We point these out not because they significantly change conclusions but because they remind us yet again that research, article preparation, and peer review processes are imperfect and can fail to identify such basic discrepancies. Q3.b What if the standard evaluation for subarachnoid hemorrhage was only a nonenhanced CT? How many patients had a brain CT in the ED? How does this percentage compare with the Ottawa rule’s 85% testing rate? What is the incremental benefit of the Ottawa subarachnoid hemorrhage rule over CT imaging alone? According to Table 1, 83% of patients had a CT. This number is similar to 85% estimated to receive a CT if the rule were used. The rule would (1) not affect the number of patients who are imaged but, to some unknown extent, change who is imaged; and (2) increase the number of lumbar punctures from 39% to just above 80% (assuming that 3% to 4% of CTs would be positive, obviating the need for lumbar puncture). Annals of Emergency Medicine 777

Journal Club We could not find a definitive value for how many patients with subarachnoid hemorrhage were missed by the clinicians in this study. See the answer to 3D for more information. Q3.c Focus on Table 4, which reports the characteristics of the 11 patients with a subarachnoid hemorrhage not identified by 1 or more of the clinical decision rules. How many of the patients who ultimately required a surgical intervention were “missed” and discharged from the ED? Does Table 4 inform the reader how many of these patients underwent CT and lumbar puncture in the ED? If the data are not available in the table, can you find these numbers elsewhere in the article? We found no specific mention about how many of the 132 patients with subarachnoid hemorrhage were missed and discharged home by the treating ED clinician. Theoretically, there might have been patients who were positive for 1 or more of the rules but not evaluated with CT or lumbar puncture in the ED. The authors do report that physicians misinterpreted the decision rules as not requiring further investigation in 5% to 6% of cases, which could have resulted in missed cases.1 Without knowing the actual percent of subarachnoid hemorrhage patients who were discharged home by the ED clinician, we have to rely on the data presented in the article’s Table 4 to answer this question. According to these limited data, one might argue that the emergency physicians did not miss any patients with subarachnoid hemorrhage. According to Table 4, only 2 were not admitted. One of these patients, a 29-year-old man, was ultimately determined to have a false-positive result. The other patient, a 30-year-old woman, had a nonaneurysmal bleeding event that required no surgical intervention. This woman was 6 days postpartum, had a normal CT angiogram result, and was evaluated by a neurosurgeon, who discharged the patient home from the ED. She returned to the ED 3 days later for persistent headache but had no subsequent complications. Table 4 does not report what diagnostic studies were performed in these patients. The investigators defined subarachnoid hemorrhage as one of the following: “subarachnoid blood on unenhanced CT of the head; xanthochromia in the cerebrospinal fluid; or red blood cells (>1106/L) in the final tube of cerebrospinal fluid, with an aneurysm or arteriovenous malformation on cerebral angiography.”1 Given that these patients were classified as having positive results for subarachnoid hemorrhage, these individuals must have undergone a CT, lumbar puncture, or angiogram. Table 2 states that all 132 patients with subarachnoid hemorrhage had CT imaging and that a lumbar puncture was performed on 22 of those patients (16.7%). Table 4 reports that 5 of the patients had nonaneurysmal bleeding events, including the aforementioned postpartum woman who had a CT and CT angiogram conducted in the ED. However, the article does not detail how many lumbar punctures or angiograms were performed in the remaining 10 patients. Q3.d The final rule has a specificity of 15.3%, meaning that 5 of 6 patients would receive a CT/lumbar puncture evaluation. Is the rule going to benefit patients or harm them? Did the authors compare the decision rules’ performance to the performance 778 Annals of Emergency Medicine

of the treating clinicians? If the investigation did not perform this comparison, opine why such a comparison was not conducted. Why is such a comparison critical to evaluating the impact of a decision rule? From the answers to the previous questions we derive the following table: Table. Comparison of clinician and Ottawa rule performance at evaluating patients with suspected subarachnoid hemorrhage. Category Patient receiving a CT (%) Patients receiving a lumbar puncture (%) Additional cases detected Additional important cases detected

These Clinicians, %

Ottawa Rule, %

83 39 — —

85 80 2 (0.09%)* 0*

—, Zero. *It is possible that there were cases picked up by all 4 rules but not by the clinicians. No mention of this is made in the article. If such cases do exist, they would increase the totals in the asterisked boxes.

From this table, we conclude that the rule, if implemented on this cohort of patients and clinicians, would double the number of lumbar punctures (and the number of spinal headaches and other consequences of such activity) without improving care. Q3.e Consider which of the following outcomes, decreasing the frequency of missed subarachnoid hemorrhage diagnoses or reducing the use of tests (ie, CT, lumbar puncture, and angiography), might be more important to practicing clinicians. Which might be more important to hospital administrators adjusting to the reimbursement reductions that will accompany the full implementation of the Patient Protection and Affordable Care Act? What about public health administrators? Practicing Emergency Physicians Clinical decision instruments become most important to emergency physicians when they are endorsed by respected groups or reach a high level of recognition and acceptance by other means. Such rules can then protect the emergency physician from a miss as long as the physician can demonstrate that the rule was followed in good faith. In that sense, it is the widespread acceptance of the rule, rather than the actual performance of the rule, that determines whether the physician likes it. Emergency physicians may believe that they are conducting too many head CTs, but this rule will not help them in this regard. They may be less excited at the prospect of conducting twice as many lumbar punctures because that procedure can be time consuming and challenging, given the American obesity epidemic. See the July 2013 Journal Club for a more detailed discussion on this topic.20 In sum, it is difficult to know how emergency physicians will feel about this rule, given that it does not seem to offer any clinical benefit. Hospital Administrators Without a clear understanding of how reimbursement will be affected by the Patient Protection and Affordable Care Act, we Volume 63, no. 6 : June 2014

Journal Club will base this answer on the theorized reductions in payments for procedures and diagnostic testing.35,36 Hospital administrators have a vested interest in measures to decrease the number of missed subarachnoid hemorrhages because these events often lead to patient morbidity, expensive malpractice judgments, and bad press for the hospital.10,37 However, hospital administrators have a competing financial obligation to their stakeholders and must also focus on optimizing their medical staff productivity, monitoring resource use, and keeping their customers (ie, patients and their families) pleased with the services. A lumbar puncture is not a pleasant procedure for many patients, especially when followed by post–lumbar puncture headache. It is our experience that patients often prefer to have a CT or CT angiogram rather than a lumbar puncture. As we discussed in the November Journal Club, the cost of a CT to the hospital is relatively low, besides the radiologist’s interpretation time.38 Therefore, the potential reduction in reimbursement for a CT study is likely small compared with the financial loss for a neurosurgeon or neuroradiologist to perform a cerebral angiogram. If the reimbursement for a lumbar puncture is reduced, the hospital might be in favor of measures that advocate that fewer of these noxious procedures be performed. Furthermore, emergency physicians might be able to treat 1 or 2 additional patients and disposition their existing patients quicker if they could safely forgo a lumbar puncture on a patient with a concerning headache. Both of these possibilities might improve ED throughput, financial productivity, and patient satisfaction. Public Health Public health administrators would likely be concerned about appropriate medical resource use and limiting potential studies and procedures that might have short- and long-term adverse effects. Decreasing the number of unnecessary lumbar punctures and CT studies can have important public health consequences. The 2012 fungal meningitis outbreak is an example of severe complications related to what was previously thought as a relatively safe procedure.39,40 Consider the potential catastrophic consequences had the contamination been in the injectable lidocaine aliquots rather than the steroid mixtures. Such a contamination could put thousands of patients who undergo lumbar punctures at risk for severe morbidity and mortality. We have discussed the risks of cancer and contrast-induced nephropathy related to CT’s ionizing radiation and contrast in previous Journal Clubs.20,41,42 We direct interested readers to those answers for a more detailed discussion.

ANSWER 4 Q4. The initial study describes how classification and regression tree (CART) analysis was applied to a single data set to derive 3 4-element rules that seemed to have similar test characteristics.2 Q4.a Why would the investigators derive 3 different rules from the same data set? What does the ability to derive 3 similarly

Volume 63, no. 6 : June 2014

performing rules from the data set suggest about the nature of the clinical question? The utility of rules for that question? The process by which the rules were developed? What special considerations arise when age is used as one of the criteria? Deriving 3 rules from the same data set may seem more like a “fishing expedition” than a systematic rule development as recommended by international experts in decision rule development.43,44 The fact that the rules all have similar performance suggests that they likely represent a common foundational concept. The rules do not identify any novel risk factors that have not previously been associated with subarachnoid hemorrhage. This multiple rule development strategy may remind the advanced biostatistician readers of factor analysis in which the observable factors are combined and rotated to try to obtain the real factors. Age is a common variable in many decision rules. It is intuitive that older patients generally are more likely to experience adverse events (especially death) at an increased frequency compared with younger patients with similar comorbidities. However, including age tends to endanger the stability of the decision rule when it is moved among settings. For example, the median age of the Ottawa cohort was 44 years. The rule when applied in an ED among several nursing homes may perform very differently than when used in an ED on a college campus. The value of the age variable may differ in different settings. Q4.b The authors then further refine the rules by augmenting one after they had seen the results of the validation study. Do you think that the new rule is valid or, because it was developed post hoc, does it need to be validated in an external data set? If it fails to validate, what would that imply about the process by which it was created? Before we answer the question about whether the Ottawa rule is valid, we should first separate it into 2 parts. Is the rule internally valid (ie, reproducible) and is it externally valid (ie, generalizable)? Readers interested in a more detailed discussion on validation are directed to the November 2009 Journal Club Answers.41 The investigators performed a 1,000-bootstrapreplication internal validation for the Ottawa rule in the original derivation data set and reported the sensitivity as 100% (95% confidence interval 100% to 100%) and specificity 20.6% (95% confidence interval 16.6% to 19.1%). This implies that the rule is internally valid, although additional information on the rule’s calibration (eg, c statistic or area under the receiver operating curve) and discrimination (eg, Somer’s Dxy) would also be informative. The Ottawa rule was a refinement of the 3 previously derived rules to develop one with maximum sensitivity for identifying patients with subarachnoid hemorrhage. Prognostic modeling experts advocate the refinement and updating of existing decision rules when the external validation studies underperform.45 However, these updated rules still should be tested in other patient populations and their effect analyzed before incorporation into clinical practice.45 The augmented rule has not been tested in an independent population (the original data set is not independent because the base rule was derived in it), and thus the Ottawa

Annals of Emergency Medicine 779

Journal Club rule has not had an external validation and is not ready for general use. Decision rules often perform worse in validation studies, and it would not be surprising if the Ottawa rule underperformed in another patient population. Toll et al45 identified 3 possible differences that may account for decreased accuracy in external validation studies. First, validation studies are often performed by independent investigators and may use different definitions and modes of measurement for the predictors and outcomes. For example, even though the same investigative team performed both the derivation and validation studies, there were still differences in the definitions of neck stiffness and “thunderclap headache.” Second, the patient case mix may be different in the derivation and validation cohorts. One cohort may be considerably older or include more women than the derivation cohort, for example. If either age or sex affects the outcome and is not included in the decision rule, then the predicted probabilities will be underestimated in the validation cohort.45 Third, validation studies typically have smaller sample sizes and fewer outcomes than derivation studies.45 Another common cause of poor validation is that the decision rule is overfit, and this may occur for a number of reasons that are discussed in detail in a previous Journal Club.41 Section editors: Tyler W. Barrett, MD, MSCI; David L. Schriger, MD, MPH Author affiliations: From the University of California, Los Angeles, CA (Gupta, Schriger); and the Vanderbilt University Medical Center, Nashville, TN (Barrett). REFERENCES 1. Perry JJ, Stiell IG, Sivilotti ML, et al. Clinical decision rules to rule out subarachnoid hemorrhage for acute headache. JAMA. 2013;310:1248-1255. 2. Perry JJ, Stiell IG, Sivilotti ML, et al. High risk clinical characteristics for subarachnoid haemorrhage in patients with acute headache: prospective cohort study. BMJ. 2010;341:c5204. 3. Vermeulen MJ, Schull MJ. Missed diagnosis of subarachnoid hemorrhage in the emergency department. Stroke. 2007;38:1216-1221. 4. Adams HP Jr, Jergenson DD, Kassell NF, et al. Pitfalls in the recognition of subarachnoid hemorrhage. JAMA. 1980;244:794-796. 5. Becker LA, Green LA, Beaufait D, et al. Detection of intracranial tumors, subarachnoid hemorrhages, and subdural hematomas in primary care patients: a report from ASPN, part 2. J Fam Pract. 1993;37:135-141. 6. Chan BS, Dorsch NW. Delayed diagnosis in subarachnoid haemorrhage. Med J Aust. 1991;154:509-511. 7. Duffy GP. The “warning leak” in spontaneous subarachnoid haemorrhage. Med J Aust. 1983;1:514-516. 8. Hauerberg J, Andersen BB, Eskesen V, et al. Importance of the recognition of a warning leak as a sign of a ruptured intracranial aneurysm. Acta Neurol Scand. 1991;83:61-64. 9. Kassell NF, Kongable GL, Torner JC, et al. Delay in referral of patients with ruptured aneurysms to neurosurgical attention. Stroke. 1985;16:587-590.

780 Annals of Emergency Medicine

10. Kowalski RG, Claassen J, Kreiter KT, et al. Initial misdiagnosis and outcome after subarachnoid hemorrhage. JAMA. 2004;291:866-869. 11. Leblanc R. The minor leak preceding subarachnoid hemorrhage. J Neurosurg. 1987;66:35-39. 12. Mayer PL, Awad IA, Todor R, et al. Misdiagnosis of symptomatic cerebral aneurysm. Prevalence and correlation with outcome at four institutions. Stroke. 1996;27:1558-1563. 13. Neil-Dwyer G, Lang D. “Brain attack”—aneurysmal subarachnoid haemorrhage: death due to delayed diagnosis. J R Coll Physicians Lond. 1997;31:49-52. 14. Schievink WI, van der Werf DJ, Hageman LM, et al. Referral pattern of patients with aneurysmal subarachnoid hemorrhage. Surg Neurol. 1988;29:367-371. 15. Vannemreddy P, Nanda A, Kelley R, et al. Delayed diagnosis of intracranial aneurysms: confounding factors in clinical presentation and the influence of misdiagnosis on outcome. South Med J. 2001;94:1108-1111. 16. Verweij RD, Wijdicks EF, van Gijn J. Warning headache in aneurysmal subarachnoid hemorrhage. A case-control study. Arch Neurol. 1988;45:1019-1020. 17. Singer RJ, Ogilvy SJ, Rordorf G. Clinical manifestations and diagnosis of aneurysmal subarachnoid hemorrhage. UpToDate. Available at: http:// wwwuptodatecom/contents/clinical-manifestations-and-diagnosis-ofaneurysmal-subarachnoid-hemorrhage?source¼search_result& search¼subarachnoidþhemorrhage&selectedTitle¼1w150(2013). Accessed December 5, 2013. 18. Camargo CA Jr, Ginde AA, Singer AH, et al. Assessment of emergency physician workforce needs in the United States, 2005. Acad Emerg Med. 2008;15:1317-1320. 19. Hoffman JR, Cooper RJ. Overdiagnosis of disease: a modern epidemic. Arch Intern Med. 2012;172:1123-1124. 20. Gupta M, Barrett TW, Schriger DL. Every peddler praises his own needle: have clinical rules in the diagnosis of subarachnoid hemorrhage supplanted lumbar punctures yet? answers to the July 2013 Journal Club questions. Ann Emerg Med. 2013;62:633-640. 21. Gilbert JW, Johnson KM, Larkin GL, et al. Atraumatic headache in US emergency departments: recent trends in CT/MRI utilisation and factors associated with severe intracranial pathology. Emerg Med J. 2012;29:576-581. 22. Brown TB, Cofield SS, Iyer A, et al. Assessment of risk tolerance for adverse events in emergency department chest pain patients: a pilot study. J Emerg Med. 2010;39:247-252. 23. Gupta M, Schriger DL, Tabas JA. The presence of outcome bias in emergency physician retrospective judgments of the quality of care. Ann Emerg Med. 2011;57:323-328.e9. 24. Wears RL, Nemeth CP. Replacing hindsight with insight: toward better understanding of diagnostic failures. Ann Emerg Med. 2007;49:206-209. 25. Donabedian A. Methods for deriving criteria for assessing the quality of medical care. Med Care Rev. 1980;37:653-698. 26. Wears RL. The error of counting “errors.”. Ann Emerg Med. 2008;52:502-503. 27. Gupta M, Schriger DL, Hiatt JR, et al. Selective use of computed tomography compared with routine whole body imaging in patients with blunt trauma. Ann Emerg Med. 2011;58:407-416.e15. 28. Kassirer JP. Our stubborn quest for diagnostic certainty. A cause of excessive testing. N Engl J Med. 1989;320:1489-1491. 29. Weber EJ, Showstack JA, Hunt KA, et al. Are the uninsured responsible for the increase in emergency department visits in the United States? Ann Emerg Med. 2008;52:108-115. 30. Available at: http://www.hcahpsonline.org. 2013. Accessed December 3, 2013. 31. Centers for Disease Control and Prevention, National Center for Heatlh Statistics. Homepage. Available at: http://www.cdc.gov/nchs/ahcd/ web_tables.htm-2010. 2013. Accessed December 3, 2013.

Volume 63, no. 6 : June 2014

Journal Club 32. Green NA, Durani Y, Brecher D, et al. Emergency Severity Index version 4: a valid and reliable tool in pediatric emergency department triage. Pediatr Emerg Care. 2012;28:753-757. 33. Tanabe P, Gimbel R, Yarnold PR, et al. The Emergency Severity Index (version 3) 5-level triage system scores predict ED resource consumption. J Emerg Nurs. 2004;30:22-29. 34. Durant E, Fahimi J. Factors associated with ambulance use among patients with low-acuity conditions. Prehosp Emerg Care. 2012;16:329-337. 35. Doctors will have to take a paycut under Obamacare. 2013. Available at: http://www.forbes.com/sites/scottgottlieb/2013/06/28/doctorswill-have-to-take-a-pay-cut-under-obamacare/. Accessed December 13, 2013. 36. Reimbursement under the ACA: hefty cuts on the horizon. UBM Medica, LLC, 2013. Available at: http://www.diagnosticimaging.com/ healthcare-reform/reimbursement-under-aca-hefty-cuts-horizon? GUID¼255F29EB-D7B0-4684-8A0D-DC9045962C9E& rememberme¼1&ts¼03082013. Accessed December 12, 2013. 37. Karcz A, Holbrook J, Burke MC, et al. Massachusetts emergency medicine closed malpractice claims: 1988-1990. Ann Emerg Med. 1993;22:553-559. 38. Schriger DL, Callaham ML, Barrett TW. Measuring and explaining computed tomography use in the United States and Canada: a consideration of health economics, use versus appropriateness, and

Volume 63, no. 6 : June 2014

39. 40.

41.

42.

43.

44.

45.

interpreting potential conflict of interest: November 2013 Annals of Emergency Medicine Journal Club. Ann Emerg Med. 2013;62:545-546. Pettit AC, Pugh ME. Index case for the fungal meningitis outbreak, United States. N Engl J Med. 2013;368:970. Smith RM, Tipple M, Chaudry MN, et al. Relapse of fungal meningitis associated with contaminated methylprednisolone. N Engl J Med. 2013;368:2535-2536. Barrett TW, Schriger DL. Annals of Emergency Medicine Journal Club. Clinical prediction rules answers to the November 2009 Journal Club. Ann Emerg Med. 2010;55:380-389. Barrett TW, Schriger DL. Annals of Emergency Medicine Journal Club. Computed tomography imaging in the emergency department: benefits, risks and risk ratios. Answers to the November 2011 Journal Club questions. Ann Emerg Med. 2012;59:328-334. Harrell FE Jr, ed. Regression Model Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. New York, NY: Springer; 2001. Steyerberg EW. Clinical Prediction Models—A Practical Approach to Development, Validation, and Updating. New York, NY: Springer; 2009. Toll DB, Janssen KJ, Vergouwe Y, et al. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol. 2008;61:1085-1094.

Annals of Emergency Medicine 781

Annals of Emergency Medicine journal club. Do more rules make us safer? Clinical decision rules, patient safety, and the role of emergency physicians in health care: answers to the January 2014 journal club questions.

Annals of Emergency Medicine journal club. Do more rules make us safer? Clinical decision rules, patient safety, and the role of emergency physicians in health care: answers to the January 2014 journal club questions. - PDF Download Free
454KB Sizes 0 Downloads 3 Views

Recommend Documents