Risk Analysis
DOI: 10.1111/risa.12159
Trust-Level Risk Evaluation and Risk Control Guidance in the NHS East of England Alan J. Card,1,∗ James R. Ward,2 and P. John Clarkson2
In recent years, the healthcare sector has adopted the use of operational risk assessment tools to help understand the systems issues that lead to patient safety incidents. But although these problem-focused tools have improved the ability of healthcare organizations to identify hazards, they have not translated into measurable improvements in patient safety. One possible reason for this is a lack of support for the solution-focused process of risk control. This article describes a content analysis of the risk management strategies, policies, and procedures at all acute (i.e., hospital), mental health, and ambulance trusts (health service organizations) in the East of England area of the British National Health Service. The primary goal was to determine what organizational-level guidance exists to support risk control practice. A secondary goal was to examine the risk evaluation guidance provided by these trusts. With regard to risk control, we found an almost complete lack of useful guidance to promote good practice. With regard to risk evaluation, the trusts relied exclusively on risk matrices. A number of weaknesses were found in the use of this tool, especially related to the guidance for scoring an event’s likelihood. We make a number of recommendations to address these concerns. The guidance assessed provides insufficient support for risk control and risk evaluation. This may present a significant barrier to the success of risk management approaches in improving patient safety. KEY WORDS: Patient safety; healthcare risk management; risk control; risk evaluation; risk matrix
1. INTRODUCTION
lem exploration; none of them support users through the process of generating, assessing, implementing, and managing solutions, even though these solutions are the intended product of the risk management process. Healthcare workers, in contrast, have not received such training. In their clinical education, healthcare personnel are trained in both diagnosis and treatment because diagnoses, by themselves, do not improve patients’ health. Similarly, healthcare personnel working beyond their training in the area of risk management cannot be expected to be successful unless they are supported in both the problem exploration stage (risk assessment) and the problemsolving stage (risk control) of the risk management process. And, although the adoption of risk assessment tools has improved the practice of diagnosing systems-level problems in healthcare,(15) the use of such problem-focused tools has not translated to improvements in the solutions design process(16) of
In recent years, many healthcare organizations have adopted operational risk management tools developed in high-reliability industries like nuclear power generation, manufacturing, and aviation to address patient safety risks. But these techniques, including root cause analysis (RCA),(1–5) failure mode and effects analysis (FMEA),(6–10) and others,(10–14) were designed to be used by engineers and others who are trained to develop robust, high-quality solutions in response to assessed risks. Perhaps as a result, all of these tools focus exclusively on prob1 Evidence-Based
Health Solutions, LLC, Notre Dame, IN, USA.
2 Engineering Design Centre, Department of Engineering, Univer-
sity of Cambridge, Cambridge, UK. correspondence to Alan J. Card, PhD, MPH, CPH, CPHQ, President/CEO, Evidence-Based Health Solutions, LLC, PO Box 62, Notre Dame, IN 46556, USA; [email protected].
∗ Address
1
C 2013 Society for Risk Analysis 0272-4332/13/0100-0001$22.00/1
2 risk control.(2) Healthcare workers continue to experience significant difficulty in devising high-quality risk control actions and in assessing risk control quality.(2,17–20) If it is clear that healthcare workers require support for the risk control process, it is less clear whether they are receiving it, and if so, what form such support might take. A systematic review of the literature on healthcare RCA found no widely used tools to support the risk control (problem-solving) component of the risk management process. This study also found that the risk controls developed in response to included RCAs tended to be weak, with an overwhelming reliance on administrative risk controls.(2) But, given the low level of detail with which most RCAs were reported, and the fact that most healthcare RCAs do not make their way into the research literature, it is not clear whether the studies included in this review provide an accurate reflection of real-world practice. Indeed, healthcare risk management practice is often characterized by an overreliance on institutional experience,(12) and insufficient use of research evidence.(21) This raises the question of whether, and to what extent, organization-level guidance provides a framework for the risk control process in healthcare. To shed light on this question, we conducted a content analysis of risk management strategies, policies, and procedures at all acute (i.e., hospital), mental health, and ambulance trusts in the East of England (EoE) region of the U.K. National Health Service (NHS). The primary goal of this work was to understand what support, if any, organization-level policies and procedures provide to assist healthcare workers in the risk control process, i.e., converting the learning generated from risk assessments (e.g., RCA, FMEA, etc.) into effective, safe, and sustainable actions to reduce risk to patients. A secondary focus of this work was on the guidance provided for risk evaluation (the process of deciding whether risk control is required, and what level of investment is warranted) because this matter of determining what constitutes a proportionate response is so tightly coupled to the risk control process. An earlier pilot study suggested that a significant level of insight could be gained into practice guidance through an analysis of these documents.(10) Current practice in healthcare risk management has failed to significantly reduce the rate of harm to patients,(22) which remains unacceptably high with
Card, Ward, and Clarkson ∼25–30% of hospital patients experiencing some level of detectable harm,(22–24) and an estimated 400,000 people dying every year as a result of patient safety incidents in the United States alone.(25) This research is important because a better understanding of the basis for current practice can help inform the development of new frameworks, tools, and techniques to improve the risk control process. More effective risk control practice will not only lead to reduced patient harm, but also to a more cost-effective risk management process, in which the investments made in risk assessment are not wasted through ineffective risk control. 2. METHODS 2.1. Data Collection Each acute, mental health, and ambulance trust in the EoE was sent a freedom of information request on October 25, 2010. The trusts were asked to provide: (1) The organization’s current risk management policy (or nearest equivalent, e.g., risk assessment policy). (2) The organization’s current risk management procedures (or nearest equivalent, e.g., risk assessment procedures). Trusts were further informed that the request pertained only to general risk management policies and procedures for protecting patient safety, and that documents focused solely on employee safety or on specific risks (e.g., patient falls, nighttime/out of hours operation, etc.) were not required. In an earlier pilot study examining risk management policies and procedures,(10) we found that these specialist documents were generally well-harmonized with trusts’ general risk management guidance, and thus contributed little to the understanding of a trust’s overall approach to patient safety risk management. 2.2. Content Analyzed 2.2.1. Risk Control Guidance Risk control guidance was assessed to determine whether any specific tools were used and what guidance (if any) was provided to support the risk control subprocesses of risk control planning (including risk control options generation and options assessment),
Trust-Level Risk Evaluation and Risk Control Guidance implementation and monitoring of risk controls, and evaluation of the success of implemented risk controls. Variables related to generating risk control options included: presence/absence of any guidance, whether a list of risk control types was provided, whether such a list was presented as an explicit hierarchy of risk controls, whether any specific methods were mentioned, and whether guidance was provided on how to use any such methods. Variables related to assessing risk control options included: whether advice was provided to consider cost, feasibility, or secondary risks, whether the guidance called for stakeholder consultation, whether a named person was to be given responsibility for implementation, whether the specific, measurable, achievable, realistic, and time-bounded (SMART) criteria was mentioned, whether a prioritized list of risk control options was to be produced, whether any such prioritized list of risk control options was reflected in the forms used for action plans/risk registers, whether any specific methods were presented for any of the preceding, and whether guidance was presented on how to use any such methods. Variables assessed for implementation and monitoring included: whether the guidance called for users to specify a timeline/date for risk control implementation, whether any implementation advice was presented, what any such advice was, whether guidance was presented on how to follow any such advice, whether any monitoring advice was presented, what any such advice was (other than periodicity tied to risk score), and whether guidance was presented on how to follow any such advice. Variables for evaluating the success of risk controls included: whether advice was presented on how to evaluate success, what any such advice was, and whether guidance was presented on how to follow any such advice.
2.2.2. Risk Evaluation Guidance Risk evaluation guidance was assessed to identify and characterize the tools were used (e.g., risk matrices), and the guidance for scoring likelihood and severity. Variables related to risk matrices included: risk matrix dimensions, number of tiers, symmetry, and (for risk matrices with asymmetrical tiers) whether a score of 4 was included in three different tiers.
3 Variables related to risk scoring guidance included: whether likelihood was measured in terms of occurrence within a specified period of time or in terms of the probability of occurrence, the percentages/proportions used for scoring opportunity-based likelihood, whether the scoring categories were discrete, the score for a probability of 20%, and whether patient safety was listed as the first category in the severity scoring guidance. 3. RESULTS AND ANALYSIS 3.1. Response Rate The response rate was 100%; it included 18 acute trusts, one ambulance trust, and six mental health trusts.: 3.2. Risk Control Guidance Risk control guidance was assessed in terms of three risk control subprocesses: planning, implementation and monitoring, and evaluation of success. These headings loosely mirror those used in ISO 31000,(2) which is probably the most commonly used standard for operational risk management. For each risk control, the standard calls for users to assess and document the “expected benefit to be gained; performance measures and constraints; persons who are accountable for approving the plan and those responsible for implementing the plan; proposed actions; reporting and monitoring requirements; resource requirements; and timing and schedule,” as well as secondary risks (negative side-effects of the risk control). It also calls for stakeholder engagement, the integration of risk control plans with broader management processes, and the production of a rankordered list of risk control recommendations to aid in decision making. The documents provided by the trusts gave users very little advice on how to conduct these functions, despite the fact that risk control is the mechanism by which all the previous stages are converted to safety improvement. One trust provided no guidance on risk control (or risk evaluation), so the analysis below pertains only to the remaining 24 trusts.: 3.2.1. Planning None of the trusts used any specific tools to support the generation of risk control options, and guidance to support options generation was very limited. Fifty percent of trusts offered no such advice. Ten
4
Card, Ward, and Clarkson
trusts provided a list of potential generic responses. An example from one trust, which appears to be drawn from ISO 31000,(26) read: Options for risk treatment, which are not necessarily mutually exclusive or appropriate in all circumstances, include the following: (1) Avoid the risk by deciding not to proceed with the activity likely to generate the risk; (2) Reduce the likelihood of the occurrence such as audits and review, inspections, training etc. (3) Reduce the consequences such as ex gratia payments, minimizing exposure to risks, public relations; (4) Transfer the risk—e.g., insurance arrangements; or (5) Retain the risk. : Four of these trusts explicitly presented the list in the form of a hierarchy, in which some risk control options were preferred on the grounds of increased effectiveness. Two trusts provided even more limited advice on generating risk control options, with one suggesting that risk elimination should be preferred over risk reduction, and the other noting that risk controls may target either the likelihood or consequence of harm. Guidance was similarly sparse on the topics of risk control design features and the assessment/selection of risk controls to be implemented. Other than the guidance provided in the risk evaluation stage, only one-third of trusts called for cost to be taken into account, and only three called for an assessment of feasibility. None called for an assessment of the potential side-effects of a risk control (i.e., new risks it might introduce or other risks it might reduce). Nor did any advise users to produce a prioritized list of risk control options. Only five trusts called for staff input to be solicited during the planning process. None explicitly called for the risk control options to be SMART. The only specific planning advice provided by a majority of trusts (n = 15) was a call for a specific person to be named as responsible for the risk control. The use of risk matrices was ubiquitous for evaluating the expected effectiveness of risk controls, but no trust provided any specific guidance to support the assessment of reductions in likelihood or consequences as a result of risk control actions.
3.2.2. Implementation and Monitoring At the border between planning and implementation, 15 (63%) explicitly called for a specific date
to be set for risk control implementation. Other than setting the periodicity of review based on risk score, no trust offered any specific guidance on monitoring risk controls. No trust called for users to monitor for potential side-effects of implemented risk controls.
3.2.3. Evaluation of Success Other than the use of risk matrices, no trusts offered any specific guidance for evaluating the success of implemented risk controls.
3.2.4. Analysis of Risk Control Guidance The healthcare sector is not alone in its emphasis on risk assessment and neglect of risk control; this is part of a broader trend in the field of risk management.(27–29) But the fact that healthcare organizations rely on risk control designers who typically have little, if any, training in this task(2) (rather than engineers or ergonomists, for instance) means that they cannot assume that successful risk assessment will necessarily translate to successful risk control. And although even highly trained engineers might benefit from tools to support risk control (just as physicians can benefit from evidence-based treatment guidelines), when risk management processes are led by those without such training, additional support is almost certainly required.(2) Thus our findings, which indicate that almost no such support is being provided, are a cause for concern. For instance, despite the safety-critical nature of the risk control function, the widely used SMART criteria(30) were not used by any of the trusts, though this might have addressed some of the common shortfalls in healthcare risk control design.(20,31) Nor did any of the trusts call for an assessment of the positive or negative side-effects of the risk control options generated. And very few called for stakeholder input in the planning stage, despite evidence suggesting that this may increase the likelihood both of a risk control being implemented, and of it proving successful.(32,33) Indeed, very few trusts called for any direct consideration of such key issues as the cost or feasibility of risk control options. And (beyond the use of risk matrices, which only measure one aspect of a risk control, the degree to which it reduced the risk of interest) none gave any guidance on evaluating the success of implemented risk controls. These deficits appear to indicate a significant lack of support for systems thinking in the risk control process.
Trust-Level Risk Evaluation and Risk Control Guidance The lack of any tool to assist users in generating high-quality risk control options in the first place is also problematic because the reflexive tendency in healthcare tends to be toward administrative controls, such as training and policy changes, which are relatively weak and nonrobust.(2,17,20,34–36) The quality of the risk controls selected and implemented can obviously not exceed the quality of the risk control options generated; thus the lack of tools to move healthcare personnel beyond this reliance on administrative controls is a critical limiting factor in the effectiveness of the healthcare risk management process. On this last point, we have developed the generating options for active risk control (GO-ARC) technique,(16) a structured brainstorming approach analogous to the structured what-if technique(11) and hazard and operability studies(37) techniques for hazard identification. The GO-ARC technique uses a series of five brainstorming prompts to elicit risk control options. These prompts are: elimination of the hazard or target, design controls (aka engineering controls), administrative controls, detection/situational awareness, and preparedness. Early evidence indicates that this approach may improve the quantity, quality, variety, and novelty of risk control options generated for consideration.(16) But, although improved risk control options generation is necessary, it is clearly not sufficient for improved risk control performance and patient safety improvement. As we have identified in this study, healthcare organizations currently lack the tools to make well-informed decisions about which options to implement, or how to determine whether implemented risk controls are successful. The lovebug diagram, a recently introduced hybrid of the fishbone diagram(38,39) and force-field analysis,(40,41) may help users better assess the forces operating for and against the success of a risk control option, thus informing decisions about which option(s) to adopt.(42) But the lack of any approaches beyond risk matrices for evaluating the eventual success of a risk control remains problematic. Although a risk matrix may (or, given the weak guidance provided by many trusts and the limitations of the tool itself, may not) be a sufficient approach for assessing the effectiveness of a risk control, it is probably inadequate to the task of evaluating success. An evaluation of success should also take into account broader issues such as cost-effectiveness (Might another risk control have had a greater effect at less cost?) and side-effects (Might the risk control have led
5 to a net increase in risk, through the introduction of new risks that outweigh the risk reduction? Or might the risk control have been even more successful than planned, because it reduced other significant risks in addition to the risk of interest?). On the primary focus of this study, the answer is clear: there is a pressing need for the development of a comprehensive framework for risk control practice in healthcare and the implementation of that framework through supporting tools and techniques. Turning to the secondary question of risk evaluation, we now examine whether these trusts are equipped to determine which risks require control.
3.3. Risk Evaluation Guidance 3.3.1. Tools The documents provided by one mental health trust did not cover the risk evaluation process; this was the same trust that did not provide any risk control guidance. Of the remaining 24, all relied exclusively on the use of risk matrices(10,43,44) for risk evaluation. The underlying theory behind the use of a risk matrix is that risk can be defined as the product of the likelihood of an event occurring and its consequence if it does occur (i.e., risk = likelihood × consequence). Thus real or potential adverse events are first assigned independent scores for likelihood and consequence (often on a scale of 1–5). The overall risk score is then computed as the product of those two scores, and lies at the intersection of the two numbers on a matrix (e.g., if likelihood = 3 and consequence = 4, risk = 12). The matrix is overlaid by predefined risk tolerability bands tied to action requirements and resource availability. For instance, a risks score of 1–7 might correspond to the green band (low risk: no action required), whereas a score of 8–15 might correspond to the amber band (medium risk: directorate-level action required, using existing directorate-level budgetary resources), and a score of 16–25 might place a risk in the red band (high risk: senior management action required and board to be informed, using organizational-level budgetary resources). See Fig. 1 for a risk matrix based on this example. Twenty-two trusts used 5 × 5 risk matrices. One used a 6 × 6 variation, in which an additional row and a column were included for scores of zero likelihood and zero consequence. And one trust used two overlapping matrices: an 8 × 9 risk matrix with a 5 × 5 matrix overlaid. A slight majority of trusts (13 of 24
6
Card, Ward, and Clarkson Likelihood Consequence 1
2
3
4
5
Unlikely
Possible
Likely
Almost Certain
5
10
15
20
25
4 Major
4
8
12
16
20
3 Moderate
3
6
9
12
15
2 Minor
2
4
6
8
10
1 Negligible
1
2
3
4
5
Rare 5 Catastrophic
Fig. 3. Scores assigned by trusts to a probability of 20%.
*light gray = green; medium gray = amber; dark gray = red
Fig. 1. A 5 × 5 risk matrix. Likelihood Consequence
1
2
3
4
5
Rare
Unlikely
Possible
Likely
Almost Certain
5 Catastrophic
5
10
15
20
25
4 Major
4
8
12
16
20
3 Moderate
3
6
9
12
15
2 Minor
2
4
6
8
10
1 Negligible
1
2
3
4
5
*white = green; light gray = yellow; medium gray = amber; dark gray = red
Fig. 2. Asymmetrical risk matrix from an acute trust.
that provided risk evaluation guidance) used matrices with four risk tolerability bands; seven used three risk tolerability bands, and four used a model with five bands. (These results treat all trusts as having 5 × 5 matrices. The 5 × 5 version was analyzed for the trust with two overlapping models, and the row and column for zero scores were ignored for the trust with a 6 × 6 matrix, which would otherwise have had five bands.) Five trusts (all of which were acute trusts) used asymmetrical risk tolerability bands, perhaps reflecting skepticism of the underlying model. In all five cases, the three means of arriving at a risk score of four (1 × 4, 2 × 2, 4 × 1) resulted in three different risk categorizations. See Fig. 2 for an example. 3.3.2. Consequence Scoring Twenty trusts provided consequence scoring guidance. This guidance was typically based on the
report “A Risk Matrix for Risk Managers” by the National Patient Safety Agency,(44) which recognizes that there are many different categories of consequences (e.g., safety, trust reputation, impact on operations, etc.). Sixteen trusts (80% of those that provided guidance) listed patient safety or general safety as the first category. Just as it is true that making patient safety the first item on the agenda for trust boards may increase the time spent engaging with the topic,(45) this may be intended to signal the importance of safety risks and lead to an increased focus on harm to patients and others. 3.3.3. Likelihood Scoring Sixteen trusts provided quantitative guidance for scoring likelihood. Two different quantitative approaches were used: assessment based on the likelihood of the event occurring within a given timeframe was used by 12 trusts (e.g., 1 = not expected to occur for years, 2 = yearly, 3 = monthly, 4 = weekly, 5 = daily), and assessment based on probability was used by 15 trusts (e.g., 1 = 90% of the time to warrant a likelihood score of 5. Further, 27% of trusts using probability-based scoring employed nondiscrete categories, which could lead to inconsistent scoring and poor allocation of scarce resources. For example, one trust used the following cut-off scores: 1 =
DOI: 10.1111/risa.12159
Trust-Level Risk Evaluation and Risk Control Guidance in the NHS East of England Alan J. Card,1,∗ James R. Ward,2 and P. John Clarkson2
In recent years, the healthcare sector has adopted the use of operational risk assessment tools to help understand the systems issues that lead to patient safety incidents. But although these problem-focused tools have improved the ability of healthcare organizations to identify hazards, they have not translated into measurable improvements in patient safety. One possible reason for this is a lack of support for the solution-focused process of risk control. This article describes a content analysis of the risk management strategies, policies, and procedures at all acute (i.e., hospital), mental health, and ambulance trusts (health service organizations) in the East of England area of the British National Health Service. The primary goal was to determine what organizational-level guidance exists to support risk control practice. A secondary goal was to examine the risk evaluation guidance provided by these trusts. With regard to risk control, we found an almost complete lack of useful guidance to promote good practice. With regard to risk evaluation, the trusts relied exclusively on risk matrices. A number of weaknesses were found in the use of this tool, especially related to the guidance for scoring an event’s likelihood. We make a number of recommendations to address these concerns. The guidance assessed provides insufficient support for risk control and risk evaluation. This may present a significant barrier to the success of risk management approaches in improving patient safety. KEY WORDS: Patient safety; healthcare risk management; risk control; risk evaluation; risk matrix
1. INTRODUCTION
lem exploration; none of them support users through the process of generating, assessing, implementing, and managing solutions, even though these solutions are the intended product of the risk management process. Healthcare workers, in contrast, have not received such training. In their clinical education, healthcare personnel are trained in both diagnosis and treatment because diagnoses, by themselves, do not improve patients’ health. Similarly, healthcare personnel working beyond their training in the area of risk management cannot be expected to be successful unless they are supported in both the problem exploration stage (risk assessment) and the problemsolving stage (risk control) of the risk management process. And, although the adoption of risk assessment tools has improved the practice of diagnosing systems-level problems in healthcare,(15) the use of such problem-focused tools has not translated to improvements in the solutions design process(16) of
In recent years, many healthcare organizations have adopted operational risk management tools developed in high-reliability industries like nuclear power generation, manufacturing, and aviation to address patient safety risks. But these techniques, including root cause analysis (RCA),(1–5) failure mode and effects analysis (FMEA),(6–10) and others,(10–14) were designed to be used by engineers and others who are trained to develop robust, high-quality solutions in response to assessed risks. Perhaps as a result, all of these tools focus exclusively on prob1 Evidence-Based
Health Solutions, LLC, Notre Dame, IN, USA.
2 Engineering Design Centre, Department of Engineering, Univer-
sity of Cambridge, Cambridge, UK. correspondence to Alan J. Card, PhD, MPH, CPH, CPHQ, President/CEO, Evidence-Based Health Solutions, LLC, PO Box 62, Notre Dame, IN 46556, USA; [email protected].
∗ Address
1
C 2013 Society for Risk Analysis 0272-4332/13/0100-0001$22.00/1
2 risk control.(2) Healthcare workers continue to experience significant difficulty in devising high-quality risk control actions and in assessing risk control quality.(2,17–20) If it is clear that healthcare workers require support for the risk control process, it is less clear whether they are receiving it, and if so, what form such support might take. A systematic review of the literature on healthcare RCA found no widely used tools to support the risk control (problem-solving) component of the risk management process. This study also found that the risk controls developed in response to included RCAs tended to be weak, with an overwhelming reliance on administrative risk controls.(2) But, given the low level of detail with which most RCAs were reported, and the fact that most healthcare RCAs do not make their way into the research literature, it is not clear whether the studies included in this review provide an accurate reflection of real-world practice. Indeed, healthcare risk management practice is often characterized by an overreliance on institutional experience,(12) and insufficient use of research evidence.(21) This raises the question of whether, and to what extent, organization-level guidance provides a framework for the risk control process in healthcare. To shed light on this question, we conducted a content analysis of risk management strategies, policies, and procedures at all acute (i.e., hospital), mental health, and ambulance trusts in the East of England (EoE) region of the U.K. National Health Service (NHS). The primary goal of this work was to understand what support, if any, organization-level policies and procedures provide to assist healthcare workers in the risk control process, i.e., converting the learning generated from risk assessments (e.g., RCA, FMEA, etc.) into effective, safe, and sustainable actions to reduce risk to patients. A secondary focus of this work was on the guidance provided for risk evaluation (the process of deciding whether risk control is required, and what level of investment is warranted) because this matter of determining what constitutes a proportionate response is so tightly coupled to the risk control process. An earlier pilot study suggested that a significant level of insight could be gained into practice guidance through an analysis of these documents.(10) Current practice in healthcare risk management has failed to significantly reduce the rate of harm to patients,(22) which remains unacceptably high with
Card, Ward, and Clarkson ∼25–30% of hospital patients experiencing some level of detectable harm,(22–24) and an estimated 400,000 people dying every year as a result of patient safety incidents in the United States alone.(25) This research is important because a better understanding of the basis for current practice can help inform the development of new frameworks, tools, and techniques to improve the risk control process. More effective risk control practice will not only lead to reduced patient harm, but also to a more cost-effective risk management process, in which the investments made in risk assessment are not wasted through ineffective risk control. 2. METHODS 2.1. Data Collection Each acute, mental health, and ambulance trust in the EoE was sent a freedom of information request on October 25, 2010. The trusts were asked to provide: (1) The organization’s current risk management policy (or nearest equivalent, e.g., risk assessment policy). (2) The organization’s current risk management procedures (or nearest equivalent, e.g., risk assessment procedures). Trusts were further informed that the request pertained only to general risk management policies and procedures for protecting patient safety, and that documents focused solely on employee safety or on specific risks (e.g., patient falls, nighttime/out of hours operation, etc.) were not required. In an earlier pilot study examining risk management policies and procedures,(10) we found that these specialist documents were generally well-harmonized with trusts’ general risk management guidance, and thus contributed little to the understanding of a trust’s overall approach to patient safety risk management. 2.2. Content Analyzed 2.2.1. Risk Control Guidance Risk control guidance was assessed to determine whether any specific tools were used and what guidance (if any) was provided to support the risk control subprocesses of risk control planning (including risk control options generation and options assessment),
Trust-Level Risk Evaluation and Risk Control Guidance implementation and monitoring of risk controls, and evaluation of the success of implemented risk controls. Variables related to generating risk control options included: presence/absence of any guidance, whether a list of risk control types was provided, whether such a list was presented as an explicit hierarchy of risk controls, whether any specific methods were mentioned, and whether guidance was provided on how to use any such methods. Variables related to assessing risk control options included: whether advice was provided to consider cost, feasibility, or secondary risks, whether the guidance called for stakeholder consultation, whether a named person was to be given responsibility for implementation, whether the specific, measurable, achievable, realistic, and time-bounded (SMART) criteria was mentioned, whether a prioritized list of risk control options was to be produced, whether any such prioritized list of risk control options was reflected in the forms used for action plans/risk registers, whether any specific methods were presented for any of the preceding, and whether guidance was presented on how to use any such methods. Variables assessed for implementation and monitoring included: whether the guidance called for users to specify a timeline/date for risk control implementation, whether any implementation advice was presented, what any such advice was, whether guidance was presented on how to follow any such advice, whether any monitoring advice was presented, what any such advice was (other than periodicity tied to risk score), and whether guidance was presented on how to follow any such advice. Variables for evaluating the success of risk controls included: whether advice was presented on how to evaluate success, what any such advice was, and whether guidance was presented on how to follow any such advice.
2.2.2. Risk Evaluation Guidance Risk evaluation guidance was assessed to identify and characterize the tools were used (e.g., risk matrices), and the guidance for scoring likelihood and severity. Variables related to risk matrices included: risk matrix dimensions, number of tiers, symmetry, and (for risk matrices with asymmetrical tiers) whether a score of 4 was included in three different tiers.
3 Variables related to risk scoring guidance included: whether likelihood was measured in terms of occurrence within a specified period of time or in terms of the probability of occurrence, the percentages/proportions used for scoring opportunity-based likelihood, whether the scoring categories were discrete, the score for a probability of 20%, and whether patient safety was listed as the first category in the severity scoring guidance. 3. RESULTS AND ANALYSIS 3.1. Response Rate The response rate was 100%; it included 18 acute trusts, one ambulance trust, and six mental health trusts.: 3.2. Risk Control Guidance Risk control guidance was assessed in terms of three risk control subprocesses: planning, implementation and monitoring, and evaluation of success. These headings loosely mirror those used in ISO 31000,(2) which is probably the most commonly used standard for operational risk management. For each risk control, the standard calls for users to assess and document the “expected benefit to be gained; performance measures and constraints; persons who are accountable for approving the plan and those responsible for implementing the plan; proposed actions; reporting and monitoring requirements; resource requirements; and timing and schedule,” as well as secondary risks (negative side-effects of the risk control). It also calls for stakeholder engagement, the integration of risk control plans with broader management processes, and the production of a rankordered list of risk control recommendations to aid in decision making. The documents provided by the trusts gave users very little advice on how to conduct these functions, despite the fact that risk control is the mechanism by which all the previous stages are converted to safety improvement. One trust provided no guidance on risk control (or risk evaluation), so the analysis below pertains only to the remaining 24 trusts.: 3.2.1. Planning None of the trusts used any specific tools to support the generation of risk control options, and guidance to support options generation was very limited. Fifty percent of trusts offered no such advice. Ten
4
Card, Ward, and Clarkson
trusts provided a list of potential generic responses. An example from one trust, which appears to be drawn from ISO 31000,(26) read: Options for risk treatment, which are not necessarily mutually exclusive or appropriate in all circumstances, include the following: (1) Avoid the risk by deciding not to proceed with the activity likely to generate the risk; (2) Reduce the likelihood of the occurrence such as audits and review, inspections, training etc. (3) Reduce the consequences such as ex gratia payments, minimizing exposure to risks, public relations; (4) Transfer the risk—e.g., insurance arrangements; or (5) Retain the risk. : Four of these trusts explicitly presented the list in the form of a hierarchy, in which some risk control options were preferred on the grounds of increased effectiveness. Two trusts provided even more limited advice on generating risk control options, with one suggesting that risk elimination should be preferred over risk reduction, and the other noting that risk controls may target either the likelihood or consequence of harm. Guidance was similarly sparse on the topics of risk control design features and the assessment/selection of risk controls to be implemented. Other than the guidance provided in the risk evaluation stage, only one-third of trusts called for cost to be taken into account, and only three called for an assessment of feasibility. None called for an assessment of the potential side-effects of a risk control (i.e., new risks it might introduce or other risks it might reduce). Nor did any advise users to produce a prioritized list of risk control options. Only five trusts called for staff input to be solicited during the planning process. None explicitly called for the risk control options to be SMART. The only specific planning advice provided by a majority of trusts (n = 15) was a call for a specific person to be named as responsible for the risk control. The use of risk matrices was ubiquitous for evaluating the expected effectiveness of risk controls, but no trust provided any specific guidance to support the assessment of reductions in likelihood or consequences as a result of risk control actions.
3.2.2. Implementation and Monitoring At the border between planning and implementation, 15 (63%) explicitly called for a specific date
to be set for risk control implementation. Other than setting the periodicity of review based on risk score, no trust offered any specific guidance on monitoring risk controls. No trust called for users to monitor for potential side-effects of implemented risk controls.
3.2.3. Evaluation of Success Other than the use of risk matrices, no trusts offered any specific guidance for evaluating the success of implemented risk controls.
3.2.4. Analysis of Risk Control Guidance The healthcare sector is not alone in its emphasis on risk assessment and neglect of risk control; this is part of a broader trend in the field of risk management.(27–29) But the fact that healthcare organizations rely on risk control designers who typically have little, if any, training in this task(2) (rather than engineers or ergonomists, for instance) means that they cannot assume that successful risk assessment will necessarily translate to successful risk control. And although even highly trained engineers might benefit from tools to support risk control (just as physicians can benefit from evidence-based treatment guidelines), when risk management processes are led by those without such training, additional support is almost certainly required.(2) Thus our findings, which indicate that almost no such support is being provided, are a cause for concern. For instance, despite the safety-critical nature of the risk control function, the widely used SMART criteria(30) were not used by any of the trusts, though this might have addressed some of the common shortfalls in healthcare risk control design.(20,31) Nor did any of the trusts call for an assessment of the positive or negative side-effects of the risk control options generated. And very few called for stakeholder input in the planning stage, despite evidence suggesting that this may increase the likelihood both of a risk control being implemented, and of it proving successful.(32,33) Indeed, very few trusts called for any direct consideration of such key issues as the cost or feasibility of risk control options. And (beyond the use of risk matrices, which only measure one aspect of a risk control, the degree to which it reduced the risk of interest) none gave any guidance on evaluating the success of implemented risk controls. These deficits appear to indicate a significant lack of support for systems thinking in the risk control process.
Trust-Level Risk Evaluation and Risk Control Guidance The lack of any tool to assist users in generating high-quality risk control options in the first place is also problematic because the reflexive tendency in healthcare tends to be toward administrative controls, such as training and policy changes, which are relatively weak and nonrobust.(2,17,20,34–36) The quality of the risk controls selected and implemented can obviously not exceed the quality of the risk control options generated; thus the lack of tools to move healthcare personnel beyond this reliance on administrative controls is a critical limiting factor in the effectiveness of the healthcare risk management process. On this last point, we have developed the generating options for active risk control (GO-ARC) technique,(16) a structured brainstorming approach analogous to the structured what-if technique(11) and hazard and operability studies(37) techniques for hazard identification. The GO-ARC technique uses a series of five brainstorming prompts to elicit risk control options. These prompts are: elimination of the hazard or target, design controls (aka engineering controls), administrative controls, detection/situational awareness, and preparedness. Early evidence indicates that this approach may improve the quantity, quality, variety, and novelty of risk control options generated for consideration.(16) But, although improved risk control options generation is necessary, it is clearly not sufficient for improved risk control performance and patient safety improvement. As we have identified in this study, healthcare organizations currently lack the tools to make well-informed decisions about which options to implement, or how to determine whether implemented risk controls are successful. The lovebug diagram, a recently introduced hybrid of the fishbone diagram(38,39) and force-field analysis,(40,41) may help users better assess the forces operating for and against the success of a risk control option, thus informing decisions about which option(s) to adopt.(42) But the lack of any approaches beyond risk matrices for evaluating the eventual success of a risk control remains problematic. Although a risk matrix may (or, given the weak guidance provided by many trusts and the limitations of the tool itself, may not) be a sufficient approach for assessing the effectiveness of a risk control, it is probably inadequate to the task of evaluating success. An evaluation of success should also take into account broader issues such as cost-effectiveness (Might another risk control have had a greater effect at less cost?) and side-effects (Might the risk control have led
5 to a net increase in risk, through the introduction of new risks that outweigh the risk reduction? Or might the risk control have been even more successful than planned, because it reduced other significant risks in addition to the risk of interest?). On the primary focus of this study, the answer is clear: there is a pressing need for the development of a comprehensive framework for risk control practice in healthcare and the implementation of that framework through supporting tools and techniques. Turning to the secondary question of risk evaluation, we now examine whether these trusts are equipped to determine which risks require control.
3.3. Risk Evaluation Guidance 3.3.1. Tools The documents provided by one mental health trust did not cover the risk evaluation process; this was the same trust that did not provide any risk control guidance. Of the remaining 24, all relied exclusively on the use of risk matrices(10,43,44) for risk evaluation. The underlying theory behind the use of a risk matrix is that risk can be defined as the product of the likelihood of an event occurring and its consequence if it does occur (i.e., risk = likelihood × consequence). Thus real or potential adverse events are first assigned independent scores for likelihood and consequence (often on a scale of 1–5). The overall risk score is then computed as the product of those two scores, and lies at the intersection of the two numbers on a matrix (e.g., if likelihood = 3 and consequence = 4, risk = 12). The matrix is overlaid by predefined risk tolerability bands tied to action requirements and resource availability. For instance, a risks score of 1–7 might correspond to the green band (low risk: no action required), whereas a score of 8–15 might correspond to the amber band (medium risk: directorate-level action required, using existing directorate-level budgetary resources), and a score of 16–25 might place a risk in the red band (high risk: senior management action required and board to be informed, using organizational-level budgetary resources). See Fig. 1 for a risk matrix based on this example. Twenty-two trusts used 5 × 5 risk matrices. One used a 6 × 6 variation, in which an additional row and a column were included for scores of zero likelihood and zero consequence. And one trust used two overlapping matrices: an 8 × 9 risk matrix with a 5 × 5 matrix overlaid. A slight majority of trusts (13 of 24
6
Card, Ward, and Clarkson Likelihood Consequence 1
2
3
4
5
Unlikely
Possible
Likely
Almost Certain
5
10
15
20
25
4 Major
4
8
12
16
20
3 Moderate
3
6
9
12
15
2 Minor
2
4
6
8
10
1 Negligible
1
2
3
4
5
Rare 5 Catastrophic
Fig. 3. Scores assigned by trusts to a probability of 20%.
*light gray = green; medium gray = amber; dark gray = red
Fig. 1. A 5 × 5 risk matrix. Likelihood Consequence
1
2
3
4
5
Rare
Unlikely
Possible
Likely
Almost Certain
5 Catastrophic
5
10
15
20
25
4 Major
4
8
12
16
20
3 Moderate
3
6
9
12
15
2 Minor
2
4
6
8
10
1 Negligible
1
2
3
4
5
*white = green; light gray = yellow; medium gray = amber; dark gray = red
Fig. 2. Asymmetrical risk matrix from an acute trust.
that provided risk evaluation guidance) used matrices with four risk tolerability bands; seven used three risk tolerability bands, and four used a model with five bands. (These results treat all trusts as having 5 × 5 matrices. The 5 × 5 version was analyzed for the trust with two overlapping models, and the row and column for zero scores were ignored for the trust with a 6 × 6 matrix, which would otherwise have had five bands.) Five trusts (all of which were acute trusts) used asymmetrical risk tolerability bands, perhaps reflecting skepticism of the underlying model. In all five cases, the three means of arriving at a risk score of four (1 × 4, 2 × 2, 4 × 1) resulted in three different risk categorizations. See Fig. 2 for an example. 3.3.2. Consequence Scoring Twenty trusts provided consequence scoring guidance. This guidance was typically based on the
report “A Risk Matrix for Risk Managers” by the National Patient Safety Agency,(44) which recognizes that there are many different categories of consequences (e.g., safety, trust reputation, impact on operations, etc.). Sixteen trusts (80% of those that provided guidance) listed patient safety or general safety as the first category. Just as it is true that making patient safety the first item on the agenda for trust boards may increase the time spent engaging with the topic,(45) this may be intended to signal the importance of safety risks and lead to an increased focus on harm to patients and others. 3.3.3. Likelihood Scoring Sixteen trusts provided quantitative guidance for scoring likelihood. Two different quantitative approaches were used: assessment based on the likelihood of the event occurring within a given timeframe was used by 12 trusts (e.g., 1 = not expected to occur for years, 2 = yearly, 3 = monthly, 4 = weekly, 5 = daily), and assessment based on probability was used by 15 trusts (e.g., 1 = 90% of the time to warrant a likelihood score of 5. Further, 27% of trusts using probability-based scoring employed nondiscrete categories, which could lead to inconsistent scoring and poor allocation of scarce resources. For example, one trust used the following cut-off scores: 1 =
