Lessons Learned From Measuring Return on Investment in Public Health Quality Improvement Initiatives Lou Anne Crawley-Stout, MBA, CLSSBB, PMP; Kerri Ann Ward, BS; Claire Herring See, MHA; Greg Randolph, MD, MPH rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Context: The Center for Public Health Quality and its partner, North Carolina State University Industrial Extension Service, used 2 existing, yet similar quality improvement (QI) programs to provide technical assistance to conduct return on investment (ROI) and economic impact (EI) analyses so that they could estimate their QI projects’ financial impacts. Objectives: The objectives of this article are to describe the approach and ongoing learning from applying ROI and EI analyses to public health QI projects and analyze the results in order to illustrate ROI potential in public health. Design: We used a before-after study design for all ROI and EI analyses, spanning a 3-year time period. Setting: The study was conducted as part of 2 existing public health QI training programs that included webinars, face-to-face workshops, on-site facilitation, and longitudinal coaching and mentoring. Participants: The QI training programs included multidisciplinary teams from local and state public health programs in North Carolina. Main Outcome Measure: Return on investment and EI calculations. Results: Numerous adaptations were made over the 3 years of the ROI program to enhance participant’s understanding and application. Results show an average EI of $149 000, and a total EI in excess of $5 million for the 35 projects studied. The average ROI per QI project was $8.56 for every $1 invested in the project. Conclusions: Adapting the ROI approach was important in helping teams successfully conduct their ROI analyses. This study suggests that ROI analyses can be effectively applied in public health settings, and the potential for financial return is substantial.
J Public Health Management Practice, 2016, 22(2), E28–E37 C 2016 Wolters Kluwer Health, Inc. All rights reserved. Copyright
KEY WORDS: economic impact, process improvement, public
health, quality improvement, return on investment
Public health practice is undergoing rapid change, facing the need to deliver more with fewer resources. The adoption of quality improvement (QI), which has been used in many industries to improve the quality of Author Affiliations: North Carolina State University Industrial Extension Service and Center for Public Health Quality, Raleigh (Ms Crawley-Stout); Center for Public Health Quality, Raleigh, North Carolina (Ms Ward and See and Dr Randolph); University of North Carolina School of Medicine, Chapel Hill (Dr Randolph); and University of North Carolina Gillings School of Global Public Health, Chapel Hill (Dr Randolph). This article was supported by grants from the Duke Endowment, Blue Cross Blue Shield of North Carolina Foundation, and by funds made available from the Centers for Disease Control and Prevention, Office for State, Tribal, Local and Territorial Support under grant number 5U58CD001291-03. The content of this article are those of the authors and do not necessarily represent the official position of or endorsement by the Centers for Disease Control and Prevention. The devoted time, effort, and follow-through from the local and state teams that participated in the QI 101 program made this article possible. Their commitment to improve processes and calculate the project economic impact and return on investment is worthwhile and greatly appreciated. In particular, the authors are grateful to the DPH Tobacco Prevention and Control Branch in supporting their ongoing EI/ROI training through the use of their team’s actual ROI calculation and results. Finally, the NC State IES Lean Healthcare expertise contributed greatly to the initiation and continuance of our ROI/EI work. The authors are especially grateful to Gene Smith and Jim Kurian for improvement changes to their approach and form formatting and for their ROI guidance and support to the QI teams. No conflicts of interest exist for the authors. Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (http://www.JPHMP.com). Correspondence: Lou Anne Crawley-Stout, MBA, CLSSBB, PMP, North Carolina State University Industrial Extension Service and Center for Public Health Quality, PO Box 18763, Raleigh, NC 27619 ([email protected]). DOI: 10.1097/PHH.0000000000000229
E28 Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Lessons Learned From Measuring Return on Investment in Public Health Quality Improvement Initiatives
services and products, is one of the major changes in the field.1 In addition, governmental fiscal austerity is placing intense pressure on public health agencies, and indeed on all aspects of government, to demonstrate the value and impact of the public funds they receive.2 Quality improvement is a definitive process that responds to community and population health needs using continuous and measurable approaches.3 It requires significant upfront investment (such as staff time, new hiring, training, etc), yet promises many potential benefits, including financial benefits. Similarly, population health improvements, a common focus of public health organizations, not only require upfront investments but also promise health benefits and potential health care savings. To justify QI efforts, and especially those involving population health improvements, leaders have to understand the potential costs and benefits of the proposed work. Return on investment (ROI) analysis is a method that is widely used in business settings to examine costs and benefits related to strategies and investments.4 Since ROI evolved as a business tool, it is not widely understood, nor frequently utilized by public health leaders and practitioners. However, largely due to the aforementioned changes and a pressing urgency to optimize program efficiency and outcomes in today’s economic climate, ROI assessment is increasingly receiving attention among public health leaders. A notable example is that ROI was the theme of the 2013 National Public Health Week.5 The objectives of this article are twofold. First, we describe the approach and ongoing learning from a crosssector partnership between the Center for Public Health Quality (CPHQ) and the North Carolina State University Industrial Extension Service (IES). This partnership has been applying ROI analysis to QI projects at local and state governmental public health organizations for 3 years. We summarize IES and CPHQ’s unique experiences in performing ROI analysis on the QI projects that they support, including how they adapted ROI from business settings to public health settings. The second objective is to analyze the ROI results of QI projects completed by IES and CPHQ in order to illustrate the potential ROI that public health QI projects can yield.
● Methods Context, setting and participants The CPHQ and the IES designed and administered 2 QI training programs. The Public Health Quality Improvement 101 (PH QI 101) program provides QI training and project support for local health departments (LHDs) and has been described in detail in previous studies.6,7 The Division of Public Health Quality Im-
❘ E29
provement 101 (DPH QI 101) is a similar program that was adapted to meet the needs of state public health programs. Both are experiential learning programs designed to build capacity to implement QI in local and state public health entities. Both programs are taught by QI experts from CPHQ and IES, hereby referred to as CPHQ/IES faculty. The QI programs are conducted in cohorts that include 5 to 10 volunteer teams that arise from local and state entities and their partners. Throughout the program, teams learn by applying QI tools and methods to a specific project within their workplace. Over the 8-month program, the teams attend educational sessions and receive coaching and technical assistance via on-site facilitation, webinars, and two 2-day workshops. During these sessions, they learn methods and tools to test, implement, spread, and sustain QI initiatives and conduct ROI analysis. Teams also participate in a 4-day rapid improvement event, or kaizen event (a formalized, focused event performed by the team over several days to achieve dramatic improvements), that includes on-site facilitation by a Lean expert from IES, hereby referred to as IES specialist. Lean is a continuous process improvement methodology that focuses on identifying and eliminating wasteful activities, “defining value from the customer’s perspective . . . .”8 The program culminates with teams developing ROI and economic impact (EI) calculations for their QI projects.
ROI calculation method We used a before-after study design9 for all ROI and EI analyses, spanning a 3-year time period. This design enabled us to compare pre- to postchange intervention results. Return on investment is a performance measure used to evaluate the efficiency of an investment. Economic impact refers to the analysis of costs and benefits of an activity. Financial benefits are tangible benefits that can be converted to monetary values. For the QI 101 programs, financial benefits have subcategories: internal benefits are related to LHD or state programmatic improvements (such as time savings due to improving patient visit time or reducing the number of process steps) and external benefits (such as reducing health care costs by preventing patients from going to the emergency department or identifying child developmental issues early that impact lifelong education costs) that reflect population or community-related benefits. Costs are distinguished as one-time costs that occur during the QI 101 program and represent the cost to implement the project (such as team time) and recurring costs, which are those that will be routinely incurred to continue the changed process and sustain the QI initiative.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
E30 ❘ Journal of Public Health Management and Practice During trainings, each team is expected to think through these EI/ROI components and apply the formulae that are listed below: EI = Total financial benefits – Total costs ROI =
EI Total financial benefits − Costs or Total costs Total costs
Table 1 shows an example analysis. In the example, the QI team incorporated into a small clinic a tobacco cessation approach—the 5As: Ask, Advise, Assess, Assist, and Arrange. The project’s internal benefits resulted from embedding a template in the clinic’s electronic medical record that prompted providers to acquire and document the appropriate 5A information. Submitting the required documentation on approximately 1300 patients per year resulted in additional Medicaid revenue. The team also achieved financial benefits related to time savings by implementing checklists and protocols that further streamlined and supported efficiencies in the new process. When the clinic began referring tobacco-using patients to the Quitline (QuitlineNC provides free cessation services to any North Carolina resident who needs help quitting tobacco use12 ), they were able to apply evidence, per the cited resources, to support the financial benefits
outlined in the external/community benefits section of Table 1. To calculate one-time project costs to implement improvements, the team applied an average salary rate to their time invested in the project. They also included costs of educational materials and supplies as one-time and recurring costs. These 3 elements—Internal Benefits, External Benefits, and Costs—are then applied to the ROI formulae and results interpreted as follows: r Internal ROI: For every $1.00 invested in this QI project, $3.31 was returned to the participating clinic (via time savings and additional Medicaid revenue). These financial returns can then be reinvested into ongoing QI opportunities at the clinic. r Combined internal and external ROI: For every $1.00 invested in this QI project, $18.36 was returned to the clinic and the community (via the internal savings mentioned previously and the reduced medical costs and increased productivity of the non–tobacco-using workforce). In addition to financial benefits, each team was required to include intangible, or nonmonetary, benefits. Although many methods exist for converting intangibles into monetary values, such as the Intangible Benefit Multiplier Pyramid Approach,13 we chose not to
TABLE 1 ● Example ROI Calculation—Tobacco Prevention: Implementing 5As in a Small Clinic
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq Project Aim Increase the use of the five major components – 5As: Ask, Advise, Assess, Assist, Arrange – in a model site with a champion provider for low income Medicaid-eligible, uninsured and underinsured tobacco users. Internal Benefits - annualized Calculations Clinic Labor/time savings $1,080 Additional Medicaid Revenue 15,509 Total $16,589 ROI Formula: Financial Benefits – Costs/Costs External/Community Benefits Reduced smokers via Quitline referrals. Year 1 – Internal only Clinic averages 102 Quitline referrals per year: 16,589 – 7,260 = 9,329/7,260 = 1.28 • 50% of paent referrals accept Quitline services and 30% of • •
10 those paents actually quit : 102 x 50% x 30% = 15.3 paents per year who quit Companies spend $3,856 per year in direct medical costs and 11 lost producvity 15 paents x $3,856 = $57,840 annual community benefit
Costs Education materials Patient folders Smoking cessation table Tobacco team time Clinic provider time Total
Year 1 $325 130 25 3,400 2,950 $6,830
Recurring $300 130
$430
Year 1 – Internal + External (16,589 + 57,840) – 7,260 = 67,169/7,260 = 9.25 Overall Results (2 years combined) – Internal Only 33,178 – 7,690 = 25,488/7,690 = 3.31 Overall Results (2 years combined) – Internal + External 33,178 + 115,680 – 7,690 = 141,168/7,690 = 18.36
Note: Year 1 includes costs incurred during the QI101 program as well as Recurring costs during the postimprovement period of one year.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Lessons Learned From Measuring Return on Investment in Public Health Quality Improvement Initiatives
quantify these benefits. First, monetizing intangibles can be complex and time-consuming,14 and we needed to keep our model simple since public health agencies have limited resources. In addition, the process for converting intangibles is subjective,14 and we did not want to decrease the ROI’s credibility with stakeholders. In our Table 1 case example, the tobacco team’s intangible benefits included the following: r Increased clinical staff/physician knowledge through 5A and tobacco cessation training that enabled them to better serve patients; r Identified smokers more readily through the new tobacco questionnaire/template that was embedded in the electronic medical record for provider use; included steps and reminders for all 5As, billing, medication, and counseling; and r Improved patient knowledge of secondhand smoke effects and health care risks due to 5A implementation. Teams are taught that costs used in the ROI calculation relate to expenses incurred while attending trainings, the kaizen event and team meetings; testing and implementing improvements; and/or purchasing supplies or materials needed for improvements. During ROI training, teams are instructed to track all expenses related to the QI project. The key expense incurred is team member time. The total team member time is easily converted into an hourly salary amount. Any other expenses, whether one-time or recurring, need to be retained by the team for the ROI analysis.
Data collection and analysis Early in the QI training, teams develop an aim statement, which defines the overall purpose and importance of the project and includes measurable goals that ultimately help teams demonstrate the effectiveness of their improvements. In addition, they develop a measurement plan that includes the following components15 : r Outcome measures—the ultimate result the team is trying to achieve r Process measures—what is done to achieve the desired outcome r Balancing measures—something that inadvertently could be harmed while trying to improve the process and outcome Teams are guided to define at least 1 goal for each measure type and document their measures in a detailed project measurement table (see measurement table, Supplemental Digital Content 0, available at http://links.lww.com/JPHMP/A129). After developing the measurement table, the baseline data were
❘ E31
gathered, change interventions were tested and implemented, and remeasurement of the goals occurred. When these postimplementation data had been collected, they were compared with the baseline data and analyzed for actual improvements derived from the QI project. From this information and the successful process changes that were adopted, teams determined the benefits that could be converted into monetary values and annualized. As the measurements related directly to the QI project’s aim, they most often became part of the data utilized in the ROI calculation. For example, in the Table 1 tobacco prevention case, they increased the number of clinic patient referrals to the Quitline. Since the clinic had not been referring patients previously, the QI team implemented a referral process that generated more than 100 patient referrals per year. Evidence10 shows that such referrals lead to an increase of patients who actually stop tobacco use, which is linked to savings in direct medical costs and lost productivity for employers.11 Thus, the referral goal derived from the aim statement became the basis for a financial benefit that can be utilized in the ROI calculation. Although teams were introduced to ROI calculation methods early in the program, they were also prompted by their IES specialist to consider the benefits of their change ideas during the kaizen event. Draft ROI reports, which were due 60 days after the final workshop, were submitted by QI teams to the CPHQ ROI team (several staff who were trained how to review submissions). This ROI team checked the benefits that were converted into monetary values and all mathematical calculations, probed for potential opportunities to monetize benefits that were overlooked, verified that sufficient evidence was given to support the findings, and ensured that all estimates of financial impact were conservative. The QI teams also submitted an ROI draft report to their senior leader for initial review and feedback. Within the next 30 days, the final report was submitted to their senior leader for approval before submitting it to CPHQ. Upon receiving the completed ROI analyses, CPHQ staff entered ROI results into a Microsoft Excel spreadsheet. The spreadsheet was designed to filter data and provide mean and cumulative results, which could be customized for CPHQ stakeholders as well as analysis for the present study.
Reporting ROI/EI results Guidance about communicating ROI results to stakeholders is provided to QI teams during the final workshop, when teams’ projects are wrapping up and they are ready to begin the calculations. Communication tips include the following:
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
E32 ❘ Journal of Public Health Management and Practice r Refer to the stakeholder analysis that was included in the QI project’s charter. Develop a plan for when, how, and what to communicate to specific stakeholder audiences. r Schedule face-to-face presentations since ROI is relatively new for public health audiences. In-person presentation allows a more thorough explanation of the methodology and a chance to emphasize meaningful aspects of the ROI analysis. It also provides opportunity to address questions immediately and dispel misinterpretations. r Illustrate with examples that the ROI calculation is based on evidence, thereby strengthening its credibility. When converting benefits and costs into monetary values, describe the conservative approach utilized. r Focus on how the ROI contributes to achievement of the organization’s mission versus solely on dollars saved. r Negate assumptions that positive results should be used to reduce budgets. This kind of thinking could starve their most innovative and effective performers. r Emphasize the value of the intangible benefits. Sometimes overlooked, but when presented effectively often carry substantial influence. r Consider stakeholders with whom ROI results should not be communicated. Some projects are important to improve although labor-intensive team time causes costs to exceed financial benefits. Misunderstood negative ROIs can lead to devaluation of intangible benefits.
Interventions Although IES had been teaching and utilizing ROI in QI efforts in manufacturing and health care settings, the adaptation and application of ROI to the QI 101 public health projects evolved through substantial iterative learning. These adaptations over the 3-year period are listed below.
Adaptation 1: Early introduction of ROI in the QI 101 program Because ROI was not included in the QI 101 training programs until 2011 (the PH QI 101 program originated in 2010 and DPH QI 101 in 2011), ROI had to be introduced to early cohorts retroactively. We learned quickly that this retroactive approach was neither effective nor sustainable and also saw inconsistencies in ROI calculation approaches, accuracy, and the number of completed submissions. In these early cohorts, processes and expectations for teams to complete and
submit ROI work were less defined, thus we had limited success in obtaining accurate and timely ROI results. The newness of ROI terms and critical thinking, coupled with learning how to apply the knowledge to calculate financial benefits accurately and credibly, was commonly reported as issues by teams. In 2011, ROI training was incorporated into QI 101 near the end of the program. After observing many teams struggle to complete ROI analyses, receiving participant feedback about needing ROI knowledge at the beginning of the program, and researching ROI best practices, we began to introduce ROI thinking early for subsequent QI 101 cohorts.
Adaptation 2: Ongoing technical assistance and coaching To align with the early introduction of ROI in the training programs, we also asked each QI team’s faculty coach to address ROI throughout the coaching experience to continue to sensitize team thinking toward this work. In addition, IES specialists helped the team identify intangible and tangible benefits and costs during the kaizen event, while process improvements are fresh in their minds.
Adaptation 3: Improved structure and guidance for QI teams To help QI 101 participants understand the entire ROI process, CPHQ/IES faculty created a model to guide them through planning, executing, and communicating ROI results. This “big picture” approach proved important to provide them with understanding early in the QI project. The “ROI Process Model for Public Health” is shown in the Figure.
Adaptation 4: New tools and templates to facilitate ROI process In 2012, we created new tools to aid QI teams in developing ROI analyses, while promoting a standardized method. The ROI Memory Jogger Questionnaire (see questionnaire, Supplemental Digital Content 1, available at http://links.lww.com/JPHMP/A130) and the ROI Memory Jogger Worksheet (see worksheet, Supplemental Digital Content 2, available at http://links. lww.com/JPHMP/A131) prompt QI teams to identify benefits and costs to include in their calculation. Participants confirmed that the tools were useful in directing their thinking and approach.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Lessons Learned From Measuring Return on Investment in Public Health Quality Improvement Initiatives
❘ E33
FIGURE ●
ROI Process Model for Public Healtha qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Abbreviation: ROI, return on investment. a Adapted with permission from Phillips.9
Adaptation 5: Changing ROI calculations to be more comprehensive
TABLE 2 ● Improvements to Increase ROI Analysis
Based on feedback from participants and stakeholders, we altered the ROI calculation method to better match the needs of QI projects in public health settings and the QI 101 program. For example, we changed the time period from 1 to 2 years and began utilizing a onetime versus recurring costs approach to offset year 1 costs. These costs were associated with team time to learn and apply QI concepts and tools at workshops, webinars, kaizen events, and so forth, and would not occur in the project’s year 2. We also added the external/community benefit section to distinguish between the benefits to the program/organization and benefits to the community (external). This change not only provided clarification as to internal and external benefits but also accentuated the far-reaching effects that some improvements had on the community. Related to these changes was the adaptation of our ROI Calculation Form (see form, Supplemental Digital Content 3, available at http://links.lww.com/JPHMP/ A132) to better provide step-by-step guidance for listing the 2-year benefits and costs as well as formulae to automatically generate the ROI and the EI.
Problem
Adaptation 6: Ensuring teams complete the ROI assignment In early cohorts, we had difficulty obtaining ROI analyses from PH QI 101 teams. This was due to several problems that are presented, along with improvements, as shown in Table 2. By implementing changes, a more recent PH QI 101 cohort achieved 100% ROI analysis completion. Since DPH QI 101 participants had access to an on-site IES specialist for coaching and follow-up,
Completion Rate qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq Insufficient collection methods and follow-up on the part of CPHQ/IES faculty and follow-through by project teams
Uncertainty as to CPHQ/IES faculty roles and responsibilities regarding ROI submission protocol and final review ROI analysis was not presented to nor required by the early PH QI 101 cohorts
Improvements Created a process and spreadsheet to track completion of ROI draft and final documents Established and shared a timeline specifying ROI milestones and due dates Created a formal process and clarified roles
Required all QI 101 participants to perform ROI analysis
Abbreviations: CPHQ/IES, Center for Public Health Quality/ Industrial Extension Service; ROI, return on investment.
high completion rates were achieved from this program’s inception.
● Results Between June 2011 and March 2014, 83 teams completed QI 101. Nineteen of these teams were not required to submit ROI results (early cohorts) or are currently in the process of developing the analyses. It is the 64 remaining teams that are included in the present study of ROI results. Approximately 55% of these 64 teams completed an ROI analysis. Table 3 illustrates the internal and external financial benefits, costs, EI, and ROI over 2 years for 35 state
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
E34 ❘ Journal of Public Health Management and Practice
TABLE 3 ● ROI Results
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Project
Internal Benefit
Overall project results for 2-year period—DPH program (in dollars) Course management process 41 880 Clinic use of 5As 33 178 Customer service response time 282 890 Early hearing screening/diagnostic 94 820 Update application process 62 280 Web site for placement services 0 Monitoring/administrative review process 108 205 Home visiting data collection 15 971 Program evaluation process 19 916 Purchasing process 15 016 Food code interpretation consistency 50 240 Total—DPH 724 396 Average—DPH 65 854 Overall project results for 2-year period—LHD program (in dollars) Child health lead time 47 252 Septic system permit process 153 048 Immunization rate 66 456 Physical examination lead time 53 219 Family planning lead time 171 248 Abnormal PAPs 158 280 Family planning lead time 56 510 Women, infant and children appointments 16 224 Patient scheduling 1288 New physical lead time 122 766 Inventory report accuracy 61 450 Clinic flow 115 676 Internal documentation 17 016 Family planning lead time 103 645 Diabetes clinic flow and caseload 129 267 Maternal health clinic lead time 13 960 Maternity lead time 20 634 Pediatric clinic lead time 385 304 Adult clinic flow 19 939 Clinic survey tools/data collection 2000 WIC clinic lead time 41 486 Children clinic lead time 222 196 Patient lead time 99 340 File server reorganization 376 258 Total—LHD 2 454 463 Average—LHD 102 269 Overall total—LHD and DPH 3 178 859 Overall average—LHD and DPH 90 825
Overall ROI (All BenefitsCosts/Costs)
External Benefit
Costs
Overall EI (Costs-Benefits)
0 184 284 0 658 440 0 533 580 0 77 136 0 0 0 1 453 440 132 131
10 050 7285 8556 24 800 20 243 24 911 16 301 20 865 11 191 17 957 26 579 188 738 17 158
31 830 210 177 274 334 728 460 42 037 508 669 91 904 72 242 8725 − 2941 23 661 1 989 098 180 827
3.17 28.85 32.06 29.37 2.08 20.42 5.64 3.46 0.78 − 0.16 0.89
0 0 116 456 0 0 0 0 825 408 259 200 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 201 064 50 044 2 654 504 75 843
9150 10 575 62 540 12 478 3871 10 115 4902 15 538 3551 18 343 12 822 11 588 12 149 16 662 8646 5163 1500 17 076 15 327 12 770 10 008 62 717 64 847 19 121 421 458 17 561 610 196 17 434
38 102 142 473 120 372 40 742 167 377 148 165 51 608 826 094 256 937 104 423 48 628 104 088 4867 86 983 120 621 8797 19 134 368 228 4613 − 10 770 31 478 159 479 34 493 357 137 3 234 069 134 753 5 223 167 149 233
4.16 13.47 1.92 3.27 43.24 14.65 10.53 53.17 72.36 5.69 3.79 8.98 0.40 5.22 13.95 1.70 12.76 21.56 0.30 − 0.84 3.15 2.54 0.53 18.68
10.54
7.67 8.56
Abbreviations: DPH; EI, economic impact; LHD, local health department; PAPs; ROI, return on investment; WIC.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Lessons Learned From Measuring Return on Investment in Public Health Quality Improvement Initiatives
and local QI projects. Remarkably, all but 2 QI projects produced a positive EI. The negative ROIs result when the project costs exceed the financial benefits. Negative ROIs in QI 101 projects were due to laborintensive projects that raised costs (such as Web site or survey-related improvements) and inability to make the changes necessary to generate financial benefits (such as QI teams that encountered cultural barriers in interdepartment initiatives). The QI projects overall yielded greater internal ($3 178 859) than external ($2 654 504) benefits, although both were substantial. Overall EI averaged $149 233 and the average ROI was $8.56. Comparing state to local results, the average internal benefits among LHD projects exceeded those from state projects. However, average external benefits for the state projects exceeded local projects. Among the LHD teams, 46% completed ROI analysis. These LHD teams achieved more than $2.4 million in internal benefits as compared with $1.2 million in external/community benefits. These combined internal and external benefits minus costs of $421 458 resulted in more than $3.2 million in EI for the 24 projects or an average EI of $134 753. Their average ROI was $7.67. The DPH teams had a 92% ROI completion rate. The 11 DPH teams completing the analysis achieved more than $700 000 in internal benefits and nearly $1.5 million in external benefits. Considering the combined benefits and total costs of $188 738, their EI was more than $1.9 million, with an average EI of more than $180 000. Their ROI averaged $10.54.
● Discussion Investing in QI projects via the QI 101 programs generated substantial financial benefits at both the state and local levels. The greatest benefit from these 6- to 9-month QI projects was internal, yet the communities also reaped significant external benefits. Not surprising, given the often greater scope of state programs, state QI projects showed the greatest community benefits. The ROI process utilized by the CPHQ/IES partnership underwent a gradual transformation over the 3 years covered by this study, requiring a number of iterative adaptations derived from QI team and CPHQ/IES faculty feedback, unexpected results or problems, and best practices from other industries. Nevertheless, this study demonstrates that ROI analyses can be adapted and applied to QI projects in public health settings. Calculating ROI for public health initiatives has proven to be difficult—lag times exist between implementation and resulting benefits, calculating community benefit and the value of a life is difficult, and prov-
❘ E35
ing causation between a project and the benefit can be cumbersome.16 Despite these challenges, ROI is beginning to be used more frequently in public health to communicate the value of public health and prevention. Return on investment analyses are currently used in 3 areas: program effectiveness, aggregate public health spending, and QI projects.17 Although the resources for calculating ROI for public health improvement projects are limited, there are notable efforts in addition to the present study that are attempting to develop the tools that public health organizations need to calculate ROI for their projects. One of these efforts is the Association of State and Territorial Health Officials’ (ASTHO) ROI tool for improvement projects undertaken through the National Public Health Improvement Initiative. Like the CPHQ/IES tool, the calculations are done for the user once the data are collected and entered into the Excel spreadsheet. Although this tool is still in a pilot phase and is being tested by a user network, ASTHO has incorporated many of the same fundamental concepts as the CPHQ/IES effort. For example, the ASTHO model stresses the importance of building ROI into the project inception, measuring baseline data, categorization of costs and benefits, and involving both the QI team and the end users (ie, senior leaders) in the creation of the ROI report. Notable differences include ASTHO’s inclusion of more complex concepts and calculations such as amortization and present value calculations.17 Another effort is the Center for Health Care Strategies ROI Forecasting Calculator and ROI Template (template was developed by researchers at the University of North Carolina at Chapel Hill). Like both the CPHQ/IES tool and the ASTHO tool, Center for Health Care Strategies’ tool also calculates the ROI for the user once the data are collected and entered. However, this tool is specific to clinical projects focused on patients served by Medicaid and requires detailed information such as risk stratification.18 Return on investment calculators also exist for specific QI projects, such as the Public Health Institute’s ROI tool for remote patientmonitoring programs19 ; however, these tools may be too project-specific to be generalized to QI projects. Although ROI is just now emerging as a priority in public health, it is more common in other industries, such as health care. However, existing health care models have industry-specific components that are not easily adapted for public health. In addition, these models are resource-intensive to complete and recommend using complex calculations to adjust for inflation, discount future money, and depreciate equipment. These models also advocate for and suggest the involvement of multiple groups within the organization such as QI program staff, financial officers, clinical staff, statisticians, data analysts, programmers, and consultants.20
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
E36 ❘ Journal of Public Health Management and Practice These resources are unlikely to be available in many public health settings. Ultimately, the approach to using ROI in public health settings must balance rigor with being simple enough to execute with the limited resources and overworked staff often seen in public health organizations.16 In the present study, we attempted to balance adequate rigor with enough simplicity and support to allow public health organizations that have never used ROI to successfully assess the EI of their QI work. In addition, we hope that the training and experience they gained will allow them to hardwire ROI calculations into other continuous improvement projects beyond the QI 101 program. As stated earlier, the QI/Lean approach utilized in the QI 101 program is a continuous improvement methodology. As program participants learn and adopt this continuous improvement mindset and apply it to future process improvements, this may perpetuate additional EI/ROI. Thus, ROI could continue to evolve depending on the impact of the changes on goals and measurements.
potential bias had little to no impact on our results because we promoted an atmosphere of accomplishment for team learning and project completion, honesty, and objectivity in reporting, and emphasized the importance of intangible benefits. The ROI Process Model for Public Health includes the communication of results to key stakeholders as the final step. We were not able to monitor how teams communicated their ROI results—assisting teams with generating their final ROI analyses and continuously improving our assistance required all of the resources available in the present study. Going forward, we intend to focus more attention to communication of ROI results. This topic may be a good one for future studies. A final potential limitation is that the ROI analyses in the present study were from the perspective of the QI teams and their organizations, thus we did not include the costs of the grant-funded QI 101 training program. These costs (a one-time cost of approximately $18 000 per team) were not included since they were not paid by the team’s organization. Thus, a societal perspective would have demonstrated a lower EI (though still substantial) than presented in the present study.
Limitations Quality improvement project team members were generally not financial officers, health economists, or accountants. Thus, their ROI results likely underestimate the actual financial benefits because of difficulty in identifying benefits and converting these to monetary values. We also stressed in all calculations that in order for ROI to be credible for stakeholders, financial estimates should be conservative. In addition, we were unable to estimate the financial benefits of the large number of public health professionals that learned how to apply QI to their work through the QI 101 program, such as the value of future QI projects they could perform and presumably would yield additional financial benefits. Thus, again, our estimates of EI and ROI are likely underestimates. Teams participating in QI 101 had a large amount of expert support, such as assistance in performing their assignments and engagement with their leadership to encourage supporting the time needed for the team to work on their QI project as well as addressing barriers. Thus, these teams’ EI/ROI results may not be generalizable to other QI projects supported with less expertise and/or time and resources that result in lesser project results. Reporting negative ROIs could be disappointing to teams that worked so diligently with hopes of showing a positive return on their time and efforts. Thus, it is possible that teams with a negative EI/ROI may not have completed/shared their results, causing us to overstate overall results. However, we believe that this
● Conclusions Return on investment and EI analysis is increasingly being applied in public health. There is hope that it will be a viable and effective methodology to evaluate performance of public health QI projects and to justify program continuance during an era of decreasing budgets. In this study, QI teams employed ROI/EI methodology to determine whether the QI project’s benefits outweighed the investment. We have described our ROI approach and how we applied many lessons along the way to improve the method. Overall results to date reflect that ROI analyses can be effectively utilized in public health settings. Our article suggests that with ongoing practice and experience, ROI can become a meaningful, recognized application to assist public health in a new economic era. REFERENCES 1. Randolph GD, Lea CS. Quality improvement in public health: moving from knowing the path to walking the path. J Public Health Manag Pract. 2012;18(1):4-8. 2. Oberlander J. Health care policy in an age of austerity. N Engl J Med. 2011;365(12):1075-1077. 3. Riley WJ, Moran JW, Corso LC, Beitsch LM, Bialek R, Cofsky A. Defining quality improvement in public health. J Public Health Manag Pract. 2010;16(10):5-7. 4. Phillips P, Phillips J. The Basics. In: Return on Investment (ROI) Basics. Alexandria, VA: ASTD; 2005:1.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Lessons Learned From Measuring Return on Investment in Public Health Quality Improvement Initiatives
5. American Public Health Association. National public health week. http://www.nphw.org/. Accessed verified June 23, 2014 6. Cornett A, Thomas M, Davis MV, et al. Early evaluation results from a statewide quality improvement training program for local public health departments in North Carolina. J Public Health Manag Pract. 2012;18(1): 43-51 7. Davis M, Cornett A, Mahanna E, See C., Randolph G. Advancing quality improvement in public health departments through a statewide training program. J Public Health Manag Practice. In press. DOI: 10.1097/PHH.0000000000000165. 8. Martin K, Osterling M. The Kaizen Event Planner: Achieving Rapid Improvement in Office, Service, and Technical Environments. Boca Raton, FL: Productivity Press; 2007. 9. Zaza, S., Briss, P. A., Harris, K. W., eds., Resources. In: The Guide to Community Preventive Services: What Works to Promote Health? Oxford, UK: Oxford University Press; 2012:477. 10. Zbikowski SM. North Carolina Tobacco Use Quitline Final Comprehensive Evaluation Reports. Seattle, WA: Free & Clear, Inc. & Alere Wellbeing Inc; 2006-2011. 11. Centers for Disease Control and Prevention. Smokingattributable mortality, years of potential life lost, and productivity losses: United States, 2000-2004. Morb Mortal Wkly Rep. 2008;57(45):1226-1228. 12. QuitlineNC.com. http://www.quitlinenc.com/home. Updated 2014. 13. Moran J, Riley B. Calculating the real value of process improvement: Factoring in intangible benefits. Process Excell Netw. 2012. http://www.processexcellencenetwork.com/l.
❘ E37
14. Phillips J, Phillips P. Measuring. In: ROI at Work: Best-Practice Case Studies From the Real World. Alexandria, VA: ASTD; 2005:7-10. 15. QI toolbox—measures. Center for Public Health Quality Web site. http://centerforpublichealthquality.org/index.php/ theqitoolbox/qi-project-profile-tool/workplace-organization/ 94-step-by-step-guide/169-measures. Updated 2014. Accessed December 18, 2014. 16. Neumann PJ, Jacobson PD, Palmer JA. Measuring the value of public health systems: the disconnect between health economists and public health practitioners. Am J Public Health. 2008;98(12):2173-2180. 17. Ensign K. Estimating return on investment for public health improvements: tutorial on using the new tool. http:// www.astho.org/Accreditation-and-Performance/Estimating -Return-on-Investment-for-Public-Health-Improvements/. Published 2013. Accessed June 23, 2014. 18. Center for Health Care Strategies, Inc. ROI forecasting calculator for quality initiatives. http://www.chcsroi.org/ Welcome.aspx. Published 2007. Accessed June 23, 2014. 19. Public Health Institute. New web-based ROI tool demonstrates return on investment of remote patient monitoring programs. http://www.phi.org/news-events/487/newweb-based-roi-tool-demonstrates-return-on-investment-ofremote-patient-monitoring-programs. Published 2013. Accessed June 23, 2014. 20. Institute for Healthcare Improvement. Adverse events prevented calculator. http://www.ihi.org/resources/Pages/ Tools/AdverseEventsPreventedCalculator.aspx. Accessed June 23, 2014.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Context: The Center for Public Health Quality and its partner, North Carolina State University Industrial Extension Service, used 2 existing, yet similar quality improvement (QI) programs to provide technical assistance to conduct return on investment (ROI) and economic impact (EI) analyses so that they could estimate their QI projects’ financial impacts. Objectives: The objectives of this article are to describe the approach and ongoing learning from applying ROI and EI analyses to public health QI projects and analyze the results in order to illustrate ROI potential in public health. Design: We used a before-after study design for all ROI and EI analyses, spanning a 3-year time period. Setting: The study was conducted as part of 2 existing public health QI training programs that included webinars, face-to-face workshops, on-site facilitation, and longitudinal coaching and mentoring. Participants: The QI training programs included multidisciplinary teams from local and state public health programs in North Carolina. Main Outcome Measure: Return on investment and EI calculations. Results: Numerous adaptations were made over the 3 years of the ROI program to enhance participant’s understanding and application. Results show an average EI of $149 000, and a total EI in excess of $5 million for the 35 projects studied. The average ROI per QI project was $8.56 for every $1 invested in the project. Conclusions: Adapting the ROI approach was important in helping teams successfully conduct their ROI analyses. This study suggests that ROI analyses can be effectively applied in public health settings, and the potential for financial return is substantial.
J Public Health Management Practice, 2016, 22(2), E28–E37 C 2016 Wolters Kluwer Health, Inc. All rights reserved. Copyright
KEY WORDS: economic impact, process improvement, public
health, quality improvement, return on investment
Public health practice is undergoing rapid change, facing the need to deliver more with fewer resources. The adoption of quality improvement (QI), which has been used in many industries to improve the quality of Author Affiliations: North Carolina State University Industrial Extension Service and Center for Public Health Quality, Raleigh (Ms Crawley-Stout); Center for Public Health Quality, Raleigh, North Carolina (Ms Ward and See and Dr Randolph); University of North Carolina School of Medicine, Chapel Hill (Dr Randolph); and University of North Carolina Gillings School of Global Public Health, Chapel Hill (Dr Randolph). This article was supported by grants from the Duke Endowment, Blue Cross Blue Shield of North Carolina Foundation, and by funds made available from the Centers for Disease Control and Prevention, Office for State, Tribal, Local and Territorial Support under grant number 5U58CD001291-03. The content of this article are those of the authors and do not necessarily represent the official position of or endorsement by the Centers for Disease Control and Prevention. The devoted time, effort, and follow-through from the local and state teams that participated in the QI 101 program made this article possible. Their commitment to improve processes and calculate the project economic impact and return on investment is worthwhile and greatly appreciated. In particular, the authors are grateful to the DPH Tobacco Prevention and Control Branch in supporting their ongoing EI/ROI training through the use of their team’s actual ROI calculation and results. Finally, the NC State IES Lean Healthcare expertise contributed greatly to the initiation and continuance of our ROI/EI work. The authors are especially grateful to Gene Smith and Jim Kurian for improvement changes to their approach and form formatting and for their ROI guidance and support to the QI teams. No conflicts of interest exist for the authors. Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (http://www.JPHMP.com). Correspondence: Lou Anne Crawley-Stout, MBA, CLSSBB, PMP, North Carolina State University Industrial Extension Service and Center for Public Health Quality, PO Box 18763, Raleigh, NC 27619 ([email protected]). DOI: 10.1097/PHH.0000000000000229
E28 Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Lessons Learned From Measuring Return on Investment in Public Health Quality Improvement Initiatives
services and products, is one of the major changes in the field.1 In addition, governmental fiscal austerity is placing intense pressure on public health agencies, and indeed on all aspects of government, to demonstrate the value and impact of the public funds they receive.2 Quality improvement is a definitive process that responds to community and population health needs using continuous and measurable approaches.3 It requires significant upfront investment (such as staff time, new hiring, training, etc), yet promises many potential benefits, including financial benefits. Similarly, population health improvements, a common focus of public health organizations, not only require upfront investments but also promise health benefits and potential health care savings. To justify QI efforts, and especially those involving population health improvements, leaders have to understand the potential costs and benefits of the proposed work. Return on investment (ROI) analysis is a method that is widely used in business settings to examine costs and benefits related to strategies and investments.4 Since ROI evolved as a business tool, it is not widely understood, nor frequently utilized by public health leaders and practitioners. However, largely due to the aforementioned changes and a pressing urgency to optimize program efficiency and outcomes in today’s economic climate, ROI assessment is increasingly receiving attention among public health leaders. A notable example is that ROI was the theme of the 2013 National Public Health Week.5 The objectives of this article are twofold. First, we describe the approach and ongoing learning from a crosssector partnership between the Center for Public Health Quality (CPHQ) and the North Carolina State University Industrial Extension Service (IES). This partnership has been applying ROI analysis to QI projects at local and state governmental public health organizations for 3 years. We summarize IES and CPHQ’s unique experiences in performing ROI analysis on the QI projects that they support, including how they adapted ROI from business settings to public health settings. The second objective is to analyze the ROI results of QI projects completed by IES and CPHQ in order to illustrate the potential ROI that public health QI projects can yield.
● Methods Context, setting and participants The CPHQ and the IES designed and administered 2 QI training programs. The Public Health Quality Improvement 101 (PH QI 101) program provides QI training and project support for local health departments (LHDs) and has been described in detail in previous studies.6,7 The Division of Public Health Quality Im-
❘ E29
provement 101 (DPH QI 101) is a similar program that was adapted to meet the needs of state public health programs. Both are experiential learning programs designed to build capacity to implement QI in local and state public health entities. Both programs are taught by QI experts from CPHQ and IES, hereby referred to as CPHQ/IES faculty. The QI programs are conducted in cohorts that include 5 to 10 volunteer teams that arise from local and state entities and their partners. Throughout the program, teams learn by applying QI tools and methods to a specific project within their workplace. Over the 8-month program, the teams attend educational sessions and receive coaching and technical assistance via on-site facilitation, webinars, and two 2-day workshops. During these sessions, they learn methods and tools to test, implement, spread, and sustain QI initiatives and conduct ROI analysis. Teams also participate in a 4-day rapid improvement event, or kaizen event (a formalized, focused event performed by the team over several days to achieve dramatic improvements), that includes on-site facilitation by a Lean expert from IES, hereby referred to as IES specialist. Lean is a continuous process improvement methodology that focuses on identifying and eliminating wasteful activities, “defining value from the customer’s perspective . . . .”8 The program culminates with teams developing ROI and economic impact (EI) calculations for their QI projects.
ROI calculation method We used a before-after study design9 for all ROI and EI analyses, spanning a 3-year time period. This design enabled us to compare pre- to postchange intervention results. Return on investment is a performance measure used to evaluate the efficiency of an investment. Economic impact refers to the analysis of costs and benefits of an activity. Financial benefits are tangible benefits that can be converted to monetary values. For the QI 101 programs, financial benefits have subcategories: internal benefits are related to LHD or state programmatic improvements (such as time savings due to improving patient visit time or reducing the number of process steps) and external benefits (such as reducing health care costs by preventing patients from going to the emergency department or identifying child developmental issues early that impact lifelong education costs) that reflect population or community-related benefits. Costs are distinguished as one-time costs that occur during the QI 101 program and represent the cost to implement the project (such as team time) and recurring costs, which are those that will be routinely incurred to continue the changed process and sustain the QI initiative.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
E30 ❘ Journal of Public Health Management and Practice During trainings, each team is expected to think through these EI/ROI components and apply the formulae that are listed below: EI = Total financial benefits – Total costs ROI =
EI Total financial benefits − Costs or Total costs Total costs
Table 1 shows an example analysis. In the example, the QI team incorporated into a small clinic a tobacco cessation approach—the 5As: Ask, Advise, Assess, Assist, and Arrange. The project’s internal benefits resulted from embedding a template in the clinic’s electronic medical record that prompted providers to acquire and document the appropriate 5A information. Submitting the required documentation on approximately 1300 patients per year resulted in additional Medicaid revenue. The team also achieved financial benefits related to time savings by implementing checklists and protocols that further streamlined and supported efficiencies in the new process. When the clinic began referring tobacco-using patients to the Quitline (QuitlineNC provides free cessation services to any North Carolina resident who needs help quitting tobacco use12 ), they were able to apply evidence, per the cited resources, to support the financial benefits
outlined in the external/community benefits section of Table 1. To calculate one-time project costs to implement improvements, the team applied an average salary rate to their time invested in the project. They also included costs of educational materials and supplies as one-time and recurring costs. These 3 elements—Internal Benefits, External Benefits, and Costs—are then applied to the ROI formulae and results interpreted as follows: r Internal ROI: For every $1.00 invested in this QI project, $3.31 was returned to the participating clinic (via time savings and additional Medicaid revenue). These financial returns can then be reinvested into ongoing QI opportunities at the clinic. r Combined internal and external ROI: For every $1.00 invested in this QI project, $18.36 was returned to the clinic and the community (via the internal savings mentioned previously and the reduced medical costs and increased productivity of the non–tobacco-using workforce). In addition to financial benefits, each team was required to include intangible, or nonmonetary, benefits. Although many methods exist for converting intangibles into monetary values, such as the Intangible Benefit Multiplier Pyramid Approach,13 we chose not to
TABLE 1 ● Example ROI Calculation—Tobacco Prevention: Implementing 5As in a Small Clinic
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq Project Aim Increase the use of the five major components – 5As: Ask, Advise, Assess, Assist, Arrange – in a model site with a champion provider for low income Medicaid-eligible, uninsured and underinsured tobacco users. Internal Benefits - annualized Calculations Clinic Labor/time savings $1,080 Additional Medicaid Revenue 15,509 Total $16,589 ROI Formula: Financial Benefits – Costs/Costs External/Community Benefits Reduced smokers via Quitline referrals. Year 1 – Internal only Clinic averages 102 Quitline referrals per year: 16,589 – 7,260 = 9,329/7,260 = 1.28 • 50% of paent referrals accept Quitline services and 30% of • •
10 those paents actually quit : 102 x 50% x 30% = 15.3 paents per year who quit Companies spend $3,856 per year in direct medical costs and 11 lost producvity 15 paents x $3,856 = $57,840 annual community benefit
Costs Education materials Patient folders Smoking cessation table Tobacco team time Clinic provider time Total
Year 1 $325 130 25 3,400 2,950 $6,830
Recurring $300 130
$430
Year 1 – Internal + External (16,589 + 57,840) – 7,260 = 67,169/7,260 = 9.25 Overall Results (2 years combined) – Internal Only 33,178 – 7,690 = 25,488/7,690 = 3.31 Overall Results (2 years combined) – Internal + External 33,178 + 115,680 – 7,690 = 141,168/7,690 = 18.36
Note: Year 1 includes costs incurred during the QI101 program as well as Recurring costs during the postimprovement period of one year.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Lessons Learned From Measuring Return on Investment in Public Health Quality Improvement Initiatives
quantify these benefits. First, monetizing intangibles can be complex and time-consuming,14 and we needed to keep our model simple since public health agencies have limited resources. In addition, the process for converting intangibles is subjective,14 and we did not want to decrease the ROI’s credibility with stakeholders. In our Table 1 case example, the tobacco team’s intangible benefits included the following: r Increased clinical staff/physician knowledge through 5A and tobacco cessation training that enabled them to better serve patients; r Identified smokers more readily through the new tobacco questionnaire/template that was embedded in the electronic medical record for provider use; included steps and reminders for all 5As, billing, medication, and counseling; and r Improved patient knowledge of secondhand smoke effects and health care risks due to 5A implementation. Teams are taught that costs used in the ROI calculation relate to expenses incurred while attending trainings, the kaizen event and team meetings; testing and implementing improvements; and/or purchasing supplies or materials needed for improvements. During ROI training, teams are instructed to track all expenses related to the QI project. The key expense incurred is team member time. The total team member time is easily converted into an hourly salary amount. Any other expenses, whether one-time or recurring, need to be retained by the team for the ROI analysis.
Data collection and analysis Early in the QI training, teams develop an aim statement, which defines the overall purpose and importance of the project and includes measurable goals that ultimately help teams demonstrate the effectiveness of their improvements. In addition, they develop a measurement plan that includes the following components15 : r Outcome measures—the ultimate result the team is trying to achieve r Process measures—what is done to achieve the desired outcome r Balancing measures—something that inadvertently could be harmed while trying to improve the process and outcome Teams are guided to define at least 1 goal for each measure type and document their measures in a detailed project measurement table (see measurement table, Supplemental Digital Content 0, available at http://links.lww.com/JPHMP/A129). After developing the measurement table, the baseline data were
❘ E31
gathered, change interventions were tested and implemented, and remeasurement of the goals occurred. When these postimplementation data had been collected, they were compared with the baseline data and analyzed for actual improvements derived from the QI project. From this information and the successful process changes that were adopted, teams determined the benefits that could be converted into monetary values and annualized. As the measurements related directly to the QI project’s aim, they most often became part of the data utilized in the ROI calculation. For example, in the Table 1 tobacco prevention case, they increased the number of clinic patient referrals to the Quitline. Since the clinic had not been referring patients previously, the QI team implemented a referral process that generated more than 100 patient referrals per year. Evidence10 shows that such referrals lead to an increase of patients who actually stop tobacco use, which is linked to savings in direct medical costs and lost productivity for employers.11 Thus, the referral goal derived from the aim statement became the basis for a financial benefit that can be utilized in the ROI calculation. Although teams were introduced to ROI calculation methods early in the program, they were also prompted by their IES specialist to consider the benefits of their change ideas during the kaizen event. Draft ROI reports, which were due 60 days after the final workshop, were submitted by QI teams to the CPHQ ROI team (several staff who were trained how to review submissions). This ROI team checked the benefits that were converted into monetary values and all mathematical calculations, probed for potential opportunities to monetize benefits that were overlooked, verified that sufficient evidence was given to support the findings, and ensured that all estimates of financial impact were conservative. The QI teams also submitted an ROI draft report to their senior leader for initial review and feedback. Within the next 30 days, the final report was submitted to their senior leader for approval before submitting it to CPHQ. Upon receiving the completed ROI analyses, CPHQ staff entered ROI results into a Microsoft Excel spreadsheet. The spreadsheet was designed to filter data and provide mean and cumulative results, which could be customized for CPHQ stakeholders as well as analysis for the present study.
Reporting ROI/EI results Guidance about communicating ROI results to stakeholders is provided to QI teams during the final workshop, when teams’ projects are wrapping up and they are ready to begin the calculations. Communication tips include the following:
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
E32 ❘ Journal of Public Health Management and Practice r Refer to the stakeholder analysis that was included in the QI project’s charter. Develop a plan for when, how, and what to communicate to specific stakeholder audiences. r Schedule face-to-face presentations since ROI is relatively new for public health audiences. In-person presentation allows a more thorough explanation of the methodology and a chance to emphasize meaningful aspects of the ROI analysis. It also provides opportunity to address questions immediately and dispel misinterpretations. r Illustrate with examples that the ROI calculation is based on evidence, thereby strengthening its credibility. When converting benefits and costs into monetary values, describe the conservative approach utilized. r Focus on how the ROI contributes to achievement of the organization’s mission versus solely on dollars saved. r Negate assumptions that positive results should be used to reduce budgets. This kind of thinking could starve their most innovative and effective performers. r Emphasize the value of the intangible benefits. Sometimes overlooked, but when presented effectively often carry substantial influence. r Consider stakeholders with whom ROI results should not be communicated. Some projects are important to improve although labor-intensive team time causes costs to exceed financial benefits. Misunderstood negative ROIs can lead to devaluation of intangible benefits.
Interventions Although IES had been teaching and utilizing ROI in QI efforts in manufacturing and health care settings, the adaptation and application of ROI to the QI 101 public health projects evolved through substantial iterative learning. These adaptations over the 3-year period are listed below.
Adaptation 1: Early introduction of ROI in the QI 101 program Because ROI was not included in the QI 101 training programs until 2011 (the PH QI 101 program originated in 2010 and DPH QI 101 in 2011), ROI had to be introduced to early cohorts retroactively. We learned quickly that this retroactive approach was neither effective nor sustainable and also saw inconsistencies in ROI calculation approaches, accuracy, and the number of completed submissions. In these early cohorts, processes and expectations for teams to complete and
submit ROI work were less defined, thus we had limited success in obtaining accurate and timely ROI results. The newness of ROI terms and critical thinking, coupled with learning how to apply the knowledge to calculate financial benefits accurately and credibly, was commonly reported as issues by teams. In 2011, ROI training was incorporated into QI 101 near the end of the program. After observing many teams struggle to complete ROI analyses, receiving participant feedback about needing ROI knowledge at the beginning of the program, and researching ROI best practices, we began to introduce ROI thinking early for subsequent QI 101 cohorts.
Adaptation 2: Ongoing technical assistance and coaching To align with the early introduction of ROI in the training programs, we also asked each QI team’s faculty coach to address ROI throughout the coaching experience to continue to sensitize team thinking toward this work. In addition, IES specialists helped the team identify intangible and tangible benefits and costs during the kaizen event, while process improvements are fresh in their minds.
Adaptation 3: Improved structure and guidance for QI teams To help QI 101 participants understand the entire ROI process, CPHQ/IES faculty created a model to guide them through planning, executing, and communicating ROI results. This “big picture” approach proved important to provide them with understanding early in the QI project. The “ROI Process Model for Public Health” is shown in the Figure.
Adaptation 4: New tools and templates to facilitate ROI process In 2012, we created new tools to aid QI teams in developing ROI analyses, while promoting a standardized method. The ROI Memory Jogger Questionnaire (see questionnaire, Supplemental Digital Content 1, available at http://links.lww.com/JPHMP/A130) and the ROI Memory Jogger Worksheet (see worksheet, Supplemental Digital Content 2, available at http://links. lww.com/JPHMP/A131) prompt QI teams to identify benefits and costs to include in their calculation. Participants confirmed that the tools were useful in directing their thinking and approach.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Lessons Learned From Measuring Return on Investment in Public Health Quality Improvement Initiatives
❘ E33
FIGURE ●
ROI Process Model for Public Healtha qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Abbreviation: ROI, return on investment. a Adapted with permission from Phillips.9
Adaptation 5: Changing ROI calculations to be more comprehensive
TABLE 2 ● Improvements to Increase ROI Analysis
Based on feedback from participants and stakeholders, we altered the ROI calculation method to better match the needs of QI projects in public health settings and the QI 101 program. For example, we changed the time period from 1 to 2 years and began utilizing a onetime versus recurring costs approach to offset year 1 costs. These costs were associated with team time to learn and apply QI concepts and tools at workshops, webinars, kaizen events, and so forth, and would not occur in the project’s year 2. We also added the external/community benefit section to distinguish between the benefits to the program/organization and benefits to the community (external). This change not only provided clarification as to internal and external benefits but also accentuated the far-reaching effects that some improvements had on the community. Related to these changes was the adaptation of our ROI Calculation Form (see form, Supplemental Digital Content 3, available at http://links.lww.com/JPHMP/ A132) to better provide step-by-step guidance for listing the 2-year benefits and costs as well as formulae to automatically generate the ROI and the EI.
Problem
Adaptation 6: Ensuring teams complete the ROI assignment In early cohorts, we had difficulty obtaining ROI analyses from PH QI 101 teams. This was due to several problems that are presented, along with improvements, as shown in Table 2. By implementing changes, a more recent PH QI 101 cohort achieved 100% ROI analysis completion. Since DPH QI 101 participants had access to an on-site IES specialist for coaching and follow-up,
Completion Rate qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq Insufficient collection methods and follow-up on the part of CPHQ/IES faculty and follow-through by project teams
Uncertainty as to CPHQ/IES faculty roles and responsibilities regarding ROI submission protocol and final review ROI analysis was not presented to nor required by the early PH QI 101 cohorts
Improvements Created a process and spreadsheet to track completion of ROI draft and final documents Established and shared a timeline specifying ROI milestones and due dates Created a formal process and clarified roles
Required all QI 101 participants to perform ROI analysis
Abbreviations: CPHQ/IES, Center for Public Health Quality/ Industrial Extension Service; ROI, return on investment.
high completion rates were achieved from this program’s inception.
● Results Between June 2011 and March 2014, 83 teams completed QI 101. Nineteen of these teams were not required to submit ROI results (early cohorts) or are currently in the process of developing the analyses. It is the 64 remaining teams that are included in the present study of ROI results. Approximately 55% of these 64 teams completed an ROI analysis. Table 3 illustrates the internal and external financial benefits, costs, EI, and ROI over 2 years for 35 state
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
E34 ❘ Journal of Public Health Management and Practice
TABLE 3 ● ROI Results
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Project
Internal Benefit
Overall project results for 2-year period—DPH program (in dollars) Course management process 41 880 Clinic use of 5As 33 178 Customer service response time 282 890 Early hearing screening/diagnostic 94 820 Update application process 62 280 Web site for placement services 0 Monitoring/administrative review process 108 205 Home visiting data collection 15 971 Program evaluation process 19 916 Purchasing process 15 016 Food code interpretation consistency 50 240 Total—DPH 724 396 Average—DPH 65 854 Overall project results for 2-year period—LHD program (in dollars) Child health lead time 47 252 Septic system permit process 153 048 Immunization rate 66 456 Physical examination lead time 53 219 Family planning lead time 171 248 Abnormal PAPs 158 280 Family planning lead time 56 510 Women, infant and children appointments 16 224 Patient scheduling 1288 New physical lead time 122 766 Inventory report accuracy 61 450 Clinic flow 115 676 Internal documentation 17 016 Family planning lead time 103 645 Diabetes clinic flow and caseload 129 267 Maternal health clinic lead time 13 960 Maternity lead time 20 634 Pediatric clinic lead time 385 304 Adult clinic flow 19 939 Clinic survey tools/data collection 2000 WIC clinic lead time 41 486 Children clinic lead time 222 196 Patient lead time 99 340 File server reorganization 376 258 Total—LHD 2 454 463 Average—LHD 102 269 Overall total—LHD and DPH 3 178 859 Overall average—LHD and DPH 90 825
Overall ROI (All BenefitsCosts/Costs)
External Benefit
Costs
Overall EI (Costs-Benefits)
0 184 284 0 658 440 0 533 580 0 77 136 0 0 0 1 453 440 132 131
10 050 7285 8556 24 800 20 243 24 911 16 301 20 865 11 191 17 957 26 579 188 738 17 158
31 830 210 177 274 334 728 460 42 037 508 669 91 904 72 242 8725 − 2941 23 661 1 989 098 180 827
3.17 28.85 32.06 29.37 2.08 20.42 5.64 3.46 0.78 − 0.16 0.89
0 0 116 456 0 0 0 0 825 408 259 200 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 201 064 50 044 2 654 504 75 843
9150 10 575 62 540 12 478 3871 10 115 4902 15 538 3551 18 343 12 822 11 588 12 149 16 662 8646 5163 1500 17 076 15 327 12 770 10 008 62 717 64 847 19 121 421 458 17 561 610 196 17 434
38 102 142 473 120 372 40 742 167 377 148 165 51 608 826 094 256 937 104 423 48 628 104 088 4867 86 983 120 621 8797 19 134 368 228 4613 − 10 770 31 478 159 479 34 493 357 137 3 234 069 134 753 5 223 167 149 233
4.16 13.47 1.92 3.27 43.24 14.65 10.53 53.17 72.36 5.69 3.79 8.98 0.40 5.22 13.95 1.70 12.76 21.56 0.30 − 0.84 3.15 2.54 0.53 18.68
10.54
7.67 8.56
Abbreviations: DPH; EI, economic impact; LHD, local health department; PAPs; ROI, return on investment; WIC.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Lessons Learned From Measuring Return on Investment in Public Health Quality Improvement Initiatives
and local QI projects. Remarkably, all but 2 QI projects produced a positive EI. The negative ROIs result when the project costs exceed the financial benefits. Negative ROIs in QI 101 projects were due to laborintensive projects that raised costs (such as Web site or survey-related improvements) and inability to make the changes necessary to generate financial benefits (such as QI teams that encountered cultural barriers in interdepartment initiatives). The QI projects overall yielded greater internal ($3 178 859) than external ($2 654 504) benefits, although both were substantial. Overall EI averaged $149 233 and the average ROI was $8.56. Comparing state to local results, the average internal benefits among LHD projects exceeded those from state projects. However, average external benefits for the state projects exceeded local projects. Among the LHD teams, 46% completed ROI analysis. These LHD teams achieved more than $2.4 million in internal benefits as compared with $1.2 million in external/community benefits. These combined internal and external benefits minus costs of $421 458 resulted in more than $3.2 million in EI for the 24 projects or an average EI of $134 753. Their average ROI was $7.67. The DPH teams had a 92% ROI completion rate. The 11 DPH teams completing the analysis achieved more than $700 000 in internal benefits and nearly $1.5 million in external benefits. Considering the combined benefits and total costs of $188 738, their EI was more than $1.9 million, with an average EI of more than $180 000. Their ROI averaged $10.54.
● Discussion Investing in QI projects via the QI 101 programs generated substantial financial benefits at both the state and local levels. The greatest benefit from these 6- to 9-month QI projects was internal, yet the communities also reaped significant external benefits. Not surprising, given the often greater scope of state programs, state QI projects showed the greatest community benefits. The ROI process utilized by the CPHQ/IES partnership underwent a gradual transformation over the 3 years covered by this study, requiring a number of iterative adaptations derived from QI team and CPHQ/IES faculty feedback, unexpected results or problems, and best practices from other industries. Nevertheless, this study demonstrates that ROI analyses can be adapted and applied to QI projects in public health settings. Calculating ROI for public health initiatives has proven to be difficult—lag times exist between implementation and resulting benefits, calculating community benefit and the value of a life is difficult, and prov-
❘ E35
ing causation between a project and the benefit can be cumbersome.16 Despite these challenges, ROI is beginning to be used more frequently in public health to communicate the value of public health and prevention. Return on investment analyses are currently used in 3 areas: program effectiveness, aggregate public health spending, and QI projects.17 Although the resources for calculating ROI for public health improvement projects are limited, there are notable efforts in addition to the present study that are attempting to develop the tools that public health organizations need to calculate ROI for their projects. One of these efforts is the Association of State and Territorial Health Officials’ (ASTHO) ROI tool for improvement projects undertaken through the National Public Health Improvement Initiative. Like the CPHQ/IES tool, the calculations are done for the user once the data are collected and entered into the Excel spreadsheet. Although this tool is still in a pilot phase and is being tested by a user network, ASTHO has incorporated many of the same fundamental concepts as the CPHQ/IES effort. For example, the ASTHO model stresses the importance of building ROI into the project inception, measuring baseline data, categorization of costs and benefits, and involving both the QI team and the end users (ie, senior leaders) in the creation of the ROI report. Notable differences include ASTHO’s inclusion of more complex concepts and calculations such as amortization and present value calculations.17 Another effort is the Center for Health Care Strategies ROI Forecasting Calculator and ROI Template (template was developed by researchers at the University of North Carolina at Chapel Hill). Like both the CPHQ/IES tool and the ASTHO tool, Center for Health Care Strategies’ tool also calculates the ROI for the user once the data are collected and entered. However, this tool is specific to clinical projects focused on patients served by Medicaid and requires detailed information such as risk stratification.18 Return on investment calculators also exist for specific QI projects, such as the Public Health Institute’s ROI tool for remote patientmonitoring programs19 ; however, these tools may be too project-specific to be generalized to QI projects. Although ROI is just now emerging as a priority in public health, it is more common in other industries, such as health care. However, existing health care models have industry-specific components that are not easily adapted for public health. In addition, these models are resource-intensive to complete and recommend using complex calculations to adjust for inflation, discount future money, and depreciate equipment. These models also advocate for and suggest the involvement of multiple groups within the organization such as QI program staff, financial officers, clinical staff, statisticians, data analysts, programmers, and consultants.20
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
E36 ❘ Journal of Public Health Management and Practice These resources are unlikely to be available in many public health settings. Ultimately, the approach to using ROI in public health settings must balance rigor with being simple enough to execute with the limited resources and overworked staff often seen in public health organizations.16 In the present study, we attempted to balance adequate rigor with enough simplicity and support to allow public health organizations that have never used ROI to successfully assess the EI of their QI work. In addition, we hope that the training and experience they gained will allow them to hardwire ROI calculations into other continuous improvement projects beyond the QI 101 program. As stated earlier, the QI/Lean approach utilized in the QI 101 program is a continuous improvement methodology. As program participants learn and adopt this continuous improvement mindset and apply it to future process improvements, this may perpetuate additional EI/ROI. Thus, ROI could continue to evolve depending on the impact of the changes on goals and measurements.
potential bias had little to no impact on our results because we promoted an atmosphere of accomplishment for team learning and project completion, honesty, and objectivity in reporting, and emphasized the importance of intangible benefits. The ROI Process Model for Public Health includes the communication of results to key stakeholders as the final step. We were not able to monitor how teams communicated their ROI results—assisting teams with generating their final ROI analyses and continuously improving our assistance required all of the resources available in the present study. Going forward, we intend to focus more attention to communication of ROI results. This topic may be a good one for future studies. A final potential limitation is that the ROI analyses in the present study were from the perspective of the QI teams and their organizations, thus we did not include the costs of the grant-funded QI 101 training program. These costs (a one-time cost of approximately $18 000 per team) were not included since they were not paid by the team’s organization. Thus, a societal perspective would have demonstrated a lower EI (though still substantial) than presented in the present study.
Limitations Quality improvement project team members were generally not financial officers, health economists, or accountants. Thus, their ROI results likely underestimate the actual financial benefits because of difficulty in identifying benefits and converting these to monetary values. We also stressed in all calculations that in order for ROI to be credible for stakeholders, financial estimates should be conservative. In addition, we were unable to estimate the financial benefits of the large number of public health professionals that learned how to apply QI to their work through the QI 101 program, such as the value of future QI projects they could perform and presumably would yield additional financial benefits. Thus, again, our estimates of EI and ROI are likely underestimates. Teams participating in QI 101 had a large amount of expert support, such as assistance in performing their assignments and engagement with their leadership to encourage supporting the time needed for the team to work on their QI project as well as addressing barriers. Thus, these teams’ EI/ROI results may not be generalizable to other QI projects supported with less expertise and/or time and resources that result in lesser project results. Reporting negative ROIs could be disappointing to teams that worked so diligently with hopes of showing a positive return on their time and efforts. Thus, it is possible that teams with a negative EI/ROI may not have completed/shared their results, causing us to overstate overall results. However, we believe that this
● Conclusions Return on investment and EI analysis is increasingly being applied in public health. There is hope that it will be a viable and effective methodology to evaluate performance of public health QI projects and to justify program continuance during an era of decreasing budgets. In this study, QI teams employed ROI/EI methodology to determine whether the QI project’s benefits outweighed the investment. We have described our ROI approach and how we applied many lessons along the way to improve the method. Overall results to date reflect that ROI analyses can be effectively utilized in public health settings. Our article suggests that with ongoing practice and experience, ROI can become a meaningful, recognized application to assist public health in a new economic era. REFERENCES 1. Randolph GD, Lea CS. Quality improvement in public health: moving from knowing the path to walking the path. J Public Health Manag Pract. 2012;18(1):4-8. 2. Oberlander J. Health care policy in an age of austerity. N Engl J Med. 2011;365(12):1075-1077. 3. Riley WJ, Moran JW, Corso LC, Beitsch LM, Bialek R, Cofsky A. Defining quality improvement in public health. J Public Health Manag Pract. 2010;16(10):5-7. 4. Phillips P, Phillips J. The Basics. In: Return on Investment (ROI) Basics. Alexandria, VA: ASTD; 2005:1.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Lessons Learned From Measuring Return on Investment in Public Health Quality Improvement Initiatives
5. American Public Health Association. National public health week. http://www.nphw.org/. Accessed verified June 23, 2014 6. Cornett A, Thomas M, Davis MV, et al. Early evaluation results from a statewide quality improvement training program for local public health departments in North Carolina. J Public Health Manag Pract. 2012;18(1): 43-51 7. Davis M, Cornett A, Mahanna E, See C., Randolph G. Advancing quality improvement in public health departments through a statewide training program. J Public Health Manag Practice. In press. DOI: 10.1097/PHH.0000000000000165. 8. Martin K, Osterling M. The Kaizen Event Planner: Achieving Rapid Improvement in Office, Service, and Technical Environments. Boca Raton, FL: Productivity Press; 2007. 9. Zaza, S., Briss, P. A., Harris, K. W., eds., Resources. In: The Guide to Community Preventive Services: What Works to Promote Health? Oxford, UK: Oxford University Press; 2012:477. 10. Zbikowski SM. North Carolina Tobacco Use Quitline Final Comprehensive Evaluation Reports. Seattle, WA: Free & Clear, Inc. & Alere Wellbeing Inc; 2006-2011. 11. Centers for Disease Control and Prevention. Smokingattributable mortality, years of potential life lost, and productivity losses: United States, 2000-2004. Morb Mortal Wkly Rep. 2008;57(45):1226-1228. 12. QuitlineNC.com. http://www.quitlinenc.com/home. Updated 2014. 13. Moran J, Riley B. Calculating the real value of process improvement: Factoring in intangible benefits. Process Excell Netw. 2012. http://www.processexcellencenetwork.com/l.
❘ E37
14. Phillips J, Phillips P. Measuring. In: ROI at Work: Best-Practice Case Studies From the Real World. Alexandria, VA: ASTD; 2005:7-10. 15. QI toolbox—measures. Center for Public Health Quality Web site. http://centerforpublichealthquality.org/index.php/ theqitoolbox/qi-project-profile-tool/workplace-organization/ 94-step-by-step-guide/169-measures. Updated 2014. Accessed December 18, 2014. 16. Neumann PJ, Jacobson PD, Palmer JA. Measuring the value of public health systems: the disconnect between health economists and public health practitioners. Am J Public Health. 2008;98(12):2173-2180. 17. Ensign K. Estimating return on investment for public health improvements: tutorial on using the new tool. http:// www.astho.org/Accreditation-and-Performance/Estimating -Return-on-Investment-for-Public-Health-Improvements/. Published 2013. Accessed June 23, 2014. 18. Center for Health Care Strategies, Inc. ROI forecasting calculator for quality initiatives. http://www.chcsroi.org/ Welcome.aspx. Published 2007. Accessed June 23, 2014. 19. Public Health Institute. New web-based ROI tool demonstrates return on investment of remote patient monitoring programs. http://www.phi.org/news-events/487/newweb-based-roi-tool-demonstrates-return-on-investment-ofremote-patient-monitoring-programs. Published 2013. Accessed June 23, 2014. 20. Institute for Healthcare Improvement. Adverse events prevented calculator. http://www.ihi.org/resources/Pages/ Tools/AdverseEventsPreventedCalculator.aspx. Accessed June 23, 2014.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
