Beyond Cost Benefit: An Assessment Approach for the '90s Erica L. Drazen Arthur D. Little, Inc. Cambridge, MA 02140 Abstract

eliminated. The approaches used to obtain a CON are focused on cost justification only--not total benefits. Most CONs only predicted benefits and never followed through to see if the expected benefits were realized. During this era, most information systems were being purchased to replace clerical tasks, and simple approaches which looked at the trade-off between personnel and computers were adequate. These approaches are not useful when examining today's systems which are focused on improving the clinical process and outcome of care delivery.

A new evaluation approach is needed to evaluate clinical and management applications of HI.S.-where the major benefits may not be related to labor savings. New evaluation approaches also need to reflect a "bottom line" business orientation. We will describe an evaluation approach which is based on TQM concepts and meets both these criteria and incorporates benefits realization into the evaluation process. The Need for a Change

An Approach for the '90s

The evaluation of hospital information systems is undergoing a revival. However, the objectives of current evaluations are different. Evaluations are no longer academic exercises or intended to meet some administrative requirement; they are undertaken to really understand whether or not a new information system will produce positive "bottom line" results for the institution. Upcoming limitations on the pass-through of capital costs for Medicare reimbursement will almost surely accelerate the evaluation trend. Although the interest in evaluation is renewed, the methods for performing cost and benefit analysis are not--and they are out of alignment with the nature of systems being purchased and the objectives of current assessments. It is time to reexamine our approaches to evaluating healthcare technology; innovate to create studies that provide real value; and then promote evaluation as a standard part of the process of purchasing information systems in healthcare.

At least four major enhancements in methodology will be critical in developing an evaluation approach for the next decade: * Driving to achieve benefits as the primary

evaluation goal. * Focusing on critical issues and using standard tools to achieve efficiencies.

* Maintaining independence.

* Fitting with the management philosophy of the institution to minimize the "extra effort" devoted to evaluation.

Driving to Achieve Benefits During the '80s there was growing recognition that when an information system--especially a clinicallyoriented system--was installed, benefits did not automatically follow. The concept of benefits realization was introduced. Now most evaluations involve a two-step effort: first looking to see if expected benefits are realized and if not, initiating a benefits realization process. These need to be integrated into one process where the initial focus is on benefits realization and then on documenting benefits achieved.

The approaches that we are using in most evaluations of hospital information systems today were developed in the '60s and '70s. During that era there were two main types of evaluations: large-scale technology assessments funded by the federal government and smaller studies funded by potential users and designed to support a certificate of need (CON) process. Although some of the techniques from the technology assessment era remain, the entire framework is impossible to replicate now that large federal programs have been

0195-4210/91/$5.00 ©) 1992 AMIA, Inc.

13

Benefits involve more than direct cost savings. They include improvement in level of service (creating more satisfied customers)--which may lead to improved future revenue, and improvements in the outcome of care--even at added expense.

vendor/user studies, with some outside oversight, will help minimize bias.

Fitting with the Institutional Philosophy

Institutions are being forced to move closer to a management philosophy that is very compatible with evaluation. With payers demanding a "pay for performance" standard, and requesting documentation that high levels of performance are achieved, evaluation measures will become more available as a result of the standard care delivery process. The growing interest in total quality management also creates a useful context for the evaluation of information systems. An information system is a process improvement...a solution to a problem with information processes. Viewed in this light, evaluation of information systems should be a natural part of any TQM effort. As a first step to improving the evaluation of information systems, we have experimented with using a TQM framework for evaluation.

As a first step, it is important to ensure that all the important benefits of the system are being achieved and then document those benefits. Better ways of measuring overall success are also needed. Clinical researchers have moved from measuring lives saved--to life years saved--to quality-adjusted life years saved for evaluating clinical treatments. HIS researchers need to be thinking about ways to measure quality-adjusted days of care or qualityadjusted outpatient visits to use in evaluating information systems.

Focusing on Important Benefits

When thinking about how to evaluate a system, it is easy to fall into the trap of first considering what you could easily measure and then to pick from those measures ones that reflect important benefits. Methodologies will advance much more quickly if this thinking is reversed, and evaluations strive to develop measures for all the important items.

A TQM Approach to Benefits Assessment

One of the core goals of any quality effort is meeting customer expectations. A quality improvement process focuses on areas where the hospital can close gaps between "customer" expectations and performance. (In this case the "customers" include patients, payors, administrators, nurses, physicians, etc.) Evaluators naturally focus on issues of value to the institution by looking to fill gaps in expectations from information systems.

Although the important benefits will vary from user to user, the types of benefits will probably overlap considerably. Mechanisms for sharing tools are needed so that each institution does not need to invent a new nurse staff satisfaction survey, or an approach to quantifying unnecessary delays in care delivery. As tools become more standardized, the likelihood that these will be built into systems by HIS vendors will increase...thus achieving even more efficiency in data collection.

TQM incorporates the concept of continuous improvement which requires that measurements be made to confirm problems and monitor progress. If evaluators use a TQM approach to design and conduct their study, they will identify ways in which the HIS can document processes, confirm issues, and track progress as well as achieve benefits. Institutions which operate in this mode are most likely to gain full value from their systems. Continuous improvement is a core concept of TQM. By introducing the concepts of TQM, the hospital will be left with tools for identifying and documenting further improvements due to the system. TQM also requires that the root causes of problems be identified. By identifying and addressing several root causes of a problem, the benefits received will increase; and there will be less need to "blame" another problem when expected benefits are not achieved.

Maintaining Independence The era of govemment-funded studies is probably over. Individual institutions and vendors are the most likely funders of future evaluations. One concern about these self-funded efforts is the bias introduced by the evaluators. Institutions have a natural desire to prove that they made the correct decision, and vendors want positive results to use in marketing. This undoubtedly has an influence on the design of the evaluation and on the interpretation of the results. It certainly has an effect on the subset of results that is disseminated outside the institution. Jointly-sponsored

14

It is difficult to convey a concept as abstract as a new evaluation methodology on paper. Therefore, we will illustrate by example how an evaluation using TQM would proceed and then discuss the differences between this and traditional approaches.

achieving improvement, and the strategic importance of the improvement. For purposes of illustration, we will assume that we have identified the process used to document care as one highpriority process where improvements are needed.

An Example TQM/Benefits Assessment Approach

Understanding Priority Processes: The next steps in the TQM process are designed to ensure that there is a common and complete understanding of the processes to be "fixed." This is done by developing a flowchart of the current process, identifying groups or individuals who supply information or resources to the process, and those that receive outputs from the process.

The overall approach to a TQM benefits assessment is illustrated in Figure 1.

Figure 1 - TQM Approach to Benefits Assessment Post npbenmfttion Date HIS Gerwratd Mtlon

HIS & Proc

knprovwmts

4

`

Identify areas for

improvement

Track progress

key i~~~~dentty customersY

Developrimpornent Devebpimplement

solutions

r

I Confirm

Baselkh Data

Determine

ncustomer"

root causes

requirements

collecion

Identiy gaps/

Find root causes of

benefit opportunities

problems

N\

. Understand priority process

Identifying Improvement Opportunities

For each group which uses outputs of the process, we consider what they expect from the process and what they currently get from the process in order to identify gaps. If major gaps are identified in the process, then a redesign effort may be required. Extemal benchmarking should be used to evaluate alternative processes. If only minor gaps are identified, modifications to existing processes probably will produce the desired results. We have assumed that the existing process can be improved. For purposes of illustration, we will assume that the steps involving nurses recording information in the patient chart and flowsheet have been identified as problems, and the "gap" involves the amount of time nurses spend in this part of the documentation process.

The first step in the benefits assessment process involves identifying information processes in the hospital that need improvement. From a "bottom line" perspective, priority information processes are those that produce outcomes that produce customer satisfaction and where your current performance does not meet customer expectations or match competitors' performance levels. These customer requirements, gaps, and process improvement opportunities can be generated in brainstorming sessions--however, a longer term goal should be to generate objective data to guide future efforts. If a large number of processes are identified, they can be set in priority by considering their importance to multiple stakeholders, the difficulty in

15

Finding the Root Causes of Problems: Next we need to be sure that we understand the root causes of the problem. In this example, assume the problem is: too much nursing time is spent in documenting care. We will find root causes by identifying the obvious causes of the problem and then asking, "Why is that?" until we have reached the root cause. We will record our fmdings on a fishbone diagram. An example of a fishbone diagram of causes of High Percentage of Nursing Time in Documentation is included as Figure 2.

causes for the problems identified. If we want to have maximum impact of time spent on documentation, we would want to look at the potential for the HIS to affect all root causes. For example, JCAHO requirements are a root cause of documentation time. We need to consider if there are any pending changes in JCAHO requirements that need to be incorporated into the system design. Another root cause of the documentation problem is that we do not know enough to adjust documentation requirements to each patient. The system could provide the tool for both collecting the information needed and implementing variable documentation based on patient need.

Contrasts with a Traditional Benefits Assessment Before describing the next steps in the process, I will contrast traditional versus the TQM approach to

Figure 2 - Fishbone Diagram High Percentage of Nursing Time in Documentation

High % of time in documentation

The third difference in taking the TQM approach is that all participants will have either learned or gained additional practice in some core TQM tools as a byproduct of the initial evaluation planning.

structuring a benefits assessment. One of the most obvious differences is that two steps--analysis of customer requirements and analysis of existing processes--are added to the evaluation. These steps can be implemented quickly as long as the participants have a good understanding of the processes in the hospital. Another contrast in the TQM approach is that we have identified many root

Conrirmation of Root Causes: The next step of a TQM process is to confirm the root causes of a problem. This typically involves collecting additional opinions and assembling data about the current situation. The data collection at this stage

16

exactly parallels the baseline data collection of a traditional evaluation. Therefore, this step is no different in content from a traditional evaluation

new ways to evaluate systems and to improve processes.

process.

The time required to identify and understand processes that need improvement and to postulate root causes took less time than we expected. Two half-day sessions per department were sufficient. The staff found it easy to learn and use the basic tools to identify gaps in meeting customer needs, in developing flowcharts, and finding root causes of problems. Many recognized that these techniques would be valuable to use in addressing other issues within the department. By involving staff who will be involved in systems use and benefits realization, we also achieved early buy-in. They are all now thinking of how processes will change and how benefits can be achieved.

Identirication of Improvement Actions: Once we have confirmed that the right problems are being addressed and baseline operations have been documented, the next step is to implement the improvements. One possible intervention, clearly, is the implementation of the HIS. Additional actions will address other root causes including: changing manual procedures, and making enhancements to existing systems. Many of these would have been part of an eventual benefits realization process. Again, TQM steps are no different in content from a traditional evaluation process. They do differ in that the actions are more comprehensive: addressing all root causes of a problem. They are also implemented sooner: benefits realization accompanies rather than follows system implementation.

The mix of staff involved in identifying benefit opportunities is critical. All major "customers" and "suppliers" of information need to be involved. Our initial sessions did not include enough administrative staff to fairly represent their views and their priorities. A corollary problem is that the time commitment--while quite small in terms of the overall HIS project--still is enough to disrupt work flow in the department. Administrative staff are especially difficult to get to commit to two half-day work sessions. It is ironic that more administrative time will probably be devoted to selecting the "right" system than to planning for, realizing, and documenting benefits. This is an area where collaboration is needed among vendors, consultants, and academicians to educate hospitals on where they can invest time to increase the value of their information systems investments.

Tracking Progress: A TQM program requires that the results of interventions (improvements) are documented. This step parallels (in content and timing) the post-implementation data collection process of a traditional evaluation. Because of the thinking that leads up to this step, postimplementation data collection will probably make more use of the system to automatically collect data and also will probably be used as a model for ongoing data collection to support the continuous improvement efforts. Continuous Improvement: The final step of a TQM program is to start all over in examining existing operations (which now include the computer system) and identifying areas for improvement. The post-implementation data collection provides the initial basis for this assessment. The difference is that benefits realization has already begun and that the hospital has a process and tools in place for carrying on the continuous improvement process.

The TQM approach also generated a long list of processes that could be improved. Even after screening for improvements that were significant and tied to the information systems, the extensive list of potential benefits results in a much longer process to set priorities for benefits realization and benefits documentation. Conclusions

Results

Despite some initial difficulties, we are convinced that the TQM approach to benefits will be successful in melding assessment and benefits realization. This result will be greater achievement of benefits, lower cost of assessment, and a process for continual improvement. We encourage others to try this new approach and report on their results.

We have completed Beta testing of the TQM approach to benefits assessment in Planning an HIS evaluation. Our experience indicates that the concept of using TQM as an approach to benefits assessment is appealing; hospitals are looking for

17

Beyond cost benefit: an assessment approach for the '90s.

A new evaluation approach is needed to evaluate clinical and management applications of H.I.S.-- where the major benefits may not be related to labor ...
703KB Sizes 0 Downloads 0 Views