Technology and Health Care 22 (2014) 729–739 DOI 10.3233/THC-140847 IOS Press
729
Development of sustainable models for technology evaluation in hospital Roberto Miniati∗, Francesco Frosini, Giulio Cecconi, Fabrizio Dori and G. Biffi Gentili Department of Information Engineering, Biomedical Laboratory, University of Florence, Florence, Italy Received 11 June 2014 Accepted 9 July 2014 Abstract. BACKGROUND: This paper reports the development of standard techniques for technology evaluation in hospital carried out at the Florence Teaching Hospital Careggi (AOUC), where, as a complex system, the technological evaluation is a strategic and essential element for the maintenance of high-quality clinical activity and maximization of available resources. OBJECTIVE: The aim of this paper has been the development of a system of economically sustainable models for the implementation of HTA and HS analyses in the hospital environment as well as presenting, in addition to a valid scientific resilience, the methodological and temporary flexibility to satisfy needs of hospital decision-makers. METHODS: The evaluation models call for 3 main phases: an initial analysis of the in-hospital request, a collection of data, and finally a draft of a specific, easily usable set of reports. RESULTS: Three standardized and tested models of evaluation were developed, which, in relation to the objective of the request and schedule of the assignment, provide for the production of a speedy report (1-week), an intermediate report (1month), or a extensive report typical of classical studies of hospital based HTA (1-year). It is then related to the evaluation model of the IORT (Intra-Operative Radiation Therapy) technology. DISCUSSIONS AND CONCLUSION: The developed models have permitted the construction, using personnel and laboratories within the hospital, of an evaluation system reliable and responsive to the HOSPITAL’s temporary needs based on the HS and HTA analyses in the hospital environment. Regarding the applicable case of IORT, this has shown how in-hospital requests have been satisfied in the preset time: although it establishes expected improvements on the social effect and weight of the illness and reveals a high territorial strategic relevance, the introduction of IORT in the hospital presents some criticalities on the impact on the healthcare organization and the necessity of specific training of medical technologist personnel. Keywords: Horizon scanning, health technology assessment, hospital based, IORT
1. Introduction Within a hospital complex framework, the technological evaluation is a strategic and essential element for the maintenance of high-quality clinical activity and maximization of available resources. Effectively, multidisciplinary evaluations of technology are ever more necessary for managing technological innovations, taking into consideration the impact on the hospital, the benefits, and the risks, as well as tending to the usual investments (and disinvestments). ∗ Corresponding author: Roberto Miniati, Department of Information Engineering, Biomedical Laboratory, University of Florence, Via S. Marta 3, 50139 Florence, Italy. E-mail: [email protected].
c 2014 – IOS Press and the authors. All rights reserved 0928-7329/14/$27.50
730
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
Fig. 1. General framework for the evaluation of HTA in the hospital environment. (Colours are visible in the online version of the article; http://dx.doi.org/10.3233/THC-140847)
For this reason, the typical approaches and methods of HTA (Health Technology Assessment) and HS (Horizon Scanning) supply the most suitable theoretical instruments to healthcare structures to respond to these needs. Therefore, the base for the development of operative instruments is formed, which can, through the drafting of specific reports, adjust and respect the ordinarily very limited business schedules [1–3]. The aim of this paper has been the development of a general framework, sustainable in terms of expenses, for the implementation of HTA and HS analyses in the hospital environment, and that, in addition to having a high scientific strength [4], it would have the necessary methodological flexibility to precisely satisfy the requests of professionals and modulate together with the necessary schedule of the hospital. The evaluation report, designed to be concise and highly usable by the general management, varies in relation to the typology of requested analyses and the time necessary for the study: speedy (1week report), intermediate (1-month report), and extensive (1-year report). Finally, as an example, the application of the model for hospital evaluation of the IORT technology is reported. 2. Methods The technological evaluation model in the hospital environment has been designed and implemented within the Joined Laboratory “Technological Observatory of Decision-Making Support for Healthcare Structures), composed of the Department of Information Engineering (DINFO) of the University of Florence’s School of Engineering and under the supervision of the Careggi Teaching Hospital of Florence (AOUC). As reported in Fig. 1, the conceptual framework considers 3 main phases: (1) analyses of the in-hospital request (2) data collection and (3) draft of report. 2.1. In-hospital request Through analyses of the in-hospital request, the objectives of the evaluation are defined: the typology and the schedule of the study (1-week, 1-month or 1-year), the composition of the evaluation team and the analyses indicators. If the evaluation request is ineligible for the specific HTA/HS activity, a consulting and/or support activity is carried out that goes beyond the HTA/HS approach (e.g., market analyses or evaluations for technical exclusivity).
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
731
As previously outlined, in case of a request coherent with the activity of HTA, the content of the request is analyzed first (e.g., comparative analyses among several technologies, evaluations of costefficiency, and/or classical evaluations for hospital based HS and HTA) and the maximum schedule of the study (1-week, 1-month, or 1-year). Consequently, the multidisciplinary team of in-hospital evaluation and the indicators to be calculated become defined. The selection of indicators represents a critical point for each evaluation system and, in the developed model, depend on the typology of the requested evaluation: – 1-year report In the case of an extensive study, the indicators to consider are those found in the core model of the European Network for Health Technology Assessment – EUnetHTA [5]. The model demonstrates the importance of HTA for the European Union, as to have the EU regulate the evaluations and integrate HTA among the European political priorities through the launch of a European Network for the evaluation of healthcare technologies, which coordinates the efforts of 35 European organizations. Such an international network also has the task of spreading a common methodology for HTA studies in such a way as to obtain a homogeneity in the carried out reports. The uniformity of the studies allows national agencies to know all analyzed aspects and the followed procedure for the completion of the carried out reports and other entities. The Core Model collects all of the elements to evaluate the aspects that influence the entire healthcare process a subdivides them into ten settings: health issue and current use of technology, technical description and characteristics of the technology, security, efficacy, accuracy, economic cost and evaluation, ethical aspects, organizational aspects, social aspects and legal aspects. The engagement of hospital users is considered through direct interviews (health clinicians and medical technologists, general management, finance management, health technology department and pharmacy department), as well as scientifically evident research present in literature. Finally, all the indicators are aggregated to calculate 5 macro-indices of convenience (Clinical ‘Ic’, Economical ‘Ie’, Social ‘Is’, Technological ‘It’ and Security and Processing ‘Isp’) through the formula shown below, in the calculation of the clinical index: Σci × pi IC = (1) Σpi Where “p” represents the priority of the specific sub-indicator “c.” The definition of the weight of “p” agrees with the importance of the sub-indicators and has been determined through the interviews with hospital experts. – 1-month report This represents the intermediate type of evaluation. Through bibliographic research, the indicators mainly used at an international level in the HS models were selected [6–11]. As reported in Table 1, they have been classified by dimensions (clinical, economical, socio-ethical, technological and organizational), marked “Y” if expert employee (that is they need an evaluation from a panel of experts in the sector) or otherwise “N” (usually the more objective indicators pertaining to technological and economic spheres). Finally, the selected indicators were prioritized through a phase of weighing carried out by the expert in-hospital personnel. – 1-week report Regarding the speedy model, the main theoretic reference is the Hospital Clinic of Barcelona model reported in scientific literature [9]. This has been preferred to the previous systems, more precise and detailed theoretic basis, including the EUnetHTA core model, inasmuch the model [9] better
732
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
Table 1 Selected evaluations indicators for intermediate HS evaluation Dimension Indicator Panel Definition (Y/N) Efficacy (4) Safety Y Patient potential undesirable effects with the new HT. Clinical benefit Y Efficacy/effectiveness of the new HT, based on scientific evidence, compared with current available HTs Patient Impact Y Ethical, psychological and quality of life issues associated with the new HT Quality of Evidence Y Quality of available scientific evidence of the new HT, using available quality scales (e.g. SIGN, GRADE; Oxford EBM). Cost (4) Incremental Cost N Increase in the cost-per-case when using the new HT, compared with cost-per-case with available alternative Net Cost N Difference between costs-per-process with the new HT and current level of funding from payers for the available alternative. Investment Effort N Impact of the acquisition of the new HT on overall hospital’s/department annual budget for HTs investment Economic impact N The direct health care costs (annual, lifetime) of the technology Burden of disease (2) Burden of disease 1 Y The prevalence, incidence, disease-adjusted life expectancy, Burden of disease 2 Y healthy years of life expectancy, or other relevant measurement of disease burden of the population with the clinical condition(s) or those that will be affected by the technology The economic burden of the clinical condition(s) in the population Hospital impact (5) Staff Requirements Y Hospital staff requirements (training, expertise. . .) for the new HT Physical space Impact N Space requirements for the new HT to be operational Process of care impact Y Impact of the new HT on the organization of healthcare services and/or work dynamics and flows within the hospital Variation in the rates of Y The coefficient of variation (standard deviation divided by the use mean) Alternatives Y The alternatives, if any, currently or soon available for the conditions that this technology treats Socio-politic (1) Ethical, legal and social Y The probability that an assessment comparing two or more techaspects nologies will help to inform important ethical, legal, or social issues Technology (2) Innovativeness N Level of novelty of the new HT
adapts to the Horizon Scanning-type evaluations that have a comparative approach and is based on a mainly qualitative (or semi-qualitative) character as defined by the cost/benefit index.” The selected indicators are shown in Table 2 and are subdivided in two macro categories: risk and benefit. Both categories are evaluated aggregating the indicators through the same formula (1) and weighing the priorities of the sub-indicators “c1” according to the reported weights in [9]. 3. Data collection Calculating in the indicators, the necessary data collection depends, as in the case of the indicators themselves, on the typology of the evaluation: extensive, intermediate or speedy. In the case of the extensive evaluation (1-year report), the data is collected through the research of national and international scientific evidence (carried out with typical epidemiological techniques) as well as through the consultation of data and in-hospital experts. Also regarding the intermediate evaluation (1-month report), this considers the use of scientific evidence and the consultancy of data and in-hospital experts. The difference from the previous case is that
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
733
Table 2 Selected evaluations indicators for speedy HS [9] CRITICALITY (C) Requests of personnel Structural impact Impact of the process Cost of operation Economical analysis Investment commitment BENEFIT (B) Safety Clinical efficacy Impact on the patient Cost/efficacy Reliability facts Innovativeness
DATA SOURCE IN-HOSPITAL DATA AND SCIENTIFIC EVIDENCE
DATA SOURCE SCIENTIFIC EVIDENCE
Table 3 Selected databases for the research of standardized evidence DATABASE LINK CENTRE FOR REVIEWS AND DISSEMINATION (UK) http://www.york.ac.uk/inst/crd/ CADTH (CANADA) http://www.cadth.ca/ HORIZON SCANNING (AU, NZ) http://www.horizonscanning.gov.au/ EUROSCAN (EU) http://euroscan.org.uk/ NATIONAL INSTITUTE FOR HEALTH AND CLINICAL EXCELLENCE http://www.nice.org.uk/ NIHR HTA (UK) http://www.hta.ac.uk/ NSC (UK) http://www.hsc.nihr.ac.uk/ CALIFORNIA TECHNOLOGY ASSESSMENT FORUM (US) http://ctaf.org/ INTERNATIONAL NETWORK OF AGENCIES OF HTA http://www.inahta.net/ COCHRANE SUMMARIES (UK) http://summaries.cochrane.org/ BANCA DATI VIHTA (ITA) http://www.ijph.it
the intermediate evaluation observes the research of the already compiled HS/HTA reports in the international and national reviews instead of launching a HS/HTA study from scratch. The primary objective is the contextualization, precisely in the hospital environment, of analyses that possibly already exist. For this reason the research of evidence becomes fundamental. In order to reduce possible system errors due to the subjectivity of the research to a minimum, it has been decided to select the strict set of electronic resources reported in Table 3, as representatives of the major DB, national and international, used in HS and HTA analyses [7–13]. It is necessary to remember, as in the intermediate evaluation, the scientific evidence is an informative support for the various in-hospital experts and not the direct way to calculate the clinical indicators. A detail of the collected data sheet sent to the expert in-hospital users of the specific technology is reported in Fig. 2. The found scientific evidence is attached to this sheet to be filled out through the use of the Lykert scale of 9 levels (completely disagree – completely agree). Lastly, for the speedy evaluation (1-week report), the significant difference from the other evaluations lies in the fact the calculation of the indicators belonging to the clinical benefits (see Table 2) primarily happens with the sole research of scientific evidence (reported DB in Table 3) and without the ordinary involvement of several in-hospital clinician experts. To calculate the indicators pertaining to the “criticality” area, in addition to the evidence in literature, specific in-hospital data is found (from informative applications or specific, direct advice with hospital experts).
734
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
Fig. 2. Detail of the model of collected data for the clinical indicators.
4. Report The objective of the report is to supply a clear tool to the decision-maker, simple to use and so complete that it can hold up in in-hospital decisions. These are the points that have represented the bases for the development of the 3 reports: 1-week, 1-month and 1-year reports. For the 1-year report, the chosen manner is the one connected to the graphic “radar” representation of the macro-indicators. In Fig. 3 the main page of the report is presented, subdivided into a “reliability” section, which evaluates the number of calculated indicators out of the totals in general and by setting; and a “results” section where the index of convenience is calculated as the total area of the pentagon identified by the macro-indices for dimensional setting. This is to have both a framework of general comparative evaluation and one for a specific setting. The example in Fig. 3 is relevant to the comparative evaluation of 3 technologies: tech1, tech2 and tech3. The intermediate evaluation report is subdivided into two parts: the clinical evaluation and the technical evaluation. For the technical one, there is a sheet that reports the data related to the economic and technological aspects (innovation, obsolescence, alternative existence, weight on the budget, Breaking Even Point), while for the clinical part, the choice is addressed towards a graphic representation (a three-level code with “red”, “yellow” and “green”) subdivided by user. In Fig. 4, the involvement of anesthetist, a surgeon, a medical physicist and a generic “other” is shown. Furthermore, as reported in the figure, it is possible to have an average for the specific area of evaluation (clinical efficacy, weight of the illness, impact on the hospital and socio-political aspect) as well as for the entire clinical scope (general average for the clinical area). Finally, it is noted that as for every evaluated index, it is possible to see its priority (weighted column), the improvement or deterioration expected from the introduction of the new technology with respect to the traditional method in use: improvement = green; deterioration = red; stable situation = yellow. The speedy evaluation (1-week report) improves the original point of reference [9] by improving the usability and completing it mainly with a series of attachments which better specify the bases of the con-
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
735
Fig. 3. 1-year report for extensive evaluation. (Colours are visible in the online version of the article; http://dx.doi.org/10.3233/ THC-140847)
ducted analyses (economic analyses, considered evidence, main motivations, etc.). The report is made up, firstly, of a part related to the description of the request and the technology under consideration. The second part contains, in addition to the considered scientific evidence and sequence used for the research of the evidence, the list of the detailed attachments and a section concerning the results of the comparatively calculated indicators based on determined decisions (affirmative, negative, same convenience or not calculated) that aim to answer the question: “With respect to traditional technologies, will the evaluated one prove more suitable?” Finally, the third section, reported in Fig. 5, shows the graphic representation of the results of the HS evaluation in terms of criticality/benefit. This permits an arrangement of more technologies within the graph, making it a multi-technological, comparative evaluation. Finally Fig. 6 reports the specific part of the economic evaluation through the calculations of two indices: the “Break Even Point” (BEP) defined by the difference Expenses-Earnings in relation to the number of operations; and the “Net Present Value” (NPV) capable of identifying the time necessary to
736
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
weight
index
surgery
anesthesia
other
medical physics
total
1 1 0,9
medium
low
medium
medium
low
low
medium
medium
low
low
medium
medium
impact on the patient
high
low
medium
medium
QALY/DALY
high
low
medium
medium
socio-economic weight
high
low
medium
medium
security
medium 0,8 0,7 1
Burden of Disease 1
medium 0,9
medium 1
Impact on the Hospital curing process
low
low
medium
medium
0,9
1
frequency of use
high
low
medium
medium
0,8
healthcare personnel
low
low
medium
medium
0,7
technical personnel
low
low
medium
medium
0,6
technological alternatives
high
low
medium
medium
ethical and legal elements
high
1
high
Socio-Political Aspects 1
low
medium
medium
medium AVERAGE FOR SPECIFIC AREA
GENERAL AVERAGE
Fig. 4. 1-month report for intermediate evaluation. (Colours are visible in the online version of the article; http://dx.doi.org/ 10.3233/THC-140847)
Fig. 5. 1-week report for speedy evaluation. (Colours are visible in the online version of the article; http://dx.doi.org/10.3233/ THC-140847)
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital Full Service
13.000
Full Risk
Full Service + Compartment Costs
737
Full Risk + Compartment Costs
Management Costs
9.800
6.600
3.400 BEP FR + CC
200 BEP FS BEP FS + CC -
3.000
5
10
15
BEP FR 20
40
Number of Surgical Interventions
Fig. 6. Economic evaluation of IORT in the hospital environment. (Colours are visible in the online version of the article; http://dx.doi.org/10.3233/THC-140847)
recover, if possible, the effectuated investment while considering the average number of yearly operations as an independent variable.
5. Results In the laboratory’s first year of activity (September 2012 – September 2013) two 1-year reports, one 1-month report and sixteen 1-week reports were developed. As an applicable example, the carried out practice on the evaluation of the addition of the IORT (Intra-Operative Radiation Therapy) technology within the hospital will be described. Following the previously mentioned methodology, the first phase reviewed the analyses of the inhospital request, which can be summarized as follows: 1. Evaluation of the use of IORT in the clinic with respect to the traditional method. 2. Evaluation of a possible, inexpensive offer to buy used equipment. 3. Schedule of an intermediate evaluation (1-month report). The second phase included the research of scientific evidence and its distribution, together with the clinical Lykert questionnaire, to the in-hospital users that have been selected from the following professional fields: general surgery and oncology, anesthesia, medical physics and radiotherapy. The response deadline given to the hospital users was two weeks. In Fig. 4, the clinical evaluation on the part of the personnel concerning oncological surgery, medical and health physics are reported. Regarding the calculation of the technical-economic indicators, it was possible to calculate the graph shown in Fig. 6. This represents the gross spending limit evaluated as “Expenses-Earnings” in relation to the number of yearly operations and the typology of the signed maintenance contract (yearly cost): full service or full risk.
738
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
Considering the scenarios of use, in relation to the expected activity and the optional maintenance contract, it is possible to identify a figure to spend on the item calculating the yearly limit for the age of the equipment. Given that the graph is calculated as “Expenses-Earnings”, a positive limit represents an annual loss while a negative one represents a yearly gain. With the term BEP (Breaking Even Point) the points of balance between the cost of technological maintenance and the DRG earnings (for the typology of the contract and considering the fixed amortized cost of the operating theatre) is identified.
6. Discussion and conclusions The models presented here have permitted the construction, using personnel and laboratories within the hospital, of a system of models that permit a response in appropriate time to the needs of the HS and HTA evaluations in the hospital environment. Furthermore, the standardization of the schedules, reports and methodologies permits the hospital management to plan the number of requests and increase the in-hospital evaluation team through a training itinerary. The evaluative activity has had a strong increase, especially for the speedy evaluations (1-week report), in the second semester of activity, after its centralization in the first semester had been for the design and development of the models themselves. Regarding the applicable case of the IORT, the requests have been completely satisfied. The evaluation had a high and punctual participation by the in-hospital users involved in the analyses (6/7), in addition to the development of a supporting report that albeit producing specific information, gives the in-hospital decision-maker a “capacity for movement and planning” for its own strategic analyses. In fact, the example supplied by the estimated cost of purchase of used equipment, shows how the report, though supplying quantitatively clear elements, allows the calculation of the precise figure for the in-hospital planning and expected and/or predicted activity (number of yearly operations). The health-based clinical evaluation of the IORT shows a clinical efficacy comparable to traditional techniques, identifies expected improvements for the impact, the weight of the illness and the social aspect as well as considering it, in the specific analyzed context, a strategic element for the territory. Finally, the criticalities of the technology are the negative impact of its installation on the in-hospital organization, for the necessity of specific training of the medical technologist personnel as well as the alterations of the itineraries and process of hospital care.
References [1] [2] [3] [4] [5]
Ehlers L., Vestergaard M., Kidholm K., Bonnevie B., Pedersen P.H., Jorgensen T., Jensen M.F., Kristensen F.B., Kjolby M. Doing mini-health technology assessment in hospitals: A new concept of decision support in healthcare? Int J Technol Assess Health Care. 2006; 22(3): 295-301. Jacob R., McGregor M. Assessing the impact of health technology assessment. Int J Technol Assess Health Care. 1997; 13: 68-80. Douw, K. and H. Vondelin. Selection of new health technologies for assessment aimed at informing decision making: A Survey among horizon scanning systems. International Journal of Technology Assessment in Health Care, 2006. 22(2): 177-183. Kidholm K., Ehlers L., Korsbek L., Kjaerby R., Beck M. Assessment of the quality of mini-HTA. International Journal of Technology Assessment in Health Care, 25: 1 (2009), 42-48. Eunehta project, joint Action on HTA (2010–2012) funded by the European Union, in the framework of the Health Programme. HTA Core Model Online. https://meka.thl.fi/htacore. National Institute for Health and Welfare.
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital [6]
[7] [8] [9] [10] [11] [12] [13]
739
Simpson S., Hiller J., Gutierrez-Ibarluzea I., Kearney B., Norderhaug I., Fay AF., Packer C., Asua J., Benguria G., Blanchard S., Blozik E., Bonnevie BM., Clifford T., Eckerlund I., Galnares L., Groeneveld K., Hae Lee Robin S., Hakak N., Husereau D., Ibargoyen N., Kaila M., KuâTłzli ¸ C., Llanos A., Luengo S., Morrison A., Mundy L., Tal O., Wallgren L., Wallin J. A toolkit for the identification and assessment of new and emerging health technologies. 2009. EuroScan. Birmingham. Effie Amanatidou, Maurits Butter, Vicente Carabias, Totti Könnölä, Miriam Leis, Ozcan Saritas, Petra Schaper-Rinkel and Victor van Rij. On concepts and methods in horizon scanning: Lessons from initiating policy dialogues on emerging issues. Science and Public Policy (2012) 39 (2): 208-221. Hussein Z. Noorani, Donald R. Husereau, Rhonda Boudreau and Becky Skidmore. Priority setting for health technology assessments: A systematic review of current practical approaches. International Journal of Technology Assessment in Health Care, 23: 03(2007), 310-315. Laura Sampietro-Colom, Irene Morilla-Bachs, Santiago Gutierrez-Moreno and Pedro Gallo. Development and test of a decision support tool for hospital health technology assessment. International Journal of Technology Assessment in Health Care, 28: 4(2012), 460-465. Iñaki Gutierrez-Ibarluzea, Sue Simpson and Gaizka Benguria-Arrate. Early Awareness and Alert Systems: An Overview of EuroScan Methods. International Journal of Technology Assessment in Health Care, 28: 03, (2012), 301-307. Guy Paré, Placide Poba-Nzaou and Claude Sicotte. Home Telemonitoring for Chronic Disease Management: An Economic Assessment. International Journal of Technology Assessment in Health Care, 29: 2, 155-161. Leonora Varela-Lema, Jeanette Punal-Riobóo, Beatriz Casal Acción, Alberto Ruano-Ravina and Marisa López García. Making processes reliable: A validated PubMed search strategy for identifying new or emerging technologies. International Journal of Technology Assessment in Health Care, 28: 4 (2012), 452–459. Claire Packer, Matthew Fung and Andrew Stevens. Analyzing 10 years of early awareness and alert activity in the united kingdom journal. International Journal of Technology Assessment in Health Care, 28: 3 (2012), 308–314.
Copyright of Technology & Health Care is the property of IOS Press and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
729
Development of sustainable models for technology evaluation in hospital Roberto Miniati∗, Francesco Frosini, Giulio Cecconi, Fabrizio Dori and G. Biffi Gentili Department of Information Engineering, Biomedical Laboratory, University of Florence, Florence, Italy Received 11 June 2014 Accepted 9 July 2014 Abstract. BACKGROUND: This paper reports the development of standard techniques for technology evaluation in hospital carried out at the Florence Teaching Hospital Careggi (AOUC), where, as a complex system, the technological evaluation is a strategic and essential element for the maintenance of high-quality clinical activity and maximization of available resources. OBJECTIVE: The aim of this paper has been the development of a system of economically sustainable models for the implementation of HTA and HS analyses in the hospital environment as well as presenting, in addition to a valid scientific resilience, the methodological and temporary flexibility to satisfy needs of hospital decision-makers. METHODS: The evaluation models call for 3 main phases: an initial analysis of the in-hospital request, a collection of data, and finally a draft of a specific, easily usable set of reports. RESULTS: Three standardized and tested models of evaluation were developed, which, in relation to the objective of the request and schedule of the assignment, provide for the production of a speedy report (1-week), an intermediate report (1month), or a extensive report typical of classical studies of hospital based HTA (1-year). It is then related to the evaluation model of the IORT (Intra-Operative Radiation Therapy) technology. DISCUSSIONS AND CONCLUSION: The developed models have permitted the construction, using personnel and laboratories within the hospital, of an evaluation system reliable and responsive to the HOSPITAL’s temporary needs based on the HS and HTA analyses in the hospital environment. Regarding the applicable case of IORT, this has shown how in-hospital requests have been satisfied in the preset time: although it establishes expected improvements on the social effect and weight of the illness and reveals a high territorial strategic relevance, the introduction of IORT in the hospital presents some criticalities on the impact on the healthcare organization and the necessity of specific training of medical technologist personnel. Keywords: Horizon scanning, health technology assessment, hospital based, IORT
1. Introduction Within a hospital complex framework, the technological evaluation is a strategic and essential element for the maintenance of high-quality clinical activity and maximization of available resources. Effectively, multidisciplinary evaluations of technology are ever more necessary for managing technological innovations, taking into consideration the impact on the hospital, the benefits, and the risks, as well as tending to the usual investments (and disinvestments). ∗ Corresponding author: Roberto Miniati, Department of Information Engineering, Biomedical Laboratory, University of Florence, Via S. Marta 3, 50139 Florence, Italy. E-mail: [email protected].
c 2014 – IOS Press and the authors. All rights reserved 0928-7329/14/$27.50
730
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
Fig. 1. General framework for the evaluation of HTA in the hospital environment. (Colours are visible in the online version of the article; http://dx.doi.org/10.3233/THC-140847)
For this reason, the typical approaches and methods of HTA (Health Technology Assessment) and HS (Horizon Scanning) supply the most suitable theoretical instruments to healthcare structures to respond to these needs. Therefore, the base for the development of operative instruments is formed, which can, through the drafting of specific reports, adjust and respect the ordinarily very limited business schedules [1–3]. The aim of this paper has been the development of a general framework, sustainable in terms of expenses, for the implementation of HTA and HS analyses in the hospital environment, and that, in addition to having a high scientific strength [4], it would have the necessary methodological flexibility to precisely satisfy the requests of professionals and modulate together with the necessary schedule of the hospital. The evaluation report, designed to be concise and highly usable by the general management, varies in relation to the typology of requested analyses and the time necessary for the study: speedy (1week report), intermediate (1-month report), and extensive (1-year report). Finally, as an example, the application of the model for hospital evaluation of the IORT technology is reported. 2. Methods The technological evaluation model in the hospital environment has been designed and implemented within the Joined Laboratory “Technological Observatory of Decision-Making Support for Healthcare Structures), composed of the Department of Information Engineering (DINFO) of the University of Florence’s School of Engineering and under the supervision of the Careggi Teaching Hospital of Florence (AOUC). As reported in Fig. 1, the conceptual framework considers 3 main phases: (1) analyses of the in-hospital request (2) data collection and (3) draft of report. 2.1. In-hospital request Through analyses of the in-hospital request, the objectives of the evaluation are defined: the typology and the schedule of the study (1-week, 1-month or 1-year), the composition of the evaluation team and the analyses indicators. If the evaluation request is ineligible for the specific HTA/HS activity, a consulting and/or support activity is carried out that goes beyond the HTA/HS approach (e.g., market analyses or evaluations for technical exclusivity).
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
731
As previously outlined, in case of a request coherent with the activity of HTA, the content of the request is analyzed first (e.g., comparative analyses among several technologies, evaluations of costefficiency, and/or classical evaluations for hospital based HS and HTA) and the maximum schedule of the study (1-week, 1-month, or 1-year). Consequently, the multidisciplinary team of in-hospital evaluation and the indicators to be calculated become defined. The selection of indicators represents a critical point for each evaluation system and, in the developed model, depend on the typology of the requested evaluation: – 1-year report In the case of an extensive study, the indicators to consider are those found in the core model of the European Network for Health Technology Assessment – EUnetHTA [5]. The model demonstrates the importance of HTA for the European Union, as to have the EU regulate the evaluations and integrate HTA among the European political priorities through the launch of a European Network for the evaluation of healthcare technologies, which coordinates the efforts of 35 European organizations. Such an international network also has the task of spreading a common methodology for HTA studies in such a way as to obtain a homogeneity in the carried out reports. The uniformity of the studies allows national agencies to know all analyzed aspects and the followed procedure for the completion of the carried out reports and other entities. The Core Model collects all of the elements to evaluate the aspects that influence the entire healthcare process a subdivides them into ten settings: health issue and current use of technology, technical description and characteristics of the technology, security, efficacy, accuracy, economic cost and evaluation, ethical aspects, organizational aspects, social aspects and legal aspects. The engagement of hospital users is considered through direct interviews (health clinicians and medical technologists, general management, finance management, health technology department and pharmacy department), as well as scientifically evident research present in literature. Finally, all the indicators are aggregated to calculate 5 macro-indices of convenience (Clinical ‘Ic’, Economical ‘Ie’, Social ‘Is’, Technological ‘It’ and Security and Processing ‘Isp’) through the formula shown below, in the calculation of the clinical index: Σci × pi IC = (1) Σpi Where “p” represents the priority of the specific sub-indicator “c.” The definition of the weight of “p” agrees with the importance of the sub-indicators and has been determined through the interviews with hospital experts. – 1-month report This represents the intermediate type of evaluation. Through bibliographic research, the indicators mainly used at an international level in the HS models were selected [6–11]. As reported in Table 1, they have been classified by dimensions (clinical, economical, socio-ethical, technological and organizational), marked “Y” if expert employee (that is they need an evaluation from a panel of experts in the sector) or otherwise “N” (usually the more objective indicators pertaining to technological and economic spheres). Finally, the selected indicators were prioritized through a phase of weighing carried out by the expert in-hospital personnel. – 1-week report Regarding the speedy model, the main theoretic reference is the Hospital Clinic of Barcelona model reported in scientific literature [9]. This has been preferred to the previous systems, more precise and detailed theoretic basis, including the EUnetHTA core model, inasmuch the model [9] better
732
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
Table 1 Selected evaluations indicators for intermediate HS evaluation Dimension Indicator Panel Definition (Y/N) Efficacy (4) Safety Y Patient potential undesirable effects with the new HT. Clinical benefit Y Efficacy/effectiveness of the new HT, based on scientific evidence, compared with current available HTs Patient Impact Y Ethical, psychological and quality of life issues associated with the new HT Quality of Evidence Y Quality of available scientific evidence of the new HT, using available quality scales (e.g. SIGN, GRADE; Oxford EBM). Cost (4) Incremental Cost N Increase in the cost-per-case when using the new HT, compared with cost-per-case with available alternative Net Cost N Difference between costs-per-process with the new HT and current level of funding from payers for the available alternative. Investment Effort N Impact of the acquisition of the new HT on overall hospital’s/department annual budget for HTs investment Economic impact N The direct health care costs (annual, lifetime) of the technology Burden of disease (2) Burden of disease 1 Y The prevalence, incidence, disease-adjusted life expectancy, Burden of disease 2 Y healthy years of life expectancy, or other relevant measurement of disease burden of the population with the clinical condition(s) or those that will be affected by the technology The economic burden of the clinical condition(s) in the population Hospital impact (5) Staff Requirements Y Hospital staff requirements (training, expertise. . .) for the new HT Physical space Impact N Space requirements for the new HT to be operational Process of care impact Y Impact of the new HT on the organization of healthcare services and/or work dynamics and flows within the hospital Variation in the rates of Y The coefficient of variation (standard deviation divided by the use mean) Alternatives Y The alternatives, if any, currently or soon available for the conditions that this technology treats Socio-politic (1) Ethical, legal and social Y The probability that an assessment comparing two or more techaspects nologies will help to inform important ethical, legal, or social issues Technology (2) Innovativeness N Level of novelty of the new HT
adapts to the Horizon Scanning-type evaluations that have a comparative approach and is based on a mainly qualitative (or semi-qualitative) character as defined by the cost/benefit index.” The selected indicators are shown in Table 2 and are subdivided in two macro categories: risk and benefit. Both categories are evaluated aggregating the indicators through the same formula (1) and weighing the priorities of the sub-indicators “c1” according to the reported weights in [9]. 3. Data collection Calculating in the indicators, the necessary data collection depends, as in the case of the indicators themselves, on the typology of the evaluation: extensive, intermediate or speedy. In the case of the extensive evaluation (1-year report), the data is collected through the research of national and international scientific evidence (carried out with typical epidemiological techniques) as well as through the consultation of data and in-hospital experts. Also regarding the intermediate evaluation (1-month report), this considers the use of scientific evidence and the consultancy of data and in-hospital experts. The difference from the previous case is that
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
733
Table 2 Selected evaluations indicators for speedy HS [9] CRITICALITY (C) Requests of personnel Structural impact Impact of the process Cost of operation Economical analysis Investment commitment BENEFIT (B) Safety Clinical efficacy Impact on the patient Cost/efficacy Reliability facts Innovativeness
DATA SOURCE IN-HOSPITAL DATA AND SCIENTIFIC EVIDENCE
DATA SOURCE SCIENTIFIC EVIDENCE
Table 3 Selected databases for the research of standardized evidence DATABASE LINK CENTRE FOR REVIEWS AND DISSEMINATION (UK) http://www.york.ac.uk/inst/crd/ CADTH (CANADA) http://www.cadth.ca/ HORIZON SCANNING (AU, NZ) http://www.horizonscanning.gov.au/ EUROSCAN (EU) http://euroscan.org.uk/ NATIONAL INSTITUTE FOR HEALTH AND CLINICAL EXCELLENCE http://www.nice.org.uk/ NIHR HTA (UK) http://www.hta.ac.uk/ NSC (UK) http://www.hsc.nihr.ac.uk/ CALIFORNIA TECHNOLOGY ASSESSMENT FORUM (US) http://ctaf.org/ INTERNATIONAL NETWORK OF AGENCIES OF HTA http://www.inahta.net/ COCHRANE SUMMARIES (UK) http://summaries.cochrane.org/ BANCA DATI VIHTA (ITA) http://www.ijph.it
the intermediate evaluation observes the research of the already compiled HS/HTA reports in the international and national reviews instead of launching a HS/HTA study from scratch. The primary objective is the contextualization, precisely in the hospital environment, of analyses that possibly already exist. For this reason the research of evidence becomes fundamental. In order to reduce possible system errors due to the subjectivity of the research to a minimum, it has been decided to select the strict set of electronic resources reported in Table 3, as representatives of the major DB, national and international, used in HS and HTA analyses [7–13]. It is necessary to remember, as in the intermediate evaluation, the scientific evidence is an informative support for the various in-hospital experts and not the direct way to calculate the clinical indicators. A detail of the collected data sheet sent to the expert in-hospital users of the specific technology is reported in Fig. 2. The found scientific evidence is attached to this sheet to be filled out through the use of the Lykert scale of 9 levels (completely disagree – completely agree). Lastly, for the speedy evaluation (1-week report), the significant difference from the other evaluations lies in the fact the calculation of the indicators belonging to the clinical benefits (see Table 2) primarily happens with the sole research of scientific evidence (reported DB in Table 3) and without the ordinary involvement of several in-hospital clinician experts. To calculate the indicators pertaining to the “criticality” area, in addition to the evidence in literature, specific in-hospital data is found (from informative applications or specific, direct advice with hospital experts).
734
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
Fig. 2. Detail of the model of collected data for the clinical indicators.
4. Report The objective of the report is to supply a clear tool to the decision-maker, simple to use and so complete that it can hold up in in-hospital decisions. These are the points that have represented the bases for the development of the 3 reports: 1-week, 1-month and 1-year reports. For the 1-year report, the chosen manner is the one connected to the graphic “radar” representation of the macro-indicators. In Fig. 3 the main page of the report is presented, subdivided into a “reliability” section, which evaluates the number of calculated indicators out of the totals in general and by setting; and a “results” section where the index of convenience is calculated as the total area of the pentagon identified by the macro-indices for dimensional setting. This is to have both a framework of general comparative evaluation and one for a specific setting. The example in Fig. 3 is relevant to the comparative evaluation of 3 technologies: tech1, tech2 and tech3. The intermediate evaluation report is subdivided into two parts: the clinical evaluation and the technical evaluation. For the technical one, there is a sheet that reports the data related to the economic and technological aspects (innovation, obsolescence, alternative existence, weight on the budget, Breaking Even Point), while for the clinical part, the choice is addressed towards a graphic representation (a three-level code with “red”, “yellow” and “green”) subdivided by user. In Fig. 4, the involvement of anesthetist, a surgeon, a medical physicist and a generic “other” is shown. Furthermore, as reported in the figure, it is possible to have an average for the specific area of evaluation (clinical efficacy, weight of the illness, impact on the hospital and socio-political aspect) as well as for the entire clinical scope (general average for the clinical area). Finally, it is noted that as for every evaluated index, it is possible to see its priority (weighted column), the improvement or deterioration expected from the introduction of the new technology with respect to the traditional method in use: improvement = green; deterioration = red; stable situation = yellow. The speedy evaluation (1-week report) improves the original point of reference [9] by improving the usability and completing it mainly with a series of attachments which better specify the bases of the con-
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
735
Fig. 3. 1-year report for extensive evaluation. (Colours are visible in the online version of the article; http://dx.doi.org/10.3233/ THC-140847)
ducted analyses (economic analyses, considered evidence, main motivations, etc.). The report is made up, firstly, of a part related to the description of the request and the technology under consideration. The second part contains, in addition to the considered scientific evidence and sequence used for the research of the evidence, the list of the detailed attachments and a section concerning the results of the comparatively calculated indicators based on determined decisions (affirmative, negative, same convenience or not calculated) that aim to answer the question: “With respect to traditional technologies, will the evaluated one prove more suitable?” Finally, the third section, reported in Fig. 5, shows the graphic representation of the results of the HS evaluation in terms of criticality/benefit. This permits an arrangement of more technologies within the graph, making it a multi-technological, comparative evaluation. Finally Fig. 6 reports the specific part of the economic evaluation through the calculations of two indices: the “Break Even Point” (BEP) defined by the difference Expenses-Earnings in relation to the number of operations; and the “Net Present Value” (NPV) capable of identifying the time necessary to
736
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
weight
index
surgery
anesthesia
other
medical physics
total
1 1 0,9
medium
low
medium
medium
low
low
medium
medium
low
low
medium
medium
impact on the patient
high
low
medium
medium
QALY/DALY
high
low
medium
medium
socio-economic weight
high
low
medium
medium
security
medium 0,8 0,7 1
Burden of Disease 1
medium 0,9
medium 1
Impact on the Hospital curing process
low
low
medium
medium
0,9
1
frequency of use
high
low
medium
medium
0,8
healthcare personnel
low
low
medium
medium
0,7
technical personnel
low
low
medium
medium
0,6
technological alternatives
high
low
medium
medium
ethical and legal elements
high
1
high
Socio-Political Aspects 1
low
medium
medium
medium AVERAGE FOR SPECIFIC AREA
GENERAL AVERAGE
Fig. 4. 1-month report for intermediate evaluation. (Colours are visible in the online version of the article; http://dx.doi.org/ 10.3233/THC-140847)
Fig. 5. 1-week report for speedy evaluation. (Colours are visible in the online version of the article; http://dx.doi.org/10.3233/ THC-140847)
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital Full Service
13.000
Full Risk
Full Service + Compartment Costs
737
Full Risk + Compartment Costs
Management Costs
9.800
6.600
3.400 BEP FR + CC
200 BEP FS BEP FS + CC -
3.000
5
10
15
BEP FR 20
40
Number of Surgical Interventions
Fig. 6. Economic evaluation of IORT in the hospital environment. (Colours are visible in the online version of the article; http://dx.doi.org/10.3233/THC-140847)
recover, if possible, the effectuated investment while considering the average number of yearly operations as an independent variable.
5. Results In the laboratory’s first year of activity (September 2012 – September 2013) two 1-year reports, one 1-month report and sixteen 1-week reports were developed. As an applicable example, the carried out practice on the evaluation of the addition of the IORT (Intra-Operative Radiation Therapy) technology within the hospital will be described. Following the previously mentioned methodology, the first phase reviewed the analyses of the inhospital request, which can be summarized as follows: 1. Evaluation of the use of IORT in the clinic with respect to the traditional method. 2. Evaluation of a possible, inexpensive offer to buy used equipment. 3. Schedule of an intermediate evaluation (1-month report). The second phase included the research of scientific evidence and its distribution, together with the clinical Lykert questionnaire, to the in-hospital users that have been selected from the following professional fields: general surgery and oncology, anesthesia, medical physics and radiotherapy. The response deadline given to the hospital users was two weeks. In Fig. 4, the clinical evaluation on the part of the personnel concerning oncological surgery, medical and health physics are reported. Regarding the calculation of the technical-economic indicators, it was possible to calculate the graph shown in Fig. 6. This represents the gross spending limit evaluated as “Expenses-Earnings” in relation to the number of yearly operations and the typology of the signed maintenance contract (yearly cost): full service or full risk.
738
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital
Considering the scenarios of use, in relation to the expected activity and the optional maintenance contract, it is possible to identify a figure to spend on the item calculating the yearly limit for the age of the equipment. Given that the graph is calculated as “Expenses-Earnings”, a positive limit represents an annual loss while a negative one represents a yearly gain. With the term BEP (Breaking Even Point) the points of balance between the cost of technological maintenance and the DRG earnings (for the typology of the contract and considering the fixed amortized cost of the operating theatre) is identified.
6. Discussion and conclusions The models presented here have permitted the construction, using personnel and laboratories within the hospital, of a system of models that permit a response in appropriate time to the needs of the HS and HTA evaluations in the hospital environment. Furthermore, the standardization of the schedules, reports and methodologies permits the hospital management to plan the number of requests and increase the in-hospital evaluation team through a training itinerary. The evaluative activity has had a strong increase, especially for the speedy evaluations (1-week report), in the second semester of activity, after its centralization in the first semester had been for the design and development of the models themselves. Regarding the applicable case of the IORT, the requests have been completely satisfied. The evaluation had a high and punctual participation by the in-hospital users involved in the analyses (6/7), in addition to the development of a supporting report that albeit producing specific information, gives the in-hospital decision-maker a “capacity for movement and planning” for its own strategic analyses. In fact, the example supplied by the estimated cost of purchase of used equipment, shows how the report, though supplying quantitatively clear elements, allows the calculation of the precise figure for the in-hospital planning and expected and/or predicted activity (number of yearly operations). The health-based clinical evaluation of the IORT shows a clinical efficacy comparable to traditional techniques, identifies expected improvements for the impact, the weight of the illness and the social aspect as well as considering it, in the specific analyzed context, a strategic element for the territory. Finally, the criticalities of the technology are the negative impact of its installation on the in-hospital organization, for the necessity of specific training of the medical technologist personnel as well as the alterations of the itineraries and process of hospital care.
References [1] [2] [3] [4] [5]
Ehlers L., Vestergaard M., Kidholm K., Bonnevie B., Pedersen P.H., Jorgensen T., Jensen M.F., Kristensen F.B., Kjolby M. Doing mini-health technology assessment in hospitals: A new concept of decision support in healthcare? Int J Technol Assess Health Care. 2006; 22(3): 295-301. Jacob R., McGregor M. Assessing the impact of health technology assessment. Int J Technol Assess Health Care. 1997; 13: 68-80. Douw, K. and H. Vondelin. Selection of new health technologies for assessment aimed at informing decision making: A Survey among horizon scanning systems. International Journal of Technology Assessment in Health Care, 2006. 22(2): 177-183. Kidholm K., Ehlers L., Korsbek L., Kjaerby R., Beck M. Assessment of the quality of mini-HTA. International Journal of Technology Assessment in Health Care, 25: 1 (2009), 42-48. Eunehta project, joint Action on HTA (2010–2012) funded by the European Union, in the framework of the Health Programme. HTA Core Model Online. https://meka.thl.fi/htacore. National Institute for Health and Welfare.
R. Miniati et al. / Development of sustainable models for technology evaluation in hospital [6]
[7] [8] [9] [10] [11] [12] [13]
739
Simpson S., Hiller J., Gutierrez-Ibarluzea I., Kearney B., Norderhaug I., Fay AF., Packer C., Asua J., Benguria G., Blanchard S., Blozik E., Bonnevie BM., Clifford T., Eckerlund I., Galnares L., Groeneveld K., Hae Lee Robin S., Hakak N., Husereau D., Ibargoyen N., Kaila M., KuâTłzli ¸ C., Llanos A., Luengo S., Morrison A., Mundy L., Tal O., Wallgren L., Wallin J. A toolkit for the identification and assessment of new and emerging health technologies. 2009. EuroScan. Birmingham. Effie Amanatidou, Maurits Butter, Vicente Carabias, Totti Könnölä, Miriam Leis, Ozcan Saritas, Petra Schaper-Rinkel and Victor van Rij. On concepts and methods in horizon scanning: Lessons from initiating policy dialogues on emerging issues. Science and Public Policy (2012) 39 (2): 208-221. Hussein Z. Noorani, Donald R. Husereau, Rhonda Boudreau and Becky Skidmore. Priority setting for health technology assessments: A systematic review of current practical approaches. International Journal of Technology Assessment in Health Care, 23: 03(2007), 310-315. Laura Sampietro-Colom, Irene Morilla-Bachs, Santiago Gutierrez-Moreno and Pedro Gallo. Development and test of a decision support tool for hospital health technology assessment. International Journal of Technology Assessment in Health Care, 28: 4(2012), 460-465. Iñaki Gutierrez-Ibarluzea, Sue Simpson and Gaizka Benguria-Arrate. Early Awareness and Alert Systems: An Overview of EuroScan Methods. International Journal of Technology Assessment in Health Care, 28: 03, (2012), 301-307. Guy Paré, Placide Poba-Nzaou and Claude Sicotte. Home Telemonitoring for Chronic Disease Management: An Economic Assessment. International Journal of Technology Assessment in Health Care, 29: 2, 155-161. Leonora Varela-Lema, Jeanette Punal-Riobóo, Beatriz Casal Acción, Alberto Ruano-Ravina and Marisa López García. Making processes reliable: A validated PubMed search strategy for identifying new or emerging technologies. International Journal of Technology Assessment in Health Care, 28: 4 (2012), 452–459. Claire Packer, Matthew Fung and Andrew Stevens. Analyzing 10 years of early awareness and alert activity in the united kingdom journal. International Journal of Technology Assessment in Health Care, 28: 3 (2012), 308–314.
Copyright of Technology & Health Care is the property of IOS Press and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
