bs_bs_banner
Journal of Evaluation in Clinical Practice ISSN 1365-2753
Can complex health interventions be evaluated using routine clinical and administrative data? – a realist evaluation approach Iiris Riippa MSc (Tech) MSc (Econ),1 Olli-Pekka Kahilakoski MSc (Tech),2 Miika Linna DSc (Tech)3 and Minni Hietala MSc (Econ)1 1 Researcher, 2Research Assistant, 3Senior Researcher, The Institute of Healthcare Engineering and Management, Department of Industrial Engineering and Management, Aalto University, Espoo, Finland
Keywords CMO approach, complex intervention, primary health care, realist evaluation, routine data Correspondence Ms Iiris Riippa The Institute of Healthcare Engineering and Management Department of Industrial Engineering and Management Aalto University PO Box 15500 FI-00076 Aalto Finland E-mail: [email protected] Accepted for publication: 14 April 2014 doi:10.1111/jep.12175
Abstract Rationale, aims and objectives Interventions aimed at improving chronic care typically consist of multiple interconnected parts, all of which are essential to the effect of the intervention. Limited attention has been paid to the use of routine clinical and administrative data in the evolution of these complex interventions. The purpose of this study is to examine the feasibility of routinely collected data when evaluating complex interventions and to demonstrate how a theory-based, realist approach to evaluation may increase the feasibility of routine data. Methods We present a case study of evaluating a complex intervention, namely, the chronic care model (CCM), in Finnish primary health care. Issues typically faced when evaluating the effects of a complex intervention on health outcomes and resource use are identified by using routine data in a natural setting, and we apply context-mechanismoutcome (CMO) approach from the realist evaluation paradigm to improve the feasibility of using routine data in evaluating complex interventions. Results From an experimentalist approach that dominates the medical literature, routine data collected from a single centre offered a poor starting point for evaluating complex interventions. However, the CMO approach offered tools for identifying indicators needed to evaluate complex interventions. Conclusions Applying the CMO approach can aid in a typical evaluation setting encountered by primary care managers: one in which the intervention is complex, the primary data source is routinely collected clinical and administrative data from a single centre, and in which randomization of patients into two research arms is too resource consuming to arrange.
Introduction In chronic health care, few interventions are truly simple. They target several levels of the organization, because most improvements in care provision require changes on the care frontline and on the organization’s administrative levels. As the importance of the patient’s role in managing chronic conditions has become evident, elements of involving patients as active managers of their conditions are typically included in chronic care interventions [1]. This requires specific skills on the part of care professionals as well as patients. Furthermore, some customization of the intervention often is called for due to patients’ differing levels of skill and ability. A complex intervention is typically described as containing several interacting components. Although there is no sharp
boundary between complex and simple interventions, some characteristics are perceived as increasing the complexity of an intervention and should be considered when evaluating it. These characteristics are (i) number of interacting components within the intervention [2]; number and difficulty of behaviours required by those delivering or receiving the intervention; (iii) number of groups or organizational levels targeted by the intervention; (iv) number and variability of outcomes; and (v) degree of flexibility or tailoring of the intervention permitted [2]. The characteristics of complex interventions cause additional challenges for evaluators ‘in addition to the practical and methodological difficulties that any successful evaluation must overcome’ ([3], p. 6). The main challenges discussed in the literature are varying implementation of the intervention in different contexts and specification of relevant outcomes. Datta and Petticrew
Journal of Evaluation in Clinical Practice 20 (2014) 1129–1136 © 2014 John Wiley & Sons, Ltd.
1129
Complex intervention evaluation
[1] state that the design of an intervention should be based on theoretical understanding of its causal effects and, therefore, the evaluator should have a hypothesis about expected effects. However, ‘it may not be possible to predict which elements of an intervention will be acceptable to healthcare staff and patients, have the desired effect and be sustainable’ ([1], p. 12). Multiple contextual factors might form barriers to implementation and integration of an intervention into usual service. For comparison purposes, evaluators must be aware of possibly tailored elements of the intervention. Such awareness contributes to understanding what kind of intervention the results are attributable to. Oakley et al. [4] demonstrate how acknowledgement of differences in implementation of a sex education intervention added to the understanding of what kind of intervention is effective in which circumstances and for which target group. However, the UK Medical Research Council [3] notes that identifying ‘the active ingredient’ of an intervention may be impossible in some cases. In these cases, analysing effects of the intervention on intermediary outcomes might disclose new understanding of the intervention’s essence. In their study on national implementation of electronic health records (EHRs), Takian et al. [5] decide to forgo quantitative evaluation of final outcomes and, instead, used qualitative methods to acquire rich data on the process effects of EHR implementation to better understand what in the intervention causes potential effects on the health care system as a whole. Disclosure of the implemented intervention and its mechanism is closely related to specification of outcome measures. Definition of ‘the length and complexity of the causal chains linking intervention with [end] outcome’ ([3], p. 6) and verification of this mechanism through changes in intermediary outcomes are important to interpreting results. For example, Campbell et al. [6] state that in the case of a negative trial result, intermediate outcomes are important to identifying the point along the causal pathway at which the intervention failed. Similarly, in the case of a positive trial result, intermediary outcomes can verify that the effect is, indeed, attributable to the expected mechanism. Realist evaluation, first introduced in the social sciences [7], offers tools to evaluate complex interventions. The approach was introduced by Pawson and Tilley [7] to criticize the use of classic experimental approach to evaluation that dominates the toolkit of evidence-based medicine for evaluating interventions [8]. The main problem of traditional experimental approach, in the case of complex interventions, is the focus on outcomes, which do not explain why an intervention works; in other words, what active ingredients and context are required for the intervention to be effective. This treatment of an intervention as a black box has led to inconsistent findings that do not explain why the intervention failed or succeeded [7]. Elissen et al. [9] further explicate that ‘[a]nalyzing complex [disease management programs] necessitates a clear framework that links expected outcomes to the characteristics of both the program and its target population, and measures effects over an adequate period of time’. In accordance with suggested solutions to the challenges of evaluating complex interventions, Pawson and Tilley [7] present the contextmechanism-outcome (CMO) approach, which directs the evaluator’s attention to the context in which the intervention is implemented and the intervention’s mechanism in addition to the final outcomes. 1130
I. Riippa et al.
Although previous studies discuss evaluation of complex interventions, limited attention has been paid to the feasibility of using routinely collected clinical and administrative data in evaluation. The purpose of this study is to examine the feasibility of using such data in evaluating complex interventions. We present a case study of evaluating a complex intervention, namely, the chronic care model (CCM) [10], in Finnish primary care to demonstrate the feasibility of using routine data and to show how the data could be improved through the CMO approach. The case study consists of two parts. In the first part, routine data are used to compare changes in health outcomes and resource use between CCM patients and usual care patients. In the second part, interviews with health care professionals are conducted to examine retrospectively the implementation of the CCM in the case organization.
Methods CCM The CCM is built on the evidence of effective system changes that improve clinical [11,12] and patient-reported [13] health outcomes and reduce the cost of chronic care [14]. The CCM was refined by Wagner et al. [15] to counterbalance the emphasis on acute care by focusing on longitudinal and patient-centred care for the chronically ill [15]. The CCM suggests that chronic care should be organized based on six interrelated components [15,16]. Two of these refer mainly to the context in which chronic care is provided: (i) the health care organization, which encourages patient-centred care through, for example, incentive programmes; and (ii) links to community resources beyond the organization providing chronic care. The remaining four components are related to the actual delivery of care: (iii) self-management support, which assists patients to better manage their own chronic condition; (iv) delivery system design to provide coordinated, multidisciplinary team care to the patient; (v) decision support for providers to assure the expertise and memory aids to routinely provide evidence-based care; and (vi) clinical information systems, which facilitate collecting, organizing and accessing relevant patient information [15–17].
Routine data analysis of CCM effects on care outcomes Routinely collected clinical and administrative data from 2007 to 2011 on all patients with type 2 diabetes treated in the city of Espoo (population c. 250 000) were used to evaluate the effects of the CCM on health outcomes and resource use. Effects of the intervention were examined by comparing two groups of patients with type 2 diabetes: a case group, which received care according to the CCM, and a control group, which received usual care. The following describes how the case and control groups and the change in health outcomes and resource use were identified in routine clinical and administrative data. The Results section discusses feasibility of using routine data. A complete report on the analysis, following STROBE guidelines for reporting observational studies, is available from the authors. Case and control groups The study cohort, 9278 patients with type 2 diabetes, was identified by patients’ recorded diagnoses (ICD-10 diagnoses in blocks
© 2014 John Wiley & Sons, Ltd.
I. Riippa et al.
E11 or E13) or visit-related information (ICPC codes T90 or T88). Among the cohort, patients exposed to the CCM and, thus, belonging to the intervention group were identified by the presence of a care plan in their visit-related records. Patients recruited to the CCM in Espoo drew up care plans jointly with their care professional to support their self-management of health. The care plan stated the main care goals based on the patient’s main healthrelated problems. The plan was saved in the EHR and could be shared with the entire care team. Therefore, care plans implemented three CCM components: self-management support, clinical information systems and a delivery system to provide coordinated, multidisciplinary team care to the patient. In addition to the care plan, the CCM consisted of counselling the patient and annual follow-up visits. The control group consisted of patients with type 2 diabetes in Espoo who had no care plans in their records and who, thus, received usual care. In the usual chronic care in Espoo, each health care professional was assumed to generally follow the care guidelines established by the Finnish Medical Society Duodecim. As is typical in retrospective, natural settings, there were some differences in descriptive characteristics between the intervention and control groups. After exclusion of patients who had moved away or died before January 2012, the intervention group included 6672 patients, and the control group 2606 patients. The intervention group consisted of older patients than those in the control group, and it had a larger proportion of men. In addition, more patients in the intervention group had co-morbid cardiovascular disease. Before enrolment in the CCM, during November 2006– November 2007, the control group averaged 1.5 mmol mol−1 lower levels of HbA1c than the intervention group and had visited a nurse 0.6 times more often, on average (Supporting Information Appendix S1). Outcome measures and controlled confounding factors Primary outcomes were HbA1c and resource use, operationalized as the number of doctor and nurse visits for each patient. In addition, routinely gathered data allowed us to control for the following confounding factors: age, gender, co-morbid cardiovascular disease, onset of CCM and health centre that the patient used most. In the case organization, another intervention, personal health record, was implemented during the study period. This intervention was also included in the controlled confounding factors. Statistical methods Because the data were not generated in a controlled research setting, statistical modelling was used to control patient heterogeneity. A Bayesian linear mixed model was used to assess CCMrelated changes in HbA1c levels, and Bayesian generalized linear mixed models were used to analyse nurse and doctor use. Observation-level variables included in the models were age, gender, co-morbid cardiovascular disease, time of enrolment in CCM, patient group, observation year, health centre that the patient used most and concomitant personal health record intervention. The models also included patient-level varying intercepts. Generalized, linear mixed models used Poisson-lognormal distri-
© 2014 John Wiley & Sons, Ltd.
Complex intervention evaluation
butions to account for overdispersion. Uniform priors were used for model parameters. Effect of the CCM was computed as the difference in predictions of the model between the two values of the CCM indicator variable, averaged over the case group. The effect was summarized on the scale of the response variable. To assess the sensitivity of the results on model selection, we also conducted the analysis without adjusting for patient- or observation-specific variables. Baseline characteristics were compared between groups using Bayesian statistics: for continuous variables, normal models with reference priors were used; for proportions, binomial models with dependent Howard priors were used [18]. Missing values in HbA1c measurements were assumed to be missing completely at random and were therefore deleted listwise. We did not remove any measurements as outliers. There were no missing values in the resource use data. Data processing and statistical computation were done using R version 2.15.1 [19]. The models for HbA1c and resource use were fitted using the rstan package [20]. Baseline characteristics for proportions were compared using the LearnBayes package [21].
Implementation of CCM in the case organization As suggested by the literature on complex interventions, we examined how the CCM was implemented in the case organization. We interviewed five physicians and six nurses working in 5 of the 11 health centres in the city of Espoo. Interviews were based on the Assessment of Chronic Illness Care (ACIC) questionnaire, which is a tool for evaluating the realization of the six dimensions of the CCM in a health care organization [22]. Interviews were recorded, and a content analysis was conducted. The case organization had designed and customized the CCM to meet its needs during 2006–2007, and had then started to implement it. Issues mentioned by interviewees reflect the status quo of the CCM at the time of the interviews (June 2012). To differentiate between the status quo and the initial plan for implementation, we examined interview findings in relation to the case organization’s documents on the initial CCM implementation plan. A full report on the implementation inquiry is available from the authors.
Results Routine data analysis of CCM effects on care outcomes During the first year after enrolment in the CCM, the difference in nurse visits due to the CCM was 1.8 visits per year (1.7–1.9) and in doctor visits 0.7 visits per year (0.5–0.7). The differences indicate that the CCM increased both nurse and doctor use. During the following 3 years, the increase in average number of nurse visits diminished to 0.5 visits per year (0.4–0.7) in the second year, to 0.4 visits per year (0.2–0.5) in the third year and to 0.3 visits per year (0.0–0.5) in the fourth year. However, the number of doctor visits remained at the baseline level after the first year (Supporting Information Appendices S2 and S3). During the first year in the CCM, the estimated mean effect of the CCM on HbA1c was −1.4 mmol mol−1 (95% Bayesian confidence interval: −1.7 to −1.2; a negative difference indicates that the 1131
Complex intervention evaluation
intervention was effective). The change was sustained, approximately, during the following 3 years: the average effect of the CCM on HbA1c during the first 4 years was −1.3 mmol mol−1 (−1.6 to −1.0) (Supporting Information Appendices S2 and S3). To evaluate the effect of confounding factors, we conducted the analysis without controlling for them. Non-adjusted analysis showed similar results, although the effects of the CCM on both resource use and HbA1c were smaller than in our primary analysis (Supporting Information Appendix S4). Routine data analysis showed that the CCM decreased patients’ HbA1c levels by an average of 1.4 mmol mol−1 (1.2–1.7) during the first year in the care model. This decrease was gained through a small increase in resource use: an increase of 1.8 nurse visits (1.7–1.9) and 0.7 doctor visits (0.6–0.7) during the first year in the CCM. This implies that the CCM model may be moderately cost-effective.
Implementation of CCM in the case organization Interview data indicated that although implementation was executed successfully, possible risks noted during the design stage were realized after implementation. The following describes findings of this qualitative inquiry, divided into the six dimensions of the ACIC questionnaire. 1 The health care organization had supported patient-centred care when it introduced the CCM, but later, support was perceived to have diminished. The aim that the CCM would become a permanent and continuously evolving practice was not fulfilled. In addition, interviewees perceived support as varying among areas, and they mentioned management and leadership issues as root causes. Furthermore, cost pressures and personnel shortages were perceived as taking precedence over CCM care issues. 2 Both at the time of implementation and during interviews, professionals did not recognize many links to community sources beyond their organization. Use of private or third-party services depended on the individual physician or, more importantly, the nurse. 3 The self-management support, which was considered a key issue in the implementation phase, had not become a working practice in the organization. For instance, although the care plan was considered comprehensive, a copy of it was not always given to the patient. In addition, the care plan did not necessarily include goals or activities for the patient to follow. 4 Design of the delivery system was not believed to fully support providing patients with coordinated, multidisciplinary team care. During implementation, increased emphasis was placed on collaboration among various health care professionals. At the time of the interviews, CCM care was mainly the responsibility of nurses, who consulted doctors when medically necessary. 5 Decision support for providers to ensure provision of evidence-based care was perceived as insufficient. During implementation, decision support took the form of documents, lectures and training. Interviewees mentioned that support ceased after implementation. 6 Some interviewees thought clinical information systems were helpful in processing relevant patient information, whereas others thought relevant information was difficult to find. Information 1132
I. Riippa et al.
systems had remained the same from implementation to the time interviews were conducted. On the bases of interviewee perceptions and documents on CCM implementation received from the case organization, two main findings on the success of CCM implementation could be made. The first was that despite initial comprehensive plans and instructions for CCM adoption, systematic guidelines for the model’s long-term use and development in daily care were perceived as insufficient. The second was that a complex intervention presents a multitude of issues to govern, and without continuous support and management, outcomes of an intervention may be short lived.
Weaknesses of the routine data analysis of CCM effects on care outcomes Although according to analysis of routine data, the CCM had a positive, albeit clinically insignificant, effect on health outcomes through small increases in resource use, the results must be interpreted in the light of three main weaknesses of the analysis. First, due to the natural setting of the study, participants were not assigned randomly to the case and control groups. Because the factors that affect recruitment and willingness to participate are not known, let alone measured, they could not be controlled for in the model to account for the selection bias. Therefore, results might be attributable to specific characteristics of those patients who ended up in the intervention group, not to the intervention itself. Second, the exposure indicator used does not accurately separate patients exposed to the intervention from those unexposed. The care plan was the best indicator available in routine administrative data for identifying patients exposed to the intervention. However, the care plan can be interpreted to detect exposure to only three of six components of the CCM. Furthermore, it is plausible that the control group had been exposed to elements of the intervention that cannot be delivered to a patient personally but that are delivered to the entire population served, such as patientcentred organization of health care. Therefore, the results cannot be interpreted as deriving from the entire CCM but, rather, from some components of it. Third, proper implementation of the intervention could not be detected in routinely gathered clinical and administrative data. Interview findings suggest that there were some deficiencies in implementation of the CCM. Again, this raises the question of whether the intervention caused the results. The first two issues, selection bias and inaccuracy of the exposure indicator, cannot be fully overcome in single-centre evaluation of a complex intervention. In such a case, evidence on the causal chain that is expected to lead to the change in outcome is particularly important. In contrast to the rather experimentalist approach applied in the first part of the case study, in the following, we show how the CMO approach from realist evaluation paradigm could aid in evaluating complex interventions using routine data.
Application of CMO approach to identify the data needs for intervention evaluation The proposition that ‘causal outcomes follow from mechanisms acting in contexts . . . is the axiomatic base upon which all realist
© 2014 John Wiley & Sons, Ltd.
I. Riippa et al.
Complex intervention evaluation
explanation builds’([6], p. 58). Pawson and Tilley [7] contrast the CMO approach with the classic experimentalist approach, OXO, which stands for ‘observe a system (O), introduce a perturbation (X) to some participants but not others, and then observe again (O)’ ([7], p. 1183). Pawson and Tilley [7] criticize OXO for its focus on outcomes at the expense of considering the mechanism and context of the intervention. In addition to changes in outcomes, the CMO approach pays attention to the mechanism that causes the effect and the context in which the effect can be observed. This approach adds to the understanding of how outcomes can be managed. To form a CMO configuration of the CCM, we start by identifying the central effect mechanisms of the model and explain how the six components of the model trigger these mechanisms. The eventual configuration with context and outcome measures that are (i) relevant in terms of the suggested mechanisms and (ii) suitable for routine data collection are presented in Table 1. The configuration is not an exhaustive list of indicators for evaluating the whole chain of causes and effects of the CCM. Rather, it attempts to show how the configuration can point out essential evaluation aspects of a complex intervention. According to Wagner and colleagues [15], the components of CCM represent ‘the enhancements to the organization and its practices that contribute to productive interactions between providers and patients’. They specify that productive interactions ‘consistently provide the assessments, support for selfmanagement, optimization of therapy, and follow-up associated with good [disease control] outcomes’. Two distinct mechanisms triggered by CCM components can thus be identified: improved clinical care and self-management support to the patient. Different components of the CCM are intertwined and some of them are more directly linked to the before mentioned mechanisms than others. Health care organization enables changes in other elements of CCM through ‘strong leadership, appropriate incentives, and effective improvement strategies’ [15]. Self-management support is listed as an independent component in CCM literature. It helps patients to ‘set limited goals for improving their management of their illness, identify barriers to reaching their goals, and develop a plan to overcome the barriers’ [15]. The four other
components are essential in the provision of both clinical care and self-management support: links to community resources complement the needed clinical care and self-management support provided by the organization; delivery system design enables the productive interactions and thus constitutes the platform for the clinical care and self-management support; decision support assures that the care teams ‘have the expertise to provide appropriate clinical and behavioral management’ [15]; and finally, clinical information systems allow the professionals to reply to the specific needs of individual patients and enable the resource allocation and service provision planning to effectively meet the needs of the population served [15]. In the case of CCM, relevant context factors that contribute to or hinder the effects of the intervention services are characteristics of the patient and the caregiver. Such measurable patient characteristics are, for example, willingness to participate and current state of clinically measured health. One caregiver characteristic that can affect the intervention’s effect on patients is the recruitment rate of the health care professional or health centre that recruited the patient to the intervention. Furthermore, because a higher number of care professionals treating the patient can affect continuity of care, the number of professionals taking care of the patient could be included as a context measure. As suggested by Campbell et al. [6] in their study on complex intervention evaluation, we gathered intermediary and final outcomes to better grasp the intervention’s causal effects. Here, process outcomes present changes in the care delivered, which, in turn, are expected to result in improvements in clinical care, including, for example, adjustments in medications and procedures to follow care guidelines and provision of better self-management support. Although improvements in both clinical care and self-management support are expected to have a positive effect on clinical outcomes, the time lag in these effects is plausibly different. Improved clinical care can be expected to affect clinical health indicators in the short term, whereas the effects of self-care support can be expected to lag, as psychosocial changes and/or changes in knowledge base of the patient are needed first to trigger lifestyle changes, which, in turn, can affect the patient’s
Table 1 Context-mechanism-outcome configuration of CCM Context
Mechanism
Outcome
Patient characteristics: • Willingness to participate (yes/no) • State of clinical health (HbA1c)
Improvements in clinical care and self-management support triggered by enhancements in the six CCM components: 1 Health care organization 2 Links to community resources 3 Self-management support 4 Delivery system design 5 Decision support 6 Clinical information systems
Process outcomes: • Assessment of Chronic Illness Care • Availability of clinical health indicators
Provider characteristics: • Recruitment rate of the caregiver (doctor, nurse, health centre) • Number of caregivers (nurses, doctors, health centres)
Clinical health outcomes: • HbA1c • LDL Self-management outcomes: • Knowledge, skills and confidence in managing one’s health (e.g. Patient Activation Measure) • Health behaviour (e.g. physical activity, diet, smoking) Resource use outcomes: • Nurse visits • Doctor visits
CCM, chronic care model; LDL, low-density lipoprotein.
© 2014 John Wiley & Sons, Ltd.
1133
Complex intervention evaluation
Patient characteristics Provider characteristics
Process outcomes Clinical health outcomes Self-management outcomes
Resource use outcome
I. Riippa et al.
Measure
Availability in routine data
Willingness to participate (yes/no) State of clinical health (HbA1c) Recruitment rate of the caregiver (doctor, nurse, health centre) Number of caregivers (nurses, doctors, health centres) Assessment of Chronic Illness Care Availability of clinical health outcomes HbA1c LDL Knowledge, skills and confidence in managing one’s health (e.g. Patient Activation Measure) Health behaviour Nurse visits Doctor visits
No Yes No
Table 2 Feasibility of using routine data in the light of CMO approach
Yes No Yes Yes Yes, but not recorded sufficiently Self-care ability and barriers to self-care were in use, but not recorded adequately Yes, but not recorded adequately Yes Yes
CMO, context-mechanism-outcome; LDL, low-density lipoprotein.
clinical health status. In order to capture the intervention’s effect on clinical care and self-care support, outcomes for both parts are needed. Two relevant clinical health outcomes in type 2 diabetes management are HbA1c and low-density lipoprotein. In short-term evaluation of self-care support outcomes, changes in a patient’s psychosocial state and health behaviour are informative. One example of a validated psychosocial instrument for measuring a patient’s knowledge, skills and confidence in managing his or her health is the Patient Activation Measure (see, e.g. [23,24]). Information on a patient’s health behaviour can be collected with questionnaires on physical activity, diet, smoking and other aspects relevant to managing specific conditions. As resource constraints are a reality in all health care provision, care outcomes of an intervention always should be considered in relation to possible changes in resource consumption due to the intervention. Applicable measures for patient-specific resource use are visits with doctors and nurses. Process outcomes suitable for CCM evaluation are the ACIC survey, through which professionals’ perceptions of CCM realization can be evaluated, and the availability of relevant clinical health indicators. Table 2 presents availability of various evaluation measures in the routine data of the case organization. Routine data proved more feasible to use in evaluating changes in clinical care provision than in self-management support. One process measure for clinical care, the availability of relevant clinical indicators, could be traced in the frequency of specific laboratory tests. In addition, for type 2 diabetics, HbA1c serves as an appropriate clinical health outcome, and its levels were recorded adequately. However, changes in self-management support provision or self-management outcomes could not be evaluated with routine data. The case organization used measures for self-care ability and barriers to it, but especially the former was rarely recorded. Furthermore, of relevant context indicators, only a patient’s clinical health and the number of caregivers were available. Two other context indicators relevant to self-management success were unavailable: (i) patient’s willingness to participate in the intervention and (ii) the recruitment rate of the caregiver who recruited the patient. Nurse and doctor visits were recorded systematically in routine data and are sufficient measures of resource outcome in CCM 1134
evaluation. In the case organization, no indicators for realization of the intervention mechanisms (e.g. ACIC) were available in the routine data. Therefore, routine data analysis cannot detect whether changes in outcome indicators were attributable to the CCM or to some other concomitant changes in the organization (Table 2).
Discussion In this study, we discussed a typical evaluation setting encountered by primary care managers, one in which the intervention is complex, the primary data source is routinely collected clinical and administrative data, and randomization of patients into two research arms is too resource consuming to arrange. Through a case study on evaluating complex intervention, we examined the feasibility of using routine data in such an evaluation setting. The findings of the study showed that the classic experimentalist approach yields few tools for evaluating complex interventions in natural settings, in which context and implementation of intervention mechanisms often vary. To suggest an alternative to the experimentalist approach, we demonstrated how the CMO approach from the realist evaluation paradigm could aid in identifying measures needed for evaluation. The CMO configuration of the care model revealed deficiencies in the context and outcome indicators available in routine data of the care organization. First, the way in which the CCM was implemented in the case organization could not be detected in the routine data. Second, an essential effect of care model mechanisms, namely, the effect on self-management support provision and final self-management outcomes, could not be traced in routine data. Furthermore, context measures relevant to self-management success, such as a patient’s willingness to participate in the intervention and the recruitment rate of the specific caregiver who recruited that patient, were not available. As far as we are aware, this is the first study to apply CMO approach in examining routine data feasibility in evaluating complex care interventions. Our demonstration shows that with the CMO approach, collection of routine data could be designed to meet the needs of complex intervention evaluation in a natural
© 2014 John Wiley & Sons, Ltd.
I. Riippa et al.
setting more adequately. In practice, there is often a trade-off between efficiency of data collection and coverage of the causal chain of effects through measurement. Therefore, the appropriate level of measurement coverage should be considered when evaluating a specific intervention, along with which measures could be added to routine data collection. In this study, we focused on utilizing and examining the usability of routine data in complex intervention evaluation. More advanced methods for complex intervention evaluation are available when the study objective allows collection of data beyond that routinely gathered (see, e.g. [25]). This study indicated that gaps in routine data collection identified using CMO approach might prove relevant not only in terms of the specific intervention that is addressed, but in terms of the care organization’s performance in a broader perspective. Global [26] and national care guidelines [27] mention a patient’s active participation as an essential part of type 2 diabetes care. Yet, in this case study, the CMO configuration revealed a lack of indicators in the case organization for self-care support and self-care outcomes. This might suggest needs for changes in the collection of routine data for managerial purposes. In the case organization, weaknesses in routine data originated from two issues: either a measure for a certain outcome or a context characteristic was not defined in the organization or, if defined, it was not recorded adequately. When the decision is made to routinely record a measure, the system should support that using the gathered data in decision making, which, in turn, further encourages individual professionals to continue recording the measure.
Conflict of interest The authors declare no conflict of interest. The authors alone are responsible for the content and writing of the paper.
Acknowledgements The authors wish to thank medical doctors Tuomo Lehtovuori and Osmo Saarelma from Espoo primary care for their valuable comments on the chronic care model in Espoo and Tieto Corporation for collection of the data. In addition, DSc (Tech) Karita Reijonsaari from Aalto University deserves special recognition for assisting in planning the study. The study was (partly) supported by the SalWe Research Program for IMO (Tekes – the Finnish Funding Agency for Technology and Innovation grant 648/10).
References 1. Datta, J. & Petticrew, M. (2013) Challenges to evaluating complex interventions: a content analysis of published papers. BMC Public Health, 13 (1), 568. 2. Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I. & Petticrew, M. (2008) Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ (Clinical Research Ed.), 337, 979. Available at: http://www.ncbi.nlm.nih.gov/ pmc/articles/PMC2769032/ (last accessed 10 April 2013). 3. Medical Research Council. (2008) Complex interventions guidance. Available at: http://www.mrc.ac.uk/Utilities/Documentrecord/index .htm?d=MRC004871 (last accessed 12 September 2013).
© 2014 John Wiley & Sons, Ltd.
Complex intervention evaluation
4. Oakley, A., Strange, V., Bonell, C., Allen, E. & Stephenson, J. (2006) Health services research: process evaluation in randomised controlled trials of complex interventions. BMJ (Clinical Research Ed.), 332 (7538), 413. 5. Takian, A., Petrakaki, D., Cornford, T., Sheikh, A., Barber, N. & National NHS Care Records Service Evaluation Team (2012) Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems. BMC Health Services Research, 12 (1), 105. 6. Campbell, N. C., Murray, E., Darbyshire, J., et al. (2007) Designing and evaluating complex interventions to improve health care. BMJ (Clinical Research Ed.), 334 (7591), 455–459. 7. Pawson, R. & Tilley, N. (1997) Realistic evaluation. Sage. Available at: http://www.google.com/books?hl=fi&lr=&id=GXagvZwC2bcC& oi=fnd&pg=PR9&dq=realist+evaluation+pawson&ots=ZQLpVA0rCx &sig=VijUxnuGyw2ZaFRqZ3sXgB0anFQ (last accessed 30 August 2013). 8. Berwick, D. M. (2008) The science of improvement. JAMA: The Journal of the American Medical Association, 299 (10), 1182–1184. 9. Elissen, A. M. J., Duimel-Peeters, I. G. P., Spreeuwenberg, C., Spreeuwenberg, M. & Vrijhoef, H. J. M. (2012) Toward tailored disease management for type 2 diabetes. The American Journal of Managed Care, 18 (10), 619–630. 10. Wagner, E. H., Austin, B. T. & Von Korff, M. (1996) Organizing care for patients with chronic illness. The Milbank Quarterly, 74 (4), 511–544. 11. Bodenheimer, T., Wagner, E. H. & Grumbach, K. (2002) Improving primary care for patients with chronic illness. JAMA: The Journal of the American Medical Association, 288 (15), 1909–1914. 12. Musacchio, N., Lovagnini Scher, A., Giancaterini, A., Pessina, L., Salis, G., Schivalocchi, F., Nicolucci, A., Pellegrini, F. & Rossi, M. C. E. (2011) Impact of a chronic care model based on patient empowerment on the management of type 2 diabetes: effects of the SINERGIA programme. Diabetic Medicine: A Journal of the British Diabetic Association, 28 (6), 724–730. 13. Battersby, M., Von Korff, M., Schaefer, J., Davis, C., Ludman, E., Greene, S. M., Parkerton, M. & Wagner, E. H. (2010) Twelve evidence-based principles for implementing self-management support in primary care. Joint Commission Journal on Quality and Patient Safety, 36 (12), 561–570. 14. Bodenheimer, T., Lorig, K., Holman, H. & Grumbach, K. (2002) Patient self-management of chronic disease in primary care. JAMA: The Journal of the American Medical Association, 288 (19), 2469–2475. 15. Wagner, E. H., Austin, B. T., Davis, C., Hindmarsh, M., Schaefer, J. & Bonomi, A. (2001) Improving chronic illness care: translating evidence into action. Health Affairs, 20 (6), 64–78. 16. Wagner, E. H., Bennett, S. M., Austin, B. T., Greene, S. M., Schaefer, J. K. & Vonkorff, M. (2005) Finding common ground: patientcenteredness and evidence-based chronic illness care. Journal of Alternative and Complementary Medicine, 11 (Suppl. 1), S7–S15. 17. De Bruin, S. R., Heijink, R., Lemmens, L. C., Struijs, J. N. & Baan, C. A. (2011) Impact of disease management programs on healthcare expenditures for patients with diabetes, depression, heart failure or chronic obstructive pulmonary disease: a systematic review of the literature. Health Policy, 101 (2), 105–121. 18. Howard, J. V. (1998) The 2 × 2 table: a discussion from a Bayesian viewpoint. Statistical Science, 13 (4), 351–367. 19. R Development Core Team (2005) The R Project for statistical computing. Available at: http://www.r-project.org/ (last accessed 12 September 2013). 20. Stan Development Team (2014) Stan: project home. Available at: http://mc-stan.org/ (last accessed 12 September 2013). 21. Albert, J. (2012) Package LearnBayes. Available at: http://cran.rproject.org/web/packages/LearnBayes/index.html (last accessed 12 September 2013).
1135
Complex intervention evaluation
22. Bonomi, A. E., Wagner, E. H., Glasgow, R. E. & VonKorff, M. (2002) Assessment of chronic illness care (ACIC): a practical tool to measure quality improvement. Health Services Research, 37 (3), 791– 820. 23. Hibbard, J. H., Mahoney, E. R., Stockard, J. & Tusler, M. (2005) Development and testing of a short form of the patient activation measure. Health Services Research, 40 (6 Pt 1), 1918–1930. 24. Fowles, J. B., Terry, P., Xi, M., Hibbard, J., Bloom, C. T. & Harvey, L. (2009) Measuring self-management of patients’ and employees’ health: further validation of the Patient Activation Measure (PAM) based on its relation to employee characteristics. Patient Education and Counseling, 77 (1), 116–122. 25. Greenhalgh, T., Humphrey, C., Hughes, J., Macfarlane, F., Butler, C. & Pawson, R. (2009) How do you modernize a health service? A
1136
I. Riippa et al.
realist evaluation of whole-scale transformation in London. The Milbank Quarterly, 87 (2), 391–416. 26. International Diabetes Federation (2012) Global guideline for type 2 diabetes. Available at: http://www.idf.org/global-guideline-type-2diabetes-2012 (last accessed 5 May 2014). 27. Laakso, M., Groop, L., Isomaa, B., et al. (2013) Diabetes. Current care guideline (in Finnish). Finnish Medical Society. Available at: http:// www.kaypahoito.fi (last accessed 5 May 2014).
Supporting information Additional supporting information may be found in the online version of this article at the publisher’s web site.
© 2014 John Wiley & Sons, Ltd.
Copyright of Journal of Evaluation in Clinical Practice is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
Journal of Evaluation in Clinical Practice ISSN 1365-2753
Can complex health interventions be evaluated using routine clinical and administrative data? – a realist evaluation approach Iiris Riippa MSc (Tech) MSc (Econ),1 Olli-Pekka Kahilakoski MSc (Tech),2 Miika Linna DSc (Tech)3 and Minni Hietala MSc (Econ)1 1 Researcher, 2Research Assistant, 3Senior Researcher, The Institute of Healthcare Engineering and Management, Department of Industrial Engineering and Management, Aalto University, Espoo, Finland
Keywords CMO approach, complex intervention, primary health care, realist evaluation, routine data Correspondence Ms Iiris Riippa The Institute of Healthcare Engineering and Management Department of Industrial Engineering and Management Aalto University PO Box 15500 FI-00076 Aalto Finland E-mail: [email protected] Accepted for publication: 14 April 2014 doi:10.1111/jep.12175
Abstract Rationale, aims and objectives Interventions aimed at improving chronic care typically consist of multiple interconnected parts, all of which are essential to the effect of the intervention. Limited attention has been paid to the use of routine clinical and administrative data in the evolution of these complex interventions. The purpose of this study is to examine the feasibility of routinely collected data when evaluating complex interventions and to demonstrate how a theory-based, realist approach to evaluation may increase the feasibility of routine data. Methods We present a case study of evaluating a complex intervention, namely, the chronic care model (CCM), in Finnish primary health care. Issues typically faced when evaluating the effects of a complex intervention on health outcomes and resource use are identified by using routine data in a natural setting, and we apply context-mechanismoutcome (CMO) approach from the realist evaluation paradigm to improve the feasibility of using routine data in evaluating complex interventions. Results From an experimentalist approach that dominates the medical literature, routine data collected from a single centre offered a poor starting point for evaluating complex interventions. However, the CMO approach offered tools for identifying indicators needed to evaluate complex interventions. Conclusions Applying the CMO approach can aid in a typical evaluation setting encountered by primary care managers: one in which the intervention is complex, the primary data source is routinely collected clinical and administrative data from a single centre, and in which randomization of patients into two research arms is too resource consuming to arrange.
Introduction In chronic health care, few interventions are truly simple. They target several levels of the organization, because most improvements in care provision require changes on the care frontline and on the organization’s administrative levels. As the importance of the patient’s role in managing chronic conditions has become evident, elements of involving patients as active managers of their conditions are typically included in chronic care interventions [1]. This requires specific skills on the part of care professionals as well as patients. Furthermore, some customization of the intervention often is called for due to patients’ differing levels of skill and ability. A complex intervention is typically described as containing several interacting components. Although there is no sharp
boundary between complex and simple interventions, some characteristics are perceived as increasing the complexity of an intervention and should be considered when evaluating it. These characteristics are (i) number of interacting components within the intervention [2]; number and difficulty of behaviours required by those delivering or receiving the intervention; (iii) number of groups or organizational levels targeted by the intervention; (iv) number and variability of outcomes; and (v) degree of flexibility or tailoring of the intervention permitted [2]. The characteristics of complex interventions cause additional challenges for evaluators ‘in addition to the practical and methodological difficulties that any successful evaluation must overcome’ ([3], p. 6). The main challenges discussed in the literature are varying implementation of the intervention in different contexts and specification of relevant outcomes. Datta and Petticrew
Journal of Evaluation in Clinical Practice 20 (2014) 1129–1136 © 2014 John Wiley & Sons, Ltd.
1129
Complex intervention evaluation
[1] state that the design of an intervention should be based on theoretical understanding of its causal effects and, therefore, the evaluator should have a hypothesis about expected effects. However, ‘it may not be possible to predict which elements of an intervention will be acceptable to healthcare staff and patients, have the desired effect and be sustainable’ ([1], p. 12). Multiple contextual factors might form barriers to implementation and integration of an intervention into usual service. For comparison purposes, evaluators must be aware of possibly tailored elements of the intervention. Such awareness contributes to understanding what kind of intervention the results are attributable to. Oakley et al. [4] demonstrate how acknowledgement of differences in implementation of a sex education intervention added to the understanding of what kind of intervention is effective in which circumstances and for which target group. However, the UK Medical Research Council [3] notes that identifying ‘the active ingredient’ of an intervention may be impossible in some cases. In these cases, analysing effects of the intervention on intermediary outcomes might disclose new understanding of the intervention’s essence. In their study on national implementation of electronic health records (EHRs), Takian et al. [5] decide to forgo quantitative evaluation of final outcomes and, instead, used qualitative methods to acquire rich data on the process effects of EHR implementation to better understand what in the intervention causes potential effects on the health care system as a whole. Disclosure of the implemented intervention and its mechanism is closely related to specification of outcome measures. Definition of ‘the length and complexity of the causal chains linking intervention with [end] outcome’ ([3], p. 6) and verification of this mechanism through changes in intermediary outcomes are important to interpreting results. For example, Campbell et al. [6] state that in the case of a negative trial result, intermediate outcomes are important to identifying the point along the causal pathway at which the intervention failed. Similarly, in the case of a positive trial result, intermediary outcomes can verify that the effect is, indeed, attributable to the expected mechanism. Realist evaluation, first introduced in the social sciences [7], offers tools to evaluate complex interventions. The approach was introduced by Pawson and Tilley [7] to criticize the use of classic experimental approach to evaluation that dominates the toolkit of evidence-based medicine for evaluating interventions [8]. The main problem of traditional experimental approach, in the case of complex interventions, is the focus on outcomes, which do not explain why an intervention works; in other words, what active ingredients and context are required for the intervention to be effective. This treatment of an intervention as a black box has led to inconsistent findings that do not explain why the intervention failed or succeeded [7]. Elissen et al. [9] further explicate that ‘[a]nalyzing complex [disease management programs] necessitates a clear framework that links expected outcomes to the characteristics of both the program and its target population, and measures effects over an adequate period of time’. In accordance with suggested solutions to the challenges of evaluating complex interventions, Pawson and Tilley [7] present the contextmechanism-outcome (CMO) approach, which directs the evaluator’s attention to the context in which the intervention is implemented and the intervention’s mechanism in addition to the final outcomes. 1130
I. Riippa et al.
Although previous studies discuss evaluation of complex interventions, limited attention has been paid to the feasibility of using routinely collected clinical and administrative data in evaluation. The purpose of this study is to examine the feasibility of using such data in evaluating complex interventions. We present a case study of evaluating a complex intervention, namely, the chronic care model (CCM) [10], in Finnish primary care to demonstrate the feasibility of using routine data and to show how the data could be improved through the CMO approach. The case study consists of two parts. In the first part, routine data are used to compare changes in health outcomes and resource use between CCM patients and usual care patients. In the second part, interviews with health care professionals are conducted to examine retrospectively the implementation of the CCM in the case organization.
Methods CCM The CCM is built on the evidence of effective system changes that improve clinical [11,12] and patient-reported [13] health outcomes and reduce the cost of chronic care [14]. The CCM was refined by Wagner et al. [15] to counterbalance the emphasis on acute care by focusing on longitudinal and patient-centred care for the chronically ill [15]. The CCM suggests that chronic care should be organized based on six interrelated components [15,16]. Two of these refer mainly to the context in which chronic care is provided: (i) the health care organization, which encourages patient-centred care through, for example, incentive programmes; and (ii) links to community resources beyond the organization providing chronic care. The remaining four components are related to the actual delivery of care: (iii) self-management support, which assists patients to better manage their own chronic condition; (iv) delivery system design to provide coordinated, multidisciplinary team care to the patient; (v) decision support for providers to assure the expertise and memory aids to routinely provide evidence-based care; and (vi) clinical information systems, which facilitate collecting, organizing and accessing relevant patient information [15–17].
Routine data analysis of CCM effects on care outcomes Routinely collected clinical and administrative data from 2007 to 2011 on all patients with type 2 diabetes treated in the city of Espoo (population c. 250 000) were used to evaluate the effects of the CCM on health outcomes and resource use. Effects of the intervention were examined by comparing two groups of patients with type 2 diabetes: a case group, which received care according to the CCM, and a control group, which received usual care. The following describes how the case and control groups and the change in health outcomes and resource use were identified in routine clinical and administrative data. The Results section discusses feasibility of using routine data. A complete report on the analysis, following STROBE guidelines for reporting observational studies, is available from the authors. Case and control groups The study cohort, 9278 patients with type 2 diabetes, was identified by patients’ recorded diagnoses (ICD-10 diagnoses in blocks
© 2014 John Wiley & Sons, Ltd.
I. Riippa et al.
E11 or E13) or visit-related information (ICPC codes T90 or T88). Among the cohort, patients exposed to the CCM and, thus, belonging to the intervention group were identified by the presence of a care plan in their visit-related records. Patients recruited to the CCM in Espoo drew up care plans jointly with their care professional to support their self-management of health. The care plan stated the main care goals based on the patient’s main healthrelated problems. The plan was saved in the EHR and could be shared with the entire care team. Therefore, care plans implemented three CCM components: self-management support, clinical information systems and a delivery system to provide coordinated, multidisciplinary team care to the patient. In addition to the care plan, the CCM consisted of counselling the patient and annual follow-up visits. The control group consisted of patients with type 2 diabetes in Espoo who had no care plans in their records and who, thus, received usual care. In the usual chronic care in Espoo, each health care professional was assumed to generally follow the care guidelines established by the Finnish Medical Society Duodecim. As is typical in retrospective, natural settings, there were some differences in descriptive characteristics between the intervention and control groups. After exclusion of patients who had moved away or died before January 2012, the intervention group included 6672 patients, and the control group 2606 patients. The intervention group consisted of older patients than those in the control group, and it had a larger proportion of men. In addition, more patients in the intervention group had co-morbid cardiovascular disease. Before enrolment in the CCM, during November 2006– November 2007, the control group averaged 1.5 mmol mol−1 lower levels of HbA1c than the intervention group and had visited a nurse 0.6 times more often, on average (Supporting Information Appendix S1). Outcome measures and controlled confounding factors Primary outcomes were HbA1c and resource use, operationalized as the number of doctor and nurse visits for each patient. In addition, routinely gathered data allowed us to control for the following confounding factors: age, gender, co-morbid cardiovascular disease, onset of CCM and health centre that the patient used most. In the case organization, another intervention, personal health record, was implemented during the study period. This intervention was also included in the controlled confounding factors. Statistical methods Because the data were not generated in a controlled research setting, statistical modelling was used to control patient heterogeneity. A Bayesian linear mixed model was used to assess CCMrelated changes in HbA1c levels, and Bayesian generalized linear mixed models were used to analyse nurse and doctor use. Observation-level variables included in the models were age, gender, co-morbid cardiovascular disease, time of enrolment in CCM, patient group, observation year, health centre that the patient used most and concomitant personal health record intervention. The models also included patient-level varying intercepts. Generalized, linear mixed models used Poisson-lognormal distri-
© 2014 John Wiley & Sons, Ltd.
Complex intervention evaluation
butions to account for overdispersion. Uniform priors were used for model parameters. Effect of the CCM was computed as the difference in predictions of the model between the two values of the CCM indicator variable, averaged over the case group. The effect was summarized on the scale of the response variable. To assess the sensitivity of the results on model selection, we also conducted the analysis without adjusting for patient- or observation-specific variables. Baseline characteristics were compared between groups using Bayesian statistics: for continuous variables, normal models with reference priors were used; for proportions, binomial models with dependent Howard priors were used [18]. Missing values in HbA1c measurements were assumed to be missing completely at random and were therefore deleted listwise. We did not remove any measurements as outliers. There were no missing values in the resource use data. Data processing and statistical computation were done using R version 2.15.1 [19]. The models for HbA1c and resource use were fitted using the rstan package [20]. Baseline characteristics for proportions were compared using the LearnBayes package [21].
Implementation of CCM in the case organization As suggested by the literature on complex interventions, we examined how the CCM was implemented in the case organization. We interviewed five physicians and six nurses working in 5 of the 11 health centres in the city of Espoo. Interviews were based on the Assessment of Chronic Illness Care (ACIC) questionnaire, which is a tool for evaluating the realization of the six dimensions of the CCM in a health care organization [22]. Interviews were recorded, and a content analysis was conducted. The case organization had designed and customized the CCM to meet its needs during 2006–2007, and had then started to implement it. Issues mentioned by interviewees reflect the status quo of the CCM at the time of the interviews (June 2012). To differentiate between the status quo and the initial plan for implementation, we examined interview findings in relation to the case organization’s documents on the initial CCM implementation plan. A full report on the implementation inquiry is available from the authors.
Results Routine data analysis of CCM effects on care outcomes During the first year after enrolment in the CCM, the difference in nurse visits due to the CCM was 1.8 visits per year (1.7–1.9) and in doctor visits 0.7 visits per year (0.5–0.7). The differences indicate that the CCM increased both nurse and doctor use. During the following 3 years, the increase in average number of nurse visits diminished to 0.5 visits per year (0.4–0.7) in the second year, to 0.4 visits per year (0.2–0.5) in the third year and to 0.3 visits per year (0.0–0.5) in the fourth year. However, the number of doctor visits remained at the baseline level after the first year (Supporting Information Appendices S2 and S3). During the first year in the CCM, the estimated mean effect of the CCM on HbA1c was −1.4 mmol mol−1 (95% Bayesian confidence interval: −1.7 to −1.2; a negative difference indicates that the 1131
Complex intervention evaluation
intervention was effective). The change was sustained, approximately, during the following 3 years: the average effect of the CCM on HbA1c during the first 4 years was −1.3 mmol mol−1 (−1.6 to −1.0) (Supporting Information Appendices S2 and S3). To evaluate the effect of confounding factors, we conducted the analysis without controlling for them. Non-adjusted analysis showed similar results, although the effects of the CCM on both resource use and HbA1c were smaller than in our primary analysis (Supporting Information Appendix S4). Routine data analysis showed that the CCM decreased patients’ HbA1c levels by an average of 1.4 mmol mol−1 (1.2–1.7) during the first year in the care model. This decrease was gained through a small increase in resource use: an increase of 1.8 nurse visits (1.7–1.9) and 0.7 doctor visits (0.6–0.7) during the first year in the CCM. This implies that the CCM model may be moderately cost-effective.
Implementation of CCM in the case organization Interview data indicated that although implementation was executed successfully, possible risks noted during the design stage were realized after implementation. The following describes findings of this qualitative inquiry, divided into the six dimensions of the ACIC questionnaire. 1 The health care organization had supported patient-centred care when it introduced the CCM, but later, support was perceived to have diminished. The aim that the CCM would become a permanent and continuously evolving practice was not fulfilled. In addition, interviewees perceived support as varying among areas, and they mentioned management and leadership issues as root causes. Furthermore, cost pressures and personnel shortages were perceived as taking precedence over CCM care issues. 2 Both at the time of implementation and during interviews, professionals did not recognize many links to community sources beyond their organization. Use of private or third-party services depended on the individual physician or, more importantly, the nurse. 3 The self-management support, which was considered a key issue in the implementation phase, had not become a working practice in the organization. For instance, although the care plan was considered comprehensive, a copy of it was not always given to the patient. In addition, the care plan did not necessarily include goals or activities for the patient to follow. 4 Design of the delivery system was not believed to fully support providing patients with coordinated, multidisciplinary team care. During implementation, increased emphasis was placed on collaboration among various health care professionals. At the time of the interviews, CCM care was mainly the responsibility of nurses, who consulted doctors when medically necessary. 5 Decision support for providers to ensure provision of evidence-based care was perceived as insufficient. During implementation, decision support took the form of documents, lectures and training. Interviewees mentioned that support ceased after implementation. 6 Some interviewees thought clinical information systems were helpful in processing relevant patient information, whereas others thought relevant information was difficult to find. Information 1132
I. Riippa et al.
systems had remained the same from implementation to the time interviews were conducted. On the bases of interviewee perceptions and documents on CCM implementation received from the case organization, two main findings on the success of CCM implementation could be made. The first was that despite initial comprehensive plans and instructions for CCM adoption, systematic guidelines for the model’s long-term use and development in daily care were perceived as insufficient. The second was that a complex intervention presents a multitude of issues to govern, and without continuous support and management, outcomes of an intervention may be short lived.
Weaknesses of the routine data analysis of CCM effects on care outcomes Although according to analysis of routine data, the CCM had a positive, albeit clinically insignificant, effect on health outcomes through small increases in resource use, the results must be interpreted in the light of three main weaknesses of the analysis. First, due to the natural setting of the study, participants were not assigned randomly to the case and control groups. Because the factors that affect recruitment and willingness to participate are not known, let alone measured, they could not be controlled for in the model to account for the selection bias. Therefore, results might be attributable to specific characteristics of those patients who ended up in the intervention group, not to the intervention itself. Second, the exposure indicator used does not accurately separate patients exposed to the intervention from those unexposed. The care plan was the best indicator available in routine administrative data for identifying patients exposed to the intervention. However, the care plan can be interpreted to detect exposure to only three of six components of the CCM. Furthermore, it is plausible that the control group had been exposed to elements of the intervention that cannot be delivered to a patient personally but that are delivered to the entire population served, such as patientcentred organization of health care. Therefore, the results cannot be interpreted as deriving from the entire CCM but, rather, from some components of it. Third, proper implementation of the intervention could not be detected in routinely gathered clinical and administrative data. Interview findings suggest that there were some deficiencies in implementation of the CCM. Again, this raises the question of whether the intervention caused the results. The first two issues, selection bias and inaccuracy of the exposure indicator, cannot be fully overcome in single-centre evaluation of a complex intervention. In such a case, evidence on the causal chain that is expected to lead to the change in outcome is particularly important. In contrast to the rather experimentalist approach applied in the first part of the case study, in the following, we show how the CMO approach from realist evaluation paradigm could aid in evaluating complex interventions using routine data.
Application of CMO approach to identify the data needs for intervention evaluation The proposition that ‘causal outcomes follow from mechanisms acting in contexts . . . is the axiomatic base upon which all realist
© 2014 John Wiley & Sons, Ltd.
I. Riippa et al.
Complex intervention evaluation
explanation builds’([6], p. 58). Pawson and Tilley [7] contrast the CMO approach with the classic experimentalist approach, OXO, which stands for ‘observe a system (O), introduce a perturbation (X) to some participants but not others, and then observe again (O)’ ([7], p. 1183). Pawson and Tilley [7] criticize OXO for its focus on outcomes at the expense of considering the mechanism and context of the intervention. In addition to changes in outcomes, the CMO approach pays attention to the mechanism that causes the effect and the context in which the effect can be observed. This approach adds to the understanding of how outcomes can be managed. To form a CMO configuration of the CCM, we start by identifying the central effect mechanisms of the model and explain how the six components of the model trigger these mechanisms. The eventual configuration with context and outcome measures that are (i) relevant in terms of the suggested mechanisms and (ii) suitable for routine data collection are presented in Table 1. The configuration is not an exhaustive list of indicators for evaluating the whole chain of causes and effects of the CCM. Rather, it attempts to show how the configuration can point out essential evaluation aspects of a complex intervention. According to Wagner and colleagues [15], the components of CCM represent ‘the enhancements to the organization and its practices that contribute to productive interactions between providers and patients’. They specify that productive interactions ‘consistently provide the assessments, support for selfmanagement, optimization of therapy, and follow-up associated with good [disease control] outcomes’. Two distinct mechanisms triggered by CCM components can thus be identified: improved clinical care and self-management support to the patient. Different components of the CCM are intertwined and some of them are more directly linked to the before mentioned mechanisms than others. Health care organization enables changes in other elements of CCM through ‘strong leadership, appropriate incentives, and effective improvement strategies’ [15]. Self-management support is listed as an independent component in CCM literature. It helps patients to ‘set limited goals for improving their management of their illness, identify barriers to reaching their goals, and develop a plan to overcome the barriers’ [15]. The four other
components are essential in the provision of both clinical care and self-management support: links to community resources complement the needed clinical care and self-management support provided by the organization; delivery system design enables the productive interactions and thus constitutes the platform for the clinical care and self-management support; decision support assures that the care teams ‘have the expertise to provide appropriate clinical and behavioral management’ [15]; and finally, clinical information systems allow the professionals to reply to the specific needs of individual patients and enable the resource allocation and service provision planning to effectively meet the needs of the population served [15]. In the case of CCM, relevant context factors that contribute to or hinder the effects of the intervention services are characteristics of the patient and the caregiver. Such measurable patient characteristics are, for example, willingness to participate and current state of clinically measured health. One caregiver characteristic that can affect the intervention’s effect on patients is the recruitment rate of the health care professional or health centre that recruited the patient to the intervention. Furthermore, because a higher number of care professionals treating the patient can affect continuity of care, the number of professionals taking care of the patient could be included as a context measure. As suggested by Campbell et al. [6] in their study on complex intervention evaluation, we gathered intermediary and final outcomes to better grasp the intervention’s causal effects. Here, process outcomes present changes in the care delivered, which, in turn, are expected to result in improvements in clinical care, including, for example, adjustments in medications and procedures to follow care guidelines and provision of better self-management support. Although improvements in both clinical care and self-management support are expected to have a positive effect on clinical outcomes, the time lag in these effects is plausibly different. Improved clinical care can be expected to affect clinical health indicators in the short term, whereas the effects of self-care support can be expected to lag, as psychosocial changes and/or changes in knowledge base of the patient are needed first to trigger lifestyle changes, which, in turn, can affect the patient’s
Table 1 Context-mechanism-outcome configuration of CCM Context
Mechanism
Outcome
Patient characteristics: • Willingness to participate (yes/no) • State of clinical health (HbA1c)
Improvements in clinical care and self-management support triggered by enhancements in the six CCM components: 1 Health care organization 2 Links to community resources 3 Self-management support 4 Delivery system design 5 Decision support 6 Clinical information systems
Process outcomes: • Assessment of Chronic Illness Care • Availability of clinical health indicators
Provider characteristics: • Recruitment rate of the caregiver (doctor, nurse, health centre) • Number of caregivers (nurses, doctors, health centres)
Clinical health outcomes: • HbA1c • LDL Self-management outcomes: • Knowledge, skills and confidence in managing one’s health (e.g. Patient Activation Measure) • Health behaviour (e.g. physical activity, diet, smoking) Resource use outcomes: • Nurse visits • Doctor visits
CCM, chronic care model; LDL, low-density lipoprotein.
© 2014 John Wiley & Sons, Ltd.
1133
Complex intervention evaluation
Patient characteristics Provider characteristics
Process outcomes Clinical health outcomes Self-management outcomes
Resource use outcome
I. Riippa et al.
Measure
Availability in routine data
Willingness to participate (yes/no) State of clinical health (HbA1c) Recruitment rate of the caregiver (doctor, nurse, health centre) Number of caregivers (nurses, doctors, health centres) Assessment of Chronic Illness Care Availability of clinical health outcomes HbA1c LDL Knowledge, skills and confidence in managing one’s health (e.g. Patient Activation Measure) Health behaviour Nurse visits Doctor visits
No Yes No
Table 2 Feasibility of using routine data in the light of CMO approach
Yes No Yes Yes Yes, but not recorded sufficiently Self-care ability and barriers to self-care were in use, but not recorded adequately Yes, but not recorded adequately Yes Yes
CMO, context-mechanism-outcome; LDL, low-density lipoprotein.
clinical health status. In order to capture the intervention’s effect on clinical care and self-care support, outcomes for both parts are needed. Two relevant clinical health outcomes in type 2 diabetes management are HbA1c and low-density lipoprotein. In short-term evaluation of self-care support outcomes, changes in a patient’s psychosocial state and health behaviour are informative. One example of a validated psychosocial instrument for measuring a patient’s knowledge, skills and confidence in managing his or her health is the Patient Activation Measure (see, e.g. [23,24]). Information on a patient’s health behaviour can be collected with questionnaires on physical activity, diet, smoking and other aspects relevant to managing specific conditions. As resource constraints are a reality in all health care provision, care outcomes of an intervention always should be considered in relation to possible changes in resource consumption due to the intervention. Applicable measures for patient-specific resource use are visits with doctors and nurses. Process outcomes suitable for CCM evaluation are the ACIC survey, through which professionals’ perceptions of CCM realization can be evaluated, and the availability of relevant clinical health indicators. Table 2 presents availability of various evaluation measures in the routine data of the case organization. Routine data proved more feasible to use in evaluating changes in clinical care provision than in self-management support. One process measure for clinical care, the availability of relevant clinical indicators, could be traced in the frequency of specific laboratory tests. In addition, for type 2 diabetics, HbA1c serves as an appropriate clinical health outcome, and its levels were recorded adequately. However, changes in self-management support provision or self-management outcomes could not be evaluated with routine data. The case organization used measures for self-care ability and barriers to it, but especially the former was rarely recorded. Furthermore, of relevant context indicators, only a patient’s clinical health and the number of caregivers were available. Two other context indicators relevant to self-management success were unavailable: (i) patient’s willingness to participate in the intervention and (ii) the recruitment rate of the caregiver who recruited the patient. Nurse and doctor visits were recorded systematically in routine data and are sufficient measures of resource outcome in CCM 1134
evaluation. In the case organization, no indicators for realization of the intervention mechanisms (e.g. ACIC) were available in the routine data. Therefore, routine data analysis cannot detect whether changes in outcome indicators were attributable to the CCM or to some other concomitant changes in the organization (Table 2).
Discussion In this study, we discussed a typical evaluation setting encountered by primary care managers, one in which the intervention is complex, the primary data source is routinely collected clinical and administrative data, and randomization of patients into two research arms is too resource consuming to arrange. Through a case study on evaluating complex intervention, we examined the feasibility of using routine data in such an evaluation setting. The findings of the study showed that the classic experimentalist approach yields few tools for evaluating complex interventions in natural settings, in which context and implementation of intervention mechanisms often vary. To suggest an alternative to the experimentalist approach, we demonstrated how the CMO approach from the realist evaluation paradigm could aid in identifying measures needed for evaluation. The CMO configuration of the care model revealed deficiencies in the context and outcome indicators available in routine data of the care organization. First, the way in which the CCM was implemented in the case organization could not be detected in the routine data. Second, an essential effect of care model mechanisms, namely, the effect on self-management support provision and final self-management outcomes, could not be traced in routine data. Furthermore, context measures relevant to self-management success, such as a patient’s willingness to participate in the intervention and the recruitment rate of the specific caregiver who recruited that patient, were not available. As far as we are aware, this is the first study to apply CMO approach in examining routine data feasibility in evaluating complex care interventions. Our demonstration shows that with the CMO approach, collection of routine data could be designed to meet the needs of complex intervention evaluation in a natural
© 2014 John Wiley & Sons, Ltd.
I. Riippa et al.
setting more adequately. In practice, there is often a trade-off between efficiency of data collection and coverage of the causal chain of effects through measurement. Therefore, the appropriate level of measurement coverage should be considered when evaluating a specific intervention, along with which measures could be added to routine data collection. In this study, we focused on utilizing and examining the usability of routine data in complex intervention evaluation. More advanced methods for complex intervention evaluation are available when the study objective allows collection of data beyond that routinely gathered (see, e.g. [25]). This study indicated that gaps in routine data collection identified using CMO approach might prove relevant not only in terms of the specific intervention that is addressed, but in terms of the care organization’s performance in a broader perspective. Global [26] and national care guidelines [27] mention a patient’s active participation as an essential part of type 2 diabetes care. Yet, in this case study, the CMO configuration revealed a lack of indicators in the case organization for self-care support and self-care outcomes. This might suggest needs for changes in the collection of routine data for managerial purposes. In the case organization, weaknesses in routine data originated from two issues: either a measure for a certain outcome or a context characteristic was not defined in the organization or, if defined, it was not recorded adequately. When the decision is made to routinely record a measure, the system should support that using the gathered data in decision making, which, in turn, further encourages individual professionals to continue recording the measure.
Conflict of interest The authors declare no conflict of interest. The authors alone are responsible for the content and writing of the paper.
Acknowledgements The authors wish to thank medical doctors Tuomo Lehtovuori and Osmo Saarelma from Espoo primary care for their valuable comments on the chronic care model in Espoo and Tieto Corporation for collection of the data. In addition, DSc (Tech) Karita Reijonsaari from Aalto University deserves special recognition for assisting in planning the study. The study was (partly) supported by the SalWe Research Program for IMO (Tekes – the Finnish Funding Agency for Technology and Innovation grant 648/10).
References 1. Datta, J. & Petticrew, M. (2013) Challenges to evaluating complex interventions: a content analysis of published papers. BMC Public Health, 13 (1), 568. 2. Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I. & Petticrew, M. (2008) Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ (Clinical Research Ed.), 337, 979. Available at: http://www.ncbi.nlm.nih.gov/ pmc/articles/PMC2769032/ (last accessed 10 April 2013). 3. Medical Research Council. (2008) Complex interventions guidance. Available at: http://www.mrc.ac.uk/Utilities/Documentrecord/index .htm?d=MRC004871 (last accessed 12 September 2013).
© 2014 John Wiley & Sons, Ltd.
Complex intervention evaluation
4. Oakley, A., Strange, V., Bonell, C., Allen, E. & Stephenson, J. (2006) Health services research: process evaluation in randomised controlled trials of complex interventions. BMJ (Clinical Research Ed.), 332 (7538), 413. 5. Takian, A., Petrakaki, D., Cornford, T., Sheikh, A., Barber, N. & National NHS Care Records Service Evaluation Team (2012) Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems. BMC Health Services Research, 12 (1), 105. 6. Campbell, N. C., Murray, E., Darbyshire, J., et al. (2007) Designing and evaluating complex interventions to improve health care. BMJ (Clinical Research Ed.), 334 (7591), 455–459. 7. Pawson, R. & Tilley, N. (1997) Realistic evaluation. Sage. Available at: http://www.google.com/books?hl=fi&lr=&id=GXagvZwC2bcC& oi=fnd&pg=PR9&dq=realist+evaluation+pawson&ots=ZQLpVA0rCx &sig=VijUxnuGyw2ZaFRqZ3sXgB0anFQ (last accessed 30 August 2013). 8. Berwick, D. M. (2008) The science of improvement. JAMA: The Journal of the American Medical Association, 299 (10), 1182–1184. 9. Elissen, A. M. J., Duimel-Peeters, I. G. P., Spreeuwenberg, C., Spreeuwenberg, M. & Vrijhoef, H. J. M. (2012) Toward tailored disease management for type 2 diabetes. The American Journal of Managed Care, 18 (10), 619–630. 10. Wagner, E. H., Austin, B. T. & Von Korff, M. (1996) Organizing care for patients with chronic illness. The Milbank Quarterly, 74 (4), 511–544. 11. Bodenheimer, T., Wagner, E. H. & Grumbach, K. (2002) Improving primary care for patients with chronic illness. JAMA: The Journal of the American Medical Association, 288 (15), 1909–1914. 12. Musacchio, N., Lovagnini Scher, A., Giancaterini, A., Pessina, L., Salis, G., Schivalocchi, F., Nicolucci, A., Pellegrini, F. & Rossi, M. C. E. (2011) Impact of a chronic care model based on patient empowerment on the management of type 2 diabetes: effects of the SINERGIA programme. Diabetic Medicine: A Journal of the British Diabetic Association, 28 (6), 724–730. 13. Battersby, M., Von Korff, M., Schaefer, J., Davis, C., Ludman, E., Greene, S. M., Parkerton, M. & Wagner, E. H. (2010) Twelve evidence-based principles for implementing self-management support in primary care. Joint Commission Journal on Quality and Patient Safety, 36 (12), 561–570. 14. Bodenheimer, T., Lorig, K., Holman, H. & Grumbach, K. (2002) Patient self-management of chronic disease in primary care. JAMA: The Journal of the American Medical Association, 288 (19), 2469–2475. 15. Wagner, E. H., Austin, B. T., Davis, C., Hindmarsh, M., Schaefer, J. & Bonomi, A. (2001) Improving chronic illness care: translating evidence into action. Health Affairs, 20 (6), 64–78. 16. Wagner, E. H., Bennett, S. M., Austin, B. T., Greene, S. M., Schaefer, J. K. & Vonkorff, M. (2005) Finding common ground: patientcenteredness and evidence-based chronic illness care. Journal of Alternative and Complementary Medicine, 11 (Suppl. 1), S7–S15. 17. De Bruin, S. R., Heijink, R., Lemmens, L. C., Struijs, J. N. & Baan, C. A. (2011) Impact of disease management programs on healthcare expenditures for patients with diabetes, depression, heart failure or chronic obstructive pulmonary disease: a systematic review of the literature. Health Policy, 101 (2), 105–121. 18. Howard, J. V. (1998) The 2 × 2 table: a discussion from a Bayesian viewpoint. Statistical Science, 13 (4), 351–367. 19. R Development Core Team (2005) The R Project for statistical computing. Available at: http://www.r-project.org/ (last accessed 12 September 2013). 20. Stan Development Team (2014) Stan: project home. Available at: http://mc-stan.org/ (last accessed 12 September 2013). 21. Albert, J. (2012) Package LearnBayes. Available at: http://cran.rproject.org/web/packages/LearnBayes/index.html (last accessed 12 September 2013).
1135
Complex intervention evaluation
22. Bonomi, A. E., Wagner, E. H., Glasgow, R. E. & VonKorff, M. (2002) Assessment of chronic illness care (ACIC): a practical tool to measure quality improvement. Health Services Research, 37 (3), 791– 820. 23. Hibbard, J. H., Mahoney, E. R., Stockard, J. & Tusler, M. (2005) Development and testing of a short form of the patient activation measure. Health Services Research, 40 (6 Pt 1), 1918–1930. 24. Fowles, J. B., Terry, P., Xi, M., Hibbard, J., Bloom, C. T. & Harvey, L. (2009) Measuring self-management of patients’ and employees’ health: further validation of the Patient Activation Measure (PAM) based on its relation to employee characteristics. Patient Education and Counseling, 77 (1), 116–122. 25. Greenhalgh, T., Humphrey, C., Hughes, J., Macfarlane, F., Butler, C. & Pawson, R. (2009) How do you modernize a health service? A
1136
I. Riippa et al.
realist evaluation of whole-scale transformation in London. The Milbank Quarterly, 87 (2), 391–416. 26. International Diabetes Federation (2012) Global guideline for type 2 diabetes. Available at: http://www.idf.org/global-guideline-type-2diabetes-2012 (last accessed 5 May 2014). 27. Laakso, M., Groop, L., Isomaa, B., et al. (2013) Diabetes. Current care guideline (in Finnish). Finnish Medical Society. Available at: http:// www.kaypahoito.fi (last accessed 5 May 2014).
Supporting information Additional supporting information may be found in the online version of this article at the publisher’s web site.
© 2014 John Wiley & Sons, Ltd.
Copyright of Journal of Evaluation in Clinical Practice is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
