577445

research-article2015

AOPXXX10.1177/1060028015577445Annals of PharmacotherapyFernandes et al

Research Report

Development of Clinical Pharmacy Key Performance Indicators for Hospital Pharmacists Using a Modified Delphi Approach

Annals of Pharmacotherapy 1­–14 © The Author(s) 2015 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1060028015577445 aop.sagepub.com

Olavo Fernandes, PharmD1,2, Sean K. Gorman, PharmD3,4, Richard S. Slavik, PharmD3,4, William M. Semchuk, PharmD5,6, Steve Shalansky, PharmD4,7, Jean-François Bussières, MSc8,9, Douglas Doucette, PharmD10,11, Heather Bannerman, PharmD12, Jennifer Lo, PharmD13, Simone Shukla, PharmD14, Winnie W. Y. Chan, PharmD15, Natalie Benninger, PharmD16, Neil J. MacKinnon, PhD17, Chaim M. Bell, MD2,18,19, Jeremy Slobodan20, Catherine Lyder, MHSA21, Peter J. Zed, PharmD4, and Kent Toombs22

Abstract Background: Key performance indicators (KPIs) are quantifiable measures of quality. There are no published, systematically derived clinical pharmacy KPIs (cpKPIs). Objective: A group of hospital pharmacists aimed to develop national cpKPIs to advance clinical pharmacy practice and improve patient care. Methods: A cpKPI working group established a cpKPI definition, 8 evidence-derived cpKPI critical activity areas, 26 candidate cpKPIs, and 11 cpKPI ideal attributes in addition to 1 overall consensus criterion. Twenty-six clinical pharmacists and hospital pharmacy leaders participated in an internetbased 3-round modified Delphi survey. Panelists rated 26 candidate cpKPIs using 11 cpKPI ideal attributes and 1 overall consensus criterion on a 9-point Likert scale. A meeting was facilitated between rounds 2 and 3 to debate the merits and wording of candidate cpKPIs. Consensus was reached if 75% or more of panelists assigned a score of 7 to 9 on the consensus criterion during the third Delphi round. Results: All panelists completed the 3 Delphi rounds, and 25/26 (96%) attended the meeting. Eight candidate cpKPIs met the consensus definition: (1) performing admission medication reconciliation (including best-possible medication history), (2) participating in interprofessional patient care rounds, (3) completing pharmaceutical care plans, (4) resolving drug therapy problems, (5) providing in-person disease and medication education to patients, (6) providing discharge patient medication education, (7) performing discharge medication reconciliation, and (8) providing bundled, proactive direct patient care activities. Conclusions: A Delphi panel of hospital pharmacists was successful in determining 8 consensus cpKPIs. Measurement and assessment of these cpKPIs will serve to advance clinical pharmacy practice and improve patient care. Keywords clinical pharmacy, key performance indicators, process-of-care indicators, Delphi consensus

Introduction Health care–related key performance indicators (KPIs) are quantifiable measures of quality used to track an organization’s progress with specific, essential processes and outcomes.1 They serve to inform policy development, improve quality of care provided, help ensure accountability to protect public safety, and inform consumers, so that they can make an educated decision on their choice of health care provider.2-5 Health care quality is defined as “the degree to which health services for individuals and populations

increase the likelihood of desired health outcomes and care consistent with current professional knowledge.”6 Quality assessment of professional practice predominantly involves measurement of processes by which health care is delivered or outcomes achieved by health care professionals’ activities.5 Currently, there is no international consensus on clinical pharmacy KPIs (cpKPIs). Although indicators for hospital pharmacy services have been available since the 1980s, they focus on workload and distribution measures and do not measure the quality of clinical pharmacy service delivery.7 Clinical pharmacy is defined as “a health science

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

2

Annals of Pharmacotherapy 

discipline in which pharmacists provide patient care that optimizes medication therapy and promotes health, wellness, and disease prevention” and “embraces the philosophy of pharmaceutical care.”8(p816) Consequently, there is a need to bridge this indicator gap as pharmacy practice has evolved over the decades from a drug and drug distribution focus toward a patient-centered focus. Indeed, there is a growing body of compelling pharmacy practice literature (evidence from randomized controlled trials [RCT], systematic reviews, meta-analyses, and observational studies) demonstrating the impact of specific clinical pharmacy patient processes of care on meaningful patient outcomes such as hospital readmissions and mortality.9-14 This recent published evidence assessing the impact of pharmacists partnering with patients and interprofessional teams can enable the selection of appropriate indicators, which was not possible to the same degree in the past. If successfully implemented, standardized cpKPIs could be used to identify significant achievements and deficiencies in clinical pharmacy care, justify the allocation of scarce resources, and facilitate continuous improvement in the quality of clinical pharmacy practice. The aim of this study was to develop cpKPIs using a systematic, evidenceinformed, national consensus-building process.

process commonly used to develop consensus health care quality indicators.15,16 It is a multiple iteration survey technique that enables anonymous, systematic refinement of expert opinion to arrive at a combined or consensual opinion in a short period of time.17 A Delphi process can be modified to incorporate discussion and live meetings.15 Three rounds of Delphi surveys were performed using an Internet-based survey (SurveyMonkey Inc, Palo Alto, CA; http://www.surveymonkey.com). An in-person panelist meeting was held between rounds 2 and 3, with the aim of enabling discussion of candidate cpKPIs and to clarify questions with respect to wording and definitions of the candidate indicators. Panelists were encouraged to suggest new candidate cpKPIs during round 1 and to provide feedback on the wording of the candidate cpKPIs during rounds 1 and 2. Prior to the subsequent Delphi round, panelists received their individual ratings for each candidate indicator for the previous round in addition to the overall group ratings and anonymized panelists’ written feedback for each candidate indicator. All candidate cpKPIs were carried forward through all 3 Delphi rounds, and consensus was determined only after the third Delphi round. This study was conducted in accordance with the ethical standards of the University Health Network’s research ethics board.

Methods Study Design

cpKPI National Collaborative and Working Group

Consensus cpKPIs were determined using a modified Delphi technique. A Delphi technique is a structured

Prior to initiating the modified Delphi survey, a national collaborative of hospital pharmacists created a working

1

University Health Network Pharmacy Department, Toronto, ON, Canada University of Toronto, Toronto, ON, Canada 3 Interior Health Pharmacy Services, Kelowna, BC, Canada 4 The University of British Columbia, Vancouver, BC, Canada 5 Regina Qu’Appelle Health Region Pharmacy Services, Regina, SK, Canada 6 University of Saskatchewan, Saskatoon, SK, Canada 7 Lower Mainland Pharmacy Services, Providence Healthcare, Vancouver, BC, Canada 8 Département de pharmacie et unité de recherche en pratique pharmaceutique, CHU Sainte-Justine, Montréal, QC, Canada 9 Université de Montréal, QC, Canada 10 Horizon Health Network Pharmacy Services, Moncton, NB, Canada 11 Dalhousie University, Halifax, NS, Canada 12 McMaster University, Hamilton, ON, Canada 13 Sunnybrook Health Sciences Centre Pharmacy Department, Toronto, ON, Canada 14 Foothills Medical Centre Pharmacy Department, Calgary, AB, Canada 15 St Michael’s Hospital Pharmacy Department, Toronto, ON, Canada 16 University Health Network—Toronto Rehabilitation Institute Pharmacy Department, Toronto, ON, Canada 17 James L Winkle College of Pharmacy, University of Cincinnati, Cincinnati, OH, USA 18 Mount Sinai Hospital, Toronto, ON, Canada 19 Institute for Clinical Evaluative Sciences (ICES), Toronto, ON, Canada 20 Alberta Health Services Pharmacy Services, Red Deer, AB, Canada 21 Canadian Society of Hospital Pharmacists, Ottawa, ON, Canada 22 Capital District Health Authority Pharmacy Department, Halifax, NS, Canada 2

Corresponding Author: Sean K. Gorman, Regional Coordinator, Clinical Quality & Research, Interior Health Pharmacy Services, No. 200-1835 Gordon Dr, Kelowna, BC V1Y3H5, Canada. Email: [email protected]

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

3

Fernandes et al group tasked to systematically develop cpKPIs. The cpKPI working group consisted of frontline clinical pharmacists and hospital pharmacy leaders from across Canada. The working group created the tools necessary to conduct a modified Delphi survey to determine consensus cpKPIs. These tools included an inventory of candidate cpKPIs, cpKPI definition, cpKPI selection criteria, evidence tables, and identification of clinical pharmacy evidence-informed thematic critical activity areas.

Candidate cpKPI Inventory Three investigators (OF, WWYC, NB) conducted a comprehensive literature search and review that focused on clinical pharmacy studies reporting process and outcome measures. Searches of MEDLINE and EMBASE (inception to July 2012) were performed using the keywords clinical pharmacy, performance, key performance indicator, standard, benchmark, metric system, and metrics. To account for existing national hospital practices, the working group held monthly meetings during which representatives from different parts of the country shared their perspectives on hospital pharmacy practice models and existing clinical pharmacy metrics. Canadian Society of Hospital Pharmacists (CSHP) 201518 and Canadian Blueprint for Pharmacy19 initiatives were also reviewed to extract potential candidate cpKPIs. National input and suggested candidate cpKPIs from CSHP members, frontline hospital staff, and leaders in pharmacy practice were gathered through facilitated workshops at regional conferences, local hospital pharmacy staff feedback sessions, and e-mail announcements to CSHP members. Based on the literature review and hospital pharmacy engagement process, the working group created a list of 137 crude candidate clinical pharmacy metrics (to be evaluated for consideration as cpKPIs).

cpKPI Definition A 5-point cpKPI definition was established to guide refinement of the inventory of candidate cpKPIs. The working group members agreed that all cpKPIs should (1) reflect a desired quality practice, (2) link to direct patient care, (3) have evidence supporting an impact on meaningful patient outcomes, (4) be pharmacy or pharmacist sensitive, and (5) be feasible to measure.

cpKPI Selection Criteria Once a definition was established, cpKPI selection criteria were developed to reflect the ideal attributes of a cpKPI. Three working group members (NB, OF, RSS) created a list of 11 cpKPI ideal attribute criteria that was informed

by 4 quality indicator parameters of the Agency for Healthcare Research and Quality (importance; scientific soundness: clinical logic; scientific soundness: measure properties; and feasibility)20 and the 5 characteristics of a cpKPI definition. The resulting 11 cpKPI selection criteria stated that an indicator is (1) supported by high-quality evidence; (2) associated with a relevant impact on clinically important outcomes; (3) a reflection of a role that is best suited for a clinical pharmacist; (4) attributable to direct patient care; (5) specific to a pharmaceutical care process; (6) aligned with professional goals, objectives, and practices of a clinical pharmacist; (7) an accepted disease-based quality indicator (if applicable); (8) feasible to measure; (9) efficient to measure; (10) a valuable quality measure; and (11) generalizable to all hospital pharmacy types (eg, rural vs urban hospitals, community vs teaching hospitals).

Evidence Summary Tables Evidence tables summarizing the key studies9-14 addressing clinical pharmacy activities were created for the Delphi panelists to review prior to Delphi round 1. Each evidence table included basic information about each study and included title, design, objective, patient population, methods, and results along with study strengths and limitations. The tables were provided to Delphi panelists prior to round 1 to ensure that all panelists had baseline knowledge of the best evidence supporting the candidate cpKPIs.

Evidence-Informed Thematic Critical Activity Areas and Final Inventory of Candidate cpKPIs In collaboration with the working group, 3 members (DD, OF, KT) reviewed the evidence tables summarizing the highest-quality hospital pharmacy practice literature and extracted 8 thematic critical activity areas that the final consensus cpKPIs should represent. These critical activities included best-possible medication history (BPMH), admission medication reconciliation, patient care rounds, pharmaceutical care, disease- or drug-specific quality indicators, patient education/discharge counseling, discharge medication reconciliation, and postdischarge follow-up. Two working group members (RSS and WMS) refined the list of 137 crude candidate clinical pharmacy metrics by applying the cpKPI consensus definition and 11 ideal attributes criteria. A description of the activities associated with each candidate cpKPI was also created. This resulted in a final inventory of 26 candidate cpKPIs representing the 8 thematic critical activity areas (Table 1). A summary of the methodology is presented in Figure 1.

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

4

Annals of Pharmacotherapy 

Table 1.  Candidate Clinical Pharmacy Key Performance Indicators (cpKPIs). Candidate cpKPI

Thematic cpKPI Critical Activity Area

1. Number (or proportion) of patients who receive a formal documented best-possible medication history (BPMH) by a pharmacist or pharmacy technician 2. Number (or proportion) of patients who receive formal documented admission medication reconciliation and the resolution of identified discrepancies by a pharmacist 3. Number (or proportion) of pharmacists who actively participate in interprofessional patient care rounds to improve medication management 4. Number (or proportion) of patients for whom clinical pharmacists have completed a pharmaceutical care plan 5. Number of total drug therapy problems (DTPs) resolved by pharmacists 6. Number of DTPs resolved for “high-alert” (ISMP) medications by pharmacists 7. Number (or proportion) of patients with health record documentation by a pharmacist 8. Number (or proportion) of patients who have received in-person education from a pharmacist about their disease(s) and medication(s) during their hospital stay 9. Number (or proportion) of hospital patients who receive medication counseling by a pharmacist at discharge 10. Number (or proportion) of hospital patients who receive formal documented seamless care activities by a pharmacist 11. Proportion of patients who receive formal documented discharge medication reconciliation and resolution of identified discrepancies by a pharmacist 12. Number (or proportion) of patients discharged with complex and high-risk medication regimens for whom pharmacists have documented assessments of the patients’ response to treatment plans by following up between 3 and 7 days postdischarge 13. Number (or proportion) of heart failure patients with an ACE inhibitor or ARB initiated or titrated to target doses prior to discharge 14. Number (or proportion) of heart failure patients with a β-blocker initiated or titrated to target doses prior to discharge 15. Number (or proportion) of ischemic heart disease patients with a β-blocker initiated or titrated to target doses prior to discharge 16. Number (or proportion) of ischemic heart disease patients who receive ASA prior to discharge 17. Number (or proportion) of ischemic heart disease patients who receive a statin prior to discharge 18. Number (or proportion) of chronic obstructive pulmonary disease (COPD) patients who receive systemic corticosteroids for acute exacerbation of COPD (AECOPD) 19. Number (or proportion) of COPD patients who receive empirical antibiotics for purulent AECOPD 20. Number (or proportion) of COPD patients who receive documented pharmacist education on COPD; medications, including inhaler technique; recognizing AECOPD; self-management plan; nicotine replacement/smoking cessation; vaccines; medication adherence; and exercise 21. Number (or proportion) of atrial fibrillation patients who receive stroke prophylaxis prior to discharge 22. Number (or proportion) of patients who receive influenza/pneumococcal/tetanus vaccination prior to discharge 23. Number (or proportion) of patients who receive nicotine replacement therapy to prevent symptoms of nicotine withdrawal during hospitalization 24. Number (or proportion) of patients who receive smoking cessation counseling/ therapy prior to discharge 25. Number (or proportion) of inpatients receiving venous thromboembolism prophylaxis

BPMH

26. Number (or proportion) of cancer inpatients receiving cytotoxic chemotherapy as determined by hospital-approved chemotherapy treatment protocol

Admission medication reconciliation Interprofessional patient care rounds Pharmaceutical care Pharmaceutical care Pharmaceutical care Pharmaceutical care Patient education/Discharge counseling Patient education/Discharge counseling Patient education/Discharge counseling Discharge medication reconciliation Postdischarge follow-up Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators Disease- or drug-specific quality indicators

Abbreviations: ACE, angiotensin-converting enzyme; ARB, angiotensin receptor blocker; ASA, acetylsalicylic acid; ISMP, Institute for Safe Medication Practices

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

5

Fernandes et al

Survey Questionnaire Development

Figure 1.  Study methodology.

Abbreviation: KPI, key performance indicator.

Expert Delphi Panel Panelists were frontline hospital pharmacists, hospital pharmacy leaders, or members of national hospital pharmacy organizations and represented all 10 Canadian provinces. They had expertise on hospital-based clinical pharmacy services. The panelists were identified and recruited by 4 investigators (OF, KT, SKG, HB). Members of the original working group were eligible to participate, in addition to other national representatives who were not previously part of the working group. The list of panel participants was created by 3 investigators (OF, KT, SKG) by identifying at least 1 participant for each of the 10 Canadian provinces. The potential panelists had to be hospital pharmacists and had to have demonstrated an interest in clinical pharmacy metrics in their current or past professional roles. A systematic review of Delphi methods used to create health care quality indicators suggested that 20 participants is an adequate Delphi panel size.15 Therefore, after institutional review board approval, 26 panelists were contacted, and all provided informed consent to participate.

The Delphi survey questionnaire required creation of the final overall consensus cpKPI criterion; selecting the Likert-type rating scale; populating the instrument with 26 candidate cpKPIs, 11 ideal cpKPI attributes, and overall consensus criterion; defining the consensus threshold; and pretesting the questionnaire. The working group drafted several versions of the final overall consensus criterion that would be used to determine whether a candidate cpKPI would be selected. It was agreed that the final overall consensus criterion would state that “measuring this cpKPI is useful in advancing clinical pharmacy practice to improve quality of patient care.” This criterion was the final determinant of whether a candidate cpKPI was selected during the Delphi procedure. A 9-point Likert scale was used for panelist ratings of the candidate cpKPIs with respect to the selection criteria and the overall consensus criterion.21 A score of 1 indicated that the panelist strongly disagreed and a score of 9 indicated that the panelist strongly agreed. A score of 5 indicated that the panelist neither agreed nor disagreed. The 26 candidate cpKPIs, 11 cpKPI selection criteria, and final overall consensus criterion were populated into an Internet-based survey tool (SurveyMonkey Inc, Palo Alto, CA; http://www.surveymonkey.com). Three hospital pharmacists not involved in the study pretested the questionnaire and provided feedback on the time required to complete the questionnaire, ease of access to the electronic survey via the e-mail link, clarity of the layout and instructions, and suggestions to improve the clarity and ease of use of the questionnaire. After receiving e-mail feedback, the electronic questionnaire was finalized.

Modified Delphi Procedure To ensure that panelists had consistent baseline knowledge of the study procedures and the key evidence supporting clinical pharmacy services, an orientation package was required to be reviewed prior to Delphi round 1. The panelists were also provided with optional reading material that included the publications9-14,22 that guided creation of the candidate cpKPIs. The original inventory of 137 crude candidate clinical pharmacy metrics was available to panelists on request. Round 1 of the modified Delphi procedure took place between December 21, 2012, and January 14, 2013. In addition to rating each candidate cpKPI using the selection criteria and overall consensus criterion, the panelists had the opportunity to provide qualitative feedback on the title and description of each candidate cpKPI and could suggest additional candidate cpKPIs for consideration in round 2.

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

6

Annals of Pharmacotherapy 

Round 2 of the modified Delphi procedure took place between January 17, 2013, and January 30, 2013. Qualitative feedback from round 1 of the Delphi process was incorporated into the survey questionnaire for round 2. Panelists were provided a copy of their individual responses from round 1, the group’s rating of the overall consensus criterion for each candidate cpKPI, and comments provided by panelists during round 1. Additional candidate cpKPIs suggested by panelists were included in the round 2 questionnaires. In addition to rating each candidate cpKPI using the selection criteria and overall consensus criterion, the panelists had the opportunity to provide qualitative feedback on the title and description of each candidate cpKPI but could not suggest additional candidate cpKPIs during this round. A meeting to share the results of the first 2 Delphi rounds and to encourage panelist discussion around clarifying candidate cpKPI wording and definitions was held between rounds 2 and 3 on February 5, 2013. Panelists were encouraged to attend the meeting in person and, if not possible, by teleconference. The meeting was cofacilitated by 3 of the study investigators (OF, KT, SKG) and their team of University of Toronto Doctor of Pharmacy students. Suggestions for new candidate cpKPIs could be made during the meeting; however, candidate cpKPIs were not rated during this meeting. Round 3 of the modified Delphi procedure took place between February 14, 2013, and March 8, 2013. Panelists were provided a copy of their individual responses from round 2, the group’s rating of the overall consensus criterion for each candidate cpKPI from round 2, and comments provided by panelists during round 2. Additional candidate cpKPIs suggested during the meeting were included. The same rating process as the previous 2 rounds was used. All candidate cpKPIs were carried forward to the final round. After the third round of Delphi ratings, consensus was reached on a candidate cpKPI if at least 75% (20 out of 26) of the panelists assigned a rating of 7 to 9 on the 9-point Likert scale for the overall consensus criterion.15,23,24

Results Expert Panel Demographics A total of 26 hospital pharmacists representing all 10 Canadian provinces consented to participate in this study. All panelists completed all Delphi rounds, and 25 (96%) attended the live meeting. The panel represented the scope of activities performed by pharmacists involved in adult and pediatric hospital pharmacy practice and included hospital pharmacy leadership, hospital pharmacists who provide inpatient clinical services, hospital pharmacists who perform drug distribution activities, hospital pharmacists with university appointments, and hospital pharmacists

Table 2.  Expert Panel Characteristics. Characteristic

n a

Practice domain   Pharmacy leadership/administration   Clinical (direct patient care)  Academia   Drug distribution Practice sitea   Teaching hospital   Tertiary care hospital   Community hospital  Academia  Clinic Primary population served  Adults  Pediatrics Experience as pharmacist   6-10 Years   11-15 Years   Greater than 15 Years Academic credentialsa  BScPharm  PharmD   Pharmacy residency   Master’s degree

24 (92%) 12 (46%) 12 (46%) 2 (8%) 18 (69%) 9 (35%) 7 (27%) 7 (27%) 3 (12%) 23 (88%) 3 (12%) 2 (8%) 5 (19%) 19 (73%) 26 (100%) 14 (54%) 14 (54%) 5 (19%)

a

Panelists could select more than 1 option.

who work in clinic-based settings. The panelists were experienced, with 19 (73%) having worked as licensed pharmacists for more than 15 years (Table 2).

First 2 Delphi Rounds and Live Meeting Of the 26 candidate cpKPIs, 7 (27%) were rated 7 to 9 on the overall consensus criterion in round 1 (Appendix A). Three additional candidate cpKPIs were suggested by panelists during this round. These included the number (or proportion) of patients who receive formal documented admission medication reconciliation by a pharmacist (includes a pharmacist BPMH or pharmacist BPMH review as part of the medication reconciliation process as well as resolution of identified discrepancies), number (or proportion) of patients receiving proactive comprehensive direct patient care by a pharmacist in collaboration with the health care team, and committed decentralized clinical pharmacist time per patient-day, per patient service area. Of the 29 candidate cpKPIs, 8 (28%) were rated 7 to 9 on the overall consensus criterion in round 2 (Appendix B). One additional candidate cpKPI was generated during the live meeting and was the number (or proportion) of patients for whom pharmacists have completed (executed/ implemented) a pharmaceutical care plan.

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

7

Fernandes et al Table 3.  Consensus Clinical Pharmacy Key Performance Indicators (cpKPIs). Thematic cpKPI Critical Activity Area

Candidate cpKPI 1. Proportion of patients who receive formal documented discharge medication reconciliation and resolution of identified discrepancies by a pharmacist 2. Number (or proportion) of patients who receive formal documented admission medication reconciliation by a pharmacist (includes a pharmacist best-possible medication history or pharmacist best-possible medication history review as part of the medication reconciliation process as well as resolution of identified discrepancies) 3. Number (or proportion) of pharmacists who actively participate in interprofessional patient care rounds to improve medication management 4. Number (proportion) of patients for whom clinical pharmacists have completed (executed/implemented) a pharmaceutical care plan 5. Number of total drug therapy problems resolved by pharmacists 6. Number (or proportion) of patients receiving proactive comprehensive direct patient care by a pharmacist in collaboration with the health care team 7. Number (or proportion) of hospital patients who receive medication counseling by a pharmacist at discharge 8. Number (or proportion) of patients who have received in-person education from a pharmacist about their disease(s) and medication(s) during their hospital stay

Consensus Ratingsa (n = 26)

Discharge medication reconciliation

25

Admission medication reconciliation and bestpossible medication history

24

Interprofessional patient care rounds Pharmaceutical care

24

Pharmaceutical care Bundle of cpKPI critical activity areas

23 22

Patient education/ discharge counseling Patient education/ discharge counseling

21

24

20

a

Number of panelists who rated the consensus criterion 7 to 9 on Likert scale in final Delphi round.

Final Consensus cpKPIs The third round ratings of the candidate cpKPIs are described in Appendix C. Of the 30 candidate cpKPIs, 8 (27%) were rated 7 to 9 on the overall consensus criterion and are considered consensus cpKPIs (Table 3). Three of the consensus cpKPIs were not part of the original inventory of 26 candidate cpKPIs; they were number (or proportion) of patients who receive formal documented admission medication reconciliation by a pharmacist (combined with BPMH), number (or proportion) of patients for whom clinical pharmacists have completed (executed/implemented) a pharmaceutical care plan, and number (or proportion) of patients receiving proactive comprehensive direct patient care by a pharmacist in collaboration with the health care team. The final suite of cpKPIs represents 6 of the 8 clinical pharmacy critical activity areas. The 2 thematic cpKPI critical activity areas not represented in the final 8 were postdischarge follow-up and disease- or drug-specific best-practice quality indicators.

Discussion An expert panel of Canadian hospital pharmacists systematically determined that 8 cpKPIs could be useful to advance clinical pharmacy practice to improve the quality of patient care. The 8 indicators represent clinical pharmacy activities

demonstrated to improve patient outcomes.9-13 These activities involve BPMH, admission medication reconciliation, patient care rounds, pharmaceutical care, patient education/ discharge counseling, and discharge medication reconciliation. Disease- or drug-specific quality indicators and postdischarge follow-up candidate cpKPIs were not selected in the final suite. Several methodological factors associated with this study strengthen the validity of the indicators selected.25 The expert panel was represented by all dimensions of hospital pharmacist practice in Canada, with a mix of frontline clinical pharmacists and hospital pharmacy leaders. The panel was composed of an experienced cohort of hospital pharmacists who completed all 3 rounds of the Delphi survey. Creation of the inventory of candidate cpKPIs was informed by published evidence that included systematic reviews, RCTs, and large epidemiological studies. Inventory development was also supported by gray literature such as professional society group vision documents. This should strengthen the external validity of the cpKPIs by aligning with our professional vision for clinical pharmacy services. Indicators that were derived from frontline pharmacists and the expert panelists were also included in the inventory and help ensure that consensus cpKPIs reflect real-world, highvalue activities.

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

8

Annals of Pharmacotherapy 

Panelists were given several opportunities to provide feedback during the Delphi rounds and live meeting. The live meeting enabled discussion among panelists on the wording and clarity of candidate cpKPIs. The cpKPI addressing pharmacist delivery of bundled, proactive, comprehensive direct patient care in collaboration with the health care team was not part of the original inventory of candidate cpKPIs primarily because it does not represent a single, isolated clinical pharmacy activity. Panelist discussion highlighted that the key RCTs demonstrating a positive impact on hospital readmissions involved pharmacist delivery of a “bundle” of interrelated process-of-care interventions rather than isolated elements.9,10 Comparing isolated discrete clinical pharmacy activities with proactive bundle-of-care activities is analogous to comparing single grapes to a grape bunch. For example, discharge education and counseling is an isolated discrete clinical pharmacy activity that can be performed proactively or reactively. Still, discharge education and counseling is one part of a larger stem of pharmaceutical care. The stem of pharmaceutical care will contain several “grapes,” such as drug therapy problem (DTP) resolution, interprofessional patient care rounds participation, and discharge counseling. Yet the value toward improving patient care would be much higher if all isolated discrete clinical pharmacy elements are bundled as part of proactive patient care (grape bunch), with the isolated discrete clinical pharmacy activities interlinked. This indicator promotes patient-centered, proactive pharmaceutical care rather than reactive delivery of discrete clinical pharmacy activities and, therefore, is an optimal indicator of pharmacy service delivery. Specifically, this final consensus bundle cpKPI included 5 interlinked processes of care from the 2 RCTs9,10: (1) pharmaceutical care, (2) admission medication reconciliation, (3) active participation on interprofessional rounds, (4) patient education, and (5) discharge medication reconciliation. There are potential limitations to the interpretation of this study. The Delphi approach may be limited, in that it may not reflect a true consensus.26 For example, it is assumed that the panelists did not know who the other panelists were prior to the meeting between rounds 2 and 3; however, there is a low likelihood that panelists communicated prior to the meeting, and that may have influenced one or more panelists to make decisions that were not entirely their own. Another potential limitation to this study design is the potential for ignoring or not exploring disagreement within the panel. However, the live meeting allowed for facilitated, collegial discussion and creative abrasion among panelists, such that we are confident that a true consensus was achieved.26 Unlike the process-of-care candidate cpKPIs that focused on discrete clinical pharmacy activities (such as

DTP resolution, medication reconciliation, or patient education), which were summarized for panelists in detailed evidence tables, for candidate cpKPIs related to specific diseases or drugs, panelists were provided links to up-todate clinical consensus guidelines (detailed summary of all clinical trials relating to each disease state indicator were beyond the scope of this investigation). Panelists provided meaningful commentary and debate on the relative merit of process of care versus disease- or drug-specific indicators as cpKPIs. Whereas many of the disease-or drug-based indicators had extensive evidence to support them, most of the outcomes were not determined solely by the hospital pharmacist and were not as pharmacist specific as the process-of-care indicators. For example, the percentage of appropriate patients receiving venous thromboembolism (VTE) prophylaxis may be largely influenced by prescriber (physician) intervention rather than solely hospital pharmacist intervention. As such, a gross assessment of the proportion of patients receiving specific medications (represented by the disease-specific indicators) may not meaningfully represent unique and independent pharmacist contributions to care unless DTPs resolved are accounted for. Consequently, panelist discussion suggested that a future consensus process to specifically account for drug or disease-specific high value action item drug therapy problems resolved14 may be a more appropriate approach to address this because it concurrently accounts for both (1) highly impactful patient therapeutic interventions and (2) pharmacist sensitivity. Interestingly, the only candidate disease- or drug-specific indicator that had at least 20 panelists rate it between 7 and 9 in any round was the number (or proportion) of inpatients receiving VTE prophylaxis. This candidate cpKPI received 20 ratings of 7 to 9 in round 1, but this number progressively dropped in following rounds. No other candidate disease- or drug-specific cpKPIs came close to being selected. VTE prophylaxis is an institutional requirement as dictated by national hospital accreditation standards27 and as such may have been more highly rated simply because of familiarity. Also, disease- or drug-specific cpKPIs would be defined using very specific criteria that could change based on evolving scientific evidence. Because the main purpose of developing cpKPIs is to allow institutional benchmarking to guide improvement initiatives targeting the quality of clinical pharmacists’ activities, static criteria associated with disease- or drugspecific cpKPIs may quickly become outdated. Panelists may have rated these candidate cpKPIs low because of the inherent risk for losing relevance over time or concerns over application of disease-based indicators in specialized practice areas. Postdischarge follow-up was a core clinical pharmacy activity included in the intervention arm of a landmark

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

9

Fernandes et al trial9 that informed candidate cpKPI inventory creation. The authors of the most comprehensive review of hospitalbased clinical pharmacy activities also reported this activity as being highly valuable to patient care.11 Despite the supporting evidence, this indicator was not selected. Panelist discussion suggested that there were workload and health-system funding concerns about the feasibility of large-scale implementation of postdischarge follow-up by inpatient pharmacists in the context of competing priorities for limited pharmacist resources and that it may be more optimally considered in the future after implementation of the other cpKPIs.28 Also, community and ambulatory care pharmacists may be better positioned to take the lead on postdischarge follow-up. The only other published study that we are aware of on cpKPI development aimed to establish a set of measurable KPIs, which demonstrate clinical pharmacy’s contribution to patient care in New Zealand hospitals.22 Although the initial phase of this study required systematic development of cpKPIs by a Delphi panel of 3 pharmacists, the main focus was stakeholder feedback on the relevance and measurability of 52 KPIs. In contrast, our study utilized an expert panel of 26 hospital pharmacists representing frontline hospital pharmacists, hospital pharmacy leaders, and academia. External stakeholder feedback (eg, patients, physicians, nurses) is an essential part of the preimplementation process for cpKPIs. However, consensus was first sought from the pharmacy profession before engaging those outside the profession. This development sequence should be beneficial in helping to translate the evidence supporting clinical pharmacy activities to nonpharmacist key stakeholders and to mobilize the results into practice. Prior to this study, we knew of no systematically developed, evidence-informed consensus cpKPIs relevant to clinical pharmacy practice. It is believed that these cpKPIs will facilitate advancement of clinical pharmacy practice in hospitals. Development and subsequent implementation of

a system to measure these indicators offers several benefits, including enhanced professional transparency and accountability, better performance toward desired quality practices, and increased allocation of resources to high-quality services.1 This core set of cpKPIs can also assist in defining minimum standards of pharmacy practice and allow meaningful comparisons of clinical pharmacy services within and between organizations.1 They can also facilitate hospital pharmacist performance management.29 Large-scale standardized measurement of these consensus cpKPIs across hospital systems can also spark the development of future research evaluating the real-world impact of pharmacist processes of care on meaningful patient outcomes such as hospital readmissions and mortality. Furthermore, future research should focus on (1) seeking external stakeholder feedback on cpKPIs, (2) establishing consensus on “high value action item DTPs resolved,” and (3) developing consensus cpKPIs for the primary care, ambulatory care, and community pharmacy settings. The real-world adoption and feasibility of measuring these cpKPIs and what their impact is on advancing clinical pharmacy practice and improving patient outcomes have yet to be determined. Finally, it is unknown whether clinical pharmacy subspecialty areas, such as critical care or cardiology, require different or supplemental cpKPIs to help advance clinical pharmacy practice and the quality of patient care in those areas. Future research should address these unanswered questions to provide a more complete understanding of the role of cpKPIs in hospital pharmacy practice.

Conclusion An expert panel of hospital pharmacists established 8 cpKPIs using a modified Delphi consensus-building process. Adoption, measurement, and assessment of these cpKPIs may advance clinical pharmacy practice and ultimately improve the quality of care for hospitalized patients.

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

10

Annals of Pharmacotherapy 

Appendix A Round 1 Expert Panel Ratings. Candidate cpKPI Number (or proportion) of patients who receive a formal documented best-possible medication history by a pharmacist or pharmacy technician Number (or proportion) of patients who receive formal documented admission medication reconciliation and the resolution of identified discrepancies by a pharmacist Number (or proportion) of pharmacists who actively participate in interprofessional patient care rounds to improve medication management Number (or proportion) of patients for whom clinical pharmacists have completed a pharmaceutical care plan Number of total drug therapy problems (DTPs) resolved by pharmacists Number of DTPs resolved for “high-alert” (ISMP) medications by pharmacists Number (or proportion) of patients with health record documentation by a pharmacist Number (or proportion) of patients who have received in-person education from a pharmacist about their disease(s) and medication(s) during their hospital stay Number (or proportion) of hospital patients who receive medication counseling by a pharmacist at discharge Number (or proportion) of hospital patients who receive formal documented seamless care activities by a pharmacist Proportion of patients who receive formal documented discharge medication reconciliation and resolution of identified discrepancies by a pharmacist Number (or proportion) of patients discharged with complex and high-risk medication regimens for whom pharmacists have documented assessments of the patients’ response to treatment plans by following up between 3 and 7 days postdischarge Number (or proportion) of heart failure patients with an ACE inhibitor or ARB initiated or titrated to target doses prior to discharge Number (or proportion) of heart failure patients with a β-blocker initiated or titrated to target doses prior to discharge Number (or proportion) of ischemic heart disease patients with a β-blocker initiated or titrated to target doses prior to discharge Number (or proportion) of ischemic heart disease patients who receive ASA prior to discharge Number (or proportion) of ischemic heart disease patients who receive a statin prior to discharge Number (or proportion) of chronic obstructive pulmonary disease (COPD) patients who receive systemic corticosteroids for acute exacerbation of COPD (AECOPD) Number (or proportion) of COPD patients who receive empirical antibiotics for purulent acute exacerbation of COPD Number (or proportion) of COPD patients who receive documented pharmacist education on COPD; medications, including inhaler technique; recognizing AECOPD; self-management plan; nicotine replacement/smoking cessation; vaccines; medication adherence; and exercise Number (or proportion) of atrial fibrillation patients who receive stroke prophylaxis prior to discharge Number (or proportion) of patients who receive influenzae/pneumococcal/tetanus vaccination prior to discharge Number (or proportion) of patients who receive nicotine replacement therapy to prevent symptoms of nicotine withdrawal during hospitalization Number (or proportion) of patients who receive smoking cessation counseling/therapy prior to discharge Number (or proportion) of inpatients receiving venous thromboembolism prophylaxis Number (or proportion) of cancer inpatients receiving cytotoxic chemotherapy as determined by hospital-approved chemotherapy treatment protocol

Ratings 1-3

Ratings 4-6

Ratings 7-9

0

7

19

1

3

22

0

5

21

1

7

18

1 1 2 0

4 5 15 7

21 20 9 19

0

4

22

0

16

10

0

3

23

0

9

17

0

15

11

0

13

13

0

13

13

0 0

11 13

15 13

0

15

11

0

13

13

0

12

14

1

13

12

0

15

11

1

19

6

3

11

12

0 2

6 16

20 8

Abbreviations: ACE, angiotensin-converting enzyme; ARB, angiotensin receptor blocker; cpKPI, clinical pharmacy key performance indicator.

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

11

Fernandes et al

Appendix B Round 2 Expert Panel Ratings. Candidate cpKPI Number (or proportion) of patients who receive a formal documented best-possible medication history (BPMH) by a pharmacist or pharmacy technician Number (or proportion) of patients who receive formal documented admission medication reconciliation and the resolution of identified discrepancies by a pharmacist Number (or proportion) of pharmacists who actively participate in interprofessional patient care rounds to improve medication management Number (or proportion) of patients for whom clinical pharmacists have completed a pharmaceutical care plan Number of total drug therapy problems (DTPs) resolved by pharmacists Number of DTPs resolved for “high-alert” (ISMP) medications by pharmacists Number (or proportion) of patients with health record documentation by a pharmacist Number (or proportion) of patients who have received in-person education from a pharmacist about their disease(s) and medication(s) during their hospital stay Number (or proportion) of hospital patients who receive medication counseling by a pharmacist at discharge Number (or proportion) of hospital patients who receive formal documented seamless care activities by a pharmacist Proportion of patients who receive formal documented discharge medication reconciliation and resolution of identified discrepancies by a pharmacist Number (or proportion) of patients discharged with complex and high-risk medication regimens for whom pharmacists have documented assessments of the patients’ response to treatment plans by following up between 3 and 7 days postdischarge Number (or proportion) of heart failure patients with an ACE inhibitor or ARB initiated or titrated to target doses prior to discharge Number (or proportion) of heart failure patients with a β-blocker initiated or titrated to target doses prior to discharge Number (or proportion) of ischemic heart disease patients with a β-blocker initiated or titrated to target doses prior to discharge Number (or proportion) of ischemic heart disease patients who receive ASA prior to discharge Number (or proportion) of ischemic heart disease patients who receive a statin prior to discharge Number (or proportion) of chronic obstructive pulmonary disease (COPD) patients who receive systemic corticosteroids for acute exacerbation of COPD (AECOPD) Number (or proportion) of COPD patients who receive empirical antibiotics for purulent acute exacerbation of COPD Number (or proportion) of COPD patients who receive documented pharmacist education on COPD; medications, including inhaler technique; recognizing AECOPD; self-management plan; nicotine replacement/smoking cessation; vaccines; medication adherence; and exercise Number (or proportion) of atrial fibrillation patients who receive stroke prophylaxis prior to discharge Number (or proportion) of patients who receive influenza/pneumococcal/tetanus vaccination prior to discharge Number (or proportion) of patients who receive nicotine replacement therapy to prevent symptoms of nicotine withdrawal during hospitalization Number (or proportion) of patients who receive smoking cessation counseling/therapy prior to discharge Number (or proportion) of inpatients receiving venous thromboembolism prophylaxis Number (or proportion) of cancer inpatients receiving cytotoxic chemotherapy as determined by hospital approved chemotherapy treatment protocol Number (or proportion) of patients who receive formal documented admission medication reconciliation by a pharmacist (includes a pharmacist BPMH or pharmacist BPMH review as part of the medication reconciliation process as well as resolution of identified discrepancies) Number (or proportion) of patients receiving proactive comprehensive direct patient care by a pharmacist in collaboration with the health care team Committed, decentralized clinical pharmacist time per patient-day per patient service area

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

Ratings 1-3

Ratings 4-6

Ratings 7-9

0

7

19

1

3

22

0

4

22

1

6

19

1 1 2 0

2 7 17 6

23 18 7 20

1

4

21

0

16

10

0

1

25

0

13

13

0

19

7

0

18

8

0

18

8

0 0 0

18 18 19

8 8 7

0

19

7

0

12

14

1 0

17 18

8 8

1

23

2

2 0 0

18 7 24

6 19 2

2

2

22

1

3

22

2

8

16

12

Annals of Pharmacotherapy 

Appendix C Round 3 Expert Panel Ratings. Candidate cpKPI Number (or proportion) of patients who receive a formal documented best-possible medication history (BPMH) by a pharmacist or pharmacy technician Number (or proportion) of patients who receive formal documented admission medication reconciliation and the resolution of identified discrepancies by a pharmacist Number (or proportion) of pharmacists who actively participate in interprofessional patient care rounds to improve medication management Number (or proportion) of patients for whom clinical pharmacists have completed a pharmaceutical care plan Number of total drug therapy problems (DTPs) resolved by pharmacists Number of DTPs resolved for “high-alert” (ISMP) medications by pharmacists Number (or proportion) of patients with health record documentation by a pharmacist Number (or proportion) of patients who have received in-person education from a pharmacist about their disease(s) and medication(s) during their hospital stay Number (or proportion) of hospital patients who receive medication counseling by a pharmacist at discharge Number (or proportion) of hospital patients who receive formal documented seamless care activities by a pharmacist Proportion of patients who receive formal documented discharge medication reconciliation and resolution of identified discrepancies by a pharmacist Number (or proportion) of patients discharged with complex and high-risk medication regimens for whom pharmacists have documented assessments of the patients’ response to treatment plans by following up between 3 and 7 days postdischarge Number (or proportion) of heart failure patients with an ACE inhibitor or ARB initiated or titrated to target doses prior to discharge Number (or proportion) of heart failure patients with a β-blocker initiated or titrated to target doses prior to discharge Number (or proportion) of ischemic heart disease patients with a β-blocker initiated or titrated to target doses prior to discharge Number (or proportion) of ischemic heart disease patients who receive ASA prior to discharge Number (or proportion) of ischemic heart disease patients who receive a statin prior to discharge Number (or proportion) of chronic obstructive pulmonary disease (COPD) patients who receive systemic corticosteroids for acute exacerbation of COPD (AECOPD) Number (or proportion) of COPD patients who receive empirical antibiotics for purulent acute exacerbation of COPD Number (or proportion) of COPD patients who receive documented pharmacist education on COPD; medications, including inhaler technique; recognizing AECOPD; self-management plan; nicotine replacement/smoking cessation; vaccines; medication adherence; and exercise Number (or proportion) of atrial fibrillation patients who receive stroke prophylaxis prior to discharge Number (or proportion) of patients who receive influenzae/pneumococcal/tetanus vaccination prior to discharge Number (or proportion) of patients who receive nicotine replacement therapy to prevent symptoms of nicotine withdrawal during hospitalization Number (or proportion) of patients who receive smoking cessation counseling/therapy prior to discharge Number (or proportion) of inpatients receiving venous thromboembolism prophylaxis Number (or proportion) of cancer inpatients receiving cytotoxic chemotherapy as determined by hospital approved chemotherapy treatment protocol Number (or proportion) of patients who receive formal documented admission medication reconciliation by a pharmacist (includes a pharmacist BPMH or pharmacist BPMH review as part of the medication reconciliation process as well as resolution of identified discrepancies) Number (or proportion) of patients receiving proactive comprehensive direct patient care by a pharmacist in collaboration with the health care team Committed, decentralized clinical pharmacist time per patient-day per patient service area Number (proportion) of patients for whom clinical pharmacists have completed (executed/implemented) a pharmaceutical care plan

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

Ratings 1-3

Ratings 4-6

Ratings 7-9

4

13

9

1

6

19

0

2

24

1

13

12

0 0 4 0

3 12 13 6

23 14 9 20

0 0

5 18

21 8

0

1

25

1

14

11

1

16

9

1

16

9

1

16

9

1 1 1

16 16 18

9 9 7

1

18

7

1

12

13

1 1

16 20

9 5

1

24

1

1 1 3

19 9 20

6 16 3

0

2

24

1

3

22

1 0

16 2

9 24

13

Fernandes et al Acknowledgments cpKPI Working Group Collaborative: Jean-François Bussières, Aleksandra Bjelajac-Mejia, Douglas Doucette, Olavo Fernandes, Sean Gorman, Nick Honcharik, Jason Kielly, Don Kuntz, Peter Loewen, Catherine Lyder, Bob MacLean, Neil MacKinnon, Mark Makowsky, Bruce Millin, Diana Mourao, Taciana Pereira, Colette Raymond, William Semchuk, Stephen Shalansky, Richard Slavik, Jeremy Slobodan, Sean Spina, Kent Toombs, Peter Zed. Delphi survey instrument pretesters: Lisa Nodwell, Karen Cameron, Nicole Bruchet. Delphi Panelists: Bob MacLean, Anne Hiltz, Scott Edwards, Iain Smith, Dylana Arsenault, Jean-Francois Guiven, Gary Wong, Allan Mills, Dawn Jennings, Donna Woloschuk, Steve Shalansky, Alice Chan, Sean Spina, William Semchuk, Bruce Millin, Nick Honcharik, Richard Slavik, Taciana Pereira, Don Kuntz, Jean-François Bussières, Jeremy Slobodan, Catherine Lyder, Douglas Doucette, Sandra Bjelajac Mejia, Carolyn Bornstein, Elaine Tom. Author experts who attended Delphi meeting: Peter Kaboli and Mark Makowsky. Administrative and logistics support: the Canadian Society of Hospital Pharmacists, University Health Network, Interior Health Authority, and Capital Health District Health Authority.

Authors’ Note This was presented as a poster presentation at 2013 ACCP Annual Meeting; October 13-16, 2013; Albuquerque, New Mexico.

Declaration of Conflicting Interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study received no grant funding support but was made possible through generous in-kind support from the University Health Network, Interior Health Authority, Capital District Health Authority, the Canadian Society of Hospital Pharmacists, and other Canadian health care organizations.

References 1. Doucette D. Should key performance indicators for clinical services be mandatory? The “pro” side. Can J Hosp Pharm. 2011;64:55-56. 2. Lindsay P. Performance Measurement 101: Introduction to Performance Measurement in Healthcare [webinar]. Toronto, ON: Centre for Effective Practice, Clinical Practice Guidelines Institute; 2010. 3. Mant J. Process versus outcome indicators in the assessment of quality of healthcare. Int J Qual Health Care. 2001;13: 475-480. 4. Ibrahim JE. Performance indicators from all perspectives. Int J Qual Health Care. 2001;13:431-432. 5. Rubin HR, Pronovost P, Diette GB. The advantages and disadvantages of process-based measures of health care quality. Int J Qual Health Care. 2001;13:469-474.

6. Institute of Medicine. Crossing the Quality Chasm: the IOM Health Care Quality Initiative. Washington, DC: National Academy of Sciences; 2001. http://www.iom.edu/Global/ News%20Announcements/Crossing-the-Quality-ChasmThe-IOM-Health-Care-Quality-Initiative.aspx. Accessed April 8, 2014. 7. Fowler A, Campbell D. Benchmarking and performance management in clinical pharmacy. Int J Operations Prod Manag. 2001;12:327-350. 8. American College of Clinical Pharmacy. The definition of clinical pharmacy. Pharmacotherapy. 2008;28:816-817. 9. Gillespie U, Alassaad A, Henrohn D, et al. A comprehensive pharmacist intervention to reduce morbidity in patients 80 years or older. Arch Intern Med. 2009:169:894-990. 10. Makowsky MJ, Koshman SL, Midodzi WK, Tsuyuki RT. Capturing outcomes of clinical activities performed by a rounding pharmacist practicing in a team environment: the COLLABORATE study. Med Care. 2009;47:642-650. 11. Kaboli PJ, Hoth AR, McClimon BJ, Schnipper JL. Clinical pharmacists and inpatient medical care: a systematic review. Arch Intern Med. 2006;166:955-964. 12. Bond CA, Raehl CL. Clinical pharmacy services, pharmacy staffing, and hospital mortality rates. Pharmacotherapy. 2007;27:481-493. 13. Chisholm-Burns MA, Kim LJ, Spivey CA. US pharmacists’ effect as team members on patient care: systematic review and meta-analyses. Med Care. 2010;48:923-933. 14. Bruchet N, Loewen P, de Lemos J. Improving the quality of clinical pharmacy services: a process to identify and capture high-value “quality actions.” Can J Hosp Pharm. 2011;64: 42-47. 15. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6:e20476. 16. Nigam R, MacKinnon NJ, Hartnell NR, et al. Development of Canadian safety indicators for medication use. Healthc Q. 2008;11(Sp):47-53. 17. Goodman CM. The Delphi technique: a critique. J Adv Nurs. 1987;12:729-734. 18. Canadian Society of Hospital Pharmacists. CSHP goals and objectives for pharmacy practice in hospitals and related healthcare settings to be achieved by 2015. Ottawa, ON, Canada: Canadian Society of Hospital Pharmacists; 2007. http://www.cshp.ca/cshp2015/docs/ CSHP2015GoalsandObjectivesStatusReportMay2011.pdf. Updated May 19, 2008. Accessed March 3, 2015. 19. Canadian Pharmacists Association. Blueprint for Pharmacy: The Vision for Pharmacy. Ottawa, ON, Canada: Canadian Pharmacists Association; 2008. http://blueprintforpharmacy.ca/docs/pdfs/2011/05/11/BlueprintVision. pdf?Status=Master. Accessed April 7, 2014. 20. National Quality Measures Clearinghouse: Desirable Attributes of a Quality Measure. Rockville, MD: Agency for Healthcare Research and Quality; 2014. http://www.qualitymeasures.ahrq. gov/tutorial/attributes.aspx. Accessed April 8, 2014. 21. Fitch K, Bernstein S, Aguilar M, et al. The RAND/UCLA Appropriateness Method User’s Manual. Santa Monica, CA: RAND; 2001.

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015

14

Annals of Pharmacotherapy 

22. Ng J, Harrison J. Key performance indicators for clinical pharmacy services in New Zealand public hospitals: stakeholder perspectives. J Pharm Health Serv Res. 2010;1:75-84. 23. Bell CM, Brener SS, Comrie R, Anderson GM, Bronskill SE. Quality measures for medication continuity in long-term care facilities, using a structured panel process. Drug Aging. 2012;29:319-327. 24. Morris AM, Brener S, Dresser L, et al. Use of a structured panel process to define quality metrics for antimicrobial stewardship programs. Infect Control Hosp Epidemiol. 2012;33:500-506. 25. Campbell SM, Braspenning J, Hutchinson A, Marshall M. Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care. 2002;11:358-364.

26. Yousuf MI. Using experts’ opinions through Delphi technique. Pract Assess Res Eval. 2007;12:1-8. 27. Accreditation Canada. Required Organizational Practices. Ottawa, ON, Canada: Accreditation Canada; 2014. http:// www.accreditation.ca/sites/default/files/rop-handbook2014-en.pdf. Accessed April 8, 2014. 28. Bussières JF. Clinical pharmacy services. In: Babich M, Bornstein C, Bussières JF, eds. Hospital Pharmacy in Canada 2009/2010 Annual Report. Indianapolis, IN: Eli Lilly; 2010:3-20. http://www.lillyhospitalsurvey.ca/hpc2/ content/2010_report/2009_2010_full_E.pdf. Accessed April 8, 2014. 29. SHPA Committee of Specialty Practice in Clinical Pharmacy. SHPA standards of practice for clinical pharmacy. J Pharm Pract Res. 2005;35:122-146.

Downloaded from aop.sagepub.com at USD & Wegner Health Science Information Center on April 9, 2015