ORIGINAL REPORTS
Clinical Assessment and Management Examination—Outpatient (CAMEO): Its Validity and Use in a Surgical Milestones Paradigm Adam B. Wilson, PhD,* Jennifer N. Choi, MD,* Laura J. Torbeck, PhD,* John D. Mellinger, MD,† Gary L. Dunnington, MD,* and Reed G. Williams, PhD* Department of Surgery, Indiana University School of Medicine, Indianapolis, Indiana; and †Division of General Surgery, Southern Illinois University School of Medicine, Springfield, Illinois *
OBJECTIVES: Clinical Assessment and Management Examination—Outpatient (CAMEO) is a metric for evaluating the clinical performance of surgery residents. The aim of this study was to investigate the measurement characteristics of CAMEO and propose how it might be used as an evaluation tool within the general surgery milestones project.
exploratory factor analysis, CAMEO was revealed to measure a single dimension of “clinical competence.” CONCLUSIONS: The findings of this research aligned
with related literature and verified that CAMEO scores have desirable measurement properties, making CAMEO an attractive resource for evaluating the clinical performance C 2014 Association of surgery residents. ( J Surg 72:33-40. J of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)
DESIGN: A total of 117 CAMEO evaluations were gathered and used for analysis. Internal consistency reliability was estimated, and item characteristics were explored. A Kruskal-Wallis procedure was performed to discern how well the instrument discriminated between training levels. An exploratory factor analysis was also conducted to understand the dimensionality of the evaluation.
KEY WORDS: CAMEO, validity, milestones, clinic
SETTING: CAMEO evaluations were collected from 2 departments of surgery geographically located in the Midwestern United States. Combined, the participating academic institutions graduate approximately 18 general surgery residents per year.
INTRODUCTION
PARTICIPANTS: In this retrospective data analysis, the
number of evaluations per resident ranged from 1 to 7, and evaluations were collected from 2006 to 2013. For the purpose of data analysis, residents were classified as interns (postgraduate year 1 [PGY1]), juniors (PGY2-3), or seniors (PGY4-5). RESULTS: CAMEO scores were found to have high internal consistency (Cronbach’s α ¼ 0.96), and all items were highly correlated (Z0.86) to composite CAMEO scores. Scores discriminated between senior residents (PGY4-5) and lower level residents (PGY1-3). Per an
Correspondence: Inquiries to Adam B. Wilson, PhD, Department of Surgery, Indiana University School of Medicine, 545 Barnhill Drive, Emerson Hall 543, Indianapolis, IN 46202; E-mail: [email protected], [email protected]
evaluation COMPETENCIES: Patient Care
The Clinical Assessment and Management Examination— Outpatient (CAMEO) is an evaluation that relies on direct observation to assess the performance of resident surgeons in clinical environments. CAMEO is intended to evaluate how well residents function independently in making initial decisions about patient assessment and management and in directing a patient encounter. Per the American Board of Surgery (ABS), “Applicants to the General Surgery Qualifying Exam [are] required to obtain during residency at least two operative performance assessments and two clinical performance assessments conducted by their program director or other faculty members.”1 The 2 assessment forms recommended by the ABS to satisfy the clinical performance requirement are the CAMEO and mini–Clinical Evaluation Exercise (mini-CEX). This study reports on the measurement characteristics of CAMEO and makes recommendations for its use within the Next Accreditation System.
Journal of Surgical Education & 2014 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2014.06.010
33
Because CAMEO is a modified adaptation of the miniCEX, to fully appreciate the significance of this work requires a brief history of mini-CEX research. In the 1990s, early investigations of the mini-CEX were conducted out of the need to correct the shortcomings of traditional CEXs. As Norcini et al.2 summarized, “The unreliability of the observer, the variation of resident performance from patient to patient, and the artificiality of the task mean that the traditional single-interaction CEX is not a dependable measure of a resident’s clinical competence.” Since the introduction of the mini-CEX, it has been reported that high levels of reproducibility can be achieved when 12 to 14 observations per resident are performed,2 evaluators and residents alike are satisfied with the format of the abbreviated instrument, mini-CEX scores have high internal consistency and reproducibility coefficients,3-6 and mini-CEX scores are known to correlate with scores from high-stakes examinations.4,5 One study has also shown that mini-CEXs discriminate between levels of seniority.7 Although, in the absence of post hoc analyses, it was unclear as to which training levels differed in performance.7 Ample literature, including a meta-analysis of published mini-CEX validity research,8 points to the widespread usability and validity of the mini-CEX instrument. Literature that focuses on an instrument’s evidence of validity and practical applications is useful on multiple levels. For instance, education program directors often rely on psychometric evidence to make judgments about the significance or weight of an evaluation compared with other metrics, to establish meaning of performance scores, and to set pass/fail thresholds. In the absence of literature on CAMEOs, surgery program directors and vice chairs of education are left to speculate on the instrument’s validity and intrinsic value. Therefore, the purpose of this study was to explore the measurement characteristics of CAMEO and to assess its usefulness as a measure of surgery resident performance in the surgical clinic. This study reports on the following: (1) CAMEO’s reliability and item characteristics, (2) its ability to discriminate between levels of surgical trainees, and (3) the instrument’s dimensionality. Recommendations for using CAMEO within the surgery milestones paradigm are also presented.
METHODS Instrument Blueprint The assessment goals of the mini-CEX and CAMEO parallel one another in that they aim to evaluate clinical skills performance and provide concise learner feedback on the patient encounter. Attending surgeons complete these evaluations either during or after having directly observed a patient-resident interaction. CAMEO comprises 6 items with descriptive anchors at the extremes and midpoint of a 5-point Likert-type scale that ranges from poor (1 point) to excellent (5 points). As Yeates et al.9 point out, “criterion 34
uncertainty” is 1 way in which measurement error between evaluators can be introduced. The descriptive anchors, therefore, are intended to minimize misinterpretation of scaled items through criterion standardization and allow evaluators to be more discerning when discriminating between performance levels. The 6 categories of evaluation on the CAMEO instrument (http://www.absurgery.org/ default.jsp?certgsqe_resassess) can be summarized as test ordering and understanding, diagnostic acumen, history taking, physical examination, communication skills, and overall performance. As part of the evaluation, faculty evaluators record the patient’s chief complaint, the presumptive diagnosis, and the difficulty of the case. Population and Data Description The Departments of Surgery at the Indiana University School of Medicine (IUSM) and at the Southern Illinois University School of Medicine (SIUSM) have had experience in using CAMEO as a resident evaluation instrument since 2012 and 2006, respectively. Combined IUSM and SIUSM graduate approximately 18 residents per annum. Faculty members at both institutions were encouraged to complete a CAMEO for each resident on a monthly basis. This recommended frequency, however, was neither closely monitored nor strictly enforced. Evaluators were introduced to CAMEO during a faculty development session in which the purpose and usage of the instrument were discussed and instruction was provided in the form of example scenarios. Information was also disseminated through a PowerPoint voiceover presentation distributed via e-mail. A total of 117 CAMEO evaluations (nIUSM ¼ 36 and nSIUSM ¼ 81), completed throughout each year from 2006 to 2013, were included for analysis. The number of CAMEO evaluations per resident ranged from 1 to 7. Levels of resident experience were consolidated to include interns (postgraduate year 1 [PGY1]), junior residents (PGY2-3), and senior (PGY4-5) residents. A total of 40 CAMEOs were submitted for interns, 30 for junior residents, and 47 for senior surgery residents. CAMEO percentage scores were derived by summing the scores of each of the 6 items (worth a maximum of 5 points each) and dividing by the highest attainable score (i.e., 30 points). The supplemental patient perceptions component of the CAMEO form was not evaluated in this study. This retrospective data analysis was approved by the Indiana University Institutional Review Board. Statistical Analyses The internal consistency of the instrument was measured using Cronbach’s alpha coefficient. Corrected item-total correlations and item-discrimination indices were also calculated to determine how each item contributed to total CAMEO scores and to evaluate the discriminatory power of each item, respectively.
Journal of Surgical Education Volume 72/Number 1 January/February 2015
Skewed score distributions warranted the use of nonparametrics. A combination of Kruskal-Wallis tests and follow-up Mann-Whitney tests with Bonferroni adjustments10 was used to compare scores between resident training levels, to evaluate differences in rank-ordered scores irrespective of training level, and to investigate the effects of case difficulty on CAMEO scores grouped by training level. All evaluations were treated as independent samples. Alpha was set at 0.05, and aggregate scores are reported as median (mean ⫾ standard deviation). An exploratory factor analysis was also conducted in SPSS (version 21) to determine the number and types of dimensions measured by the CAMEO instrument. Principal axis factoring with Varimax (i.e., orthogonal) rotation was used. Kaiser’s criterion (i.e., factors with Eigenvalues 41.00) was used to arrive at a parsimonious number of extracted factors. To evaluate sampling adequacy and whether sample correlation matrices were appropriate for factor analysis, a Kaiser-Meyer-Olkin coefficient was computed, and values Z0.60 were considered sufficient for conducting an exploratory factor analysis. Factor loadings Z0.4 were considered significant and were used to interpret factor structure.11
RESULTS Of the 117 evaluations, scores ranged from 40% to 100% with a median of 93.3% (mean ¼ 87.2% ⫾ 14.5%).
Internal consistency reliability for the 6-item CAMEO was 0.96. Corrected item-total correlations were Z0.86. All items were able to moderately discriminate between the lowest and highest performing residents (discrimination index range: 0.29-0.31). After ranking composite scores and equally dividing scores into top, middle, and lower thirds, CAMEO was found to discriminate between the highest, middle, and lowest performing residents, irrespective of resident training level. The highest ranked 1/3 of the evaluations (median ¼ 100% and mean ¼ 100% ⫾ 0.0%) differed significantly (p o 0.001) from the middle ranked 1/3 of evaluations (median ¼ 93.3% and mean ¼ 91.5% ⫾ 6.6%), which differed significantly (p o 0.001) from the lowest ranked 1/3 of evaluations (median ¼ 73.3%, mean ¼ 70.1% ⫾ 10.3%). CAMEO scores were found to vary significantly (p o 0.001) between training levels of the 3 resident groups. In particular, senior residents significantly outperformed (p o 0.001) junior residents and interns (Fig. 1). Senior residents scored a median of 100% (mean ¼ 95.2% ⫾ 7.7%), with a raw score range of 21 to 30; junior residents scored 81.7% (mean ¼ 83.4% ⫾ 13.2%), with a raw score range of 18 to 30; and intern residents scored 80.0% (mean ¼ 80.6% ⫾ 16.9%), with a raw score range of 12 to 30. No difference (p ¼ 0.524) in scores was identified between intern and junior residents (Fig. 1). Concerning case difficulty, no statistical differences were reported (p Z 0.136) between case difficulty level
p
Clinical Assessment and Management Examination—Outpatient (CAMEO): Its Validity and Use in a Surgical Milestones Paradigm Adam B. Wilson, PhD,* Jennifer N. Choi, MD,* Laura J. Torbeck, PhD,* John D. Mellinger, MD,† Gary L. Dunnington, MD,* and Reed G. Williams, PhD* Department of Surgery, Indiana University School of Medicine, Indianapolis, Indiana; and †Division of General Surgery, Southern Illinois University School of Medicine, Springfield, Illinois *
OBJECTIVES: Clinical Assessment and Management Examination—Outpatient (CAMEO) is a metric for evaluating the clinical performance of surgery residents. The aim of this study was to investigate the measurement characteristics of CAMEO and propose how it might be used as an evaluation tool within the general surgery milestones project.
exploratory factor analysis, CAMEO was revealed to measure a single dimension of “clinical competence.” CONCLUSIONS: The findings of this research aligned
with related literature and verified that CAMEO scores have desirable measurement properties, making CAMEO an attractive resource for evaluating the clinical performance C 2014 Association of surgery residents. ( J Surg 72:33-40. J of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)
DESIGN: A total of 117 CAMEO evaluations were gathered and used for analysis. Internal consistency reliability was estimated, and item characteristics were explored. A Kruskal-Wallis procedure was performed to discern how well the instrument discriminated between training levels. An exploratory factor analysis was also conducted to understand the dimensionality of the evaluation.
KEY WORDS: CAMEO, validity, milestones, clinic
SETTING: CAMEO evaluations were collected from 2 departments of surgery geographically located in the Midwestern United States. Combined, the participating academic institutions graduate approximately 18 general surgery residents per year.
INTRODUCTION
PARTICIPANTS: In this retrospective data analysis, the
number of evaluations per resident ranged from 1 to 7, and evaluations were collected from 2006 to 2013. For the purpose of data analysis, residents were classified as interns (postgraduate year 1 [PGY1]), juniors (PGY2-3), or seniors (PGY4-5). RESULTS: CAMEO scores were found to have high internal consistency (Cronbach’s α ¼ 0.96), and all items were highly correlated (Z0.86) to composite CAMEO scores. Scores discriminated between senior residents (PGY4-5) and lower level residents (PGY1-3). Per an
Correspondence: Inquiries to Adam B. Wilson, PhD, Department of Surgery, Indiana University School of Medicine, 545 Barnhill Drive, Emerson Hall 543, Indianapolis, IN 46202; E-mail: [email protected], [email protected]
evaluation COMPETENCIES: Patient Care
The Clinical Assessment and Management Examination— Outpatient (CAMEO) is an evaluation that relies on direct observation to assess the performance of resident surgeons in clinical environments. CAMEO is intended to evaluate how well residents function independently in making initial decisions about patient assessment and management and in directing a patient encounter. Per the American Board of Surgery (ABS), “Applicants to the General Surgery Qualifying Exam [are] required to obtain during residency at least two operative performance assessments and two clinical performance assessments conducted by their program director or other faculty members.”1 The 2 assessment forms recommended by the ABS to satisfy the clinical performance requirement are the CAMEO and mini–Clinical Evaluation Exercise (mini-CEX). This study reports on the measurement characteristics of CAMEO and makes recommendations for its use within the Next Accreditation System.
Journal of Surgical Education & 2014 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2014.06.010
33
Because CAMEO is a modified adaptation of the miniCEX, to fully appreciate the significance of this work requires a brief history of mini-CEX research. In the 1990s, early investigations of the mini-CEX were conducted out of the need to correct the shortcomings of traditional CEXs. As Norcini et al.2 summarized, “The unreliability of the observer, the variation of resident performance from patient to patient, and the artificiality of the task mean that the traditional single-interaction CEX is not a dependable measure of a resident’s clinical competence.” Since the introduction of the mini-CEX, it has been reported that high levels of reproducibility can be achieved when 12 to 14 observations per resident are performed,2 evaluators and residents alike are satisfied with the format of the abbreviated instrument, mini-CEX scores have high internal consistency and reproducibility coefficients,3-6 and mini-CEX scores are known to correlate with scores from high-stakes examinations.4,5 One study has also shown that mini-CEXs discriminate between levels of seniority.7 Although, in the absence of post hoc analyses, it was unclear as to which training levels differed in performance.7 Ample literature, including a meta-analysis of published mini-CEX validity research,8 points to the widespread usability and validity of the mini-CEX instrument. Literature that focuses on an instrument’s evidence of validity and practical applications is useful on multiple levels. For instance, education program directors often rely on psychometric evidence to make judgments about the significance or weight of an evaluation compared with other metrics, to establish meaning of performance scores, and to set pass/fail thresholds. In the absence of literature on CAMEOs, surgery program directors and vice chairs of education are left to speculate on the instrument’s validity and intrinsic value. Therefore, the purpose of this study was to explore the measurement characteristics of CAMEO and to assess its usefulness as a measure of surgery resident performance in the surgical clinic. This study reports on the following: (1) CAMEO’s reliability and item characteristics, (2) its ability to discriminate between levels of surgical trainees, and (3) the instrument’s dimensionality. Recommendations for using CAMEO within the surgery milestones paradigm are also presented.
METHODS Instrument Blueprint The assessment goals of the mini-CEX and CAMEO parallel one another in that they aim to evaluate clinical skills performance and provide concise learner feedback on the patient encounter. Attending surgeons complete these evaluations either during or after having directly observed a patient-resident interaction. CAMEO comprises 6 items with descriptive anchors at the extremes and midpoint of a 5-point Likert-type scale that ranges from poor (1 point) to excellent (5 points). As Yeates et al.9 point out, “criterion 34
uncertainty” is 1 way in which measurement error between evaluators can be introduced. The descriptive anchors, therefore, are intended to minimize misinterpretation of scaled items through criterion standardization and allow evaluators to be more discerning when discriminating between performance levels. The 6 categories of evaluation on the CAMEO instrument (http://www.absurgery.org/ default.jsp?certgsqe_resassess) can be summarized as test ordering and understanding, diagnostic acumen, history taking, physical examination, communication skills, and overall performance. As part of the evaluation, faculty evaluators record the patient’s chief complaint, the presumptive diagnosis, and the difficulty of the case. Population and Data Description The Departments of Surgery at the Indiana University School of Medicine (IUSM) and at the Southern Illinois University School of Medicine (SIUSM) have had experience in using CAMEO as a resident evaluation instrument since 2012 and 2006, respectively. Combined IUSM and SIUSM graduate approximately 18 residents per annum. Faculty members at both institutions were encouraged to complete a CAMEO for each resident on a monthly basis. This recommended frequency, however, was neither closely monitored nor strictly enforced. Evaluators were introduced to CAMEO during a faculty development session in which the purpose and usage of the instrument were discussed and instruction was provided in the form of example scenarios. Information was also disseminated through a PowerPoint voiceover presentation distributed via e-mail. A total of 117 CAMEO evaluations (nIUSM ¼ 36 and nSIUSM ¼ 81), completed throughout each year from 2006 to 2013, were included for analysis. The number of CAMEO evaluations per resident ranged from 1 to 7. Levels of resident experience were consolidated to include interns (postgraduate year 1 [PGY1]), junior residents (PGY2-3), and senior (PGY4-5) residents. A total of 40 CAMEOs were submitted for interns, 30 for junior residents, and 47 for senior surgery residents. CAMEO percentage scores were derived by summing the scores of each of the 6 items (worth a maximum of 5 points each) and dividing by the highest attainable score (i.e., 30 points). The supplemental patient perceptions component of the CAMEO form was not evaluated in this study. This retrospective data analysis was approved by the Indiana University Institutional Review Board. Statistical Analyses The internal consistency of the instrument was measured using Cronbach’s alpha coefficient. Corrected item-total correlations and item-discrimination indices were also calculated to determine how each item contributed to total CAMEO scores and to evaluate the discriminatory power of each item, respectively.
Journal of Surgical Education Volume 72/Number 1 January/February 2015
Skewed score distributions warranted the use of nonparametrics. A combination of Kruskal-Wallis tests and follow-up Mann-Whitney tests with Bonferroni adjustments10 was used to compare scores between resident training levels, to evaluate differences in rank-ordered scores irrespective of training level, and to investigate the effects of case difficulty on CAMEO scores grouped by training level. All evaluations were treated as independent samples. Alpha was set at 0.05, and aggregate scores are reported as median (mean ⫾ standard deviation). An exploratory factor analysis was also conducted in SPSS (version 21) to determine the number and types of dimensions measured by the CAMEO instrument. Principal axis factoring with Varimax (i.e., orthogonal) rotation was used. Kaiser’s criterion (i.e., factors with Eigenvalues 41.00) was used to arrive at a parsimonious number of extracted factors. To evaluate sampling adequacy and whether sample correlation matrices were appropriate for factor analysis, a Kaiser-Meyer-Olkin coefficient was computed, and values Z0.60 were considered sufficient for conducting an exploratory factor analysis. Factor loadings Z0.4 were considered significant and were used to interpret factor structure.11
RESULTS Of the 117 evaluations, scores ranged from 40% to 100% with a median of 93.3% (mean ¼ 87.2% ⫾ 14.5%).
Internal consistency reliability for the 6-item CAMEO was 0.96. Corrected item-total correlations were Z0.86. All items were able to moderately discriminate between the lowest and highest performing residents (discrimination index range: 0.29-0.31). After ranking composite scores and equally dividing scores into top, middle, and lower thirds, CAMEO was found to discriminate between the highest, middle, and lowest performing residents, irrespective of resident training level. The highest ranked 1/3 of the evaluations (median ¼ 100% and mean ¼ 100% ⫾ 0.0%) differed significantly (p o 0.001) from the middle ranked 1/3 of evaluations (median ¼ 93.3% and mean ¼ 91.5% ⫾ 6.6%), which differed significantly (p o 0.001) from the lowest ranked 1/3 of evaluations (median ¼ 73.3%, mean ¼ 70.1% ⫾ 10.3%). CAMEO scores were found to vary significantly (p o 0.001) between training levels of the 3 resident groups. In particular, senior residents significantly outperformed (p o 0.001) junior residents and interns (Fig. 1). Senior residents scored a median of 100% (mean ¼ 95.2% ⫾ 7.7%), with a raw score range of 21 to 30; junior residents scored 81.7% (mean ¼ 83.4% ⫾ 13.2%), with a raw score range of 18 to 30; and intern residents scored 80.0% (mean ¼ 80.6% ⫾ 16.9%), with a raw score range of 12 to 30. No difference (p ¼ 0.524) in scores was identified between intern and junior residents (Fig. 1). Concerning case difficulty, no statistical differences were reported (p Z 0.136) between case difficulty level
p
