Med Teach Downloaded from informahealthcare.com by Ohio State University Libraries on 12/07/14 For personal use only.
Medical Teacher, Vol. 14, No. 213, 1992
179
An examination of the examinations: the reliability of the objective structured clinical examination and clinical examination
ABDULBASIT A-LATIF, Department of Surgery, King Saud University College of Medicine, Abha, Saudi Arabia
This study evaluates the correlation that exists between the objective structured clinical examination and the traditional clinical examination. An analysis is pevormed on the results of four groups of students assessed by these two methods. The internal consistency of the OSCE is also described by correlating the results by rank difference correlation using the split-half method of grading the even and odd numbered stations separately. Also 46 surgical interns were asked to indicate their preference regarding the eficacy of five diflzrent methods of assessment of clinical competence. The results indicate a high reliability of OSCE and traditional clinical examination. In the opinion of surgical interns, the OSCE and clinical examination both scored high for their ability to assess clinical competence. SUMMARY
Introduction Conducting a reliable and valid clinical examination for undergraduate medical students is a formidable task. Traditionally medical students have always faced clinical examinations usually one long case and several short cases with an oral examination. But of late this kind of assessment has been questioned because it has been regarded as too narrow, selective and for being insufficiently comprehensive to test all the knowledge, skills and attitudes needed by the doctors (Harden et al., 1975). Similarly the validity of multiple choice questions in assessing the degree of clinical competence have been challenged by several studies (Levine et al., 1970; Newble et al., 1978; Lazarus & Kent, 1983). It has been suggested that for performance evaluation at least five to seven patients are required for a reasonably accurate estimate of general competence of [emergency] physicians (Maatsch et al., 1987). The objective structured clinical examination (OSCE) employs such larger number of patients for the student’s evaluation. There is a general consensus of opinion that OSCE constitutes a definite improvement on the traditional examination and this type of examination has now been
Med Teach Downloaded from informahealthcare.com by Ohio State University Libraries on 12/07/14 For personal use only.
180
Abdulbasit A-Latif
used for several years as in-course assessment or forms an integral part of the final examination (Cuscheri et al., 1979). Roberts & Norman (1990), demonstrated a high reliability of teacher’s observations of the OSCE clinical skills stations and a very good test-retest internal reliability of different stations. Harden & Mulholland (1990), have observed a good correlation between the students’ scores on history-taking stations and have concluded that one advantage of the OSCE is that it provides a measure of students’ competence in history taking. Cox (1990), however has suggested the extension of OSCE for final clinical examination as inadequate because its rigid examination structure fails to test clinical tasks requiring eclectic, responsive skills controlled by clinical judgment. This study aims at examining the reliability of the two major components of assessment instrument namely the OSCE and the traditional clinical examination, also it examines the internal reliability of the OSCE. The study was then completed by asking the fresh graduates who were nearing the completion of their internship year to give their opinion regarding the ability of different types of examinations to test the clinical competence necessary for doctors.
Methods At the King Saud University College of Medicine in Abha, for the last five years we have employed OSCE, with seminar presentations as forming the mid term in-course assessment followed a few weeks later by a traditional long case, short cases with table viva examination as the end-course examination in surgery. The surgery course is conducted over a period of eight weeks. Mid term OSCE is held usually in week five and the traditional clinical examination in week eight. Since there is only an interval of three weeks between the two examinations, they are regarded as two components of the same examination. The reliability of the whole examination was measured by correlating the two sets of scores obtained for each student and measuring the degree of relationship between them by means of a correlation coefficient using the Spearman rank-difference method. Similarly the internal reliability of the OSCE was calculated by adding the scores of even numbered stations and odd numbered stations separately and computing the rank-difference correlation as for a split half method for internal consistency. Four examinations were thus analysed. After graduation each student was asked to complete a questionnaire anonymously at the end of his surgical internship. They were asked to indicate their preference regarding the efficacy of five different methods of assessment. A Likert type questionnaire containing 12 given criteria considered to be important components of clinical competence, grouped into those concerned with objectivity, knowledge, skills and attitude (Table I), invited the surgical interns to indicate their preference on a 5-point scale (5 =agree strongly; 4 =agree; 3=neutral; 2 =disagree; 1=disagree strongly).
Results There was high correlation between OSCE and the final clinical examination in all four groups of students (Table 11). When both examinations were considered as forming one whole examination for the same group of students the reliability was high. In OSCE I, the internal consistency was low (0.34). One possible reason could be that in
Reliability of OSCE and clinical examination
181
TABLE I. Interns’ preference for methods of assessment.
Med Teach Downloaded from informahealthcare.com by Ohio State University Libraries on 12/07/14 For personal use only.
Components of clinical competence
MCQ’s
Methods of assessment Orals Essays Clinical
OSCE
Objectivity All students assessed in similar conditions Little examiner subjectivity Wide coverage
4.3
3.0
2.8
4.1
4.3
4.6 4.0
2.3 2.8
3.3 3.8
3.1 2.3
4.2 4.4
Mean
4.3
2.7
3.3
3.1
4.3
4.0
3.7
3.0
3.4
3.8
Knowledge Factual recall Comprehension and application
3.1
4.0
3.7
3.8
3.9
3.5
3.8
3.3
3.6
3.8
History taking Physical examination Problem-solving Clinical reasoning Communication
2.2 2.1 2.8 2.7 1.9
3.4 3.1 3.0 4.0 2.6
2.7 1.9 2.9 3.0 2.4
4.0 4.4 4.1 4.3 4.0
4.2 4.5 4.3 4.2 4.4
Mean
2.3
3.2
2.6
4.1
4.3
Attitudes Interpersonal skills Patient education
1.7 1.4
2.5 2.1
2.7 1.5
4.1 4.3
4.2 4.0
Mean
Skills
Mean
1.5
2.3
2.1
4.2
4.1
Overall mean
2.9
3.0
2.8
3.8
4.2
Two most preferred methods in each category are underlined.
this 21 station OSCE only 4 patients were used. In OSCE 11, I11 & IV, with a similar number of total stations at least 7 patients were used and the results showed average to high internal consistency and reliability (Table 111). TABLE 11. Rank difference correlation p OSCE vs. final clinical. Reliability whole
Exam no. I
I1 111 IV
No. of students
Sum rank (D2)
p
examination*
21 18 11 15
672 564 58 108
0.57 0.42 0.74 0.81
0.72 0.59 0.85 0.89
*Spearman-Brown formula
Forty-six surgical interns completed the questionnaire. Mean scores for preference of surgical interns for various methods of assessment are shown in Table I. The interns regarded OSCE and MCQs as being equally objective. The knowledge was best
182
Abdulbasit A-Latif
measured by OSCE and orals and skills were best assessed by OSCE and clinicals. Although the interns marginally preferred clinicals for the measurement of attitudes, both OSCE and clinicals were regarded as important components for overall measurement of clinical competence. TABLE 111. Internal consistency of OSCE even vs odd stations
Med Teach Downloaded from informahealthcare.com by Ohio State University Libraries on 12/07/14 For personal use only.
Exam no. I I1 111
IV
No. of students
21 18 11 15
Sum Rank (D2) 1019 568 62 124
p
Reliability whole examination
0.34 0.42 0.72 0.78
0.50 0.59 0.83 0.87
Discussion Evaluation of clinical competence of undergraduate medical students continues to be a challenge for medical teachers. The results of this study indicate that the objective structured clinical examination and the traditional clinical examination have a high reliability. This provides an important feedback for the teachers for designing teaching programmes and to vary their methods of teaching. It has been said that the type of learning activity in which students will engage is primarily determined by the type of assessment used (Guilbert, 1977). Without doubt, the assessment procedure is the most potent factor influencing student learning behaviour (Elton & Laurillard 1979). A change in assessment procedure can result in a change in learning behaviour. Not only does assessment profoundly affect the knowledge and skills the student acquires (Marton & Saljo, 1976), but it also influences the approach to learning. For example, tests requiring only the recall of factual knowledge tend to introduce the surface approach even in those whose preference is to adopt a deep approach (Newble & Entwistle, 1986). If a student expects to be examined in a variety of clinical skills he will wish to learn these from his tutors before the examination (Smith et al., 1984). The OSCE stations when compared by the split-half method of scoring even and odd stations separately showed a good range of internal consistency in all except OSCE I. This may be in part due to the fact that only four patients were used in this examination as compared with seven or more patient stations in the other three examinations. Roberts & Norman (1990) have suggested that one reason for the low observed correlation across different stations in OSCE is a result of different skills being assessed, rather than any inherent variability in performance. The method of assessment preferred by the surgical interns varied with the component of clinical competence being tested. For objectivity the MCQ's and OSCE were chosen. Knowledge was considered to be better tested by orals and OSCE. Skills and attitudes were favourably assessed by OSCE and the clinicals. These views suggest that there is no single most appropriate method of assessment which can adequately test all the desired components of clinical competence. Although OSCE has been shown as a powerful tool in testing a large variety of skills, other methods such as orals, MCQ's and clinicals have also been found to be favourite. Advantages of OSCE
Reliability of OSCE and clinical examination
183
over other forms of assessment procedures have been well recognized and one of particular value is the provision of feedback by providing the students with the marked checklists during or after the examination (Black & Harden, 1986), or to provide them with a profile which expresses the student’s ability across a range of qualities and skills.
Med Teach Downloaded from informahealthcare.com by Ohio State University Libraries on 12/07/14 For personal use only.
Conclusion In conclusion, the objective structured clinical examination and the traditional clinical examination correlate very well in assessing undergraduate medical students. Students have also indicated their preference for the MCQ’s and orals. It is suggested that several of these methods of assessment should be used in combination to complement each other and provide a broad based assessment of student’s performance.
Correspondence: Associate Professor Abdulbasit A-Latif, Department of Surgery, King Saud University, College of Medicine, Abha, P.O. Box 641, Saudi Arabia.
REFERENCES BLACK,N.M.I. & HARDEN,R.M. (1986) Providing feedback to students on clinical skills by using the objective structured clinical examination, Medical Education, 20, pp. 48-52. Cox, K. (1990) No Oscar for OSCA, Medical Education, 24, pp. 540-545. CUSCHIERI,A., GLEFSON, F.A., HARDEN, R.M. & WOOD, R.A.B. (1979) A new approach to a final examination in surgery: use of the objective structured clinical examination, Annals of the Royal College of Surgeons of England, 61, pp. 400-405. ELTON,L.R.B. & LAURILLARD, D.M. (1979) Trends in research on student learning, Studies in Higher Education, 4, pp. 87-102. GUILBERT, J.J. (1977) Educational handbook for health personnel, p. 310 (Geneva, World Health Organization). HARDEN,R.M., STEVENSON, M., DOWNIE,W.W. & WILSON,G.M. (1975) Assessment of clinical competence using objective structured examination, British Medical Journal, l, pp. 447-451. HARDEN,R.M. & MULHOLLAND, H. (1990) The assessment of competence in history-taking, Medical Education, 24, p. 183. LAZARUS, J. & KENT, A.P. (1983) Student attitudes towards the objective structured clinical examination (OSCE) and conventional methods of assessment, South Afncan Medical Journal, 64, pp. 390-394. LEVINE,H.G., MCGUIRE,C.H. & NATTRFS, L.W. (1970) The validity of multiple choice achievement tests as measure of competence in medicine, American Educational Research Journal, 1, pp. 69-82. ~ ~ A A T S C H J., , JUANG, R.R., DOWNING, S.M. & MUNGER,B.S. (1987) Examiner assessments of clinical performance: what do they tell us about clinical competence? Evaluation and Programme Planning, 10, pp. 13-17. MARTON,F. & SALJO,R. (1976) On qualitative differences in learning. 11. Outcome as a function of the learners’ conception of the task, British Journal of Educational Psychology, 46, pp. 115-127. NEWBLE,D.I., ELMSLIE,R.G. & BAXTER,A. (1978) A problem-based criterion-referenced examination of clinical competence, Journal of Medical Education, 53, pp. 720-726. NEWBLE,D.I. & ENTWISTLE, N.J. (1986) Learning styles and approaches: implications for medical education, Medical Education, 20, pp. 162-1 75. ROBERTS,J. & NORMAN, G. (1990) Reliability and learning from the objective structured clinical examintion, Medical Education, 24, pp. 219-223. SMITH,L.J., PRICE, D.A. & HOUSTON,I.B. (1984) Objective structured clinical examination compared with other forms of student assessment, Archives of Diseases in Childhood, 59, pp. 1173-1176.
Medical Teacher, Vol. 14, No. 213, 1992
179
An examination of the examinations: the reliability of the objective structured clinical examination and clinical examination
ABDULBASIT A-LATIF, Department of Surgery, King Saud University College of Medicine, Abha, Saudi Arabia
This study evaluates the correlation that exists between the objective structured clinical examination and the traditional clinical examination. An analysis is pevormed on the results of four groups of students assessed by these two methods. The internal consistency of the OSCE is also described by correlating the results by rank difference correlation using the split-half method of grading the even and odd numbered stations separately. Also 46 surgical interns were asked to indicate their preference regarding the eficacy of five diflzrent methods of assessment of clinical competence. The results indicate a high reliability of OSCE and traditional clinical examination. In the opinion of surgical interns, the OSCE and clinical examination both scored high for their ability to assess clinical competence. SUMMARY
Introduction Conducting a reliable and valid clinical examination for undergraduate medical students is a formidable task. Traditionally medical students have always faced clinical examinations usually one long case and several short cases with an oral examination. But of late this kind of assessment has been questioned because it has been regarded as too narrow, selective and for being insufficiently comprehensive to test all the knowledge, skills and attitudes needed by the doctors (Harden et al., 1975). Similarly the validity of multiple choice questions in assessing the degree of clinical competence have been challenged by several studies (Levine et al., 1970; Newble et al., 1978; Lazarus & Kent, 1983). It has been suggested that for performance evaluation at least five to seven patients are required for a reasonably accurate estimate of general competence of [emergency] physicians (Maatsch et al., 1987). The objective structured clinical examination (OSCE) employs such larger number of patients for the student’s evaluation. There is a general consensus of opinion that OSCE constitutes a definite improvement on the traditional examination and this type of examination has now been
Med Teach Downloaded from informahealthcare.com by Ohio State University Libraries on 12/07/14 For personal use only.
180
Abdulbasit A-Latif
used for several years as in-course assessment or forms an integral part of the final examination (Cuscheri et al., 1979). Roberts & Norman (1990), demonstrated a high reliability of teacher’s observations of the OSCE clinical skills stations and a very good test-retest internal reliability of different stations. Harden & Mulholland (1990), have observed a good correlation between the students’ scores on history-taking stations and have concluded that one advantage of the OSCE is that it provides a measure of students’ competence in history taking. Cox (1990), however has suggested the extension of OSCE for final clinical examination as inadequate because its rigid examination structure fails to test clinical tasks requiring eclectic, responsive skills controlled by clinical judgment. This study aims at examining the reliability of the two major components of assessment instrument namely the OSCE and the traditional clinical examination, also it examines the internal reliability of the OSCE. The study was then completed by asking the fresh graduates who were nearing the completion of their internship year to give their opinion regarding the ability of different types of examinations to test the clinical competence necessary for doctors.
Methods At the King Saud University College of Medicine in Abha, for the last five years we have employed OSCE, with seminar presentations as forming the mid term in-course assessment followed a few weeks later by a traditional long case, short cases with table viva examination as the end-course examination in surgery. The surgery course is conducted over a period of eight weeks. Mid term OSCE is held usually in week five and the traditional clinical examination in week eight. Since there is only an interval of three weeks between the two examinations, they are regarded as two components of the same examination. The reliability of the whole examination was measured by correlating the two sets of scores obtained for each student and measuring the degree of relationship between them by means of a correlation coefficient using the Spearman rank-difference method. Similarly the internal reliability of the OSCE was calculated by adding the scores of even numbered stations and odd numbered stations separately and computing the rank-difference correlation as for a split half method for internal consistency. Four examinations were thus analysed. After graduation each student was asked to complete a questionnaire anonymously at the end of his surgical internship. They were asked to indicate their preference regarding the efficacy of five different methods of assessment. A Likert type questionnaire containing 12 given criteria considered to be important components of clinical competence, grouped into those concerned with objectivity, knowledge, skills and attitude (Table I), invited the surgical interns to indicate their preference on a 5-point scale (5 =agree strongly; 4 =agree; 3=neutral; 2 =disagree; 1=disagree strongly).
Results There was high correlation between OSCE and the final clinical examination in all four groups of students (Table 11). When both examinations were considered as forming one whole examination for the same group of students the reliability was high. In OSCE I, the internal consistency was low (0.34). One possible reason could be that in
Reliability of OSCE and clinical examination
181
TABLE I. Interns’ preference for methods of assessment.
Med Teach Downloaded from informahealthcare.com by Ohio State University Libraries on 12/07/14 For personal use only.
Components of clinical competence
MCQ’s
Methods of assessment Orals Essays Clinical
OSCE
Objectivity All students assessed in similar conditions Little examiner subjectivity Wide coverage
4.3
3.0
2.8
4.1
4.3
4.6 4.0
2.3 2.8
3.3 3.8
3.1 2.3
4.2 4.4
Mean
4.3
2.7
3.3
3.1
4.3
4.0
3.7
3.0
3.4
3.8
Knowledge Factual recall Comprehension and application
3.1
4.0
3.7
3.8
3.9
3.5
3.8
3.3
3.6
3.8
History taking Physical examination Problem-solving Clinical reasoning Communication
2.2 2.1 2.8 2.7 1.9
3.4 3.1 3.0 4.0 2.6
2.7 1.9 2.9 3.0 2.4
4.0 4.4 4.1 4.3 4.0
4.2 4.5 4.3 4.2 4.4
Mean
2.3
3.2
2.6
4.1
4.3
Attitudes Interpersonal skills Patient education
1.7 1.4
2.5 2.1
2.7 1.5
4.1 4.3
4.2 4.0
Mean
Skills
Mean
1.5
2.3
2.1
4.2
4.1
Overall mean
2.9
3.0
2.8
3.8
4.2
Two most preferred methods in each category are underlined.
this 21 station OSCE only 4 patients were used. In OSCE 11, I11 & IV, with a similar number of total stations at least 7 patients were used and the results showed average to high internal consistency and reliability (Table 111). TABLE 11. Rank difference correlation p OSCE vs. final clinical. Reliability whole
Exam no. I
I1 111 IV
No. of students
Sum rank (D2)
p
examination*
21 18 11 15
672 564 58 108
0.57 0.42 0.74 0.81
0.72 0.59 0.85 0.89
*Spearman-Brown formula
Forty-six surgical interns completed the questionnaire. Mean scores for preference of surgical interns for various methods of assessment are shown in Table I. The interns regarded OSCE and MCQs as being equally objective. The knowledge was best
182
Abdulbasit A-Latif
measured by OSCE and orals and skills were best assessed by OSCE and clinicals. Although the interns marginally preferred clinicals for the measurement of attitudes, both OSCE and clinicals were regarded as important components for overall measurement of clinical competence. TABLE 111. Internal consistency of OSCE even vs odd stations
Med Teach Downloaded from informahealthcare.com by Ohio State University Libraries on 12/07/14 For personal use only.
Exam no. I I1 111
IV
No. of students
21 18 11 15
Sum Rank (D2) 1019 568 62 124
p
Reliability whole examination
0.34 0.42 0.72 0.78
0.50 0.59 0.83 0.87
Discussion Evaluation of clinical competence of undergraduate medical students continues to be a challenge for medical teachers. The results of this study indicate that the objective structured clinical examination and the traditional clinical examination have a high reliability. This provides an important feedback for the teachers for designing teaching programmes and to vary their methods of teaching. It has been said that the type of learning activity in which students will engage is primarily determined by the type of assessment used (Guilbert, 1977). Without doubt, the assessment procedure is the most potent factor influencing student learning behaviour (Elton & Laurillard 1979). A change in assessment procedure can result in a change in learning behaviour. Not only does assessment profoundly affect the knowledge and skills the student acquires (Marton & Saljo, 1976), but it also influences the approach to learning. For example, tests requiring only the recall of factual knowledge tend to introduce the surface approach even in those whose preference is to adopt a deep approach (Newble & Entwistle, 1986). If a student expects to be examined in a variety of clinical skills he will wish to learn these from his tutors before the examination (Smith et al., 1984). The OSCE stations when compared by the split-half method of scoring even and odd stations separately showed a good range of internal consistency in all except OSCE I. This may be in part due to the fact that only four patients were used in this examination as compared with seven or more patient stations in the other three examinations. Roberts & Norman (1990) have suggested that one reason for the low observed correlation across different stations in OSCE is a result of different skills being assessed, rather than any inherent variability in performance. The method of assessment preferred by the surgical interns varied with the component of clinical competence being tested. For objectivity the MCQ's and OSCE were chosen. Knowledge was considered to be better tested by orals and OSCE. Skills and attitudes were favourably assessed by OSCE and the clinicals. These views suggest that there is no single most appropriate method of assessment which can adequately test all the desired components of clinical competence. Although OSCE has been shown as a powerful tool in testing a large variety of skills, other methods such as orals, MCQ's and clinicals have also been found to be favourite. Advantages of OSCE
Reliability of OSCE and clinical examination
183
over other forms of assessment procedures have been well recognized and one of particular value is the provision of feedback by providing the students with the marked checklists during or after the examination (Black & Harden, 1986), or to provide them with a profile which expresses the student’s ability across a range of qualities and skills.
Med Teach Downloaded from informahealthcare.com by Ohio State University Libraries on 12/07/14 For personal use only.
Conclusion In conclusion, the objective structured clinical examination and the traditional clinical examination correlate very well in assessing undergraduate medical students. Students have also indicated their preference for the MCQ’s and orals. It is suggested that several of these methods of assessment should be used in combination to complement each other and provide a broad based assessment of student’s performance.
Correspondence: Associate Professor Abdulbasit A-Latif, Department of Surgery, King Saud University, College of Medicine, Abha, P.O. Box 641, Saudi Arabia.
REFERENCES BLACK,N.M.I. & HARDEN,R.M. (1986) Providing feedback to students on clinical skills by using the objective structured clinical examination, Medical Education, 20, pp. 48-52. Cox, K. (1990) No Oscar for OSCA, Medical Education, 24, pp. 540-545. CUSCHIERI,A., GLEFSON, F.A., HARDEN, R.M. & WOOD, R.A.B. (1979) A new approach to a final examination in surgery: use of the objective structured clinical examination, Annals of the Royal College of Surgeons of England, 61, pp. 400-405. ELTON,L.R.B. & LAURILLARD, D.M. (1979) Trends in research on student learning, Studies in Higher Education, 4, pp. 87-102. GUILBERT, J.J. (1977) Educational handbook for health personnel, p. 310 (Geneva, World Health Organization). HARDEN,R.M., STEVENSON, M., DOWNIE,W.W. & WILSON,G.M. (1975) Assessment of clinical competence using objective structured examination, British Medical Journal, l, pp. 447-451. HARDEN,R.M. & MULHOLLAND, H. (1990) The assessment of competence in history-taking, Medical Education, 24, p. 183. LAZARUS, J. & KENT, A.P. (1983) Student attitudes towards the objective structured clinical examination (OSCE) and conventional methods of assessment, South Afncan Medical Journal, 64, pp. 390-394. LEVINE,H.G., MCGUIRE,C.H. & NATTRFS, L.W. (1970) The validity of multiple choice achievement tests as measure of competence in medicine, American Educational Research Journal, 1, pp. 69-82. ~ ~ A A T S C H J., , JUANG, R.R., DOWNING, S.M. & MUNGER,B.S. (1987) Examiner assessments of clinical performance: what do they tell us about clinical competence? Evaluation and Programme Planning, 10, pp. 13-17. MARTON,F. & SALJO,R. (1976) On qualitative differences in learning. 11. Outcome as a function of the learners’ conception of the task, British Journal of Educational Psychology, 46, pp. 115-127. NEWBLE,D.I., ELMSLIE,R.G. & BAXTER,A. (1978) A problem-based criterion-referenced examination of clinical competence, Journal of Medical Education, 53, pp. 720-726. NEWBLE,D.I. & ENTWISTLE, N.J. (1986) Learning styles and approaches: implications for medical education, Medical Education, 20, pp. 162-1 75. ROBERTS,J. & NORMAN, G. (1990) Reliability and learning from the objective structured clinical examintion, Medical Education, 24, pp. 219-223. SMITH,L.J., PRICE, D.A. & HOUSTON,I.B. (1984) Objective structured clinical examination compared with other forms of student assessment, Archives of Diseases in Childhood, 59, pp. 1173-1176.
