CLINICAL AND MATERIALS SCIENCES
Examiner Standardization for Caries Studies STUART FISCHMAN, ANTHONY PICOZZI, DANIEL JULIANO, MALCOLM SLAKTER, and JAMES ENGLISH School of Dentistry, State University of New York at Buffalo, Buffalo, New York 14214, USA and School of Dentistry, Fairleigh Dickinson University, Hackensack, New Jersey 07601
Examiners in clinical control programs customarily undergo an intensive period of training to standardize their interpretation of diagnostic criteria. Data are presented summarizing examiner error rates in calibration examinations and reversal rates in a field study. An important aspect of experimental design is the application of appropriate methods for controlling, or reducing, the effects of those variables that are extraneous to the purposes of the study. One of the most troublesome variables affecting large-scale clinical caries incidence studies, especially those using multiple examiners, is the variation in the diagnosis of the carious lesion that exists between examiners and within examiners. Since clinical examinations form the basis for comparing caries incidence in field trials of potential anticaries treatments, it is obviously important that the examinations be made with sufficient accuracy to be dependable and reproducible. Accordingly, before undertaking field trials, it is customary to train examiners to make comparable and consistent examinations. Several authors have recognized that the assessment of dental caries is associated with considerable interexaminer and intraexaminer variability.1-6 At a conference held in 1968, Radike7 discussed "reversals in diagnosis." He noted that examiner error is Government sponsorship of the research on which this publication is based does not constitute endorsement of the results or conclusions presented here. This investigation was supported, in part, by NIHNIDR Contract 72-2406, National Institutes of Health, Bethesda, Md. Received for publication December 19, 1975. Accepted for publication April 23, 1976.
926
largely due to the inability of examiners to draw a sharp line between sound and unsound tooth structure, particularly those surfaces that are in transition from sound to carious. The resultant variability in diagnosis would theoretically give an equal number of positive and negative examiner errors, but only the negative half would be observable as reversals, that is, surfaces diagnosed as carious or filled on one examination and sound on a subsequent examination. Obviously, the unbiased examiner must resist any attempt to manipulate the data in any way that might eliminate reversals. At the same conference, Horowitz2 stated that it was "essential that before examinations, the examiners undergo an intensive period of training to standardize their interpretation of the criteria to be used and that, during the examinations, they calibrate their examining techniques frequently." He also recommended that examiners should conduct a duplicate examination of patients not included in the clinical investigation before the commencement of the clinical trial. This calibration should help to provide consistent, uniform interpretation and understanding of the diagnostic criteria, and should maximize reproducibility of clinical findings among different examiners as well as within the same examiner. A review of several published studies showed a wide range of reversal rates. For example, values ranged from 0.0258 to 1.629 reversals per subject. In several caries studies neither the reversal rate nor any other parameter of examiner error rate was reported. As a part of a program to evaluate caries preventive measures, a study was undertaken to determine the consistency of each exam-
Downloaded from jdr.sagepub.com at Karolinska Institutets Universitetsbibliotek on May 25, 2015 For personal use only. No other uses without permission.
Vol 55 No. 6
EXAMINER STANDARDIZATION FOR CARIES STUDIES
927
TABLE 1 MEANS AND SD FOR EXAMINER INCONSISTENCIES AT ANNUAL CALIBRATION SESSIONS Yr of Exam
No. of Sub-
Examiner 1
Examiner 2
jects
x
SD
x
SD
1972 1973
27 20
0.59 0.40
0.617 0.293
0.42 0.33
0.681 0.157
1974
19
0.97
1.366
0.76
0.201
iner in caries diagnosis. This article reports on the results of the study. Materials and Methods Subjects were sixth grade students in a fluoridated area at the time the study began. Radiographs were not used and only permanent teeth were studied. The mean DMFS score at original examination was 4.30. The evaluation included the following phases: (1) examiner calibration, (2) replicate examinations, and (3) reversal rate analysis. CALIBRATION SESSIONS-Methods.-Before the baseline examination and at each annual examination thereafter, the two examining dentists and the recorders were trained in the procedures to be followed, and the diagnostic criteria for dental caries were established and reviewed. Each training or standardization session required two days. The morning of the first day was used for the establishment and review of the clinical criteria for the diagnosis of caries and for a discussion of the application of these criteria in clinical situations. After this discussion, three or four students were examined by each examiner. After a given student had been examined by both examiners, he was recalled and the scores of each examiner were compared with the clinical findings in the patient. Differences in the criteria of caries diagnosis were then discussed with the examiners. On the afternoon of the first day and on the morning and afternoon of the second day, a panel of approximately 20 students was examined by each examiner. Each student, therefore, received three examinations by each examiner. The students were not participants in the field trial but were of a similar age. A different set of students was examined each year. Results.-The mean number of diagnostic inconsistencies per student calculated at the three annual examinations is given in Ta-
ble 1. A diagnostic inconsistency is defined as a tooth surface which was called sound on one calibration examination and carious or filled on the other calibration examination. Both examiners had a relatively low rate of inconsistencies, that is, in five of six cases, considerably less than one surface per subject. For each annual calibration examination, examiner 1 seemed to have more inconsistencies than his colleague, but these differences were not statistically significant at the 0.05 level. Table 2 gives the mean numbers of surfaces (averaged across three examinations) found to be decayed and filled by each examiner at the annual calibration sessions. The data suggest that examiner 1 diagnosed more caries than examiner 2; however, this difference was only significant in 1974 (the t test for correlated samples was used and the t value was calculated equal to 4.727; df = 18; P < 0.05). It should be noted that different subjects were used for the calibration sessions each year. REPLICATE EXAMINATIONS.-Methods.-Dur-
ing the course of the field study to evaluate caries preventive measures, a 5% sample of the test panel was examined twice by each examiner on the same day. The examiner was not aware of which students had been selected for reexamination, and no attempt was made to identify the "repeat" subjects to the examiner. Each examiner had his own set of subjects for the replicate examinations. Diagnostic inconsistency is again defined as a tooth surface which was called sound on one of the two examinations and carious or filled on a subsequent or previous examination. Results.-The mean number of diagnostic inconsistencies per subject in these replicate examinations is given in Table 3. The inconsistency rate remained relatively low for
Downloaded from jdr.sagepub.com at Karolinska Institutets Universitetsbibliotek on May 25, 2015 For personal use only. No other uses without permission.
FISCHMAN ET AL
928
J Dent Res November-December 1976
TABLE 2 MEANS AND SD OF SURFACES REPORTED AS DECAYED OR FILLED BY EACH EXAMINER AT ANNUAL CALIBRATION SESSIONS
Yr of Exam-
(averaged across three examinations) Yr of Exam-
Examiner 1
Examiner 2
ination/ Surface
x
SD
x
SD
1972/D 1972/F 1973/D 1973/F 1974/D 1974/F
1.18 3.20 1.68 1.13
1.32 3.71 1.90 1.75 3.23
.60 3.05 1.38 1.28 2.83 1.35
1.19 3.59 1.65 1.95 2.63 1.90
4.20 1.28
2.07
TABLE 3 MEAN EXAMINER INCONSISTENCIES EXAMINATIONS
Note: A different set of participants was examined each year and they are the same participants as those reported on in Table 1. D, decayed; F, filled surfaces.
each examiner, and was considerably less than one surface per subject. REVERSAL ANALYSIS.-1lethod.-The annual examinations which were conducted on those participants in the caries prevention study provided an opportunity to determine the mean number of "reversals" between the two examinations. A reversal is defined as a tooth surface which was found to be filled or carious on the initial examination and reported to be sound on the second examination. If one accepts the premise that caries, as defined in the clinical or epidemiologic sense, is essentially an irreversible process, these represent "true" errors. Results.-It should be noted that the reversal rate reported between the annual examinations reflects only errors in one direction. In theory, an equal number of errors occurred in the "opposite" direction and therefore the examiners' inconsistencies would be double the reversal rate. The reversal rate between the two annual examinations was 0.440 for examiner 1 (605 subjects) and 0.303 for examiner 2 (614 subjects). This reversal rate of less than one half surface per subject would suggest a theoretical inconsistency rate of less than one surface per subject. Discussion and Conclusions The wide range of published examiner reversal rates is a function of many factors. Certainly the skill of the examiner and accuracy of the reporter are important components. In addition, the age of the sub-
Examiner
AT
REPLICATE
Examiner 2
1
ination
N*
5X
N
X
1972 1973
78 65
0.179 0.477
78 62
0.269 0.435
Note: Each examiner had his own set of participants for the replicate examination. *
N is the number of participants.
jects, the use of radiographs in making a diagnosis, the prevalence of caries in the study population, the annual increment seen, the "field" conditions at the time of examination, and similar technical details are also quite important. From a review of the data obtained during this study, we can conclude that examiner error rate in the calibration exams approximated the error rate or reversal rate in the actual field trial. Inconsistency rates were generally less than one surface per subject. This suggests that the data obtained at standardization or calibration examinations could be predictive of an examiner's reversal rates in a field study and could therefore be used to screen examiners before the initiation of a large-scale study. Furthermore, the data suggest that examiner differences in the scoring of surfaces as decayed or filled on the calibration exams can be used as an indicator of a trend of examiner scoring during the field study. This study was of three years' duration and the inconsistencies between examiners were persistent, though not significant. References Conduct of Clinical Assays of Caries Controlling Agents, in HARRIs, R.S. (ed) Art and Sciences of Dental Caries Research, New York: Academic Press, Inc., 1968, pp 163-175. 2. HOROWITZ, H.: Inter- and Intra- Examiner Variability, in Proceedings of the Conference on Clinicat Testing of Cariostatic Agents (October 1968), Chicago: American Dental Association, 1972, pp 96-97. 3. HOROWITz, H.: Clinical Trial of Preventives for Dental Caries, J Public Health Dent 32: 229-233, 1972. 1. FINN, S.D.:
4. HOROWITZ, H.; BAUME, L.; BACKER-DIRKS, O.; DAVIES, G.; and SLACK, G.: Principal Require-
Downloaded from jdr.sagepub.com at Karolinska Institutets Universitetsbibliotek on May 25, 2015 For personal use only. No other uses without permission.
Vol 55 No. 6
EXAMINER STANDARDIZATION FOR CARIES STUDIES
ments for Controlled Clinical Trials of Caries
Preventive Agents and Procedures, Int Dent J 23: 506-516, 1973. 5. DEPAOLA, P.: Examiner Inconsistency in Clinical Caries Research, J Oral Ther 4: 200204, 1968. 6. Picozzi, A., and ALMAN, J.: Predicting Examiner Reversal Rates: Empirical Results from Five Caries Trials, abstracted IADR Prograin and Abstracts of Papers, No. 335, 1970. 7. RADIKE, A.W.: Examiner Error and Reversals in Diagnosis, in Proceedings of the Conference
929
Clinical Testing of Cariostatic Agents (October 1968), Chicago: American Dental Association, 1972, pp 92-95. 8. HoRoWITZ, H.; LAW, F.; THOMPSON, M.; and CHAMBERLIN, S.: Evaluation of a Stannous Fluoride Dentifrice for Use in Dental Public Health Programs-Basic Findings, JADA 72: 408-422, 1966. 9. THOMAS, A., and JAMISON, H.: Effects of SnF2 Dentifrices in Caries in Children: Two-Year Clinical Study of Supervised Brushing in Children's Homes, JADA 73: 844-852, 1966. on
Downloaded from jdr.sagepub.com at Karolinska Institutets Universitetsbibliotek on May 25, 2015 For personal use only. No other uses without permission.
Examiner Standardization for Caries Studies STUART FISCHMAN, ANTHONY PICOZZI, DANIEL JULIANO, MALCOLM SLAKTER, and JAMES ENGLISH School of Dentistry, State University of New York at Buffalo, Buffalo, New York 14214, USA and School of Dentistry, Fairleigh Dickinson University, Hackensack, New Jersey 07601
Examiners in clinical control programs customarily undergo an intensive period of training to standardize their interpretation of diagnostic criteria. Data are presented summarizing examiner error rates in calibration examinations and reversal rates in a field study. An important aspect of experimental design is the application of appropriate methods for controlling, or reducing, the effects of those variables that are extraneous to the purposes of the study. One of the most troublesome variables affecting large-scale clinical caries incidence studies, especially those using multiple examiners, is the variation in the diagnosis of the carious lesion that exists between examiners and within examiners. Since clinical examinations form the basis for comparing caries incidence in field trials of potential anticaries treatments, it is obviously important that the examinations be made with sufficient accuracy to be dependable and reproducible. Accordingly, before undertaking field trials, it is customary to train examiners to make comparable and consistent examinations. Several authors have recognized that the assessment of dental caries is associated with considerable interexaminer and intraexaminer variability.1-6 At a conference held in 1968, Radike7 discussed "reversals in diagnosis." He noted that examiner error is Government sponsorship of the research on which this publication is based does not constitute endorsement of the results or conclusions presented here. This investigation was supported, in part, by NIHNIDR Contract 72-2406, National Institutes of Health, Bethesda, Md. Received for publication December 19, 1975. Accepted for publication April 23, 1976.
926
largely due to the inability of examiners to draw a sharp line between sound and unsound tooth structure, particularly those surfaces that are in transition from sound to carious. The resultant variability in diagnosis would theoretically give an equal number of positive and negative examiner errors, but only the negative half would be observable as reversals, that is, surfaces diagnosed as carious or filled on one examination and sound on a subsequent examination. Obviously, the unbiased examiner must resist any attempt to manipulate the data in any way that might eliminate reversals. At the same conference, Horowitz2 stated that it was "essential that before examinations, the examiners undergo an intensive period of training to standardize their interpretation of the criteria to be used and that, during the examinations, they calibrate their examining techniques frequently." He also recommended that examiners should conduct a duplicate examination of patients not included in the clinical investigation before the commencement of the clinical trial. This calibration should help to provide consistent, uniform interpretation and understanding of the diagnostic criteria, and should maximize reproducibility of clinical findings among different examiners as well as within the same examiner. A review of several published studies showed a wide range of reversal rates. For example, values ranged from 0.0258 to 1.629 reversals per subject. In several caries studies neither the reversal rate nor any other parameter of examiner error rate was reported. As a part of a program to evaluate caries preventive measures, a study was undertaken to determine the consistency of each exam-
Downloaded from jdr.sagepub.com at Karolinska Institutets Universitetsbibliotek on May 25, 2015 For personal use only. No other uses without permission.
Vol 55 No. 6
EXAMINER STANDARDIZATION FOR CARIES STUDIES
927
TABLE 1 MEANS AND SD FOR EXAMINER INCONSISTENCIES AT ANNUAL CALIBRATION SESSIONS Yr of Exam
No. of Sub-
Examiner 1
Examiner 2
jects
x
SD
x
SD
1972 1973
27 20
0.59 0.40
0.617 0.293
0.42 0.33
0.681 0.157
1974
19
0.97
1.366
0.76
0.201
iner in caries diagnosis. This article reports on the results of the study. Materials and Methods Subjects were sixth grade students in a fluoridated area at the time the study began. Radiographs were not used and only permanent teeth were studied. The mean DMFS score at original examination was 4.30. The evaluation included the following phases: (1) examiner calibration, (2) replicate examinations, and (3) reversal rate analysis. CALIBRATION SESSIONS-Methods.-Before the baseline examination and at each annual examination thereafter, the two examining dentists and the recorders were trained in the procedures to be followed, and the diagnostic criteria for dental caries were established and reviewed. Each training or standardization session required two days. The morning of the first day was used for the establishment and review of the clinical criteria for the diagnosis of caries and for a discussion of the application of these criteria in clinical situations. After this discussion, three or four students were examined by each examiner. After a given student had been examined by both examiners, he was recalled and the scores of each examiner were compared with the clinical findings in the patient. Differences in the criteria of caries diagnosis were then discussed with the examiners. On the afternoon of the first day and on the morning and afternoon of the second day, a panel of approximately 20 students was examined by each examiner. Each student, therefore, received three examinations by each examiner. The students were not participants in the field trial but were of a similar age. A different set of students was examined each year. Results.-The mean number of diagnostic inconsistencies per student calculated at the three annual examinations is given in Ta-
ble 1. A diagnostic inconsistency is defined as a tooth surface which was called sound on one calibration examination and carious or filled on the other calibration examination. Both examiners had a relatively low rate of inconsistencies, that is, in five of six cases, considerably less than one surface per subject. For each annual calibration examination, examiner 1 seemed to have more inconsistencies than his colleague, but these differences were not statistically significant at the 0.05 level. Table 2 gives the mean numbers of surfaces (averaged across three examinations) found to be decayed and filled by each examiner at the annual calibration sessions. The data suggest that examiner 1 diagnosed more caries than examiner 2; however, this difference was only significant in 1974 (the t test for correlated samples was used and the t value was calculated equal to 4.727; df = 18; P < 0.05). It should be noted that different subjects were used for the calibration sessions each year. REPLICATE EXAMINATIONS.-Methods.-Dur-
ing the course of the field study to evaluate caries preventive measures, a 5% sample of the test panel was examined twice by each examiner on the same day. The examiner was not aware of which students had been selected for reexamination, and no attempt was made to identify the "repeat" subjects to the examiner. Each examiner had his own set of subjects for the replicate examinations. Diagnostic inconsistency is again defined as a tooth surface which was called sound on one of the two examinations and carious or filled on a subsequent or previous examination. Results.-The mean number of diagnostic inconsistencies per subject in these replicate examinations is given in Table 3. The inconsistency rate remained relatively low for
Downloaded from jdr.sagepub.com at Karolinska Institutets Universitetsbibliotek on May 25, 2015 For personal use only. No other uses without permission.
FISCHMAN ET AL
928
J Dent Res November-December 1976
TABLE 2 MEANS AND SD OF SURFACES REPORTED AS DECAYED OR FILLED BY EACH EXAMINER AT ANNUAL CALIBRATION SESSIONS
Yr of Exam-
(averaged across three examinations) Yr of Exam-
Examiner 1
Examiner 2
ination/ Surface
x
SD
x
SD
1972/D 1972/F 1973/D 1973/F 1974/D 1974/F
1.18 3.20 1.68 1.13
1.32 3.71 1.90 1.75 3.23
.60 3.05 1.38 1.28 2.83 1.35
1.19 3.59 1.65 1.95 2.63 1.90
4.20 1.28
2.07
TABLE 3 MEAN EXAMINER INCONSISTENCIES EXAMINATIONS
Note: A different set of participants was examined each year and they are the same participants as those reported on in Table 1. D, decayed; F, filled surfaces.
each examiner, and was considerably less than one surface per subject. REVERSAL ANALYSIS.-1lethod.-The annual examinations which were conducted on those participants in the caries prevention study provided an opportunity to determine the mean number of "reversals" between the two examinations. A reversal is defined as a tooth surface which was found to be filled or carious on the initial examination and reported to be sound on the second examination. If one accepts the premise that caries, as defined in the clinical or epidemiologic sense, is essentially an irreversible process, these represent "true" errors. Results.-It should be noted that the reversal rate reported between the annual examinations reflects only errors in one direction. In theory, an equal number of errors occurred in the "opposite" direction and therefore the examiners' inconsistencies would be double the reversal rate. The reversal rate between the two annual examinations was 0.440 for examiner 1 (605 subjects) and 0.303 for examiner 2 (614 subjects). This reversal rate of less than one half surface per subject would suggest a theoretical inconsistency rate of less than one surface per subject. Discussion and Conclusions The wide range of published examiner reversal rates is a function of many factors. Certainly the skill of the examiner and accuracy of the reporter are important components. In addition, the age of the sub-
Examiner
AT
REPLICATE
Examiner 2
1
ination
N*
5X
N
X
1972 1973
78 65
0.179 0.477
78 62
0.269 0.435
Note: Each examiner had his own set of participants for the replicate examination. *
N is the number of participants.
jects, the use of radiographs in making a diagnosis, the prevalence of caries in the study population, the annual increment seen, the "field" conditions at the time of examination, and similar technical details are also quite important. From a review of the data obtained during this study, we can conclude that examiner error rate in the calibration exams approximated the error rate or reversal rate in the actual field trial. Inconsistency rates were generally less than one surface per subject. This suggests that the data obtained at standardization or calibration examinations could be predictive of an examiner's reversal rates in a field study and could therefore be used to screen examiners before the initiation of a large-scale study. Furthermore, the data suggest that examiner differences in the scoring of surfaces as decayed or filled on the calibration exams can be used as an indicator of a trend of examiner scoring during the field study. This study was of three years' duration and the inconsistencies between examiners were persistent, though not significant. References Conduct of Clinical Assays of Caries Controlling Agents, in HARRIs, R.S. (ed) Art and Sciences of Dental Caries Research, New York: Academic Press, Inc., 1968, pp 163-175. 2. HOROWITZ, H.: Inter- and Intra- Examiner Variability, in Proceedings of the Conference on Clinicat Testing of Cariostatic Agents (October 1968), Chicago: American Dental Association, 1972, pp 96-97. 3. HOROWITz, H.: Clinical Trial of Preventives for Dental Caries, J Public Health Dent 32: 229-233, 1972. 1. FINN, S.D.:
4. HOROWITZ, H.; BAUME, L.; BACKER-DIRKS, O.; DAVIES, G.; and SLACK, G.: Principal Require-
Downloaded from jdr.sagepub.com at Karolinska Institutets Universitetsbibliotek on May 25, 2015 For personal use only. No other uses without permission.
Vol 55 No. 6
EXAMINER STANDARDIZATION FOR CARIES STUDIES
ments for Controlled Clinical Trials of Caries
Preventive Agents and Procedures, Int Dent J 23: 506-516, 1973. 5. DEPAOLA, P.: Examiner Inconsistency in Clinical Caries Research, J Oral Ther 4: 200204, 1968. 6. Picozzi, A., and ALMAN, J.: Predicting Examiner Reversal Rates: Empirical Results from Five Caries Trials, abstracted IADR Prograin and Abstracts of Papers, No. 335, 1970. 7. RADIKE, A.W.: Examiner Error and Reversals in Diagnosis, in Proceedings of the Conference
929
Clinical Testing of Cariostatic Agents (October 1968), Chicago: American Dental Association, 1972, pp 92-95. 8. HoRoWITZ, H.; LAW, F.; THOMPSON, M.; and CHAMBERLIN, S.: Evaluation of a Stannous Fluoride Dentifrice for Use in Dental Public Health Programs-Basic Findings, JADA 72: 408-422, 1966. 9. THOMAS, A., and JAMISON, H.: Effects of SnF2 Dentifrices in Caries in Children: Two-Year Clinical Study of Supervised Brushing in Children's Homes, JADA 73: 844-852, 1966. on
Downloaded from jdr.sagepub.com at Karolinska Institutets Universitetsbibliotek on May 25, 2015 For personal use only. No other uses without permission.
