264
Journal of Public Health Dentistry
Assessing Dental Practice Quality by Evaluating Radiology Items Ronald J. Hunt, DDS, MS Shen Jang Fann, MS Preventive and Community Dentistry College of Dentistry University of Iowa Iowa City, IA
Me1 L. Kantor, DDS Oral and Maxillofacial Radiology School of Dental Medicine University of Connecticut Farmington, CT
Alvin L. Morris, DDS, PhD Department of Dental Ecology School of Dentistry University of North Carolina Chapel Hill, NC
Abstract The DEMCAD dental office assessment instrument was developed to evaluate practice quality using Donebedian 's quality assessment model of structure, process, and outcome. This previously validated instmment takes about six hours to complete. Subsequent analysis was undertaken to determine whether an abbreviated office assessment based on the evaluation of radiology items was sufficiently sensitive, specific, and practical to be used as a screening instrument for identifying dental offices with very low evaluationscores. Data for this analysis were obtained from 300 volunteer general dental practices evaluated in the field testing of the DEMCAD instrument. The nine radiology structure items predicted very poorly the overall Structure scores. However, 73 radiology process items predicted overall process scores quite accurately. Four of the 73 radiology process items (periodontaldiagnosesrecorded,interdentalbonesho wnon x-rays, caries diagnosesrecorded, and current x-rays mounted)produced a combined R2 Of .58. These four radiology variables predicted the 70 percent of the dental practices with the lowest overall process score with 87percent sensitivity and 93 percent specificity. This analysis showed that an abbreviated dental practice process quality assessment using oral radiology Send correspondenceand reprint requests to Dr. Hunt, Department of Dental Ecology, CB #7450, School of Dentistry, University of North Carolina, Chapel Hill, NC 27599. Manuscript received: 9/17/90; returned to authors for revision: 10/31/90; accepted for publication: 2/17/92.
J Public Health Dent 1992;52(5):264-8
items in an audit of patients' records may be feasible as a screening test for dental office assessment. Key Words: dental health services, quality assurance, oral radiology. The provision of quality dental care has long been a primary goal of the dental profession. More recently, the public also has become more active in demandingquality of care, although the public's interest in quality is often confused with an interest in cost containment. What was once a matter of ethics within the professionhas evolved into a need to respond also to pressures for quality assurance from external forces (1).As a result, various systems and criteria have been investigated for potential use in assessing the quality of dental practices. Most systems for quality assessment in dentistry have been based on the dimensions of structure, process, and outcome as described for evaluation of care provided in medical and hospital settings by Donabedian (2) and adapted to dental settings by Bailit (3). The structure dimensionincludes such facetsof dental practiceas licensure, facilities, equipment, organization, and personnel. The process dimension includes evaluationof the results of the procedures and tasks the dentist performs, such as thorough record keeping, fully developed treatment plans, compliancewith infectioncontrol procedures, creation of good diagnostic radiographs, and accurateinterpretation of the radiographs. The outcome dimension
265
Vol. 52, No. 5, Fall 1992
includes direct evaluation of the care the patient receives, such as patient satisfaction, technical quality of restorations, oral health status of the patient, and completion of treatment. It is generally recognized that the measurement of outcomes of care among patients is preferable to the measurement of processes within the practice itself (4). However, outcomes of care typically are difficult to measure (3). Moreover, outcome assessment requires the direct examination of a number of patients, which greatly increases the monetary costs and the potential psychological costs of the assessment to the patients and the dental practices. Recently developed practice assessments have tended to focus on the process dimension. In 1982 the W. K. Kellogg Foundation funded a largescale dental practice assessment project entitled “Development of Evaluation Methods and Computer Applications in Dentistry (DEMCAD).”One of the primary goals of this project was to develop an objective, practical, and professionally acceptable dental practice assessment method using in-office visits (5,6).The DEMCAD quality assessment instrument, when used as designed, provides a comprehensive evaluation of a dental office structure, process, and outcome. However, the full office assessment requires six hours to complete and it may not always be economically feasible for a given evaluation program to conduct the full assessment. The DEMCAD instrument has not yet been fully validated through use in additional samples of dental practices. Thus, the findings of this analysis cannot be generalized beyond the measures contained in the DEMCAD assessment. This analysis merely explores whether the DEMCAD itself could be abbreviated and still identify the lowest scoring practices as identified by the entire DEMCAD instrument. If so, evidence would be provided that suggests that it may possible to construct shorter dental practice assessment methods. A previous evaluation of the radiographic equipment and procedures in the 300 dental offices volunteering to participate in the development phase of the DEMCAD project subsequently led to the question of whether an abbreviated office assessmentbased on the evaluation of the oral radiology items alone could be used as a screening instrument for identifying dental offices that had very low overall DEMCAD practice assessment scores. An abbreviated assessment, if sufficiently sensitive and specific, could be more practical in situations where limited resources prevented the implementation of a full assessment program. This article summarizes a secondary analysis of data from the development phase of the DEMCAD project to investigate the utility of using radiology data alone to predict overall assessment scores.
Methods The DEMCAD dental office assessment instrument was developed with the assistance of a panel of eight
prominent dental practitioners (5,6).The result was a 248-item survey instrument, designed and intended to assess the quality of practice as defined by leading dental practitioners. When the assessment instrument had been developed and refined, it was tested for use in a nationwide, nonrandom, volunteer sample of 300 dental offices. The purpose of this test was to examine the feasibilityof using the proposed assessment instrument in the dental office setting. Each participating dental office was visited by one evaluator from a cadre of 10 geographically distributed general practitioner evaluators. These evaluators were trained and calibrated for six days at the beginning of the assessment and again after two years. Levels of comparability between pairs of evaluators visiting five offices during calibration exceeded 96 percent (5,6).Completion of the entire quality assessment instrument for a single dental office required about a six-hour visit by an evaluator. The structure dimension contained nine items on radiographic equipment: x-ray equipment in good repair, high-speed timer present, adequate shielding present, size of x-ray area adequate, two x-ray units and a panoramic machine present, automatic processor present, viewboxes present at chairside, written radiology policies available, and radiation monitoring badges used. The process dimension contained 13 items on radiographic procedures and practices (Table 1).There were
TABLE 1 Scores on Oral Radiology Process Items in DEMCAD Dental Practice Quality Assessment Instrument Practices Correct in 5 of 5 Records
Policy concerns Films taken at correct frequency All x-ray films dated Most current films mounted Radiographictechnique Density/contrast of exposure Processing technique Angulation of beam Tissue coverage on films Interproximals of teeth shown Tooth-bearing areas shown All intercrestal areas shown All periapical areas shown No missed radiographic diagnoses Caries diagnoses Bone lesion diagnoses Periodontal diagnoses
n
%
219 206 199
69 66
73
82 77 34
27
44 35
15
31 15
10 5
178
59
102 69
34 23
26 11
12
Journal of Public Health Dentistry
266
no radiographic items in the outcome category. Since the data were collected as part of a larger overall evaluation of the quality assessment instrument, the items included may not be fully descriptive of all aspects of the radiographic procedures employed in dental practice. DEMCAD instrument items that pertained to radiographic equipment or general radiologic practices required only a nominal yes or no response. Scores for instrument items that assessed radiographic policies, radiographic techniques, and tissue coverage of films were determined froma review of five patient charts fromeach office. To qualify for inclusion in the study, each of these five charts chosen nonrandomly by the evaluator had to represent a patient who had at least 22 teeth and had received extensive restorative treatment. For each of the radiology criteria, the office was scored on a range of 1 to 5, depending upon the number of records out of five that met the criterion. To assess the quality of the radiographic interpretation, the full-mouth series from the five patient charts were reviewed for pathology and compared with notations in the chart. Any carious, periodontal, or bone lesion noted in the radiographs by the evaluator that was not accounted for in the record was scored as a missed diagnosis. If no films were available, a zero score for diagnosis was recorded. In the determination of practice quality scores, the DEMCAD instrument is designed to attribute disproportionate weights to various aspects of the instrument. For example, the panel of eight experts who helped design the instrument determined that the radiographic items should be given greater weight than many other items. There were also selective weights assigned within the radiology items. However, in this analysis, all factor weights were removed, so that all items had equal opportunity to contribute to the overall score. SAS software was used for all data analyses. Stepwise multiple linear regression models were constructed to determine which combination of radiographic items resulted in the most parsimonious models for predicting the overall structure Score and the overall process score. Then discriminant function analyses were used to investigate the sensitivity and specificity of these models in predicting the practices with the lowest overall structure and process scores. Unequal variances were assumed and tested by the SAS procedure. Analyses were directed toward prediction of the 20 percent of the practices with the lowest scores and the 10 percent with the lowest scores. To validate the regression equation in terms of predictive ability in other populations, a double cross-validation was completed. The data set was randomly divided into two subsets and separate equations were developed for each subset. The same predictor variables were identified in each corresponding, but separately developed, model. Corresponding models also had similar sensitiv-
ities and specificities. It should be noted that the 20 percent and 10 percent cutoffs for practice scores were purely arbitrary, and merely meant these practices had the lowest Scoreson the DEMCAD assessment. Since the data were collected during a feasibility study and there were no established ranges of acceptable or unacceptable quality scores, having a score below a particular cutoff does not necessarily mean that these practices were substandard. Results Stepwise regression analyses showed that three of the nine radiology structure items (two x-ray units and a panoramic machine present, automatic processor present, and viewboxes present at chairside) were significantly correlated with the overall structure score. However, in terms of potential predictive ability, this threevariable model had a cumulative R2 of only .29. Neither this three-variablemodel nor the full nine-variable model predicted the overall structure scores very well. So the radiology items in the structure dimension of the DEMCAD assessment were not accurate in predicting the overall structure scores for the practices. The remainder of the analysis focused on attempts to predict the process dimension of the assessment, using the 20 percent and 10 percent cutoffs. Descriptive statistics for the radiology process items in the participating dental offices have been described previously (7)and are summarized in Table 1. The frequency distribution of the overall process scores is shown in Table 2. Attempts to develop prediction models for identifylng practices in the lowest 20 percent of the overall process scores ylelded models with modest R-squares, sensitivities, and specificities. These parameters were much greater for models predicting practices below the 10 percent cutoff.
TABLE 2 Distribution of Overall Process Quality Scores in ParticipatingDental Offices
Dental Off ices Process Score
n
Less than 350 350-399
12 23
4 8
400-449 450-499 500-549 550-599 600-649
53 57 55 51
18 19 18
650-699 700 or more Total
18 5
17 9 8 2
300
100
26
55
267
Vol. 52, No. 5, Fall 1992
TABLE 3 Discriminant Function Analysis of Overall DEMCAD Process Scores as Predicted by All 13 Radiology Items Predicted Process Score
Actual Process Score Lowest 10%
Higher 90%
Total
28 2 30
14 256 270
42 258 300
Lowest 10% Higher 90% Total
Sensitivity for 10%lowest scores=28/30=93%. Specificity for 90% higher scores=256/270=95%.
TABLE 4 Stepwise Regression Model from 13 Radiology Process Items to Identify Best Predictors of Overall DEMCAD Process Scores
percent. The results of a stepwise regression to determine which subset of these 13 items could predict overall process scores without undue decreases in sensitivity and specificity is shown in Table 4.The best predictors, in order of decreasing contribution to the prediction model, were: all periodontal conditions noted on the record, all interdental bone shown on the films, all carious lesions noted, and most current films mounted. The four-variable model had a cumulative R2 of .58. Table 5 shows the results of the discriminant analysis that used only these four radiographic process items to predict the overall process score. The sensitivity of these four items in identifying the 10 percent of the practices with lowest overall process scores was 87 percent. This was a decrease of 6 percent from the 13-item model. The specificityof these four items in identifying the90percent of the practices with higher scores was 93 percent. This was a decrease of 2 percent from the 13-item model.
~
Radiology Process Item Entered in Regression Model All periodontal conditions noted All interdental bone shown All carious lesions noted Most current films mounted
Beta
Cumulative R2
P
22.19
.47
c.0001
12.57 11.22 8.84
.53 .56 .58
c.0001
c.0005
Journal of Public Health Dentistry
Assessing Dental Practice Quality by Evaluating Radiology Items Ronald J. Hunt, DDS, MS Shen Jang Fann, MS Preventive and Community Dentistry College of Dentistry University of Iowa Iowa City, IA
Me1 L. Kantor, DDS Oral and Maxillofacial Radiology School of Dental Medicine University of Connecticut Farmington, CT
Alvin L. Morris, DDS, PhD Department of Dental Ecology School of Dentistry University of North Carolina Chapel Hill, NC
Abstract The DEMCAD dental office assessment instrument was developed to evaluate practice quality using Donebedian 's quality assessment model of structure, process, and outcome. This previously validated instmment takes about six hours to complete. Subsequent analysis was undertaken to determine whether an abbreviated office assessment based on the evaluation of radiology items was sufficiently sensitive, specific, and practical to be used as a screening instrument for identifying dental offices with very low evaluationscores. Data for this analysis were obtained from 300 volunteer general dental practices evaluated in the field testing of the DEMCAD instrument. The nine radiology structure items predicted very poorly the overall Structure scores. However, 73 radiology process items predicted overall process scores quite accurately. Four of the 73 radiology process items (periodontaldiagnosesrecorded,interdentalbonesho wnon x-rays, caries diagnosesrecorded, and current x-rays mounted)produced a combined R2 Of .58. These four radiology variables predicted the 70 percent of the dental practices with the lowest overall process score with 87percent sensitivity and 93 percent specificity. This analysis showed that an abbreviated dental practice process quality assessment using oral radiology Send correspondenceand reprint requests to Dr. Hunt, Department of Dental Ecology, CB #7450, School of Dentistry, University of North Carolina, Chapel Hill, NC 27599. Manuscript received: 9/17/90; returned to authors for revision: 10/31/90; accepted for publication: 2/17/92.
J Public Health Dent 1992;52(5):264-8
items in an audit of patients' records may be feasible as a screening test for dental office assessment. Key Words: dental health services, quality assurance, oral radiology. The provision of quality dental care has long been a primary goal of the dental profession. More recently, the public also has become more active in demandingquality of care, although the public's interest in quality is often confused with an interest in cost containment. What was once a matter of ethics within the professionhas evolved into a need to respond also to pressures for quality assurance from external forces (1).As a result, various systems and criteria have been investigated for potential use in assessing the quality of dental practices. Most systems for quality assessment in dentistry have been based on the dimensions of structure, process, and outcome as described for evaluation of care provided in medical and hospital settings by Donabedian (2) and adapted to dental settings by Bailit (3). The structure dimensionincludes such facetsof dental practiceas licensure, facilities, equipment, organization, and personnel. The process dimension includes evaluationof the results of the procedures and tasks the dentist performs, such as thorough record keeping, fully developed treatment plans, compliancewith infectioncontrol procedures, creation of good diagnostic radiographs, and accurateinterpretation of the radiographs. The outcome dimension
265
Vol. 52, No. 5, Fall 1992
includes direct evaluation of the care the patient receives, such as patient satisfaction, technical quality of restorations, oral health status of the patient, and completion of treatment. It is generally recognized that the measurement of outcomes of care among patients is preferable to the measurement of processes within the practice itself (4). However, outcomes of care typically are difficult to measure (3). Moreover, outcome assessment requires the direct examination of a number of patients, which greatly increases the monetary costs and the potential psychological costs of the assessment to the patients and the dental practices. Recently developed practice assessments have tended to focus on the process dimension. In 1982 the W. K. Kellogg Foundation funded a largescale dental practice assessment project entitled “Development of Evaluation Methods and Computer Applications in Dentistry (DEMCAD).”One of the primary goals of this project was to develop an objective, practical, and professionally acceptable dental practice assessment method using in-office visits (5,6).The DEMCAD quality assessment instrument, when used as designed, provides a comprehensive evaluation of a dental office structure, process, and outcome. However, the full office assessment requires six hours to complete and it may not always be economically feasible for a given evaluation program to conduct the full assessment. The DEMCAD instrument has not yet been fully validated through use in additional samples of dental practices. Thus, the findings of this analysis cannot be generalized beyond the measures contained in the DEMCAD assessment. This analysis merely explores whether the DEMCAD itself could be abbreviated and still identify the lowest scoring practices as identified by the entire DEMCAD instrument. If so, evidence would be provided that suggests that it may possible to construct shorter dental practice assessment methods. A previous evaluation of the radiographic equipment and procedures in the 300 dental offices volunteering to participate in the development phase of the DEMCAD project subsequently led to the question of whether an abbreviated office assessmentbased on the evaluation of the oral radiology items alone could be used as a screening instrument for identifying dental offices that had very low overall DEMCAD practice assessment scores. An abbreviated assessment, if sufficiently sensitive and specific, could be more practical in situations where limited resources prevented the implementation of a full assessment program. This article summarizes a secondary analysis of data from the development phase of the DEMCAD project to investigate the utility of using radiology data alone to predict overall assessment scores.
Methods The DEMCAD dental office assessment instrument was developed with the assistance of a panel of eight
prominent dental practitioners (5,6).The result was a 248-item survey instrument, designed and intended to assess the quality of practice as defined by leading dental practitioners. When the assessment instrument had been developed and refined, it was tested for use in a nationwide, nonrandom, volunteer sample of 300 dental offices. The purpose of this test was to examine the feasibilityof using the proposed assessment instrument in the dental office setting. Each participating dental office was visited by one evaluator from a cadre of 10 geographically distributed general practitioner evaluators. These evaluators were trained and calibrated for six days at the beginning of the assessment and again after two years. Levels of comparability between pairs of evaluators visiting five offices during calibration exceeded 96 percent (5,6).Completion of the entire quality assessment instrument for a single dental office required about a six-hour visit by an evaluator. The structure dimension contained nine items on radiographic equipment: x-ray equipment in good repair, high-speed timer present, adequate shielding present, size of x-ray area adequate, two x-ray units and a panoramic machine present, automatic processor present, viewboxes present at chairside, written radiology policies available, and radiation monitoring badges used. The process dimension contained 13 items on radiographic procedures and practices (Table 1).There were
TABLE 1 Scores on Oral Radiology Process Items in DEMCAD Dental Practice Quality Assessment Instrument Practices Correct in 5 of 5 Records
Policy concerns Films taken at correct frequency All x-ray films dated Most current films mounted Radiographictechnique Density/contrast of exposure Processing technique Angulation of beam Tissue coverage on films Interproximals of teeth shown Tooth-bearing areas shown All intercrestal areas shown All periapical areas shown No missed radiographic diagnoses Caries diagnoses Bone lesion diagnoses Periodontal diagnoses
n
%
219 206 199
69 66
73
82 77 34
27
44 35
15
31 15
10 5
178
59
102 69
34 23
26 11
12
Journal of Public Health Dentistry
266
no radiographic items in the outcome category. Since the data were collected as part of a larger overall evaluation of the quality assessment instrument, the items included may not be fully descriptive of all aspects of the radiographic procedures employed in dental practice. DEMCAD instrument items that pertained to radiographic equipment or general radiologic practices required only a nominal yes or no response. Scores for instrument items that assessed radiographic policies, radiographic techniques, and tissue coverage of films were determined froma review of five patient charts fromeach office. To qualify for inclusion in the study, each of these five charts chosen nonrandomly by the evaluator had to represent a patient who had at least 22 teeth and had received extensive restorative treatment. For each of the radiology criteria, the office was scored on a range of 1 to 5, depending upon the number of records out of five that met the criterion. To assess the quality of the radiographic interpretation, the full-mouth series from the five patient charts were reviewed for pathology and compared with notations in the chart. Any carious, periodontal, or bone lesion noted in the radiographs by the evaluator that was not accounted for in the record was scored as a missed diagnosis. If no films were available, a zero score for diagnosis was recorded. In the determination of practice quality scores, the DEMCAD instrument is designed to attribute disproportionate weights to various aspects of the instrument. For example, the panel of eight experts who helped design the instrument determined that the radiographic items should be given greater weight than many other items. There were also selective weights assigned within the radiology items. However, in this analysis, all factor weights were removed, so that all items had equal opportunity to contribute to the overall score. SAS software was used for all data analyses. Stepwise multiple linear regression models were constructed to determine which combination of radiographic items resulted in the most parsimonious models for predicting the overall structure Score and the overall process score. Then discriminant function analyses were used to investigate the sensitivity and specificity of these models in predicting the practices with the lowest overall structure and process scores. Unequal variances were assumed and tested by the SAS procedure. Analyses were directed toward prediction of the 20 percent of the practices with the lowest scores and the 10 percent with the lowest scores. To validate the regression equation in terms of predictive ability in other populations, a double cross-validation was completed. The data set was randomly divided into two subsets and separate equations were developed for each subset. The same predictor variables were identified in each corresponding, but separately developed, model. Corresponding models also had similar sensitiv-
ities and specificities. It should be noted that the 20 percent and 10 percent cutoffs for practice scores were purely arbitrary, and merely meant these practices had the lowest Scoreson the DEMCAD assessment. Since the data were collected during a feasibility study and there were no established ranges of acceptable or unacceptable quality scores, having a score below a particular cutoff does not necessarily mean that these practices were substandard. Results Stepwise regression analyses showed that three of the nine radiology structure items (two x-ray units and a panoramic machine present, automatic processor present, and viewboxes present at chairside) were significantly correlated with the overall structure score. However, in terms of potential predictive ability, this threevariable model had a cumulative R2 of only .29. Neither this three-variablemodel nor the full nine-variable model predicted the overall structure scores very well. So the radiology items in the structure dimension of the DEMCAD assessment were not accurate in predicting the overall structure scores for the practices. The remainder of the analysis focused on attempts to predict the process dimension of the assessment, using the 20 percent and 10 percent cutoffs. Descriptive statistics for the radiology process items in the participating dental offices have been described previously (7)and are summarized in Table 1. The frequency distribution of the overall process scores is shown in Table 2. Attempts to develop prediction models for identifylng practices in the lowest 20 percent of the overall process scores ylelded models with modest R-squares, sensitivities, and specificities. These parameters were much greater for models predicting practices below the 10 percent cutoff.
TABLE 2 Distribution of Overall Process Quality Scores in ParticipatingDental Offices
Dental Off ices Process Score
n
Less than 350 350-399
12 23
4 8
400-449 450-499 500-549 550-599 600-649
53 57 55 51
18 19 18
650-699 700 or more Total
18 5
17 9 8 2
300
100
26
55
267
Vol. 52, No. 5, Fall 1992
TABLE 3 Discriminant Function Analysis of Overall DEMCAD Process Scores as Predicted by All 13 Radiology Items Predicted Process Score
Actual Process Score Lowest 10%
Higher 90%
Total
28 2 30
14 256 270
42 258 300
Lowest 10% Higher 90% Total
Sensitivity for 10%lowest scores=28/30=93%. Specificity for 90% higher scores=256/270=95%.
TABLE 4 Stepwise Regression Model from 13 Radiology Process Items to Identify Best Predictors of Overall DEMCAD Process Scores
percent. The results of a stepwise regression to determine which subset of these 13 items could predict overall process scores without undue decreases in sensitivity and specificity is shown in Table 4.The best predictors, in order of decreasing contribution to the prediction model, were: all periodontal conditions noted on the record, all interdental bone shown on the films, all carious lesions noted, and most current films mounted. The four-variable model had a cumulative R2 of .58. Table 5 shows the results of the discriminant analysis that used only these four radiographic process items to predict the overall process score. The sensitivity of these four items in identifying the 10 percent of the practices with lowest overall process scores was 87 percent. This was a decrease of 6 percent from the 13-item model. The specificityof these four items in identifying the90percent of the practices with higher scores was 93 percent. This was a decrease of 2 percent from the 13-item model.
~
Radiology Process Item Entered in Regression Model All periodontal conditions noted All interdental bone shown All carious lesions noted Most current films mounted
Beta
Cumulative R2
P
22.19
.47
c.0001
12.57 11.22 8.84
.53 .56 .58
c.0001
c.0005
