The Cleft Palate–Craniofacial Journal 53(1) pp. 30–37 January 2016 Ó Copyright 2016 American Cleft Palate–Craniofacial Association
ORIGINAL ARTICLE The Americleft Project: A Proposed Expanded Nasolabial Appearance Yardstick for 5- to 7-Year-Old Patients With Complete Unilateral Cleft Lip and Palate (CUCLP) A.M. Mercado, D.M.D., Ph.D., K.A. Russell, D.D.S., M.Sc., J. Daskalogiannakis, D.D.S., M.Sc., F.R.C.D(C), R.R. Hathaway, D.D.S., M.S., G. Semb, D.D.S., Ph.D., T. Ozawa, D.D.S., A. Smith, D.D.S., A.Y. Lin, M.D., R.E. Long Jr., D.M.D., M.S., Ph.D. Objective: To develop a yardstick of reference photographs for nasolabial appearance assessments of 5- to 7-year-old patients with complete unilateral cleft lip and palate (CUCLP). Design: Blind retrospective analysis of clinical records and comparison to historical controls. Patients: Subjects were two groups of 6- to 12-year-olds (n ¼ 124 and n ¼ 135) and one group of 5- to 7-year-olds (n ¼ 149) with nonsyndromic CUCLP from three previous Americleft studies, including cohorts from seven different cleft/craniofacial centers. Interventions: All patients received the infant management protocols of their respective centers. Eleven trained and calibrated judges (five participated in all three studies) did blind ratings of nasolabial appearance using the Asher-McDade method. Main Outcome Measures: Patients receiving the most consistent ratings between judges, selected first from the groups of 6- to 12-year-olds, were used to create a pilot yardstick for eventual use in the third study of 5- to 7-year-olds. For each of the Asher-McDade categories, 8 of the 5- to 7-year-old patients receiving the most consistent scores between raters were ranked by 10 judges for a final elimination to leave three per category. Results: Using this method of successive changes in rating methods, a new reference yardstick for nasolabial appearance rating was established and linked to the original AsherMcDade method as well as the single examples in a previously published yardstick for patients with CUCLP. Pilot testing using the yardstick improved reliabilities. Conclusions: Use of an expanded nasolabial yardstick of reference photographs representative of the range of possibilities of each of the five Asher-McDade categories is now available to see if reliability of these ratings can be improved. KEY WORDS:
CUCLP, nasolabial appearance, yardstick
Intercenter outcome comparisons have increased value if the outcomes being compared are identical and a common benchmark is used. For outcomes using ordinal scales for ratings, the Goslon Yardstick for dental arch relationships in patients with complete unilateral cleft lip and palate (CUCLP) (Mars et al., 1987) represents one of the best examples of this concept. Used most notably in the original Eurocleft study (Mars et al., 1982), the availability of a reference set of dental models representative of the five rating categories of dental arch relationship (1 ¼ best to 5 ¼ worst) contributed to the very high inter- and intrareliability of the ratings. Furthermore, with the countless additional intercenter comparisons of dental arch relationship that followed, using the same reference yardstick has enabled valid comparison of results between studies (Hathaway et al., 2013).
Dr. Mercado is Clinical Assistant Professor of Orthodontics, Division of Orthodontics, The Ohio State University, Columbus, Ohio. Dr. Russell is Professor and Head, Department of Orthodontics and Cleft Palate Team, Dalhousie University/IWK Health Centre, Halifax, Nova Scotia, Canada. Dr. Daskalogiannakis is Staff Orthodontist, SickKids Hospital, and Associate Professor, Department of Orthodontics, University of Toronto, Toronto, Ontario, Canada. Dr. Hathaway is Professor, Division of Craniofacial Plastic Surgery, Cincinnati Children’s Hospital, Cincinnati, Ohio. Dr. Semb is Professor, University of Manchester, Manchester, United Kingdom. Dr. Ozawa is Orthodontist, Hospital de Reabilitaca ¸ ˜ o de Anomalias Craniofaciais Universidade de Sa˜o Paulo, Bauru, Brazil. Dr. Smith is Clinical Director and Maxillofacial Prosthodontist, Lancaster Cleft Palate Clinic, Lancaster, Pennsylvania. Dr. Lin is Director, St. Louis Cleft-Craniofacial Center, SSM Cardinal Glennon Children’s Medical Center, St. Louis University, St. Louis, Missouri. Dr. Long is Executive Director, Lancaster Cleft Palate Clinic, Lancaster, Pennsylvania. Presented at the 12th International Congress on Cleft Lip/Palate and Related Craniofacial Anomalies, Orlando, Florida, May 5–10, 2013. Submitted February 2014; Revised September 2014; Accepted September 2014. Address correspondence to: Dr. Ross E. Long Jr., Lancaster Cleft Palate Clinic, 223 North Lime Street, Lancaster, PA 17602. E-mail [email protected].
DOI: 10.1597/14-017 30
Mercado et al., NASOLABIAL APPEARANCE YARDSTICK FOR 5- TO 7-YEAR-OLDS WITH CUCLP
Yardsticks of reference photographs for nasolabial appearance assessments have not been as helpful as the Goslon Yardstick and may be one of the reasons contributing to poorer reliability scores. Facial appearance is one of the most relevant measures of success of cleft lip and palate treatment (Asher-McDade et al., 1991). Having a method to reliably assess the nasolabial appearance outcomes of various treatment protocols is critical. Residual anomalies in nasal form, nasal asymmetry, and distortion of the upper lip (Asher-McDade et al., 1991) may affect self-perception, self-esteem, and social interactions. Currently, there is no widely accepted standard rating method to assess facial aesthetics in patients with CUCLP that has demonstrated a high degree of reliability (Nollet et al., 2007). The ability to evaluate and compare the impact of various treatment protocols on nasolabial appearance relies on having a reliable and valid outcome measure. Outcome comparisons of facial appearance are especially difficult because of the subjective nature of the assessment. Use of two-dimensional photography has been the most common method upon which ratings are based because of convenience, cost, and common usage among centers as part of routine record-taking. Several different scoring or rating systems have been developed. The Asher-McDade method (Asher-McDade et al., 1991) is one of the most commonly used. Similar to the Goslon Yardstick, this method was based on an ordinal scale of rive categories of outcomes (1 ¼ best to 5 ¼ worst) for four different nasolabial appearance features: nasal form, nasal symmetry, nasolabial profile, and vermilion border. After the method was first proposed, it was also subsequently used in the landmark Eurocleft study (AsherMcDade et al., 1992), with an overall reliability of 0.60 using the weighted Kappa statistic. More recently, using the same method, two studies from the Americleft Project reported an average intrarater reliability of 0.739 and an average interrater reliability of 0.645 in the first study (Mercado et al., 2011) and an average intrarater reliability of 0.648 and an average interrater reliability of 0.485 in the second study (Mercado et al., 2012). While reliabilities of 0.61 to 0.80 are considered good agreement (Altman, 1991), a rating method with reliability in the excellent range would allow for stronger and more accurate evaluations of nasolabial appearance outcomes. The lack of a reference scale such as the Goslon Yardstick, which includes multiple examples of each of its five categories, might also contribute to the low reliability for nasolabial appearance ratings. A study by KuijpersJagtman et al. (2009) presented reference photographs to illustrate a single example of each of the five categories for each of the four views of the Asher-McDade rating scale (20 photographs in total). The 2011 Americleft study (Mercado et al., 2011) utilized the Kuijpers-Jagtman reference photographs but did not report a significant increase in reliability scores (Mercado et al., 2011). However, given the wide range of nasolabial appearances in children with clefts,
TABLE 1 Yardstick
31
Summary of Studies Used in Developing the Nasolabial
No. Mean age (y) Age range (y)
Study 1 (Mercado et al., 2011)
Study 2 (Mercado et al., 2012)
Study 3 (Daskalogiannakis et al., 2013)
124 8.4 6–12
135 8.6 6–12
149 5.5 5–7
having only one reference photograph for each category makes it difficult to identify the appropriate rating for an individual patient. In contrast, the Goslon Yardstick has been shown to lead to consistently better reliability. Of significance is the fact that the Goslon Yardstick includes multiple examples of cases representative of the range of possibilities in each category, and these are available at rating sessions to view in comparison to a particular patient being rated. Therefore, the purpose of this study was to identify cases from previous studies for which there was the greatest amount of agreement between raters and from that subset to choose cases that represented the range of possibilities within a category to represent a new expanded nasolabial yardstick for future ratings. In addition, in an attempt to maintain some continuity and comparability with the previous Eurocleft and Americleft efforts, the age ranges and stages of development that were determinants in the original outcome comparison studies for both dental arch relationship (Goslon) and nasolabial appearance (AsherMcDade) were followed (7- to 12-year-olds in mixed dentition to early permanent dentition). With a trend toward comparing outcomes at a younger age and the development of a Goslon-type dental arch relationship yardstick for 5- to 6-year-olds (late primary to early mixed dentition) (Atack et al., 1997) and so the nasolabial yardstick could be used in the comparison of samples of younger patients (5- to 7-year-olds), an additional purpose was to include this age range in creating the expanded yardstick. MATERIALS
AND
METHODS
Data from three consecutive studies of nasolabial appearance ratings by the Americleft group (Mercado et al., 2011; Mercado et al., 2012; Daskalogiannakis et al., 2013) were used in a sequential process of identifying patients rated in these studies who received unanimous or near unanimous agreement between raters for their scores. These studies are summarized in Table 1. Study 1 comprised 6- to 12-year-old patients (n ¼ 124; mean age ¼ 8.4 years) with CUCLP, study 2 comprised similarly aged patients (6 to 12 years old) with CUCLP (n¼ 135; mean age ¼8.6 years), and study 3 comprised 5- to 7-year-old patients (n ¼ 149; mean age ¼ 5.5 years) with CUCLP. Study 1 and study 2 used the Asher-McDade method for nasolabial
32
Cleft Palate–Craniofacial Journal, January 2016, Vol. 53 No. 1
FIGURE 1 Vermilion Border rating 1 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 2 Vermilion Border rating 2 (lower left with permission from Kuijpers-Jagtman et al., 2009).
appearance ratings (Asher-McDade et al., 1991), similar to the original Eurocleft assessment of nasolabial appearance (Asher-McDade et al.,1992). The first two Americleft studies used the same reference photographs used in the Eurocleft study (Mercado et al., 2011; Mercado et al., 2012). There was just one example of each scoring category for each view to be used as a reference photograph. From the 259 patients in study 1 and study 2, cases were selected from the Americleft samples that received the most consistent and unanimous or closest to unanimous scores from all raters; these were used to create a new set of reference photographs. Obviously, unanimous agreement was desired, that is, all raters scoring a case identically. When sufficient unanimously rated cases were not available to complete the desired number of reference photos, additional reference cases were chosen based on those in which the mean score from all raters was closest to unanimous. Thus, a final set of four reference photos of 6to 12-year-old patients for each of the five rating categories for each of the four views (80 total) was selected from the total pool of 259 patients from these initial two studies. The third intercenter study of nasolabial appearance involved evaluation of 149 patients with CUCLP with an age range of 5 to 7 years (Daskalogiannakis et al., 2013). For this study, the only reference photos available for this age range were those in a publication by Kuijpers-Jagtman et al. (2009). Because this publication included only one yardstick reference photo per scoring category, we thought better reliability might be possible if there were multiple reference photographs representing a range of possibilities for each score for each of the four views. Therefore, the four reference photographs for each scoring category and each view that were selected from the original two Americleft studies were added to the new reference yardstick along with the single reference photograph from the KuijpersJagtman (2009) publication. The Kuijpers-Jagtman photograph was included to ensure continuity with the previous
Eurocleft study (Asher-McDade et al., 1992) and the reference photos created by Kuijpers-Jagtman et al. (2009) for use with the Asher-McDade method by including a common reference photograph for each of the scores of the Asher-McDade categories. Note, however, that the new yardstick was being used to rate a sample of 5- to 7-yearolds. To the single Kuijpers-Jagtman reference photographs of 5- to 7-year-olds were added the additional reference photographs from the samples of 6- to 12-yearolds. This was done with the assumption that with no other reference photographs available for 5- to 7-year-olds to expand the Kuijpers-Jagtman yardstick, the age difference would not be visible in the cropped images versus the fullface images, and the deviations in form affecting scoring would be the same in both age groups. Although this assumption cannot be validated, there is likewise no published evidence that a 3-year mean age difference in early to mid-childhood would represent a significant confounder in identifying and discriminating between good and poor nasolabial appearance results. Two other significant modifications were made to the traditional Asher-McDade method that were incorporated into this Americleft study of 5- to 7-year-olds with CUCLP. First, in order to shorten and simplify the ratings in this study, the Asher-McDade categories of nasolabial form and nasolabial symmetry, each evaluated separately in the frontal view, were combined into one category called nasal frontal to be consistent with the single category of nasolabial profile. The merging of the two categories allowed for an overall assessment of the aesthetics from the frontal view. In addition, one less category results in faster completion of the ratings and less rater fatigue, which may contribute to lower inter- and intrarater reliability. The additional reference photographs for the new nasolabial frontal view were chosen from the two previous Americleft studies from those patients showing the closest agreement
Mercado et al., NASOLABIAL APPEARANCE YARDSTICK FOR 5- TO 7-YEAR-OLDS WITH CUCLP
33
FIGURE 3 Vermilion Border rating 3 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 5 Vermilion Border rating 5 (lower left with permission from Kuijpers-Jagtman et al., 2009).
between raters for the nasolabial symmetry and nasolabial form categories merged. The second modification was the use of a modified Q-sort method (Daskalogiannakis et al., 2013; Long et al., 2013) rather than the traditional method of PowerPoint projection of one case at a time and simultaneous rating of all views. The Q-sort method was introduced by Stephenson (1953) to provide a foundation for the systematic study of subjective objects (Brown, 1993). Each object is judged relative to the other objects in the sample. As applied to this investigation, with the Q-sort method raters are provided with their own pack of cards with the cropped views of each patient on separate cards. The cards are then sequentially separated into five piles representative of the five categories of outcomes. Once completed the raters are able to reevaluate their choices by comparing the patients within and between the piles.
From this rating, a subset of 85 from the entire sample of 149 was generated by the selection of 5 to 8 patients in each category for whom there was the greatest agreement between raters and the average scores were exactly or closest to the category number. At a separate session 10 experienced judges (seven orthodontists, two surgeons, and one prosthodontist) rank-ordered the cases from best to worst within each category and for each of the three features being assessed (nasolabial profile, nasal frontal and vermilion border). Based on an average rank and the most consistent rank across raters, three cases were selected for each rated feature and for each category. These were the case ranked as best for that category, the case ranked lowest, the case that represented the median rank. To maintain continuity and commonality with the previous published yardstick, the single representative case from the Kuijpers-Jagtman (2009) published yardstick was combined with these three examples for each feature and
FIGURE 4 Vermilion Border rating 4 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 6 Nasal Frontal rating 1 (lower left with permission from Kuijpers-Jagtman et al., 2009).
34
Cleft Palate–Craniofacial Journal, January 2016, Vol. 53 No. 1
FIGURE 7 Nasal Frontal rating 2 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 9 Nasal Frontal rating 4 (lower left with permission from Kuijpers-Jagtman et al., 2009).
category to create an expanded yardstick for nasolabial appearance for 5- to 7-year-olds with a range of representative cases within each group. This new expanded nasolabial appearance yardstick for 5- to 7-year-olds with CUCLP was then pilot-tested on a small (n ¼ 30) sample of 5- to 7-year-old patients with CUCLP. The modified Q-sort method was again used because raters from the previous study unanimously preferred this method over the traditional PowerPoint method. The panel of seven raters for this pilot study comprised four orthodontists, two surgeons, and one prosthodontist. All 30 cases were rated twice on two separate days. Intra- and interrater reliability were tested using the weighted Kappa statistic. Intrarater reliability was calculated by comparing each rater’s first ratings to their second ratings. Interrater reliability was calculated by comparing each rater to the others for both ratings. Reliabilities were then compared to all three previous
Americleft studies to see if the expanded yardstick was associated with improvement.
FIGURE 8 Nasal Frontal rating 3 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 10 Nasal Frontal rating 5 (lower left with permission from Kuijpers-Jagtman et al., 2009).
RESULTS Figures 1 through 15 (available online in color) show the new expanded yardstick for 5- to 7-year-olds with CUCLP for each rating category (one through five) for each of the three features being rated: vermilion border, nasal frontal, and nasolabial profile. The lower left reference photo in each figure is taken from the original publication of reference photos for patients with CUCLP (KuijpersJagtman et al., 2009, with permission from the Journal of Craniofacial Surgery). The full-color yardstick is available for download for use in outcome comparisons from the Americleft Task Force Study Guide, which can be accessed through the Research tab on the website for the American
Mercado et al., NASOLABIAL APPEARANCE YARDSTICK FOR 5- TO 7-YEAR-OLDS WITH CUCLP
35
FIGURE 11 Nasolabial Profile rating 1 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 13 Nasolabial Profile rating 3 (lower left with permission from Kuijpers-Jagtman et al., 2009).
Cleft Palate–Craniofacial Association website (www. acpa-cpf.org). Table 2 compares the average intra- and interrater Kappa values for the three previous Americleft nasolabial appearance studies (Mercado et al., 2011; Mercado et al., 2012; Daskalogiannakis et al., 2013) as well as the pilot study carried out as part of this report. Table 3 lists the interpretation of the Kappa statistic (Altman, 1991).
This report provides, for the first time, a nasolabial yardstick for 5- to 7-year-old patients with CUCLP that consists of an expanded number of reference photographs indicative of a range of possibilities for each of the five rating categories for each feature. The expanded yardstick was attempted for the purpose of improving the reliability of the ratings for use in intercenter outcome comparisons.
Because of the extreme subjectivity of nasolabial appearance ratings, in many previous investigations the reliability of the scoring between and within raters has always been noticeably poorer than that of comparisons of dental arch relationships using the Goslon Yardstick (Mars et al., 1987). One reason for this may be the fact that the Goslon Yardstick contains several examples of differing appearances for cases rated in each of the five scoring categories. With additional examples against which to compare a case being scored, the raters have a greater likelihood of finding similar-appearing reference cases to match the one being scored. As it is likely that variations in nasolabial appearance and the additional subjectivity in rating these exceed what would be encountered in a dental arch relationship study, having only one reference case available for each of the nasolabial appearance categories would logically only increase the subjectivity of the rating.
FIGURE 12 Nasolabial Profile rating 2 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 14 Nasolabial Profile rating 4 (lower left with permission from Kuijpers-Jagtman et al., 2009).
DISCUSSION
36
Cleft Palate–Craniofacial Journal, January 2016, Vol. 53 No. 1
TABLE 3
Interpretation of Kappa Values
Kappa Value
Strength of Agreement
,0.20 0.21–0.40 0.41–0.60 0.61–0.80 0.80–1.00
Poor Fair Moderate Good Very Good
increasing the numbers of samples involved in the comparison. CONCLUSIONS
FIGURE 15 Nasolabial Profile rating 5 (lower left with permission from Kuijpers-Jagtman et al., 2009).
The results of the pilot study using the new yardstick are encouraging, this being the first of the Americleft studies where intrarater reliability reached the highest level of agreement using the Kappa statistic, while moving interrater agreement from moderate to good. Clearly, the use of the Q-sort method may also have played a role in the improved agreement, but regardless, all raters in the Daskalogiannakis et al. (2013) study and the current pilot study were unanimous in their preference of this over the previous PowerPoint slide-presentation method. Additionally, although merging the nasolabial symmetry and nasolabial form categories into one feature, (nasal frontal), clearly simplified and shortened the time required to do a Qsort rating, it remains to be seen whether too much useful information is lost by not keeping these features separate. Finally, the use of a common reference photograph from a previous publication for the various features and rating categories is essential if our intercenter outcome comparison studies are going to be suitable for cross-comparisons. The value of the current regional and national initiatives for collaboration between centers willing to compare their outcomes in well-controlled and blinded studies, as has been done in Euroclett and Americleft, is exponentially increased if all studies are using a common benchmark for measuring and scoring. By so doing, the results of studies carried out separately but using the same records; prepared identically; and rated, scored, or measured similarly using the same benchmark reference yardstick can be compared more confidently to one another, thereby significantly TABLE 2 Comparison of Average Kappa Values From Previous Americleft Studies Study 3 Study 2 Study 1 (Mercado (Daskalogiannakis (Mercado et al., 2013) Current Study et al., 2011) et al., 2012) Interrater Intrarater
0.674 0.803
0.645 0.739
0.485 0.648
0.594 0.761
The use of an expanded nasolabial yardstick of reference photographs representative of the range of possibilities in each of the five Asher-McDade categories is now available to see if reliability of these ratings can be improved. Acknowledgments. The authors and Americleft Group would like to acknowledge and thank the American Cleft Palate-Craniofacial Association and the Cleft Palate Foundation for their financial support of the Americleft Task Force. Additional funding was provided by the Trout Family Trust and the Mellinger Medical Research Fund, Lancaster, Pennsylvania.
REFERENCES Altman DG. Practical Statistics for Medical Research. London: Chapman & Hall; 1991:403–409. Asher-McDade C, Brattstrom V, Dahl E, McWilliam J, Molsted K, Plint DA, Prahl-Andersen B, Semb G, Shaw WC. A six center international study of treatment outcome in patients with clefts of the lip and palate: part 4. Assessment of nasolabial appearance. Cleft Palate Craniofac J. 1992:29:409–412. Asher-McDade C, Roberts C, Shaw WC, Gallagher C. The development of a method for rating nasolabial appearance in patients with clefts of the lip and palate. Cleft Palate J. 1991;28:380–390. Atack NE, Hathorn IS, Semb, G, Dowell T, Sandy JR. A new index for assessing surgical outcomes in unilateral cleft lip and palate subjects aged 5—reproducibility and reliability. Cleft Palate Craniofac J. 1997;34:242–246. Brown SR. A primer on Q methodology. Operant Subjectivity. 1993;16:91–138. Daskalogiannakis J, Russell KA, Mercado A, Hathaway RR, Singer E, Stoutland A, Long RE Jr, Semb G, Shaw WC. The Americleft Project: burden of care analysis of various infant orthopedic protocols for improvement of nasolabial aesthetics in patients with CUCLP. Presented at the 12th International Congress on Cleft Lip/ Palate and Related Craniofacial Anomalies; May 2013; Orlando, Florida. Hathaway RR, Long RE Jr, Mercado AM, Daskalogiannakis J, Russell KA, Semb G, Shaw WC, Gregory J. The Americleft Project: use of a standardized outcome measure of dental arch relationships (Goslon) to allow international interstudy comparisons. Presented at the 12th International Congress on Cleft Lip/ Palate and Related Craniofacial Anomalies; May 2013; Orlando, Florida Kuijpers-Jagtman AM, Nollet PJ, Semb G, Bronkhorst EM, Shaw WC, Katsaros C. Reference photographs for nasolabial appearance
Mercado et al., NASOLABIAL APPEARANCE YARDSTICK FOR 5- TO 7-YEAR-OLDS WITH CUCLP
rating in unilateral cleft lip and palate. J Craniofac Surg. 2009;20:1683–1686. Long RE Jr, Mercado A, Hathaway RR, Daskalogiannakis J, Russell KA, Stoutland A, Singer E, Semb G, Shaw WC, Fessler J. The Americleft Project: a modification of Asher-McDade method for rating nasolabial esthetics in patients with CUCLP. Presented at the 12th International Congress on Cleft Lip/Palate and Related Craniofacial Anomalies; May 2013; Orlando, Florida. Mars M, Asher-McDade C, Brattstrom V, Dahl E, McWilliam J, ¨ Mølsted K, Plint DA, Prahl-Andersen B, Semb G, Shaw WC, The RPS. A six-center international study of treatment outcome in patients with clefts of lip and palate: Part 3: Dental arch relationships. Cleft Palate-Craniofac J, 1992;29:405–408. Mars M, Plint DA, Houston WJ, Bergland O, Semb G. The Goslon yardstick: a new system of assessing dental arch relationships in children with unilateral clefts of the lip and palate. Cleft Palate J. 1987;24:314–322.
37
Mercado A, Peanchitlertkajorn S, Daskalogiannakis J, Hathaway R, Lamichane M, Russell K, Semb G, Long RE Jr. The Americleft Project: an expansion of the Americleft intercenter comparisons of nasolabial appearance outcomes to include a center using NAM as part of its primary protocol. Presented at the 68th Annual Meeting of the American Cleft Palate-Craniofacial Association; April 2012; San Juan, Puerto Rico. Mercado A, Russell K, Hathaway R, Daskalogiannakis J, Sadek H, Long RE, Cohen M, Semb G, Shaw WC. The Americleft Study: an inter-center study of treatment outcomes for patients with unilateral cleft lip and palate. Part 4. Nasolabial aesthetics. Cleft Palate Craniofac J. 2011;48:259–264 Nollet PJ, Kuijpers-Jagtman AM, Chatzigianni A, Semb G, Shaw WC, Bronkhorst EM, Katsaros C. Nasolabial appearance in unilateral cleft lip, alveolus and palate: a comparison with Eurocleft. J Craniomaxillofac Surg. 2007;35:278–286. Stephenson W. The Study of Behavior. Chicago: University of Chicago Press; 1953.
ORIGINAL ARTICLE The Americleft Project: A Proposed Expanded Nasolabial Appearance Yardstick for 5- to 7-Year-Old Patients With Complete Unilateral Cleft Lip and Palate (CUCLP) A.M. Mercado, D.M.D., Ph.D., K.A. Russell, D.D.S., M.Sc., J. Daskalogiannakis, D.D.S., M.Sc., F.R.C.D(C), R.R. Hathaway, D.D.S., M.S., G. Semb, D.D.S., Ph.D., T. Ozawa, D.D.S., A. Smith, D.D.S., A.Y. Lin, M.D., R.E. Long Jr., D.M.D., M.S., Ph.D. Objective: To develop a yardstick of reference photographs for nasolabial appearance assessments of 5- to 7-year-old patients with complete unilateral cleft lip and palate (CUCLP). Design: Blind retrospective analysis of clinical records and comparison to historical controls. Patients: Subjects were two groups of 6- to 12-year-olds (n ¼ 124 and n ¼ 135) and one group of 5- to 7-year-olds (n ¼ 149) with nonsyndromic CUCLP from three previous Americleft studies, including cohorts from seven different cleft/craniofacial centers. Interventions: All patients received the infant management protocols of their respective centers. Eleven trained and calibrated judges (five participated in all three studies) did blind ratings of nasolabial appearance using the Asher-McDade method. Main Outcome Measures: Patients receiving the most consistent ratings between judges, selected first from the groups of 6- to 12-year-olds, were used to create a pilot yardstick for eventual use in the third study of 5- to 7-year-olds. For each of the Asher-McDade categories, 8 of the 5- to 7-year-old patients receiving the most consistent scores between raters were ranked by 10 judges for a final elimination to leave three per category. Results: Using this method of successive changes in rating methods, a new reference yardstick for nasolabial appearance rating was established and linked to the original AsherMcDade method as well as the single examples in a previously published yardstick for patients with CUCLP. Pilot testing using the yardstick improved reliabilities. Conclusions: Use of an expanded nasolabial yardstick of reference photographs representative of the range of possibilities of each of the five Asher-McDade categories is now available to see if reliability of these ratings can be improved. KEY WORDS:
CUCLP, nasolabial appearance, yardstick
Intercenter outcome comparisons have increased value if the outcomes being compared are identical and a common benchmark is used. For outcomes using ordinal scales for ratings, the Goslon Yardstick for dental arch relationships in patients with complete unilateral cleft lip and palate (CUCLP) (Mars et al., 1987) represents one of the best examples of this concept. Used most notably in the original Eurocleft study (Mars et al., 1982), the availability of a reference set of dental models representative of the five rating categories of dental arch relationship (1 ¼ best to 5 ¼ worst) contributed to the very high inter- and intrareliability of the ratings. Furthermore, with the countless additional intercenter comparisons of dental arch relationship that followed, using the same reference yardstick has enabled valid comparison of results between studies (Hathaway et al., 2013).
Dr. Mercado is Clinical Assistant Professor of Orthodontics, Division of Orthodontics, The Ohio State University, Columbus, Ohio. Dr. Russell is Professor and Head, Department of Orthodontics and Cleft Palate Team, Dalhousie University/IWK Health Centre, Halifax, Nova Scotia, Canada. Dr. Daskalogiannakis is Staff Orthodontist, SickKids Hospital, and Associate Professor, Department of Orthodontics, University of Toronto, Toronto, Ontario, Canada. Dr. Hathaway is Professor, Division of Craniofacial Plastic Surgery, Cincinnati Children’s Hospital, Cincinnati, Ohio. Dr. Semb is Professor, University of Manchester, Manchester, United Kingdom. Dr. Ozawa is Orthodontist, Hospital de Reabilitaca ¸ ˜ o de Anomalias Craniofaciais Universidade de Sa˜o Paulo, Bauru, Brazil. Dr. Smith is Clinical Director and Maxillofacial Prosthodontist, Lancaster Cleft Palate Clinic, Lancaster, Pennsylvania. Dr. Lin is Director, St. Louis Cleft-Craniofacial Center, SSM Cardinal Glennon Children’s Medical Center, St. Louis University, St. Louis, Missouri. Dr. Long is Executive Director, Lancaster Cleft Palate Clinic, Lancaster, Pennsylvania. Presented at the 12th International Congress on Cleft Lip/Palate and Related Craniofacial Anomalies, Orlando, Florida, May 5–10, 2013. Submitted February 2014; Revised September 2014; Accepted September 2014. Address correspondence to: Dr. Ross E. Long Jr., Lancaster Cleft Palate Clinic, 223 North Lime Street, Lancaster, PA 17602. E-mail [email protected].
DOI: 10.1597/14-017 30
Mercado et al., NASOLABIAL APPEARANCE YARDSTICK FOR 5- TO 7-YEAR-OLDS WITH CUCLP
Yardsticks of reference photographs for nasolabial appearance assessments have not been as helpful as the Goslon Yardstick and may be one of the reasons contributing to poorer reliability scores. Facial appearance is one of the most relevant measures of success of cleft lip and palate treatment (Asher-McDade et al., 1991). Having a method to reliably assess the nasolabial appearance outcomes of various treatment protocols is critical. Residual anomalies in nasal form, nasal asymmetry, and distortion of the upper lip (Asher-McDade et al., 1991) may affect self-perception, self-esteem, and social interactions. Currently, there is no widely accepted standard rating method to assess facial aesthetics in patients with CUCLP that has demonstrated a high degree of reliability (Nollet et al., 2007). The ability to evaluate and compare the impact of various treatment protocols on nasolabial appearance relies on having a reliable and valid outcome measure. Outcome comparisons of facial appearance are especially difficult because of the subjective nature of the assessment. Use of two-dimensional photography has been the most common method upon which ratings are based because of convenience, cost, and common usage among centers as part of routine record-taking. Several different scoring or rating systems have been developed. The Asher-McDade method (Asher-McDade et al., 1991) is one of the most commonly used. Similar to the Goslon Yardstick, this method was based on an ordinal scale of rive categories of outcomes (1 ¼ best to 5 ¼ worst) for four different nasolabial appearance features: nasal form, nasal symmetry, nasolabial profile, and vermilion border. After the method was first proposed, it was also subsequently used in the landmark Eurocleft study (AsherMcDade et al., 1992), with an overall reliability of 0.60 using the weighted Kappa statistic. More recently, using the same method, two studies from the Americleft Project reported an average intrarater reliability of 0.739 and an average interrater reliability of 0.645 in the first study (Mercado et al., 2011) and an average intrarater reliability of 0.648 and an average interrater reliability of 0.485 in the second study (Mercado et al., 2012). While reliabilities of 0.61 to 0.80 are considered good agreement (Altman, 1991), a rating method with reliability in the excellent range would allow for stronger and more accurate evaluations of nasolabial appearance outcomes. The lack of a reference scale such as the Goslon Yardstick, which includes multiple examples of each of its five categories, might also contribute to the low reliability for nasolabial appearance ratings. A study by KuijpersJagtman et al. (2009) presented reference photographs to illustrate a single example of each of the five categories for each of the four views of the Asher-McDade rating scale (20 photographs in total). The 2011 Americleft study (Mercado et al., 2011) utilized the Kuijpers-Jagtman reference photographs but did not report a significant increase in reliability scores (Mercado et al., 2011). However, given the wide range of nasolabial appearances in children with clefts,
TABLE 1 Yardstick
31
Summary of Studies Used in Developing the Nasolabial
No. Mean age (y) Age range (y)
Study 1 (Mercado et al., 2011)
Study 2 (Mercado et al., 2012)
Study 3 (Daskalogiannakis et al., 2013)
124 8.4 6–12
135 8.6 6–12
149 5.5 5–7
having only one reference photograph for each category makes it difficult to identify the appropriate rating for an individual patient. In contrast, the Goslon Yardstick has been shown to lead to consistently better reliability. Of significance is the fact that the Goslon Yardstick includes multiple examples of cases representative of the range of possibilities in each category, and these are available at rating sessions to view in comparison to a particular patient being rated. Therefore, the purpose of this study was to identify cases from previous studies for which there was the greatest amount of agreement between raters and from that subset to choose cases that represented the range of possibilities within a category to represent a new expanded nasolabial yardstick for future ratings. In addition, in an attempt to maintain some continuity and comparability with the previous Eurocleft and Americleft efforts, the age ranges and stages of development that were determinants in the original outcome comparison studies for both dental arch relationship (Goslon) and nasolabial appearance (AsherMcDade) were followed (7- to 12-year-olds in mixed dentition to early permanent dentition). With a trend toward comparing outcomes at a younger age and the development of a Goslon-type dental arch relationship yardstick for 5- to 6-year-olds (late primary to early mixed dentition) (Atack et al., 1997) and so the nasolabial yardstick could be used in the comparison of samples of younger patients (5- to 7-year-olds), an additional purpose was to include this age range in creating the expanded yardstick. MATERIALS
AND
METHODS
Data from three consecutive studies of nasolabial appearance ratings by the Americleft group (Mercado et al., 2011; Mercado et al., 2012; Daskalogiannakis et al., 2013) were used in a sequential process of identifying patients rated in these studies who received unanimous or near unanimous agreement between raters for their scores. These studies are summarized in Table 1. Study 1 comprised 6- to 12-year-old patients (n ¼ 124; mean age ¼ 8.4 years) with CUCLP, study 2 comprised similarly aged patients (6 to 12 years old) with CUCLP (n¼ 135; mean age ¼8.6 years), and study 3 comprised 5- to 7-year-old patients (n ¼ 149; mean age ¼ 5.5 years) with CUCLP. Study 1 and study 2 used the Asher-McDade method for nasolabial
32
Cleft Palate–Craniofacial Journal, January 2016, Vol. 53 No. 1
FIGURE 1 Vermilion Border rating 1 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 2 Vermilion Border rating 2 (lower left with permission from Kuijpers-Jagtman et al., 2009).
appearance ratings (Asher-McDade et al., 1991), similar to the original Eurocleft assessment of nasolabial appearance (Asher-McDade et al.,1992). The first two Americleft studies used the same reference photographs used in the Eurocleft study (Mercado et al., 2011; Mercado et al., 2012). There was just one example of each scoring category for each view to be used as a reference photograph. From the 259 patients in study 1 and study 2, cases were selected from the Americleft samples that received the most consistent and unanimous or closest to unanimous scores from all raters; these were used to create a new set of reference photographs. Obviously, unanimous agreement was desired, that is, all raters scoring a case identically. When sufficient unanimously rated cases were not available to complete the desired number of reference photos, additional reference cases were chosen based on those in which the mean score from all raters was closest to unanimous. Thus, a final set of four reference photos of 6to 12-year-old patients for each of the five rating categories for each of the four views (80 total) was selected from the total pool of 259 patients from these initial two studies. The third intercenter study of nasolabial appearance involved evaluation of 149 patients with CUCLP with an age range of 5 to 7 years (Daskalogiannakis et al., 2013). For this study, the only reference photos available for this age range were those in a publication by Kuijpers-Jagtman et al. (2009). Because this publication included only one yardstick reference photo per scoring category, we thought better reliability might be possible if there were multiple reference photographs representing a range of possibilities for each score for each of the four views. Therefore, the four reference photographs for each scoring category and each view that were selected from the original two Americleft studies were added to the new reference yardstick along with the single reference photograph from the KuijpersJagtman (2009) publication. The Kuijpers-Jagtman photograph was included to ensure continuity with the previous
Eurocleft study (Asher-McDade et al., 1992) and the reference photos created by Kuijpers-Jagtman et al. (2009) for use with the Asher-McDade method by including a common reference photograph for each of the scores of the Asher-McDade categories. Note, however, that the new yardstick was being used to rate a sample of 5- to 7-yearolds. To the single Kuijpers-Jagtman reference photographs of 5- to 7-year-olds were added the additional reference photographs from the samples of 6- to 12-yearolds. This was done with the assumption that with no other reference photographs available for 5- to 7-year-olds to expand the Kuijpers-Jagtman yardstick, the age difference would not be visible in the cropped images versus the fullface images, and the deviations in form affecting scoring would be the same in both age groups. Although this assumption cannot be validated, there is likewise no published evidence that a 3-year mean age difference in early to mid-childhood would represent a significant confounder in identifying and discriminating between good and poor nasolabial appearance results. Two other significant modifications were made to the traditional Asher-McDade method that were incorporated into this Americleft study of 5- to 7-year-olds with CUCLP. First, in order to shorten and simplify the ratings in this study, the Asher-McDade categories of nasolabial form and nasolabial symmetry, each evaluated separately in the frontal view, were combined into one category called nasal frontal to be consistent with the single category of nasolabial profile. The merging of the two categories allowed for an overall assessment of the aesthetics from the frontal view. In addition, one less category results in faster completion of the ratings and less rater fatigue, which may contribute to lower inter- and intrarater reliability. The additional reference photographs for the new nasolabial frontal view were chosen from the two previous Americleft studies from those patients showing the closest agreement
Mercado et al., NASOLABIAL APPEARANCE YARDSTICK FOR 5- TO 7-YEAR-OLDS WITH CUCLP
33
FIGURE 3 Vermilion Border rating 3 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 5 Vermilion Border rating 5 (lower left with permission from Kuijpers-Jagtman et al., 2009).
between raters for the nasolabial symmetry and nasolabial form categories merged. The second modification was the use of a modified Q-sort method (Daskalogiannakis et al., 2013; Long et al., 2013) rather than the traditional method of PowerPoint projection of one case at a time and simultaneous rating of all views. The Q-sort method was introduced by Stephenson (1953) to provide a foundation for the systematic study of subjective objects (Brown, 1993). Each object is judged relative to the other objects in the sample. As applied to this investigation, with the Q-sort method raters are provided with their own pack of cards with the cropped views of each patient on separate cards. The cards are then sequentially separated into five piles representative of the five categories of outcomes. Once completed the raters are able to reevaluate their choices by comparing the patients within and between the piles.
From this rating, a subset of 85 from the entire sample of 149 was generated by the selection of 5 to 8 patients in each category for whom there was the greatest agreement between raters and the average scores were exactly or closest to the category number. At a separate session 10 experienced judges (seven orthodontists, two surgeons, and one prosthodontist) rank-ordered the cases from best to worst within each category and for each of the three features being assessed (nasolabial profile, nasal frontal and vermilion border). Based on an average rank and the most consistent rank across raters, three cases were selected for each rated feature and for each category. These were the case ranked as best for that category, the case ranked lowest, the case that represented the median rank. To maintain continuity and commonality with the previous published yardstick, the single representative case from the Kuijpers-Jagtman (2009) published yardstick was combined with these three examples for each feature and
FIGURE 4 Vermilion Border rating 4 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 6 Nasal Frontal rating 1 (lower left with permission from Kuijpers-Jagtman et al., 2009).
34
Cleft Palate–Craniofacial Journal, January 2016, Vol. 53 No. 1
FIGURE 7 Nasal Frontal rating 2 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 9 Nasal Frontal rating 4 (lower left with permission from Kuijpers-Jagtman et al., 2009).
category to create an expanded yardstick for nasolabial appearance for 5- to 7-year-olds with a range of representative cases within each group. This new expanded nasolabial appearance yardstick for 5- to 7-year-olds with CUCLP was then pilot-tested on a small (n ¼ 30) sample of 5- to 7-year-old patients with CUCLP. The modified Q-sort method was again used because raters from the previous study unanimously preferred this method over the traditional PowerPoint method. The panel of seven raters for this pilot study comprised four orthodontists, two surgeons, and one prosthodontist. All 30 cases were rated twice on two separate days. Intra- and interrater reliability were tested using the weighted Kappa statistic. Intrarater reliability was calculated by comparing each rater’s first ratings to their second ratings. Interrater reliability was calculated by comparing each rater to the others for both ratings. Reliabilities were then compared to all three previous
Americleft studies to see if the expanded yardstick was associated with improvement.
FIGURE 8 Nasal Frontal rating 3 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 10 Nasal Frontal rating 5 (lower left with permission from Kuijpers-Jagtman et al., 2009).
RESULTS Figures 1 through 15 (available online in color) show the new expanded yardstick for 5- to 7-year-olds with CUCLP for each rating category (one through five) for each of the three features being rated: vermilion border, nasal frontal, and nasolabial profile. The lower left reference photo in each figure is taken from the original publication of reference photos for patients with CUCLP (KuijpersJagtman et al., 2009, with permission from the Journal of Craniofacial Surgery). The full-color yardstick is available for download for use in outcome comparisons from the Americleft Task Force Study Guide, which can be accessed through the Research tab on the website for the American
Mercado et al., NASOLABIAL APPEARANCE YARDSTICK FOR 5- TO 7-YEAR-OLDS WITH CUCLP
35
FIGURE 11 Nasolabial Profile rating 1 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 13 Nasolabial Profile rating 3 (lower left with permission from Kuijpers-Jagtman et al., 2009).
Cleft Palate–Craniofacial Association website (www. acpa-cpf.org). Table 2 compares the average intra- and interrater Kappa values for the three previous Americleft nasolabial appearance studies (Mercado et al., 2011; Mercado et al., 2012; Daskalogiannakis et al., 2013) as well as the pilot study carried out as part of this report. Table 3 lists the interpretation of the Kappa statistic (Altman, 1991).
This report provides, for the first time, a nasolabial yardstick for 5- to 7-year-old patients with CUCLP that consists of an expanded number of reference photographs indicative of a range of possibilities for each of the five rating categories for each feature. The expanded yardstick was attempted for the purpose of improving the reliability of the ratings for use in intercenter outcome comparisons.
Because of the extreme subjectivity of nasolabial appearance ratings, in many previous investigations the reliability of the scoring between and within raters has always been noticeably poorer than that of comparisons of dental arch relationships using the Goslon Yardstick (Mars et al., 1987). One reason for this may be the fact that the Goslon Yardstick contains several examples of differing appearances for cases rated in each of the five scoring categories. With additional examples against which to compare a case being scored, the raters have a greater likelihood of finding similar-appearing reference cases to match the one being scored. As it is likely that variations in nasolabial appearance and the additional subjectivity in rating these exceed what would be encountered in a dental arch relationship study, having only one reference case available for each of the nasolabial appearance categories would logically only increase the subjectivity of the rating.
FIGURE 12 Nasolabial Profile rating 2 (lower left with permission from Kuijpers-Jagtman et al., 2009).
FIGURE 14 Nasolabial Profile rating 4 (lower left with permission from Kuijpers-Jagtman et al., 2009).
DISCUSSION
36
Cleft Palate–Craniofacial Journal, January 2016, Vol. 53 No. 1
TABLE 3
Interpretation of Kappa Values
Kappa Value
Strength of Agreement
,0.20 0.21–0.40 0.41–0.60 0.61–0.80 0.80–1.00
Poor Fair Moderate Good Very Good
increasing the numbers of samples involved in the comparison. CONCLUSIONS
FIGURE 15 Nasolabial Profile rating 5 (lower left with permission from Kuijpers-Jagtman et al., 2009).
The results of the pilot study using the new yardstick are encouraging, this being the first of the Americleft studies where intrarater reliability reached the highest level of agreement using the Kappa statistic, while moving interrater agreement from moderate to good. Clearly, the use of the Q-sort method may also have played a role in the improved agreement, but regardless, all raters in the Daskalogiannakis et al. (2013) study and the current pilot study were unanimous in their preference of this over the previous PowerPoint slide-presentation method. Additionally, although merging the nasolabial symmetry and nasolabial form categories into one feature, (nasal frontal), clearly simplified and shortened the time required to do a Qsort rating, it remains to be seen whether too much useful information is lost by not keeping these features separate. Finally, the use of a common reference photograph from a previous publication for the various features and rating categories is essential if our intercenter outcome comparison studies are going to be suitable for cross-comparisons. The value of the current regional and national initiatives for collaboration between centers willing to compare their outcomes in well-controlled and blinded studies, as has been done in Euroclett and Americleft, is exponentially increased if all studies are using a common benchmark for measuring and scoring. By so doing, the results of studies carried out separately but using the same records; prepared identically; and rated, scored, or measured similarly using the same benchmark reference yardstick can be compared more confidently to one another, thereby significantly TABLE 2 Comparison of Average Kappa Values From Previous Americleft Studies Study 3 Study 2 Study 1 (Mercado (Daskalogiannakis (Mercado et al., 2013) Current Study et al., 2011) et al., 2012) Interrater Intrarater
0.674 0.803
0.645 0.739
0.485 0.648
0.594 0.761
The use of an expanded nasolabial yardstick of reference photographs representative of the range of possibilities in each of the five Asher-McDade categories is now available to see if reliability of these ratings can be improved. Acknowledgments. The authors and Americleft Group would like to acknowledge and thank the American Cleft Palate-Craniofacial Association and the Cleft Palate Foundation for their financial support of the Americleft Task Force. Additional funding was provided by the Trout Family Trust and the Mellinger Medical Research Fund, Lancaster, Pennsylvania.
REFERENCES Altman DG. Practical Statistics for Medical Research. London: Chapman & Hall; 1991:403–409. Asher-McDade C, Brattstrom V, Dahl E, McWilliam J, Molsted K, Plint DA, Prahl-Andersen B, Semb G, Shaw WC. A six center international study of treatment outcome in patients with clefts of the lip and palate: part 4. Assessment of nasolabial appearance. Cleft Palate Craniofac J. 1992:29:409–412. Asher-McDade C, Roberts C, Shaw WC, Gallagher C. The development of a method for rating nasolabial appearance in patients with clefts of the lip and palate. Cleft Palate J. 1991;28:380–390. Atack NE, Hathorn IS, Semb, G, Dowell T, Sandy JR. A new index for assessing surgical outcomes in unilateral cleft lip and palate subjects aged 5—reproducibility and reliability. Cleft Palate Craniofac J. 1997;34:242–246. Brown SR. A primer on Q methodology. Operant Subjectivity. 1993;16:91–138. Daskalogiannakis J, Russell KA, Mercado A, Hathaway RR, Singer E, Stoutland A, Long RE Jr, Semb G, Shaw WC. The Americleft Project: burden of care analysis of various infant orthopedic protocols for improvement of nasolabial aesthetics in patients with CUCLP. Presented at the 12th International Congress on Cleft Lip/ Palate and Related Craniofacial Anomalies; May 2013; Orlando, Florida. Hathaway RR, Long RE Jr, Mercado AM, Daskalogiannakis J, Russell KA, Semb G, Shaw WC, Gregory J. The Americleft Project: use of a standardized outcome measure of dental arch relationships (Goslon) to allow international interstudy comparisons. Presented at the 12th International Congress on Cleft Lip/ Palate and Related Craniofacial Anomalies; May 2013; Orlando, Florida Kuijpers-Jagtman AM, Nollet PJ, Semb G, Bronkhorst EM, Shaw WC, Katsaros C. Reference photographs for nasolabial appearance
Mercado et al., NASOLABIAL APPEARANCE YARDSTICK FOR 5- TO 7-YEAR-OLDS WITH CUCLP
rating in unilateral cleft lip and palate. J Craniofac Surg. 2009;20:1683–1686. Long RE Jr, Mercado A, Hathaway RR, Daskalogiannakis J, Russell KA, Stoutland A, Singer E, Semb G, Shaw WC, Fessler J. The Americleft Project: a modification of Asher-McDade method for rating nasolabial esthetics in patients with CUCLP. Presented at the 12th International Congress on Cleft Lip/Palate and Related Craniofacial Anomalies; May 2013; Orlando, Florida. Mars M, Asher-McDade C, Brattstrom V, Dahl E, McWilliam J, ¨ Mølsted K, Plint DA, Prahl-Andersen B, Semb G, Shaw WC, The RPS. A six-center international study of treatment outcome in patients with clefts of lip and palate: Part 3: Dental arch relationships. Cleft Palate-Craniofac J, 1992;29:405–408. Mars M, Plint DA, Houston WJ, Bergland O, Semb G. The Goslon yardstick: a new system of assessing dental arch relationships in children with unilateral clefts of the lip and palate. Cleft Palate J. 1987;24:314–322.
37
Mercado A, Peanchitlertkajorn S, Daskalogiannakis J, Hathaway R, Lamichane M, Russell K, Semb G, Long RE Jr. The Americleft Project: an expansion of the Americleft intercenter comparisons of nasolabial appearance outcomes to include a center using NAM as part of its primary protocol. Presented at the 68th Annual Meeting of the American Cleft Palate-Craniofacial Association; April 2012; San Juan, Puerto Rico. Mercado A, Russell K, Hathaway R, Daskalogiannakis J, Sadek H, Long RE, Cohen M, Semb G, Shaw WC. The Americleft Study: an inter-center study of treatment outcomes for patients with unilateral cleft lip and palate. Part 4. Nasolabial aesthetics. Cleft Palate Craniofac J. 2011;48:259–264 Nollet PJ, Kuijpers-Jagtman AM, Chatzigianni A, Semb G, Shaw WC, Bronkhorst EM, Katsaros C. Nasolabial appearance in unilateral cleft lip, alveolus and palate: a comparison with Eurocleft. J Craniomaxillofac Surg. 2007;35:278–286. Stephenson W. The Study of Behavior. Chicago: University of Chicago Press; 1953.
