Review article
Bone age assessment: automated techniques coming of age? RR van Rijn1 and HH Thodberg2 1
Department of Radiology, Academic Medical Centre/Emma Children’s Hospital Amsterdam, the Netherlands; 2Visiana, Holte, Denmark Correspondence to: RR van Rijn. Email: [email protected]
Abstract Bone age determination from hand radiographs is one of the oldest radiographic procedures. The first atlas was published by Poland in 1898, and to date the Greulich Pyle atlas, although it dates from 1959, is still the most commonly used method. Bone age rating is time-consuming, suffers from an unsatisfactorily high rater variability, and therefore already 25 years ago it was proposed to replace the manual rating by an automated, computerized method, a field nowadays referred to as computer-aided diagnosis (CAD). The pursuit of this goal reached a first stage of accomplishment in 1992–1996 with the presentation of several systems. However, they had limited clinical value, and efforts in CAD research were increasingly focused on lesion detection for cancer screening. It was only in 2008 that a fully-automated bone age method was presented, which appears to be clinically acceptable. In this paper we consider the requirements that should be met by an automated bone age method and review the state of the art. Integration in PACS and saving time are important factors for radiologists. But it is the validation of the methods which poses the greatest challenge, because there is no gold standard for bone age rating, and the direct comparison to manual rating is therefore not sufficient for demonstrating that manual rating can be replaced by automated rating. One needs additional studies assessing the precision of a method and its accuracy when used for adult height prediction, which serves as an objective.
Keywords: Age determination by skeleton, automated pattern recognition, computer-assisted diagnosis Submitted June 27, 2012; accepted for publication October 26, 2012
Soon after the discovery of X-rays by Wilhelm Conrad Ro¨ntgen in November 1895 the first application of radiographs in the depiction of skeletal maturation was published. In 1898 John Poland (1855 –1937), an orthopedic surgeon working at Guy’s Hospital in London, UK, published the first bone age atlas: “skiagraphic atlas showing the development of bones of the wrist and hand” (1). This atlas depicts positive reprints (skiagraphs) of hand radiographs of 19 British children, aged 1 –17 years, with an elaborate description of each radiograph. In the following decade, Pryor (2) and Rotch (3) published scientific publications on this subject. The two most influential publications in this field were published in 1959 and 1962 when Greulich & Pyle (GP) and Tanner & Whitehouse & Healy (TW) respectively published their atlases (4, 5). In pediatric radiology today, the GP atlas is most widely adopted both by radiologists and pediatricians (6). The TW method, which takes a longer time to apply because it rates bones separately, is mostly used by endocrinologists.
These methods have several drawbacks. As visual grading techniques they have an inherent inter- and intraobserver variation. Several studies have addressed this issue and it has been shown that the standard error on a single determination in inter-observer studies ranges from 0.45 to 0.83 years (7 – 10). The GP method is relatively easy to learn, and after a short learning curve, readers can achieve an intra-observer variation comparable to that of experienced readers (10). For the GP method a second drawback is the lack of standardization in how the bones are weighted. In clinical practice, some raters assign one-third or even one-half weight to the carpals, while others ignore the carpals completely. Raters using the carpals reduce their importance at higher maturity but again not in a standardized manner (10). Finally, one issue which is specifically of interest to those who are involved in multicenter studies, is the variance in bone age assessments between centers (11). All studies in the inter-observer variation have been performed in single centers. As one can assume that within one center
Acta Radiologica 2013; 54: 1024–1029. DOI: 10.1258/ar.2012.120443 Downloaded from acr.sagepub.com at University of Otago Library on July 13, 2015
Bone age assessment
1025
.................................................................................................................................................
Fig 1. Hand radiograph analyzed using BoneXpert. Bone age using both Greulich and Pyle as well as Tanner Whitehouse are calculated. Additionally the bone health index is calculated
in general the same approach to bone age assessment is used, this might underestimate the difference between radiologists and pediatricians trained in different environments.
Historical perspective A solution to these problems could be the implementation of a (semi)-automated technique to assess bone age. This is not a new idea: already 25 years ago, several research groups proposed that bone age rating was a task that lends itself well to automation by computers, and started work towards achieving this goal. Thus in 1989, after having worked on computerized methods for several years, Tanner wrote with confidence: “Surely this is the way forward, eliminating the all-too-fallible rater entirely. Surely someone will eventually put up the money for the requisite hardware” (12). To date several systems, with a variable penetration into clinical practice have been introduced (13 –17). The first three papers, representing the birth of the topic, were HANDX (18) from 1989, Pietka (19) from 1991, and Tanner and Gibbons’ CASAS system from 1992 (20). CASAS is an interactive system analyzing the 13 bones of the TW RUS system. The individual bones were located manually, aided by templates presented on the screen,
and once identified, the maturation rating was done by the computer. This procedure had to be performed for each individual bone. Tanner showed that CASAS indeed yields a more consistent bone age assessment compared to manual rating (21). The major drawback of this technique was that the interactive method took more time than a manual TW bone age assessment. Although CASAS has been used in several studies it never became a widely accepted clinical tool (22, 23). Thus bone age rating was one of the first radiological procedures considered for automation. The reasons for this are that bone age rating is tedious, time-consuming and – most importantly – humans are not good at it, as reflected in the considerable inter- and intra-rater variability. The cause of this variability is that the radiologist does not look for a particular lesion, but rather at a plethora of maturity indicators – the more the better – to average out differences among bones. A consensus of these findings is made and expressed as a numerical value, the bone age, which places the image on a continuous maturity scale. Thus it is an example of quantitative image analysis, which is in general difficult for humans, who are more proficient at making qualitative statements about, for instance, the presence of a fracture. To go beyond CASAS it was necessary to achieve complete automation of the process, but this turned out to be much more difficult than thought initially. The area received much less research resources than computer-aided detection/diagnosis (CAD) for cancer screening, so this difficulty left the area in a state of “prolonged prepuberty” with very little progress for over a decade. In the last decade several publications on this topic have appeared (24 – 28). However, these publications have not led to commercially available tools, with one exception: in 2008 a fully automated method was developed and marketed under the name BoneXpert (14). The system locates the same 13 RUS bones of the hand as CASAS: Radius, ulna, and the bones in rays 1 and 5, but the system determines both Tanner-Whitehouse (TW2 or TW3) and Greulich-Pyle (GP) bone ages (Fig. 1).
Requirements to CAD So while the need for automated bone age analysis was realized 25 years ago, it has had a remarkably long childhood. The similar vision of CAD for mammography screening grew up much faster and resulted in the first FDA-approved system already in 1998 (29). Recently, a general analysis of CAD was presented in a paper entitled “Computer-aided diagnosis: how to move from the laboratory to the clinic” and we will use the concepts in this paper in the continued discussion (29). In this publication van Ginneken et al. define CAD in a wider sense, including not only detection and diagnosis but also quantitative image analysis, because it is based on the same technology and has increased the interest in CAD in recent years. According to the authors, “Computerized quantification may hold more potential than computerized detection. When radiologists are asked to name aspects of
Downloaded from acr.sagepub.com at University of Otago Library on July 13, 2015
1026
RR van Rijn and HH Thodberg
.................................................................................................................................................
their work that are common, time consuming, and could be automated, they usually do not mention detection but rather documentation and quantification.” The paper proposes four requirements which should be met in order for a CAD method to be successful in clinical practice: (a) (b) (c) (d)
CAD should improve radiologists’ performance; CAD should save time; CAD must be seamlessly integrated into the workflow; CAD should not impose liability concerns and the incremental costs should be negligible or reimbursed.
The CASAS system did indeed demonstrate improvements in precision of the rating. But it scored poorly on the timesaving, as it actually required more time. It took place on a dedicated workstation, where a camera was used to manually align each bone to a template, so it was not integrated in the workflow. The fully automated BoneXpert method conforms better, although not fully, to van Ginneken et al.’s requirements. Compared to manual rating it makes fewer gross errors, it has better precision, and it yields a better accuracy when used for adult height prediction, and these improvements are clinically significant (30, 31). It saves the time used for manual rating, estimated between 1 and 10 min and can be implemented as a PACS plug-in leading to seamless integration in the workflow. In Europe it is a CE-marked medical device intended to replace or supplement the radiologist, and validated as such. However, the costs are currently not reimbursed. The most significant other research group, which pursues the development of an automated method, has published several papers over more than a decade (16, 19, 24). They have addressed the issue of clinical workflow with a system, which is not commercially available, but which is intended to assist the radiologist in the bone age rating by suggesting similar images from a large atlas. The precise use and the performance of the system has not yet been published, but what has been revealed is a detailed description of the integration in the PACS workflow, which as van Ginneken mentions, is essential (32). The same group has also presented a method for rating bone age of carpals with promising results (23).
Bone age assessment can also be useful in the diagnosis of disorders/syndromes and/or skeletal dysplasias. The hand radiograph can also be of help in the diagnosis of disorders/syndromes, irrespective of the presence of abnormal bone maturation. For this it is essential that the radiograph is reported by a ( pediatric) radiologist. Examples of such disorders/syndromes are Turner syndrome (squared off carpals and irregular metaphysis), rickets (fraying and splaying of metaphysis), hypochondroplasia (with broader and stump bones), and pseudohyporathyroidism (short fourth and fifth metacarpal bones). Chronic diseases can also manifest themselves on the hand radiograph; examples of these are renal osteodystrophy (subperiosteal bone resorption) and juvenile idiopathic arthritis ( periarticular osteopenia). In children with skeletal dysplasias radiographs of the left hand are a part of the skeletal survey and as such play a role in the diagnostic work-up. In these cases the assessment of bone age, due to a disturbed anatomy and development, is difficult and the results should be interpreted with caution. Finally, bone age assessment is mandatory in all patients treated for growth disorders in order to monitor therapy. In the individual patient automated bone age assessment has the advantage that it is an unbiased tool, whereas the radiologist/clinician (if aware of treatment and/or chronological age of the patient) might be influenced in his/her bone age assessment. It could be argued that in longitudinal studies only the first hand radiograph has to be reviewed by a ( pediatric) radiologist, in order to detect anomalies which will not be detected by an automated bone age assessment. Subsequent radiographs would not necessarily be read and the assessment could be done in a completely automated fashion.
Validation studies of automated bone rating Van Ginneken et al. pointed out that validation of CAD systems is often a major challenge, which may require development of new types of studies and new metrics to quantify the clinical value. Automated bone age rating is no exception, and this section reviews the recent development in this area. Accuracy
Clinical application In 2009 a group of experts met for the first International Workshop on Skeletal Maturity in Tu¨bingen, Germany. Partly based on the discussions at this meeting two papers on the use of bone age in clinical practice were published (11, 33). There are two main reasons for the initial assessment of bone age: short and tall stature. Besides these it can be useful in clinical pediatric oncology trials, in order to assess late effects of chemotherapy and radiotherapy. In these groups bone age assessment is the main objective of the study and automated systems can have a significant impact in patient care.
For assessment of accuracy, automatic bone age methods should be compared with manual rating. Table 1 presents six such studies performed for the BoneXpert method. They used Bland-Altman plots and revealed an SD of the difference of typically 0.72 years. This is of the same magnitude as the SD between two manual raters so this seems to indicate that the automated method is valid. The studies cover the major disorders where bone age is used and also cover four different ethnicities, and the SDs listed in Table 1 are more or less the same across these groups. Table 1 also lists the biases, defined as the average difference between automated and manual rating, and they are small. However, according to Bland-Altman’s methodology, such studies only indicate that manual and automated rating
Downloaded from acr.sagepub.com at University of Otago Library on July 13, 2015
Bone age assessment
1027
.................................................................................................................................................
Table 1 Studies comparing manual and automated rating
Population Normal Caucasian Short stature Tall stature Normal, Four ethnicities Precocious puberty Congenital adrenal hyperplasia
Radiographs (n) 538 1084 812 708 741 892
SD (y) 0.71 0.72 0.55 0.61† 0.76 0.72
Bias (y)
0 0 0.13 0.07 – 0.19 – 0.02
Fraction of dispsuted cases where the reference is closer to automated than to manual rating
Reference
7/7 18/27 15/19 12/15 23/41 18/20
(31) (40) (41) (8) (35) (36)
These two studies where used to adjust the bias of the method so the bias is close to zero by construction The manual rating was the average of two observers’ ratings A positive bias indicates that the automated method on average gives larger values than manual rating †
are the same with a tolerance of 2 SD ¼ 1.44 years, quite a large margin, so clearly these studies alone are not sufficient for validation of a new method. In order to challenge the automated method more, these studies went a step further and focused on the cases where the deviation between the two methods was larger than a certain threshold typically set to 1.5 years. These “disputed cases” were carefully rerated blindly by two to four raters and the average of these ratings was defined as “the reference rating”. Then it was assessed whether the reference ratings agreed better with the automated rating than with the original manual rating, and Table 1 lists proportion of the disputed cases where this was true. The automated method was the more correct method in five of six studies, while in the one study, on children with precocious puberty there was no significant difference. In the study on precocious puberty the authors did not specifically address this issue and did not present an explanation for this finding. Table 1 demonstrates that replacing manual by automated rating leads to a safer rating, in the sense that there will be fewer cases which are rated very wrong. This supports the intended use of the method as a stand-alone device, with no need of manual supervision.
the rater precision to be 0.58 years (calculation based on the same data, private communication with the first author), so the precision of the automated rating is three times better compared to manual rating (38). This is the most conspicuous advantage of the automated method. The dilemma for automated bone age assessment The studies reviewed so far have shown that the new automated GP bone age method agrees as well with manual rating as one could expect (SD 0.72 y) for a completely new bone age method, and that it makes less severe errors than manual raters, and that it has an excellent precision. But the devil’s advocate would still point to one dilemma: the automated method could be measuring something slightly else than GP bone age. To address this dilemma, it should first be pointed out that manual GP bone age is not the ultimate “truth” about bone age, rather it is one possible convention, and there are other conventions in use, namely TW and Fels bone ages. There is no direct way to determine the true maturity of a hand; one cannot, for example, dissect the hand and perform a measurement by any reference method. So the deviation of the automated rating of + 1.44 y (2 SD) from manual GP rating should not necessarily be taken against automated rating.
Precision Two types of studies assessing the precision of a bone age method have been defined as its ability to yield the same result on a repeated X-ray image. The first type utilized longitudinal series of X-rays taken at, for instance, 6-month intervals (34). The smoothness of the series was used to estimate the precision of the automated bone age to 0.17 years, meaning the standard error on a single measurement. A study in precocious puberty children found 0.23 years, and in a study on children with Congenital Adrenal Hypoplasia a standard error of 0.21 years was found (35, 36). The second type used 2700 pairs of X-rays of the left and right hands obtained on the same day to estimate the precision to 0.18 years (37). The study found no average difference between the bone age of the left and the right hand. The precision of manual rating varies considerable with the experience of the raters and the time spent on the rating. A recent study compared 12 raters and found
Adult height prediction To resolve the dilemma, a new framework for validating bone age methods was developed, based on the ability to predict the adult height (39). Adult height prediction is indeed the most important application of bone age, and the framework proposed to assess the validity (or trueness) of a bone age method by its ability to predict the adult height. This is an objective and highly relevant endpoint. By using the Zurich Longitudinal study of 232 normal children born in 1954 – 1956 it was found that automated GP bone age was at least as good as manual GP bone age for predicting adult height (39). It was also found that manual GP bone age performed significantly better than manual TW bone age, which came as a surprise to many. This shows that the new automated method does indeed measure true bone age, and not some other value with unknown biological meaning. Another paper continued along these lines by constructing a new adult height prediction model based on the
Downloaded from acr.sagepub.com at University of Otago Library on July 13, 2015
1028
RR van Rijn and HH Thodberg
.................................................................................................................................................
same longitudinal data from Zurich (30). The new method can be seen as a successor of the classical Bayley-Pinneau method, but using better mathematics and more recent data. The model was validated on a later study from Zurich of children born in 1973 – 1991, and the model was found to predict with the same accuracy as on the older data.
Conclusion on the clinical use of automated bone age assessment There is now good scientific evidence that automated bone age assessment has come of age. It has been shown that automated systems have a better precision and accuracy compared to radiologists’ reading. The integration in the PACS workflow is essential, and the next step would be to develop a standardized way to enable the bone age value to flow into other hospital information systems, e.g. EPJ, where it can be used for adult height prediction. As a result, the automated bone age assessment can support and in some cases replace the radiologists’ report. As automated bone age systems are not developed to detect anomalies in skeletal anatomy, radiological assessment of at least the first radiograph of each child is mandatory. In studies in which surgical intervention, such as epiphysiodesis has taken place, manual assessment is also mandatory as this influences automatic bone age assessment. In multicenter studies the use of automated systems will obviate the need for independent central readers. In some cases, however, the referring physician might request a general diagnosis of the child from the image, and the radiologist then needs to report the radiograph for that specific request. Additionally, automated systems will not be able to rate all radiographs, either due to age constraints or anatomical variation or malpositioning, in these cases skeletal age assessment will remain a manual task. Limitations of the reported systems are that in some, the carpals and in other the phalanges are not included. Another disadvantage is that the bone age range is limited to specific age groups, where none of the published automated techniques covers the whole age range. Another valuable extension would be to diagnose various conditions from the bone lengths and other morphological features, but this is potentially much more complex, and there seems to be a long way to go before computers can replace the radiologist’s general review of the hand radiograph. Conflict of interest: The BoneXpert technology is proprietary to Visiana, a company owned by HH Thodberg. BoneXpert has been marketed as a medical device. A US patent on BoneXpert has been issued.
REFERENCES 1 Poland J. Skiagraphic atlas showing the development of bones of the wrist and hand. London: Smith, Elder and Co., 1898 2 Pryor JW. The heriditary nature of variation in the ossification of bones. Anat Rec 1907;1:84– 88
3 Rotch TM. A study of the development of bones in childhood by the roentgen method with the view of establishing a development index for the grading of and the protection of early life. Trans Amer Ass Phys 1909;24:603 –30 4 Greulich WW, Pyle SI. Radiographic atlas of skeletal development of the hand and wrist. Stanford: Stanford University Press, 1959 5 Tanner JM, Whitehouse RH, Healy MJR. A new system for estimating skeletal maturity from the hand and wrist with standards derived from a study of 2600 healthy british children. Paris: Centre International de L’enfance, 1962. 6 Buckler JM. How to make the most of bone ages. Arch Dis Child 1983;58:761 –3 7 Bull RK, Edwards PD, Kemp PM, et al. Bone age assessment: a large scale comparison of the Greulich and Pyle, and Tanner and Whitehouse (TW2) methods. Arch Dis Child 1999; 81:172 –3 8 Thodberg HH, Savendahl L. Validation and reference values of automated bone age determination for four ethnicities. Acad Radiol 2010;17:1425 – 32 9 Johnson GF, Dorst JP, Kuhn JP, et al. Reliability of skeletal age assessments. Am J Roentgenol Radium Ther Nucl Med 1973; 118:320 – 7 10 Roche AF, Rohmann CG, French NY, et al. Effect of training on replicability of assessments of skeletal maturity (Greulich-Pyle). Am J Roentgenol Radium Ther Nucl Med 1970;108:511 – 5 11 Martin DD, Wit JM, Hochberg Z, et al. The use of bone age in clinical practice - part 1. Horm Res Paediatr 2011;76:1– 9 12 Tanner JM. Assessing the skeletal maturity of the hand-wrist: FELS method. Am J Human Biol 1989;1:493 –4 13 Tanner JM, Gibbons RD. Automatic bone age measurement using computerized image analysis. J Pediatr Endocrinol 1994; 7:141 –5 14 Thodberg HH, Kreiborg S, Juul A, et al. The BoneXpert method for automated determination of skeletal maturity. IEEE Trans Med Imaging 2009;28:52 –66 15 Hill K, Pynsent PB. A fully automated bone-ageing system. Acta Paediatr Suppl 1994;406:81 –3 16 Pietka E, Gertych A, Pospiech S, et al. Computer-assisted bone age assessment: image preprocessing and epiphyseal/metaphyseal ROI extraction. IEEE Trans Med Imaging 2001;20:715 – 29 17 Liu J, Qi J, Liu Z, et al. Automatic bone age assessment based on intelligent algorithms and comparison with TW3 method. Comput Med Imaging Graph 2008;32:678 – 84 18 Michael DJ, Nelson AC. HANDX: a model-based system for automatic segmentation of bones from digital hand radiographs. IEEE Trans Med Imaging 1989;8:64 –9 19 Pietka E, McNitt-Gray MF, Kuo ML, et al. Computer-assisted phalangeal analysis in skeletal age assessment. IEEE Trans Med Imaging 1991;10:616 –20 20 Tanner JM, Gibbons RD, Bock RD. An image analysis system for TW skeletal maturity. Horm Res 1992;37:11 21 Tanner JM, Oshman D, Lindgren G, et al. Reliability and validity of computer-assisted estimates of Tanner-Whitehouse skeletal maturity (CASAS): comparison with the manual method. Horm Res 1994;42:288 –94 22 Frisch H, Riedl S, Waldhor T. Computer-aided estimation of skeletal age and comparison with bone age evaluations by the method of Greulich-Pyle and Tanner-Whitehouse. Pediatr Radiol 1996;26:226 –31 23 Van TA, De WW, Roks A, et al. Computer-aided skeletal age scores in healthy children, girls with Turner syndrome, and in children with constitutionally tall stature. Pediatr Res 1996;39:360 –7 24 Zhang A, Gertych A, Liu BJ. Automatic bone age assessment for young children from newborn to 7-year-old using carpal bones. Comput Med Imaging Graph 2007;31:299 – 310 25 Somkantha K, Theera-Umpon N, Auephanwiriyakul S. Bone age assessment in young children using automatic carpal bone feature extraction and support vector regression. J Digit Imaging 2011;24:1044 – 58 26 Hsieh CW, Liu TC, Jong TL, et al. A fuzzy-based growth model with principle component analysis selection for carpal bone-age assessment. Med Biol Eng Comput 2010;48:579 – 88
Downloaded from acr.sagepub.com at University of Otago Library on July 13, 2015
Bone age assessment
1029
.................................................................................................................................................
27 Hsieh CW, Jong TL, Tiu CM. Bone age estimation based on phalanx information with fuzzy constrain of carpals. Med Biol Eng Comput 2007;45:283 –95 28 Rucci M, Coppini G, Nicoletti I, et al. Automatic analysis of hand radiographs for the assessment of skeletal age: a subsymbolic approach. Comput Biomed Res 1995;28:239 –56 29 van GB, Schaefer-Prokop CM, Prokop M. Computer-aided diagnosis: how to move from the laboratory to the clinic. Radiology 2011;261:719 – 32 30 Thodberg HH, Jenni OG, Caflisch J, et al. Prediction of adult height based on automated determination of bone age. J Clin Endocrinol Metab 2009;94:4868 – 74 31 van Rijn RR, Lequin MH, Thodberg HH. Automatic determination of Greulich and Pyle bone age in healthy Dutch children. Pediatr Radiol 2009;39:591 –7 32 Gertych A, Zhang A, Sayre J, et al. Bone age assessment of children using a digital hand atlas. Comput Med Imaging Graph 2007; 31:322 –31 33 Martin DD, Wit JM, Hochberg Z, et al. The use of bone age in clinical practice - part 2. Horm Res Paediatr 2011;76:10– 6 34 Martin DD, Sato K, Sato M, et al. Validation of a new method for automated determination of bone age in Japanese children. Horm Res Paediatr 2010;73:398 – 404
35 Martin DD, Meister K, Schweizer R, et al. Validation of automatic bone age rating in children with precocious and early puberty. J Pediatr Endocrinol Metab 2011;24:1009 –14 36 Martin DD, Heil K, Heckmann C, et al. Validation of automatic bone age determination in children with congenital adrenal hyperplasia. Horm Res 2010;74:116 37 Martin DD, Neuhof J, Jenni OG, et al. Automatic determination of leftand right-hand bone age in the First Zurich Longitudinal Study. Horm Res Paediatr 2010;74:50 –5 38 Kaplowitz P, Srinivasan S, He J, et al. Comparison of bone age readings by pediatric endocrinologists and pediatric radiologists using two bone age atlases. Pediatr Radiol 2011;41:690 – 3 39 Thodberg HH, Neuhof J, Ranke MB, et al. Validation of bone age methods by their ability to predict adult height. Horm Res Paediatr 2010;74:15 –22 40 Martin DD, Deusch D, Schweizer R, et al. Clinical application of automated Greulich-Pyle bone age determination in children with short stature. Pediatr Radiol 2009;39:598 –607 41 Martin DD, Schweizer R, Schriever L, et al. Validation of Automatic Greulich-Pyle Bone Age in Children with Tall Stature. Gemeinsame Jahrestagung der Arbeitsgemeinschaften fu¨r Pa¨diatrische Endokrinologie und Pa¨diatrische Diabetologie. 2010. Augsburg, Germany.
Downloaded from acr.sagepub.com at University of Otago Library on July 13, 2015
Bone age assessment: automated techniques coming of age? RR van Rijn1 and HH Thodberg2 1
Department of Radiology, Academic Medical Centre/Emma Children’s Hospital Amsterdam, the Netherlands; 2Visiana, Holte, Denmark Correspondence to: RR van Rijn. Email: [email protected]
Abstract Bone age determination from hand radiographs is one of the oldest radiographic procedures. The first atlas was published by Poland in 1898, and to date the Greulich Pyle atlas, although it dates from 1959, is still the most commonly used method. Bone age rating is time-consuming, suffers from an unsatisfactorily high rater variability, and therefore already 25 years ago it was proposed to replace the manual rating by an automated, computerized method, a field nowadays referred to as computer-aided diagnosis (CAD). The pursuit of this goal reached a first stage of accomplishment in 1992–1996 with the presentation of several systems. However, they had limited clinical value, and efforts in CAD research were increasingly focused on lesion detection for cancer screening. It was only in 2008 that a fully-automated bone age method was presented, which appears to be clinically acceptable. In this paper we consider the requirements that should be met by an automated bone age method and review the state of the art. Integration in PACS and saving time are important factors for radiologists. But it is the validation of the methods which poses the greatest challenge, because there is no gold standard for bone age rating, and the direct comparison to manual rating is therefore not sufficient for demonstrating that manual rating can be replaced by automated rating. One needs additional studies assessing the precision of a method and its accuracy when used for adult height prediction, which serves as an objective.
Keywords: Age determination by skeleton, automated pattern recognition, computer-assisted diagnosis Submitted June 27, 2012; accepted for publication October 26, 2012
Soon after the discovery of X-rays by Wilhelm Conrad Ro¨ntgen in November 1895 the first application of radiographs in the depiction of skeletal maturation was published. In 1898 John Poland (1855 –1937), an orthopedic surgeon working at Guy’s Hospital in London, UK, published the first bone age atlas: “skiagraphic atlas showing the development of bones of the wrist and hand” (1). This atlas depicts positive reprints (skiagraphs) of hand radiographs of 19 British children, aged 1 –17 years, with an elaborate description of each radiograph. In the following decade, Pryor (2) and Rotch (3) published scientific publications on this subject. The two most influential publications in this field were published in 1959 and 1962 when Greulich & Pyle (GP) and Tanner & Whitehouse & Healy (TW) respectively published their atlases (4, 5). In pediatric radiology today, the GP atlas is most widely adopted both by radiologists and pediatricians (6). The TW method, which takes a longer time to apply because it rates bones separately, is mostly used by endocrinologists.
These methods have several drawbacks. As visual grading techniques they have an inherent inter- and intraobserver variation. Several studies have addressed this issue and it has been shown that the standard error on a single determination in inter-observer studies ranges from 0.45 to 0.83 years (7 – 10). The GP method is relatively easy to learn, and after a short learning curve, readers can achieve an intra-observer variation comparable to that of experienced readers (10). For the GP method a second drawback is the lack of standardization in how the bones are weighted. In clinical practice, some raters assign one-third or even one-half weight to the carpals, while others ignore the carpals completely. Raters using the carpals reduce their importance at higher maturity but again not in a standardized manner (10). Finally, one issue which is specifically of interest to those who are involved in multicenter studies, is the variance in bone age assessments between centers (11). All studies in the inter-observer variation have been performed in single centers. As one can assume that within one center
Acta Radiologica 2013; 54: 1024–1029. DOI: 10.1258/ar.2012.120443 Downloaded from acr.sagepub.com at University of Otago Library on July 13, 2015
Bone age assessment
1025
.................................................................................................................................................
Fig 1. Hand radiograph analyzed using BoneXpert. Bone age using both Greulich and Pyle as well as Tanner Whitehouse are calculated. Additionally the bone health index is calculated
in general the same approach to bone age assessment is used, this might underestimate the difference between radiologists and pediatricians trained in different environments.
Historical perspective A solution to these problems could be the implementation of a (semi)-automated technique to assess bone age. This is not a new idea: already 25 years ago, several research groups proposed that bone age rating was a task that lends itself well to automation by computers, and started work towards achieving this goal. Thus in 1989, after having worked on computerized methods for several years, Tanner wrote with confidence: “Surely this is the way forward, eliminating the all-too-fallible rater entirely. Surely someone will eventually put up the money for the requisite hardware” (12). To date several systems, with a variable penetration into clinical practice have been introduced (13 –17). The first three papers, representing the birth of the topic, were HANDX (18) from 1989, Pietka (19) from 1991, and Tanner and Gibbons’ CASAS system from 1992 (20). CASAS is an interactive system analyzing the 13 bones of the TW RUS system. The individual bones were located manually, aided by templates presented on the screen,
and once identified, the maturation rating was done by the computer. This procedure had to be performed for each individual bone. Tanner showed that CASAS indeed yields a more consistent bone age assessment compared to manual rating (21). The major drawback of this technique was that the interactive method took more time than a manual TW bone age assessment. Although CASAS has been used in several studies it never became a widely accepted clinical tool (22, 23). Thus bone age rating was one of the first radiological procedures considered for automation. The reasons for this are that bone age rating is tedious, time-consuming and – most importantly – humans are not good at it, as reflected in the considerable inter- and intra-rater variability. The cause of this variability is that the radiologist does not look for a particular lesion, but rather at a plethora of maturity indicators – the more the better – to average out differences among bones. A consensus of these findings is made and expressed as a numerical value, the bone age, which places the image on a continuous maturity scale. Thus it is an example of quantitative image analysis, which is in general difficult for humans, who are more proficient at making qualitative statements about, for instance, the presence of a fracture. To go beyond CASAS it was necessary to achieve complete automation of the process, but this turned out to be much more difficult than thought initially. The area received much less research resources than computer-aided detection/diagnosis (CAD) for cancer screening, so this difficulty left the area in a state of “prolonged prepuberty” with very little progress for over a decade. In the last decade several publications on this topic have appeared (24 – 28). However, these publications have not led to commercially available tools, with one exception: in 2008 a fully automated method was developed and marketed under the name BoneXpert (14). The system locates the same 13 RUS bones of the hand as CASAS: Radius, ulna, and the bones in rays 1 and 5, but the system determines both Tanner-Whitehouse (TW2 or TW3) and Greulich-Pyle (GP) bone ages (Fig. 1).
Requirements to CAD So while the need for automated bone age analysis was realized 25 years ago, it has had a remarkably long childhood. The similar vision of CAD for mammography screening grew up much faster and resulted in the first FDA-approved system already in 1998 (29). Recently, a general analysis of CAD was presented in a paper entitled “Computer-aided diagnosis: how to move from the laboratory to the clinic” and we will use the concepts in this paper in the continued discussion (29). In this publication van Ginneken et al. define CAD in a wider sense, including not only detection and diagnosis but also quantitative image analysis, because it is based on the same technology and has increased the interest in CAD in recent years. According to the authors, “Computerized quantification may hold more potential than computerized detection. When radiologists are asked to name aspects of
Downloaded from acr.sagepub.com at University of Otago Library on July 13, 2015
1026
RR van Rijn and HH Thodberg
.................................................................................................................................................
their work that are common, time consuming, and could be automated, they usually do not mention detection but rather documentation and quantification.” The paper proposes four requirements which should be met in order for a CAD method to be successful in clinical practice: (a) (b) (c) (d)
CAD should improve radiologists’ performance; CAD should save time; CAD must be seamlessly integrated into the workflow; CAD should not impose liability concerns and the incremental costs should be negligible or reimbursed.
The CASAS system did indeed demonstrate improvements in precision of the rating. But it scored poorly on the timesaving, as it actually required more time. It took place on a dedicated workstation, where a camera was used to manually align each bone to a template, so it was not integrated in the workflow. The fully automated BoneXpert method conforms better, although not fully, to van Ginneken et al.’s requirements. Compared to manual rating it makes fewer gross errors, it has better precision, and it yields a better accuracy when used for adult height prediction, and these improvements are clinically significant (30, 31). It saves the time used for manual rating, estimated between 1 and 10 min and can be implemented as a PACS plug-in leading to seamless integration in the workflow. In Europe it is a CE-marked medical device intended to replace or supplement the radiologist, and validated as such. However, the costs are currently not reimbursed. The most significant other research group, which pursues the development of an automated method, has published several papers over more than a decade (16, 19, 24). They have addressed the issue of clinical workflow with a system, which is not commercially available, but which is intended to assist the radiologist in the bone age rating by suggesting similar images from a large atlas. The precise use and the performance of the system has not yet been published, but what has been revealed is a detailed description of the integration in the PACS workflow, which as van Ginneken mentions, is essential (32). The same group has also presented a method for rating bone age of carpals with promising results (23).
Bone age assessment can also be useful in the diagnosis of disorders/syndromes and/or skeletal dysplasias. The hand radiograph can also be of help in the diagnosis of disorders/syndromes, irrespective of the presence of abnormal bone maturation. For this it is essential that the radiograph is reported by a ( pediatric) radiologist. Examples of such disorders/syndromes are Turner syndrome (squared off carpals and irregular metaphysis), rickets (fraying and splaying of metaphysis), hypochondroplasia (with broader and stump bones), and pseudohyporathyroidism (short fourth and fifth metacarpal bones). Chronic diseases can also manifest themselves on the hand radiograph; examples of these are renal osteodystrophy (subperiosteal bone resorption) and juvenile idiopathic arthritis ( periarticular osteopenia). In children with skeletal dysplasias radiographs of the left hand are a part of the skeletal survey and as such play a role in the diagnostic work-up. In these cases the assessment of bone age, due to a disturbed anatomy and development, is difficult and the results should be interpreted with caution. Finally, bone age assessment is mandatory in all patients treated for growth disorders in order to monitor therapy. In the individual patient automated bone age assessment has the advantage that it is an unbiased tool, whereas the radiologist/clinician (if aware of treatment and/or chronological age of the patient) might be influenced in his/her bone age assessment. It could be argued that in longitudinal studies only the first hand radiograph has to be reviewed by a ( pediatric) radiologist, in order to detect anomalies which will not be detected by an automated bone age assessment. Subsequent radiographs would not necessarily be read and the assessment could be done in a completely automated fashion.
Validation studies of automated bone rating Van Ginneken et al. pointed out that validation of CAD systems is often a major challenge, which may require development of new types of studies and new metrics to quantify the clinical value. Automated bone age rating is no exception, and this section reviews the recent development in this area. Accuracy
Clinical application In 2009 a group of experts met for the first International Workshop on Skeletal Maturity in Tu¨bingen, Germany. Partly based on the discussions at this meeting two papers on the use of bone age in clinical practice were published (11, 33). There are two main reasons for the initial assessment of bone age: short and tall stature. Besides these it can be useful in clinical pediatric oncology trials, in order to assess late effects of chemotherapy and radiotherapy. In these groups bone age assessment is the main objective of the study and automated systems can have a significant impact in patient care.
For assessment of accuracy, automatic bone age methods should be compared with manual rating. Table 1 presents six such studies performed for the BoneXpert method. They used Bland-Altman plots and revealed an SD of the difference of typically 0.72 years. This is of the same magnitude as the SD between two manual raters so this seems to indicate that the automated method is valid. The studies cover the major disorders where bone age is used and also cover four different ethnicities, and the SDs listed in Table 1 are more or less the same across these groups. Table 1 also lists the biases, defined as the average difference between automated and manual rating, and they are small. However, according to Bland-Altman’s methodology, such studies only indicate that manual and automated rating
Downloaded from acr.sagepub.com at University of Otago Library on July 13, 2015
Bone age assessment
1027
.................................................................................................................................................
Table 1 Studies comparing manual and automated rating
Population Normal Caucasian Short stature Tall stature Normal, Four ethnicities Precocious puberty Congenital adrenal hyperplasia
Radiographs (n) 538 1084 812 708 741 892
SD (y) 0.71 0.72 0.55 0.61† 0.76 0.72
Bias (y)
0 0 0.13 0.07 – 0.19 – 0.02
Fraction of dispsuted cases where the reference is closer to automated than to manual rating
Reference
7/7 18/27 15/19 12/15 23/41 18/20
(31) (40) (41) (8) (35) (36)
These two studies where used to adjust the bias of the method so the bias is close to zero by construction The manual rating was the average of two observers’ ratings A positive bias indicates that the automated method on average gives larger values than manual rating †
are the same with a tolerance of 2 SD ¼ 1.44 years, quite a large margin, so clearly these studies alone are not sufficient for validation of a new method. In order to challenge the automated method more, these studies went a step further and focused on the cases where the deviation between the two methods was larger than a certain threshold typically set to 1.5 years. These “disputed cases” were carefully rerated blindly by two to four raters and the average of these ratings was defined as “the reference rating”. Then it was assessed whether the reference ratings agreed better with the automated rating than with the original manual rating, and Table 1 lists proportion of the disputed cases where this was true. The automated method was the more correct method in five of six studies, while in the one study, on children with precocious puberty there was no significant difference. In the study on precocious puberty the authors did not specifically address this issue and did not present an explanation for this finding. Table 1 demonstrates that replacing manual by automated rating leads to a safer rating, in the sense that there will be fewer cases which are rated very wrong. This supports the intended use of the method as a stand-alone device, with no need of manual supervision.
the rater precision to be 0.58 years (calculation based on the same data, private communication with the first author), so the precision of the automated rating is three times better compared to manual rating (38). This is the most conspicuous advantage of the automated method. The dilemma for automated bone age assessment The studies reviewed so far have shown that the new automated GP bone age method agrees as well with manual rating as one could expect (SD 0.72 y) for a completely new bone age method, and that it makes less severe errors than manual raters, and that it has an excellent precision. But the devil’s advocate would still point to one dilemma: the automated method could be measuring something slightly else than GP bone age. To address this dilemma, it should first be pointed out that manual GP bone age is not the ultimate “truth” about bone age, rather it is one possible convention, and there are other conventions in use, namely TW and Fels bone ages. There is no direct way to determine the true maturity of a hand; one cannot, for example, dissect the hand and perform a measurement by any reference method. So the deviation of the automated rating of + 1.44 y (2 SD) from manual GP rating should not necessarily be taken against automated rating.
Precision Two types of studies assessing the precision of a bone age method have been defined as its ability to yield the same result on a repeated X-ray image. The first type utilized longitudinal series of X-rays taken at, for instance, 6-month intervals (34). The smoothness of the series was used to estimate the precision of the automated bone age to 0.17 years, meaning the standard error on a single measurement. A study in precocious puberty children found 0.23 years, and in a study on children with Congenital Adrenal Hypoplasia a standard error of 0.21 years was found (35, 36). The second type used 2700 pairs of X-rays of the left and right hands obtained on the same day to estimate the precision to 0.18 years (37). The study found no average difference between the bone age of the left and the right hand. The precision of manual rating varies considerable with the experience of the raters and the time spent on the rating. A recent study compared 12 raters and found
Adult height prediction To resolve the dilemma, a new framework for validating bone age methods was developed, based on the ability to predict the adult height (39). Adult height prediction is indeed the most important application of bone age, and the framework proposed to assess the validity (or trueness) of a bone age method by its ability to predict the adult height. This is an objective and highly relevant endpoint. By using the Zurich Longitudinal study of 232 normal children born in 1954 – 1956 it was found that automated GP bone age was at least as good as manual GP bone age for predicting adult height (39). It was also found that manual GP bone age performed significantly better than manual TW bone age, which came as a surprise to many. This shows that the new automated method does indeed measure true bone age, and not some other value with unknown biological meaning. Another paper continued along these lines by constructing a new adult height prediction model based on the
Downloaded from acr.sagepub.com at University of Otago Library on July 13, 2015
1028
RR van Rijn and HH Thodberg
.................................................................................................................................................
same longitudinal data from Zurich (30). The new method can be seen as a successor of the classical Bayley-Pinneau method, but using better mathematics and more recent data. The model was validated on a later study from Zurich of children born in 1973 – 1991, and the model was found to predict with the same accuracy as on the older data.
Conclusion on the clinical use of automated bone age assessment There is now good scientific evidence that automated bone age assessment has come of age. It has been shown that automated systems have a better precision and accuracy compared to radiologists’ reading. The integration in the PACS workflow is essential, and the next step would be to develop a standardized way to enable the bone age value to flow into other hospital information systems, e.g. EPJ, where it can be used for adult height prediction. As a result, the automated bone age assessment can support and in some cases replace the radiologists’ report. As automated bone age systems are not developed to detect anomalies in skeletal anatomy, radiological assessment of at least the first radiograph of each child is mandatory. In studies in which surgical intervention, such as epiphysiodesis has taken place, manual assessment is also mandatory as this influences automatic bone age assessment. In multicenter studies the use of automated systems will obviate the need for independent central readers. In some cases, however, the referring physician might request a general diagnosis of the child from the image, and the radiologist then needs to report the radiograph for that specific request. Additionally, automated systems will not be able to rate all radiographs, either due to age constraints or anatomical variation or malpositioning, in these cases skeletal age assessment will remain a manual task. Limitations of the reported systems are that in some, the carpals and in other the phalanges are not included. Another disadvantage is that the bone age range is limited to specific age groups, where none of the published automated techniques covers the whole age range. Another valuable extension would be to diagnose various conditions from the bone lengths and other morphological features, but this is potentially much more complex, and there seems to be a long way to go before computers can replace the radiologist’s general review of the hand radiograph. Conflict of interest: The BoneXpert technology is proprietary to Visiana, a company owned by HH Thodberg. BoneXpert has been marketed as a medical device. A US patent on BoneXpert has been issued.
REFERENCES 1 Poland J. Skiagraphic atlas showing the development of bones of the wrist and hand. London: Smith, Elder and Co., 1898 2 Pryor JW. The heriditary nature of variation in the ossification of bones. Anat Rec 1907;1:84– 88
3 Rotch TM. A study of the development of bones in childhood by the roentgen method with the view of establishing a development index for the grading of and the protection of early life. Trans Amer Ass Phys 1909;24:603 –30 4 Greulich WW, Pyle SI. Radiographic atlas of skeletal development of the hand and wrist. Stanford: Stanford University Press, 1959 5 Tanner JM, Whitehouse RH, Healy MJR. A new system for estimating skeletal maturity from the hand and wrist with standards derived from a study of 2600 healthy british children. Paris: Centre International de L’enfance, 1962. 6 Buckler JM. How to make the most of bone ages. Arch Dis Child 1983;58:761 –3 7 Bull RK, Edwards PD, Kemp PM, et al. Bone age assessment: a large scale comparison of the Greulich and Pyle, and Tanner and Whitehouse (TW2) methods. Arch Dis Child 1999; 81:172 –3 8 Thodberg HH, Savendahl L. Validation and reference values of automated bone age determination for four ethnicities. Acad Radiol 2010;17:1425 – 32 9 Johnson GF, Dorst JP, Kuhn JP, et al. Reliability of skeletal age assessments. Am J Roentgenol Radium Ther Nucl Med 1973; 118:320 – 7 10 Roche AF, Rohmann CG, French NY, et al. Effect of training on replicability of assessments of skeletal maturity (Greulich-Pyle). Am J Roentgenol Radium Ther Nucl Med 1970;108:511 – 5 11 Martin DD, Wit JM, Hochberg Z, et al. The use of bone age in clinical practice - part 1. Horm Res Paediatr 2011;76:1– 9 12 Tanner JM. Assessing the skeletal maturity of the hand-wrist: FELS method. Am J Human Biol 1989;1:493 –4 13 Tanner JM, Gibbons RD. Automatic bone age measurement using computerized image analysis. J Pediatr Endocrinol 1994; 7:141 –5 14 Thodberg HH, Kreiborg S, Juul A, et al. The BoneXpert method for automated determination of skeletal maturity. IEEE Trans Med Imaging 2009;28:52 –66 15 Hill K, Pynsent PB. A fully automated bone-ageing system. Acta Paediatr Suppl 1994;406:81 –3 16 Pietka E, Gertych A, Pospiech S, et al. Computer-assisted bone age assessment: image preprocessing and epiphyseal/metaphyseal ROI extraction. IEEE Trans Med Imaging 2001;20:715 – 29 17 Liu J, Qi J, Liu Z, et al. Automatic bone age assessment based on intelligent algorithms and comparison with TW3 method. Comput Med Imaging Graph 2008;32:678 – 84 18 Michael DJ, Nelson AC. HANDX: a model-based system for automatic segmentation of bones from digital hand radiographs. IEEE Trans Med Imaging 1989;8:64 –9 19 Pietka E, McNitt-Gray MF, Kuo ML, et al. Computer-assisted phalangeal analysis in skeletal age assessment. IEEE Trans Med Imaging 1991;10:616 –20 20 Tanner JM, Gibbons RD, Bock RD. An image analysis system for TW skeletal maturity. Horm Res 1992;37:11 21 Tanner JM, Oshman D, Lindgren G, et al. Reliability and validity of computer-assisted estimates of Tanner-Whitehouse skeletal maturity (CASAS): comparison with the manual method. Horm Res 1994;42:288 –94 22 Frisch H, Riedl S, Waldhor T. Computer-aided estimation of skeletal age and comparison with bone age evaluations by the method of Greulich-Pyle and Tanner-Whitehouse. Pediatr Radiol 1996;26:226 –31 23 Van TA, De WW, Roks A, et al. Computer-aided skeletal age scores in healthy children, girls with Turner syndrome, and in children with constitutionally tall stature. Pediatr Res 1996;39:360 –7 24 Zhang A, Gertych A, Liu BJ. Automatic bone age assessment for young children from newborn to 7-year-old using carpal bones. Comput Med Imaging Graph 2007;31:299 – 310 25 Somkantha K, Theera-Umpon N, Auephanwiriyakul S. Bone age assessment in young children using automatic carpal bone feature extraction and support vector regression. J Digit Imaging 2011;24:1044 – 58 26 Hsieh CW, Liu TC, Jong TL, et al. A fuzzy-based growth model with principle component analysis selection for carpal bone-age assessment. Med Biol Eng Comput 2010;48:579 – 88
Downloaded from acr.sagepub.com at University of Otago Library on July 13, 2015
Bone age assessment
1029
.................................................................................................................................................
27 Hsieh CW, Jong TL, Tiu CM. Bone age estimation based on phalanx information with fuzzy constrain of carpals. Med Biol Eng Comput 2007;45:283 –95 28 Rucci M, Coppini G, Nicoletti I, et al. Automatic analysis of hand radiographs for the assessment of skeletal age: a subsymbolic approach. Comput Biomed Res 1995;28:239 –56 29 van GB, Schaefer-Prokop CM, Prokop M. Computer-aided diagnosis: how to move from the laboratory to the clinic. Radiology 2011;261:719 – 32 30 Thodberg HH, Jenni OG, Caflisch J, et al. Prediction of adult height based on automated determination of bone age. J Clin Endocrinol Metab 2009;94:4868 – 74 31 van Rijn RR, Lequin MH, Thodberg HH. Automatic determination of Greulich and Pyle bone age in healthy Dutch children. Pediatr Radiol 2009;39:591 –7 32 Gertych A, Zhang A, Sayre J, et al. Bone age assessment of children using a digital hand atlas. Comput Med Imaging Graph 2007; 31:322 –31 33 Martin DD, Wit JM, Hochberg Z, et al. The use of bone age in clinical practice - part 2. Horm Res Paediatr 2011;76:10– 6 34 Martin DD, Sato K, Sato M, et al. Validation of a new method for automated determination of bone age in Japanese children. Horm Res Paediatr 2010;73:398 – 404
35 Martin DD, Meister K, Schweizer R, et al. Validation of automatic bone age rating in children with precocious and early puberty. J Pediatr Endocrinol Metab 2011;24:1009 –14 36 Martin DD, Heil K, Heckmann C, et al. Validation of automatic bone age determination in children with congenital adrenal hyperplasia. Horm Res 2010;74:116 37 Martin DD, Neuhof J, Jenni OG, et al. Automatic determination of leftand right-hand bone age in the First Zurich Longitudinal Study. Horm Res Paediatr 2010;74:50 –5 38 Kaplowitz P, Srinivasan S, He J, et al. Comparison of bone age readings by pediatric endocrinologists and pediatric radiologists using two bone age atlases. Pediatr Radiol 2011;41:690 – 3 39 Thodberg HH, Neuhof J, Ranke MB, et al. Validation of bone age methods by their ability to predict adult height. Horm Res Paediatr 2010;74:15 –22 40 Martin DD, Deusch D, Schweizer R, et al. Clinical application of automated Greulich-Pyle bone age determination in children with short stature. Pediatr Radiol 2009;39:598 –607 41 Martin DD, Schweizer R, Schriever L, et al. Validation of Automatic Greulich-Pyle Bone Age in Children with Tall Stature. Gemeinsame Jahrestagung der Arbeitsgemeinschaften fu¨r Pa¨diatrische Endokrinologie und Pa¨diatrische Diabetologie. 2010. Augsburg, Germany.
Downloaded from acr.sagepub.com at University of Otago Library on July 13, 2015
