Journal of Autism and Childhood Schizophrenia, Vol. 6, No. 3, 1976
The Reliability and Diagnostic Validity of the Physical and Neurological Examination for Soft Signs (PANESS) 1 John S. Werry and Michael G. A m a n ~ University of Auckland, Auckland, New Zealand
Twenty-one children, mean age o f 8 years, were each examined on separate occasions by two pediatric residents, blind to diagnosis, using the neurological examination (PANESS) included in the group o f instruments recommended by the National Institute o f Mental Health f o r psychotropic drug studies in children. Half the children were hyperactive/aggressive, one quarter were normal, and one quarter had histories or signs strongly presumptive o f brain damage. Many o f the signs, though reliable, did not occur in the majority o f children. Examiners did achieve a high level o f agreement about global neurological status. It was concluded that the neurological examination probably contains a substantial number o f noncontributory items and should be regarded as experimental rather than definitive. INTRODUCTION
It has long been recognized that children with psychiatric disorders, while seldom having major neurological signs, often have a cluster of what have come to be called " s o f t " or equivocal signs involving minor abnormalities of reflexes and tone, but above all of sensorimotor coordination (Werry, 1972). Thus, to have any potential usefulness in child psychiatry, a system of examination must include minor as well as major signs. 'This study was supported in part by a grant to Professor Werry from the Medical Research Council of New Zealand and USPHS grant #MH 18909 from the National Institute of Health to R. L. Sprague, Ph.D. Drs. M. Hudson and M. Morris performed the examinations. We should like to pay particular tribute to Dr. Thelma Becroft, a school doctor in Auckland, who supplied the normal and neurological subjects. 2Requests for reprints should be addressed to Prof. J. S. Werry, Department of Psychiatry, School of Medicine, University of Auckland, P.B., Auckland, New Zealand. 253 9 1976 Plenum Publishing Corporation, 227 West 17th Street, New Y o r k , N . Y . 1 0 0 1 1 . NO part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, microfilming, recording, or otherwise, without written permission of the publisher.
254
Werry and Aman
While there have been efforts in the past (e.g., Paine & Oppe, 1966; Ozer, 1969; Rutter, Graham, & Yule, 1970; Werry, Minde, Guzman, Weiss, Dogan, & Hoy, 1970) to systematize wide-range neurological examinations for child psychiatric patients, there is as yet no generally accepted method (Werry et al., 1970). Also, conspicuously lacking with only a few exceptions (Ozer, 1969; Rutter et al., 1970; Werry et al., 1970) are psychometric studies of the reliability and validity of this type of neurological examination in children (Werry, 1972). The most sophisticated examination of all, a children's version of the Reitan Battery (Reitan & Heinemann, 1969) is cumbersome, requires expensive equipment, and is difficult to score and interpret. One of the problem areas in pediatric psychopharmacology is the prediction of those children who are likely to respond to medication. Broad psychiatric diagnostic pointers are known, as are behavioral target symptoms (Close, 1973), but there has been interest in predictors which relate more directly to central nervous function (Wender, 1971). A recurrent theme through the literature is that children with neurological signs or "organic" children respond (or perhaps equally often do not respond) to medication better than children who lack these signs (Conners, 1972; Kornetsky, 1970; Wender, 1971; Werry, 1972). Close (1973) has compiled a neurological examination especially for drug studies in children and this has been incorporated into the recently published (Psychopharmacology Bulletin, 1973) children's battery of psychopharmacological measures compiled by the Early Clinical Drug Evaluation Unit, (ECDEU) of the National Institute of Mental Health where it appears as PANESS (Physical and Neurological Examination for Soft Signs). It is important, therefore, since none appears to be available yet for this now official instrument, that data attesting to the reliability and validity be acquired before another test of unknown scientific worth becomes in d extricably molded into the technique and literature of pediatric psychopharmacology. The types of reliability of interest are interexaminer and test-retest reliability. The validities of concern are those of ability to discriminate among children who are normal, have minimal brain dysfunction, and are neurologically disordered and to make predictions about drug response. The study to be described in this paper is concerned with interexaminer reliability and discriminative power of the individual signs and the examination as a whole. However, as will become apparent below, certain admittedly unproven assumptions about test-retest reliability were necessarily made.
PANESS Examination for Soft Signs
255
METHOD
Subjects The children for this study were selected from three sources to provide, it was hoped, a wide spectrum of both type and number of signs. Six subjects were normal children in the local school system. Ten were in an ongoing project involved with the evaluation and treatment of hyperactive/ aggressive children, and five were children from the local school system considered by the school doctor to have major neurological impairment but without mental retardation. The hyperactive/aggressive children employed in the study were fairly extreme behaviorally as judged by Conners' Teacher Questionnaire (1969). Their standard scores (relative to a group of normals) (Sprague, Christensen, & Werry, 1974) on the Conduct, Inattentive, and Hyperactive factors were 4.07, 2.10, and 3.49, respectively. Background details of the three groups are presented in Table I.
Procedure Two senior residents in pediatrics at one of the University of Auckland's teaching hospitals served as the examiners. Due to logistical problems, 62% of the children were seen first by Examiner A then by Examiner B while the remainder were first seen by Examiner B. For similar reasons the interval between examinations varied from 1 to 110 days (median of 5 days). A stipend for each child examined was paid to the doctors since the examinations were necessarily somewhat tedious. In order to enhance the prospects of obtaining reliability, a number of features of the procedure were maintained constant. None of the children Table I. Characteristics of the Three Groups
N Mean age (months) Age range Median time between exares (days) Percent examined by A first
Normal
Hyperactive
Neurological
Total
6 119.8
10 90.2
5 105.6
21 96.6
104-143
61-121
72-146
61-146
4
8.5
2
5
83.3
50.0
60.0
61.9
256
Werry and Aman
received psychotropic medication on or immediately preceding the days they were tested. All examinations were conducted in the same room and the time o f day o f the two tests was always the same for a given child. Neither doctor had any knowledge o f the personal history or the diagnosis of the subjects. The system of examination was strictly according to the published protocol (Close, 1973) and the two doctors practiced the examinations on other children until they felt reasonably familiar with it. Each child was offered a reward for his cooperation over the two half-hour examinations.
RESULTS AND DISCUSSION
Reliability and Occurrence of Items The examination consists of 43 items, some of which are scored in more than one dimension to yield a total of 56 scores. Most range in some way or other over a 4-point scale of " n o impairment" through "severe impairment." To be useful, an item should be reliable and contribute to the ability o f the examination to discriminate between different diagnostic groups. The subjects o f this study were chosen on the assumption, based on previous studies (Werry, 1972; Werry et al., 1970), that they would present a range o f scores enabling judgments to be made about the items. The following possibilities obtain for a given item: (1) It is unreliable. (2) It is reliable, but it does not occur or it occurs too infrequently to be of much use and simply prolongs the examination. (3) It is reliable and occurs in a range o f values but does not discriminate between diagnostic groups. (4) It is reliable, occurs, and discriminates. Eighty-six percent o f the total signs occurred at least once in the judgments o f both examiners. However, most o f these signs appeared in only one or two children, and only 36% o f the items were judged as occurring in a sample as small as 20% of the children. Only 12% o f the signs occurred when the criterion was raised to 50% of the children. This suggests that a large number of items are probably noncontributory. The level of occurrence for each item of PANESS has been listed in Table II. One way of depicting the qualities o f reliability and occurrence is by constructing a four by four contingency table (such as is done for a X2 test) for each item in which each child's position is plotted with one examiner's score along the abscissa and the other along the ordinate. The ideal diagnostic sign then emerges with scores along the diagonal while a reliable but nonoccurring sign clusters in one corner and an unreliable one has scores
PANESS Examination for Soft Signs
257
Table II. Level of Occurrence for Items of PANESS % of subjects in whom sign was observed by both examiners 0-10%
11-20% 21-30% 31-40% 41-50% 51 60% 61-70% 71-80% 81-90% 91-100%
Itemsa 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21, 25, 27, 28, 29, 30, 31, 32, 35A, 35B,36B, 38B, 38C, 43A, 43B 10, 23, 33, 36A, 37B, 37C, 40B 13, 24, 26, 34, 39B, 42B 22, 40C 38A, 39C, 41B None 37A, 42C None 40A, 41C, 42A 39A, 41A
aWhere letters occur following numbers, they refer to the respective subcomponent of that item. Items falling into the highest frequency categorieswould appear to occur too often. Inspection of the data indicated that the degree of "pathology" was more extreme in the hyperactive and neurological groups than in the normal group. Items axe identified in the appendix. scattered randomly around the table. Examples of these three possibilities are set out in Table III. Unfortunately, Example 1 (reliable but not occurring) represents by far the commonest situation. It is customary, however, to rely on some numerical method of describing this p h e n o m e n o n of reliability and occurrence. A crude estimate o f reliability was made by calculating the percentage of items in which there w~ts agreement within one point for 75% or more o f the children. Ninetythree percent o f items met this criterion of agreement within one point. However, this takes no cognizance o f occurrence and, as pointed out by Fleiss, Spitzer, Endicott, and Cohen (1972), makes no allowance for agreement based simply on chance. Selection o f a statistic presented problems due to the ordinal nature of the scale and the small number o f gradations. TB, a correlation coefficient proposed by Kendall and Stewart (1961) to measure the association between two ranking_s with many ties appeared particularly suitable for these data, though it is vulnerable to near perfect nonoccurrence exemplified in Example I, Table III. Compared to the other T Statistics, T C and TG, it has the property o f reaching an intermediate magnitude, and after inspection o f the data, T B was selected as the most appropriate summary statistic o f the three (Table IV).
258
Werry and Aman Table III a Example 1: Reliable b u t n o t occurring
2~ < X
4 3 2
0 0 0
0 0 1
0 0 0
0 0 0
T B = .548 T C = .109
1
18
2
0
0
T G = 1.00
1
2 3 EXAMINER A
4
Example 2: Reliable and occurring
< X
4 3 2 1
0 0 3 1 1
0 2 6 2
0 3 1 1
1 0 0 0
2
3
4
T B = .411 TC'=- .353 T G = .582
EXAMINER A Example 3: Unreliable r 2:
The Reliability and Diagnostic Validity of the Physical and Neurological Examination for Soft Signs (PANESS) 1 John S. Werry and Michael G. A m a n ~ University of Auckland, Auckland, New Zealand
Twenty-one children, mean age o f 8 years, were each examined on separate occasions by two pediatric residents, blind to diagnosis, using the neurological examination (PANESS) included in the group o f instruments recommended by the National Institute o f Mental Health f o r psychotropic drug studies in children. Half the children were hyperactive/aggressive, one quarter were normal, and one quarter had histories or signs strongly presumptive o f brain damage. Many o f the signs, though reliable, did not occur in the majority o f children. Examiners did achieve a high level o f agreement about global neurological status. It was concluded that the neurological examination probably contains a substantial number o f noncontributory items and should be regarded as experimental rather than definitive. INTRODUCTION
It has long been recognized that children with psychiatric disorders, while seldom having major neurological signs, often have a cluster of what have come to be called " s o f t " or equivocal signs involving minor abnormalities of reflexes and tone, but above all of sensorimotor coordination (Werry, 1972). Thus, to have any potential usefulness in child psychiatry, a system of examination must include minor as well as major signs. 'This study was supported in part by a grant to Professor Werry from the Medical Research Council of New Zealand and USPHS grant #MH 18909 from the National Institute of Health to R. L. Sprague, Ph.D. Drs. M. Hudson and M. Morris performed the examinations. We should like to pay particular tribute to Dr. Thelma Becroft, a school doctor in Auckland, who supplied the normal and neurological subjects. 2Requests for reprints should be addressed to Prof. J. S. Werry, Department of Psychiatry, School of Medicine, University of Auckland, P.B., Auckland, New Zealand. 253 9 1976 Plenum Publishing Corporation, 227 West 17th Street, New Y o r k , N . Y . 1 0 0 1 1 . NO part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, microfilming, recording, or otherwise, without written permission of the publisher.
254
Werry and Aman
While there have been efforts in the past (e.g., Paine & Oppe, 1966; Ozer, 1969; Rutter, Graham, & Yule, 1970; Werry, Minde, Guzman, Weiss, Dogan, & Hoy, 1970) to systematize wide-range neurological examinations for child psychiatric patients, there is as yet no generally accepted method (Werry et al., 1970). Also, conspicuously lacking with only a few exceptions (Ozer, 1969; Rutter et al., 1970; Werry et al., 1970) are psychometric studies of the reliability and validity of this type of neurological examination in children (Werry, 1972). The most sophisticated examination of all, a children's version of the Reitan Battery (Reitan & Heinemann, 1969) is cumbersome, requires expensive equipment, and is difficult to score and interpret. One of the problem areas in pediatric psychopharmacology is the prediction of those children who are likely to respond to medication. Broad psychiatric diagnostic pointers are known, as are behavioral target symptoms (Close, 1973), but there has been interest in predictors which relate more directly to central nervous function (Wender, 1971). A recurrent theme through the literature is that children with neurological signs or "organic" children respond (or perhaps equally often do not respond) to medication better than children who lack these signs (Conners, 1972; Kornetsky, 1970; Wender, 1971; Werry, 1972). Close (1973) has compiled a neurological examination especially for drug studies in children and this has been incorporated into the recently published (Psychopharmacology Bulletin, 1973) children's battery of psychopharmacological measures compiled by the Early Clinical Drug Evaluation Unit, (ECDEU) of the National Institute of Mental Health where it appears as PANESS (Physical and Neurological Examination for Soft Signs). It is important, therefore, since none appears to be available yet for this now official instrument, that data attesting to the reliability and validity be acquired before another test of unknown scientific worth becomes in d extricably molded into the technique and literature of pediatric psychopharmacology. The types of reliability of interest are interexaminer and test-retest reliability. The validities of concern are those of ability to discriminate among children who are normal, have minimal brain dysfunction, and are neurologically disordered and to make predictions about drug response. The study to be described in this paper is concerned with interexaminer reliability and discriminative power of the individual signs and the examination as a whole. However, as will become apparent below, certain admittedly unproven assumptions about test-retest reliability were necessarily made.
PANESS Examination for Soft Signs
255
METHOD
Subjects The children for this study were selected from three sources to provide, it was hoped, a wide spectrum of both type and number of signs. Six subjects were normal children in the local school system. Ten were in an ongoing project involved with the evaluation and treatment of hyperactive/ aggressive children, and five were children from the local school system considered by the school doctor to have major neurological impairment but without mental retardation. The hyperactive/aggressive children employed in the study were fairly extreme behaviorally as judged by Conners' Teacher Questionnaire (1969). Their standard scores (relative to a group of normals) (Sprague, Christensen, & Werry, 1974) on the Conduct, Inattentive, and Hyperactive factors were 4.07, 2.10, and 3.49, respectively. Background details of the three groups are presented in Table I.
Procedure Two senior residents in pediatrics at one of the University of Auckland's teaching hospitals served as the examiners. Due to logistical problems, 62% of the children were seen first by Examiner A then by Examiner B while the remainder were first seen by Examiner B. For similar reasons the interval between examinations varied from 1 to 110 days (median of 5 days). A stipend for each child examined was paid to the doctors since the examinations were necessarily somewhat tedious. In order to enhance the prospects of obtaining reliability, a number of features of the procedure were maintained constant. None of the children Table I. Characteristics of the Three Groups
N Mean age (months) Age range Median time between exares (days) Percent examined by A first
Normal
Hyperactive
Neurological
Total
6 119.8
10 90.2
5 105.6
21 96.6
104-143
61-121
72-146
61-146
4
8.5
2
5
83.3
50.0
60.0
61.9
256
Werry and Aman
received psychotropic medication on or immediately preceding the days they were tested. All examinations were conducted in the same room and the time o f day o f the two tests was always the same for a given child. Neither doctor had any knowledge o f the personal history or the diagnosis of the subjects. The system of examination was strictly according to the published protocol (Close, 1973) and the two doctors practiced the examinations on other children until they felt reasonably familiar with it. Each child was offered a reward for his cooperation over the two half-hour examinations.
RESULTS AND DISCUSSION
Reliability and Occurrence of Items The examination consists of 43 items, some of which are scored in more than one dimension to yield a total of 56 scores. Most range in some way or other over a 4-point scale of " n o impairment" through "severe impairment." To be useful, an item should be reliable and contribute to the ability o f the examination to discriminate between different diagnostic groups. The subjects o f this study were chosen on the assumption, based on previous studies (Werry, 1972; Werry et al., 1970), that they would present a range o f scores enabling judgments to be made about the items. The following possibilities obtain for a given item: (1) It is unreliable. (2) It is reliable, but it does not occur or it occurs too infrequently to be of much use and simply prolongs the examination. (3) It is reliable and occurs in a range o f values but does not discriminate between diagnostic groups. (4) It is reliable, occurs, and discriminates. Eighty-six percent o f the total signs occurred at least once in the judgments o f both examiners. However, most o f these signs appeared in only one or two children, and only 36% o f the items were judged as occurring in a sample as small as 20% of the children. Only 12% o f the signs occurred when the criterion was raised to 50% of the children. This suggests that a large number of items are probably noncontributory. The level of occurrence for each item of PANESS has been listed in Table II. One way of depicting the qualities o f reliability and occurrence is by constructing a four by four contingency table (such as is done for a X2 test) for each item in which each child's position is plotted with one examiner's score along the abscissa and the other along the ordinate. The ideal diagnostic sign then emerges with scores along the diagonal while a reliable but nonoccurring sign clusters in one corner and an unreliable one has scores
PANESS Examination for Soft Signs
257
Table II. Level of Occurrence for Items of PANESS % of subjects in whom sign was observed by both examiners 0-10%
11-20% 21-30% 31-40% 41-50% 51 60% 61-70% 71-80% 81-90% 91-100%
Itemsa 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21, 25, 27, 28, 29, 30, 31, 32, 35A, 35B,36B, 38B, 38C, 43A, 43B 10, 23, 33, 36A, 37B, 37C, 40B 13, 24, 26, 34, 39B, 42B 22, 40C 38A, 39C, 41B None 37A, 42C None 40A, 41C, 42A 39A, 41A
aWhere letters occur following numbers, they refer to the respective subcomponent of that item. Items falling into the highest frequency categorieswould appear to occur too often. Inspection of the data indicated that the degree of "pathology" was more extreme in the hyperactive and neurological groups than in the normal group. Items axe identified in the appendix. scattered randomly around the table. Examples of these three possibilities are set out in Table III. Unfortunately, Example 1 (reliable but not occurring) represents by far the commonest situation. It is customary, however, to rely on some numerical method of describing this p h e n o m e n o n of reliability and occurrence. A crude estimate o f reliability was made by calculating the percentage of items in which there w~ts agreement within one point for 75% or more o f the children. Ninetythree percent o f items met this criterion of agreement within one point. However, this takes no cognizance o f occurrence and, as pointed out by Fleiss, Spitzer, Endicott, and Cohen (1972), makes no allowance for agreement based simply on chance. Selection o f a statistic presented problems due to the ordinal nature of the scale and the small number o f gradations. TB, a correlation coefficient proposed by Kendall and Stewart (1961) to measure the association between two ranking_s with many ties appeared particularly suitable for these data, though it is vulnerable to near perfect nonoccurrence exemplified in Example I, Table III. Compared to the other T Statistics, T C and TG, it has the property o f reaching an intermediate magnitude, and after inspection o f the data, T B was selected as the most appropriate summary statistic o f the three (Table IV).
258
Werry and Aman Table III a Example 1: Reliable b u t n o t occurring
2~ < X
4 3 2
0 0 0
0 0 1
0 0 0
0 0 0
T B = .548 T C = .109
1
18
2
0
0
T G = 1.00
1
2 3 EXAMINER A
4
Example 2: Reliable and occurring
< X
4 3 2 1
0 0 3 1 1
0 2 6 2
0 3 1 1
1 0 0 0
2
3
4
T B = .411 TC'=- .353 T G = .582
EXAMINER A Example 3: Unreliable r 2:
