Percephral and Motor Skills, 1992, 75, 27-34.
O Perceptual and Motor Skills 1992
PERFORMANCE O F STUDENTS W I T H LEARNING DISABILITIES O N VALIDITY INDEXES O F T H E PERCEPTION O F ABILITY SCALE FOR STUDENTS ',' JAMES W. CHAPMAN Marsey Universi~
AND
FREDERIC J. BOERSMA Universi~of Alberh
Summary.-The present study examined the performance of 78 students with learning disabilities and 71 normally achieving students in regular Form 1 (Grade 6) classes on three validity indexes of the Perception of Ability Scale for Students, a measure of academic self-concept. The three indexes assess consistency of responding, negative or positive response biases, and misrepresentation of self-perceptions in terms of unrealistic perceptions of perfection in school. Analysis showed that learning disabled students obtained significantly lower Full Scale scores than the normal students, but no significant differences appeared on the three validity indexes. Users of the test can be confident that learning disabled students respond to items in as valid a manner as other students. Having specific learning problems in school should not interfere with response patterns on this scale.
Interpretations of scores on measures of self-concept are based on the assumption that responses on individual items are valid indicators of the way respondents feel about themselves. With this important consideration in mind, developers of measures of self-concept often include additional procedures or scales for estimating the validity of individuals' responses. The frequently used Piers-Harris Children's Self-concept Scale (Piers, 1984), for example, has three indexes that provide information on faked responses, acquiescence or negative response patterns, and random responding. Similarly, Marsh (1988) gives information on six control scores for the Self-description Questionnaire-1. These scores are designed to assess noncontingent responding (e.g., random responding) and positivity and negativity biases. Although both of these instruments yield information on measures of internal validity, little or none is available on how different populations perform on these indexes. More recently, the developers of the Perception of Ability Scale for Students (Boersma & Chapman, 1992a, 1992b) have followed the suggestions of Marsh (1988), Piers (1984), and Wylie (1974) by providing procedures to estimate the internal validity of individuals' responses. These procedures are validity indexes; the Item Inconsistency Index, the Response Bias Index, and the Misrepresentation Index. Although the manual (Boersma & Chapman, 1992a) contains normative data for the indexes, no research is 'The Perception of Ability Scale for Students was formerly known as the Student's Perception
yf Ability Scale.
Correspondence concerning this article should be addressed to James W. Chapman, Department of Education, Massey University, Paherston North, New Zealand.
28
J. W. CHAPMAN & F. J. BOERSMA
reported on their characteristics for special populations. The present study was designed for this purpose. Such a study is timely because the scale has been used extensively to assess academic self-concept of children with learning disabilities (Chapman, 1988a). Although studies show that learning disabled children tend to obtain full scale scores about one standard deviation below the means of average achievers, the internal validity of their responses is not known. Such students may not respond as consistently as normally achieving students, or they may adopt a particular response set that biases towards lower scores.
PERCEPTION OF ABILITYSCALEFOR STUDENTS This scale is a self-report instrument designed to assess children's feelings about their academic abihties and school-related achievement. Academic self-concept here is defined as a relatively stable set of attitudes and feelings reflecting a self-perception or self-evaluation of performance on tasks like reading, writing, spelling, and mathematics. Following Shavelson, Hubner, and Stanton's (1976) multifaceted model of self-concept, academic self-concept is seen as related to yet distinct from general self-concept (e.g., Boersma & Chapman, 1992a; Marsh, 1988). The scale contains 70 forced-choice "Yes-No" items relating to feelings and attitudes about performance in five academic areas (reading, writing, spelling, language arts, mathematics, and printinglwriting), and also to school in general. Six subscales, derived through factor analysis, include Perception of General Ability, Perception of Mathematics Abllity, Perception of Reading and Spelling Ability, Perception of Penmanship and Neatness, General School Satisfaction (each of which contains 12 items), and Confidence in Academic Ability (10 items). Detailed psychometric characteristics are reported fully in the manual (Boersma & Chapman, 1992a) and elsewhere (e.g., Boersma, Chapman, & Maguire, 1979; Chapman, 1989; Chapman & Boersma, 1991). The main psychometric properties based on data from a standardization sample of 831 US children in Grades 3 to 6 include a F d Scale internal reliability coefficient of .93, a mean of 46.49 (SD = 12.59), and an SEM of 3.35. Studies have shown the scale is meaningfully related to other measures of academic self-concept (Boersma & Chapman, 1992a), as well as to academic locus of control for success and to students', teachers', and mothers' achievement expectations (Chapman & Boersma, 1980). The scale makes significant contributions to prediction of school achievement (e.g., Chapman, Cullen, Boersma, & Maguire, 1981; Chapman, 1988a), usefully discriminates between students with learning disabilities and normal achievers (Chapman, 1988b), differentiates between gifted students, students with learning disabilities, and normal achievers (Chapman & McAlpine, 1988), and is sensitive to change following remedial intervention (e.g., Boersma, Chapman, &
LEARNING DISABLED CHILDREN: PERCEIVED ABILITY
29
Battle, 1979). The scale has been used with children from Grade 2 (Boersma & Chapman, 1992a) to Grade 10 (Hiebert, Wong, & Hunter, 1982), in countries that include the United States, Canada, Australia, New Zealand, England, Czechoslovakia, and Brazil, and with various ethnic groups, such as American Mexicans, Canadian Indians, and New Zealand Maoris. The scale appears to be stable across national and ethnic groups, has good internal reliability, and relates appropriately to other affective measures and to school achievement (Boersma & Chapman, 1992a). Description of the Validity Indexes The Inconsistency Index gives a check on random response patterns. The index was developed on the observation that certain pairs of responses are inconsistent, unlikely, or illogical. Academic self-concept scores of children who have a large number of inconsistent responses are likely to be of doubtful validity. Developed in a manner similar to that described for the Piers-Harris Children's Self-concept Scale (Piers, 1984) and described fully in the manual, the index essentially involved identifying pairs of items which appeared to represent a basic consistency or inconsistency between items. For example, Item 26: "I have problems printing neatly" and Item 32: "I am good at printing," can yield an inconsistent response if a child answers "yes" or "no" to both items. Only those pairs of items which correlated .40 or greater and occurred in fewer than 10% of the normative sample were included. Fifteen pairs of items comprise the index; the mean is 1.03 and the standard deviation is 1.13. Most children in the normative sample obtained low scores: 41.2% scored 0, and 30.3% scored 1. The cut-off score was set at + 2 standard deviation units, which rounded to the nearest whole number is 4. For the normative sample, the cumulative frequency at this point was 99.3%, meaning that only 0.7% of the children in the sample would have been identified as responding randomly or inconsistently. The Response Bias Index is a measure of positive or negative response bias, i.e., the number of items to which an individual responded yes. The score can range from 0 (based on all no responses) to 70 (based on all yes responses). This index was standardzed on the US normative sample, for whom the mean was 29.99 (SD = 5.36). Cut-off points were set at k 2 standard deviation units, w h c h translated to scores of 19 and 41, respectively. These two points were selected because they are conservative and because moderate deviations in response set in either direction should not pose a major threat to the validity of individuals' responses. Of the 831 children in the normative sample, only 12 showed a significant number of "no" responses, and 17 as having answered "yes" to a large number of items.
*
J. W.CHAPMAN & F. J. BOERSMA
The Misrepresentation Index was developed to identify respondents who appeared to be faking responses in a positive manner. Some test manuals refer to such an index as a Lie Scale (e.g., Reynolds & Richmond, 1985), but here connotations are that students may consciously attempt to misrepresent their true feelings and self-perceptions about school. Without further information from students differentiation between students who deliberately attempt to deceive and those who subconsciously misrepresent their self-perceptions would be difficult. The index comprises items that refer to perfect work in school, or to situations in which students feel that they always do well or get work right. Items such as "My printing is perfect" and "My spelling is always rightMd o not typify real achievement behavior, so a high score (i.e., 2 5 on a scale of O to 6) may be indicative of inaccurate self-report or possibly denial. The mean for the normative sample was 1.70 (SD = 1.63), but only 4.7% obtained a score of 5 or more.
METI-IOD Sample The sample was from a larger study on cognitive-motivational characteristics of New Zealand Form 1 (Grade 6) children with learning disabilities from five intermediate (junior h g h ) schools. Because New Zealand schools do not recognise learning disabilities as a distinct category of special and remedial education, it was necessary to adopt our own identification procedure. To allow comparison with other research in the learning disabilities area, an ability-achievement discrepancy method for identifying learning disabled children was used. Although the method is not without problems (e.g., Algozzine & Ysseldyke, 1986; Stanovich, 1991), this approach has been noted as virtually the only common sampling characteristic in research on learning disabilities (Kavale, Forness, & Lorsbach, 1991). The procedure for identifying the children with learning disabdities is more fully described by Chapman (1988a). To summarise, these students were selected from a cohort of 1,220 Form 1 (Grade 6) children on the basis of their having average or above IQ and major problems in one or more subject areas. Estimates of ability were obtained from scores on the WISC-R. Intellectual assessments are not routinely administered to children in New Zealand, and because it was not feasible to administer the full WISC-R to a large number of children, a short form comprising four subscales was used. Although not an ideal procedure, it was considered an acceptable procedure for research purposes. The subtests included were Information, Vocabulary, Picture Completion, and Block Design. They were chosen because they correlate highly (.94) with the Full Scale IQ (Sattler, 1974) and because administration is relatively easy. Full Scale IQs were prorated from summed
LEARNING DISABLED CHILDREN: PERCErVED ABILITY
'
31
scale scores, in accordance with the procedure recommended by Tellegen and Briggs (1967). Achievement was assessed with four tests in the Progressive Achievement Test series. These were Reading Comprehension (Elley & Reid, 1969), Reading Vocabulary (Elley & Reid, 19691, Listening Comprehension (Elley & Reid, 1971), and Mathematics (Reid, 1974). These tests are the most frequently used standardized measures of achievement in the New Zealand school system. Selection of learning disabled children was made on the basis of their having a prorated WISC-R IQ of 90 or above and at least one standardized achievement score below the 16th percentile. Also, following the recommendations of Cone and Wilson (19811, a l l such children met a regression method criterion for establishing an ability-achievement discrepancy: they had at least one standardized achievement score that was below the lower bound of the confidence interval (at the 10% level) for the standard error of estimate predicted from their IQs. The children were d in regular classrooms. No child with a major disability, with major social-emotional problems, or having difficulties with English as a second language was included. The selection of normal achievers was made from those students who had a prorated WISC-R IQ of 90 or above and who had obtained scores on standardized achievement tests that were at or above the 30th percentile. Attempts were made to match the groups on four variables in the following order of priority: IQ, gender, and socioeconomic status. Sample Characteristics The sample comprised 78 students with learning disabilities (48 boys, 30 girls) and 71 normal students (42 boys, 29 girls). The mean ages at the start of the academic year for these students were, respectively, 11.34 (SD = 0.40) and 11.30 (SD = 0.38). Mean WISC-R prorated IQs were 99.77 (SD = 7.07) for the learning disabled group and 100.70 (SD = 6.16) for the normal group. Socioeconomic status rankings based on fathers' occupations using the Elley-Irving Scale (Elley & Irving, 1976) indicated that the learning disabled students came from slightly lower socioeconomic backgrounds than the normal students. On a scale of 1 (professional, managerial) to 6 (unskilled), both median and modal rankings were 4 for the learning disabled group, and 3 for the normal group. Unemployed fathers were more frequent among the normal students (3) than those with learning disabilities (0). Families with fathers absent, however, were represented in both groups with about the same frequency (6 for students with learning disabilities and 7 for normal students).
Procedure The Perception of Abllity Scale for Students was administered in regu-
32
J. W. CHAPMAN
&
F. J. BOERSMA
lar class groups towards the end of the school year, as part of a larger study of cognitive-motivational and achievement characteristics of students with learning disabilities (Chapman, 1788a). Children were informed that the purpose of the questionnaire was to find out about the "thoughts and feelings" they had about school; they were assured confidentiality. Instructions and items were read aloud. No child's assessment was related to achievement.
RESULTS AND DISCUSSION Because differences between boys and girls have been consistently found on our scale (Boersma & Chapman, 1992a), it was decided to analyze scores for the Full Scale and for the three indexes by means of a 2 x 2 analysis of variance, with group and gender as the factors. A significant main effect for group was found (F,,,,,= 14.07, p
O Perceptual and Motor Skills 1992
PERFORMANCE O F STUDENTS W I T H LEARNING DISABILITIES O N VALIDITY INDEXES O F T H E PERCEPTION O F ABILITY SCALE FOR STUDENTS ',' JAMES W. CHAPMAN Marsey Universi~
AND
FREDERIC J. BOERSMA Universi~of Alberh
Summary.-The present study examined the performance of 78 students with learning disabilities and 71 normally achieving students in regular Form 1 (Grade 6) classes on three validity indexes of the Perception of Ability Scale for Students, a measure of academic self-concept. The three indexes assess consistency of responding, negative or positive response biases, and misrepresentation of self-perceptions in terms of unrealistic perceptions of perfection in school. Analysis showed that learning disabled students obtained significantly lower Full Scale scores than the normal students, but no significant differences appeared on the three validity indexes. Users of the test can be confident that learning disabled students respond to items in as valid a manner as other students. Having specific learning problems in school should not interfere with response patterns on this scale.
Interpretations of scores on measures of self-concept are based on the assumption that responses on individual items are valid indicators of the way respondents feel about themselves. With this important consideration in mind, developers of measures of self-concept often include additional procedures or scales for estimating the validity of individuals' responses. The frequently used Piers-Harris Children's Self-concept Scale (Piers, 1984), for example, has three indexes that provide information on faked responses, acquiescence or negative response patterns, and random responding. Similarly, Marsh (1988) gives information on six control scores for the Self-description Questionnaire-1. These scores are designed to assess noncontingent responding (e.g., random responding) and positivity and negativity biases. Although both of these instruments yield information on measures of internal validity, little or none is available on how different populations perform on these indexes. More recently, the developers of the Perception of Ability Scale for Students (Boersma & Chapman, 1992a, 1992b) have followed the suggestions of Marsh (1988), Piers (1984), and Wylie (1974) by providing procedures to estimate the internal validity of individuals' responses. These procedures are validity indexes; the Item Inconsistency Index, the Response Bias Index, and the Misrepresentation Index. Although the manual (Boersma & Chapman, 1992a) contains normative data for the indexes, no research is 'The Perception of Ability Scale for Students was formerly known as the Student's Perception
yf Ability Scale.
Correspondence concerning this article should be addressed to James W. Chapman, Department of Education, Massey University, Paherston North, New Zealand.
28
J. W. CHAPMAN & F. J. BOERSMA
reported on their characteristics for special populations. The present study was designed for this purpose. Such a study is timely because the scale has been used extensively to assess academic self-concept of children with learning disabilities (Chapman, 1988a). Although studies show that learning disabled children tend to obtain full scale scores about one standard deviation below the means of average achievers, the internal validity of their responses is not known. Such students may not respond as consistently as normally achieving students, or they may adopt a particular response set that biases towards lower scores.
PERCEPTION OF ABILITYSCALEFOR STUDENTS This scale is a self-report instrument designed to assess children's feelings about their academic abihties and school-related achievement. Academic self-concept here is defined as a relatively stable set of attitudes and feelings reflecting a self-perception or self-evaluation of performance on tasks like reading, writing, spelling, and mathematics. Following Shavelson, Hubner, and Stanton's (1976) multifaceted model of self-concept, academic self-concept is seen as related to yet distinct from general self-concept (e.g., Boersma & Chapman, 1992a; Marsh, 1988). The scale contains 70 forced-choice "Yes-No" items relating to feelings and attitudes about performance in five academic areas (reading, writing, spelling, language arts, mathematics, and printinglwriting), and also to school in general. Six subscales, derived through factor analysis, include Perception of General Ability, Perception of Mathematics Abllity, Perception of Reading and Spelling Ability, Perception of Penmanship and Neatness, General School Satisfaction (each of which contains 12 items), and Confidence in Academic Ability (10 items). Detailed psychometric characteristics are reported fully in the manual (Boersma & Chapman, 1992a) and elsewhere (e.g., Boersma, Chapman, & Maguire, 1979; Chapman, 1989; Chapman & Boersma, 1991). The main psychometric properties based on data from a standardization sample of 831 US children in Grades 3 to 6 include a F d Scale internal reliability coefficient of .93, a mean of 46.49 (SD = 12.59), and an SEM of 3.35. Studies have shown the scale is meaningfully related to other measures of academic self-concept (Boersma & Chapman, 1992a), as well as to academic locus of control for success and to students', teachers', and mothers' achievement expectations (Chapman & Boersma, 1980). The scale makes significant contributions to prediction of school achievement (e.g., Chapman, Cullen, Boersma, & Maguire, 1981; Chapman, 1988a), usefully discriminates between students with learning disabilities and normal achievers (Chapman, 1988b), differentiates between gifted students, students with learning disabilities, and normal achievers (Chapman & McAlpine, 1988), and is sensitive to change following remedial intervention (e.g., Boersma, Chapman, &
LEARNING DISABLED CHILDREN: PERCEIVED ABILITY
29
Battle, 1979). The scale has been used with children from Grade 2 (Boersma & Chapman, 1992a) to Grade 10 (Hiebert, Wong, & Hunter, 1982), in countries that include the United States, Canada, Australia, New Zealand, England, Czechoslovakia, and Brazil, and with various ethnic groups, such as American Mexicans, Canadian Indians, and New Zealand Maoris. The scale appears to be stable across national and ethnic groups, has good internal reliability, and relates appropriately to other affective measures and to school achievement (Boersma & Chapman, 1992a). Description of the Validity Indexes The Inconsistency Index gives a check on random response patterns. The index was developed on the observation that certain pairs of responses are inconsistent, unlikely, or illogical. Academic self-concept scores of children who have a large number of inconsistent responses are likely to be of doubtful validity. Developed in a manner similar to that described for the Piers-Harris Children's Self-concept Scale (Piers, 1984) and described fully in the manual, the index essentially involved identifying pairs of items which appeared to represent a basic consistency or inconsistency between items. For example, Item 26: "I have problems printing neatly" and Item 32: "I am good at printing," can yield an inconsistent response if a child answers "yes" or "no" to both items. Only those pairs of items which correlated .40 or greater and occurred in fewer than 10% of the normative sample were included. Fifteen pairs of items comprise the index; the mean is 1.03 and the standard deviation is 1.13. Most children in the normative sample obtained low scores: 41.2% scored 0, and 30.3% scored 1. The cut-off score was set at + 2 standard deviation units, which rounded to the nearest whole number is 4. For the normative sample, the cumulative frequency at this point was 99.3%, meaning that only 0.7% of the children in the sample would have been identified as responding randomly or inconsistently. The Response Bias Index is a measure of positive or negative response bias, i.e., the number of items to which an individual responded yes. The score can range from 0 (based on all no responses) to 70 (based on all yes responses). This index was standardzed on the US normative sample, for whom the mean was 29.99 (SD = 5.36). Cut-off points were set at k 2 standard deviation units, w h c h translated to scores of 19 and 41, respectively. These two points were selected because they are conservative and because moderate deviations in response set in either direction should not pose a major threat to the validity of individuals' responses. Of the 831 children in the normative sample, only 12 showed a significant number of "no" responses, and 17 as having answered "yes" to a large number of items.
*
J. W.CHAPMAN & F. J. BOERSMA
The Misrepresentation Index was developed to identify respondents who appeared to be faking responses in a positive manner. Some test manuals refer to such an index as a Lie Scale (e.g., Reynolds & Richmond, 1985), but here connotations are that students may consciously attempt to misrepresent their true feelings and self-perceptions about school. Without further information from students differentiation between students who deliberately attempt to deceive and those who subconsciously misrepresent their self-perceptions would be difficult. The index comprises items that refer to perfect work in school, or to situations in which students feel that they always do well or get work right. Items such as "My printing is perfect" and "My spelling is always rightMd o not typify real achievement behavior, so a high score (i.e., 2 5 on a scale of O to 6) may be indicative of inaccurate self-report or possibly denial. The mean for the normative sample was 1.70 (SD = 1.63), but only 4.7% obtained a score of 5 or more.
METI-IOD Sample The sample was from a larger study on cognitive-motivational characteristics of New Zealand Form 1 (Grade 6) children with learning disabilities from five intermediate (junior h g h ) schools. Because New Zealand schools do not recognise learning disabilities as a distinct category of special and remedial education, it was necessary to adopt our own identification procedure. To allow comparison with other research in the learning disabilities area, an ability-achievement discrepancy method for identifying learning disabled children was used. Although the method is not without problems (e.g., Algozzine & Ysseldyke, 1986; Stanovich, 1991), this approach has been noted as virtually the only common sampling characteristic in research on learning disabilities (Kavale, Forness, & Lorsbach, 1991). The procedure for identifying the children with learning disabdities is more fully described by Chapman (1988a). To summarise, these students were selected from a cohort of 1,220 Form 1 (Grade 6) children on the basis of their having average or above IQ and major problems in one or more subject areas. Estimates of ability were obtained from scores on the WISC-R. Intellectual assessments are not routinely administered to children in New Zealand, and because it was not feasible to administer the full WISC-R to a large number of children, a short form comprising four subscales was used. Although not an ideal procedure, it was considered an acceptable procedure for research purposes. The subtests included were Information, Vocabulary, Picture Completion, and Block Design. They were chosen because they correlate highly (.94) with the Full Scale IQ (Sattler, 1974) and because administration is relatively easy. Full Scale IQs were prorated from summed
LEARNING DISABLED CHILDREN: PERCErVED ABILITY
'
31
scale scores, in accordance with the procedure recommended by Tellegen and Briggs (1967). Achievement was assessed with four tests in the Progressive Achievement Test series. These were Reading Comprehension (Elley & Reid, 1969), Reading Vocabulary (Elley & Reid, 19691, Listening Comprehension (Elley & Reid, 1971), and Mathematics (Reid, 1974). These tests are the most frequently used standardized measures of achievement in the New Zealand school system. Selection of learning disabled children was made on the basis of their having a prorated WISC-R IQ of 90 or above and at least one standardized achievement score below the 16th percentile. Also, following the recommendations of Cone and Wilson (19811, a l l such children met a regression method criterion for establishing an ability-achievement discrepancy: they had at least one standardized achievement score that was below the lower bound of the confidence interval (at the 10% level) for the standard error of estimate predicted from their IQs. The children were d in regular classrooms. No child with a major disability, with major social-emotional problems, or having difficulties with English as a second language was included. The selection of normal achievers was made from those students who had a prorated WISC-R IQ of 90 or above and who had obtained scores on standardized achievement tests that were at or above the 30th percentile. Attempts were made to match the groups on four variables in the following order of priority: IQ, gender, and socioeconomic status. Sample Characteristics The sample comprised 78 students with learning disabilities (48 boys, 30 girls) and 71 normal students (42 boys, 29 girls). The mean ages at the start of the academic year for these students were, respectively, 11.34 (SD = 0.40) and 11.30 (SD = 0.38). Mean WISC-R prorated IQs were 99.77 (SD = 7.07) for the learning disabled group and 100.70 (SD = 6.16) for the normal group. Socioeconomic status rankings based on fathers' occupations using the Elley-Irving Scale (Elley & Irving, 1976) indicated that the learning disabled students came from slightly lower socioeconomic backgrounds than the normal students. On a scale of 1 (professional, managerial) to 6 (unskilled), both median and modal rankings were 4 for the learning disabled group, and 3 for the normal group. Unemployed fathers were more frequent among the normal students (3) than those with learning disabilities (0). Families with fathers absent, however, were represented in both groups with about the same frequency (6 for students with learning disabilities and 7 for normal students).
Procedure The Perception of Abllity Scale for Students was administered in regu-
32
J. W. CHAPMAN
&
F. J. BOERSMA
lar class groups towards the end of the school year, as part of a larger study of cognitive-motivational and achievement characteristics of students with learning disabilities (Chapman, 1788a). Children were informed that the purpose of the questionnaire was to find out about the "thoughts and feelings" they had about school; they were assured confidentiality. Instructions and items were read aloud. No child's assessment was related to achievement.
RESULTS AND DISCUSSION Because differences between boys and girls have been consistently found on our scale (Boersma & Chapman, 1992a), it was decided to analyze scores for the Full Scale and for the three indexes by means of a 2 x 2 analysis of variance, with group and gender as the factors. A significant main effect for group was found (F,,,,,= 14.07, p
