PerceptuaI and Motor Skills, 1992, 75, 25-26.
O Percepmal and Motor Skills 1992
DESIGN A N D ANALYSIS CONSIDERATIONS I N RESEARCH ' BIKKAR S. RANDHAWA Universiry of Saskatchewan Summary.-This short note emphasizes the need For multivariate analysis when multiple correlated dependent variables are used in a study. The use of multivariate analysis and the consequences of not using it are illustrated in relation to a previously published study that used self-concept subscales as dependent variables.
Theoretical rationale for a research study is no doubt very important, for without it a study can hardly be justified. Having taken the trouble to plan a study and having embedded it in an historical context, it is worthwhile to pay utmost attention, especially for an empirical study, to the design and the analysis of the data. Even when a study is a case study or an ethnographic study, the design and analysis of the gathered information or field notes are important considerations. Surely, a flawed design and an improper data analysis can lead one into a situation that may complicate the interpretation of results and how the findings of the study support or contradict previous conclusions. I intend to elaborate upon these aspects in relation to research on self-concepts of gifted students, specifically the study by Yong and McIntyre (1991). I n the rationale of their study, Yong and McIntyre (1991) cited a brief report of our investigation, Hunt and Randhawa (1980), and stated that it, among others, examined self-concept of gifted students. In fact, our study reported personality factor differences of academically talented and retarded groups. Therefore, it is not clear from their general rationale of the study whether the authors regard the personality factors of the 16 PF analogous to the dimensions of the self-concept. Surely, there is a distinction between what is measured by the 16 PF in our study and what they measured using the Piers-Harris Children's Self-concept Scale. This kind of confusion, if there were one, was also evident in other citations as well. For instance, the Karnes and Wherry (1981) study, which incidentally was incorrectly referenced for volume and page numbers, examined the consensus of the gifted students on the items of the Piers-Harris scale. Hence, these studies should not be used to conclude, as they did, that "these studies have yielded apparently conflicting results" (p. 443). Now turning to t h e design and analysis of the Yong and McIntyre study, we note that they tested 40 "gifted" and 40 "regular" 8th graders
'Address enquiries to B. S. Randhawa, Department of Educational Psychology, College of Education, University of Saskatchewan, Saskatoon, Canada S7N OWO.
26
B. S. RANDHAWA
from two public junior high schools. I t is not clear whether the subjects were selected randomly from the two populations, gifted and regular 8th graders, or whether the subjects were conveniently taken from intact classes. Furthermore, the authors state that "subjects were identified as gifted according to the standard guidelines established by the Illinois State Board of Education" (p. 444) and describe the various instruments schools use for this purpose. Again, it is not clear what criteria were used for identifying the gifted subjects from these two schools and who did the screening, the authors or the schools. No descriptive information on any standardized instruments is provided for either the gifted or the regular students. All we are told is that "all subjects were white" (p. 443) and that "they were from communities representative of middle socioeconomic class status" (p. 443). The authors wanted to compare the means of the gifted group with those of the regular group on the six subscales and the total score of the Piers-Harris scale. They carried out six independent t tests for the six scales and found that on only one subscale, Behavior, the two groups were significantly different. Similarly, a t test was made on the total score which also produced a nonsignificant difference. The significant result was interpreted as a reliable difference in spite of the fact that only one of the six tests performed was significant. The probability of this happening by chance alone is 0.094. But, the probability of finding at least one significant difference by chance is 0.109. O n the other hand, if an a probability, say, of 0.05 is specified for testing a null hypothesis and six independent t tests are made, then the probability of Type I error, a,is in fact 0.265. Under these considerations the interpretation of a significant difference on one of the six subscales of the Piers-Harris scale as a reliable difference is not defensible. Had the authors conducted a multivariate T test, they would have found that the null hypothesis was plausible. That would have been the end of the story for reporting the results. Then they could have offered some justification for this situation in relation to what they expected from the literature. In conclusion, it is clear from the above discussion that design and analysis are crucial ingredients for research to provide defensible and interpretable results. Specifically, multivariate analysis should be used when correlated multiple dependent variables are used in a study. The size of the sample and its selection a r t equally important considerations whether a univariate or multivariate analysis is used. REFERENCES HUNT,D., & RANDFLAWA, B. S. (1980) Personality factors and ability groups. Perceptual and Motor Skills, 50, 902. KARNES, F. A , , & WHERRY, J. N . (1981) SelF-concepts of gifted students as measured by Piers-Harris Children's Self-concept Scale. Psychological Reporfr, 49, 903-906. YONG,F. L.,& MCINTYRE, J. D. (1991) Comparison of self-concepts of students identified as gifted and regular students. Perceptual and Motor Skills, 7 3 , 443-446.
Accepted June 5, 1992
O Percepmal and Motor Skills 1992
DESIGN A N D ANALYSIS CONSIDERATIONS I N RESEARCH ' BIKKAR S. RANDHAWA Universiry of Saskatchewan Summary.-This short note emphasizes the need For multivariate analysis when multiple correlated dependent variables are used in a study. The use of multivariate analysis and the consequences of not using it are illustrated in relation to a previously published study that used self-concept subscales as dependent variables.
Theoretical rationale for a research study is no doubt very important, for without it a study can hardly be justified. Having taken the trouble to plan a study and having embedded it in an historical context, it is worthwhile to pay utmost attention, especially for an empirical study, to the design and the analysis of the data. Even when a study is a case study or an ethnographic study, the design and analysis of the gathered information or field notes are important considerations. Surely, a flawed design and an improper data analysis can lead one into a situation that may complicate the interpretation of results and how the findings of the study support or contradict previous conclusions. I intend to elaborate upon these aspects in relation to research on self-concepts of gifted students, specifically the study by Yong and McIntyre (1991). I n the rationale of their study, Yong and McIntyre (1991) cited a brief report of our investigation, Hunt and Randhawa (1980), and stated that it, among others, examined self-concept of gifted students. In fact, our study reported personality factor differences of academically talented and retarded groups. Therefore, it is not clear from their general rationale of the study whether the authors regard the personality factors of the 16 PF analogous to the dimensions of the self-concept. Surely, there is a distinction between what is measured by the 16 PF in our study and what they measured using the Piers-Harris Children's Self-concept Scale. This kind of confusion, if there were one, was also evident in other citations as well. For instance, the Karnes and Wherry (1981) study, which incidentally was incorrectly referenced for volume and page numbers, examined the consensus of the gifted students on the items of the Piers-Harris scale. Hence, these studies should not be used to conclude, as they did, that "these studies have yielded apparently conflicting results" (p. 443). Now turning to t h e design and analysis of the Yong and McIntyre study, we note that they tested 40 "gifted" and 40 "regular" 8th graders
'Address enquiries to B. S. Randhawa, Department of Educational Psychology, College of Education, University of Saskatchewan, Saskatoon, Canada S7N OWO.
26
B. S. RANDHAWA
from two public junior high schools. I t is not clear whether the subjects were selected randomly from the two populations, gifted and regular 8th graders, or whether the subjects were conveniently taken from intact classes. Furthermore, the authors state that "subjects were identified as gifted according to the standard guidelines established by the Illinois State Board of Education" (p. 444) and describe the various instruments schools use for this purpose. Again, it is not clear what criteria were used for identifying the gifted subjects from these two schools and who did the screening, the authors or the schools. No descriptive information on any standardized instruments is provided for either the gifted or the regular students. All we are told is that "all subjects were white" (p. 443) and that "they were from communities representative of middle socioeconomic class status" (p. 443). The authors wanted to compare the means of the gifted group with those of the regular group on the six subscales and the total score of the Piers-Harris scale. They carried out six independent t tests for the six scales and found that on only one subscale, Behavior, the two groups were significantly different. Similarly, a t test was made on the total score which also produced a nonsignificant difference. The significant result was interpreted as a reliable difference in spite of the fact that only one of the six tests performed was significant. The probability of this happening by chance alone is 0.094. But, the probability of finding at least one significant difference by chance is 0.109. O n the other hand, if an a probability, say, of 0.05 is specified for testing a null hypothesis and six independent t tests are made, then the probability of Type I error, a,is in fact 0.265. Under these considerations the interpretation of a significant difference on one of the six subscales of the Piers-Harris scale as a reliable difference is not defensible. Had the authors conducted a multivariate T test, they would have found that the null hypothesis was plausible. That would have been the end of the story for reporting the results. Then they could have offered some justification for this situation in relation to what they expected from the literature. In conclusion, it is clear from the above discussion that design and analysis are crucial ingredients for research to provide defensible and interpretable results. Specifically, multivariate analysis should be used when correlated multiple dependent variables are used in a study. The size of the sample and its selection a r t equally important considerations whether a univariate or multivariate analysis is used. REFERENCES HUNT,D., & RANDFLAWA, B. S. (1980) Personality factors and ability groups. Perceptual and Motor Skills, 50, 902. KARNES, F. A , , & WHERRY, J. N . (1981) SelF-concepts of gifted students as measured by Piers-Harris Children's Self-concept Scale. Psychological Reporfr, 49, 903-906. YONG,F. L.,& MCINTYRE, J. D. (1991) Comparison of self-concepts of students identified as gifted and regular students. Perceptual and Motor Skills, 7 3 , 443-446.
Accepted June 5, 1992
