This article was downloaded by: [McMaster University] On: 23 December 2014, At: 06:53 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Multivariate Behavioral Research Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hmbr20
Not Only But Also R.P. McDonald Published online: 10 Jun 2010.
To cite this article: R.P. McDonald (1995) Not Only But Also, Multivariate Behavioral Research, 30:1, 113-115, DOI: 10.1207/s15327906mbr3001_12 To link to this article: http://dx.doi.org/10.1207/s15327906mbr3001_12
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan,
Downloaded by [McMaster University] at 06:53 23 December 2014
sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Multivariaie Behavioral Research, 30 (I), 1 13-1 15 Copyright O 1995, Lawrence Erlbaum Associates, Inc.
Not Only But Also R.P. McDonald
Downloaded by [McMaster University] at 06:53 23 December 2014
University of Illinois
There is, indeed, an error in the final paragraph of my previous comment on Labouvie and Ruetsch's (1995) article. While a necessary condition for the factors in two or more groups to be the same is that the loadings, suitably scaled, are the same, this condition is not, strictly speaking, sufficient. The sufficient condition, by McDonald & Mulaik (1979) is that the loadings should be the same in the infinite set of items (the test of infinite length) of which the actual test contains only a subset. However, most post-Popperians, including Weimer and myself, would allow that it is rational to behave as though the condition I stated was sufficient, since we cannot in applications exhaust the behavior domain, so otherwise could not proceed at all. In any case, the pedantic correction I have just allowed does not strengthen the case Labouvie and Ruetsch (1995) have made, but quite the contrary. If the loadings are not the same (in the finite item-sample), the factors are certainly not the same (in the domain), and it is the necessary condition that ~ a b o u v i eand Ruetsch choose not to satisfy. (No one can satisfy the sufficient condition.) Precisely what has not been shown by Labouvie and Ruetsch is that if neither condition for the factors to be the same is satisfied, "quantitative comparisons of the resulting latent variables [are] meaningful." If the factors are open to distinct interpretations, the comparison cannot possibly be "meaningful", the test is biased, and in important applications subject to action in law. As I pointed out, Labouvie and Ruetsch are willing to compare means in a data-set in which loadings of the variables on a single factor are as different as possible. They do not have any condition to be satisfied in such a case except unidimensionality. Given the interpretive axiom of factor analysis and item response theory, that a common factor is "that quantitative characteristic of the examinees that is measured by tests [/items] that have high loadings on it" (McDonald, 1981, p.105), if a set of items has distinct profiles of loadings in distinct groups, this fact should excite our scientific curiosity, not be swept under the carpet. One possibility is that we have fit the wrong model, and a different factor Requests for reprints should be sent to R.P. McDonald, Department of Psychology, University of Illinois, 603 E. Daniel St. Champaign, lL 61820.
MULTIVARIATE BEHAVIORAL RESEARCH
113
Downloaded by [McMaster University] at 06:53 23 December 2014
R. McDonald
model exists, with interpretable factors, having invariant loadings. If this is not the case, we would hope to find, in items reflecting perceptions, a convincing account of the loadings in terms of differential perception, or, in cognitive items, differences in opportunity to learn. Theory for this class of problems needs work. Perhaps we need not only a plain mathematical statement of this psychometric truism but also an empirical illustration of what may happen. Consider Labouvie and Ruetsch's (1995) example. From their Table 3, the first six items appear to define "the" factor mainly by gentle, helpful, kind, for adolescent boys, and mainly by the conceptually distinct items awareness of others 'feelings, understanding of others, warmth in relating to others, for adolescent girls. Given the differential socialization practices employed on boys and girls, this makes sense. Similarly, the last four items show notably low loadings for the girls in respect of Z am good and Z am important. Admittedly, given sexist acculturation practices, it is easier to explain why perceiving oneself as important is not a major facet of girls' self esteem than why perceiving oneself as good might not be. Anyway, we combine substantive considerations and the apparent disagreement in the loading profiles to consider the hypothesis that V1-V3 define a conventionul prosocial seK while V4-V6 define a distinct warmly relating self; and these factors will be more distinct (have a lower correlation) for the girls than the boys. Similarly, though less confidently, perhaps, we consider the hypothesis that V7-V8 define noncognitzve selfesteem, while V9-V10 define cognitive selfesteerrz, and again these are more distinct for the girls than the boys. Table 1 gives fit statistics from four re-analyses of Labouvie and Ruetsch's (1995) covariance matrices (by COSAN, Fraser & McDonald, 1988), together with their M1 results for comparison. In each of these models the nonsalient loadings are fixed at zero, and the salient loadings are constrained, as by Labouvie and Ruetsch, to average unity in both groups. Models M2E and M2N fit Labouvie and Ruetsch's two factors, with loadings equated in M2E and not equated in M2N. Similarly, models M4E and M4N fit the conjectured four-factor model with equated and not equated loadings respectively. Chronologically, I first tried M4E and considered the clear improvement in fit over Labouvie and Ruetsch's M1 a satisfying confirmation of the conjecture that any failure of factorial invariance came from fitting an over restrictive and conceptually oversimplified model. Further, the correlations of the two Social Self f a c t o r e . 9 4 8 for the boys and ,756 for the g i r l s a n d of the two self esteem factors,--.809 for the boys and .580 for the girls, further suggest that Labouvie and Ruetsch's factors split for the girls more clearly than for the boys. However, the further analyses show that Labouvie and Ruetsch's data set is quite 114
MULTIVARIATE BEHAVIORAL RESEARCH
R. McDonald
Table 1
Downloaded by [McMaster University] at 06:53 23 December 2014
Model
df
chisqu
d
RMSEA
URFI
remarkable for its lack of interest, since, in their two-factor model, equating factor loadings does not give a significantly worse fit than not equating them, and indeed gives a slightly better fit as measured by the goodness-offit indices tabulated. These include the unbiassed estimate o f the noncentrality parameter, d (McDonald, 1988), the Root Mean Squared Error of Approximation, RMSEA (Browne & Cudeck, 1992), and the Unbiassed Relative Fit Index, URFI (McDonald & Marsh, 1990). The same is true in the four-factor model. Thus, there was no problem of failure of factorial invariance in the first place, and no motive for our curiosity. However, the methodological point remains, that even an apparent failure of factorial invariance is a basis for further investigation, not to be ignored. Some thirty years of consulting on applications of factor analysis have led me to conclude that precisely because our concepts are always less precise than we would wish, and hence less precisely instantiated in items than we would hope, we must be as clear about their semantics as possible. Fuzzy sets may be isomorphic with fuzzy thinking, but this fact does not provide any rational justification for either. References Browne, M. W. & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods and Research, 21,230-258. Fraser, C. & McDonald, R. P. (1988). COSAN: Covariance structure analysis. Multivariate Behavioral Research, 23,263-265. Labouvie, E. & Ruetsch, C. (1995). Testing for equivalence of measurement scales: Simple structure and metric equivalence reconsidered. Multivariate Behavioral Research, 30, 63-76. McDonald, R. P. (1981). The dimensionality of tests and items. British Journal of Mathematical and SfafisficalPsychology, 34, 1 10-1 17. McDonald, R. P. (1988). An index of goodness of fit based on noncentrality. Journal of Class~ficafion,6,97- 103. McDonald, R. P. & Marsh, H. W. (1990). Choosing a multivariate model: Noncentrality and goodness of fit. Psychological Bulletin, 107, 247-255. McDonald, R. P. & Mulaik, S. A. (1979), The determinacy of common factors: A nontechnical review. Psychological Bullefm, 86, 297-306. MULTIVARIATE BEHAVIORAL RESEARCH
115
Multivariate Behavioral Research Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hmbr20
Not Only But Also R.P. McDonald Published online: 10 Jun 2010.
To cite this article: R.P. McDonald (1995) Not Only But Also, Multivariate Behavioral Research, 30:1, 113-115, DOI: 10.1207/s15327906mbr3001_12 To link to this article: http://dx.doi.org/10.1207/s15327906mbr3001_12
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan,
Downloaded by [McMaster University] at 06:53 23 December 2014
sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Multivariaie Behavioral Research, 30 (I), 1 13-1 15 Copyright O 1995, Lawrence Erlbaum Associates, Inc.
Not Only But Also R.P. McDonald
Downloaded by [McMaster University] at 06:53 23 December 2014
University of Illinois
There is, indeed, an error in the final paragraph of my previous comment on Labouvie and Ruetsch's (1995) article. While a necessary condition for the factors in two or more groups to be the same is that the loadings, suitably scaled, are the same, this condition is not, strictly speaking, sufficient. The sufficient condition, by McDonald & Mulaik (1979) is that the loadings should be the same in the infinite set of items (the test of infinite length) of which the actual test contains only a subset. However, most post-Popperians, including Weimer and myself, would allow that it is rational to behave as though the condition I stated was sufficient, since we cannot in applications exhaust the behavior domain, so otherwise could not proceed at all. In any case, the pedantic correction I have just allowed does not strengthen the case Labouvie and Ruetsch (1995) have made, but quite the contrary. If the loadings are not the same (in the finite item-sample), the factors are certainly not the same (in the domain), and it is the necessary condition that ~ a b o u v i eand Ruetsch choose not to satisfy. (No one can satisfy the sufficient condition.) Precisely what has not been shown by Labouvie and Ruetsch is that if neither condition for the factors to be the same is satisfied, "quantitative comparisons of the resulting latent variables [are] meaningful." If the factors are open to distinct interpretations, the comparison cannot possibly be "meaningful", the test is biased, and in important applications subject to action in law. As I pointed out, Labouvie and Ruetsch are willing to compare means in a data-set in which loadings of the variables on a single factor are as different as possible. They do not have any condition to be satisfied in such a case except unidimensionality. Given the interpretive axiom of factor analysis and item response theory, that a common factor is "that quantitative characteristic of the examinees that is measured by tests [/items] that have high loadings on it" (McDonald, 1981, p.105), if a set of items has distinct profiles of loadings in distinct groups, this fact should excite our scientific curiosity, not be swept under the carpet. One possibility is that we have fit the wrong model, and a different factor Requests for reprints should be sent to R.P. McDonald, Department of Psychology, University of Illinois, 603 E. Daniel St. Champaign, lL 61820.
MULTIVARIATE BEHAVIORAL RESEARCH
113
Downloaded by [McMaster University] at 06:53 23 December 2014
R. McDonald
model exists, with interpretable factors, having invariant loadings. If this is not the case, we would hope to find, in items reflecting perceptions, a convincing account of the loadings in terms of differential perception, or, in cognitive items, differences in opportunity to learn. Theory for this class of problems needs work. Perhaps we need not only a plain mathematical statement of this psychometric truism but also an empirical illustration of what may happen. Consider Labouvie and Ruetsch's (1995) example. From their Table 3, the first six items appear to define "the" factor mainly by gentle, helpful, kind, for adolescent boys, and mainly by the conceptually distinct items awareness of others 'feelings, understanding of others, warmth in relating to others, for adolescent girls. Given the differential socialization practices employed on boys and girls, this makes sense. Similarly, the last four items show notably low loadings for the girls in respect of Z am good and Z am important. Admittedly, given sexist acculturation practices, it is easier to explain why perceiving oneself as important is not a major facet of girls' self esteem than why perceiving oneself as good might not be. Anyway, we combine substantive considerations and the apparent disagreement in the loading profiles to consider the hypothesis that V1-V3 define a conventionul prosocial seK while V4-V6 define a distinct warmly relating self; and these factors will be more distinct (have a lower correlation) for the girls than the boys. Similarly, though less confidently, perhaps, we consider the hypothesis that V7-V8 define noncognitzve selfesteem, while V9-V10 define cognitive selfesteerrz, and again these are more distinct for the girls than the boys. Table 1 gives fit statistics from four re-analyses of Labouvie and Ruetsch's (1995) covariance matrices (by COSAN, Fraser & McDonald, 1988), together with their M1 results for comparison. In each of these models the nonsalient loadings are fixed at zero, and the salient loadings are constrained, as by Labouvie and Ruetsch, to average unity in both groups. Models M2E and M2N fit Labouvie and Ruetsch's two factors, with loadings equated in M2E and not equated in M2N. Similarly, models M4E and M4N fit the conjectured four-factor model with equated and not equated loadings respectively. Chronologically, I first tried M4E and considered the clear improvement in fit over Labouvie and Ruetsch's M1 a satisfying confirmation of the conjecture that any failure of factorial invariance came from fitting an over restrictive and conceptually oversimplified model. Further, the correlations of the two Social Self f a c t o r e . 9 4 8 for the boys and ,756 for the g i r l s a n d of the two self esteem factors,--.809 for the boys and .580 for the girls, further suggest that Labouvie and Ruetsch's factors split for the girls more clearly than for the boys. However, the further analyses show that Labouvie and Ruetsch's data set is quite 114
MULTIVARIATE BEHAVIORAL RESEARCH
R. McDonald
Table 1
Downloaded by [McMaster University] at 06:53 23 December 2014
Model
df
chisqu
d
RMSEA
URFI
remarkable for its lack of interest, since, in their two-factor model, equating factor loadings does not give a significantly worse fit than not equating them, and indeed gives a slightly better fit as measured by the goodness-offit indices tabulated. These include the unbiassed estimate o f the noncentrality parameter, d (McDonald, 1988), the Root Mean Squared Error of Approximation, RMSEA (Browne & Cudeck, 1992), and the Unbiassed Relative Fit Index, URFI (McDonald & Marsh, 1990). The same is true in the four-factor model. Thus, there was no problem of failure of factorial invariance in the first place, and no motive for our curiosity. However, the methodological point remains, that even an apparent failure of factorial invariance is a basis for further investigation, not to be ignored. Some thirty years of consulting on applications of factor analysis have led me to conclude that precisely because our concepts are always less precise than we would wish, and hence less precisely instantiated in items than we would hope, we must be as clear about their semantics as possible. Fuzzy sets may be isomorphic with fuzzy thinking, but this fact does not provide any rational justification for either. References Browne, M. W. & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods and Research, 21,230-258. Fraser, C. & McDonald, R. P. (1988). COSAN: Covariance structure analysis. Multivariate Behavioral Research, 23,263-265. Labouvie, E. & Ruetsch, C. (1995). Testing for equivalence of measurement scales: Simple structure and metric equivalence reconsidered. Multivariate Behavioral Research, 30, 63-76. McDonald, R. P. (1981). The dimensionality of tests and items. British Journal of Mathematical and SfafisficalPsychology, 34, 1 10-1 17. McDonald, R. P. (1988). An index of goodness of fit based on noncentrality. Journal of Class~ficafion,6,97- 103. McDonald, R. P. & Marsh, H. W. (1990). Choosing a multivariate model: Noncentrality and goodness of fit. Psychological Bulletin, 107, 247-255. McDonald, R. P. & Mulaik, S. A. (1979), The determinacy of common factors: A nontechnical review. Psychological Bullefm, 86, 297-306. MULTIVARIATE BEHAVIORAL RESEARCH
115
