when i say When I say . . . response process validity Clarence Kreiter

It’s easy to miss the forest for the trees when first attempting to comprehend the concept of test validity. It is not too hard to imagine a perplexed student, who, having read the rhetorically complex literature that defines modern validity theory,1 looks up from her assigned text and asks: ‘What does validity really mean?’ To which her teacher answers: ‘It’s simply about whether a test measures what it’s supposed to measure.’ Although this answer is vaguely correct, it leaves the student wondering why she was assigned such difficult reading material, and merely postpones any attempt to address the far more difficult and important question of how one might establish this thing called ‘validity’. Unfortunately, generating and interpreting validity evidence is far from simple. The broader literature discusses many different types of validity study (predictive, concurrent, internal consistency, content, criterion. . .) that might provide evidence on what is being measured and how precisely ‘it’ (whatever it is) is being measured. As modern validity theory maintains that evidence produced by each type of study is really part of construct validity, researchers are required to generate, interpret and integrate multiple sources of evidence to support a construct interpretation. The importance of each type of evidence is dependent on the test’s characteristics and what it purports to measure (the construct). Construct validity is established by generating a validity argument that provides support for a particular interpretation of scores generated by the test. If all of this sounds a bit convoluted and tangential to answering the student’s simple question, it should be comforting to learn that there is at least one type of validity evidence that is very direct in Iowa City, Iowa, USA

Medical Education 2015: 49: 247–248

addressing whether a test measures what it’s supposed to measure. Response process validity is more qualitative than quantitative, and is often a matter of simple common sense rather than complex statistical argument. The Standards for Educational and Psychological Testing defines response process validity as an analysis of whether ‘the response processes of test takers can provide evidence concerning the fit between the construct and the detailed nature of the performance or response actually engaged in by examinees’.2 This implies that when measurement objectives are defined by the sorts of behaviours that coincide with sound educational objectives (ability to explain, understanding terminology, ability to predict), rather than hypothetical mental functions or constructs (scientific thinking, synthesis, analysis), response process validity is practically a sufficient source of evidence. In an educational achievement test, response process validity is concerned with establishing whether the items, problems or tasks on a test are likely to require the behavioural skill we seek to assess. If they do, the skill will have a causal role in determining the quantitative outcome of the test and the test can be regarded as valid. In some ways, this is a ‘retro’ pre-construct type of validity evidence that reverts to addressing whether the test itself is valid. Critics of construct validity have discussed the impracticality of interpretative arguments that require sophisticated nomological networks.3 Such networks don’t exist for the achievement tests used in medical education. Response process validity evidence is well aligned with achievement testing because instructors are most interested in whether a behavioural skill plays a causal role in determining the score outcome. Such evidence may be derived Correspondence: Clarence Kreiter, Department of Family Medicine, University of Iowa, 1204 MEB, Office of Consultation and Research in Medical Education, Iowa City, Iowa 52242, USA. Tel: 00 1 319 335 8906; E-mail: [email protected]

doi: 10.1111/medu.12572

ª 2015 John Wiley & Sons Ltd. MEDICAL EDUCATION 2015; 49: 247–248

247

C Kreiter from something as simple as asking examinees what strategy they used to respond to an item, or by analysing the steps required to produce and score the ‘correct’ answer. It is instructive to consider a recent example of response process validity in action. Over the last decade there has emerged a large literature aimed at documenting the construct validity of the script concordance test (SCT) item. A review of that research suggested that the evidence generally supported its proposed use, but also noted that the response process evidence was not robust.4 To address this, researchers examined the logical steps required to solve an SCT problem and their relation to the methods used to score them.5,6 This research relied on a ‘common sense’ response process approach and analysed what the SCT question format was actually asking an examinee to do and how a correct response was defined and recorded. Whereas the evidence for internal consistency, relationships with other variables and concurrent validity had all proven quite positive, the evidence for response process validity clearly showed there to be serious problems when the SCT item was used for educational achievement testing. This example demonstrates that response process validity is a useful and practical concept in the psychometrician’s toolbox.

248

REFERENCES 1 Kane MT. Validation. In: Brennan RL, ed. Educational Measurement, 4th edn. West Port, CT: American Council on Education, Praeger Publishing 2006;17–64. 2 American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington, DC: American Psychological Association 1999;12–3. 3 Colliver JC, Conlee MJ, Verhulst SJ. From test validity to construct validity. . .and back? Teach Learn Med 2012;46:366–71. 4 Lubarsky S, Charlin B, Cook DA, Chalk C, van der Vleuten CPM. Script concordance testing: a review of published validity evidence. Med Educ 2011;45:329–38. 5 Kreiter CD. Commentary: the response process validity of a script concordance test item. Adv Health Sci Educ Theory Pract 2012;17:7–9. 6 Lineberry M, Kreiter CD, Bordage G. Threats to validity in the use and interpretation of script concordance test scores. Med Educ 2013;47:1175–83. Received 26 June 2014; editorial comments to author 22 July 2014; accepted for publication 29 July 2014

ª 2015 John Wiley & Sons Ltd. MEDICAL EDUCATION 2015; 49: 247–248

Copyright of Medical Education is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.

When I say response process validity.

When I say response process validity. - PDF Download Free
44KB Sizes 0 Downloads 4 Views