Psychological Reports, 1991, 69, 543-549. O Psychological Reports 1991
T H E BASIC H I V DISEASE KNOWLEDGE QUESTIONNAIRE: A RASCH-SCALED INSTRUMENT TO MEASURE ESSENTIAL H I V KNOWLEDGE ' WILLIAM McCOWN Hahnemann Universily
AND
JUDITH JOHNSON Vilhnova Universily Hahnemann Universiiy
Summary.-The need for improvement in HIV disease-related psychometric questionnaires is noted. In response to this need a Rasch-scaled rneasure to ascertain basic HIV disease knowledge was developed and calibrated on 1092 subjects. Four studies conducted with the measure suggest adequate test-retest reliability. Three vahdity studies also conducted with this questionnaire suggest that the measure is sensitive to knowledge obtained from HIV-related training and education. Potential uses and limitations of this new questionnaire are discussed.
H I V disease is one of the dominant public health concerns of this century. The only foreseeable strategy to reduce the spread of this virus involves a campaign of education and subsequent behavioral change (US Surgeon General's Report on Acquired Immune Deficiency Syndrome, 1986). Following the Surgeon General's recommendations, several comprehensive and theoretically derived interventions have been formulated to reduce HIV spread (Rugg, 1990). Essential to these programs are psychometric instruments that can assess pre- and postintervention HIV knowledge and prevention s k d s . At least 40 questionnaires have been developed to fill this role, most of which were developed for specific and time-limited interventions. To date, no single inventory or instrument for H I V knowledge and skills assessment has received wide use. The reason for the variety of instruments presently available may be linked with the dissatisfaction regarding existing questionnaires which concern HIV knowledge and prevention skills. Little is known about the reliability of most HIV-related psychometric instruments. Secondly, even if data regarding reliability are favorable, many of these measures are of questionable or otherwise limited validity. Too frequently such questionnaires have been developed in an ad hoc manner with specific and inappropriate samples, such as those normed on small groups of college students or on w h t e homosexual men from large epicenters in the early stages of the epidemic. Some HIV questionnaires are superfluous to most persons' immediate needs of avoiding the infection, since they often include assessment of knowledge of pathogenesis or seroprevalence that are less relevant to lay populations. Fur-
'Address corres ondence to both authors at Hahnemann University, Mail Stop 626, Broad and Vine, ~hiladelp&a,PA 19012.
544
W. McCOWN & J. JOHNSON
thermore, a few instruments are lengthy to administer, w h c h reduces their clinical utility. Other questionnaires may be politicized, ethnically or culturally insensitive, or even offensive, further restricting utility. However, the major problem of most HIV-related psychometric instruments may be the degree of literacy required on the part of the respondent. Tests regarding HIV knowledge or behavior often include the use of medical and technical terms that are not part of typical parlance. These include vague and potentially confusing phrases such as "nonejaculatory, mutual oral sexual activity," "spermatocide barrier products," "unprotected receptive anal relations," and "serotransmissive risk occurrences." As a response to this concern, some test constructors have attempted to develop specific instruments that are more culturally sensitive to the needs of particular groups, for example, through the inclusion of the use of slang (Mantell & Devittis, 1990). Unfortunately, the idiosyncratic features of each unique instrument may make comparisons between HIV knowledge in different racial, ethnic, age, or other groups difficult. A useful H I V knowledge test should have a number of characteristics. It should be brief so it can be used quickly by the physician or mental health professional. Optimally, such a test would be reliable and valid across a variety of populations. For example, it should require minimum education, literacy, or intellectual functioning on the part of the examinee. I t should also include an oral and written form, both of which should be scorable by persons with limited training. The purpose of this study is to report the construction of an H I V knowledge and prevention skills test with some of these characteristics. ~
-
Test Construction Items regarding H I V knowledge and prevention skills were solicited from 10 medical personnel consultants involved in HIV prevention. These included three physicians, three public health nurses, an epidemiologist, and three public health educators. Six of these consultants had been recommended by community-based H I V prevention organization leaders for both their general level of knowledge and their sensitivity to the problems encountered in HIV education. The other four consultants were recommended by the initial group of six, also on the basis of their knowledge and sensitivity. Each consultant was asked to include up to 25 "facts or skds" necessary for mastery by the general population to prevent HIV disease and to generate a scoring key regarding evidence of correct knowledge for each item. Generated items were examined for duplication and were reduced to a list of 17. A second group of five medical personnel (two physicians and three public health nurses) involved in H I V education reviewed the 17 items and were requested to generate additional items if needed. Two additional items were
HIV KNOWLEDGE: RASCH-SCALE MEASURE
545
produced. Each of these 19 items was then written with minor wording variation and included as additional items in the pool of test items, resulting in 38 items being included in the item pool. All test items were reviewed by-three special education instructors to increase the comprehension for individuals within about three standard deviations below mean intellectual functioning. This was done so items could be understood by people functioning in the mild range of mental retardation. Each of these 38 items, along with the scoring key, was then resubmitted to the first group of experts for accuracy and veracity. A Rasch scaling procedure (Wright & Stone, 1979) was employed for test construction and subsequent "item calibration." The Rasch model was chosen rather than more traditional psychometric test construction techniques because the Rasch model produces test items that are relatively "sample free" or applicable across diverse groups (KLine, 1986). The goals of the Rasch model are (1) to provide a procedure where the internal consistency of a scale can be estimated regardless of the variance of the population under study and (2) to identify any item-population interactions which would render dubious any comparisons among populations. Using a Rasch model, scores for subjects can be obtained without substantial reference to ability because item difficulty can be distinguished from subjects' abilities. Rasch scaling is particularly useful for test-retest situations and situations where one measure needs to be developed and applied to several populations of varying ability. I t is most useful where there is a well-defined item pool (KLine, 1986), such as with skills necessary to prevent the spread of H I V and where subjects cannot guess correct responses. Data were collected from 1092 individuals (664 men) who visited one of three hospitals in two cities. Subjects were solicited upon hospital entry and then asked to answer anonymously the written version of t h s test. This took approximately 10 minutes for each subject. Desiring to maintain anonymity, no information was collected regarding age, race, socioeconomic status, or other variables of potential identification. However, each of these hospitals provided services for multiethnic, multiracial, multicultural communities, including women, minorities, homosexual men and lesbians, and indigent patients. Therefore, it can be assumed that respondents-individuals visitingthese patients-were also racially, culturally, and economically diverse. Items from this sample were scored by two paraprofessionals (high school diploma) who received approximately 90 minutes of training, based on a scoring manual developed by the first author. Interrater agreement for the entire sample was .93. Items were Rasch-calibrated using the program RASCAL (Assessment Systems Corporation, 1988) which is based on Wright and Stone's (1979) general unconditional calibration method. Items without interrater agreement were counted as "failed responses" for the
546
W. McCOWN & J. JOHNSON
purposes of analysis. As is usual in Rasch item calibration, omitted items were also treated as incorrect responses. Thirteen items were fitted to the Rasch model, as indicated by the nonsignificant x 2 associated with total responses for the item. Table 1 indicates these items. TABLE 1
TESTITEMSFITTING RASCHMODEL Item Number and Content 1. 2. 3. 4. 5.
6.
7. 8.
9. 10.
11. 12.
13.
What does HIV do to the body? Correct if states any answer regarding immune system. Is HIV disease serious? Correct if states that it is a fatal disorder. How do you cure HIV disease? Correct if indicates there is no present cure. How do you know if you have HIV infection? Correct if knows necessity of blood test. What does HIV have to do with blood? Correct if demonstrates understanding that human blood may harbor HIV virus. How do you et HIV infection through sex? Correct if iemonstrates understanding of exchange of bodily fluid or states necessity of barrier methods to prevent viral transmission. What does HIV have to do with dru needles? Correct if demonstrates understanLg that HIV is transmitted through needle sharing. How do you use a condom? Correct if demonstrates understandin both that it (a) goes on penis and goes on (b) before sexuafcontact. If you use drugs, what should you do with the needle? Correct if demonstrates understanding that you do not share needles. Tell me why you use a condom and/or foam? Correct if demonstrates understanding of barrier concept. Tell me some ways you do not get HIV disease. Correct if subject can generate two or more answers. Tell me some ways you can have sex and not get HIV d e c t i o n . Correct if subject can generate answers involving any of the following: (a) abstinence, (b) barrier methods, (c) noninsertive sexual contact, (d) sex only with HIV negative persons. What two ~ h n g sshould you do if you have the HIV infection? Correct if srate, (a) avoidance of sex with noninfected partner and (b) necessity of medical treatment.
x2* 22.8 18.2 25.4 23.2
12.5
15.1 17.2
24.3 23.2 15.5 11.8
13.2 25.5
*Cn't ~' c a lvalue for item rejection X,,2>26.3
RESULTS Although some advocates of the Rasch model and other variants of Item Response Theory state that traditional reliability and validity measures are irrelevant for Rasch models, indices of stability and criterion validity were obtained in five different studies involving diverse populations. Four reliability and three validity studies were conducted. These are discussed briefly below.
HIV KNOWLEDGE: RASCH-SCALE MEASURE
547
Reliability Studies I n Reliability Study 1 we attempted to estimate the stability of the instrument in a general population. The Basic HIV Disease Questionnaire constructed above was administered via telephone at 1-mo. intervals to 30 men and 30 women (mean age = 43.4 yr.) who did not have prior experience with the disease, either professionally or personally. The test-retest reliability for this group was .74 at one month and .61 at six months. This reliability may have been attenuated somewhat by the fact that 11 of the respondents indicated that they had sought additional HIV information following the first administration of the test. In Reliability Study 2 we attempted to estimate the test-retest coefficient for a written form of the Basic HIV Disease Questionnaire. Sixty individuals (41 men) were contacted at random from a telephone directory and solicited to participate in this study. Subjects completed the questionnaire in writing twice, at 1-mo. intervals. The reliability for one month was .78, with seven subjects stating that they had pursued additional knowledge regarding HIV disease on the basis of their experience with the first presentation of the test. Reliability for six months was .69. I n Reliabihty Study 3 we attempted to estimate the test-retest reliability for mildly retarded individuals. The questionnaire was administered orally to 30 individuals in the mildly retarded range of mental functioning (WAIS-R or Stanford-Binet 55 to 70, as indicated by school records, and with significant evidence of psychosocial adaptive impairment, as indicated by placement in special education). The test-retest correlations for this group were at one month .81 and .72 for six months. I n Reliability Study 4 we attempted to estimate the questionnaire's stability with psychiatric inpatients. The questionnaire was administered to a group of 34 mixed hospitalized psychiatric patients (all men) at a VA psychiatric unit following a one-week interval. Test-retest correlation was .58. While this number is rather low, it probably reflects attenuation due to an improvement in patients' conditions associated with enhanced mental status functioning. One-month test-retest interval (obtained after patients were discharged) r was .53. N o data were obtained for a six-month period, as many of the patients had experienced rehospitalization or were otherwise unlocatable. Validity Studies Validity studies were conducted to ascertain whether scores on the questionnaire increased following HIV knowledge training. I n Validity Study 1 the Basic HIV Knowledge Questionnaire was administered to a group of 39 nursing-level health care professionals before and after four hours of HIV prevention training. The questionnaire was administered immediately before and after training, as part of participants' evaluations. Scores increased from a mean of 8.41 (SD = 3.3) to 11.01 (SD = 2.2; 2-tailed t,, = -5.01, p
T H E BASIC H I V DISEASE KNOWLEDGE QUESTIONNAIRE: A RASCH-SCALED INSTRUMENT TO MEASURE ESSENTIAL H I V KNOWLEDGE ' WILLIAM McCOWN Hahnemann Universily
AND
JUDITH JOHNSON Vilhnova Universily Hahnemann Universiiy
Summary.-The need for improvement in HIV disease-related psychometric questionnaires is noted. In response to this need a Rasch-scaled rneasure to ascertain basic HIV disease knowledge was developed and calibrated on 1092 subjects. Four studies conducted with the measure suggest adequate test-retest reliability. Three vahdity studies also conducted with this questionnaire suggest that the measure is sensitive to knowledge obtained from HIV-related training and education. Potential uses and limitations of this new questionnaire are discussed.
H I V disease is one of the dominant public health concerns of this century. The only foreseeable strategy to reduce the spread of this virus involves a campaign of education and subsequent behavioral change (US Surgeon General's Report on Acquired Immune Deficiency Syndrome, 1986). Following the Surgeon General's recommendations, several comprehensive and theoretically derived interventions have been formulated to reduce HIV spread (Rugg, 1990). Essential to these programs are psychometric instruments that can assess pre- and postintervention HIV knowledge and prevention s k d s . At least 40 questionnaires have been developed to fill this role, most of which were developed for specific and time-limited interventions. To date, no single inventory or instrument for H I V knowledge and skills assessment has received wide use. The reason for the variety of instruments presently available may be linked with the dissatisfaction regarding existing questionnaires which concern HIV knowledge and prevention skills. Little is known about the reliability of most HIV-related psychometric instruments. Secondly, even if data regarding reliability are favorable, many of these measures are of questionable or otherwise limited validity. Too frequently such questionnaires have been developed in an ad hoc manner with specific and inappropriate samples, such as those normed on small groups of college students or on w h t e homosexual men from large epicenters in the early stages of the epidemic. Some HIV questionnaires are superfluous to most persons' immediate needs of avoiding the infection, since they often include assessment of knowledge of pathogenesis or seroprevalence that are less relevant to lay populations. Fur-
'Address corres ondence to both authors at Hahnemann University, Mail Stop 626, Broad and Vine, ~hiladelp&a,PA 19012.
544
W. McCOWN & J. JOHNSON
thermore, a few instruments are lengthy to administer, w h c h reduces their clinical utility. Other questionnaires may be politicized, ethnically or culturally insensitive, or even offensive, further restricting utility. However, the major problem of most HIV-related psychometric instruments may be the degree of literacy required on the part of the respondent. Tests regarding HIV knowledge or behavior often include the use of medical and technical terms that are not part of typical parlance. These include vague and potentially confusing phrases such as "nonejaculatory, mutual oral sexual activity," "spermatocide barrier products," "unprotected receptive anal relations," and "serotransmissive risk occurrences." As a response to this concern, some test constructors have attempted to develop specific instruments that are more culturally sensitive to the needs of particular groups, for example, through the inclusion of the use of slang (Mantell & Devittis, 1990). Unfortunately, the idiosyncratic features of each unique instrument may make comparisons between HIV knowledge in different racial, ethnic, age, or other groups difficult. A useful H I V knowledge test should have a number of characteristics. It should be brief so it can be used quickly by the physician or mental health professional. Optimally, such a test would be reliable and valid across a variety of populations. For example, it should require minimum education, literacy, or intellectual functioning on the part of the examinee. I t should also include an oral and written form, both of which should be scorable by persons with limited training. The purpose of this study is to report the construction of an H I V knowledge and prevention skills test with some of these characteristics. ~
-
Test Construction Items regarding H I V knowledge and prevention skills were solicited from 10 medical personnel consultants involved in HIV prevention. These included three physicians, three public health nurses, an epidemiologist, and three public health educators. Six of these consultants had been recommended by community-based H I V prevention organization leaders for both their general level of knowledge and their sensitivity to the problems encountered in HIV education. The other four consultants were recommended by the initial group of six, also on the basis of their knowledge and sensitivity. Each consultant was asked to include up to 25 "facts or skds" necessary for mastery by the general population to prevent HIV disease and to generate a scoring key regarding evidence of correct knowledge for each item. Generated items were examined for duplication and were reduced to a list of 17. A second group of five medical personnel (two physicians and three public health nurses) involved in H I V education reviewed the 17 items and were requested to generate additional items if needed. Two additional items were
HIV KNOWLEDGE: RASCH-SCALE MEASURE
545
produced. Each of these 19 items was then written with minor wording variation and included as additional items in the pool of test items, resulting in 38 items being included in the item pool. All test items were reviewed by-three special education instructors to increase the comprehension for individuals within about three standard deviations below mean intellectual functioning. This was done so items could be understood by people functioning in the mild range of mental retardation. Each of these 38 items, along with the scoring key, was then resubmitted to the first group of experts for accuracy and veracity. A Rasch scaling procedure (Wright & Stone, 1979) was employed for test construction and subsequent "item calibration." The Rasch model was chosen rather than more traditional psychometric test construction techniques because the Rasch model produces test items that are relatively "sample free" or applicable across diverse groups (KLine, 1986). The goals of the Rasch model are (1) to provide a procedure where the internal consistency of a scale can be estimated regardless of the variance of the population under study and (2) to identify any item-population interactions which would render dubious any comparisons among populations. Using a Rasch model, scores for subjects can be obtained without substantial reference to ability because item difficulty can be distinguished from subjects' abilities. Rasch scaling is particularly useful for test-retest situations and situations where one measure needs to be developed and applied to several populations of varying ability. I t is most useful where there is a well-defined item pool (KLine, 1986), such as with skills necessary to prevent the spread of H I V and where subjects cannot guess correct responses. Data were collected from 1092 individuals (664 men) who visited one of three hospitals in two cities. Subjects were solicited upon hospital entry and then asked to answer anonymously the written version of t h s test. This took approximately 10 minutes for each subject. Desiring to maintain anonymity, no information was collected regarding age, race, socioeconomic status, or other variables of potential identification. However, each of these hospitals provided services for multiethnic, multiracial, multicultural communities, including women, minorities, homosexual men and lesbians, and indigent patients. Therefore, it can be assumed that respondents-individuals visitingthese patients-were also racially, culturally, and economically diverse. Items from this sample were scored by two paraprofessionals (high school diploma) who received approximately 90 minutes of training, based on a scoring manual developed by the first author. Interrater agreement for the entire sample was .93. Items were Rasch-calibrated using the program RASCAL (Assessment Systems Corporation, 1988) which is based on Wright and Stone's (1979) general unconditional calibration method. Items without interrater agreement were counted as "failed responses" for the
546
W. McCOWN & J. JOHNSON
purposes of analysis. As is usual in Rasch item calibration, omitted items were also treated as incorrect responses. Thirteen items were fitted to the Rasch model, as indicated by the nonsignificant x 2 associated with total responses for the item. Table 1 indicates these items. TABLE 1
TESTITEMSFITTING RASCHMODEL Item Number and Content 1. 2. 3. 4. 5.
6.
7. 8.
9. 10.
11. 12.
13.
What does HIV do to the body? Correct if states any answer regarding immune system. Is HIV disease serious? Correct if states that it is a fatal disorder. How do you cure HIV disease? Correct if indicates there is no present cure. How do you know if you have HIV infection? Correct if knows necessity of blood test. What does HIV have to do with blood? Correct if demonstrates understanding that human blood may harbor HIV virus. How do you et HIV infection through sex? Correct if iemonstrates understanding of exchange of bodily fluid or states necessity of barrier methods to prevent viral transmission. What does HIV have to do with dru needles? Correct if demonstrates understanLg that HIV is transmitted through needle sharing. How do you use a condom? Correct if demonstrates understandin both that it (a) goes on penis and goes on (b) before sexuafcontact. If you use drugs, what should you do with the needle? Correct if demonstrates understanding that you do not share needles. Tell me why you use a condom and/or foam? Correct if demonstrates understanding of barrier concept. Tell me some ways you do not get HIV disease. Correct if subject can generate two or more answers. Tell me some ways you can have sex and not get HIV d e c t i o n . Correct if subject can generate answers involving any of the following: (a) abstinence, (b) barrier methods, (c) noninsertive sexual contact, (d) sex only with HIV negative persons. What two ~ h n g sshould you do if you have the HIV infection? Correct if srate, (a) avoidance of sex with noninfected partner and (b) necessity of medical treatment.
x2* 22.8 18.2 25.4 23.2
12.5
15.1 17.2
24.3 23.2 15.5 11.8
13.2 25.5
*Cn't ~' c a lvalue for item rejection X,,2>26.3
RESULTS Although some advocates of the Rasch model and other variants of Item Response Theory state that traditional reliability and validity measures are irrelevant for Rasch models, indices of stability and criterion validity were obtained in five different studies involving diverse populations. Four reliability and three validity studies were conducted. These are discussed briefly below.
HIV KNOWLEDGE: RASCH-SCALE MEASURE
547
Reliability Studies I n Reliability Study 1 we attempted to estimate the stability of the instrument in a general population. The Basic HIV Disease Questionnaire constructed above was administered via telephone at 1-mo. intervals to 30 men and 30 women (mean age = 43.4 yr.) who did not have prior experience with the disease, either professionally or personally. The test-retest reliability for this group was .74 at one month and .61 at six months. This reliability may have been attenuated somewhat by the fact that 11 of the respondents indicated that they had sought additional HIV information following the first administration of the test. In Reliability Study 2 we attempted to estimate the test-retest coefficient for a written form of the Basic HIV Disease Questionnaire. Sixty individuals (41 men) were contacted at random from a telephone directory and solicited to participate in this study. Subjects completed the questionnaire in writing twice, at 1-mo. intervals. The reliability for one month was .78, with seven subjects stating that they had pursued additional knowledge regarding HIV disease on the basis of their experience with the first presentation of the test. Reliability for six months was .69. I n Reliabihty Study 3 we attempted to estimate the test-retest reliability for mildly retarded individuals. The questionnaire was administered orally to 30 individuals in the mildly retarded range of mental functioning (WAIS-R or Stanford-Binet 55 to 70, as indicated by school records, and with significant evidence of psychosocial adaptive impairment, as indicated by placement in special education). The test-retest correlations for this group were at one month .81 and .72 for six months. I n Reliability Study 4 we attempted to estimate the questionnaire's stability with psychiatric inpatients. The questionnaire was administered to a group of 34 mixed hospitalized psychiatric patients (all men) at a VA psychiatric unit following a one-week interval. Test-retest correlation was .58. While this number is rather low, it probably reflects attenuation due to an improvement in patients' conditions associated with enhanced mental status functioning. One-month test-retest interval (obtained after patients were discharged) r was .53. N o data were obtained for a six-month period, as many of the patients had experienced rehospitalization or were otherwise unlocatable. Validity Studies Validity studies were conducted to ascertain whether scores on the questionnaire increased following HIV knowledge training. I n Validity Study 1 the Basic HIV Knowledge Questionnaire was administered to a group of 39 nursing-level health care professionals before and after four hours of HIV prevention training. The questionnaire was administered immediately before and after training, as part of participants' evaluations. Scores increased from a mean of 8.41 (SD = 3.3) to 11.01 (SD = 2.2; 2-tailed t,, = -5.01, p
