COMMUNITY HEALTH STUDIES VOLUME XW,NUMBER 2,1990 IS PROBABILITY SAMPLING ALWAYS BETTER? A COMPARISON OF RESULTS FROM A QUOTA AND A PROBABILITY SAMPLE SURVEY Robert Graham Cumming
Department of Community Medicine, University of Sydney, Westmead Hospital. Westmead. 2145
Abstract Two surveys in the same defiied population in Sydney's western suburbs in 1986 and 1987 provided the opportunity to compare results obtained from a quota and a probability sample survey. These surveys were designed to provide information for the planning of local health promotion programs. The quota sample survey was conducted in shopping centres and used quota sampling to select 1727 respondents. In the second survey, area probability sampling was used to select 484 respondents. This survey had a response rate of 65 per cent. There were 15 questions common to both surveys; results of only three differed significantly (peO.05) between surveys. None of these differences was important from a public health perspective. The agreement between the results of these two surveys probably reflects the fact that the same selection bias has operated in both. Unless a very high response rate can be achieved, quota sample surveys with age and sex quota controls may be an acceptable alternative to probability sample surveys for gathering local data relevant to the development of health programs. Introduction The aim of descriptive epidemiological studies is to measure particular health characteristics in defined populations of interest. Except rarely, when the entire population is studied, these characteristics must be measured in a representative sample. Probability sampling methods, of which random selection of subjects is best known. rue the accepted way of achieving representativeness.la In probability samples each member of the population of interest has a known non-zero chance of inclusion. Studies also appear in the literature which have used non-probability sampling methods such as convenience, purposive and quota sampling.' Quota sampling is widely used by market research companies. It involves selection of subjects into various sub-groups of the study sample. These
CUMMING
sub-groups are defined by criteria, known as quota controls, such as age and sex. Census data is often used to estimate the number of subjects required in each sub-group. The choice of the actual subjects selected is left to the interviewer. Quota sampling ensures representativeness in terms of the quota controls but there is doubt as to how well it performs on other characteristics. For example, Moser and Stuart found an underrepresentation in their quota sample of people who left school before the age of 15 years.' There are major problems with quota sampling. Because nothing is known about the probability of selection into the sample, the application of statistical theory is impossible. This means that the variance around prevalence estimates from a quota sample cannot b e calculated. Selection bias is also a major concern. Response rates cannot be calculated, nor can characteristics of non-respondents be identified. Hence, it is difficult to make a judgement on the likely effects of selection bias. This problem is compounded by the unknown criteria used by the interviewer to select subjects. Despite these problems. Kish concedes that "quota samples probably produce some good results" and that they are generally superior to other forms of nonprobability sampling.' Choosing the type of sampling method to use in a health survey involves three main issues: precision, bias and cost. As noted above. quota sampling methods do not allow estimation of precision, and selection bias is also a problem of unknown size in this type of survey. In probability samples, precision is mainly a function of sample size: the larger the number of subjects, the tighter the confidence intervals around point estimates of prevalence. I t is important to recognize that selection bias can also occur in probability samples if a high response rate is not achieved. For example, smokers tend to be under-represented among survey respondents.' This means that a health survey with a low response rate is likely to under-estimate the prevalence of smoking.
132
COMMUNITY HEALTH STUDIES
If an accurate estimate of the occurrence of some health characteristic is required, then a probability sample of adequate size and with a high response rate is essential. However, for purposes of planning local health programs such a high order of accuracy may be unnecessary. Moreover, practical constraints of time and money often preclude a high response rate. In these circumstances consideration might be given to using a non-probability sampling method. The opportunity to see whether the results of a health survey using quota sampling differed from those of a probability sample survey with a less than ideal response rate arose recently in the Western Sydney Health Study. This two-phase descriptive study was designed to provide policyrelevant information on health status, health service utilisation and health-related knowledge, attitudes and behaviours, in a defiied population in the western suburbs of Sydney. It consisted of two separate surveys conducted in the same population: the first survey, in November 1986. was of a sample of shoppers' and the second. in May 1987, used area probability sampling methods.' Because many of the questions asked were common to both surveys, it is possible to compnre results.
census collector's districts (CDs) were selected with equal probability from each of the three Health Areas. The number of CDs in each area was 448.248 and 330. and the average number of dwellings in selected CDs in these areas was 271, 282 and 259. From a random start point in each CD, households were systematically selected with a one in six skip interval. One eligible subject was selected for interview from each of these households using a selection grid which took into account the rank age order of subjects previously selected in households of the same size. Quatiomires The questionnaires used in each survey were interviewer-administered and about half of the 15 interviewers used in the area probability survey had also been involved in the shopping centre survey. The quota sample survey questionnaire comprised 32 questions and required about 15 minutes to complete; the questionnaire used in the probability sample survey comprised 97 questions and took about 45 minutes to complete. Apart from questions about socio-demographic characteristics, 15 questions had identical wording in both surveys. Statistical Issues and Analysis The sample size for each survey was calculated assuming simple random sampling. A sample size of 500 results in 95 per cent confidence intervals of 7.4 to 12.6 per cent and 45.6 to 54.4 per cent around point estimates of 10 and 50 per cent, respectively. This degree of precision was felt to be adequate for the purposes of these surveys. In the quota sample survey it was planned to examine results separately for each of the three areas surveyed and so the required sample size was a minimum of 1500. Results of the probability sample were weighted for the number of eligible respondents in the household and post-stratification' was used to adjust the sample to the age and sex distribution of the 1986 census for the target population. Responses to all questions were dichotomized and the statistical significance of differences between prevalence rates in the two surveys was calculated with the z-test for differences between proportions.' This assumes simple random sampling, which is not strictly statistically valid here. The statistical power to detect differences was assessed with methods described by Cohen.*O With sample sizes of 484 and 1727 and an alpha of 0.05. power varied from as low as 60 per cent for the difference between 3 and 5 per cent to over 95 per cent for the difference between 65 and 55 per cent.
Methods The population targetted for study in the Western Sydney Health Study was all persons aged 14 years and over resident in three Health Areas (Cumberland. Prospect and Whitlam) in Sydney's western suburbs. The total population so defined was 570.973 in the June 30. 1986 Australian Census.' Quota Sample Subjects were interviewed in six shopping centres, two in each of the three Health Areas surveyed. These shopping centres were chosen with the aid of local community health workers to be representative of the Health Areas (on the basis of their catchment areas and customer throughput). The survey was conducted over a six day period and interviewing times were arranged so that a wide variety of shopping times was covered, including evening and Saturday shopping. A quota sampling method was used to select respondents, with age and sex (based on the 1981 census) as the quota controls. For the purposes of this paper, respondents not resident in the target population defiied above, have been excluded (n=337).
Probability Sample Multi-stage area probability sampling was used to select subjects in the second survey. Five
CUMMING
133
COMMUNITY HEALTH STUDIES
TABLE 1 Socio-demographiccharacterlstlcs of quota and probability' samples compared with 1!%6
Quota Sample %
Probability Sample %
1986 Census %
(n-1
(n=l727) Sex2 Male Female
census data
50.6 49.4
41.7 58.3
49.8 50.2
14.4 25.0 20.3 17.1 11.5 7.9 3.8
11.6 19.4 21.9 14.9 13.4 11.2 7.6
15.0 21.8 21.0 16.5 11.8 8.0 5.8
speaking Countries'
17.6
16.9
27.6
No post-school qualifications'
72.1
76.3
62.7
Age OremY
14-19 20-29 30-39 4049 50-59 60-69 7ot Born m non-English
1. The results for the probability sample are unweighted. 2. The denominator for the 1986 census data is the population of the three Health Areas of the target population aged 14 years and over. 3. The denominator for the two surveys and the 1986 census data is those aged 15 years and over.
cost
The costs of the surveys were compared by calculating how much more it would have cost to use the area probability sampling method to gather the same data as was collected in shopping centres. The major factor affecting this calculation was the time it took in the probability sample SUNCY to make contact with selected respondents.
Results The study sample for the probability sample survey comprised 484 respondents. Excluding those respondents resident outside of the defied target area, there was a total of 1727 respondents in the quota sample. Comparison of the sociodemographic characteristics of each sample and the June 30, 1986 Australian Census findings for the target population is shown in table 1. Both samples were broadly reprcsmtative of the defied population of interest. However, there are some important differences which merit comment. Most striking is the under-representationof males in the probability sample. In contrast, the quota sampling method used to select subjects in shopping centres ensured equal numbers of males
CUMMING
and females. For the same reason, the shopping centre sample was more representative than the probability sample in terms of age, them being an over-representation of older people in the latter. Note that post-stratification by age and sex has the effect of resolving these problems in the probability sample. In both samples there was an underrepresentation of persons born in non-English speaking countries and of persons with qualifications beyond school level. However, the educational achievement of the quota sample was slightly closer to the m m s figure than was that of the probability sample. The response rate in the probability sample was low: 65 per cent. Unfortunately, only limited information is available about non-respondents. Twenty per cent of the non-response was due to no contact at all being made with anyone in the selected household, despite four call-backs. The remaining non-response was due to refusal to participate. The age and sex of 144 of these refusals is known and on these two characteristics there were no important differences from respondents.
134
COMMUNllY HEALTH STUDIES
TABLE 2 Comparison between results of a quota and a probability' sample survey conducted in the same target population in western Sydney.
Quota sample % (n=1727)
Probablllty sample % (n=484)
Self-rated health (Excellent, good)
68.3
74.6
6.3
(1.8,10.8)*
No.days in bed
89.5
90.1
0.6
(-2.4.3.6)
79.2
80.8
1.6
(-2.4,5.6)
Doctor perceived as main source of health information
33.7
41.5
7.8
(2.9.12.7)*
Control over health (A lot, complete)
67.7
65.0
-2.7
(-7.5.2.1)
Breast self-exam at least monthly
38.8
40.6
1.8
(-3.1.6.7)
Dlfference between samples
(Mper cent Confidence Limits)
inpast2weeks
No.days of reduced activity due to illness in past 2 weeks
(-6.lJ.3)
Mammogram (ever)
17.7
15.3
-2.4
Pap smeac in past 5 years
65.3
55.2
-10.1
Current smoker
38.5
33.9
4.6
(-9.4,0.2)
Hypertensian
23.9
22.9
-1.8
(-6.0,2.4)
Raised lipids
7.2
7.4
0.2
(-2.4.2.8)
Heart Attack
4.3
3.0
-1..3
(-3.1,0.5)
Angina
5.3
3.4
-1.9
(-3.8.0.0)
Stroke
1.2
1.9
0.7
(-0.6.2.0)
Diabetes
3.9
2.5
-1.4
(-3.1.0.3)
(-15.1.-5.1)*
Self-reported morbidity and risk factors:
* 1.
p < 0.05 for difference between proportions by z-test. The results for the probability sample are weighted for household size and post-stratified by age and sex.
the probability sample respondents. They were a little more likely to be smokers and to report having had a heart attack, angina or diabetes. However, none of these differences were statistically significant at the 0.05 level. The most important difference between the two surveys was their cost. In the probability sample survey it took about one hour on average for interviewers to make contact with selected subjects and begin interviewing. For 1727 subjects (the sample size in the shopping centre survey), at a wage of $10 an hour, this would
There was close agreement between the findings of the two surveys for those 15 questions common to both questionnaires (see table 2). Only three differences in prevalences were significant (pc0.05): the proportion who perceived their own health to be excellent or good, the proportion who felt doctors were their main source of health information and the proportion of women who reported a Pap smear in the past five years. Overall, it appears that those interviewed in shopping centres reported poorer health than did
CUMMMG
135
COMMUNITY HEALTH STUDIES
surveys. However, the health of the probability sample was still far worse than that reported in the National Heart Foundation Risk Factor Prevalence Survey, conducted nationally in 1983.1aSo, the difference in health status results of the two surveys described in this paper does not alter the conclusion that health in western Sydney is poor. The major difference between the two surveys was cost: it would have cost about four times as much in interviewers' wages to collect the shopping centre data if the probability sample methods had been used. Of course, the costs of both probability and quota sample surveys will vary widely depending on the particular survey. For example, in quota samples, the more quota controls used, the greater the cost. The cost of a probability sample survey relates to factors such as stratification, cluster sampling and the number of call-backs. Nevertheless. the cost findings of this study are consistent with the general consensus: the cost per data item is much lower in a quota sample survey.lJJ There is another way of comparing the costs of surveys and that is in terms of the cost per unit variance. This approach was not possible here because the variance of the quota sample cannot be determined. However, as Kish has stressed, the precision per dollar may indicate less value than simply the number of interviews per dollar.' The lack of important differences between the results of the two surveys discussed in this paper does not necessarily mean that they provide an accurate description of the prevalence of measured characteristics in the population studied. The agreement between results may simply reflect the fact that the same selection bias has operated in the assembling of both study samples. Nonprobability samples are well known for consisting of 'volunteer' subjects. It could be argued that probability samples from surveys with high nonresponse rates suffer from the same problem. Selection bias occurs when respondents and non-respondents differ on relevant characteristics. Criqui et al. studied the characteristics of nonrespondents in a cardiovascular disease-related study in Southern California." They found that non-respondents tended to have higher levels of cardiovascular disease and were more likely to smoke. However, they were less likely to have raised blood lipids or a family history of heart disease. They suggested that respondents may represent a "worried well" population. Cobb et al.. in a study of arthritis prevalence, made a similar ~bservation.~~ More generally, it has been shown that respondents and non-respondents tend to differ on a wide range of psychological and social variables."
represent about $17,000 on top of the cost of actually conducting the interview. Interviewers' wages in the survey conducted in shopping centres were about $5,000. Thus, using area probability sampling instead of quota sampling in shopping centres would have cost over four times as much in interviewers' wages and produced essentially the same results.
Discussion Comparison of the results of a probability sample survey and a quota sample survey conducted within six months of each other in the same target population revealed few important differences. I t should be noted that the comparison here was not between the ideal probability sample survey with a 100 per cent response rate and a typical convenience sample. Instead, the findings relate directly to the common situation in descriptive epidemiological research where a less than ideal response rate is obtained. How do such studies perform in relation to quota sample surveys using age and sex controls? Respondents selected in shopping centres were more representative of the defined population of interest than were respondents in the probability sample, at least on those socio-demographic characteristics that were measured. The use of age and sex quota controls is chiefly responsible for this and helped overcome the common problem in shopping centre surveys of an over-representation of women in their 20s and 30s. The significance of the differences between the results of the two surveys can be judged in two ways: statistically and from a public health perspective. The surveys described in this paper were designed to provide information for the planning of local health promotion programs. The statistically significant differences between the two survey's findings do not affect the direction these programs might take. Thus. the fact that 42 per cent of respondents in the probability sample gave doctors and nurses as their main source of health information compared to 34 per cent in the quota sample is statistically highly significant (p
Department of Community Medicine, University of Sydney, Westmead Hospital. Westmead. 2145
Abstract Two surveys in the same defiied population in Sydney's western suburbs in 1986 and 1987 provided the opportunity to compare results obtained from a quota and a probability sample survey. These surveys were designed to provide information for the planning of local health promotion programs. The quota sample survey was conducted in shopping centres and used quota sampling to select 1727 respondents. In the second survey, area probability sampling was used to select 484 respondents. This survey had a response rate of 65 per cent. There were 15 questions common to both surveys; results of only three differed significantly (peO.05) between surveys. None of these differences was important from a public health perspective. The agreement between the results of these two surveys probably reflects the fact that the same selection bias has operated in both. Unless a very high response rate can be achieved, quota sample surveys with age and sex quota controls may be an acceptable alternative to probability sample surveys for gathering local data relevant to the development of health programs. Introduction The aim of descriptive epidemiological studies is to measure particular health characteristics in defined populations of interest. Except rarely, when the entire population is studied, these characteristics must be measured in a representative sample. Probability sampling methods, of which random selection of subjects is best known. rue the accepted way of achieving representativeness.la In probability samples each member of the population of interest has a known non-zero chance of inclusion. Studies also appear in the literature which have used non-probability sampling methods such as convenience, purposive and quota sampling.' Quota sampling is widely used by market research companies. It involves selection of subjects into various sub-groups of the study sample. These
CUMMING
sub-groups are defined by criteria, known as quota controls, such as age and sex. Census data is often used to estimate the number of subjects required in each sub-group. The choice of the actual subjects selected is left to the interviewer. Quota sampling ensures representativeness in terms of the quota controls but there is doubt as to how well it performs on other characteristics. For example, Moser and Stuart found an underrepresentation in their quota sample of people who left school before the age of 15 years.' There are major problems with quota sampling. Because nothing is known about the probability of selection into the sample, the application of statistical theory is impossible. This means that the variance around prevalence estimates from a quota sample cannot b e calculated. Selection bias is also a major concern. Response rates cannot be calculated, nor can characteristics of non-respondents be identified. Hence, it is difficult to make a judgement on the likely effects of selection bias. This problem is compounded by the unknown criteria used by the interviewer to select subjects. Despite these problems. Kish concedes that "quota samples probably produce some good results" and that they are generally superior to other forms of nonprobability sampling.' Choosing the type of sampling method to use in a health survey involves three main issues: precision, bias and cost. As noted above. quota sampling methods do not allow estimation of precision, and selection bias is also a problem of unknown size in this type of survey. In probability samples, precision is mainly a function of sample size: the larger the number of subjects, the tighter the confidence intervals around point estimates of prevalence. I t is important to recognize that selection bias can also occur in probability samples if a high response rate is not achieved. For example, smokers tend to be under-represented among survey respondents.' This means that a health survey with a low response rate is likely to under-estimate the prevalence of smoking.
132
COMMUNITY HEALTH STUDIES
If an accurate estimate of the occurrence of some health characteristic is required, then a probability sample of adequate size and with a high response rate is essential. However, for purposes of planning local health programs such a high order of accuracy may be unnecessary. Moreover, practical constraints of time and money often preclude a high response rate. In these circumstances consideration might be given to using a non-probability sampling method. The opportunity to see whether the results of a health survey using quota sampling differed from those of a probability sample survey with a less than ideal response rate arose recently in the Western Sydney Health Study. This two-phase descriptive study was designed to provide policyrelevant information on health status, health service utilisation and health-related knowledge, attitudes and behaviours, in a defiied population in the western suburbs of Sydney. It consisted of two separate surveys conducted in the same population: the first survey, in November 1986. was of a sample of shoppers' and the second. in May 1987, used area probability sampling methods.' Because many of the questions asked were common to both surveys, it is possible to compnre results.
census collector's districts (CDs) were selected with equal probability from each of the three Health Areas. The number of CDs in each area was 448.248 and 330. and the average number of dwellings in selected CDs in these areas was 271, 282 and 259. From a random start point in each CD, households were systematically selected with a one in six skip interval. One eligible subject was selected for interview from each of these households using a selection grid which took into account the rank age order of subjects previously selected in households of the same size. Quatiomires The questionnaires used in each survey were interviewer-administered and about half of the 15 interviewers used in the area probability survey had also been involved in the shopping centre survey. The quota sample survey questionnaire comprised 32 questions and required about 15 minutes to complete; the questionnaire used in the probability sample survey comprised 97 questions and took about 45 minutes to complete. Apart from questions about socio-demographic characteristics, 15 questions had identical wording in both surveys. Statistical Issues and Analysis The sample size for each survey was calculated assuming simple random sampling. A sample size of 500 results in 95 per cent confidence intervals of 7.4 to 12.6 per cent and 45.6 to 54.4 per cent around point estimates of 10 and 50 per cent, respectively. This degree of precision was felt to be adequate for the purposes of these surveys. In the quota sample survey it was planned to examine results separately for each of the three areas surveyed and so the required sample size was a minimum of 1500. Results of the probability sample were weighted for the number of eligible respondents in the household and post-stratification' was used to adjust the sample to the age and sex distribution of the 1986 census for the target population. Responses to all questions were dichotomized and the statistical significance of differences between prevalence rates in the two surveys was calculated with the z-test for differences between proportions.' This assumes simple random sampling, which is not strictly statistically valid here. The statistical power to detect differences was assessed with methods described by Cohen.*O With sample sizes of 484 and 1727 and an alpha of 0.05. power varied from as low as 60 per cent for the difference between 3 and 5 per cent to over 95 per cent for the difference between 65 and 55 per cent.
Methods The population targetted for study in the Western Sydney Health Study was all persons aged 14 years and over resident in three Health Areas (Cumberland. Prospect and Whitlam) in Sydney's western suburbs. The total population so defined was 570.973 in the June 30. 1986 Australian Census.' Quota Sample Subjects were interviewed in six shopping centres, two in each of the three Health Areas surveyed. These shopping centres were chosen with the aid of local community health workers to be representative of the Health Areas (on the basis of their catchment areas and customer throughput). The survey was conducted over a six day period and interviewing times were arranged so that a wide variety of shopping times was covered, including evening and Saturday shopping. A quota sampling method was used to select respondents, with age and sex (based on the 1981 census) as the quota controls. For the purposes of this paper, respondents not resident in the target population defiied above, have been excluded (n=337).
Probability Sample Multi-stage area probability sampling was used to select subjects in the second survey. Five
CUMMING
133
COMMUNITY HEALTH STUDIES
TABLE 1 Socio-demographiccharacterlstlcs of quota and probability' samples compared with 1!%6
Quota Sample %
Probability Sample %
1986 Census %
(n-1
(n=l727) Sex2 Male Female
census data
50.6 49.4
41.7 58.3
49.8 50.2
14.4 25.0 20.3 17.1 11.5 7.9 3.8
11.6 19.4 21.9 14.9 13.4 11.2 7.6
15.0 21.8 21.0 16.5 11.8 8.0 5.8
speaking Countries'
17.6
16.9
27.6
No post-school qualifications'
72.1
76.3
62.7
Age OremY
14-19 20-29 30-39 4049 50-59 60-69 7ot Born m non-English
1. The results for the probability sample are unweighted. 2. The denominator for the 1986 census data is the population of the three Health Areas of the target population aged 14 years and over. 3. The denominator for the two surveys and the 1986 census data is those aged 15 years and over.
cost
The costs of the surveys were compared by calculating how much more it would have cost to use the area probability sampling method to gather the same data as was collected in shopping centres. The major factor affecting this calculation was the time it took in the probability sample SUNCY to make contact with selected respondents.
Results The study sample for the probability sample survey comprised 484 respondents. Excluding those respondents resident outside of the defied target area, there was a total of 1727 respondents in the quota sample. Comparison of the sociodemographic characteristics of each sample and the June 30, 1986 Australian Census findings for the target population is shown in table 1. Both samples were broadly reprcsmtative of the defied population of interest. However, there are some important differences which merit comment. Most striking is the under-representationof males in the probability sample. In contrast, the quota sampling method used to select subjects in shopping centres ensured equal numbers of males
CUMMING
and females. For the same reason, the shopping centre sample was more representative than the probability sample in terms of age, them being an over-representation of older people in the latter. Note that post-stratification by age and sex has the effect of resolving these problems in the probability sample. In both samples there was an underrepresentation of persons born in non-English speaking countries and of persons with qualifications beyond school level. However, the educational achievement of the quota sample was slightly closer to the m m s figure than was that of the probability sample. The response rate in the probability sample was low: 65 per cent. Unfortunately, only limited information is available about non-respondents. Twenty per cent of the non-response was due to no contact at all being made with anyone in the selected household, despite four call-backs. The remaining non-response was due to refusal to participate. The age and sex of 144 of these refusals is known and on these two characteristics there were no important differences from respondents.
134
COMMUNllY HEALTH STUDIES
TABLE 2 Comparison between results of a quota and a probability' sample survey conducted in the same target population in western Sydney.
Quota sample % (n=1727)
Probablllty sample % (n=484)
Self-rated health (Excellent, good)
68.3
74.6
6.3
(1.8,10.8)*
No.days in bed
89.5
90.1
0.6
(-2.4.3.6)
79.2
80.8
1.6
(-2.4,5.6)
Doctor perceived as main source of health information
33.7
41.5
7.8
(2.9.12.7)*
Control over health (A lot, complete)
67.7
65.0
-2.7
(-7.5.2.1)
Breast self-exam at least monthly
38.8
40.6
1.8
(-3.1.6.7)
Dlfference between samples
(Mper cent Confidence Limits)
inpast2weeks
No.days of reduced activity due to illness in past 2 weeks
(-6.lJ.3)
Mammogram (ever)
17.7
15.3
-2.4
Pap smeac in past 5 years
65.3
55.2
-10.1
Current smoker
38.5
33.9
4.6
(-9.4,0.2)
Hypertensian
23.9
22.9
-1.8
(-6.0,2.4)
Raised lipids
7.2
7.4
0.2
(-2.4.2.8)
Heart Attack
4.3
3.0
-1..3
(-3.1,0.5)
Angina
5.3
3.4
-1.9
(-3.8.0.0)
Stroke
1.2
1.9
0.7
(-0.6.2.0)
Diabetes
3.9
2.5
-1.4
(-3.1.0.3)
(-15.1.-5.1)*
Self-reported morbidity and risk factors:
* 1.
p < 0.05 for difference between proportions by z-test. The results for the probability sample are weighted for household size and post-stratified by age and sex.
the probability sample respondents. They were a little more likely to be smokers and to report having had a heart attack, angina or diabetes. However, none of these differences were statistically significant at the 0.05 level. The most important difference between the two surveys was their cost. In the probability sample survey it took about one hour on average for interviewers to make contact with selected subjects and begin interviewing. For 1727 subjects (the sample size in the shopping centre survey), at a wage of $10 an hour, this would
There was close agreement between the findings of the two surveys for those 15 questions common to both questionnaires (see table 2). Only three differences in prevalences were significant (pc0.05): the proportion who perceived their own health to be excellent or good, the proportion who felt doctors were their main source of health information and the proportion of women who reported a Pap smear in the past five years. Overall, it appears that those interviewed in shopping centres reported poorer health than did
CUMMMG
135
COMMUNITY HEALTH STUDIES
surveys. However, the health of the probability sample was still far worse than that reported in the National Heart Foundation Risk Factor Prevalence Survey, conducted nationally in 1983.1aSo, the difference in health status results of the two surveys described in this paper does not alter the conclusion that health in western Sydney is poor. The major difference between the two surveys was cost: it would have cost about four times as much in interviewers' wages to collect the shopping centre data if the probability sample methods had been used. Of course, the costs of both probability and quota sample surveys will vary widely depending on the particular survey. For example, in quota samples, the more quota controls used, the greater the cost. The cost of a probability sample survey relates to factors such as stratification, cluster sampling and the number of call-backs. Nevertheless. the cost findings of this study are consistent with the general consensus: the cost per data item is much lower in a quota sample survey.lJJ There is another way of comparing the costs of surveys and that is in terms of the cost per unit variance. This approach was not possible here because the variance of the quota sample cannot be determined. However, as Kish has stressed, the precision per dollar may indicate less value than simply the number of interviews per dollar.' The lack of important differences between the results of the two surveys discussed in this paper does not necessarily mean that they provide an accurate description of the prevalence of measured characteristics in the population studied. The agreement between results may simply reflect the fact that the same selection bias has operated in the assembling of both study samples. Nonprobability samples are well known for consisting of 'volunteer' subjects. It could be argued that probability samples from surveys with high nonresponse rates suffer from the same problem. Selection bias occurs when respondents and non-respondents differ on relevant characteristics. Criqui et al. studied the characteristics of nonrespondents in a cardiovascular disease-related study in Southern California." They found that non-respondents tended to have higher levels of cardiovascular disease and were more likely to smoke. However, they were less likely to have raised blood lipids or a family history of heart disease. They suggested that respondents may represent a "worried well" population. Cobb et al.. in a study of arthritis prevalence, made a similar ~bservation.~~ More generally, it has been shown that respondents and non-respondents tend to differ on a wide range of psychological and social variables."
represent about $17,000 on top of the cost of actually conducting the interview. Interviewers' wages in the survey conducted in shopping centres were about $5,000. Thus, using area probability sampling instead of quota sampling in shopping centres would have cost over four times as much in interviewers' wages and produced essentially the same results.
Discussion Comparison of the results of a probability sample survey and a quota sample survey conducted within six months of each other in the same target population revealed few important differences. I t should be noted that the comparison here was not between the ideal probability sample survey with a 100 per cent response rate and a typical convenience sample. Instead, the findings relate directly to the common situation in descriptive epidemiological research where a less than ideal response rate is obtained. How do such studies perform in relation to quota sample surveys using age and sex controls? Respondents selected in shopping centres were more representative of the defined population of interest than were respondents in the probability sample, at least on those socio-demographic characteristics that were measured. The use of age and sex quota controls is chiefly responsible for this and helped overcome the common problem in shopping centre surveys of an over-representation of women in their 20s and 30s. The significance of the differences between the results of the two surveys can be judged in two ways: statistically and from a public health perspective. The surveys described in this paper were designed to provide information for the planning of local health promotion programs. The statistically significant differences between the two survey's findings do not affect the direction these programs might take. Thus. the fact that 42 per cent of respondents in the probability sample gave doctors and nurses as their main source of health information compared to 34 per cent in the quota sample is statistically highly significant (p
