American Journal of Community Psychology, VoL 6, No. 4, 1978
A Comparison of Three Survey Methods to Obtain Data for Community Mental Health Program Planning Andrew Lee Hinkle and Glen D. King 2 Auburn University
A perennial problem for mental health planners is assessing community needs and existing services. The three most common methods used to obtain this data are the telephone survey, the mail-out questionnaire, and the face-to-face interview. However, there are advantages and disadvantages associated with each approach in terms o f sampling, response rates, and economic costs. The present study utilized all three methods to survey the same community population in order to determine the comparability o f obtained data and relative efficacy o f the methods. A standard 21-item questionnaire was developed to obtain demographic and epidemioIogical data. This included nine Likert-type items to obtain opinions on a typical question such as "To what extent would you have confidence in recommending the Mental Health Center to members o f your immediate family?" In the first method, survey teams made door-todoor interviews to complete 449 questionnaires on a random sample. In the second method, 1,000 questionnaires were mailed to a random sample with returns requested. In the third method, 224 people were randomly selected from the telephone directory and asked to respond to the questionnaire over the phone. Precautions were taken in all methods to ensure confidentiality o f responses. All respondents were classified according to a two-factor index based on occupation and education. The data were analyzed to determine whether comparable data were obtained through divergent methods. Results are discussed and implications are given for community mental health planners.
1 This research was supported in part by Auburn University Research Council grant 73-50, Glen D. King, principal investigator. 2 All correspondence should be sent to Glen D. King, Department of Psychology, Auburn University, Auburn, Alabama 36830. 389 0091-0562/78/0800-0389505.00/0 © 1978 Plenum Publishing Corporation
390
Hinkle and King
The function of community mental health is to provide comprehensive mental health services according to National Institute of Mental Health guidelines to geographically defined populations (Scherl, 1970). Planning for the delivery of these services is complex, as numerous studies have shown that great variations exist in the prevalence of mental illness and the availability of psychiatric care (Auld & Meyers, 1954; Hollingshead & Redlich, 1958; Hunt, 1960; Imber, Nash, & Stone, 1955; McMahon, 1964; Meyers & Bean, 1968; Meyers & Schaffer, 1954; Winder & Hersko, 1955). Hollingshead and Redlich have reported in their classic study (1958) that both the prevalence of mental illness and type of psychiatric care received are directly related to socioeconomic status (SES), although these findings have been disputed by Gursslin, Hunt, and Roach (1964). Nevertheless, the demographic and,epidemiological characteristics of any mental health catchment area are likely to vary widely, and the implementation of a comprehensive community mental health program must take this into account. While exhaustive and detailed demographic data are readily available from U.S. Bureau of Census and other reports, epidemiological factors are more difficult to specify. Even more difficult are attempts to relate the occurrence of these two factors in the population. Thus, community mental health program planners at the local, state, and federal levels must often conduct their own surveys to obtain accurate and up-to-date information. The three most common ways to obtain such first-hand information are: interview, mail-out questionnaire, and telephone survey. The relative advantages and disadvantages of these techniques can best be evaluated in terms of complete, accurate, and representative information obtained versus the expense of time, personnel, and money. Studies using these different techniques to gather information have reported a wide range of response rates. Eddy, Paap, and Glad (1970) obtained as high as a 98% response rate on selected questions in a survey of mental health problems by personal interview. However this is generally considered the most expensive technique in terms of time and money. An alternate method is the use of the mail-out questionnaire, and although the expense is thought to be less than the interview method, the maximum return rate of questionnaires is usually no more than 50% (Plog, 1963), and on mental health surveys, may be as low as 11% (Johnson, Boutwell, & Hinkle, 1975). The telephone survey is generally considered the least expensive, and return rates of 58% have been recently reported (Weaver, Holmes, & Glenn, 1975). In obtaining information for mental health program planning it is obvious that the choice of the optimal survey technique depends on the quality and quantity of the data in addition to the cost incurred. The present study was designed to obtain the same data from mutually exclusive samples from the same population using the three techniques described. The results obtained
Comparison of Survey Methods
3 91
provide some guidelines as to the most efficient technique to obtain information on mental health problems and how they are served in their community.
METHOD
Subjects The area surveyed by all three methods was the town of Auburn, Alabama with a 1976 estimated population of 27,000. To obtain a representative sample of the population, a map was divided into seven sectors which were estimated to represent the SES characteristics of the area. Interview teams of two persons were assigned to each sector with instructions to contact every residence on randomly selected streets within their sector until they had fulfilled their quota. They recorded names and addresses from mail boxes of houses not contacted to provide a mailing list for the mail-out questionnaires. This list provided about half of the mailing addresses and was supplemented by an additional 500 names selected at random from the telephone directory. An additional 224 members were selected at random from the telephone directory for use in the telephone survey.
Questionnaire A standard two-page questionnaire was developed for use in all methods by "debugging" a preliminary form on a sample of 100 college students. The final version requested demographic information such as occupation, income range, and educational level. It also asked whether any one in the household had experienced a personal problem requiring assistance from a community agency, the nature of the problem, and the type of service obtained. In addition, this included nine Likert-type items to obtain opinions on a 5-point scale to a typical question such as "To what extent would you have confidence in recommending the Mental Health Center to members of your immediate family?"
Procedure In all three methods, preliminary remarks were used to identify the survey party, explain the content and purpose of the questionnaire, and ensure the confidentiality of the respondent. In the interview method, these general remarks were made prior to the presentation of the questionnaire. If the person responded, he was given an envelope in which to seal his reply before
392
Hinkle and King
returning it to the survey team. In the marl-out method these points were described in a cover letter with instructions to return the completed questionnaire in the enclosed self-addressed stamped envelope. For the telephone survey, the interviewer informed the potential respondent of these conditions and additionally said that their telephone number was taken from an anonymous list that did not identify the number by name. In all survey methods considerable emphasis was given to prospective respondents that their replies would be anonymous and not identified in any individual way.
Results All respondents were classified by survey method, a two-factor socioeconomic index based on their occupation and education (Meyers & Bean, 1968) and whether they had experienced an emotional problem requiring community assistance or not. A comparison of the total responses from the three survey methods shows there are striking differences in the quantity and quality of data obtained. From Table I it can be seen that the highest attempt/completion ratio was the 70% response rate for interview, followed by 57% and 19% for telephone and mail-out methods, respectively. Perhaps even a more important comparison is the relative economic and time costs for the three methods. While the marl-out is far more efficient in time required, it is the most expensive technique in terms of cost per completed questionnaire (see Table II). The telephone survey is the least expensive method but has a combined refusal and no contact rate of 43% which may bias the results. A further inspection of the responses to scaled questions shows that there are systematic response biases as a function of the three classification factors. Table III presents the percentage of respondents per interview method Table I. Response Rates for Interview, Mail Questionnaire, and Telephone Survey Interviewa Attempts No contact Refusal Completed
N
%
641 141 b 51 449
100 22 8 70
Mail N
%
1000 100 278 c 28 532 53 190 19
Telephone N
%
224 50 47 127
100 22 21 57
aThese data were extrapolated from the actual returned questionnaires as some interviewers did not keep complete records. bperson not at home. CQuestionnaire returned by post office due to incorrect/ incomplete address.
Comparison of Survey Methods
393
Table II. Relative Costs for Interview, Mail Questionnaire, and Telephone Survey
Personnel @ $2.00/hour Postage @ 13c/letter Printing/paper @ $21.00/1000 c Envelopes @ $25.00/1000 Total cost Number completed Cost per completed questionnaire
Interview
Mail
Telephone
$592.00 a
$34.00 b
$112.00
0
$260.00
0
$9,40
$27.00 d
$4.20
0
$25.00
0
$601.40 449 $1.34
$346.00 190 $1.82
$116.20 127 $0.92
a Interviews conducted by teams of two. bAddressing and stuffing questionnaires/return envelopes. tEach questionnaire consists of two 81A× 14 papers. dlncludes additional cost of 1000 cover letters. by income level compared to the percent of the population at each income level as reported in the census. It is immediately noticeable that the various methods differ substantially in the correspondence of the income levels of their respective samples of respondents to the census figures. Table IV presents the responses of subjects to a particular question by method, SES, help received or not, and opinion to amplify the point that systematic response biases by interview method exist. The data pertaining to the specific question "To what extent are you aware of the East Alabama Mental Health Center and its services?" are representative of data generated for others questions. Referring to Table IV, of those people who had received help for emotional problems Table III. Percentage of Respondents at Each Income Level for Interview (I), Mail Questionnaire (M), and Telephone Survey (T) Procedures in Comparison to Census Income-Level Data (C) Income level ($)
C
I
M
T
2,999 & below 3,000-4,999 5,000-7,999 8,000-9,999 10,000-14,999 15,000-24,999 25,000 & above Uncoded Total (%)
18 14 22 09 19 16 03 00 101
10 06 07 08 16 32 15 07 101
03 07 09 06 21 27 22 04 99
11 07 08 06 10 28 12 18 I00
394
Hinkle and King
Table IV. Percentage of Responses by Interview (I), Mail Questionnaire (M), and Telephone Survey (T) by SES I-VIII a SES
a +b I
M
c T
I
M
d+e T
I
Total/method
M
T
I
M
T
22 0 3 3 5 32
29 7 0 7 7 50
45 13 8 9 24 99
76 8 3 3 11 101
41 30 0 12 12 100
16 1 0 4 2 24
7 5 3 3 5 23
54 10 10 9 16 99
56 13 5 12 13 99
36 15 15 13 22 101
Class A: People who had received help for emotional problems I & II
III IV&V VI VII & VIII Total/method
13 7 4 4 11 39
35 5 0 0 3 43
12 12 0 7 7 36
12 3 3 1 7 25
19 3 0 0 3 24
0 12 0 0 0 12
20 4 1 4 7 36
Class B: People who had not had problems requiring assistance I & II 25 28 20 19 12 9 10 III 6 7 6 2 5 4 2 IV & V 6 5 9 2 0 3 2 VI 5 7 9 2 2 1 2 VII & VIII 7 9 16 7 3 1 1 Total/method 50 56 60 33 20 18 17
aThese data were obtained from an exemplary question, "To what extent are you aware of the East Alabama Mental Health Center and its services?" The responses are represented on the following 5 point scale: a = not at all; b = less than average; c = average; d = more than average; e = much more than average.
(Class A), an average of 45% of the respondents on both the interview and telephone survey were in the highest SES groups (I & II). However, on the mail survey an average of 75% of the respondents in Class A were in groups I and II. Of those people who had not reported problems requiring assistance (Class B), an average of about 50% of the interviews and 60% of the mail-outs were in groups I and II, while 35% of the telephone survey data were in the same group. In looking at the range of responses across the 5-point scale in Table IV, it appears that SES groups I and II mail-out respondents give more neutral (c) and negative (a and b) responses, especially if they had required assistance. Group I and II respondents on interview and telephone surveys show the opposite trend, endorsing more positive attitudes, especially if they have required assistance. The remaining SES groups are moderately well represented and symmetrically distributed except for the middle SES group (III) and the unemployed/other group (VII and VIII). For telephone surveys under Class A, an obviously high average of about 30% fall into group III. For interviews under Class A, a disparate average of 25% fall into the unemployed/other group.
Comparison of Survey Methods
395
DISCUSSION It seems clear that there are relatively different expenses and response characteristics associated with the three different techniques. Perhaps the clearest finding is that, contrary to popular belief, the mail-out questionnaire is the most costly of the survey methods commonly used. Telephone surveys appear to be the most economical with the cost of door-to-door survey teams falling about midway between. However, these survey teams included two persons each for security precautions and thus almost doubled our total cost for that technique. If security risks for a particular survey area were considered low and utilization of one-person survey teams feasible, the door-to-door survey would be by far the most economical procedure, perhaps more than a dollar less per completed questionnaire than mail surveys. Perhaps the high use of the mail technique by budget-minded surveyors is due to consideration of absolute cost rather than the amount of data received relative to the expenditure. While it is true that more money was spent on the door-to-door interview, we received about 2½ times more data than in the mail questionnaire. In addition to economic costs, it was found that there were emotional costs as well. While the door-to-door teams experienced little emotional discomfort, a low refusal rate (8%), and reported having enjoyed the experience, the opposite was true for the telephone survey. Over one-fifth (21%) of the prospective telephone respondents refused to participate with the survey and hung-up, or expressed hostile and suspicious comments. Of those that did participate, many refused to answer some questions, e.g., approximately 20% refused to report their income. The telephone surveyors reported that this was an unpleasant experience for them, and they disliked the task. On one occasion a male surveyor was so distraught by the experience that he approached one of the researchers and asked to withdraw from the project. While this was an extreme case, it seems to represent the typical emotional effect on many of the phone surveyors. Another difference among the three methods were the characteristics of respondents. While all methods seemed to overrepresent the upper SES (I & II), the mail techniques greatly exaggerated this group, particularly for those who had received services. However, the interview data tended to give a better representation of all the SES groups, especially the lowest group, while the telephone survey tended to overrepresent SES III, especially for those who had experienced personal problems. Although more than 90% of the residences in the survey area have telephones, there may be a systematic bias which excludes upper SES respondents who have an unlisted telephone number, unavailable to the researcher, while the lower SES group may not have a telephone at all. While a random-digit dialing procedure suggested by Gtasser and Metzger (.1972) may serve to eliminate
396
Hinkle and King
the bias introduced by unlisted telephone numbers, it obviously does not solve the problem of excluding those who do not have a phone. Another interesting trend was in the ratings given by upper SES respondents by method. For marl respondents, there was a much greater proportion giving neutral and negative evaluations, especially if they had received assistance for some problem. Thus, a typical respondent on the marl survey seemed to be an upper SES individual who is critical of community services, especially if he has used them to assist him in resolving his personal problems. This suggests that community mental health surveyors who evaluate their program by marl questionnaire are more likely to receive a negative to neutral evaluation than a positive one. Since marl-out is also likely to cost more, with this method one pays more to look worse! Since none of these methods appear to give a stratified sample of the population, an alternate procedure to correct this bias would be to construct a stratified sample from the target population to match known demographic characteristics. For example, extrapolation from th income characteristics of the population according to the U.S. Census income data for this city, with a minimum of n = 10 per cell for statistical considerations, would require the following samples. The interview method would require 1,096 completed questionnaires at a cost of $1,467.42, to provide a data pool from which a stratified sample could be constructed to match the census income population characteristics. Similarly, the marl-out would require a pool of 1,898 completed questionnaires at a cost of $3,462.03, and the telephone survey would require 930 questionnaires at a cost of $851.75. The costs per completed questionnaire are based on data presented in Table II. Thus, to obtain the minimum sample, it is clear that the marl-out technique is exorbitantly costly. While the telephone method is ostensibly most economical, it is worth emphasizing again that the interview team employing only one surveyor costs about half as much as teams of two. In this case, the minimum interview sample collected by interviewers working alone rather than in pairs, would only cost $733.71, a savings of about $100 over the telephone survey. The results of the study clearly demonstrate that choice of survey method should be determined by two primary factors, the cost (total and per completed unit), and the representation of the population by the survey technique employed. Obviously, the marl-out survey technique fails both considerations. It is too expensive and appears to sample only upper SES individuals who express dissatisfaction. Both the telephone and facto-to-face interview conducted by a single interviewer are attractive in terms of financial costs, but the telephone method may be too expensive in terms of emotional cost to the interviewer. Both telephone and interview methods yield comparable data which is more representative of population than that generated by the marl-out survey
Comparison of Survey Methods
397
m e t h o d . It is apparent that either the t e l e p h o n e or interview m e t h o d m a y be used w i t h equal c o n f i d e n c e to assess c o m m u n i t y attitudes, for which the mailo u t survey m e t h o d appears totally i n a d e q u a t e .
REFERENCES Auld, F., Jr. & Meyers, J. K. Contributions to a theory for selecting psychotherapy patients. Journal of Clinical Psychology, 1954, 10, 56-60 Eddy, W. B., Paap, S. M., & Glad, D. D. Solving problems in living: The citizen's viewpoint. MentaIHygiene, 1970, 54(1), 64-71. Glasser, G. J., & Metzger, G. D. Random-digit dialing as a method of telephone sampling. Journal o f Marketing Research, 1972, 9, 59-64. Gursslin, O. R., Hunt, R. G., & Roach, J. L. Social class and the mental health movement. In F. Riessman, J. Cohen, & A. Pearl (Eds.), Mental health o f the poor. New York: Free Press, 1964, 57-67. Hollingshead, A. B., & Redlich, F. C. Social class and mental illness: A community study. New York: John Wiley & Sons, Inc., 1958. Hunt, R. G. Social class and mental illness: Some implications for clinical theory and practice. American Journal o f Psychiatry, 1960, 116, 1065-1069. Imber, S. D., Nash, E. H., & Stone, A. R. Social class and duration of psychotherapy. Journal o f ClinicaI Psychology, 1955, 11, 281-284. Johnson, T. I. Boutwell, C. R., & Hinkle, A. L. CMHC evaluation: A survey of mental health problems and professional services. Evaluation, 1975,2(2), 18-19. Meyers, J. K., & Bean, L. L. A decade later: A follow-up of social classand mental illness. New York: John Wiley & Sons, 1968. Meyers, J. K., & Schaffer, L. Social stratification and psychiatric practice: A study of an outpatient clinic. American Sociological Review, 1954, 19, 307-310. McMahon, J. T. The working class psychiatric patient: A clinical view. In F. Reissman, J. Cohen, & A. Pearl (Eds.), Mental health ofthepoor. New York: Free Press, 1964, 283-302. Plog, S. C. Explanation for a high return rate on a mail questionnaire. Public Opinion Quarterly, 1963, 27(2), 197-298. Scherl, D. J. The community mental health center and mental health services for the poor. In H. Grunebaum (Ed.), The practice o f community mental health. Boston: Little, Brown, & Co., 1970. Weaver, C. N., Holmes, S. L., & Glenn, N. D. Some characteristics of inaccessible respondents in a telephone survey. Journal of Applied Psychology, 1975, 60, 260-262. Winder, A. E., & Hersko, M. The effect of social class on the length and type of psychotherapy in a Veterans Administration mental hygiene clinic. Journal o f Clinical Psychology, 1955, 11, 77-79.
A Comparison of Three Survey Methods to Obtain Data for Community Mental Health Program Planning Andrew Lee Hinkle and Glen D. King 2 Auburn University
A perennial problem for mental health planners is assessing community needs and existing services. The three most common methods used to obtain this data are the telephone survey, the mail-out questionnaire, and the face-to-face interview. However, there are advantages and disadvantages associated with each approach in terms o f sampling, response rates, and economic costs. The present study utilized all three methods to survey the same community population in order to determine the comparability o f obtained data and relative efficacy o f the methods. A standard 21-item questionnaire was developed to obtain demographic and epidemioIogical data. This included nine Likert-type items to obtain opinions on a typical question such as "To what extent would you have confidence in recommending the Mental Health Center to members o f your immediate family?" In the first method, survey teams made door-todoor interviews to complete 449 questionnaires on a random sample. In the second method, 1,000 questionnaires were mailed to a random sample with returns requested. In the third method, 224 people were randomly selected from the telephone directory and asked to respond to the questionnaire over the phone. Precautions were taken in all methods to ensure confidentiality o f responses. All respondents were classified according to a two-factor index based on occupation and education. The data were analyzed to determine whether comparable data were obtained through divergent methods. Results are discussed and implications are given for community mental health planners.
1 This research was supported in part by Auburn University Research Council grant 73-50, Glen D. King, principal investigator. 2 All correspondence should be sent to Glen D. King, Department of Psychology, Auburn University, Auburn, Alabama 36830. 389 0091-0562/78/0800-0389505.00/0 © 1978 Plenum Publishing Corporation
390
Hinkle and King
The function of community mental health is to provide comprehensive mental health services according to National Institute of Mental Health guidelines to geographically defined populations (Scherl, 1970). Planning for the delivery of these services is complex, as numerous studies have shown that great variations exist in the prevalence of mental illness and the availability of psychiatric care (Auld & Meyers, 1954; Hollingshead & Redlich, 1958; Hunt, 1960; Imber, Nash, & Stone, 1955; McMahon, 1964; Meyers & Bean, 1968; Meyers & Schaffer, 1954; Winder & Hersko, 1955). Hollingshead and Redlich have reported in their classic study (1958) that both the prevalence of mental illness and type of psychiatric care received are directly related to socioeconomic status (SES), although these findings have been disputed by Gursslin, Hunt, and Roach (1964). Nevertheless, the demographic and,epidemiological characteristics of any mental health catchment area are likely to vary widely, and the implementation of a comprehensive community mental health program must take this into account. While exhaustive and detailed demographic data are readily available from U.S. Bureau of Census and other reports, epidemiological factors are more difficult to specify. Even more difficult are attempts to relate the occurrence of these two factors in the population. Thus, community mental health program planners at the local, state, and federal levels must often conduct their own surveys to obtain accurate and up-to-date information. The three most common ways to obtain such first-hand information are: interview, mail-out questionnaire, and telephone survey. The relative advantages and disadvantages of these techniques can best be evaluated in terms of complete, accurate, and representative information obtained versus the expense of time, personnel, and money. Studies using these different techniques to gather information have reported a wide range of response rates. Eddy, Paap, and Glad (1970) obtained as high as a 98% response rate on selected questions in a survey of mental health problems by personal interview. However this is generally considered the most expensive technique in terms of time and money. An alternate method is the use of the mail-out questionnaire, and although the expense is thought to be less than the interview method, the maximum return rate of questionnaires is usually no more than 50% (Plog, 1963), and on mental health surveys, may be as low as 11% (Johnson, Boutwell, & Hinkle, 1975). The telephone survey is generally considered the least expensive, and return rates of 58% have been recently reported (Weaver, Holmes, & Glenn, 1975). In obtaining information for mental health program planning it is obvious that the choice of the optimal survey technique depends on the quality and quantity of the data in addition to the cost incurred. The present study was designed to obtain the same data from mutually exclusive samples from the same population using the three techniques described. The results obtained
Comparison of Survey Methods
3 91
provide some guidelines as to the most efficient technique to obtain information on mental health problems and how they are served in their community.
METHOD
Subjects The area surveyed by all three methods was the town of Auburn, Alabama with a 1976 estimated population of 27,000. To obtain a representative sample of the population, a map was divided into seven sectors which were estimated to represent the SES characteristics of the area. Interview teams of two persons were assigned to each sector with instructions to contact every residence on randomly selected streets within their sector until they had fulfilled their quota. They recorded names and addresses from mail boxes of houses not contacted to provide a mailing list for the mail-out questionnaires. This list provided about half of the mailing addresses and was supplemented by an additional 500 names selected at random from the telephone directory. An additional 224 members were selected at random from the telephone directory for use in the telephone survey.
Questionnaire A standard two-page questionnaire was developed for use in all methods by "debugging" a preliminary form on a sample of 100 college students. The final version requested demographic information such as occupation, income range, and educational level. It also asked whether any one in the household had experienced a personal problem requiring assistance from a community agency, the nature of the problem, and the type of service obtained. In addition, this included nine Likert-type items to obtain opinions on a 5-point scale to a typical question such as "To what extent would you have confidence in recommending the Mental Health Center to members of your immediate family?"
Procedure In all three methods, preliminary remarks were used to identify the survey party, explain the content and purpose of the questionnaire, and ensure the confidentiality of the respondent. In the interview method, these general remarks were made prior to the presentation of the questionnaire. If the person responded, he was given an envelope in which to seal his reply before
392
Hinkle and King
returning it to the survey team. In the marl-out method these points were described in a cover letter with instructions to return the completed questionnaire in the enclosed self-addressed stamped envelope. For the telephone survey, the interviewer informed the potential respondent of these conditions and additionally said that their telephone number was taken from an anonymous list that did not identify the number by name. In all survey methods considerable emphasis was given to prospective respondents that their replies would be anonymous and not identified in any individual way.
Results All respondents were classified by survey method, a two-factor socioeconomic index based on their occupation and education (Meyers & Bean, 1968) and whether they had experienced an emotional problem requiring community assistance or not. A comparison of the total responses from the three survey methods shows there are striking differences in the quantity and quality of data obtained. From Table I it can be seen that the highest attempt/completion ratio was the 70% response rate for interview, followed by 57% and 19% for telephone and mail-out methods, respectively. Perhaps even a more important comparison is the relative economic and time costs for the three methods. While the marl-out is far more efficient in time required, it is the most expensive technique in terms of cost per completed questionnaire (see Table II). The telephone survey is the least expensive method but has a combined refusal and no contact rate of 43% which may bias the results. A further inspection of the responses to scaled questions shows that there are systematic response biases as a function of the three classification factors. Table III presents the percentage of respondents per interview method Table I. Response Rates for Interview, Mail Questionnaire, and Telephone Survey Interviewa Attempts No contact Refusal Completed
N
%
641 141 b 51 449
100 22 8 70
Mail N
%
1000 100 278 c 28 532 53 190 19
Telephone N
%
224 50 47 127
100 22 21 57
aThese data were extrapolated from the actual returned questionnaires as some interviewers did not keep complete records. bperson not at home. CQuestionnaire returned by post office due to incorrect/ incomplete address.
Comparison of Survey Methods
393
Table II. Relative Costs for Interview, Mail Questionnaire, and Telephone Survey
Personnel @ $2.00/hour Postage @ 13c/letter Printing/paper @ $21.00/1000 c Envelopes @ $25.00/1000 Total cost Number completed Cost per completed questionnaire
Interview
Telephone
$592.00 a
$34.00 b
$112.00
0
$260.00
0
$9,40
$27.00 d
$4.20
0
$25.00
0
$601.40 449 $1.34
$346.00 190 $1.82
$116.20 127 $0.92
a Interviews conducted by teams of two. bAddressing and stuffing questionnaires/return envelopes. tEach questionnaire consists of two 81A× 14 papers. dlncludes additional cost of 1000 cover letters. by income level compared to the percent of the population at each income level as reported in the census. It is immediately noticeable that the various methods differ substantially in the correspondence of the income levels of their respective samples of respondents to the census figures. Table IV presents the responses of subjects to a particular question by method, SES, help received or not, and opinion to amplify the point that systematic response biases by interview method exist. The data pertaining to the specific question "To what extent are you aware of the East Alabama Mental Health Center and its services?" are representative of data generated for others questions. Referring to Table IV, of those people who had received help for emotional problems Table III. Percentage of Respondents at Each Income Level for Interview (I), Mail Questionnaire (M), and Telephone Survey (T) Procedures in Comparison to Census Income-Level Data (C) Income level ($)
C
I
M
T
2,999 & below 3,000-4,999 5,000-7,999 8,000-9,999 10,000-14,999 15,000-24,999 25,000 & above Uncoded Total (%)
18 14 22 09 19 16 03 00 101
10 06 07 08 16 32 15 07 101
03 07 09 06 21 27 22 04 99
11 07 08 06 10 28 12 18 I00
394
Hinkle and King
Table IV. Percentage of Responses by Interview (I), Mail Questionnaire (M), and Telephone Survey (T) by SES I-VIII a SES
a +b I
M
c T
I
M
d+e T
I
Total/method
M
T
I
M
T
22 0 3 3 5 32
29 7 0 7 7 50
45 13 8 9 24 99
76 8 3 3 11 101
41 30 0 12 12 100
16 1 0 4 2 24
7 5 3 3 5 23
54 10 10 9 16 99
56 13 5 12 13 99
36 15 15 13 22 101
Class A: People who had received help for emotional problems I & II
III IV&V VI VII & VIII Total/method
13 7 4 4 11 39
35 5 0 0 3 43
12 12 0 7 7 36
12 3 3 1 7 25
19 3 0 0 3 24
0 12 0 0 0 12
20 4 1 4 7 36
Class B: People who had not had problems requiring assistance I & II 25 28 20 19 12 9 10 III 6 7 6 2 5 4 2 IV & V 6 5 9 2 0 3 2 VI 5 7 9 2 2 1 2 VII & VIII 7 9 16 7 3 1 1 Total/method 50 56 60 33 20 18 17
aThese data were obtained from an exemplary question, "To what extent are you aware of the East Alabama Mental Health Center and its services?" The responses are represented on the following 5 point scale: a = not at all; b = less than average; c = average; d = more than average; e = much more than average.
(Class A), an average of 45% of the respondents on both the interview and telephone survey were in the highest SES groups (I & II). However, on the mail survey an average of 75% of the respondents in Class A were in groups I and II. Of those people who had not reported problems requiring assistance (Class B), an average of about 50% of the interviews and 60% of the mail-outs were in groups I and II, while 35% of the telephone survey data were in the same group. In looking at the range of responses across the 5-point scale in Table IV, it appears that SES groups I and II mail-out respondents give more neutral (c) and negative (a and b) responses, especially if they had required assistance. Group I and II respondents on interview and telephone surveys show the opposite trend, endorsing more positive attitudes, especially if they have required assistance. The remaining SES groups are moderately well represented and symmetrically distributed except for the middle SES group (III) and the unemployed/other group (VII and VIII). For telephone surveys under Class A, an obviously high average of about 30% fall into group III. For interviews under Class A, a disparate average of 25% fall into the unemployed/other group.
Comparison of Survey Methods
395
DISCUSSION It seems clear that there are relatively different expenses and response characteristics associated with the three different techniques. Perhaps the clearest finding is that, contrary to popular belief, the mail-out questionnaire is the most costly of the survey methods commonly used. Telephone surveys appear to be the most economical with the cost of door-to-door survey teams falling about midway between. However, these survey teams included two persons each for security precautions and thus almost doubled our total cost for that technique. If security risks for a particular survey area were considered low and utilization of one-person survey teams feasible, the door-to-door survey would be by far the most economical procedure, perhaps more than a dollar less per completed questionnaire than mail surveys. Perhaps the high use of the mail technique by budget-minded surveyors is due to consideration of absolute cost rather than the amount of data received relative to the expenditure. While it is true that more money was spent on the door-to-door interview, we received about 2½ times more data than in the mail questionnaire. In addition to economic costs, it was found that there were emotional costs as well. While the door-to-door teams experienced little emotional discomfort, a low refusal rate (8%), and reported having enjoyed the experience, the opposite was true for the telephone survey. Over one-fifth (21%) of the prospective telephone respondents refused to participate with the survey and hung-up, or expressed hostile and suspicious comments. Of those that did participate, many refused to answer some questions, e.g., approximately 20% refused to report their income. The telephone surveyors reported that this was an unpleasant experience for them, and they disliked the task. On one occasion a male surveyor was so distraught by the experience that he approached one of the researchers and asked to withdraw from the project. While this was an extreme case, it seems to represent the typical emotional effect on many of the phone surveyors. Another difference among the three methods were the characteristics of respondents. While all methods seemed to overrepresent the upper SES (I & II), the mail techniques greatly exaggerated this group, particularly for those who had received services. However, the interview data tended to give a better representation of all the SES groups, especially the lowest group, while the telephone survey tended to overrepresent SES III, especially for those who had experienced personal problems. Although more than 90% of the residences in the survey area have telephones, there may be a systematic bias which excludes upper SES respondents who have an unlisted telephone number, unavailable to the researcher, while the lower SES group may not have a telephone at all. While a random-digit dialing procedure suggested by Gtasser and Metzger (.1972) may serve to eliminate
396
Hinkle and King
the bias introduced by unlisted telephone numbers, it obviously does not solve the problem of excluding those who do not have a phone. Another interesting trend was in the ratings given by upper SES respondents by method. For marl respondents, there was a much greater proportion giving neutral and negative evaluations, especially if they had received assistance for some problem. Thus, a typical respondent on the marl survey seemed to be an upper SES individual who is critical of community services, especially if he has used them to assist him in resolving his personal problems. This suggests that community mental health surveyors who evaluate their program by marl questionnaire are more likely to receive a negative to neutral evaluation than a positive one. Since marl-out is also likely to cost more, with this method one pays more to look worse! Since none of these methods appear to give a stratified sample of the population, an alternate procedure to correct this bias would be to construct a stratified sample from the target population to match known demographic characteristics. For example, extrapolation from th income characteristics of the population according to the U.S. Census income data for this city, with a minimum of n = 10 per cell for statistical considerations, would require the following samples. The interview method would require 1,096 completed questionnaires at a cost of $1,467.42, to provide a data pool from which a stratified sample could be constructed to match the census income population characteristics. Similarly, the marl-out would require a pool of 1,898 completed questionnaires at a cost of $3,462.03, and the telephone survey would require 930 questionnaires at a cost of $851.75. The costs per completed questionnaire are based on data presented in Table II. Thus, to obtain the minimum sample, it is clear that the marl-out technique is exorbitantly costly. While the telephone method is ostensibly most economical, it is worth emphasizing again that the interview team employing only one surveyor costs about half as much as teams of two. In this case, the minimum interview sample collected by interviewers working alone rather than in pairs, would only cost $733.71, a savings of about $100 over the telephone survey. The results of the study clearly demonstrate that choice of survey method should be determined by two primary factors, the cost (total and per completed unit), and the representation of the population by the survey technique employed. Obviously, the marl-out survey technique fails both considerations. It is too expensive and appears to sample only upper SES individuals who express dissatisfaction. Both the telephone and facto-to-face interview conducted by a single interviewer are attractive in terms of financial costs, but the telephone method may be too expensive in terms of emotional cost to the interviewer. Both telephone and interview methods yield comparable data which is more representative of population than that generated by the marl-out survey
Comparison of Survey Methods
397
m e t h o d . It is apparent that either the t e l e p h o n e or interview m e t h o d m a y be used w i t h equal c o n f i d e n c e to assess c o m m u n i t y attitudes, for which the mailo u t survey m e t h o d appears totally i n a d e q u a t e .
REFERENCES Auld, F., Jr. & Meyers, J. K. Contributions to a theory for selecting psychotherapy patients. Journal of Clinical Psychology, 1954, 10, 56-60 Eddy, W. B., Paap, S. M., & Glad, D. D. Solving problems in living: The citizen's viewpoint. MentaIHygiene, 1970, 54(1), 64-71. Glasser, G. J., & Metzger, G. D. Random-digit dialing as a method of telephone sampling. Journal o f Marketing Research, 1972, 9, 59-64. Gursslin, O. R., Hunt, R. G., & Roach, J. L. Social class and the mental health movement. In F. Riessman, J. Cohen, & A. Pearl (Eds.), Mental health o f the poor. New York: Free Press, 1964, 57-67. Hollingshead, A. B., & Redlich, F. C. Social class and mental illness: A community study. New York: John Wiley & Sons, Inc., 1958. Hunt, R. G. Social class and mental illness: Some implications for clinical theory and practice. American Journal o f Psychiatry, 1960, 116, 1065-1069. Imber, S. D., Nash, E. H., & Stone, A. R. Social class and duration of psychotherapy. Journal o f ClinicaI Psychology, 1955, 11, 281-284. Johnson, T. I. Boutwell, C. R., & Hinkle, A. L. CMHC evaluation: A survey of mental health problems and professional services. Evaluation, 1975,2(2), 18-19. Meyers, J. K., & Bean, L. L. A decade later: A follow-up of social classand mental illness. New York: John Wiley & Sons, 1968. Meyers, J. K., & Schaffer, L. Social stratification and psychiatric practice: A study of an outpatient clinic. American Sociological Review, 1954, 19, 307-310. McMahon, J. T. The working class psychiatric patient: A clinical view. In F. Reissman, J. Cohen, & A. Pearl (Eds.), Mental health ofthepoor. New York: Free Press, 1964, 283-302. Plog, S. C. Explanation for a high return rate on a mail questionnaire. Public Opinion Quarterly, 1963, 27(2), 197-298. Scherl, D. J. The community mental health center and mental health services for the poor. In H. Grunebaum (Ed.), The practice o f community mental health. Boston: Little, Brown, & Co., 1970. Weaver, C. N., Holmes, S. L., & Glenn, N. D. Some characteristics of inaccessible respondents in a telephone survey. Journal of Applied Psychology, 1975, 60, 260-262. Winder, A. E., & Hersko, M. The effect of social class on the length and type of psychotherapy in a Veterans Administration mental hygiene clinic. Journal o f Clinical Psychology, 1955, 11, 77-79.
