AMERICAN JOURNAL or EPIDEMIOLOGY
Copyright © 1979 by The Johns Hopkins Univereity School of Hygiene and Public Health All rights reserved
Vol. 109, No. 1 Printed in
ASSURING THE QUALITY OF QUESTIONNAIRE DATA IN EPIDEMIOLOGIC RESEARCH LEON GORDIS
A major challenge to epidemiologic research today is to assure the quality of the "raw data" which are generated for analysis in our studies. A serious potential hazard confronting us is that, as epidemiologists, we may become so enamored of increasingly sophisticated statistical techniques and data processing capabilities that we may pay inadequate attention to the quality of the data obtained in our investigations—data which, in fact, become the raw material for these statistical and data processing methods. For no scientific discipline can be any better than the quality of its raw data. Improvements in study design or in analytic techniques cannot compensate for data of questionable quality generated by epidemiologic investigations. Unfortunately, we have often paid too little attention to the quality of epidemiologic data, and particularly those obtained through interviews and questionnaires. For example, one has only to contrast the situations in epidemiology and in biochemistry. In general, in a biochemical research paper, the "Materials and Methods" section will describe all details of the techniques used, such as the enzyme preparation, substrate, buffer, centrifugationa, homogenizations and specific assays employed. As a result, other investigators who wish to replicate the biochemical experiment can, in principle, apply the same methods and attempt to confirm the findings reported.
This is generally not the situation, however, with epidemiologic investigations which utilize interviews or selfadministered questionnaires. Interview or questionnaire instruments are rarely subjected to peer review and often are not available for use by others for replication in different populations. The instrument is generally not available at the time a grant application is submitted and, indeed, often the application itself quite reasonably includes a request for the funds needed to develop and pilot test the instrument. Consequently, there is no peer review of the instrument at this point. After the work has been completed and the paper is submitted to a reputable peer review journal, the manuscript is generally not accompanied by the instrument, and the description of the instrument in the paper is often extremely sketchy. As a result, the journal's reviewers do not see the instrument, and in their review they are in effect expressing faith in the investigator and in the probable quality of his instrument. Finally, when the paper does appear in print, the interview or questionnaire is generally not published with it because of the expenses involved in printing and publication. Furthermore, even when information regarding the instrument is provided, often the type of training which the interviewers received, or the extent to which the interviewers were instructed to probe in order to elicit a response to a specific question, is not described. The reader is therefore unable to assess the quality of the instrument or of the data derived from it, and he does not have the instrument available for replication if he should so desire.
From the Department of Epidemiology, Johns Hopkins School of Hygiene and Public Health, 615 N. Wolfe St., Baltimore, MD 21205 (send reprint requests to Dr. Gordis at this address). Supported in part by grant #5 T32 HL07024 from the National Heart, Lung and Blood Institute. 21
22
LEON GORDIS
The issue of the validity of interview or questionnaire data is a difficult one. hi speaking of more "sensitive" or "specific" instruments there is an implication that "truth" is known by some other independent means. This is generally, however, not the case. For certain variables, such as occupation, it may be possible to validate the histories obtained, but for many variables this may not be so. However, small-scale pilot studies to validate interviews may be possible even for other types of variables. The findings from such exploratory studies could then be generalized to other studies in which the same interview instrument is used in different populations. In any event, there is a need to develop further the techniques for validating interview and questionnaire instruments. In the interim, however, we should at least be concerned with the extent to which various instruments are comparable and when there is a lack of comparability, the reasons for this should be explored. If the details of the interview instruments and the ways in which they were administered were available in published papers, we would at least be able to make progress in addressing the issue of instrument comparability and ultimately deal with that of validity itself. As epidemiologists, most of us have encountered these problems in many of our own studies in relation to a number of different categories of data obtained in interviews and questionnaires. Many of us often devise our own questions and formats regarding diet, occupation and even histories of smoking and of radiation exposure. Specifically, in the case of smoking, where countless interviews have dealt with this subject, it is surprising that after so many years epidemiologists have not arrived at a consensus of what the best instrument may be for obtaining sensitive and specific information regarding smoking history. There is also tremendous variability in questions about
occupation, ranging from open-ended questions regarding what occupations individuals may have had and ranging to the other extreme of a detailed listing both of occupations and of specific chemical or other occupational exposures. While dietary histories are notoriously difficult to obtain, we are unlikely to make significant progress in this area if each investigator asks the questions in his own way and does not at least compare the responses he obtains with those obtained by alternate means. It is interesting to consider, as an example, several reported investigations of the relationship of bladder cancer and artificial sweeteners. A valid history of intake of artificial sweeteners is probably extremely difficult to elicit since these substances are present in so many foods, drugs, cosmetics and other products that the respondents themselves may not even be aware that they have used these substances. In addition, even if they are aware, they may very well not know whether the substance was cyclamate or saccharin and may be unable to give a quantitative estimate of their intake. However, these problems in eliciting valid histories are further compounded by the variability in the interview instruments used in different studies and the impossibility of assessing the comparability of these studies because of the lack of sufficient information in published papers regarding the instruments employed. For example, in one study of bladder cancer mortality, the complete description of the method used for ascertaining saccharin consumption was: The questionnaire covered the present and past use of artificial sweeteners: in tablet or liquid form added to food during meal preparation in already prepared low-calorie foods in conventional and low calorie soft drinks.
Another paper presented a somewhat more detailed methodologic description:
ASSURING THE QUALITY OF QUESTIONNAIRE DATA
23
iables has by now been sought often enough in epidemiologic interviews that sufficient experience has been generated to warrant some form of standardization of the questions. Among these variables are many components of past medical history such as childhood diseases, past surgical procedures and other hospitalizations, residential mobility, occupational history, including places of employment, In another study, the description was as specific radiation or chemical exposures, follows: and history of pet ownership, to name but a few. . . . the patients were subjected to intensive personal interview with respect to their occuOther areas such as dietary histories pational history, smoking habits, and use of and histories of alcohol intake still reartificial sweeteners. Non-nutritive sweetener intake was probed in terms of: 1) quire considerable work. However, progtable sweeteners, 2) diet beverages, 3) diet ress will not be made in these areas if as foods and 4) total intake in all forms. For each investigators we continue to generate our specific, non-nutritive-sweetener-containing substance, information was sought on the freown interviews de novo without attemptquency, quality and duration of use. ing to compare different methods of data These three citations do not, of course, collection. The fact that we may not have reflect in any way on the methodologic the optimal method for obtaining such insoundness of the studies they describe but formation should not justify an individual serve only to demonstrate that the descrip- entrepreneurial approach to gathering tions are not sufficiently detailed to permit these data. Some standardization of inother investigators either to assess the in- terviews and questionnaires is urgently struments used or to replicate these needed. While investigators should have the flexibility to deviate from any stanstudies in other populations. There are, of course, exceptions. For dard which is established, there should be example, Simon et al. (1), in a paper pub- a clear understanding that the point of lished in 1975, presented their actual origin was a standard recommended questionnaire as an appendix to their method; any deviation from the standard paper, and a 1974 paper by Morgan and could then be described and the effects of deviations on the quality and validity Jain (2) made the questionnaire available the of the data collected might be assessed. to readers on request. However, in genA number of steps therefore seem aperal, we are not given the wording of the propriate if we are to improve the quality questions, whether the questions were of the data we collect through interviews open- or closed-ended, and the amount of and questionnaires: detail in the questions. In most instances, 1. The relevant parts of the interview therefore, it is not possible to discuss the significance of the differences found by instrument including the way it was addifferent investigators in regard to a rela- ministered should be published with the tionship of artificial sweeteners to blad- paper as an appendix. This will generally der cancer because any differences could involve publishing only a few questions in actuality reflect differences in the in- from the instrument with any given struments used rather than differences in paper. 2. If, for financial or other reasons, step the actual consumption of artificial 1 is not possible, at the very least the relsweeteners. evant parts of the interview or questionInformation regarding a number of varTo test the hypothesis, based on animal data, that use of both saccharin and cyclamatea might increase risk of bladder cancer, three questions were asked. The first was: Do you now or have you ever used sugar substitutes?" If "yes", the number of tablets or drops usually used and the frequency and duration of using that brand was determined for each brand or type used. The second and third questions related to similar data for the use of diet drinks and for dietetic foods such as puddings, salad dressings and confectionery.
24
LEONGORDIS
naire instrument should be made available upon request and should be included with reprints when they are sent out. 3. The Society for Epidemiologic Research or some other comparable organization should establish a bank of interviews and questionnaires (perhaps in microfilm form) so that they can subsequently be retrieved for validation or replication. 4. A committee should be set up by the Society for Epidemiologic Research or some other organization to explore the feasibility of establishing recommended methods of obtaining information on any variables for which satisfactory validity and reliability data exist or can be obtained and should specify the procedures which should be employed by different investigators for modifying these instruments as needed and for assessing the effects of such modifications on the validity of the information obtained. In summary, as epidemiologists, we must confront the urgent issue of the quality of the data we obtain in our research. While more sophisticated techniques for data analysis and processing will undoubtedly be developed in the
future, and are awaited with great interest, the progress of epidemiology in coming years will probably depend more on our giving careful attention to improving the quality of the data which are generated in these studies. While this discussion has dealt specifically with questionnaire data, many of the issues raised also apply to data obtained from different types of examinations. Epidemiologic investigations are rapidly expanding into new areas. With the increased involvement of a variety of federal regulatory agencies in occupational and environmental epidemiology, epidemiologic data are increasingly serving as the basis for health policy decisions and the setting of environmental standards. Consequently, the need for a critical examination of the quality of epidemiologic data and the methods which can be used to improve the validity of these data is particularly urgent. REFERENCES
1. Simon D, Yen S, Cole P: Coffee drinking and cancer of the lower urinary tract. J Natl Cancer Inst 54:587-591, 1975 2. Morgan RW, Jain MG: Bladder Cancer: Smoking, beverages and artificial sweeteners. Can MedAssoc J 111:1067-1070, 1974
Copyright © 1979 by The Johns Hopkins Univereity School of Hygiene and Public Health All rights reserved
Vol. 109, No. 1 Printed in
ASSURING THE QUALITY OF QUESTIONNAIRE DATA IN EPIDEMIOLOGIC RESEARCH LEON GORDIS
A major challenge to epidemiologic research today is to assure the quality of the "raw data" which are generated for analysis in our studies. A serious potential hazard confronting us is that, as epidemiologists, we may become so enamored of increasingly sophisticated statistical techniques and data processing capabilities that we may pay inadequate attention to the quality of the data obtained in our investigations—data which, in fact, become the raw material for these statistical and data processing methods. For no scientific discipline can be any better than the quality of its raw data. Improvements in study design or in analytic techniques cannot compensate for data of questionable quality generated by epidemiologic investigations. Unfortunately, we have often paid too little attention to the quality of epidemiologic data, and particularly those obtained through interviews and questionnaires. For example, one has only to contrast the situations in epidemiology and in biochemistry. In general, in a biochemical research paper, the "Materials and Methods" section will describe all details of the techniques used, such as the enzyme preparation, substrate, buffer, centrifugationa, homogenizations and specific assays employed. As a result, other investigators who wish to replicate the biochemical experiment can, in principle, apply the same methods and attempt to confirm the findings reported.
This is generally not the situation, however, with epidemiologic investigations which utilize interviews or selfadministered questionnaires. Interview or questionnaire instruments are rarely subjected to peer review and often are not available for use by others for replication in different populations. The instrument is generally not available at the time a grant application is submitted and, indeed, often the application itself quite reasonably includes a request for the funds needed to develop and pilot test the instrument. Consequently, there is no peer review of the instrument at this point. After the work has been completed and the paper is submitted to a reputable peer review journal, the manuscript is generally not accompanied by the instrument, and the description of the instrument in the paper is often extremely sketchy. As a result, the journal's reviewers do not see the instrument, and in their review they are in effect expressing faith in the investigator and in the probable quality of his instrument. Finally, when the paper does appear in print, the interview or questionnaire is generally not published with it because of the expenses involved in printing and publication. Furthermore, even when information regarding the instrument is provided, often the type of training which the interviewers received, or the extent to which the interviewers were instructed to probe in order to elicit a response to a specific question, is not described. The reader is therefore unable to assess the quality of the instrument or of the data derived from it, and he does not have the instrument available for replication if he should so desire.
From the Department of Epidemiology, Johns Hopkins School of Hygiene and Public Health, 615 N. Wolfe St., Baltimore, MD 21205 (send reprint requests to Dr. Gordis at this address). Supported in part by grant #5 T32 HL07024 from the National Heart, Lung and Blood Institute. 21
22
LEON GORDIS
The issue of the validity of interview or questionnaire data is a difficult one. hi speaking of more "sensitive" or "specific" instruments there is an implication that "truth" is known by some other independent means. This is generally, however, not the case. For certain variables, such as occupation, it may be possible to validate the histories obtained, but for many variables this may not be so. However, small-scale pilot studies to validate interviews may be possible even for other types of variables. The findings from such exploratory studies could then be generalized to other studies in which the same interview instrument is used in different populations. In any event, there is a need to develop further the techniques for validating interview and questionnaire instruments. In the interim, however, we should at least be concerned with the extent to which various instruments are comparable and when there is a lack of comparability, the reasons for this should be explored. If the details of the interview instruments and the ways in which they were administered were available in published papers, we would at least be able to make progress in addressing the issue of instrument comparability and ultimately deal with that of validity itself. As epidemiologists, most of us have encountered these problems in many of our own studies in relation to a number of different categories of data obtained in interviews and questionnaires. Many of us often devise our own questions and formats regarding diet, occupation and even histories of smoking and of radiation exposure. Specifically, in the case of smoking, where countless interviews have dealt with this subject, it is surprising that after so many years epidemiologists have not arrived at a consensus of what the best instrument may be for obtaining sensitive and specific information regarding smoking history. There is also tremendous variability in questions about
occupation, ranging from open-ended questions regarding what occupations individuals may have had and ranging to the other extreme of a detailed listing both of occupations and of specific chemical or other occupational exposures. While dietary histories are notoriously difficult to obtain, we are unlikely to make significant progress in this area if each investigator asks the questions in his own way and does not at least compare the responses he obtains with those obtained by alternate means. It is interesting to consider, as an example, several reported investigations of the relationship of bladder cancer and artificial sweeteners. A valid history of intake of artificial sweeteners is probably extremely difficult to elicit since these substances are present in so many foods, drugs, cosmetics and other products that the respondents themselves may not even be aware that they have used these substances. In addition, even if they are aware, they may very well not know whether the substance was cyclamate or saccharin and may be unable to give a quantitative estimate of their intake. However, these problems in eliciting valid histories are further compounded by the variability in the interview instruments used in different studies and the impossibility of assessing the comparability of these studies because of the lack of sufficient information in published papers regarding the instruments employed. For example, in one study of bladder cancer mortality, the complete description of the method used for ascertaining saccharin consumption was: The questionnaire covered the present and past use of artificial sweeteners: in tablet or liquid form added to food during meal preparation in already prepared low-calorie foods in conventional and low calorie soft drinks.
Another paper presented a somewhat more detailed methodologic description:
ASSURING THE QUALITY OF QUESTIONNAIRE DATA
23
iables has by now been sought often enough in epidemiologic interviews that sufficient experience has been generated to warrant some form of standardization of the questions. Among these variables are many components of past medical history such as childhood diseases, past surgical procedures and other hospitalizations, residential mobility, occupational history, including places of employment, In another study, the description was as specific radiation or chemical exposures, follows: and history of pet ownership, to name but a few. . . . the patients were subjected to intensive personal interview with respect to their occuOther areas such as dietary histories pational history, smoking habits, and use of and histories of alcohol intake still reartificial sweeteners. Non-nutritive sweetener intake was probed in terms of: 1) quire considerable work. However, progtable sweeteners, 2) diet beverages, 3) diet ress will not be made in these areas if as foods and 4) total intake in all forms. For each investigators we continue to generate our specific, non-nutritive-sweetener-containing substance, information was sought on the freown interviews de novo without attemptquency, quality and duration of use. ing to compare different methods of data These three citations do not, of course, collection. The fact that we may not have reflect in any way on the methodologic the optimal method for obtaining such insoundness of the studies they describe but formation should not justify an individual serve only to demonstrate that the descrip- entrepreneurial approach to gathering tions are not sufficiently detailed to permit these data. Some standardization of inother investigators either to assess the in- terviews and questionnaires is urgently struments used or to replicate these needed. While investigators should have the flexibility to deviate from any stanstudies in other populations. There are, of course, exceptions. For dard which is established, there should be example, Simon et al. (1), in a paper pub- a clear understanding that the point of lished in 1975, presented their actual origin was a standard recommended questionnaire as an appendix to their method; any deviation from the standard paper, and a 1974 paper by Morgan and could then be described and the effects of deviations on the quality and validity Jain (2) made the questionnaire available the of the data collected might be assessed. to readers on request. However, in genA number of steps therefore seem aperal, we are not given the wording of the propriate if we are to improve the quality questions, whether the questions were of the data we collect through interviews open- or closed-ended, and the amount of and questionnaires: detail in the questions. In most instances, 1. The relevant parts of the interview therefore, it is not possible to discuss the significance of the differences found by instrument including the way it was addifferent investigators in regard to a rela- ministered should be published with the tionship of artificial sweeteners to blad- paper as an appendix. This will generally der cancer because any differences could involve publishing only a few questions in actuality reflect differences in the in- from the instrument with any given struments used rather than differences in paper. 2. If, for financial or other reasons, step the actual consumption of artificial 1 is not possible, at the very least the relsweeteners. evant parts of the interview or questionInformation regarding a number of varTo test the hypothesis, based on animal data, that use of both saccharin and cyclamatea might increase risk of bladder cancer, three questions were asked. The first was: Do you now or have you ever used sugar substitutes?" If "yes", the number of tablets or drops usually used and the frequency and duration of using that brand was determined for each brand or type used. The second and third questions related to similar data for the use of diet drinks and for dietetic foods such as puddings, salad dressings and confectionery.
24
LEONGORDIS
naire instrument should be made available upon request and should be included with reprints when they are sent out. 3. The Society for Epidemiologic Research or some other comparable organization should establish a bank of interviews and questionnaires (perhaps in microfilm form) so that they can subsequently be retrieved for validation or replication. 4. A committee should be set up by the Society for Epidemiologic Research or some other organization to explore the feasibility of establishing recommended methods of obtaining information on any variables for which satisfactory validity and reliability data exist or can be obtained and should specify the procedures which should be employed by different investigators for modifying these instruments as needed and for assessing the effects of such modifications on the validity of the information obtained. In summary, as epidemiologists, we must confront the urgent issue of the quality of the data we obtain in our research. While more sophisticated techniques for data analysis and processing will undoubtedly be developed in the
future, and are awaited with great interest, the progress of epidemiology in coming years will probably depend more on our giving careful attention to improving the quality of the data which are generated in these studies. While this discussion has dealt specifically with questionnaire data, many of the issues raised also apply to data obtained from different types of examinations. Epidemiologic investigations are rapidly expanding into new areas. With the increased involvement of a variety of federal regulatory agencies in occupational and environmental epidemiology, epidemiologic data are increasingly serving as the basis for health policy decisions and the setting of environmental standards. Consequently, the need for a critical examination of the quality of epidemiologic data and the methods which can be used to improve the validity of these data is particularly urgent. REFERENCES
1. Simon D, Yen S, Cole P: Coffee drinking and cancer of the lower urinary tract. J Natl Cancer Inst 54:587-591, 1975 2. Morgan RW, Jain MG: Bladder Cancer: Smoking, beverages and artificial sweeteners. Can MedAssoc J 111:1067-1070, 1974
