Postgraduate Medicine

ISSN: 0032-5481 (Print) 1941-9260 (Online) Journal homepage: http://www.tandfonline.com/loi/ipgm20

Verifying the validity of outcome Richard Riegelman To cite this article: Richard Riegelman (1979) Verifying the validity of outcome, Postgraduate Medicine, 65:6, 149-153, DOI: 10.1080/00325481.1979.11715180 To link to this article: http://dx.doi.org/10.1080/00325481.1979.11715180

Published online: 07 Jul 2016.

Submit your article to this journal

View related articles

Citing articles: 2 View citing articles

Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=ipgm20 Download by: [Australian Catholic University]

Date: 12 August 2017, At: 19:10

INTERPRETING MEDICAL STUDIES A Series

Downloaded by [Australian Catholic University] at 19:10 12 August 2017

Verifying the validity of outcome The first article in this series, published in May (page 241 ), dealt with selection bias and described how to tell ifit had crept into a clinical study. This article focuses on the next step in critically evaluating a study: How to assess validity of outcome. Examples illustrate the consequences of failure to fulfill each of the four criteria of valid measurement of outcome.

Richard Riegelman, MD, MPH

To assess validity of outcome means to examine the extent to which a study measures what it purports to measure. In explaining this concept, I will first define the outcome investigators intend to measure in prospective and retrospective studies and then outline the criteria used for validly assessing outcome.

Criteria of valid outcome

What constitutes a valid measure of outcome? Each of the following criteria must be fulfilled: (l) The measure of outcome used must be appropriate to the question to be answered, (2) measurement of outcome must be precise, (3) measurement of outcome must be complete, and (4) the outcome must not be

What is outcome?

Prospective studies begin with a study group which possesses the characteristic under study and a control group which does not. I Individuals in the study group are followed forward in time to determine if the disease under study develops. The disease under study is known as the outcome. The investigators must employ a valid measure of the presence of the disease. Retrospective studies begin with cases in which a certain disease has already developed and a control group in which it has not.2 The investigators review the history of individuals in both case and control groups and determine whether they had similar rates of exposure to or possession of the characteristic under study. In retrospective studies the prior characteristic is the outcome. A valid measure of the prior characteristic must be emphasized.

VOL 65/NO 6/JUNE 1979/POSTGRADUATE MEDICINE

Second of a series of five articles on Interpreting medical studies

influenced by the process of observation. Inappropriate measure of outcome- How use of an inappropriate measure of outcome can invalidate a study is illustrated in the following example. An investigator attempts to study the relationship between urban living conditions and development of active tuberculosis in a country with a high rate of the disease. He selects 500 infants from a poor urban area

and 500 infants from a poor rural area and follows them for ten years. During that time the tuberculin skin test becomes positive in 70% of the urban children, compared with 10% of the rural children. The investigator concludes that urban living is associated with development of active tuberculosis. The tuberculin skin test does not determine the presence of active tuberculosis. Rather, the investigator in this study has assessed the children's exposure to tuberculosis. That is, he has not measured what he intended to measure. The experiment may have established more precise evidence for increased exposure to tuberculosis in urban areas, but since the investigator intended to measure the development of active tuberculosis, he has used an inappropriate measure of outcome. Imprecise measurement-Information for measuring outcome may come from three sources: (I) the study individuals, (2) measurements by the study investigators, and (3) testing instruments. Any of these sources may produce imprecise results. Information from study individuals is subject to recall and reporting errors. 3 Recall errors imply defects in memory, especially defects in which one group is more likely to recall events than is the other group. Reporting errors occur continued 149

MEDICAL STUDIES CONTINUED

Reporting errors are more likely when the nature

of the information sought is confidential and/ or when

Downloaded by [Australian Catholic University] at 19:10 12 August 2017

one group is under more pressure than the other to accurately report events.

when one group of study subjects is more accurate in reporting what they remember than is another. The following example shows how recall errors can occur. In a retrospective study of the cause of spina bifida, 100 mothers of infants born with the disease and 100 mothers of infants born without the disease are studied. Fifty percent of the mothers of infants with spina bifida report having had a sore throat during pregnancy, compared with 5% of the mothers of infants without spina bifida. The investigators conclude that they have shown an association between sore throat during pregnancy and spina bifida. Before accepting the results of this study, one must ask whether recall error could explain the findings. It may be argued that mothers who experience the emotional trauma of having an infant with spina bifida are likely to search their memory intensely and remember events not usually recalled by other women. That is, recall bias is more likely to occur when the events subsequent to a reference point are traumatic and those prior to it are common and easily forgotten. Thus, the results of this study may be attributed to recall error. An error that may result from failure to report what is remembered is illustrated in the next example.

Richard Riegelman Or Riegelman is assistant professor and director of research, department of health care sciences, George Washington University Medical Center, Washington, DC.

150

A study of the relationship between gonorrhea and multiple sexual partners is conducted in which 100 women who have newly diagnosed gonorrhea are compared with 100 women from the same clinical population who are found to be free of gonorrhea. The women with gonorrhea are informed that the serious consequences of the disease may be prevented only by locating and treating sexual partners. Both groups of women are asked about their number of sexual partners during the preceding two months. The gonorrhea group reports an average of two times as many sexual partners as the nongonorrhea group. The investigators conclude that women with gonorrhea have twice as many sexual partners as women without gonorrhea. It may be argued that the women in this study with gonorrhea felt a greater obligation, and hence less hesitation, to report their sexual partners than did the women without the disease. Reporting errors are more likely when the nature of the information sought is confidential and/or when one group is under more pressure to accurately report previous events than is another group. In short, reporting error along with recall error may operate to impair the precision of assessment. In addition to the study subjects, the investigators and the testing instruments may be sources of imprecision. How the latter two can invalidate a study is illustrated in the following example.

A new antiinflammatory drug is evaluated by giving half the members of a rheumatoid arthritis study population the new drug and the other half placebos. The attending physicians prescribing the medications are the only ones who know which patients are receiving the new drug. These same physicians assess the frequency of the side effect of gastritis in both groups, using clinical evaluation and upper gastrointestinal series. They find no difference in the frequency of gastritis between the group receiving the new drug and the group receiving placebos. There are two potential sources of imprecision in this study. First, the physicians knew who received the new drug; yet, they were asked to make a subjective assessment of who had gastritis. Those assessing the outcome should not know who received the medication. This is known as blind assessment. Blind assessment of outcome is critical, especially when subjective symptoms are being evaluated. Second, an upper gastrointestinal series is a poor test instrument for precisely measuring gastritis. Thus, the critical reader must ask whether assessment by the investigators and by the testing instruments provided precise enough measures of the study outcome. Incomplete measurementWhenever follow-up of patients is incomplete, the possibility exists that those not included in the final assessment have a different outcome from that of those included.4 The

VOL 65/NO 6/JUNE 1979/POSTGAADUATE MEDICINE

Downloaded by [Australian Catholic University] at 19:10 12 August 2017

Blind a-ssment of outcome is critical, especially when subjective symptoms are being evaluated.

following example illustrates an error due to incomplete assessment. In a study of the effect ofa new alcohol treatment program, 100 patients are assigned to the new program and 100 patients to conventional treatment. The investigators visit the homes of all the patients at 9:00 on a Saturday night and draw blood for measurement of alcohol levels from all available patients. Among the new treatment group, 30 patients are home and 10 have alcohol in their blood. Among the conventionally treated patients, 40 are home and 20 have alcohol in their blood. The investigators conclude that the new treatment reduced subsequent drinking more than did conventional treatment. In this study, it may be argued that a proportion of those who could not be located were not available because they were out drinking. If this were the case, the results of the study could be dramatically altered. Incomplete follow-up can invalidate Illustration: Niculae Asciu an investigation. Incomplete follow-up does not necessarily mean that the patients having headaches, 90 fatigue, and 60 are lost to follow-up, as in the previ- depression. Among nonpill users, 25 ous example. They may be followed report having headaches, 30 fatigue, with different degrees of intensity, as and 20 depression. The average numthe following example illustrates.s ber of visits for the pill users is three, A study of the side effects of birth compared with one for the nonpill control pills is conducted by com- users. The investigators conclude that paring 1,000 young women taking use of birth control pills is associated the pill with 1,000 young women with an increasedfrequency of headusing other forms of birth control. ache,fatigue, and depression. Data are collected from the records In this study, unequal intensity of of their private physicians for office observation may have invalidated visits during the subsequent year. the results. The fact that pill users Among users of the pill, 75 report made three times as many visits to

VOL 65/NO 6/JUNE 1979/POSTGRADUATE MEDICINE

j

their physicians as did nonpill users may account for the more frequent recordings of headache, fatigue, and depression. Stated simply, the greater the intensity of observation, the more likely are frequently occurring subjective symptoms to be recorded. Process of observation-Even if a study adequately meets the difficult criteria of appropriate, precise, and complete assessment, there is one more threat to its validity. Investigators intend to measure events as they continued 151

PRELUDIN®



Downloaded by [Australian Catholic University] at 19:10 12 August 2017

(phenmetrazine hydrochloride NF) Indication: The drug is indicated in the management of exogenous obesity as a short-term (a few weeks) adjunct in a regimen of weight reduction based on caloric restriction. The limited usefulness of agents of this class should be measured against possible risk factors inherent in their use such as those described below. Contraindications: Advanced arteriosclerosis, symptomatic cardiovascular disease. moderate to severe hypertension. hyperthyroidism. known hypersensitivity or idiosyncrasy to sympathomimetic amines. glaucoma. Agitated states. Patients with a history of drug abuse. Concomitant use of CNS "stimulants. During or within 14 days following the administration of monoamine oxidase inhibitors (hypertensive crises may result). Warnings: Tolerance usually develops within a few weeks. When this occurs. the recommended dose should not be exceeded in an attempt to increase anorectic effect; rather. the drug should be discontinued. The drug may impair ability to engage in potentially hazardous activities such as operating machinery ..or driving a motor vehicle; caution the patient accordingly. Drug Dependence: The drug is related chemically and pharmacologically to the amphetamines. Amphetamines and related stimulant drugs have been extensively abused, and the possibility of abuse of this drug should be kept in mind when considering the desirability of including it as part of a weight reduction program. Abuse of amphetamines and related drugs may be associated with intense psycho·logical dependence and severe social dysfunction. 1"here are reports of patients who have increased the dosage to many times that recommended. Abrupt cessation following prolonged high dosage · administration results in extreme fatigue and mental depression; changes are also noted on the sleep EEG. Manifestations of chronic intoxication with these drugs include severe dermatoses. marked in.somnia, irritability, hyperactivity, and personality changes. The most severe manifestation of chronic intoxication is psychosis often clinically indistinguishable from schizophrenia. Usage in Pregnancy: Safe use in pregnancy has not been established. Animal reproductive studies demonstrated no teratogenic effects. However. the conception rate was adversely affected, as well as survival and body weight of pups at weaning. There have been clinical reports of congenital malformation associated with the use of this compound but a causal relationship has not been proved. Until more information is available. the drug should not be used by women who are or may become pregnant, particularly in the first trimester. unless in ihe opinion of the prescribing physician the potential benefits outweigh the possible risks. Usage in Children: Not recommended for use in children under 12 years of age. Precautions: Caution should be exercised in prescribing this drug for patients with even mild hypertension. Insulin requirements in diabetes mellitus may be altered in association with this drug and the concomitant dietary regimen. The drug may decrease the hypotensive effect of guanethidine. The least amount feasible should be prescribed or dis'pensed at one time in order to minimize the possibility of overdosage. Adverse Reactions: Palpitation. tachycardia, elevation of blood pressure. Overstimulation. restlessness. dizziness. insomnia, euphoria, dysphoria, tremor. headache; rarely psychotic episodes at recommended doses. Dryness of the mouth, unpleasant taste, diarrhea, constipation. other gastrointestinal disturbances. Urticaria. Impotence. changes in libido. Overdosage: Manifestations of acute overdosage with phenmetrazine hydrochloride include restlessness. tremor. hyperreflexia, rapid respiration. confusion, assaultiveness. hallucinations, panic states. Fatigue and depression usually follow the central stimulation. Cardiovascular effects include arrhythmias. hypertension. or hypotension and circulatory collapse. Gastrointestinal symptoms include nausea, vomiting, diarrhea and abdominal cramps. Poisoning may result in convulsions. coma. and death. Management of acute phenmetrazine hydrochloride intoxication is largely symptomatic and includes lavage and sedation with a barbiturate. Experience with hemodialysis or peritoneal dialysis is inadequate to permit recommendation in this regard. As with the amphetamines, acidification of the urine should increase phenmetrazine hydrochloride excretion. Intravenous phentolamine has been suggested for possible acute, severe hypertension if this complicates phenmetrazine overdosage. How Supplied: Tablets of 25 mg. Endurets prolonged-actiOn tablets of 50 mg and 75 mg. For complete details. please see the full prescribing information.

MEDICAL STUDIES CONTINUED

Whenever it is possible for the study subjects to switch groups or alter their behavior, the effects of observation may invalidate an investigation.

would have occurred if no one were watching. Unfortunately the very process of conducting a study involves observation or assessment of events. Hence, in verifying the validity of outcome, the reviewer must assess whether the investigation itself altered the outcome. 6

A study of the relationship between smoking and coronary artery disease is conducted in which 1,000 male smokers aged 50 and 1,000 male nonsmokers aged 50 are followed for ten years. Other risk factors are equally distributed between the two groups. At the end of ten years, there is no difference in the rate of coronary artery disease between the groups. However, on reevaluation it is discovered that 60% of the smokers have stopped smoking. Whenever it is possible for the study subjects to switch groups or alter their behavior, the effects of observation may invalidate an investigation. This is most likely when the study subjects are also aware of the possibility that their behavior may harm their health. In this study the additional element of observation may have tipped the scales, causing smokers to do what they knew they should do anyway.

References I. Ulienfeld AM: Foundations of Epidemiology. New York, Oxford University Press, 1976, pp 194-195 2. Lllienfeld, 1 pp 164-165 3. Mausner JS, Bahn AK: Epidemiology: An Introductory Text. Philadelphia, WB Saunders Co, 1974, p 142

Summary The precarious path toward assessing outcome is full of potholes. Valid assessment of outcome requires that: 1. The investigators have chosen an appropriate measure of outcome, one that measures what they intend to measure. 2. They have performed a precise assessment of outcome. 3. Their measures of outcome have been complete and of equal intensity. 4. The investigation itself has not altered what has occurred; that is, the study results have not been affected by the process of observation. Thus, to verify the validity of outcome the critical reader must decide whether the assessment of outcome has been appropriate, precise, complete, and unaffected by the process of observation. Next: The importance of significance and the significance of importance. Address reprint requests to Richard Riegelman, MD, Department of Health Care Sciences, George Washington University Medical Center, 1229 25th St NW, Washington, DC20037.

4. MacMahon B, Pugh TF: Epidemiology: Principles and Methods. Boston, Little Brown & Co, 1970, pp 229-232 5. Feinstein AR: Clinical Biostatistics. St Louis, CV M os by Co, 1977, p 79 6. Ulienfeld,1 p 2i4

153

Verifying the validity of outcome.

Postgraduate Medicine ISSN: 0032-5481 (Print) 1941-9260 (Online) Journal homepage: http://www.tandfonline.com/loi/ipgm20 Verifying the validity of o...
614KB Sizes 0 Downloads 0 Views