European Journal of Radiology, 10 (1990) 92-97 Elsevier
92
EURRAD
000 18
Publications on clinical trials with X-ray contrast media: differences in quality between journals and decades E. Andrew, H. Eide, P. Fuglerud, E.K. Hagen, D.T. Kristoffersen, M. Lambrechts, A. Waaler and M. Weibye Research and DevelopmentDivision,Nycomed A.S., Oslo, Norway (Received
1 September
1989; accepted
12 October
Key words: Clinical trial; Contrast media; Reporting standard;
1989)
Clinical trial, methodology
Abstract The objective of the present study was to investigate the quality of clinical trial publications on X-ray contrast media by use of a simple criteria list with 11 items. The publication quality in the 196Os, 1970s and 1980s and in five radiological journals was compared. One hundred and three articles retrieved from the literature and published in Br J Radio1 (British), Acta Radio1 (Scandinavian), Radiology (American), RBFo (German) and in Ann Radio1 (French) were finally included. The adapted method seemed to be suitable for roughly assessing the quality of contrast medium publications. The present reporting standard has increased considerably since the 196Os, however a higher standard is still needed. Although the limited material gathered in our investigation does not allow unequivocal statements, the results indicate that the reporting standard of the 1980s in the selected American, British and Scandinavian radiological journals was somewhat better than in the German journal and better than in the French journal. Use of the present or other assessment methods is one tool to improve the reporting standard.
Introduction Use of highly scientific methodology is a prerequisite for obtaining reliability in published clinical trials. Although the general principles for conducting and publishing clinical trials have been broadly accepted for years [7,9], the results produced rather than the methodology behind the trials are usually focused. The publication is an appropriate means for other researchers to evaluate and analyze the standard of the trial. By standard we mean the scientific quality of the presentation in the publication of aim, design, materials and methods, results and conclusion. It is this written information which forms the platform for future assessment and utilization of a trial. Since Badgley in 1961 published his assessment of the research methods in 103 biomedical articles [3], various procedures and systems for evaluating published data are reported (for an overview see Ref. 15). Address for reprints: Erik Andrew, Ph.D. Clinical R & D Nycomed Imaging, 0401 Oslo 4, Norway. 0720-048X/90/$03.50
0 1990 Elsevier Science Publishers
These approaches of critical assessment of publications usually comprise checklists, series of questions, or eligible features for characterization of the methodological strength. A simple method for evaluating the reporting standard of clinical trials with X-ray contrast media was developed by Andrew [ 11. As clinical research coordinators and statisticians working with contrast medium trials, we used a unsignificantly revised version of this method for roughly assessing the quality of clinical trial publications on contrast media from the periods 196Os, 197Os, and 1980s. The main purpose of the present work was to study whether any change in quality over the years and between five international radiological journals were indicated. Material and Methods Material We planned to evaluate 150 publications in journals printed in English and German describing clinical trials with contrast media in the period 1960-1985. Publi-
B.V. (Biomedical
Division)
93
cations from the following five journals were randomly selected : Diagnosis (Acta Radiol), Scandinavian “Fortschritte auf dem Gebiet der Rontgenstrahlen und der Nuklearmedizin” (ROFo), German The British Journal of Radiology (Br J Radiol), British “Annales de Radiologie” (Ann Radiol), French Radiology, American
TABLE
II
Characteristics
of the trial publications
- Acta Radiologica
We planned to select about ten articles from each journal in each of the three periods, 1961-1965, 1971-1975 and 1981-1985. From a practical point of view, the aim was to retrieve 150 publications which, after expected exclusions, were regarded as representative of the three decades. Publications on phase II-IV trials in children or adults on water-soluble contrast media for vascular, body cavity and myelographic use were included. Case reports and multinational studies were excluded. The name of the author and journal, year of publication and trial centre were erased from the copies received by the evaluators. The articles chosen were randomly numbered. The publications were selected by literature retrieval through on-line data bases (Biological Abstracts (Biosis), National Library of Medicine (MEDLINE) and Excerpta Medica (EMBASE)). Since few contrast medium articles were retrieved, particularly from Ann Radio1 and RoFo, selection through the tables of contents of the journals was also attempted. As shown in Table I, 103 publications were finally included and assessed. Additionally, one publication originally included for evaluation was excluded from the material due to a complete lack of data regarding the variables to be assessed. Due to a lack of papers satisfying the inclusion/exclusion criteria selected, particularly in 1961-1965, the number of papers aimed at was not TABLE
I
Number of included publications Journal
divided on journal and period
Periods
Total
1980s
1970s
1960s
Acta Radio1 Radiology Br J Radio1 Ann Radio1 RoFo
14 10 11 6 8
10 5 8 0 11
3 8 4 0 5
27 23 23 6 24
Total
49
34
20
103
evaluated
Periods
Total
1980s
1970s
1960s
Trial indication Body cavities Myelography Vascular
1 9 39
1 10 23
0 2 18
2 21 80
Age group per trial” Children only Adults only Both children and adults
7 42 0
1 29 4
1 10 8
9 81 12
Study type” Comparative Non-comparative
37 12
20 13
5 15
62 40
a Information
lacking for one trial each.
achieved. Faulty or doubtful indexing of publications complicating their retrieval through the data bases might be the main reason for the suprisingly low number of publications traced. Trial indication, age group of the patients per trial and study design are stated in Table II. The contrast medium or media used in each trial were also recorded, and 14 well-known ionic and non-ionic media were involved in the 103 publications. Evaluation method A slightly revised version of a previous evaluation method [ l] was used for assessing the trial publications (Table III). This criteria list with 11 items is a simple and practical method for roughly assessing the quality of publications. The evaluation was performed by two groups of three researchers of whom one was a statistician. Each group evaluated one half of the selected publications. To verify the reproducibility of the evaluation procedure, the same five publications were evaluated by both groups before the start of the study. The six evaluators first read and evaluated the five articles alone and then the members of each group met and discussed the results. After discussion, the group gave a common score for each question. The two groups compared their scores and if the deviations between the groups was greater than 30x, five new articles would be evaluated by both groups. This procedure was continued until the deviation was less than 30 %, Each group then evaluated its half of the trial publications selected on their own as described earlier: each member of the group read the publications, then the
94 TABLE III Criteria list for assessing the publications
of contrast medium trials (revised from Ref. 1)
1. Aim of the trial Not defined Not adequately defined Relevant and well defined 2. Study design (type, model, blinding)” Poor (e.g. open, no comparative groups) Inadequate (e.g. comparative, single blind or open) Adequate (comparative, double blind) 3. Randomization” None Inadequate (or no randomization in noncomparative studies) Adequate (e.g., complete, block, stratified) 4. Patient material (size, inclusion/nonentrance criteria, demografic data) Not defined Inadequately defined Well defined 5. Ethics and informed consent Not mentioned Inadequately described Adequately described (ethics evaluated, consent obtained)
0
1 2 0 1 2 0 1 2
0 1 2 0 1 2
6. Patients excluded after entrance Not mentioned Only number of exclusions mentioned Stated with reason for exclusion (or stated no exclusions) I. Contrast media and dosage Poorly described Inadequately described Adequately described
0
1 2 0 1 2
8. Injection and other relevant procedural Poorly described Inadequately described Adequately described
conditions
9. Effect/side-effect parameters recorded Irrelevant or poorly defined Partly relevant and partly defined Relevant and well defined
0 1 2
0 1 2
10. Statistical analysis of the results Statistical analysis wrongly or not performed Doubtful statistics Performed with statements of adequate methods used (if desirable)
0 1 2
11. Conclusion of the publication Wrong or irrelevant Inadequate Adequate
0 1 2
Total score maximum
according to aim and data presented
22
a In a few studies, comparative, double-blind design and/or randomization particulary whether box 1 or 2 should be crossed in the question.
may not be necessary according to the aim of the trial. If so, evaluate
group adjusted their scores to give one common score for each question. When all articles had been evaluated, five randomly selected publications were evaluated by both groups to control the reproducibility. The results from the two groups were finally coded into a computer to perform the statistical analysis before the code was broken. The statistical analysis was performed by a statistician (P.F.) not involved in the evaluation process.
Results
Statistical methods Based on the assumption of normality, analysis of variance (ANOVA) was performed on the total scores obtained from the evaluated publications. For testing the difference between journals, mean values were compared using an F-test. When significant differences were found, the journals were tested in pairs by the Tukey’s studentized range test. In this multiple test situation, the overall level of significance was 0.05. The same method was used for testing differences between periods.
When the two groups of evaluators compared their scores of the first live publications, the difference between the two groups was about 32%) thus more than the 30% demanded for starting the assessment study. However, this criterium was fulfilled when five other publications were evaluated by the two groups. This requirement of methodological validation was also fulfilled when five publications were assessed by both groups at the end of the evaluation process. From a maximum of 22 credits (Table III), the publications representing the 1960s obtained 9.1, those representing the 197Os, 12.3 and those the 198Os, 15.4 credits in a total mean score (Table IV). Publications from Ann Radio1 are not included in Table 4 nor in the statistical analysis, because data were available from the last period only. There is a statistically significant improvement in the mean total score comparing the 1970s to the 1960s and the 1980s to the 1970s. In order to study the difference in scores between the
95 TABLE
Mean total scores
IV
Mean total scores of a maximum of 22 credits obtainable (see Table III) divided on published decade ofthe contrast medium trials independent of the journal Periods
1980s 1970s 1960s
Total score mean
std.
min.
max.
N
15.4 12.3 9.1
3.2 2.4 2.3
5.0 7.0 5.0
21.0 17.0 13.0
49 34 20
decades on the various items in the criteria list, the distribution on 0, 1 and 2 credits are given in Fig. 1. Although great improvement over decades is obtained concerning the information about ethics and informed consent from the first period, this item still scores badly. As seen from Fig. 1, other items showing particular improvement over the decades are description of study design, patient material, statistical analysis and conclusion. Data on exclusions after trial entrance are frequently missing in all three periods. Fig. 2 shows the mean total score for each journal in the three decades. No statistically significant difference in total score is found between the journals published in the 1960s and those published in the 1970s. The mean total scores of the 1980s show a statistically significant difference between Radiology and Ann Radiol, between Br J Radio1 and Ann Radio1 as well as between Acta Radio1 and Ann Radiol. However, if the variable ‘ethics and informed consent’ is omitted from the list of criteria there is no longer a statistically significant difference in mean total score between the journals from this decade. For the four journals covered in all three decades, a
Evaluated item in the criteria list
1980s N = 49
1970s N = 34
1960s N = 20
1) Aim of the trial 2) Study design 3) Randomization 4) Patient material 5) Ethical considerations 6) Patients excluded 7) Contrast and dosage 8) Injection/procedure 9) Parameter description 10) Statistical analysis 11) Conclusion wTz_=
Fig. 1. Number of the 103 publications on contrast medium trials given various scores (credits 0, 1 or 2) broken down into each item in the criteria list (Table III) for each decade.
1970s
1960s
1980s
Periods
Fig. 2. Mean total scores (maximum obtainable 22 credits) for 103 contrast medium trials published in five radiological journals in three decades.
definite improvement the early 1960s.
in quality has taken place since
Discussion Quality of a publication is a relative concept and the adapted criteria list is a tool to obtain a certain objectivity. Although somewhat simple and rough, the method seemed to imply a reasonable attempt at a compromise between validity and ease of use [ 1,2]. More detailed and rigorous methods for analysing the quality of X-ray contrast medium trials are also desirable, such as those of Powe et al. [ 131 and Kinnison et al. [lo] published after we had practically completed our assessment. A recent commentary by Wolf [ 171 provides an excellent scientific critique of methodological issues concerning manuscript writing. In the present work, we also found the method suitable and sufficiently reliable when two separate groups each assessed halfof the publications. A prerequisite for such a procedure is a large extent of co-ordination and common reference values between the two evaluating groups. However, this agreement must be checked properly as in our case, before adapting and treating separate evaluations together. Without using such internal controls, we would recommend that all evaluating groups (evaluators) assess all reports to be compared. Surprisingly few contrast medium publications fultilling our inclusion criteria were traced, particularly in the early 1960s (Table I). As frequently experienced also in clinical trials of contrast media, the numbers finally included were in practice less than we had planned. The reduction or absence of publications from some journals, affects our conclusion only to a limited extent. As seen from Table II, indication and age group
96
per trial show a similar pattern for all decades. The variable distribution of study type between the three decades, or the expected shift in preference from noncomparative to comparative trials during the time period covered, have little influence on the main conclusions. In the present investigations, an increase in mean total scores from 9.1 in the 1960s to 15.4 in the 1980s (Table IV) shows a marked improvement of the publication quality over the years. Such quality improvements with time were also found when the reports from the iohexol clinical trial programme were compared to the metrizamide trial carried out about 5 years previously [ 21. These results are in contrast to the findings of Fletcher and Fletcher (1979) showing that the frequency of published therapeutic trials with poor research design had increased in medical journals over the past 30 years [6]. However, according to Temple [ 161 and Hemminki [8] the standard of the published drug trials have increased during the last decades. The present method is more appropriate for relative rather than absolute evaluations [ 11. However, the total mean scores in the present investigation from 9.1 to 15.4 (41-70% of the maximum possible score) were lower than those previously obtained with this method [2]. When Powe et al. [ 131 assessed the quality of 100 randomized controlled trials of contrast media using a more detailed but more cumbersome method than ours, the mean quality score was 39% of the maximum possible score. Using the value system of less than 11 credits per publication regarded as poor, between 11 and 16 as fair and above 16 credits as a high standard [2], the quality of the evaluated publications was rather disappointing, also for the 1980s which reached 15.4 credits. These results indicate that quality improvement is strongly required, as is also emphasized by wolf [ 171. No statistically significant difference in the total scores was seen between the four journals with papers from the 1960s and the 1970s (Fig. 2), while significantly better quality was found for publications in the 1980s from Radiology, Br J Radio1 and Acta Radio1 than those from Ann Radiol. These findings may reflect the difference in clinical trial documentation between the French and Anglo-American culture. As seen from Fig. 2, American-based Radiology seems to have had the most beneficial improvement over the years. However, no ditference in reporting standard was found between the American, British and Scandinavian journals. The improvements in score are most pronounced for the item ‘ethics and informed consent’, but, also study
design, patient material and statistical analysis of the results show marked improvements (Fig. 1). However, the items ‘ethics and informed consent’ and ‘patient excluded after entrance’ are still scoring very low for the 1980s’ publications. Although experienced previously [ 21, these findings are somewhat depressing, since ethical considerations to be included are very strongly stressed in the requirements to publications [4,7,9] and exlusions/drop outs have to be accounted for in detail [2,15]. The fact that ethical and statistical statements are still lacking in the publications from the 198Os, indicate that implementation of changes in scientific and editorial practise need time. The most crucial point in a publication is to present a correct conclusion. Conclusion is part of the abstract, the only part of a publication frequently retrieved and/or read. Use of improved and standardized trial plans for contrast medium trials is one means of raising the trial quality [ 3,5,14]. More professional practical conductance of the trials is also necessary [ 31. Presentation of critically written comments on published investigations [ 111 and letters to the Editor are other tools which can be used to improve the reporting standard. Nevertheless, it is usually the written conclusions in the publication which are later utilized and referred to. In the present assessment, the item ‘conclusion according to aim and results’ obtained better scores in the last two decades than in the 1960s. In our judgement, a still higher score is desired before this item reaches the scientific standard necessary for credible documentation for the 1990s. Although not assessed in detail, we feel that the conclusion in comparative studies often is, consciously or unconsciously, too much in favor of the newest contrast medium. In general, the reporting of the clinical trials appears to be biased towards an exaggaration of treatment differences [ 121. To conclude, the adapted procedure seemed to be suitable for roughly assessing the quality of contrast medium publications. The present reporting standard has increased considerably from the 1960s. Although the limited material gathered in our investigations does not allow unequivocal statements, the results indicate that the reporting standard in the American, British and Scandinavian radiological journals selected was somewhat better than in the German journal and better than in the French journal. However, a higher quality of the published literature in general is required to bring the radiological publications up to an acceptable scientific standard. Use of the present or a similar evaluation method is one tool which can be used to improve the reporting standard.
97
References 1 Andrew E. Method for assessment of the reporting standard of clinical trails with roentgen contrast media. Acta Radio1 Diagn 1984; 25: 55-88. 2 Andrew E. Evaluation of the vascular clinical trial program of metrizamide and iohexol. Acta Radio1 Diagn 1984; 25: 59-64. 3 Badgley RF. An assessment of research methods reported in 103 scientific articles from two Canadian medical journals. Can Med Ass J 1965; 85: 246-250. 4 Barni M. Ethical and medical legal problems in clinical research of contrast media. Radio1 Med (Suppl No 2) 1986; 72: 65-68. 5 Belloni G, Cerutti R, Fiori MG, Moschini L, Porta M. Clinical trials of contrast media in radiology (neuroradiology). A methodological proposal. Radio1 Med 1986; 72: 72-76. 6 Fletcher RH, Fletcher SW. Clinical research in general medical journals. New Engl J Med 1979; 301: 180-183. 7 Friedman LM, Furberg CD, DeMets DL. Fundamentals of clinical trials. John Wright, Boston, 1981. 8 Hemminki E. Quality for clinical trials. A concern of three decades. Methods Inform Med 1982; 21: 81-82.
9 International Committee of Medical Journal Editors. Uniform requirements for manuscripts submitted to biomedical journals. Br Med J 1988; 296: 401-405. 10 Kinnison ML, Powe NR, Steinberg EP. Results of randomized controlled trials of low-versus-high osmolality contrast media. Radiology 1989; 170: 381-389. 11 Moss AA. Critical reviews. Invest Radio1 1988; 23: 327-330. 12 Pocock SJ, Hughes MD, Lee RJ. Statistical problems in reporting of clinical trials. New Eng J Med 1987; 317: 426-432. 13 Powe NR, Kinnison ML, Steinberg EP. Quality assessment of randomized controlled trials of contrast media. Radiology 1989; 170: 377-380. 14 Sovak M. Towards uniform clinical testing of new contrast media. Radio1 Med (Suppl. No 2) 1986; 72: 37-39. 15 Spilker B. Guide to clinical interpretation of data. Section V. Publishing clinical data and evaluation of published literature. Raven Press, New York, 1986. 16 Temple R. Goverment viewpoint of clinical trials. Drug Information Journal Jan/June 1982; 10. 17 Wolf GL. Poor quality of clinical research in radiology: another indictment. Radiology 1989; 170: 311-312.
92
EURRAD
000 18
Publications on clinical trials with X-ray contrast media: differences in quality between journals and decades E. Andrew, H. Eide, P. Fuglerud, E.K. Hagen, D.T. Kristoffersen, M. Lambrechts, A. Waaler and M. Weibye Research and DevelopmentDivision,Nycomed A.S., Oslo, Norway (Received
1 September
1989; accepted
12 October
Key words: Clinical trial; Contrast media; Reporting standard;
1989)
Clinical trial, methodology
Abstract The objective of the present study was to investigate the quality of clinical trial publications on X-ray contrast media by use of a simple criteria list with 11 items. The publication quality in the 196Os, 1970s and 1980s and in five radiological journals was compared. One hundred and three articles retrieved from the literature and published in Br J Radio1 (British), Acta Radio1 (Scandinavian), Radiology (American), RBFo (German) and in Ann Radio1 (French) were finally included. The adapted method seemed to be suitable for roughly assessing the quality of contrast medium publications. The present reporting standard has increased considerably since the 196Os, however a higher standard is still needed. Although the limited material gathered in our investigation does not allow unequivocal statements, the results indicate that the reporting standard of the 1980s in the selected American, British and Scandinavian radiological journals was somewhat better than in the German journal and better than in the French journal. Use of the present or other assessment methods is one tool to improve the reporting standard.
Introduction Use of highly scientific methodology is a prerequisite for obtaining reliability in published clinical trials. Although the general principles for conducting and publishing clinical trials have been broadly accepted for years [7,9], the results produced rather than the methodology behind the trials are usually focused. The publication is an appropriate means for other researchers to evaluate and analyze the standard of the trial. By standard we mean the scientific quality of the presentation in the publication of aim, design, materials and methods, results and conclusion. It is this written information which forms the platform for future assessment and utilization of a trial. Since Badgley in 1961 published his assessment of the research methods in 103 biomedical articles [3], various procedures and systems for evaluating published data are reported (for an overview see Ref. 15). Address for reprints: Erik Andrew, Ph.D. Clinical R & D Nycomed Imaging, 0401 Oslo 4, Norway. 0720-048X/90/$03.50
0 1990 Elsevier Science Publishers
These approaches of critical assessment of publications usually comprise checklists, series of questions, or eligible features for characterization of the methodological strength. A simple method for evaluating the reporting standard of clinical trials with X-ray contrast media was developed by Andrew [ 11. As clinical research coordinators and statisticians working with contrast medium trials, we used a unsignificantly revised version of this method for roughly assessing the quality of clinical trial publications on contrast media from the periods 196Os, 197Os, and 1980s. The main purpose of the present work was to study whether any change in quality over the years and between five international radiological journals were indicated. Material and Methods Material We planned to evaluate 150 publications in journals printed in English and German describing clinical trials with contrast media in the period 1960-1985. Publi-
B.V. (Biomedical
Division)
93
cations from the following five journals were randomly selected : Diagnosis (Acta Radiol), Scandinavian “Fortschritte auf dem Gebiet der Rontgenstrahlen und der Nuklearmedizin” (ROFo), German The British Journal of Radiology (Br J Radiol), British “Annales de Radiologie” (Ann Radiol), French Radiology, American
TABLE
II
Characteristics
of the trial publications
- Acta Radiologica
We planned to select about ten articles from each journal in each of the three periods, 1961-1965, 1971-1975 and 1981-1985. From a practical point of view, the aim was to retrieve 150 publications which, after expected exclusions, were regarded as representative of the three decades. Publications on phase II-IV trials in children or adults on water-soluble contrast media for vascular, body cavity and myelographic use were included. Case reports and multinational studies were excluded. The name of the author and journal, year of publication and trial centre were erased from the copies received by the evaluators. The articles chosen were randomly numbered. The publications were selected by literature retrieval through on-line data bases (Biological Abstracts (Biosis), National Library of Medicine (MEDLINE) and Excerpta Medica (EMBASE)). Since few contrast medium articles were retrieved, particularly from Ann Radio1 and RoFo, selection through the tables of contents of the journals was also attempted. As shown in Table I, 103 publications were finally included and assessed. Additionally, one publication originally included for evaluation was excluded from the material due to a complete lack of data regarding the variables to be assessed. Due to a lack of papers satisfying the inclusion/exclusion criteria selected, particularly in 1961-1965, the number of papers aimed at was not TABLE
I
Number of included publications Journal
divided on journal and period
Periods
Total
1980s
1970s
1960s
Acta Radio1 Radiology Br J Radio1 Ann Radio1 RoFo
14 10 11 6 8
10 5 8 0 11
3 8 4 0 5
27 23 23 6 24
Total
49
34
20
103
evaluated
Periods
Total
1980s
1970s
1960s
Trial indication Body cavities Myelography Vascular
1 9 39
1 10 23
0 2 18
2 21 80
Age group per trial” Children only Adults only Both children and adults
7 42 0
1 29 4
1 10 8
9 81 12
Study type” Comparative Non-comparative
37 12
20 13
5 15
62 40
a Information
lacking for one trial each.
achieved. Faulty or doubtful indexing of publications complicating their retrieval through the data bases might be the main reason for the suprisingly low number of publications traced. Trial indication, age group of the patients per trial and study design are stated in Table II. The contrast medium or media used in each trial were also recorded, and 14 well-known ionic and non-ionic media were involved in the 103 publications. Evaluation method A slightly revised version of a previous evaluation method [ l] was used for assessing the trial publications (Table III). This criteria list with 11 items is a simple and practical method for roughly assessing the quality of publications. The evaluation was performed by two groups of three researchers of whom one was a statistician. Each group evaluated one half of the selected publications. To verify the reproducibility of the evaluation procedure, the same five publications were evaluated by both groups before the start of the study. The six evaluators first read and evaluated the five articles alone and then the members of each group met and discussed the results. After discussion, the group gave a common score for each question. The two groups compared their scores and if the deviations between the groups was greater than 30x, five new articles would be evaluated by both groups. This procedure was continued until the deviation was less than 30 %, Each group then evaluated its half of the trial publications selected on their own as described earlier: each member of the group read the publications, then the
94 TABLE III Criteria list for assessing the publications
of contrast medium trials (revised from Ref. 1)
1. Aim of the trial Not defined Not adequately defined Relevant and well defined 2. Study design (type, model, blinding)” Poor (e.g. open, no comparative groups) Inadequate (e.g. comparative, single blind or open) Adequate (comparative, double blind) 3. Randomization” None Inadequate (or no randomization in noncomparative studies) Adequate (e.g., complete, block, stratified) 4. Patient material (size, inclusion/nonentrance criteria, demografic data) Not defined Inadequately defined Well defined 5. Ethics and informed consent Not mentioned Inadequately described Adequately described (ethics evaluated, consent obtained)
0
1 2 0 1 2 0 1 2
0 1 2 0 1 2
6. Patients excluded after entrance Not mentioned Only number of exclusions mentioned Stated with reason for exclusion (or stated no exclusions) I. Contrast media and dosage Poorly described Inadequately described Adequately described
0
1 2 0 1 2
8. Injection and other relevant procedural Poorly described Inadequately described Adequately described
conditions
9. Effect/side-effect parameters recorded Irrelevant or poorly defined Partly relevant and partly defined Relevant and well defined
0 1 2
0 1 2
10. Statistical analysis of the results Statistical analysis wrongly or not performed Doubtful statistics Performed with statements of adequate methods used (if desirable)
0 1 2
11. Conclusion of the publication Wrong or irrelevant Inadequate Adequate
0 1 2
Total score maximum
according to aim and data presented
22
a In a few studies, comparative, double-blind design and/or randomization particulary whether box 1 or 2 should be crossed in the question.
may not be necessary according to the aim of the trial. If so, evaluate
group adjusted their scores to give one common score for each question. When all articles had been evaluated, five randomly selected publications were evaluated by both groups to control the reproducibility. The results from the two groups were finally coded into a computer to perform the statistical analysis before the code was broken. The statistical analysis was performed by a statistician (P.F.) not involved in the evaluation process.
Results
Statistical methods Based on the assumption of normality, analysis of variance (ANOVA) was performed on the total scores obtained from the evaluated publications. For testing the difference between journals, mean values were compared using an F-test. When significant differences were found, the journals were tested in pairs by the Tukey’s studentized range test. In this multiple test situation, the overall level of significance was 0.05. The same method was used for testing differences between periods.
When the two groups of evaluators compared their scores of the first live publications, the difference between the two groups was about 32%) thus more than the 30% demanded for starting the assessment study. However, this criterium was fulfilled when five other publications were evaluated by the two groups. This requirement of methodological validation was also fulfilled when five publications were assessed by both groups at the end of the evaluation process. From a maximum of 22 credits (Table III), the publications representing the 1960s obtained 9.1, those representing the 197Os, 12.3 and those the 198Os, 15.4 credits in a total mean score (Table IV). Publications from Ann Radio1 are not included in Table 4 nor in the statistical analysis, because data were available from the last period only. There is a statistically significant improvement in the mean total score comparing the 1970s to the 1960s and the 1980s to the 1970s. In order to study the difference in scores between the
95 TABLE
Mean total scores
IV
Mean total scores of a maximum of 22 credits obtainable (see Table III) divided on published decade ofthe contrast medium trials independent of the journal Periods
1980s 1970s 1960s
Total score mean
std.
min.
max.
N
15.4 12.3 9.1
3.2 2.4 2.3
5.0 7.0 5.0
21.0 17.0 13.0
49 34 20
decades on the various items in the criteria list, the distribution on 0, 1 and 2 credits are given in Fig. 1. Although great improvement over decades is obtained concerning the information about ethics and informed consent from the first period, this item still scores badly. As seen from Fig. 1, other items showing particular improvement over the decades are description of study design, patient material, statistical analysis and conclusion. Data on exclusions after trial entrance are frequently missing in all three periods. Fig. 2 shows the mean total score for each journal in the three decades. No statistically significant difference in total score is found between the journals published in the 1960s and those published in the 1970s. The mean total scores of the 1980s show a statistically significant difference between Radiology and Ann Radiol, between Br J Radio1 and Ann Radio1 as well as between Acta Radio1 and Ann Radiol. However, if the variable ‘ethics and informed consent’ is omitted from the list of criteria there is no longer a statistically significant difference in mean total score between the journals from this decade. For the four journals covered in all three decades, a
Evaluated item in the criteria list
1980s N = 49
1970s N = 34
1960s N = 20
1) Aim of the trial 2) Study design 3) Randomization 4) Patient material 5) Ethical considerations 6) Patients excluded 7) Contrast and dosage 8) Injection/procedure 9) Parameter description 10) Statistical analysis 11) Conclusion wTz_=
Fig. 1. Number of the 103 publications on contrast medium trials given various scores (credits 0, 1 or 2) broken down into each item in the criteria list (Table III) for each decade.
1970s
1960s
1980s
Periods
Fig. 2. Mean total scores (maximum obtainable 22 credits) for 103 contrast medium trials published in five radiological journals in three decades.
definite improvement the early 1960s.
in quality has taken place since
Discussion Quality of a publication is a relative concept and the adapted criteria list is a tool to obtain a certain objectivity. Although somewhat simple and rough, the method seemed to imply a reasonable attempt at a compromise between validity and ease of use [ 1,2]. More detailed and rigorous methods for analysing the quality of X-ray contrast medium trials are also desirable, such as those of Powe et al. [ 131 and Kinnison et al. [lo] published after we had practically completed our assessment. A recent commentary by Wolf [ 171 provides an excellent scientific critique of methodological issues concerning manuscript writing. In the present work, we also found the method suitable and sufficiently reliable when two separate groups each assessed halfof the publications. A prerequisite for such a procedure is a large extent of co-ordination and common reference values between the two evaluating groups. However, this agreement must be checked properly as in our case, before adapting and treating separate evaluations together. Without using such internal controls, we would recommend that all evaluating groups (evaluators) assess all reports to be compared. Surprisingly few contrast medium publications fultilling our inclusion criteria were traced, particularly in the early 1960s (Table I). As frequently experienced also in clinical trials of contrast media, the numbers finally included were in practice less than we had planned. The reduction or absence of publications from some journals, affects our conclusion only to a limited extent. As seen from Table II, indication and age group
96
per trial show a similar pattern for all decades. The variable distribution of study type between the three decades, or the expected shift in preference from noncomparative to comparative trials during the time period covered, have little influence on the main conclusions. In the present investigations, an increase in mean total scores from 9.1 in the 1960s to 15.4 in the 1980s (Table IV) shows a marked improvement of the publication quality over the years. Such quality improvements with time were also found when the reports from the iohexol clinical trial programme were compared to the metrizamide trial carried out about 5 years previously [ 21. These results are in contrast to the findings of Fletcher and Fletcher (1979) showing that the frequency of published therapeutic trials with poor research design had increased in medical journals over the past 30 years [6]. However, according to Temple [ 161 and Hemminki [8] the standard of the published drug trials have increased during the last decades. The present method is more appropriate for relative rather than absolute evaluations [ 11. However, the total mean scores in the present investigation from 9.1 to 15.4 (41-70% of the maximum possible score) were lower than those previously obtained with this method [2]. When Powe et al. [ 131 assessed the quality of 100 randomized controlled trials of contrast media using a more detailed but more cumbersome method than ours, the mean quality score was 39% of the maximum possible score. Using the value system of less than 11 credits per publication regarded as poor, between 11 and 16 as fair and above 16 credits as a high standard [2], the quality of the evaluated publications was rather disappointing, also for the 1980s which reached 15.4 credits. These results indicate that quality improvement is strongly required, as is also emphasized by wolf [ 171. No statistically significant difference in the total scores was seen between the four journals with papers from the 1960s and the 1970s (Fig. 2), while significantly better quality was found for publications in the 1980s from Radiology, Br J Radio1 and Acta Radio1 than those from Ann Radiol. These findings may reflect the difference in clinical trial documentation between the French and Anglo-American culture. As seen from Fig. 2, American-based Radiology seems to have had the most beneficial improvement over the years. However, no ditference in reporting standard was found between the American, British and Scandinavian journals. The improvements in score are most pronounced for the item ‘ethics and informed consent’, but, also study
design, patient material and statistical analysis of the results show marked improvements (Fig. 1). However, the items ‘ethics and informed consent’ and ‘patient excluded after entrance’ are still scoring very low for the 1980s’ publications. Although experienced previously [ 21, these findings are somewhat depressing, since ethical considerations to be included are very strongly stressed in the requirements to publications [4,7,9] and exlusions/drop outs have to be accounted for in detail [2,15]. The fact that ethical and statistical statements are still lacking in the publications from the 198Os, indicate that implementation of changes in scientific and editorial practise need time. The most crucial point in a publication is to present a correct conclusion. Conclusion is part of the abstract, the only part of a publication frequently retrieved and/or read. Use of improved and standardized trial plans for contrast medium trials is one means of raising the trial quality [ 3,5,14]. More professional practical conductance of the trials is also necessary [ 31. Presentation of critically written comments on published investigations [ 111 and letters to the Editor are other tools which can be used to improve the reporting standard. Nevertheless, it is usually the written conclusions in the publication which are later utilized and referred to. In the present assessment, the item ‘conclusion according to aim and results’ obtained better scores in the last two decades than in the 1960s. In our judgement, a still higher score is desired before this item reaches the scientific standard necessary for credible documentation for the 1990s. Although not assessed in detail, we feel that the conclusion in comparative studies often is, consciously or unconsciously, too much in favor of the newest contrast medium. In general, the reporting of the clinical trials appears to be biased towards an exaggaration of treatment differences [ 121. To conclude, the adapted procedure seemed to be suitable for roughly assessing the quality of contrast medium publications. The present reporting standard has increased considerably from the 1960s. Although the limited material gathered in our investigations does not allow unequivocal statements, the results indicate that the reporting standard in the American, British and Scandinavian radiological journals selected was somewhat better than in the German journal and better than in the French journal. However, a higher quality of the published literature in general is required to bring the radiological publications up to an acceptable scientific standard. Use of the present or a similar evaluation method is one tool which can be used to improve the reporting standard.
97
References 1 Andrew E. Method for assessment of the reporting standard of clinical trails with roentgen contrast media. Acta Radio1 Diagn 1984; 25: 55-88. 2 Andrew E. Evaluation of the vascular clinical trial program of metrizamide and iohexol. Acta Radio1 Diagn 1984; 25: 59-64. 3 Badgley RF. An assessment of research methods reported in 103 scientific articles from two Canadian medical journals. Can Med Ass J 1965; 85: 246-250. 4 Barni M. Ethical and medical legal problems in clinical research of contrast media. Radio1 Med (Suppl No 2) 1986; 72: 65-68. 5 Belloni G, Cerutti R, Fiori MG, Moschini L, Porta M. Clinical trials of contrast media in radiology (neuroradiology). A methodological proposal. Radio1 Med 1986; 72: 72-76. 6 Fletcher RH, Fletcher SW. Clinical research in general medical journals. New Engl J Med 1979; 301: 180-183. 7 Friedman LM, Furberg CD, DeMets DL. Fundamentals of clinical trials. John Wright, Boston, 1981. 8 Hemminki E. Quality for clinical trials. A concern of three decades. Methods Inform Med 1982; 21: 81-82.
9 International Committee of Medical Journal Editors. Uniform requirements for manuscripts submitted to biomedical journals. Br Med J 1988; 296: 401-405. 10 Kinnison ML, Powe NR, Steinberg EP. Results of randomized controlled trials of low-versus-high osmolality contrast media. Radiology 1989; 170: 381-389. 11 Moss AA. Critical reviews. Invest Radio1 1988; 23: 327-330. 12 Pocock SJ, Hughes MD, Lee RJ. Statistical problems in reporting of clinical trials. New Eng J Med 1987; 317: 426-432. 13 Powe NR, Kinnison ML, Steinberg EP. Quality assessment of randomized controlled trials of contrast media. Radiology 1989; 170: 377-380. 14 Sovak M. Towards uniform clinical testing of new contrast media. Radio1 Med (Suppl. No 2) 1986; 72: 37-39. 15 Spilker B. Guide to clinical interpretation of data. Section V. Publishing clinical data and evaluation of published literature. Raven Press, New York, 1986. 16 Temple R. Goverment viewpoint of clinical trials. Drug Information Journal Jan/June 1982; 10. 17 Wolf GL. Poor quality of clinical research in radiology: another indictment. Radiology 1989; 170: 311-312.
