EDITORIAL * EDITORIAL
Citation rate: A measure of excellence? Bruce P. Squires, MD, PhD
C anadian medical scientists must have been astonished by the recent newspaper headline "McMaster tops in study of clinical medical research" (Globe and Mail, Toronto, Dec. 13, 1991: A10). Science reporter Stephen Strauss based his controversial article on information from the Science Indicators Database, 1986-90, of the Institute for Scientific Information (ISI), Philadelphia. From 1986 through 1990 medical researchers at McMaster University reportedly produced 1158 clinical articles, which were subsequently cited 6100 times in other articles (in the nearly 900 clinical medical journals that ISI monitors); the number of citations per paper was 5.27 on average. The next four highest-ranking schools were the University of Manitoba (4.37), the University of Toronto (4.03), the University of British Columbia (3.92) and Queen's University (3.69). The schools traditionally recognized as the leaders in Canadian clinical medical research were surely not pleased. It is foolhardy to base a medical school's overall research excellence solely on an index as crude as the ratio of the number of articles cited to the number published. Too many factors confound interpretation of the ratio. For example, review articles or articles describing methods are more likely to be cited than others. Indeed, an article may be cited often only because other investigators have discovered its weaknesses. A citation rate for an entire medical school may be deceptive because one or two highly cited papers have raised the rate substantially, especially if relatively few papers were produced by the school. (Only five medical schools in Canada published more than 1000 papers in clinical medicine from 1986 through 1990.) Conversely, a centre publishing many papers may have a deceptively low citation rate. For instance, the University of Toronto's clinical medical scientists published 3244 clinical articles from 1986 through 1990, twice-that of the second most productive school. One or two "hot" papers would have not
nearly the same effect on Toronto's citation rate as they would on the rates of less prolific schools. Do one or two bright research stars give a school cause to be ranked higher than a school with many effective, albeit less brilliant, lights? It is equally concerning that the citation rate is based only on citations of articles in the 900 or so journals that ISI monitors.' Significant research may be reported (or cited) in another language or in a narrowly specialized journal not monitored by ISI. Indeed, reports of very specialized research may contribute substantially to scientific knowledge in that field but not be cited often, simply because there are few researchers in the field. Conversely, an article of relatively less significance may be cited often because it is relevant to several disciplines or of interest to readers in many countries. Citation rates for a 5-year period present special problems. Articles published at the beginning of the period are far more likely to have been cited often than those published at the end. Thus, an important recently published paper may have been cited rarely during the period, even though in a few years its citation rate may turn it into a hot paper. Ranking a medical school solely by its citation rate is highly misleading and unnecessarily provocative. One must take other measures into account: research funds generated, types of research, language, breadth of interest in the research, relevance and, most important, quality. As Dr. Janet Halliwell, president of the Science Council of Canada, reportedly told Ottawa science writer Rhonda Birenbaum, "Data is [sic] like a bikini. What it reveals is interesting. What it conceals is often vital."
References 1. Clinical medicine: the top 50 US universities ranked by citation impact, 1986-90 (among those publishing > 1,000 papers). Sci Watch 1991; Nov-Dec: 7 2. Birenbaum R: Behind the data: a look at Canadian science. UnivAffairs 1992: 33 (1): 3-4
Dr. Squires is editor-in-chief and scientific editor ofCMAJ.
Reprint requests to: Dr. Bruce P. Squires, CMAJ, PO Box 8650, Ottawa, ON KIG OG8 FEBRUARY 1, 1992
CAN MED ASSOC J 1992; 146 (3)
341
Citation rate: A measure of excellence? Bruce P. Squires, MD, PhD
C anadian medical scientists must have been astonished by the recent newspaper headline "McMaster tops in study of clinical medical research" (Globe and Mail, Toronto, Dec. 13, 1991: A10). Science reporter Stephen Strauss based his controversial article on information from the Science Indicators Database, 1986-90, of the Institute for Scientific Information (ISI), Philadelphia. From 1986 through 1990 medical researchers at McMaster University reportedly produced 1158 clinical articles, which were subsequently cited 6100 times in other articles (in the nearly 900 clinical medical journals that ISI monitors); the number of citations per paper was 5.27 on average. The next four highest-ranking schools were the University of Manitoba (4.37), the University of Toronto (4.03), the University of British Columbia (3.92) and Queen's University (3.69). The schools traditionally recognized as the leaders in Canadian clinical medical research were surely not pleased. It is foolhardy to base a medical school's overall research excellence solely on an index as crude as the ratio of the number of articles cited to the number published. Too many factors confound interpretation of the ratio. For example, review articles or articles describing methods are more likely to be cited than others. Indeed, an article may be cited often only because other investigators have discovered its weaknesses. A citation rate for an entire medical school may be deceptive because one or two highly cited papers have raised the rate substantially, especially if relatively few papers were produced by the school. (Only five medical schools in Canada published more than 1000 papers in clinical medicine from 1986 through 1990.) Conversely, a centre publishing many papers may have a deceptively low citation rate. For instance, the University of Toronto's clinical medical scientists published 3244 clinical articles from 1986 through 1990, twice-that of the second most productive school. One or two "hot" papers would have not
nearly the same effect on Toronto's citation rate as they would on the rates of less prolific schools. Do one or two bright research stars give a school cause to be ranked higher than a school with many effective, albeit less brilliant, lights? It is equally concerning that the citation rate is based only on citations of articles in the 900 or so journals that ISI monitors.' Significant research may be reported (or cited) in another language or in a narrowly specialized journal not monitored by ISI. Indeed, reports of very specialized research may contribute substantially to scientific knowledge in that field but not be cited often, simply because there are few researchers in the field. Conversely, an article of relatively less significance may be cited often because it is relevant to several disciplines or of interest to readers in many countries. Citation rates for a 5-year period present special problems. Articles published at the beginning of the period are far more likely to have been cited often than those published at the end. Thus, an important recently published paper may have been cited rarely during the period, even though in a few years its citation rate may turn it into a hot paper. Ranking a medical school solely by its citation rate is highly misleading and unnecessarily provocative. One must take other measures into account: research funds generated, types of research, language, breadth of interest in the research, relevance and, most important, quality. As Dr. Janet Halliwell, president of the Science Council of Canada, reportedly told Ottawa science writer Rhonda Birenbaum, "Data is [sic] like a bikini. What it reveals is interesting. What it conceals is often vital."
References 1. Clinical medicine: the top 50 US universities ranked by citation impact, 1986-90 (among those publishing > 1,000 papers). Sci Watch 1991; Nov-Dec: 7 2. Birenbaum R: Behind the data: a look at Canadian science. UnivAffairs 1992: 33 (1): 3-4
Dr. Squires is editor-in-chief and scientific editor ofCMAJ.
Reprint requests to: Dr. Bruce P. Squires, CMAJ, PO Box 8650, Ottawa, ON KIG OG8 FEBRUARY 1, 1992
CAN MED ASSOC J 1992; 146 (3)
341
