Sci Eng Ethics DOI 10.1007/s11948-015-9625-5 ORIGINAL PAPER

Ensuring the Quality, Fairness, and Integrity of Journal Peer Review: A Possible Role of Editors David B. Resnik • Susan A. Elmore

Received: 31 October 2014 / Accepted: 22 January 2015 Ó Springer Science+Business Media Dordrecht (outside the USA) 2015

Abstract A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions that have a significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making. Keywords Peer review  Quality  Fairness  Integrity  Ethics  Reliability  Bias  Editors  Publication

Introduction Nearly all scientific journals make publication decisions based on the recommendations of peer reviewers. The editors of Philosophical Transactions of the Royal Society of London instituted the first peer review procedures in the seventeenth century, but peer review did not become a common practice in scientific publishing D. B. Resnik (&) National Institute of Environmental Health Sciences, National Institutes of Health, Box 12233, Mail Drop CU 03, Research Triangle Park, NC 27709, USA e-mail: [email protected] S. A. Elmore National Toxicology Program, National Institute of Environmental Health Sciences, National Institutes of Health, Research Triangle Park, NC 27709, USA

123

D. B. Resnik, S. A. Elmore

until the late 19th century (Lee et al. 2012). Since the early 20th century, peer review has been used to evaluate grant applications, make personnel decisions, and award scientific prizes (Resnik 2011). Journal peer review performs several functions in science. First, it serves as a type of quality control by helping to ensure that published research meets standards of methodology and ethics (Smith 2006). Peer review is a form of scientific gate-keeping. Second, it serves as a form of quality improvement because authors usually make positive changes in their manuscripts in response to useful comments from reviewers and editors (Armstrong 1997). Third, peer review serves as a form of education if authors learn more about how to do good science and write good papers through their interactions with reviewers and editors (Shamoo and Resnik 2015). Peer review is far from perfect— low quality papers that are rejected by one journal may eventually be published by another, and reviewers may fail to make suggestions that improve papers or teach authors about doing good science—but most researchers agree that peer review is an indispensable part of scientific inquiry (Merton 1973; Smith 2006; Resnik 2011; Lee et al. 2012; Ho et al. 2013). A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review. Some of these include inadequate, inconsistent or biased review, and ethical transgressions by reviewers (Resnik et al. 2008; Lee et al. 2012). In an online survey of scientists from top universities from around the world, less than half of 1,340 respondents agreed that peer review is fair or scientific, and 60 % said that reviewers are often incompetent (Ho et al. 2013). While the empirical data indicates that problems can occur in peer review, in many cases the evidence for them is weak or inconclusive (Lee et al. 2012). Scientists and journal editors have suggested or implemented several reforms to address these problems, such as double-blind or open review procedures (Resnik 2011; Lee et al. 2012). Under the more traditional anonymous or single-blind peer review process, used by the majority of scientific journals, the reviewer’s name is not disclosed to the author. The identities of the authors and reviewers are concealed from each other in a double-blind review whereas in an open peer review process no identities are concealed. While the double-blind or open review proposals may or may not help to improve peer review, they overlook the key players in the entire process— the editors. Editors play a crucial role in peer review because they manage the review process, and peer review cannot function well unless editors adhere to the highest standards of science and ethics. This article will summarize some of the problems with the peer review system, discuss proposed reforms, and describe the role that journal editors should play in safeguarding and improving peer review.

Inadequate Review Some studies have found that reviewers often fail to catch simple errors or methodological flaws in submitted manuscripts (Lee et al. 2012). This may be due to poor choice of reviewers. Reviewers that provide inadequate assessments may be inexperienced in the review process or may lack in depth knowledge of the type of study under review. Reviewers should be experts in the field and their numbers may

123

Ensuring the Quality, Fairness, and Integrity

be few, depending on the area of study. Articles may also be complex, incorporating research from different areas of expertise, so no one expert reviewer would be able to provide a thorough and knowledgeable review (Shamoo and Resnik 2015). One study sought to evaluate the effects of reviewer characteristics and publication language on manuscript assessment. The researchers sent two inauthentic but realistic papers with obvious methodological flaws, such as inappropriate statistical tests, to 180 reviewers. Reviewers who caught all of the errors received a grade of 4 on a 1 to 4 scale. The 156 reviewers who returned reviews received an average score of 1.4. Only 25 % identified wrong sampling unit as a methodological flaw, and 34 % made no comments about the methodology (Nylenna et al. 1994). Another study aimed to determine the frequency of detection of major and minor errors at a general medical journal, the types of errors missed, and the impact of training on error detection. The conclusions were that, under the conditions of this study, training appeared to have only a minimal impact on reviewer performance (Schroter et al. 2008). Many commentators have observed that peer reviewers seldom detect data fabrication, falsification or other types of misconduct (LaFollette 1992; Resnik 2011; Shamoo and Resnik 2015). Some well-known fraudulent papers that made it through the peer review system include a study on the generation of human embryonic stem cells via therapeutic cloning (Hwang et al. 2005), experiments demonstrating how to derive pluripotent stem cells from somatic cells (Obokata et al. 2014), and a paper linking the measles vaccine to autism (Wakefield et al. 1998). While the rate of misconduct in published papers is not known, it is thought to be low. A meta-analysis of 18 surveys found that an average of 1.97 % of scientists admitted to fabricating or falsifying data at least once in their careers (Fanelli 2009). It is not fair to blame reviewers for failing to detect misconduct in submitted manuscripts, however, since they usually do not have access to the materials needed to detect fabrication or falsification, such as the original research records. Reviewers usually only examine the submitted manuscript and occasionally some supporting documents or data (Lee et al. 2012). Clever, intentional manipulation of data can be extremely difficult to detect, especially if the reviewer is relying solely upon his or her own reading of the manuscript. Some journals have begun using computer programs to check submitted manuscripts for suspected plagiarism. The programs compare manuscripts to a database of published articles (Butler 2010). One publisher, Taylor and Francis, screened all articles submitted to three journals during a six month period for plagiarism. One journal had a 23 % rate of suspected plagiarism (including selfplagiarism) (Butler 2010). Other journals have begun using computer programs to check digital images for deceptive image manipulation. These journals have developed guidelines for image manipulation and require authors to submit original images for review, so the editors can determine whether changes that have been made to the original images are appropriate (Lee et al. 2012). Two journals that implemented these procedures found that 20-25 % of authors failed to follow image manipulation guidelines and that 1 % of submitted manuscripts had problems that were so significant that the editors decided to contact the authors’ institution for suspected misconduct (Cromey 2013).

123

D. B. Resnik, S. A. Elmore

Inconsistent Review Lack of consistency is another problem with peer review. Some scientists have complained that review is so inconsistent that getting a paper published is more a function of luck than talent (Lee et al. 2012). Whether a paper is accepted or rejected may depend less on the quality of one’s work and more on the reviewers who agree to evaluate the manuscript. Because reviewers are invited, and not assigned, to review manuscripts, even if they are members of an editorial review board, it can be difficult at times to find the desired number of reviewers and this may affect the level of reviewer expertise. It is also not unusual for reviewers to have diametrically opposed opinions of the same manuscript (Lee et al. 2012). Since reviewers may have different interpretations of how the review criteria apply to scientific manuscripts, some variation should be expected (Lee 2013). However, significant divergence in reviewer assessments of manuscripts can indicate inadequate reviewer expertise and/or problems with the clarity of review standards that can undermine the quality, fairness, and integrity of review. In multidisciplinary research, reviewers from different areas of expertise may have divergent opinions of manuscripts based on divergent review standards (Lee et al. 2012). So this would not reflect deficiencies in expertise, but rather differences in areas of expertise pertaining to different sections or data reviewed by each expert. One of the reasons why reviewer reports are often inconsistent is that reviewers are usually asked to evaluate subjective characteristics of the research, such as its significance, novelty, or overall quality (Lee et al. 2012). Reviewers may have different opinions concerning significance, novelty, or overall quality because they have different levels of experience, theoretical or methodological assumptions or ideas about the direction the field should take. For example, a reviewer who accepts a controversial hypothesis concerning the treatment of prostate cancer might find a paper that reinforces this hypothesis to be significant, but a reviewer who rejects this hypothesis might not. Because subjective assessments of papers are a key part of the review process, it may be difficult to reduce inconsistency in peer review (Park et al. 2014). In one of the earliest studies on consistency in peer review, the researchers resubmitted twelve articles to the same psychology journals that had refereed and published them 18 to 32 months earlier. The articles were the same, except the names and affiliations had been changed. Only 3 out of the 38 editors and reviewers (7.9 %) detected the resubmission. Nine articles went through the review process again. 16 out of 18 reviewers recommended rejecting these articles based on ‘‘serious methodological flaws’’ and the editors concurred (Peters and Ceci 1982). While this study produced some interesting (and potentially embarrassing) results, its main flaw was the small sample size. Inter-rater reliability (IRR) for journal peer reviews is the degree to which two or more independent reviews of the same scientific document agree (Cicchetti 1991). A number of large studies on IRR for peer reviewers have been published. One study examined 5,881 reviewer reports for the Journal of General Internal Medicine from 2004 to 2008 and found that IRR for the overall decision (reject vs. accept/ revise) was barely better than chance (Kravitz et al. 2010). Another study examined

123

Ensuring the Quality, Fairness, and Integrity

IRR for papers submitted to two neuroscience journals. For one journal, agreement was very low (Cohen’s Kappa statistic = 0.08, where 0.0 = equals agreement no better than chance and 1.0 = perfect agreement). For the other journal, agreement was also poor (Cohen’s Kappa = 0.28) (Rothwell and Martyn 2000). However, another study of 554 reviews of 206 manuscripts submitted to a general medical journal between 2008 and 2009 found that while IRR was low as measured by the Cohen’s Kappa statistic (\ 0.2), it was higher when using Gwet’s Kappa (0.63). The authors concluded that IRR was high enough for the purposes of editorial decisionmaking (Baethge et al. 2013). Another study examined reviewer assessments of 495 abstracts submitted to a scientific meeting from 2001 to 2004 and found significant agreement about the overall quality of the submissions. IRR was higher for methodological criteria, such as statistical considerations, than for opinion-based criteria, such as overall significance of the research (Rowe et al. 2006). Finally, a meta-analysis of 48 journal articles published between 1966 and 2008 that examined reviewer agreement found that mean IRR was low (mean Kappa = 0.17) (Bornmann et al. 2010). As shown here, accurate interpretation of various journal IRR data may be challenging. The overall low IRRs among different journals may be due to a variety of factors such as the difficulty that editors have in finding sufficient reviewer expertise for submitted manuscripts, the complexity of manuscript data and thus the need for different expert reviewers, or that the data used to determine the IRR is inappropriate. The Journal of the American Child and Adolescent Psychiatry tested various methods to improve the IRR of reviewers’ ratings of submitted manuscripts over a 1-year period with 296 pairs of ratings studied. Their findings determined that, under the conditions tested, the IRR could be improved by using a multi-item rating scale, using training manuals, obtaining the ratings of solid, concrete items rather than judgments or opinions, and averaging the scores of multiple reviewers (Strayhorn et al. 1993).

Biased Review Bias is a third problem with the peer review process and is considered a violation of impartiality in the evaluation of a submission. Because peer review is conducted by people who may have different ideas about what constitutes good science, no review will be completely free from methodological or theoretical biases (Lee et al. 2012). However, certain types of strong biases can undermine the quality, fairness, and integrity of peer review. Some examples of what may influence reviewer bias include a preference for positive versus negative findings, gender of author, gender of reviewer, institutional affiliations, author nationalities, controversial or innovative research, author recommended reviewers, reviewer or editor conflict of interest, and authors’ reputation. There is a well-documented bias toward publication of positive findings as opposed to negative ones. One of the earliest studies of this type of bias examined the fate of 285 research projects approved by a human subjects ethics committee

123

D. B. Resnik, S. A. Elmore

between 1984 and 1987 and found that studies showing a statistically significant difference between treatment groups in clinical trials were much more likely to be published than those showing no difference (Easterbrook et al. 1991). Another study with a similar design found that clinical trials reporting statistically significant positive results were much more likely to be published than trials with negative results, and their time to publication was also shorter (Stern and Simes 1997). A systematic review and meta-analysis of twenty studies on positive biases found strong evidence for an association between statistically significant positive results and publication (Dwan et al. 2013). Although the evidence supports a bias toward publishing positive results, it is not clear that the peer review system is responsible for this bias. For example, one study found that the bias toward publication of positive findings was due to decisions made by authors, not editors or reviewers (Dickersin et al. 1992). A study of 745 manuscripts submitted to the Journal of the American Medical Association between 1996 and 1999 found that manuscripts reporting positive results were not more likely to be published than those reporting negative results (Olson et al. 2002). However, the results of this study may not be generalizable, since it focused on one high-impact journal. Furthermore, it may be the case that authors refrain from submitting papers with negative results because they expect that they will be rejected (Olson et al. 2002). Clearly, more research on the causes of the bias toward publication of positive findings is needed. A number of studies have identified gender biases in grant peer review (Wenneras and Wold 1997; Bornmann et al. 2007; Lee et al. 2012), but evidence for gender bias in journal peer review is weak. One study sent an identical manuscript to 126 male and female reviewers from behavioral science journals. The only difference in the manuscripts was that half had a first author with a male name and the other had one with a female name. 65 reviews (51.6 %) were returned. Female reviewers accepted significantly more female-authored papers than male reviewers and they accepted significantly more female-authored than male-authored papers (Lloyd 1990). A significant limitation of this study was its small sample size. A larger study with a similar design sent the same paper (with male vs. female first author names) to biology students and faculty to review. 989 responses were suitable for data analysis. The study found that the gender of the first author name had no effect on the review assessment or the overall recommendation, although female post-graduate students who reviewed papers had significantly higher rejection rates, suggesting that they may apply different standards to peer review (Borsuk et al. 2009). A 2001 study that provided evidence of gender bias reported that the percentage of female first authors on papers significantly increased after Trends in Ecology and Evolution adopted a double-blind review system, in which neither authors nor reviewers are informed of each other’s identity (Grod et al. 2008; Budden et al. 2008). However, some commentators on this study objected to its statistical analysis and presented data challenging the notion that peer review is gender biased (Whittaker et al. 2008). Another study of 2,680 papers from five journals found no significant difference in the acceptance rates of female first-authored papers and male first-authored papers (Tregenza 2002), and a study of 713 papers submitted to

123

Ensuring the Quality, Fairness, and Integrity

the Journal of Neurophysiology in 2007 reported similar results (Lane and Linden 2009). An article reviewing the causes of the underrepresentation of women in science concluded that there is no evidence of gender bias in journal peer review (Ceci and Williams 2011). Some studies have examined biases related to the institutional affiliations and nationalities of authors. One study examined abstracts submitted to the American Heart Association’s Scientific Sessions from 2000 to 2005. From 2000 to 2001, the organization used an open review system, so that information concerning the authors’ names and institutional affiliations was shared with reviewers. In 2001, the organization instituted a blind review policy, so the reviewers were not told the authors’ names or affiliations from 2002 to 2005. 27,311 abstracts were submitted from 2001 to 2002, and 39,964 abstracts were submitted from 2002 to 2005. The study found that, during open review, there was evidence of possible bias in favor of authors from the U.S. and other English-speaking countries, prestigious academic institutions, and U.S. government agencies. There was evidence of possible bias against authors from private industry but no evidence of gender bias. Blinded peer review partially reduced evidence of reviewer biases (Ross et al. 2006). A study of 17,205 abstracts submitted to a scientific meeting between 1992 and 1995 found that North American authors were more likely to have their abstracts accepted than authors from other regions (Timmer et al. 2001), and a study of 2,680 papers submitted to five journals found that authors from English-speaking countries were more likely to have their papers accepted than authors from non-English-speaking countries. Another study found that journals were more likely to accept papers from authors who have the same nationality as the journal, with the strongest effect being in favor of U.S. authors (Ernst and Kienbacher 1991). A retrospective review of 258 major papers and 147 brief reports published in a medical journal in 1994 found that there was no bias in favor of papers from more prestigious institutions although there was some bias in favor of brief reports (Garfunkel et al. 1994). However, one must also consider that apparent bias in favor of or against authors of a particular nationality or affiliation may be due to differences in scientific rigor or writing proficiency between various countries and/or institutions. Moreover, if the science appears sound but the manuscript is unacceptable due to poor writing style or excessive grammatical errors, then a journal may reject with a recommendation to resubmit once the editing has been completed. So a true account of the rejection rate for any particular journal may be difficult to determine. Several commentators have criticized the grant peer review process for being conservative, claiming that reviewers are biased against research that is controversial, innovative, interdisciplinary, or otherwise challenges their theoretical frameworks, methodologies, or assumptions (Hull 1988; Chubin and Hackett 1990; Shamoo and Resnik 2015). A few studies have found that journal peer review may also be conservative, but this claim is difficult to assess (Lee et al. 2012). One study randomly assigned 398 reviewers to receive either a paper on a conventional therapy for obesity or one on an unconventional therapy. The papers were identical except for the type intervention. 141 reviewers returned useable assessment forms. The study found significant evidence of bias in favor of the conventional therapy, suggesting that reviewers may be biased against unorthodox research (Resch et al.

123

D. B. Resnik, S. A. Elmore

2000). Other studies have found that reviewers tend to favor papers that fit their theoretical assumptions (Mahoney 1977; Hull 1988). Various studies have examined other types of biases in peer review. One study that assessed the reviews from 156 reviewers from a Scandinavian journal found evidence of a bias toward manuscripts written in English compared to the same manuscripts written in the author’s native language (Nylenna et al. 1994). Two observational studies comparing reviewers suggested by authors vs. editors found no difference in the quality of review, although author-suggested reviewers were more likely to recommend publication (Schroter et al. 2006; Rivara et al. 2007). Some studies have produced evidence that female reviewers are less likely to recommend acceptance of manuscripts than male reviewers (Gilbert et al. 1994; Borsuk et al. 2009; Wing et al. 2010). An important potential source of bias that has not been well-studied is reviewer or editor conflict of interest. Reviewers or editors may have financial interests, such as stock ownership, intellectual property, or consulting arrangements with private sponsors, which are related to the research that they are asked to review (Gasparyan et al. 2013; Shamoo and Resnik 2015). While there is a considerable body of evidence showing that authors’ financial interests can create bias in research (Sismondo 2008; Shamoo and Resnik 2015), there are no published studies on the impact of reviewers’ or editors’ financial interests. It may difficult to study this topic because many journals to do not require editors or reviewers to disclose conflicts of interest and journals that have a disclosure policy may not enforce it or have a way to verify the reviewer’s stated lack of conflict. It may therefore be difficult to determine whether reviewers or editors have undisclosed conflicts of interest that could impact review. Another potential source of bias that has not been well-studied pertains to the authors’ reputation. Several decades ago, sociologist of science Robert Merton (1973) hypothesized that science is subject to a bias which he dubbed the Matthew Effect after the passage from the Holy Bible, King James Version: ‘‘For unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken even that which he hath (Matthew 25:29).’’ The basic idea here is that researchers with well-established reputations tend to increase their funding and publications, while those without well-established reputations do not. While many scientists have complained that an ‘‘old boys’ network’’ shapes grant review in science (McCullough 1989; Shamoo and Resnik 2015), the influence of the author’s reputation on journal publication decisions has not been well-studied (Lee et al. 2012).

Unethical Review It is important for reviewers to adhere to ethical standards to protect the integrity and trustworthiness of peer review. Authors who submit manuscripts to a journal must be able to trust that reviewers will not use their ideas, methods, or data without permission and will treat their manuscripts as confidential (Shamoo and Resnik 2015). Journal editors must be able to trust that reviewers will disclose conflicts of

123

Ensuring the Quality, Fairness, and Integrity

interest and conduct reviews professionally. The Committee on Publication Ethics (2014) has developed ethical standards for authors, editors, and reviewers. There is some anecdotal evidence that reviewers sometimes act unethically, but there is little empirical data to support this claim (Nature Editors 2001; Rennie 2003; Smith 2006; Resnik 2011). One study administered a survey on perceptions of unethical practices in peer review to biomedical scientists (Resnik et al. 2008). Respondents said that comments from reviewers have included personal attacks, that a reviewer had delayed the review so that he/she could publish an article on the same topic, that a reviewer breached confidentiality, and that a reviewer used ideas, data, or methods without permission (Resnik et al. 2008). An online survey also found that scientists have some concerns about ethical problems in peer review such as personal attacks from reviewers, reviewer breach of confidentiality, and reviewers requiring unnecessary references to their own research (Ho et al. 2013). One limitation of both of these studies is that they asked respondents to provide information about their experiences with journal peer review but they did not confirm whether alleged ethical transgressions actually occurred. The low response rate is a limitation of the larger survey, since scientists who experienced problems with peer review may have been more likely to respond. Nevertheless, these surveys indicate that reviewers sometimes violate ethical norms and that there is a perception among scientists that this does occur with some frequency.

Proposals for Reforming Journal Peer Review Editors, scientists, and commentators have discussed various proposals for reforming the journal peer review process to address the problems that have been identified. Proposals range from minor tweaks to radical revisions. A modest proposal for reforming the system is to increase the number of reviewers used to evaluate manuscripts (Neff and Olden 2006). Most journals use two reviewers but some use more. The thinking behind this proposal is that adding additional reviewers will increase the probability of converging on the correct assessment of the manuscript (Neff and Olden 2006). While using more reviewers could help promote consistency and reduce bias, it might not, since additional reviewers might reinforce each other’s biases or increase the divergence of opinions without convergence. Since this proposal has not been wellstudied, we cannot say for certain whether adding additional reviewers improves peer review (Newton 2010). However, it is the editor’s responsibility to ensure, to the best of his/her ability, that reviewers’ edits and comments are not biased but rather provide the author with a detailed report of the strengths and weaknesses of the paper so that the author may make appropriate revisions. Another modest proposal is to train reviewers in review standards and techniques to improve peer review (Lee et al. 2012). Some scientists have published papers on how to review a manuscript (Salasche 1997; Benos et al. 2007; Lovejoy et al. 2011) and most journals provide reviewers with some amount of instruction. While training sounds like a good idea, there is little published evidence that it works. One study randomly assigned 609 reviewers to receive one day of face-to-face training in review standards and techniques, self-study training materials with the same content

123

D. B. Resnik, S. A. Elmore

as the face-to-face training, or no training (control). The reviewers received three manuscripts to review containing some deliberately introduced errors. Reviewer performance was measured at baseline and after receiving the training (or no training). The study found that both intervention groups caught more errors than the control group, and that the self-study training materials had a minor impact on reviewer performance, but that these were slight improvements in reviewer performance that faded over time (Schroter et al. 2004). Other studies have found that training and mentoring have little impact on reviewer performance (Callaham et al. 1998; Callaham and Tercier 2007; Houry et al. 2012). Although these studies have not shown evidence of a positive effect of training on reviewer performance, there are no studies that show a negative impact. We believe that the use of training materials, either self-study or face-to-face, should not be discouraged, since training can help the novice reviewer understand the methodology of manuscript review as well as encourage consistency in the preview process. A more radical proposal is to use double-blind review. As noted above, most scientific journals use a single-blind system. Some have argued that a double-blind system, in which authors and reviewers are both anonymous, would improve the peer review process. 58.2 % of respondents in a survey of peer review favored a system in which authors are anonymous (Ho et al. 2013). Some journals have adopted a double-blind system in an attempt to improve their peer review process (Resnik 2011; Lee et al. 2012). Several studies have conducted randomized controlled trials to answer the question of whether double-blinding improves the quality of review. While two small studies (McNutt et al. 1990; Fisher et al. 1994) found that blinding reviewers improves the quality of review, several larger ones found that it does not (van Rooyen et al. 1998; Justice et al. 1998; Godlee et al. 1998). Moreover, there is evidence that reviewers can frequently identify authors from their references and topics, especially in small fields in which most scientists know each other or are familiar with each other’s work (Lee et al. 2012). One study found that blinded reviewers could correctly identify first authors 42 % of the time (van Rooyen et al. 1998). Other studies have yielded similar results (Fisher et al. 1994; Cho et al. 1998; Justice et al. 1998; Baggs et al. 2008). One of the potential dangers of double-blinding is that it may not provide adequate management of reviewer conflict of interest (COI). Most journals ask reviewers to disclose COIs related to the review. A reviewer who is not told the authors’ identities may see no reason to disclose a COI, even if the reviewer can identify the authors of the paper. One way to deal with this problem would be for editors to ask reviewers to decline an invitation to review if they can identify the authors and they have a COI. Although double-blinding may not work all the time, when it does work it may help to promote the ideal that publications be judged on the quality of the work with no bias for or against the author. In our opinion, a double-blind system would probably reduce biases related to reputation, gender, nationality, and institutional affiliation but it probably would not reduce other biases or address defective or inconsistent review, or ethical transgressions by reviewers. The most radical proposal for reforming peer review is to remove blinders from authors and reviewers so the entire process is open, and some journals have already

123

Ensuring the Quality, Fairness, and Integrity

implemented such an open review process (Smith 1999; Lee et al. 2012). Open review could reduce ethical transgressions by holding reviewers accountable for their conduct (Resnik 2011). Reviewers might be more likely to review manuscripts more carefully under an open system, since their identities would be revealed to the authors. They also might be less likely to make personal attacks in their reviews, breach confidentiality, or use information without permission. Authors might also regard an open process as fairer than a blinded one because there would be more transparency. So far the evidence for the effectiveness of open review is inconclusive. Several randomized controlled trials of open review have found that open review does not significantly improve the quality of peer review (van Rooyen et al. 1999; Godlee et al. 1998; van Rooyen et al. 2010). However, one study found reviewers who participated in open review performed better reviews and were more courteous (Walsh et al. 2000). One of the biggest drawbacks of an open review system is the potential that reviewers may decline to perform reviews because they want the protection from potential reprisal afforded by anonymity (Lee et al. 2012). Also, reviewers may provide a less-than-honest opinion of the work if they know that their identities will be revealed to the authors. Junior scientists might fear reprisals from more senior researchers if they make negative or critical comments about their papers (Smith 1999) and authors may feel compelled to be more responsive to senior reviewers than junior ones. A narrow field with a limited number of experts may require colleagues to review each other’s work, and they may not want to provide negative reviews for each other under an open review system. Some studies have shown that most reviewers prefer to remain anonymous (Lee et al. 2012). For example, 64.4 % of respondents in a study of scientists’ views on the peer system said that reviewers’ identities should not be disclosed to authors (Ho et al. 2013). In a study comparing blinded vs. un-blinded peer review, potential reviewers were more likely to decline to participate in the study if their identities would be revealed to the authors (van Rooyen et al. 1999). Another study of open review found that 55 % of potential reviewers refused to participate in a study in which their identities would be shared with authors (van Rooyen et al. 2010). The main reason why experts often decline to participate in open review is that they fear retaliation from authors (Ho et al. 2013). Some commentators have recommended that journals adopt radically open systems to enhance transparency and accountability. Under this proposal, the entire peer review process would be open to the public so that scientists could critique it. Review reports, including identifying information, would be published along with papers. Readers of the journal would be allowed to critique the papers, review reports, and the editorial decision-making process (Wicherts et al. 2012). One advantage of a radically open system is that it would make it possible to discover undisclosed conflicts of interest (Benos et al. 2007), which can be difficult to detect. A radically open system might also improve the quality and integrity of review by holding reviewers accountable to the scientific community, but qualified reviewers might refuse to participate out of fear of reprisal. Another problem with radically open review is that it would make information available to the public, such as data that could be used to identify human subjects, which should be kept confidential.

123

D. B. Resnik, S. A. Elmore

Some journals have also begun using hybrid systems in which the initial review is double-blinded, but then invited open peer commentaries on the article are posted once it is published. Authors can read and respond to the commentaries in the same issue of the journal. In theory, hybrid systems could help to reduce bias while encouraging free exchange of ideas. However, the impact of hybrid systems on peer review has not been studied (Lee et al. 2012).

The Role of Editors in Promoting Quality, Integrity, and Fairness in Review Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions that have a significant impact on the process and its outcome (Newton 2010). Editors invite experts to review manuscripts, communicate with reviewers and authors, provide guidance for reviewers, read reviewer reports, ensure that policies regarding conflict of interest and confidentiality are upheld, and provide opinions to the editor-in-chief concerning acceptance, revisions, or rejection (Chubin and Hackett 1990). Editors can compensate for the flaws inherent in the peer review system or they can exacerbate them. Journals may have different personnel organizations, depending on their size and affiliation. Larger journals may have a team of full-time professional editors. Other journals, such as those affiliated with a professional society, may have an outside volunteer editorial board composed of associate editors to assign reviewers and oversee individual manuscripts, and a larger group of volunteer reviewers, with a variety of expertise, who review papers on specialized topics. This helps to ensure that they have enough expertise to handle a variety of reviews. But in all cases, the editor-in-chief (chief editor or executive editor) is the editorial leader with final responsibility for all operations and policies and ultimately decides whether or not a submitted manuscript will be accepted or rejected. In the following sections, we offer some opinions about the steps that editors should take to minimize and mitigate the problems identified in previous sections and improve the quality, integrity, and fairness of peer review. These opinions are based on our professional experience as journal editors. We will organize our discussion of editorial responsibilities around the different stages of the peer review process, from selecting reviewers to dealing with post-publication issues.

Selecting Reviewers Editors have an opportunity to significantly impact the quality, fairness, and integrity of the peer review process when they select reviewers. Editors may use different methods for choosing potential reviewers, such as checking the paper’s reference list for subject matter experts or utilizing a database containing the names and expertises of people who have agreed to review papers for the journal. If editors use reviewers suggested by authors, they should include some independent reviewers to ensure balance. Editors should avoid stacking the deck for or against

123

Ensuring the Quality, Fairness, and Integrity

a manuscript. To ensure that manuscripts receive a fair hearing, editors should seek an appropriate balance between different types of reviewers with different qualifications. Editors may have information concerning an individual’s potential bias for or against a particular theoretical orientation, methodology, research group, discipline, nationality, or institution. Editors may also know whether some reviewers are more demanding or lenient than others. If a submission crosses disciplinary boundaries, editors should invite reviewers from different disciplines to ensure that it receives a fair and adequate review. If a manuscript is highly innovative or controversial, editors should consider inviting reviewers who they believe are open to new ideas (Armstrong 1997). Of course, inexperienced editors may not be familiar with reviewer penchants and biases, but they should gain experience as they continue to manage manuscript review. If editors invite reviewers who are not well-qualified to review the paper, then their reviewer reports may be inadequate. In many research fields, those who are best qualified to review a paper may also have biases for or against the paper. Editors should balance fairness and reviewer competence when choosing reviewers. In some cases, it may be necessary to use reviewers who may have a bias in order to obtain a high-quality review. It may be necessary to invite reviewers with special qualifications beyond the topic of the paper to ensure that the paper is properly reviewed. For example, a paper that includes digital images of the structure of a protein may need to be reviewed by someone with expertise in digital imaging, and a paper that uses an innovative method for analyzing the results of a clinical trial may need to be reviewed by a statistician.

Providing Guidance for Reviewers Editors should promote the fairness, quality, and integrity of peer review by providing reviewers with appropriate guidance. When editors provide guidance for reviewers, they should inform them of the criteria used to evaluate manuscripts (such as originality, significance, rigor, etc.), journal review policies (including ethics and conflict of interest), the journal’s policies on authorship criteria, and the timetable for returning reviews. Editors should remind reviewers of deadlines to prevent tardiness and they should consider inviting a different reviewer if a reviewer has not completed an assignment on time. It is important to note, however, that reviewing a paper can take a great deal of time and reviewers may be busy with other projects, so editors should take this into account when communicating with them. Editors should also be on the alert for possible ethical transgressions by reviewers and take steps to address them if they occur. When a reviewer discloses a conflict of interest, editors must decide whether to invite another reviewer or to solicit input from the reviewer despite the conflict. An editor might decide to seek input from a reviewer with a disclosed conflict of interest because he or she judges that the conflict is not likely to significantly impact the reviewer’s judgment and there are very few people who are qualified or willing to review the manuscript.

123

D. B. Resnik, S. A. Elmore

Evaluating Reviewer Reports When editors read reviewer reports, they should assess them for quality, fairness, thoroughness, and professionalism. Ideally, editors should have enough knowledge of the relevant area of research so they can form their own independent judgment of the merits of the manuscript, which will allow them to assess reviewer reports. Editors should take an active role in the review process by engaging with reviewers and critically evaluating their reports (Newton 2010). To do this, they should read the manuscript carefully and have some familiarity with the research. When editors receive a report, they should determine whether it is accurate, thorough, fair, and professional. They should assess reviewer comments to decide whether they are useful and appropriate. Editors should ask reviewers to clarify ambiguous comments. An editor may decide to ignore or downgrade a reviewer report if he or she judges that a review is inadequate, incompetent, and/or biased. If a reviewer makes a personal attack in a report, the editor should delete those remarks from comments that are passed on to the authors. However, editors should be careful when deleting reviewer portions of comments, since this is a form of censorship, and deciding what counts as inappropriate is a matter of interpretation. Larger journals may use multiple associate editors and multiple reviewers that review papers on specialized topics to ensure that they have enough expertise to handle the review. If editors lack sufficient expertise in a domain of research they are overseeing, they should assign the manuscript to another editor, or they should consult with other editors or editorial board members who have the requisite expertise. When editors make decisions concerning a manuscript, they should weigh and consider the reviewer reports and be mindful of their own biases for or against a paper, to the extent that this is possible (Newton 2010). For example, editors should be careful to avoid basing a judgment of a paper on an author’s or an institution’s reputation (Lee et al. 2012). When reports are conflicting, editors may invite another reviewer to break the tie or make a decision based on their overall assessment of the manuscript and the reports. Journals with very high standards for publication may require that all of the reviewer reports are positive, whereas journals with lower standards may decide to accept an article even if some reports are negative. In difficult cases, editors should consider holding a conference with each other to decide upon the outcome of review. Editors may also share other reviewer reports with the reviewers after the review process is complete to provide them with different perspectives on the paper and help them learn more about peer review. As an example, some journals blind copy the reviewers on the decision letter emails to the authors so that the reviewers will have such access to all reviews.

Communicating Decisions to Authors Editors should promote fairness and integrity when they communicate with authors about editorial decisions. If authors do not agree with editorial decisions or reviewer reports, they should be given the opportunity to respond to reviewer comments and

123

Ensuring the Quality, Fairness, and Integrity

explain why one or more reviewer recommendations are inaccurate, unfair or unnecessary. Such discussions may go back and forth between authors, editors, and reviewers, and other experts or chief editors may need to be consulted. Chief editors have the final responsibility for deciding whether requested changes should be made and whether manuscripts should be accepted for publication. When editors convey their decisions to authors, they must decide upon the conditions for acceptance. For example, an editor may decide not to require an author to include a reference recommended by a reviewer if he or she deems it to be unnecessary. Editors should provide instructions to authors for revising the manuscript, distinguishing between revisions that are necessary and those that are optional. Editors should require authors to submit a letter detailing the revisions that have been made when they submit a revision. They may also require the authors to submit a tracked changes version of the manuscript with the revision. They can also address any questions or concerns authors may have about the review process or the editorial decision.

Post-publication Editorial Decisions Editors should continue to promote the quality, integrity, and fairness of the peer review process after an article is published. When a paper appears in print or online, chief editors must be prepared to respond to commentaries on the paper from other scientists. Chief editors may decide to publish a commentary or letter that criticizes a paper and give the authors an opportunity to respond in a concurrently published letter. If there are serious problems with a paper, such as errors that undermine the validity of the data, fabrication or falisification of data, or plagiarism, chief editors must decide whether to require that the authors print a correction or retraction of the paper. Editors may sometimes need to decide whether to retract a paper even if not all of the authors agree to a retraction. Retracting a paper without the consent of all of the authors opens the editors and the journal to legal liability, since the authors may claim that their reputation has been unfairly damaged (Williams and Wager 2013). Editors must decide how to balance liability concerns against the need to maintain the integrity of the published record. Some editors may choose to publish an editorial expression of concern if a paper with a serious problem is not retracted or corrected. In cases of suspected misconduct that cannot be confirmed or denied by the editor alone, the editor may contact the author’s institution about the allegation. The institution may or may not respond to the allegation. If institutional officials conclude that misconduct has occurred they may ask the editors to retract the paper. To promote the integrity of the published record, chief editors and their publishing companies should develop procedures for clearly marking retracted or corrected papers and linking them to research databases. Chief editors should develop policies and procedures pertaining to peer review and publication, such as policies on public availability of supporting data and information, conflict of interest (for authors, reviewers, and editors), authorship, prior publication of data or text, misconduct (including data fabrication or falsification or plagiarism), retractions and corrections, human and animal research

123

D. B. Resnik, S. A. Elmore

protections, and clinical trial registration. The Committee on Publication Ethics (2014) and the International Committee of Medical Journal Editors (2014) have developed guidelines for editorial policies. Editorial policies can help authors and reviewers better understand the expectations for publication within a journal. Editors should ensure that statistics are compiled on their review process so they can make improvements. They could keep track of how often individuals are asked to review manuscripts, the value of their recommendations, and how often they return a report on time. Editors may also decide to assess reviewer agreement and potential bias in their review process. Some journals keep thorough statistics on the peer review process and others have begun to move in this direction (Lee et al. 2012). The electronic submission systems that most journals have adopted can make it easier to keep track of information related to the review process.

Conclusion Evidence shows that several different problems—inadequate, inconsistent, or biased review, and ethical transgressions by reviewers—can have a negative impact on journal peer review. Since most researchers agree that peer review is an essential part of scientific research, the question is not whether to abandon peer review but how to improve it (Smith 2006; Lee et al. 2012). It is not known whether implementing reform proposals, such as reviewer training or double-blind or open review, will effectively address these problems. More research is therefore needed on peer review’s problems and potential solutions. Since editors have a major impact on the review process and its outcome, they will continue to play a key role in any reform efforts, and they should take appropriate steps to promote quality, consistency, fairness, and ethics in peer review. Acknowledgments David B. Resnik is Associate Editor of Accountability in Research. Susan A. Elmore is Editor and Editor-in-Chief of Toxicologic Pathology. This research is supported by the National Institute of Environmental Health Sciences (NIEHS), National Institutes of Health (NIH). It does not represent the views of the NIEHS, NIH, or U.S. government. Conflict of interest The authors disclose no conficts of interest.

References Armstrong, J. S. (1997). Peer review for journals: Evidence on quality control, fairness, and innovation. Science and Engineering Ethics, 3(1), 63–84. Baethge, C., Franklin, J., & Mertens, S. (2013). Substantial agreement of referee recommendations at a ¨ rzteblatt International. PLoS ONE, general medical journal–A peer review evaluation at Deutsches A 8(5), e61401. Baggs, J. G., Broome, M. E., Dougherty, M. C., Freda, M. C., & Kearney, M. H. (2008). Blinding in peer review: The preferences of reviewers for nursing journals. Journal of Advances in Nursing, 64(2), 131–138. Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M., et al. (2007). The ups and downs of peer review. Advances in Physiology Education, 31(2), 145–152.

123

Ensuring the Quality, Fairness, and Integrity Bornmann, L., Mutz, R., & Daniela, H. D. (2010). Reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE, 5(12), e14331. Bornmann, L., Mutza, R., & Daniela, H. D. (2007). Gender differences in grant peer review: A metaanalysis. Journal of Informetrics, 1(3), 226–238. Borsuk, R. M., Aarssen, L. W., Budden, A. E., Koricheva, J., Leimu, R., Tregenza, T., & Lortie, C. J. (2009). To name or not to name: the effect of changing author gender on peer review. BioScience, 59(11), 985–989. Budden, A. E., Tregenza, T., Aarssen, L. W., Koricheva, J., Leimu, R., & Lortie, C. J. (2008). Doubleblind review favours increased representation of female authors. Trends in Ecology & Evolution, 23(1), 4–6. Butler, D. (2010). Journals step up plagiarism policing. Nature, 466(7303), 167. Callaham, M. L., & Tercier, J. (2007). The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS Med, 4(1), e40. Callaham, M. L., Wears, R. L., & Waeckerle, J. F. (1998). Effect of attendance at a training session on peer reviewer quality and performance. Annals of Emergency Medicine, 32(3 Pt 1), 318–322. Ceci, S. J., & Williams, W. M. (2011). Understanding current causes of women’s underrepresentation in science. Proceedings of the National Academy of Sciences of the United States of America, 108(8), 3157–3162. Cho, M. K., Justice, A. C., Winker, M. A., Berlin, J. A., Waeckerle, J. F., Callaham, M. L., & Rennie, D. (1998). Masking author identity in peer review: What factors influence masking success? PEER Investigators. Journal of the American Medical Association, 280(3), 243–245. Chubin, D. E., & Hackett, E. J. (1990). Peerless science: Peer review and U.S. Science Policy. Albany, NY: State University of New York Press. Cicchetti, D. V. (1991). The reliability of peer review for manuscript and grant submissions: A crossdisciplinary investigation. Behavioral and Brain Sciences, 14(1), 119–135. Committee on Publication Ethics. (2014). Code of Conduct. http://publicationethics.org/resources/codeconduct. Accessed 31 Oct 2014. Cromey, D. W. (2013). Digital images are data: And should be treated as such. Methods in Molecular Biology, 931, 1–27. Dickersin, K., Min, Y. I., & Meinert, C. L. (1992). Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. Journal of the American Medical Association, 267(3), 374–378. Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & Reporting Bias Group. (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias—An updated review. PLoS ONE, 8(7), e66844. Easterbrook, P. J., Berlin, J. A., Gopalan, R., & Matthews, D. R. (1991). Publication bias in clinical research. Lancet, 337(8746), 867–872. Editors, Nature. (2001). Editorial: Bad peer reviewers. Nature, 413(6852), 93. Ernst, E., & Kienbacher, T. (1991). Chauvinism. Nature, 352(6336), 560. Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and metaanalysis of survey data. PLoS ONE, 4(5), e5738. Fisher, M., Friedman, S. B., & Strauss, B. (1994). The effects of blinding on acceptance of research papers by peer review. Journal of the American Medical Association, 272(2), 143–146. Garfunkel, J. M., Ulshen, M. H., Hamrick, H. J., & Lawson, E. E. (1994). Effect of institutional prestige on reviewers’ recommendations and editorial decisions. Journal of the American Medical Association, 272(2), 137–138. Gasparyan, A. Y., Ayvazyan, L., Akazhanov, N. A., & Kitas, G. D. (2013). Conflicts of interest in biomedical publications: Considerations for authors, peer reviewers, and editors. Croatian Medical Journal, 54(6), 600–608. Gilbert, J. R., Williams, E. S., & Lundberg, G. D. (1994). Is there gender bias in JAMA’s peer-review process? Journal of the American Medical Association, 272(2), 139–142. Godlee, F., Gale, C. R., & Martyn, C. N. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: A randomized controlled trial. Journal of the American Medical Association, 280(3), 237–240. Grod, O. N., Budden, A. E., Tregenza, T., Koricheva, J., Leimu, R., Aarssen, L. W., & Lortie, C. J. (2008). Systematic variation in reviewer practice according to country and gender in the field of ecology and evolution. PLoS ONE, 3(9), e3202.

123

D. B. Resnik, S. A. Elmore Ho, R. C., Mak, K. K., Tao, R., Lu, Y., Day, J. R., & Pan, F. (2013). Views on the peer review system of biomedical journals: An online survey of academics from high-ranking universities. BMC Medical Research Methodology, 13, 74. Holy Bible, King James Version. (1991). New York: Random House. Houry, D., Green, S., & Callaham, M. (2012). Does mentoring new peer reviewers improve review quality? A randomized trial. BMC Medical Education, 12, 83. Hull, D. (1988). Science as a process. Chicago: University of Chicago Press. Hwang, W. S., Roh, S. I., Lee, B. C., Kang, S. K., Kwon, D. K., Kim, S., et al. (2005). Patient-specific embryonic stem cells derived from human SCNT blastocysts. Science, 308(5729), 1777–1783. International Committee of Medical Journal Editors. (2014). Recommendations. http://www.icmje.org. Accessed 31 Oct 2014. Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., & Rennie, D. (1998). Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. Journal of the American Medical Association, 280(3), 240–242. Kravitz, R. L., Franks, P., Feldman, M. D., Gerrity, M., Byrne, C., & Tierney, W. M. (2010). Editorial peer reviewers’ recommendations at a general medical journal: Are they reliable and do editors care? PLoS ONE, 5(4), e10072. LaFollette, M. (1992). Stealing Into Print: Fraud, plagiarism, and misconduct in scientific publishing. Berkeley, CA: University of California Press. Lane, J. A., & Linden, D. J. (2009). Is there gender bias in the peer review process at Journal of Neurophysiology? Journal of Neurophysiology, 101(5), 2195–2196. Lee, C. J. (2013). A Kuhnian critique of psychometric research on peer review. Philosophy of Science, 79(5), 859–870. Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2012). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17. Lloyd, M. E. (1990). Gender factors in reviewer recommendations for manuscript publication. Journal of Applied Behavior Analysis, 23(4), 539–543. Lovejoy, T. I., Revenson, T. A., & France, C. R. (2011). Reviewing manuscripts for peer-review journals: A primer for novice and seasoned reviewers. Annals of Behavioral Medicine, 42(1), 1–13. Mahoney, M. J. (1977). Publication preferences: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2), 161–175. McCullough, J. (1989). First comprehensive survey of NSF applicants focuses on their concerns about proposal review. Science, Technology and Human Values, 14(1), 78–88. McNutt, R. A., Evans, A. T., Fletcher, R. H., & Fletcher, S. W. (1990). The effects of blinding on the quality of peer review. A randomized trial. Journal of the American Medical Association, 263(10), 1371–1376. Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago Press. Neff, B. D., & Olden, J. D. (2006). Is peer review a game of chance? BioScience, 56(4), 333–340. Newton, D. P. (2010). Quality and peer review of research: An adjudicating role for editors. Accountability in Research, 17(3), 130–145. Nylenna, M., Riis, P., & Karlsson, Y. (1994). Multiple blinded reviews of the same two manuscripts. Effects of referee characteristics and publication language. Journal of the American Medical Association, 272(2), 149–151. Obokata, H., Wakayama, T., Sasai, Y., Kojima, K., Vacanti, M. P., Niwa, H., et al. (2014). Stimulustriggered fate conversion of somatic cells into pluripotency. Nature, 505(7485), 641–647. Olson, C. M., Rennie, D., Cook, D., Dickersin, K., Flanagin, A., Hogan, J. W., et al. (2002). Publication bias in editorial decision making. Journal of the American Medical Association, 287(21), 2825–2828. Park, I. U., Peacey, M. W., & Munafo`, M. R. (2014). Modelling the effects of subjective and objective decision making in scientific peer review. Nature, 506(7486), 93–96. Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: the fate of published articles, submitted again. Behavioral and Brain Sciences, 5(2), 187–195. Rennie, D. (2003). Misconduct and journal peer review. In F. Godlee & T. Jefferson (Eds.), Peer review in health sciences (2nd ed., pp. 118–129). London: BMJ Books. Resch, K. I., Ernst, E., & Garrow, J. (2000). A randomized controlled study of reviewer bias against an unconventional therapy. Journal of the Royal Society of Medicine, 93(4), 164–167. Resnik, D. B. (2011). A troubled tradition. American Scientist, 99(1), 24–28.

123

Ensuring the Quality, Fairness, and Integrity Resnik, D. B., Gutierrez-Ford, C., & Peddada, S. (2008). Perceptions of ethical problems with scientific journal peer review: An exploratory study. Science and Engineering Ethics, 14(3), 305–310. Rivara, F. P., Cummings, P., Ringold, S., Bergman, A. B., Joffe, A., & Christakis, D. A. (2007). A comparison of reviewers selected by editors and reviewers suggested by authors. Journal of Pediatrics, 151(2), 202–205. Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., et al. (2006). Effect of blinded peer review on abstract acceptance. Journal of the American Medical Association, 295(14), 1675–1680. Rothwell, P. M., & Martyn, C. N. (2000). Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? Brain, 123(Pt 9), 1964–1969. Rowe, B. H., Strome, T. L., Spooner, C., Blitz, S., Grafstein, E., & Worster, A. (2006). Reviewer agreement trends from four years of electronic submissions of conference abstract. BMC Medical Research Methodology, 6, 14. Salasche, S. J. (1997). How to ‘‘peer review’’ a medical journal manuscript. Dermatological Surgery, 23(6), 423–428. Schroter, S., Black, N., Evans, S., Carpenter, J., Godlee, F., & Smith, R. (2004). Effects of training on quality of peer review: Randomised controlled trial. British Medical Journal, 328(7441), 673. Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., & Smith, R. (2008). What errors do peer reviewers detect, and does training improve their ability to detect them? Journal of the Royal Society of Medicine, 101(10), 507–514. Schroter, S., Tite, L., Hutchings, A., & Black, N. (2006). Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. Journal of the American Medical Association, 295(3), 314–317. Shamoo, A. E., & Resnik, D. B. (2015). Responsible conduct of research (3rd ed.). New York: Oxford University Press. Sismondo, S. (2008). Pharmaceutical company funding and its consequences: A qualitative systematic review. Contempory Clinical Trials, 29(2), 109–113. Smith, R. (1999). Opening up BMJ peer review: A beginning that should lead to complete transparency. British Medical Journal, 318(7175), 4–5. Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182. Stern, J. M., & Simes, R. J. (1997). Publication bias: evidence of delayed publication in a cohort study of clinical research projects. British Medical Journal, 315(7109), 640–645. Strayhorn, J., McDermott, J. F., & Tanguay, P. (1993). An intervention to improve the reliability of manuscript reviews for the Journal of the American Acacemy of Child and Adolescent Psychiatry. Journal of the American Acacemy of Child and Adolescent Psychiatry, 150(6), 947–952. Timmer, A., Hilsden, R. J., & Sutherland, L. R. (2001). Determinants of abstract acceptance for the Digestive Diseases Week–A cross sectional study. BMC Medical Research Methodology, 1, 13. Tregenza, T. (2002). Gender bias in the refereeing process? Trends in Ecology & Evolution, 17(8), 349–350. van Rooyen, S., Delamothe, T., & Evans, S. J. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: Randomised controlled trial. British Medical Journal, 341, c5729. van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers’ recommendations: A randomised trial. British Medical Journal, 318(7175), 23–27. van Rooyen, S., Godlee, F., Evans, S., Smith, R., & Black, N. (1998). Effect of blinding and unmasking on the quality of peer review: A randomized trial. Journal of the American Medical Association, 280(3), 234–237. Wakefield, A. J., Murch, S. H., Anthony, A., Linnell, J., Casson, D. M., Malik, M., et al. (1998). Ileal– lymphoid–nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet, 351(9103), 637–641. Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: a randomised controlled trial. British Journal of Psychiatry, 176, 47–51. Wenneras, C., & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387(6631), 341–343. Whittaker, R. J., et al. (2008). Journal review and gender equality: A critical comment on Budden et al. Trends in Ecology & Evolution, 23, 478–479.

123

D. B. Resnik, S. A. Elmore Wicherts, J. M., Kievit, R. A., Bakker, M., & Borsboom, D. (2012). Letting the daylight in reviewing the reviewers and other ways to maximize transparency in science. Frontiers in Computational Neuroscience, 6, 20. Williams, P., & Wager, E. (2013). Exploring why and how journal editors retract articles: Findings from a qualitative study. Science and Engineering Ethics, 19(1), 1–11. Wing, D. A., Benner, R. S., Petersen, R., Newcomb, R., & Scott, J. R. (2010). Differences in editorial board reviewer behavior based on gender. Journal of Women’s Health, 19(10), 1919–1923.

123

Ensuring the Quality, Fairness, and Integrity of Journal Peer Review: A Possible Role of Editors.

A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, includ...
266KB Sizes 0 Downloads 5 Views