EDITORIAL
Some thoughts on how to do good quality peer review In a previous editorial, I commented on the importance of peer review in maintaining the integrity of scholarly journals (Pierson, 2014). Subsequent to writing that editorial I came across an interesting blog post (Raff, 2013) outlining a systematic approach to performing good peer review. In my capacity as an elected member of the Council for the Committee on Publication Ethics (COPE) I have also reviewed cases of serious breaches of ethics regarding the conduct of peer review; those cases stimulated the development of the COPE Guidelines for Peer Reviewers (Hames, 2013). The common elements among these documents are the importance of doing a thorough job of peer review, and doing a thorough job requires a significant investment of time and effort. Most reviewers are not compensated for this time and effort, yet journal editors do find willing and competent reviewers in most cases. After thinking about this issue of peer review for JAANP, I decided to develop a sort of checklist that would be relevant to research and review articles typically submitted to JAANP. I based the elements of this checklist on Raff’s (2013) posting, the COPE guidelines (Hames, 2013), and my own experiences reviewing and editing the JAANP. Some journals do use checklists for reviews, but we typically do not. We use an open comments box for reviewers to post their reviews. It is sometimes difficult for reviewers to organize their comments, so I offer this “checklist” along with my comments (in italics) as a guide to the major issues that should be addressed in reviewing research and review articles for JAANP. A request to serve as a peer reviewer is not a request to edit a manuscript – that is the job of the editor. It is important that reviewers make substantive statements about many aspects of the manuscript in their reviews to help the editor make a decision and to guide authors in making their revisions. Many reviewers have difficulty organizing their comments when they review research and reviews; the checklist below is a guide to help focus attention on some critical aspects of a review. (Note: research includes original qualitative and quantitative research and can be either a primary or secondary analysis; reviews include systematic and non-systematic reviews and may or may not include a meta-analysis.)
Reviewing checklist for JAANP for Research and Review articles
r
r
r
r
r
C 2014 The Author(s) Journal of the American Association of Nurse Practitioners 26 (2014) 57–58
C 2014 American Association of Nurse Practitioners
Is the research/review original and of sufficient importance to advance practice in a meaningful way? If the question has already been answered and widely accepted and the paper offers no new insights or challenges to accepted practice, the paper does not need to be published in a scholarly journal. Is this paper appropriate for JAANP? Consider the practice of NPs and the content of the paper to determine if there is a match, even if the match only applies to a small subset of specialty NPs. Reviewers can determine that the paper is appropriate to NP practice but the editor must decide if there is broad applicability to a majority of the readers. Is the current state of knowledge about the topic accurately represented in the manuscript? If not, what specific area is missing? The introduction and background should clearly state what is known about the topic to justify the value of the research/review. Both what is known and what is not known are important and help to place the current work in perspective. What specific question or gap in knowledge does this research/review address? The authors should clearly state the question or purpose of the study/review based on what is known and not known at the current time. Were the methods adequately described and was the method appropriate to answer the question posed? ◦ For reviews, was the method for article retrieval systematic and was the evidence rated in a standard manner? Systematic reviews are preferred as the most unbiased way to answer clinical questions so that negative or conflicting results are not dismissed. The preferred method for reporting systematic reviews is the PRISMA guideline (http://www.prismastatement.org/). ◦ For research, were the design, sample selection, and analysis plan appropriate to answer the research question? A basic understanding of methods and analysis is helpful here, but common sense can go a long way. For example, distributing
57
Editorial
r
r
r r
r
r
58
a set of 5 questionnaires to 12 people at a health fair is unlikely to be an adequate methodology to answer a research question. Sometimes the research is well designed but not well written; using appropriate reporting guidelines such as those found at the Equator Network (http://www.equator-network.org/) can support the writing process. Were ethical procedures followed? ◦ For research was there adequate protection of human subjects and patient confidentiality? Research must be approved by an appropriate review board and informed consent must be addressed in the methods. Individual subjects should not be identifiable in the report. ◦ For reviews, was there any evidence of bias for or against specific outcomes (positive or negative)? Deliberately ignoring contradictory findings to answer a clinical question is unethical; negative results may be more difficult to find but some attempt should be made, for example by contacting an expert in the field. Were the statistical analyses appropriate to the data collected? Usually when advanced statistics are reported, the editor will find a reviewer who is an expert in methodology and statistics. If you are not knowledgeable about the analysis used and cannot offer any opinion, state this in your review. If the manuscript reports a meta-analysis, were the subjects appropriately pooled to do the analysis? If you are not knowledgeable about meta-analysis and cannot offer any opinion, state this in your review. Are the results reasonable given the design and are they correctly interpreted by the author? Overall, you should have developed a sense of the research or review given your expertise and your knowledge of how science is produced. Sometimes authors go beyond their findings in their enthusiasm and interpretation of their results; this is not acceptable for a scholarly article. Do figures and tables contribute to the paper? Is there anything that is potentially misleading to readers? Tables and figures are a convenient way to display data; however, they can mislead readers into thinking something is more important than it is. Careful reporting of error margins and confidence intervals in all graphics assure that results will not be misrepresented. Are the interpretations of the findings backed by the data? Authors must not go beyond the data in their
C. A. Pierson
r
r
interpretations of the findings or the significance of any findings, either positive or negative. One study rarely proves a theory. Have the authors correctly placed the results in the context of current knowledge? The discussion section should relate the authors’ work to the extant literature showing how their findings concur or deviate from the current state of knowledge. This cannot be adequately done in a summary sentence (or conclusion). There should also be a summary of what remains to be discovered (future research) on the topic. Have the authors included and correctly interpreted the findings in relation to NP practice? In a practice-driven discipline, scholarly journals like JAANP strive to present useful information to drive evidencebased practice. All articles we publish foster this practice and when implications for NP practice are missing, authors need to be reminded that as NPs, we value good research and reviews that help us make the best decisions for our patients.
In summary, editors value good reviewers and authors are grateful for helpful suggestions to make their articles more readable, relevant, and scholarly. Reviewing is a good way to learn about good writing and to stay current in your area of expertise. Students may or may not learn how to review for journals in their graduate programs, but many are mentored in the process by faculty or coworkers. If you have been mentored or taught how to review, you are fortunate; if you have mentored or taught others how to review, you are remembered and highly valued by those who have benefited from your teaching. Charon A. Pierson, PhD, GNP, FAANP, FAAN Editor-in-Chief
References Hames, I., on behalf of COPE Council. (March 2013, version 1.) Committee on publication ethics: Guidelines for Peer Reviewers http://publicationethics.org/files/u7140/Peer%20review%20guidelines.pdf. Accessed December 22, 2013. Pierson, C. (2014). Fake science and peer review: Who is minding the gate? Journal of the American Association of Nurse Practitioners, 26 (1), 1–2. Raff, J. (December 13, 2013) How to become good at peer review: A guide for young scientists. Violent metaphors.com http://violentmetaphors.com/2013/12/13/how-to-become-good-at-peerreview-a-guide-for-young-scientists/#more-971. Accessed December 22, 2013.
Some thoughts on how to do good quality peer review In a previous editorial, I commented on the importance of peer review in maintaining the integrity of scholarly journals (Pierson, 2014). Subsequent to writing that editorial I came across an interesting blog post (Raff, 2013) outlining a systematic approach to performing good peer review. In my capacity as an elected member of the Council for the Committee on Publication Ethics (COPE) I have also reviewed cases of serious breaches of ethics regarding the conduct of peer review; those cases stimulated the development of the COPE Guidelines for Peer Reviewers (Hames, 2013). The common elements among these documents are the importance of doing a thorough job of peer review, and doing a thorough job requires a significant investment of time and effort. Most reviewers are not compensated for this time and effort, yet journal editors do find willing and competent reviewers in most cases. After thinking about this issue of peer review for JAANP, I decided to develop a sort of checklist that would be relevant to research and review articles typically submitted to JAANP. I based the elements of this checklist on Raff’s (2013) posting, the COPE guidelines (Hames, 2013), and my own experiences reviewing and editing the JAANP. Some journals do use checklists for reviews, but we typically do not. We use an open comments box for reviewers to post their reviews. It is sometimes difficult for reviewers to organize their comments, so I offer this “checklist” along with my comments (in italics) as a guide to the major issues that should be addressed in reviewing research and review articles for JAANP. A request to serve as a peer reviewer is not a request to edit a manuscript – that is the job of the editor. It is important that reviewers make substantive statements about many aspects of the manuscript in their reviews to help the editor make a decision and to guide authors in making their revisions. Many reviewers have difficulty organizing their comments when they review research and reviews; the checklist below is a guide to help focus attention on some critical aspects of a review. (Note: research includes original qualitative and quantitative research and can be either a primary or secondary analysis; reviews include systematic and non-systematic reviews and may or may not include a meta-analysis.)
Reviewing checklist for JAANP for Research and Review articles
r
r
r
r
r
C 2014 The Author(s) Journal of the American Association of Nurse Practitioners 26 (2014) 57–58
C 2014 American Association of Nurse Practitioners
Is the research/review original and of sufficient importance to advance practice in a meaningful way? If the question has already been answered and widely accepted and the paper offers no new insights or challenges to accepted practice, the paper does not need to be published in a scholarly journal. Is this paper appropriate for JAANP? Consider the practice of NPs and the content of the paper to determine if there is a match, even if the match only applies to a small subset of specialty NPs. Reviewers can determine that the paper is appropriate to NP practice but the editor must decide if there is broad applicability to a majority of the readers. Is the current state of knowledge about the topic accurately represented in the manuscript? If not, what specific area is missing? The introduction and background should clearly state what is known about the topic to justify the value of the research/review. Both what is known and what is not known are important and help to place the current work in perspective. What specific question or gap in knowledge does this research/review address? The authors should clearly state the question or purpose of the study/review based on what is known and not known at the current time. Were the methods adequately described and was the method appropriate to answer the question posed? ◦ For reviews, was the method for article retrieval systematic and was the evidence rated in a standard manner? Systematic reviews are preferred as the most unbiased way to answer clinical questions so that negative or conflicting results are not dismissed. The preferred method for reporting systematic reviews is the PRISMA guideline (http://www.prismastatement.org/). ◦ For research, were the design, sample selection, and analysis plan appropriate to answer the research question? A basic understanding of methods and analysis is helpful here, but common sense can go a long way. For example, distributing
57
Editorial
r
r
r r
r
r
58
a set of 5 questionnaires to 12 people at a health fair is unlikely to be an adequate methodology to answer a research question. Sometimes the research is well designed but not well written; using appropriate reporting guidelines such as those found at the Equator Network (http://www.equator-network.org/) can support the writing process. Were ethical procedures followed? ◦ For research was there adequate protection of human subjects and patient confidentiality? Research must be approved by an appropriate review board and informed consent must be addressed in the methods. Individual subjects should not be identifiable in the report. ◦ For reviews, was there any evidence of bias for or against specific outcomes (positive or negative)? Deliberately ignoring contradictory findings to answer a clinical question is unethical; negative results may be more difficult to find but some attempt should be made, for example by contacting an expert in the field. Were the statistical analyses appropriate to the data collected? Usually when advanced statistics are reported, the editor will find a reviewer who is an expert in methodology and statistics. If you are not knowledgeable about the analysis used and cannot offer any opinion, state this in your review. If the manuscript reports a meta-analysis, were the subjects appropriately pooled to do the analysis? If you are not knowledgeable about meta-analysis and cannot offer any opinion, state this in your review. Are the results reasonable given the design and are they correctly interpreted by the author? Overall, you should have developed a sense of the research or review given your expertise and your knowledge of how science is produced. Sometimes authors go beyond their findings in their enthusiasm and interpretation of their results; this is not acceptable for a scholarly article. Do figures and tables contribute to the paper? Is there anything that is potentially misleading to readers? Tables and figures are a convenient way to display data; however, they can mislead readers into thinking something is more important than it is. Careful reporting of error margins and confidence intervals in all graphics assure that results will not be misrepresented. Are the interpretations of the findings backed by the data? Authors must not go beyond the data in their
C. A. Pierson
r
r
interpretations of the findings or the significance of any findings, either positive or negative. One study rarely proves a theory. Have the authors correctly placed the results in the context of current knowledge? The discussion section should relate the authors’ work to the extant literature showing how their findings concur or deviate from the current state of knowledge. This cannot be adequately done in a summary sentence (or conclusion). There should also be a summary of what remains to be discovered (future research) on the topic. Have the authors included and correctly interpreted the findings in relation to NP practice? In a practice-driven discipline, scholarly journals like JAANP strive to present useful information to drive evidencebased practice. All articles we publish foster this practice and when implications for NP practice are missing, authors need to be reminded that as NPs, we value good research and reviews that help us make the best decisions for our patients.
In summary, editors value good reviewers and authors are grateful for helpful suggestions to make their articles more readable, relevant, and scholarly. Reviewing is a good way to learn about good writing and to stay current in your area of expertise. Students may or may not learn how to review for journals in their graduate programs, but many are mentored in the process by faculty or coworkers. If you have been mentored or taught how to review, you are fortunate; if you have mentored or taught others how to review, you are remembered and highly valued by those who have benefited from your teaching. Charon A. Pierson, PhD, GNP, FAANP, FAAN Editor-in-Chief
References Hames, I., on behalf of COPE Council. (March 2013, version 1.) Committee on publication ethics: Guidelines for Peer Reviewers http://publicationethics.org/files/u7140/Peer%20review%20guidelines.pdf. Accessed December 22, 2013. Pierson, C. (2014). Fake science and peer review: Who is minding the gate? Journal of the American Association of Nurse Practitioners, 26 (1), 1–2. Raff, J. (December 13, 2013) How to become good at peer review: A guide for young scientists. Violent metaphors.com http://violentmetaphors.com/2013/12/13/how-to-become-good-at-peerreview-a-guide-for-young-scientists/#more-971. Accessed December 22, 2013.
