RESEARCH METHODOLOGY
Instrument development and validation of a quality scale for historical research papers (QSHRP): a pilot study Jacinta Kelly & Roger Watson Accepted for publication 22 February 2014
Correspondence to J. Kelly: e-mail: [email protected] Jacinta Kelly MSc RGN Senior Lecturer Acute Care Department, Faculty of Health, Social Care & Education, Anglia Ruskin University, Cambridge, UK Roger Watson PhD FRCN FAAN Professor in Nursing School of Nursing and Midwifery, University of Hull, UK
K E L L Y J . & W A T S O N R . ( 2 0 1 4 ) Instrument development and validation of a quality scale for historical research papers (QSHRP): a pilot study. Journal of Advanced Nursing 70(12), 2964–2967. doi: 10.1111/jan.12395
Abstract Aim. To report a pilot study for the development and validation of an instrument to measure quality in historical research papers. Background. There are no set criteria to assess historical papers published in nursing journals. Design. A three phase mixed method sequential confirmatory design. Methods. In 2012, we used a three-phase approach to item generation and content evaluation. In phase 1, we consulted nursing historians using an online survey comprising three open-ended questions and revised the items. In phase 2, we evaluated the revised items for relevance with expert historians using a 4-point Likert scale and Content Validity Index calculation. In phase 3, we conducted reliability testing of the instrument using a 3-point Likert scale. Results. In phase 1, 121 responses were generated via the online survey and revised to 40 interrogatively phrased items. In phase 2, five items with an Item Content Validity Index score of ≥07 remained. In phase 3, responses from historians resulted in 100% agreement to questions 1, 2 and 4 and 89% and 78%, respectively, to questions 3 and 5. Conclusion. Items for the QSHRP have been identified, content validated and reliability tested. This scale improves on previous scales, which over-emphasized source criticism. However, a full-scale study is needed with nursing historians to increase its robustness. Keywords: historical research papers, instrument development, midwifery, nursing, pilot, quality, validation
Aim To report the development of an instrument to measure the quality of published historical research papers.
Background Fairman et al. (2009) first drew attention to shortcomings in published historical research where they designated two 2964
papers based on historical research appearing in the American Academy of Nursing’s organ of scholarship Nursing Outlook as ahistorical. According to Fairman, the papers contained factual inaccuracies, selectively drawn sources, poor arguments and badly drawn conclusions. The authors advocated for papers based on historical methods to be reviewed by trained historians. Lynaugh (1996), however, indicated that historians constitute a tiny fraction of all nurses and an equally small fraction of historians. Similarly © 2014 John Wiley & Sons Ltd
JAN: RESEARCH METHODOLOGY
in the UK, a systematic review of historical research papers published from 1976–2011 in JAN concluded that the majority of historical research papers, despite increasingly prescriptive guidelines, failed to report on aspects of methodological rigour specific to historical research (Fealy et al. 2013). A literature review indicated that various tools exist to evaluate historical research (Garraghan 1946, Gottschalk 1950, Hackett Fisher 1970, Shafer 1974, McCullagh 1984, Howell & Prevenier 2001); however, the majority focus on the evaluation of the sources of evidence or source criticism. While it is acknowledged that nothing can replace knowledge, experience and intelligence when reviewing historical research papers, in the absence of a dedicated tool and insufficient trained historians to evaluate historical research papers, this paper presents the development and evaluation of an instrument to assess the quality of historical research papers.
Design A mixed method sequential confirmatory design in three phases was employed (Table 1). The first phase was dedicated to item generation and construction, developed from expert review and compared with the literature on criteria for evaluating historical research. The second phase consisted of content evaluation of the items. In the third phase, the revised instrument was subjected to reliability testing.
Methods We used an online survey to generate items. Using a Likert scale in a subsequent online survey, we evaluated the relevance, clarity and completeness of the items using Item Content Validity Index (I-CVI) with expert historians. We tested reliability using our developed scale and a 3-point
Instrument development
Likert scale to evaluate the quality of a nursing history paper.
Sample The first phase of the study was conducted with a purposeful sample of six established nursing and midwifery historians. An established historian was operationally defined as someone who has had training in the methods of historical research and has conducted large-scale historical research. Respondents were recruited with the aid of the United Kingdom Association for History of Nursing and the Irish Centre for the History of Nursing Midwifery who kindly distributed study flyers to their members. In the second phase, we used 10 expert historians recruited via the Biomed Experts website and a further 17 with the aid of the delegates list of the History of Nursing conference programme held in Denmark in 2012. Expert historians had conducted extensive research using historical methods.
Ethical considerations The relevant institutional ethics committee was consulted. Research Ethics Committee approval was obtained (REF 11/060). Completion of the surveys was viewed as implied consent. Response anonymity was assured.
Data collection In phase 1, applying three open-ended questions in 2012, we invited established nursing and midwifery historians to participate in an online survey. The questions were fixed as follows: What do you believe are the essentials of historical research? What do you like to see in an historical nursing research paper? What elements satisfy you that an historical research paper is of high quality? The lead
Table 1 Summary of methods and results of 3 phases. Phase 1 Item generation and construction Methods – 6 historians respond to 3 open-ended questions via online survey and 121 responses are generated Analysis – Responses are analysed thematically and compared to literature on existing scales Findings – Responses are revised to 40 interrogatively phrased items Phase 2 Content evaluation Methods – 10 historians evaluate online the relevance of the 40 items Analysis – A 4-point Likert scale is used to calculate the I-CVI Findings – Five items with I-CVI of > 07 remained in the scale Phase 3 Reliability testing Methods – 4 historians read a historical paper and rate its quality using the newly developed 5-item scale Analysis – Using a 3-point Likert scale, a percentage agreement is calculated Findings – Percentage agreement to questions 1, 2 and 4 is 100% with 89% and 78% to questions 3 and 5
© 2014 John Wiley & Sons Ltd
2965
J. Kelly and R. Watson
researcher compiled the online responses. In phase 2, expert nursing historians were invited to evaluate the instrument item by item. The relevance, clarity and completeness of the items relating to the concept of quality in historical research papers was adjudicated by the expert panel with the aid of a 4-point Likert scale from: Mandatory (4); Desirable (3); Useful (2); to Unnecessary (1). Participants were charged with the task of eliminating totally irrelevant items from the instrument and also to re-phrase or supply new wording for items related to the measured constructs where necessary. We refined the instrument further and with the subsequent version we used a 3-point questionnaire, inviting 17 expert historians to test the reliability of the instrument. In this third phase, we invited experts to read a nursing paper commonly held to be of excellent quality based on historical research. Having read the accompanying paper, we asked experts to identify the extent to which they thought each aspect in the instrument was apparent in the paper using a 3-point scale: ‘yes; to a limited extent’ and ‘no’.
Data analysis In the first phase, both the first and second researcher analysed qualitatively the data from the six historians’ open-ended questions. We compared the responses with the available literature on instruments and revised the items accordingly. Statements rendering clarity and guidance to the concept of quality in historical research were entered into the first draft of the instrument. Responses were examined using thematic analysis. Objectives were written and a table of specifications was devised. In phase two, we analysed quantitatively the relevance, clarity and completeness of the revised items. We used the 4-point Likert scale to calculate an I-CVI. According to DeVellis (2011), the minimum number of experts to calculate I-CVI is 5. I-CVI was calculated taking the guidelines of Polit and Beck (2006) into account whereby the proportion of content experts giving items one of the two highest ratings (in this case ‘Mandatory’ and ‘Desirable’) was used. Items with I-CVI ≥07 were retained. Phase 3 data were analysed by percentage agreement with the statements.
Results The distribution of three open-ended questions in phase 1 generated a total of 121 responses from six established nursing historians and a table of specifications consisting of five domains. The revised responses were reduced to 40 items and formatted in the interrogative. In phase 2, the revised items were content evaluated by 10 expert historians. Five items with I-CVI of >07 remained in the scale 2966
Table 2 Quality Scale for Historical Research Papers (QSHRP) 1. Is the report original, containing new material, new answers to old questions, or new questions? 2. Does the report draw on a rich pool and combination of critically treated sources? 3. Is the analysis perceptive, penetrating and open-minded, including author’s cognisance of own possible bias? 4. Is the report written in a lucid, accessible and fair manner, acknowledging historical context and previous scholarship in the field? 5. In a study of contemporary history have ethical issues been addressed?
(Table 2). Only four people responded in phase 3, and the percentage agreement to questions 1, 2 and 4 was 100% with 89% and 78% to questions 3 and 5.
Conclusion Items for the QSHRP have been identified, content validated and reliability tested in consultation with historians. In comparison with existing scales, this scale is beneficial as there is less emphasis on source criticism. However, this scale is limited by the fact that only four historians participated in reliability testing. A larger study is needed with nursing historians to increase its robustness.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Conflict of interest No conflict of interest was declared by the authors in relation to the study itself. Note that Roger Watson is Editorin-Chief of JAN but, in line with usual practice, this paper was subjected to double-blind peer review and was edited by another editor.
Author contributions All authors have agreed on the final version and meet at least one of the following criteria [recommended by the ICMJE (http://www.icmje.org/ethical_1author.html)]:
• •
substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; drafting the article or revising it critically for important intellectual content. © 2014 John Wiley & Sons Ltd
JAN: RESEARCH METHODOLOGY
References DeVellis R. (2011) Scale Development: Theory and Applications. Sage, London. Fairman J., D’Antonio P., Lynaugh J., Keeling A., Lewenson S., Connolly C. & Whelan J. (2009) Letter to the Editor. Nursing Outlook 58, 68. Fealy G., Kelly J. & Watson R. (2013) Legitimacy in legacy: historical scholarship published in Journal of Advanced Nursing 1976–2011. Journal of Advanced Nursing 69(8), 1881–1894. Garraghan G. (1946) A Guide to Historical Method. Fordham University Press, New York. Gottschalk L. (1950) Understanding History: A Primer of Historical Method. Alfred A. Knopf, New York.
Instrument development Hackett Fisher D. (1970) Historians Fallacies: Towards a Logic of Historical Thought. First Harper, NewYork Howell M. & Prevenier W. (2001) From Reliable Sources: An Introduction to Historical Methods. Cornell University Press, Ithaca. Lynaugh J. (1996) Editorial. Nursing History Review 4, 1. McCullagh C.B. (1984) Justifying Historical Descriptions. Cambridge University Press, New York. Polit D.F. & Beck C.T. (2006) The Content Validity Index: are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health 29, 489–497. Shafer R.J. (1974) A Guide to Historical Method. The Dorsey Press, Illinois.
The Journal of Advanced Nursing (JAN) is an international, peer-reviewed, scientific journal. JAN contributes to the advancement of evidence-based nursing, midwifery and health care by disseminating high quality research and scholarship of contemporary relevance and with potential to advance knowledge for practice, education, management or policy. JAN publishes research reviews, original research reports and methodological and theoretical papers. For further information, please visit JAN on the Wiley Online Library website: www.wileyonlinelibrary.com/journal/jan Reasons to publish your work in JAN:
• High-impact forum: the world’s most cited nursing journal, with an Impact Factor of 1·527 – ranked 14/101 in the 2012 ISI Journal Citation Reports © (Nursing (Social Science)).
• Most read nursing journal in the world: over 3 million articles downloaded online per year and accessible in over 10,000 libraries worldwide (including over 3,500 in developing countries with free or low cost access).
• • • •
Fast and easy online submission: online submission at http://mc.manuscriptcentral.com/jan. Positive publishing experience: rapid double-blind peer review with constructive feedback. Rapid online publication in five weeks: average time from final manuscript arriving in production to online publication. Online Open: the option to pay to make your article freely and openly accessible to non-subscribers upon publication on Wiley Online Library, as well as the option to deposit the article in your own or your funding agency’s preferred archive (e.g. PubMed).
© 2014 John Wiley & Sons Ltd
2967
Instrument development and validation of a quality scale for historical research papers (QSHRP): a pilot study Jacinta Kelly & Roger Watson Accepted for publication 22 February 2014
Correspondence to J. Kelly: e-mail: [email protected] Jacinta Kelly MSc RGN Senior Lecturer Acute Care Department, Faculty of Health, Social Care & Education, Anglia Ruskin University, Cambridge, UK Roger Watson PhD FRCN FAAN Professor in Nursing School of Nursing and Midwifery, University of Hull, UK
K E L L Y J . & W A T S O N R . ( 2 0 1 4 ) Instrument development and validation of a quality scale for historical research papers (QSHRP): a pilot study. Journal of Advanced Nursing 70(12), 2964–2967. doi: 10.1111/jan.12395
Abstract Aim. To report a pilot study for the development and validation of an instrument to measure quality in historical research papers. Background. There are no set criteria to assess historical papers published in nursing journals. Design. A three phase mixed method sequential confirmatory design. Methods. In 2012, we used a three-phase approach to item generation and content evaluation. In phase 1, we consulted nursing historians using an online survey comprising three open-ended questions and revised the items. In phase 2, we evaluated the revised items for relevance with expert historians using a 4-point Likert scale and Content Validity Index calculation. In phase 3, we conducted reliability testing of the instrument using a 3-point Likert scale. Results. In phase 1, 121 responses were generated via the online survey and revised to 40 interrogatively phrased items. In phase 2, five items with an Item Content Validity Index score of ≥07 remained. In phase 3, responses from historians resulted in 100% agreement to questions 1, 2 and 4 and 89% and 78%, respectively, to questions 3 and 5. Conclusion. Items for the QSHRP have been identified, content validated and reliability tested. This scale improves on previous scales, which over-emphasized source criticism. However, a full-scale study is needed with nursing historians to increase its robustness. Keywords: historical research papers, instrument development, midwifery, nursing, pilot, quality, validation
Aim To report the development of an instrument to measure the quality of published historical research papers.
Background Fairman et al. (2009) first drew attention to shortcomings in published historical research where they designated two 2964
papers based on historical research appearing in the American Academy of Nursing’s organ of scholarship Nursing Outlook as ahistorical. According to Fairman, the papers contained factual inaccuracies, selectively drawn sources, poor arguments and badly drawn conclusions. The authors advocated for papers based on historical methods to be reviewed by trained historians. Lynaugh (1996), however, indicated that historians constitute a tiny fraction of all nurses and an equally small fraction of historians. Similarly © 2014 John Wiley & Sons Ltd
JAN: RESEARCH METHODOLOGY
in the UK, a systematic review of historical research papers published from 1976–2011 in JAN concluded that the majority of historical research papers, despite increasingly prescriptive guidelines, failed to report on aspects of methodological rigour specific to historical research (Fealy et al. 2013). A literature review indicated that various tools exist to evaluate historical research (Garraghan 1946, Gottschalk 1950, Hackett Fisher 1970, Shafer 1974, McCullagh 1984, Howell & Prevenier 2001); however, the majority focus on the evaluation of the sources of evidence or source criticism. While it is acknowledged that nothing can replace knowledge, experience and intelligence when reviewing historical research papers, in the absence of a dedicated tool and insufficient trained historians to evaluate historical research papers, this paper presents the development and evaluation of an instrument to assess the quality of historical research papers.
Design A mixed method sequential confirmatory design in three phases was employed (Table 1). The first phase was dedicated to item generation and construction, developed from expert review and compared with the literature on criteria for evaluating historical research. The second phase consisted of content evaluation of the items. In the third phase, the revised instrument was subjected to reliability testing.
Methods We used an online survey to generate items. Using a Likert scale in a subsequent online survey, we evaluated the relevance, clarity and completeness of the items using Item Content Validity Index (I-CVI) with expert historians. We tested reliability using our developed scale and a 3-point
Instrument development
Likert scale to evaluate the quality of a nursing history paper.
Sample The first phase of the study was conducted with a purposeful sample of six established nursing and midwifery historians. An established historian was operationally defined as someone who has had training in the methods of historical research and has conducted large-scale historical research. Respondents were recruited with the aid of the United Kingdom Association for History of Nursing and the Irish Centre for the History of Nursing Midwifery who kindly distributed study flyers to their members. In the second phase, we used 10 expert historians recruited via the Biomed Experts website and a further 17 with the aid of the delegates list of the History of Nursing conference programme held in Denmark in 2012. Expert historians had conducted extensive research using historical methods.
Ethical considerations The relevant institutional ethics committee was consulted. Research Ethics Committee approval was obtained (REF 11/060). Completion of the surveys was viewed as implied consent. Response anonymity was assured.
Data collection In phase 1, applying three open-ended questions in 2012, we invited established nursing and midwifery historians to participate in an online survey. The questions were fixed as follows: What do you believe are the essentials of historical research? What do you like to see in an historical nursing research paper? What elements satisfy you that an historical research paper is of high quality? The lead
Table 1 Summary of methods and results of 3 phases. Phase 1 Item generation and construction Methods – 6 historians respond to 3 open-ended questions via online survey and 121 responses are generated Analysis – Responses are analysed thematically and compared to literature on existing scales Findings – Responses are revised to 40 interrogatively phrased items Phase 2 Content evaluation Methods – 10 historians evaluate online the relevance of the 40 items Analysis – A 4-point Likert scale is used to calculate the I-CVI Findings – Five items with I-CVI of > 07 remained in the scale Phase 3 Reliability testing Methods – 4 historians read a historical paper and rate its quality using the newly developed 5-item scale Analysis – Using a 3-point Likert scale, a percentage agreement is calculated Findings – Percentage agreement to questions 1, 2 and 4 is 100% with 89% and 78% to questions 3 and 5
© 2014 John Wiley & Sons Ltd
2965
J. Kelly and R. Watson
researcher compiled the online responses. In phase 2, expert nursing historians were invited to evaluate the instrument item by item. The relevance, clarity and completeness of the items relating to the concept of quality in historical research papers was adjudicated by the expert panel with the aid of a 4-point Likert scale from: Mandatory (4); Desirable (3); Useful (2); to Unnecessary (1). Participants were charged with the task of eliminating totally irrelevant items from the instrument and also to re-phrase or supply new wording for items related to the measured constructs where necessary. We refined the instrument further and with the subsequent version we used a 3-point questionnaire, inviting 17 expert historians to test the reliability of the instrument. In this third phase, we invited experts to read a nursing paper commonly held to be of excellent quality based on historical research. Having read the accompanying paper, we asked experts to identify the extent to which they thought each aspect in the instrument was apparent in the paper using a 3-point scale: ‘yes; to a limited extent’ and ‘no’.
Data analysis In the first phase, both the first and second researcher analysed qualitatively the data from the six historians’ open-ended questions. We compared the responses with the available literature on instruments and revised the items accordingly. Statements rendering clarity and guidance to the concept of quality in historical research were entered into the first draft of the instrument. Responses were examined using thematic analysis. Objectives were written and a table of specifications was devised. In phase two, we analysed quantitatively the relevance, clarity and completeness of the revised items. We used the 4-point Likert scale to calculate an I-CVI. According to DeVellis (2011), the minimum number of experts to calculate I-CVI is 5. I-CVI was calculated taking the guidelines of Polit and Beck (2006) into account whereby the proportion of content experts giving items one of the two highest ratings (in this case ‘Mandatory’ and ‘Desirable’) was used. Items with I-CVI ≥07 were retained. Phase 3 data were analysed by percentage agreement with the statements.
Results The distribution of three open-ended questions in phase 1 generated a total of 121 responses from six established nursing historians and a table of specifications consisting of five domains. The revised responses were reduced to 40 items and formatted in the interrogative. In phase 2, the revised items were content evaluated by 10 expert historians. Five items with I-CVI of >07 remained in the scale 2966
Table 2 Quality Scale for Historical Research Papers (QSHRP) 1. Is the report original, containing new material, new answers to old questions, or new questions? 2. Does the report draw on a rich pool and combination of critically treated sources? 3. Is the analysis perceptive, penetrating and open-minded, including author’s cognisance of own possible bias? 4. Is the report written in a lucid, accessible and fair manner, acknowledging historical context and previous scholarship in the field? 5. In a study of contemporary history have ethical issues been addressed?
(Table 2). Only four people responded in phase 3, and the percentage agreement to questions 1, 2 and 4 was 100% with 89% and 78% to questions 3 and 5.
Conclusion Items for the QSHRP have been identified, content validated and reliability tested in consultation with historians. In comparison with existing scales, this scale is beneficial as there is less emphasis on source criticism. However, this scale is limited by the fact that only four historians participated in reliability testing. A larger study is needed with nursing historians to increase its robustness.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Conflict of interest No conflict of interest was declared by the authors in relation to the study itself. Note that Roger Watson is Editorin-Chief of JAN but, in line with usual practice, this paper was subjected to double-blind peer review and was edited by another editor.
Author contributions All authors have agreed on the final version and meet at least one of the following criteria [recommended by the ICMJE (http://www.icmje.org/ethical_1author.html)]:
• •
substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; drafting the article or revising it critically for important intellectual content. © 2014 John Wiley & Sons Ltd
JAN: RESEARCH METHODOLOGY
References DeVellis R. (2011) Scale Development: Theory and Applications. Sage, London. Fairman J., D’Antonio P., Lynaugh J., Keeling A., Lewenson S., Connolly C. & Whelan J. (2009) Letter to the Editor. Nursing Outlook 58, 68. Fealy G., Kelly J. & Watson R. (2013) Legitimacy in legacy: historical scholarship published in Journal of Advanced Nursing 1976–2011. Journal of Advanced Nursing 69(8), 1881–1894. Garraghan G. (1946) A Guide to Historical Method. Fordham University Press, New York. Gottschalk L. (1950) Understanding History: A Primer of Historical Method. Alfred A. Knopf, New York.
Instrument development Hackett Fisher D. (1970) Historians Fallacies: Towards a Logic of Historical Thought. First Harper, NewYork Howell M. & Prevenier W. (2001) From Reliable Sources: An Introduction to Historical Methods. Cornell University Press, Ithaca. Lynaugh J. (1996) Editorial. Nursing History Review 4, 1. McCullagh C.B. (1984) Justifying Historical Descriptions. Cambridge University Press, New York. Polit D.F. & Beck C.T. (2006) The Content Validity Index: are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health 29, 489–497. Shafer R.J. (1974) A Guide to Historical Method. The Dorsey Press, Illinois.
The Journal of Advanced Nursing (JAN) is an international, peer-reviewed, scientific journal. JAN contributes to the advancement of evidence-based nursing, midwifery and health care by disseminating high quality research and scholarship of contemporary relevance and with potential to advance knowledge for practice, education, management or policy. JAN publishes research reviews, original research reports and methodological and theoretical papers. For further information, please visit JAN on the Wiley Online Library website: www.wileyonlinelibrary.com/journal/jan Reasons to publish your work in JAN:
• High-impact forum: the world’s most cited nursing journal, with an Impact Factor of 1·527 – ranked 14/101 in the 2012 ISI Journal Citation Reports © (Nursing (Social Science)).
• Most read nursing journal in the world: over 3 million articles downloaded online per year and accessible in over 10,000 libraries worldwide (including over 3,500 in developing countries with free or low cost access).
• • • •
Fast and easy online submission: online submission at http://mc.manuscriptcentral.com/jan. Positive publishing experience: rapid double-blind peer review with constructive feedback. Rapid online publication in five weeks: average time from final manuscript arriving in production to online publication. Online Open: the option to pay to make your article freely and openly accessible to non-subscribers upon publication on Wiley Online Library, as well as the option to deposit the article in your own or your funding agency’s preferred archive (e.g. PubMed).
© 2014 John Wiley & Sons Ltd
2967
