A simple guide to the Research Excellence Framework Professor Alan Glasper and Professor Mandy Fader from the University of Southampton explain the Research Excellence Framework (REF), the new system for assessing the quality of nursing and other subject research in UK higher education. They explain why nursing and healthcare faculties are carrying out this exercise, to be completed on 29 November 2013.
1192
the Bay of Biscay, he decided to investigate the cause of scurvy, a condition caused by nutritional deficiency that killed many sailors and passengers on long voyages at that time (Harvie, 2002). Lind carried out a trial on 12 men with scurvy: one group (n=2) received two oranges and one lemon a day, and the other group (n=12) received non-citrus treatments, including seawater. The two men made a good recovery, whereas the other ten did not. Sadly, despite the evidence, it was to be another 48 years before the British admiralty began issuing lemon juice to naval seamen to prevent scurvy. The contribution of the nursing profession to evidence-based practice can be traced back to Florence Nightingale (McDonald, 2001), who was committed to systematic data collection. She presented data in a way that even a non-statistician could understand and illustrated her work with diagrams. Nightingale is credited with the development of the polar area pie chart to underpin her findings on soldiers’ death rates from infection. Although Nightingale’s contribution to the evidence-based practice movement was significant, much of nursing languished in the doldrums of care delivery based not on best evidence but on custom and practice. Some readers may recall applying surgical spirit to the pressure areas of patients, which, when rubbed vigorously, probably caused the decubitus ulcers (bed sores) it was supposed to prevent—and which were then treated with egg white!
How is evidence sourced? Evidence-based practice is a thoughtful integration of the best available evidence, coupled with clinical expertise and the patient’s wishes. It enables health practitioners such as nurses to address healthcare questions with an evaluative and quantitative approach. It allows the practitioner to assess current and
past research, clinical guidelines, and other information resources to identify relevant literature, while differentiating between high-quality and low-quality findings. It is the production of this literature, and the climate in which it is fostered, that is the source of the new knowledge that the REF seeks to assess.
Assessing the quality of nursing research Every 4–5 years, the Government puts a process of research assessment in place to judge the scope and quality of the research being carried out within UK institutes of higher education. This is not new: the first research assessment exercise (RAE) was done in 1986 and the most recent results were published in 2008. The RAE has been replaced by the REF, which introduces some new measures of excellence that were not considered by the old system. Essentially, the information that is collected is done so over a period of time (in this case from 2008 onwards), whereby a university provides details about the grant funding it has received to do research; about the number of postgraduate students who have completed doctoral studies; and about the research environment it provides, including training and development opportunities for students. The REF also requires information pertinent to the numbers of research-active staff and, importantly, their outputs. This is a very important part of the REF, which measures output primarily as empirical papers published in peer-reviewed academic nursing journals that have the highest impact factor. This, in turn, signposts the average number of references (citations) to recently published articles in, for example, a particular nursing journal, such as the Journal of Advanced Nursing. It is the impact factor of a particular nursing journal that is often used as a barometer of the journal’s importance within the field.
© 2013 MA Healthcare Ltd
C
linicians in nursing practice will no doubt have wondered what their academic colleagues have been doing in recent months, as they have prepared themselves for the forthcoming Research Excellence Framework (REF). Although evidence-based practice has been part of the nursing nomenclature for what seems like some time, it is in fact a relatively new guiding principle. It was Sackett et al (1997) who were among the early proponents of what is now referred to as the ‘evidencebased practice movement’ (Bick and Graham, 2010). Their definition—‘the conscientious, explicit and judicious use of current best evidence about the care of individual patients’ (Sackett et al, 1997)—has since become the mantra of the health professions. As their patients’ advocates, nurses owe it to them to strive for care based on best evidence. In fact, professional regulators such as the Nursing and Midwifery Council (NMC) have this principle at the heart of their constitution. Building health care on the basis of best evidence is not new. According to Reid (2008), the first recorded randomised controlled trial (RCT) is cited in the Old Testament, in the book of Daniel, chapter 1, verses 1–21. It describes the comparison of two groups of Israeli children: the first group eats the meat and wine of the Persian King Nebuchadnezzar; the second group, comprising Daniel and three friends, eats a vegetarian diet of pulses and drinks only water. The aim of the trial is to establish differences in ‘countenance’, which usually refers to appearance, especially of the face, and how it is perceived by others. After 10 days, Daniel and his friends have a better countenance than the children who enjoyed the diet of the king. The first modern RCT to produce ‘evidence’ to answer a clinical question was done by James Lind, a young naval surgeon’s assistant. In 1747, while at sea in
British Journal of Nursing, 2013, Vol 22, No 20
British Journal of Nursing. Downloaded from magonlinelibrary.com by 138.253.100.121 on December 4, 2015. For personal use only. No other uses without permission. . All rights reserved.
© 2013 MA Healthcare Ltd
healthcare policy Consequently, journals with higher impact factors are considered by the REF to be more important than those with lower ones. For nurse academics to be considered in the REF, the usual criterion is that they are classed as someone who is doing research as part of their work. Staff are asked to submit the four best papers that they have generated since 2008. These papers are judged (and rated) in a variety of ways, from being 4*, which is regarded as world-leading research, down to ‘unclassified’, meaning it probably does not count as research. In between, there are various other star ratings, indicating, for example, internationally and nationally recognised research, and research that is of more local significance. The primary aim of an academic nursing unit is to publish as many top-quality papers (3* or 4*) in order to be awarded the optimal rating and a good grade point average. The outcome of the 2013 REF is that faculties will be given a grade point average over all—this represents the sum total of papers submitted in the exercise, which is combined with other metrics of achievement, such as completed PhDs and the amount of research funding received. Individual university nursing and healthcare faculties will also complete a template where they can describe their achievements, their research environment and, importantly, the ways in which research is sustained and supported through the university infrastructure. This written application describes how nursing research is effectively managed and outlines the achievements of that particular faculty over the previous 5 years. The most striking innovation for the 2013 REF is the requirement to provide ‘impact case studies’. For every 10 members of staff submitted in the REF, one impact case study needs to be written.These studies are designed to describe how the nursing research that is being carried out within the institution has made a difference, indirectly or directly, to patient care. Hence the impact case study is not a research outcome in the manner of a research paper: it is a description of how things have changed, for example, in practice or in policy. Fundamentally, this part of the REF seeks assurances that the research being carried out can be shown to have made a difference. Thus ‘impact’ will be credited with significant weight in the overall assessment. This too will be star rated and combined with the other metrics of achievement to allow for the calculation and formulation of the grade
British Journal of Nursing, 2013, Vol 22, No 20
point average awarded to a particular nursing or healthcare faculty. The grade point average will therefore be awarded for an individual unit of assessment. Units of research assessment in the REF range from 1 through to 36 (1 being ‘Clinical Medicine’ and 36 being ‘Communication, Cultural and Media Studies, Library and Information Management’). In this forthcoming REF, nursing research is to be assessed within unit 3, ‘Allied Health Professions, Dentistry, Nursing and Pharmacy’. An expert subpanel for each unit of assessment will assess individual submissions, working under the leadership and guidance of four main panels (A–D). For example, panel A, which will lead on the assessment of subjects such as nursing, medicine and biological sciences, will be chaired by Professor Stephen Holgate from the University of Southampton. Professor Hugh McKenna, Pro-Vice-Chancellor for Research and Innovation at the University of Ulster, will lead unit 3. The 2014 REF is a major change from that of the past. Academic nurses are now required to think more about their research from the grant-gaining stage through to how it will make a difference to their profession in practice. In the past, nurse researchers would write a paper, present it at conferences and for many academics, that was the job finished. It was akin to throwing research over a wall and letting a community of people who might gain from that research make the best use of it. That is not good enough in today’s world of health care. If someone is going to fund research, it has to have an impact on practice. This part of the REF assessment now puts the onus on nurse researchers and the faculties in which they work to consider fully how the implementation of findings can be articulated to the communities who are the intended recipients of the research.
The impact factor of the REF is going to change the nature of nursing research. It will also affect how researchers think about their research and how it becomes part of practice. Unlike the Royal Navy of the 18th century, nursing cannot wait 48 years before published research is translated into a change in practice delivery.
Why it is important for a nursing faculty to have a high REF score Nursing is a research-based and now allgraduate entry profession. Its primary mission is to offer the very best patient care, based on the very best evidence. Research-based teaching can only be made possible if undergraduates are taught by nurses who are actually carrying out research that pushes at the frontiers of care delivery. The REF is a very important measure of the success of the nursing profession. In the future, research funders, many of them operating in a climate of financial austerity, will want to know how the intended research will benefit patients or BJN change practice. Bick D and Graham I (2010) Evaluating the Impact of Implementing Evidence-Based Practice. Wiley-Blackwell, Chichester Harvie D (2002) Limeys: the True Story of One Man’s War Against Ignorance, the Establishment and the Deadly Scurvy. Sutton Publishing, Stroud McDonald L (2001) Florence Nightingale and the early origins of evidence-based nursing. Evid Based Nurs 4(3): 68–9 Reid S (2008) Nothing new under the sun. Evid Based Ment Health 11(2): 33–4 Sackett DL, Rosenberg WMC, Haynes RB (1997) EvidenceBased. Medicine: How to Teach EBM. Churchill Livingstone, Edinburgh
Key points n Evidence-based practice is a vital part of contemporary nursing n The Research Excellence Framework (REF) is the new system for assessing the quality of nursing research in UK higher education institutions n The contribution of the nursing profession to evidence-based practice can be traced back to Florence Nightingale n The REF seeks assurances that the nursing research being carried out will make a difference to nursing practice n The REF assessment now puts an onus on nurse researchers and the faculties in which they work to fully consider how implementation of findings can be articulated to the communities who are the intended recipients of the research
1193
British Journal of Nursing. Downloaded from magonlinelibrary.com by 138.253.100.121 on December 4, 2015. For personal use only. No other uses without permission. . All rights reserved.
1192
the Bay of Biscay, he decided to investigate the cause of scurvy, a condition caused by nutritional deficiency that killed many sailors and passengers on long voyages at that time (Harvie, 2002). Lind carried out a trial on 12 men with scurvy: one group (n=2) received two oranges and one lemon a day, and the other group (n=12) received non-citrus treatments, including seawater. The two men made a good recovery, whereas the other ten did not. Sadly, despite the evidence, it was to be another 48 years before the British admiralty began issuing lemon juice to naval seamen to prevent scurvy. The contribution of the nursing profession to evidence-based practice can be traced back to Florence Nightingale (McDonald, 2001), who was committed to systematic data collection. She presented data in a way that even a non-statistician could understand and illustrated her work with diagrams. Nightingale is credited with the development of the polar area pie chart to underpin her findings on soldiers’ death rates from infection. Although Nightingale’s contribution to the evidence-based practice movement was significant, much of nursing languished in the doldrums of care delivery based not on best evidence but on custom and practice. Some readers may recall applying surgical spirit to the pressure areas of patients, which, when rubbed vigorously, probably caused the decubitus ulcers (bed sores) it was supposed to prevent—and which were then treated with egg white!
How is evidence sourced? Evidence-based practice is a thoughtful integration of the best available evidence, coupled with clinical expertise and the patient’s wishes. It enables health practitioners such as nurses to address healthcare questions with an evaluative and quantitative approach. It allows the practitioner to assess current and
past research, clinical guidelines, and other information resources to identify relevant literature, while differentiating between high-quality and low-quality findings. It is the production of this literature, and the climate in which it is fostered, that is the source of the new knowledge that the REF seeks to assess.
Assessing the quality of nursing research Every 4–5 years, the Government puts a process of research assessment in place to judge the scope and quality of the research being carried out within UK institutes of higher education. This is not new: the first research assessment exercise (RAE) was done in 1986 and the most recent results were published in 2008. The RAE has been replaced by the REF, which introduces some new measures of excellence that were not considered by the old system. Essentially, the information that is collected is done so over a period of time (in this case from 2008 onwards), whereby a university provides details about the grant funding it has received to do research; about the number of postgraduate students who have completed doctoral studies; and about the research environment it provides, including training and development opportunities for students. The REF also requires information pertinent to the numbers of research-active staff and, importantly, their outputs. This is a very important part of the REF, which measures output primarily as empirical papers published in peer-reviewed academic nursing journals that have the highest impact factor. This, in turn, signposts the average number of references (citations) to recently published articles in, for example, a particular nursing journal, such as the Journal of Advanced Nursing. It is the impact factor of a particular nursing journal that is often used as a barometer of the journal’s importance within the field.
© 2013 MA Healthcare Ltd
C
linicians in nursing practice will no doubt have wondered what their academic colleagues have been doing in recent months, as they have prepared themselves for the forthcoming Research Excellence Framework (REF). Although evidence-based practice has been part of the nursing nomenclature for what seems like some time, it is in fact a relatively new guiding principle. It was Sackett et al (1997) who were among the early proponents of what is now referred to as the ‘evidencebased practice movement’ (Bick and Graham, 2010). Their definition—‘the conscientious, explicit and judicious use of current best evidence about the care of individual patients’ (Sackett et al, 1997)—has since become the mantra of the health professions. As their patients’ advocates, nurses owe it to them to strive for care based on best evidence. In fact, professional regulators such as the Nursing and Midwifery Council (NMC) have this principle at the heart of their constitution. Building health care on the basis of best evidence is not new. According to Reid (2008), the first recorded randomised controlled trial (RCT) is cited in the Old Testament, in the book of Daniel, chapter 1, verses 1–21. It describes the comparison of two groups of Israeli children: the first group eats the meat and wine of the Persian King Nebuchadnezzar; the second group, comprising Daniel and three friends, eats a vegetarian diet of pulses and drinks only water. The aim of the trial is to establish differences in ‘countenance’, which usually refers to appearance, especially of the face, and how it is perceived by others. After 10 days, Daniel and his friends have a better countenance than the children who enjoyed the diet of the king. The first modern RCT to produce ‘evidence’ to answer a clinical question was done by James Lind, a young naval surgeon’s assistant. In 1747, while at sea in
British Journal of Nursing, 2013, Vol 22, No 20
British Journal of Nursing. Downloaded from magonlinelibrary.com by 138.253.100.121 on December 4, 2015. For personal use only. No other uses without permission. . All rights reserved.
© 2013 MA Healthcare Ltd
healthcare policy Consequently, journals with higher impact factors are considered by the REF to be more important than those with lower ones. For nurse academics to be considered in the REF, the usual criterion is that they are classed as someone who is doing research as part of their work. Staff are asked to submit the four best papers that they have generated since 2008. These papers are judged (and rated) in a variety of ways, from being 4*, which is regarded as world-leading research, down to ‘unclassified’, meaning it probably does not count as research. In between, there are various other star ratings, indicating, for example, internationally and nationally recognised research, and research that is of more local significance. The primary aim of an academic nursing unit is to publish as many top-quality papers (3* or 4*) in order to be awarded the optimal rating and a good grade point average. The outcome of the 2013 REF is that faculties will be given a grade point average over all—this represents the sum total of papers submitted in the exercise, which is combined with other metrics of achievement, such as completed PhDs and the amount of research funding received. Individual university nursing and healthcare faculties will also complete a template where they can describe their achievements, their research environment and, importantly, the ways in which research is sustained and supported through the university infrastructure. This written application describes how nursing research is effectively managed and outlines the achievements of that particular faculty over the previous 5 years. The most striking innovation for the 2013 REF is the requirement to provide ‘impact case studies’. For every 10 members of staff submitted in the REF, one impact case study needs to be written.These studies are designed to describe how the nursing research that is being carried out within the institution has made a difference, indirectly or directly, to patient care. Hence the impact case study is not a research outcome in the manner of a research paper: it is a description of how things have changed, for example, in practice or in policy. Fundamentally, this part of the REF seeks assurances that the research being carried out can be shown to have made a difference. Thus ‘impact’ will be credited with significant weight in the overall assessment. This too will be star rated and combined with the other metrics of achievement to allow for the calculation and formulation of the grade
British Journal of Nursing, 2013, Vol 22, No 20
point average awarded to a particular nursing or healthcare faculty. The grade point average will therefore be awarded for an individual unit of assessment. Units of research assessment in the REF range from 1 through to 36 (1 being ‘Clinical Medicine’ and 36 being ‘Communication, Cultural and Media Studies, Library and Information Management’). In this forthcoming REF, nursing research is to be assessed within unit 3, ‘Allied Health Professions, Dentistry, Nursing and Pharmacy’. An expert subpanel for each unit of assessment will assess individual submissions, working under the leadership and guidance of four main panels (A–D). For example, panel A, which will lead on the assessment of subjects such as nursing, medicine and biological sciences, will be chaired by Professor Stephen Holgate from the University of Southampton. Professor Hugh McKenna, Pro-Vice-Chancellor for Research and Innovation at the University of Ulster, will lead unit 3. The 2014 REF is a major change from that of the past. Academic nurses are now required to think more about their research from the grant-gaining stage through to how it will make a difference to their profession in practice. In the past, nurse researchers would write a paper, present it at conferences and for many academics, that was the job finished. It was akin to throwing research over a wall and letting a community of people who might gain from that research make the best use of it. That is not good enough in today’s world of health care. If someone is going to fund research, it has to have an impact on practice. This part of the REF assessment now puts the onus on nurse researchers and the faculties in which they work to consider fully how the implementation of findings can be articulated to the communities who are the intended recipients of the research.
The impact factor of the REF is going to change the nature of nursing research. It will also affect how researchers think about their research and how it becomes part of practice. Unlike the Royal Navy of the 18th century, nursing cannot wait 48 years before published research is translated into a change in practice delivery.
Why it is important for a nursing faculty to have a high REF score Nursing is a research-based and now allgraduate entry profession. Its primary mission is to offer the very best patient care, based on the very best evidence. Research-based teaching can only be made possible if undergraduates are taught by nurses who are actually carrying out research that pushes at the frontiers of care delivery. The REF is a very important measure of the success of the nursing profession. In the future, research funders, many of them operating in a climate of financial austerity, will want to know how the intended research will benefit patients or BJN change practice. Bick D and Graham I (2010) Evaluating the Impact of Implementing Evidence-Based Practice. Wiley-Blackwell, Chichester Harvie D (2002) Limeys: the True Story of One Man’s War Against Ignorance, the Establishment and the Deadly Scurvy. Sutton Publishing, Stroud McDonald L (2001) Florence Nightingale and the early origins of evidence-based nursing. Evid Based Nurs 4(3): 68–9 Reid S (2008) Nothing new under the sun. Evid Based Ment Health 11(2): 33–4 Sackett DL, Rosenberg WMC, Haynes RB (1997) EvidenceBased. Medicine: How to Teach EBM. Churchill Livingstone, Edinburgh
Key points n Evidence-based practice is a vital part of contemporary nursing n The Research Excellence Framework (REF) is the new system for assessing the quality of nursing research in UK higher education institutions n The contribution of the nursing profession to evidence-based practice can be traced back to Florence Nightingale n The REF seeks assurances that the nursing research being carried out will make a difference to nursing practice n The REF assessment now puts an onus on nurse researchers and the faculties in which they work to fully consider how implementation of findings can be articulated to the communities who are the intended recipients of the research
1193
British Journal of Nursing. Downloaded from magonlinelibrary.com by 138.253.100.121 on December 4, 2015. For personal use only. No other uses without permission. . All rights reserved.
