Applied Nursing Research 26 (2013) e5–e9
Contents lists available at ScienceDirect
Applied Nursing Research journal homepage: www.elsevier.com/locate/apnr
Clinical Methods
Content validity of the Spanish version of the Practice Environment Scale of the Nursing Work Index María Isabel Orts-Cortés, PHD a,⁎, Teresa Moreno-Casbas, PHD b, Allison Squires, PHD c, Carmen Fuentelsaz-Gallego, PHD d, Loreto Maciá-Soler, PHD e, Esther González-María, MSc f en nombre del consorcio RN4CAST-España and RN4CAST Team 1 a
Department of Nursing, University Jaume I, Castellón, Spain Unidad de Investigación en Cuidados (Investén-isciii), Instituto de Salud Carlos III, Madrid, Spain c New York University College of Nursing-Global Health Division, New York, NY 10003 d Hospital Universitario Vall d'Hebron, Universidad Autónoma de Barcelona, Barcelona, Spain e Department of Nursing, University Jaume I, Castellón, Spain f Unidad de Investigación en Cuidados (Investén-isciii), Instituto de Salud Carlos III, Madrid, Spain b
a r t i c l e
i n f o
Article history: Received 24 May 2013 Revised 19 August 2013 Accepted 28 August 2013 Keywords: Content validity indexing Working environment Translating Language Nursing
a b s t r a c t Objective: The objective of this study is to evaluate the content validity of the Iberian Spanish version of the questionnaire The Practice Environment Scale of the Nursing Work Index (PES-NWI) by using the Content Validity Indexing (CVI). Methods: A descriptive cross-sectional observational study was conducted. The Spanish version of the questionnaire was translated from the American English instrument through forward and back translation processes. Experts evaluated the translated items through content validity indexing. Once the assessments were completed, CVI indicators were calculated: number of agreements, item Content Validity Index and overall content validity and modified kappa coefficient of the instrument. Results: The overall content validity of the instrument was 0.82. The average modified kappa coefficient of the items was 0.80, with a rating of ‘excellent’. Only 4 of the items were rated as weak or poor. Conclusions: The study demonstrates that the content validity of the Spanish version of the PES-NWI is acceptable. Some results indicate that some items have cross-cultural applicability challenges that need to be addressed in future research studies. Use of the instrument in other Spanish language speaking countries should be taken with caution since some words may not reflect the language of the healthcare systems there. © 2013 Elsevier Inc. All rights reserved.
1. Introduction The Practice Environment Scale-Nursing Work Index (PES-NWI) is a scale used to measure the nursing practice environment. It was developed by Lake in 2002 (Lake, 2002) from two previous This article is part of the doctoral thesis content validity of the practice environment scale of the nursing work index (PES-NWI) in Europe, presented in the doctoral program of health sciences, University of Alicante. Finding: By the European Union's Seventh Framework Programme FP7/2007-2013 under grant agreement no. 223468 and the Spanish Ministry of Science and Innovation (FIS PI080599). ⁎ Corresponding author. University Jaume I Department of Nursing Avgda. Sos Baynat, s/n 12071 - Castellón de la Plana Castellón de la Plana Spain. Tel.: +34 964387728. E-mail address: [email protected] (M.I. Orts-Cortés). 1 For the RN4CAST-España Consortium: Carmen Fuentelsaz-Gallego, Teresa MorenoCasbas, Esther González-María, Teresa Gómez-García, Ana María González-Martín, Beatriz Gómez-González, Cristina Quesada-Ramos, Eduardo Candel-Parra, Enrique Oltra-Rodríguez, Enrique Terol-García, Eva Abad-Corpa, Francisco Javier Guerra-Bernal, Francisco Lozano-Olea, José Rodríguez-Escobar, Lara Martínez-Gimeno, Lena FerrúsEstopa, Leticia San Martín-Rodríguez, María Isabel Orts-Cortés, Manuel Rich-Ruiz, Marta Alonso-Carreño, Martín Rodríguez-Álvaro, Mónica Contreras-Moreira, Pablo Uriel-Latorre, Silvia Esteban-Sepúlveda y Tomás Sebastián-Viana. For the RN4CAST team, please visit (http://www.rn4cast.eu/). 0897-1897/$ – see front matter © 2013 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.apnr.2013.08.006
instruments: the NWI, developed in 1989 (Kramer & Hafner, 1989) and the NWI-R, published in 2000 (Aiken, Havens, & Sloane, 2000). This index seems to be one of the best instruments to measure the nursing practice environment due to its appropriateness, its structure, and its ability to discriminate Magnet-like hospitals. It has also been able to relate patients' health outcomes with the nursing practice by means of cross-sectional studies. The PES-NWI seems to be one of the most promising instruments to cross-culturally measure the work environments of nurses. It appears cross-culturally applicable (content validity), the structure is adequate to measure differences between health systems and hospital organization (construct validity), it has the ability to discriminate Magnet hospitals (MH) (discriminant validity) and it has an association with health outcomes in cross-sectional studies (concurrent validity). More studies, however, are necessary to establish the predictive validity of the instrument outside the United States where it was first developed (Bonneterre, Liaudy, Chatellier, Lang, & de Gaudemaris, 2008). Researchers have used the PES-NWI-R in many different countries and contexts of clinical practice to evaluate the practice environments of nurses (Warshawsky & Havens, 2011).
e6
M.I. Orts-Cortés et al. / Applied Nursing Research 26 (2013) e5–e9
Researchers had previously translated and evaluated a Spanish version of the questionnaire (Fuentelsaz-Gallego, Moreno-Casbas & Gonzalez-Maria, 2013). Content validity of the translated instrument, however, was not well established in the previous study. For the European project, Nurse Forecasting: Human Resources Planning in Nursing (RN4CAST) 2—which aimed to establish how the features of nursing work environment and training impact on productivity, nurse retention and patient outcomes in general acute hospitals from 11 European countries (Spain, Belgium, Germany, Finland, Greece, Ireland, Netherlands, Poland, Switzerland, Sweden and United Kingdom) (Sermeus et al., 2011)—a more rigorous translation was required to ensure that the instrument produced comparable results across the 11 countries involved in the study. A sample of hospitals from the United States was also included for comparison in the study (Aiken et al., 2012). This study describes the process involved with establishing the content validity of the Iberian Spanish version of the PES-NWI for the RN4CAST study. Researchers increasingly translate instruments for cross-cultural research in the field of nursing, but the translation approach for these instruments is not always rigorous. A review study that identified and classified translated instruments used in quantitative research in the field of nursing and indexed in the CINAHL database concluded that widely different translation approaches were used in the included studies and many lacked an appropriate assessment of the quality of the instrument's translation in the target language (Maneesriwongul & Dixon, 2004). The development of cross-culturally relevant and rigorously translated instruments for international research is necessary and has been theoretically and empirically studied by researchers from many disciplines for over 25 years (Chen & Boore, 2010; Maneesriwongul & Dixon, 2004). First, the use of an instrument in different cultures requires an assessment of its validity or the cultural equivalence of the translated instrument. Flaherty and colleagues (Flaherty et al., 1988) proposed five dimensions to establish crosscultural equivalence: content, semantic, technical, criterion and conceptual. In addition to Flaherty and colleagues' recommendations, experts agree that the selection of an instrument for international research must be carried out in accordance with consideration of the following: instruments showing cross-cultural equivalence; instruments with good psychometric properties thoroughly tested in a particular context/culture, but not in others; and instruments with an apparent high validity but which require further psychometric studies in the country of origin where it was developed in order to make a subsequent cross-cultural adaptation. In the absence of such instruments, a new one must be developed by researchers, testing its psychometric properties and its subsequent adaptation process. The quality of the data derived from translated instruments is based on the accuracy of the adaptation process applied (Martin, Vincenzi, & Spirig, 2007). Therefore, the selection of the measuring instrument for international research must be carried out in accordance with the following priorities: instruments showing cross-cultural equivalence; instruments with good psychometric properties thoroughly tested in a particular context/culture, but not in others; instruments with an apparent high validity but which require further psychometric studies in the country of origin where it was developed in order to make a subsequent cross-cultural adaptation; and in the absence of such instruments, a new one must be developed by researchers, testing its psychometric properties and its subsequent adaptation process. Content Validity Indexing (CVI) is one of the most important aspects in the adaptation and validation of measuring instruments, especially if they are to be used in international research with samples from different languages, or are to be applied in different contexts (Squires et al., 2013). There is a general agreement on the definition of content validity, which refers to the degree to which a sample of 2 Funded by the Seventh Framework Programme: Expt: FP7-HEALTH-2007-B: OPI 1448/08; Proposal Number 223468.
items, as a whole, constitutes an adequate operational definition of a construct (Polit & Beck, 2006). Determining the content validity of an instrument depends largely on the investigator's judgement, although two different methodological phases have been described. First, an a priori approach involves the creation of a new instrument that begins with developing a comprehensive conceptualization of the construct of interest so that the measure would adequately capture the entire domain. Such a conceptualization might come from deep first-hand knowledge, but is more likely to come from a thorough literature review or a qualitative research study (Polit & Beck, 2006). The content validity of an instrument, in contrast, is based on the investigator's discretion since there are no totally objective methods for ensuring the adequate content coverage of an instrument. Including panels of experts in a review of the content area adds rigor to the process (Polit & Beck, 2006). The process requires a minimum of five experts. However, a greater number is usually recommended if the construct is complex. The panel members are asked to evaluate the items of the instrument both separately and as a whole. Some researchers use Content Validity Indexing, an indicator based on the assessment made by expert evaluators about the relevance of the instrument items, which measures their quality separately and as a whole. This, together with the modified kappa coefficient calculation, which establishes the validity of each questionnaire item by adjusting the proper correlation to account for chance agreement among rates, would quantitatively assess this psychometric property (Polit & Beck, 2006; Polit, Beck, & Owen, 2007). This indicator has been used in numerous studies published in nursing journals (Orts-Cortés, 2011) to assess the content validity in the field of health sciences. Finally, the CVI process is an important step in the instrument translation and cross-cultural adaptation process when the target language of translation is spoken in multiple countries, like Spanish. Subtle variations in the language mean that not all words from the translation will work in other countries. This is especially true when the language of health systems is involved. For example, the name for “nurse manager” will be different in Spain than it is in Colombia, Mexico, or Argentina because of how their health systems are organized. In the PES-NWI, ensuring that the role of “nurse manager” is correctly translated to reflect the immediate supervisory role that the American English version asks about is critical for the overall reliability and validity of the instrument. 1.1. Objective The objective of the study is to evaluate the content validity of the Iberian Spanish version of the questionnaire The Practice Environment Scale of the Nursing Work Index (PES-NWI) by using Content Validity Indexing (CVI). 2. Methods For the RN4CAST study, certain modifications were made to Lake's original questionnaire—in American English—of 31 items by the group led by European and American researchers (including the creators of the NWI-R and the PES-NWI). The final version used for the study consisted of 32 items divided into five subscales: (1) Nurse participation in hospital affairs. In this, the role and value of nurses within the hospital setting is analyzed. It aims to identify nurses' perception of their contribution to hospital management, their participation in committees, the role of nursing management and the promotion opportunities (eight items: 5, 6, 11, 16, 18, 23, 25 and 29). (2) Nursing foundations for quality of care. This subscale gathers the issues related to the healthcare philosophy, the care model
M.I. Orts-Cortés et al. / Applied Nursing Research 26 (2013) e5–e9
(physician vs. nurse), the care planning within a unit, the quality assurance and the aspects related to nursing continuing education and guidance offered to newly hired nurses (nine items: 4, 15, 19, 20, 24, 27, 28, 31 and 32). (3) Nurse manager ability, leadership and support of nurses. This subscale is focused on the nursing supervisor's role and the support and recognition they receive (four items: 3, 10, 14 and 22). (4) Staffing and resource adequacy. In this, the key elements to provide quality care in relation to the human resources available are investigated. Sufficient number of nursing staff, allocation of sufficient time for patient care and communication among nurses for patient care (four items: 1, 8, 9 and 12). (5) Collegial nurse–physician relationships. In this subscale, nurses' perception of teamwork and nurse–physician collaboration are examined (seven items: 2, 7, 13, 17, 21, 26 and 30). The Spanish version of the questionnaire was translated using a systematic translation process directed by experts in cross-cultural adaptation of instruments (Squires et al., 2013). The process developed to obtain the experts' assessments and, subsequently, the CVI—the Spanish version of the instrument—is described below: - Identification of ten experts in accordance with the following selection criteria: bilingual nursing graduates (English and first language), who had worked as nurses in their home countries for the past 5 years. Two to three additional assessors were identified to replace the failures (those not being able to complete the questionnaire) among those originally selected. - Once assessors were identified for each research team, they were formally invited to participate in the project by means of a participation request letter sent via email. - A computer application was developed for experts to complete the form on-line or in a Word document. - Evaluators had 15 days to complete the assessment form. - Each expert determined whether each of the items was suitable for the study population and whether the question format was appropriate. This required the items to be evaluated using a four-point ordinal scale from 1 (irrelevant) to 4 (extremely important). The model questionnaire with the question 1 is shown in Table 1.
2.1. Data analysis CVI calculations were performed for each of the items of the instrument (I-CVI). The final average of the I-CVI scores produces a scale-level content validity score (S-CVI). According to the methodology proposed by Lynn (1986) and Polit (Polit & Beck, 2006; Polit et al., 2007), as well as Liu, Squires, & You (2011), the scores for the I-CVI could quantify the five types of validity or equivalence described by Flaherty et al. (1988) in order to evaluate the cross-cultural validity of an instrument. Calculations for obtaining the CVI are shown in Fig. 1.
Table 1 Example of the instrument used to assess the relevance of the PES-NWI items. Use the following scale to evaluate the elements of the item: Item in American English Adequate support services allow me to spend time with my patients
Relevancea Translation (1–4)
Comments/Suggestions
Existen servicios de apoyo adecuados que me permiten pasar tiempo con mis pacientes
a 1 = Irrelevant; 2 = somewhat important; 3 = quite important and 4 = extremely important.
e7
Fig. 1. Calculations for the Content Validity Indexing (Polit et al., 2007).
An item with an I-CVI value of ≥ 0.78 is considered acceptable. The S-CVI minimum scores for evaluating the overall validity of the instrument or of each subscale are ≥0.80 and ≥ 0.90. Scores are then converted to a modified kappa value to account for chance agreement among raters. Fleiss (1981) developed a set of scores for the modified kappa coefficient. The scores divide into a four-part rating scale: excellent (≥0.74), good (0.60 to 0.73), moderate (0.40 to 0.59) and poor (≤0.39). Potentially problematic items (PPI) were questions with a kappa coefficient of less than 0.60. That threshold was set to identify items that might produce outlier responses due to cross-cultural instrument adaptation issues. Microsoft Excel 2007 software was used to for all CVI calculations in the study. 2.1.1. Ethical considerations All ethical principles of scientific biomedical research were observed. The consent of all the participants in the study was also requested. All the projects were approved by the various research ethics committees of the institutions from where the key investigators of each participating country came. 3. Results The overall content validity of the Iberian Spanish version of the PES-NWI (S-CVI) was 0.82, with a minimum I-CVI of 0.20 and the maximum of 1.00. The average modified kappa coefficient of the 32 items was 0.80, with a rating of 'excellent'. An impressive 81.3% (n = 26) of the items were rated excellent, 6.3% (2) were rated good, and a 12.5% (n = 4) of the items were viewed as weak or poor. The S-CVI obtained for each of the subscales was: 0.81 for nurse participation in hospital affairs; 0.86 for nursing foundations for quality of care; 0.85 for nurse manager ability leadership and support of nurses; 0.93 for staffing and resource adequacy; and 0.69 for collegial nurse–physician relationships. Four potentially problematic items (with a modified kappa value b0.60) were identified in the process. One was in the subscale “Nurse participation in hospital affairs”: item 18 - There are promotion opportunities and in three items of subscale “collegial nurse–physician relationships”: item 7 Physicians appreciate nurses' observations and judgements; item 13 Physicians recognize the contribution of nurses to patient care; and item 30 - Physicians have high regard for nurses. The results of the content validity indicators of all the items, based on the subscale they belong to, are shown in Table 2. 4. Discussion The content validation of the Iberian Spanish version of the PESNWI proved a useful exercise before conducting data collection in
e8
M.I. Orts-Cortés et al. / Applied Nursing Research 26 (2013) e5–e9
Table 2 Results of the content validity of the Spanish version of the PES-NWI. Subscale
Item
I-CVI
pc
k
Ratinga
Nurse participation in hospital affairs 5 0.80 0.044 0.79 6 0.90 0.010 0.90 11 0.80 0.044 0.79 16 0.70 0.117 0.66 18 0.50 0.246 0.34 23 1.00 0.001 1.00 25 0.90 0.010 0.90 29 0.90 0.010 0.90 S-CVI (subscale 1) = 0,81
Excellent Excellent Excellent Good Poor Excellent Excellent Excellent
4 1.00 0.001 1.00 15 0.80 0.044 0.79 19 0.80 0.044 0.79 20 0.90 0.010 0.90 24 0.80 0.044 0.79 27 0.90 0.010 0.90 28 0.90 0.010 0.90 31 0.70 0.117 0.66 32 0.90 0.010 0.90 S-CVI (subscale 2) = 0,86 Nurse manager ability leadership and support of nurses 3 0.80 0.044 0.79 10 0.90 0.010 0, 90 14 0.80 0.044 0.79 22 0.90 0.010 0, 90 S-CVI (subscale 3) = 0,85 Staffing and resource adequacy 1 0.80 0.044 0.79 8 0.90 0.010 0, 90 9 1.00 0.001 1.00 12 1.00 0.001 1.00 S-CVI (subscale 4) = 0,93 Collegial nurse–physician relationships 2 0.80 0.044 0.79 - Physicians appreciate nurses' 7 0.60 0.205 0.50 observations and judgements - Physicians recognize the contribution 13 0.50 0.246 0.34 of nurses to patient care 17 0.90 0.010 0.90 21 0.80 0.044 0.79 26 1.00 0.001 1.00 - Physicians have high regard for nurses. 30 0.20 0.044 0.16 S-CVI (subscale 5) = 0,69
Excellent Excellent Excellent Excellent Excellent Excellent Excellent Good Excellent
- There are promotion opportunities
Nursing foundations for quality of care
Excellent Excellent Excellent Excellent
Excellent Excellent Excellent Excellent
Excellent Weak Poor Excellent Excellent Excellent Poor
I-CVI: Item Content Validity Index; Pc: probability of random agreement; K: modified kappa coefficient obtained by designing the relevant proportion of agreements: a Evaluation criteria of k: poor ≤0.39, weak = 0.40–0.59; good = 0.60–0.73; excellent ≥0,74 according to Fleiss (Fleiss, 1981); S-CVI: I-CVI average of the items in the subscale.
Spain for the larger RN4CAST study. Almost the 90% of the items of the instrument were rated as excellent/good. We propose several explanations for the PPIs. First, the item “There are promotion opportunities” obtained a low content validity score. This item is included in the subscale “Nurse participation in hospital affairs”, which is strongly related to the clinical management models and the hierarchical structure within the institution. In the Spanish National Health Service (SNHS) hospitals, management positions are usually appointed, and promotion opportunities are very low. Therefore, it is not a surprise that raters might not consider this item as “relevant” to the Spanish nursing context if overall opportunities for advancement are limited. The other subscale, in which almost half of the items were potentially problematic and resulted in a lower overall (but still “good”) S-CVI rating, was “collegial nurse–physician relationships.” This subscale also had one of the highest percentages of PPI in other language versions of the European study (Orts-Cortés, 2011). During the development of this subscale of the original instrument regarding the nurse practice environment, Kramer & Schalenberg (2004) stated
that staff nurses in Magnet hospitals (MH) described five types of relationships between nurses and physicians: collegial, collaborative, student–teacher, friendly stranger and hostile/adversarial. Four-fifths of the MH nurses in the study reported that the relationship between these two professionals should be a collaborative or collegial one in order to improve patient care. Given that this subscale obtained a lower overall content validity, it would be necessary to investigate the types of relationships in SNHS hospitals and determine if these differ from those in MH, which could be one of the possible reasons for these results. Interestingly, the overall content validity (S-CVI) of the instrument was acceptable and higher than the American English version, which scored 0.70 in the European study (Squires et al., 2013). Overall, the content validity of the Spanish version, according to the modified Kappa coefficient, was rated as excellent, like other versions of the instrument: German (Germany), Dutch (Belgium), Finnish, Greek, Irish, Dutch (Netherlands), English (United Kingdom), Swedish, Norwegian, and Italian and German (Switzerland). Swiss and Belgian French versions obtained a good evaluation. The initial assessment of the Polish version was the only one rated as poor (Orts-Cortés, 2011; Squires et al., 2013). The studies found in literature, in which the cross-cultural adaptation of the PES-NWI or the NWI-R into other languages was carried out, were the following: Spanish (De Pedro Gómez et al., 2009); French (McCusker, Dendukuri, Cardinal, Laplante, & Bambonye, 2004), Chinese (Chiang & Lin, 2009), Iceland (Gunnarsdóttir, Clarke, Rafferty, & Nutbeam, 2009), Portuguese (Gasparino & Guirardello, 2009), Australian English (Middleton, Griffiths, Fernandez, & Smith, 2008) and Dutch (Van Bogaert, Clarke, Vermeyen, Meulemans & Van de Heyning, 2009). From these, only the Chinese version analyzed the content validity of the PES-NWI using the CVI, which resulted in an overall score of 0.94. The assessment was conducted by a total of 5 experts. The smaller number of experts used in the study of Chiang & Lin (2009) may explain the excellent rating obtained by the Chinese version given that the smaller the number of experts, the higher the CVI scores and the modified kappa coefficient (Beckstead, 2009; Polit et al., 2007). A small number of experts also means that all raters must produce 100% agreement to receive an “excellent” score for an item. The limitations of the study relate mostly to the limitations of the CVI approach. Its application as a pre-data collection strategy for analyzing an instrument is relatively new and requires additional replication with other instruments. The predictive validity of the PPIs score has also not been established, but is undergoing further analysis. The CVI approach also does not allow for modifying the content of an item to better fit the context, therefore potentially increasing the CVI score, since the utility of the approach described in this study is in establishing the content validity of the instrument in cross cultural research where producing comparable results are part of the study's design. In a forthcoming research, and in order to improve the content validity of the items with low I-CVI scores or the subscales with scores below the minimum acceptable, qualitative methods should be combined with panels of experts to determine whether the removal or reformulation of items, based on the contextual or cultural characteristics of the place where the instrument is to be used, would be appropriate. An example of this process is included in the study in which the translation and validation process of the Manchester Clinical Supervision Scale (MCSS) from English into Finnish (Hyrkäs, Appelqvist-Schmidlechner, & Oksa, 2003) was analyzed. The content validity of the instrument was analysed using both the quantitative approach, by means of the CVI, and the qualitative approach with a panel of eleven experts. This triangulation process determined the items to be removed, which were rated irrelevant and unsuitable by the experts based on the cultural and contextual characteristics of the target population the instrument was intended for.
M.I. Orts-Cortés et al. / Applied Nursing Research 26 (2013) e5–e9
It is also advisable for each country to establish the content validity by using the methods described in this article and by Squires et al. (2013). Management systems and different resources on the development of the nursing role may change the relevance of questions. In addition, each context may have their own terms for the performance of administrative roles which, if not equivalent, may result in what may seem like “odd” participants' responses when compared to other studies. For example, a nursing director may be responsible for the entire hospital or, in large hospitals, for a group of units or services. It is highly important to ensure that the terms are equivalent in order to obtain comparable results. That is why it is advisable that each country, even if it has the same language as the instrument, conducts its own assessment. Spanish speaking countries in Latin America, therefore, would need to conduct individual content validations of the Iberian translation to ensure that the language conforms to local norms. 5. Conclusions Overall, the content validity of the Iberian Spanish version of the Practice Environment Scale of the Nursing Work Index has been proven acceptable. Additional confirmatory factor analyses will provide further support for the reliability and validity of the instrument in the Iberian context. The results of this research paper indicate the need to further improve the instrument without compromising its cross-cultural applicability. This requires reanalyzing the potentially problematic items and those with a low I-CVI from a qualitative perspective in order to examine the possible reasons for why the experts scored the items and further consideration of the items that could be edited to better suit the cultural, professional and training context of the target population. Other analytic alternatives could include whether the overall results of the scale and subscales are adjusted depending on the removal or modification of these items from the instrument as a whole and, at a more micro level, determining whether these items rate differently in relation to the items which obtained an acceptable or high scores. References Aiken, L., Havens, D., & Sloane, D. (2000). The magnet nursing services recognition program. The American Journal of Nursing, 100(3), 26–35 [quiz 35–6]. Aiken, L. H., Sermeus, W., Van den Heede, K., Sloane, D. M., Busse, R., McKee, M., & Kutney-Lee, A. (2012). Patient safety, satisfaction, and quality of hospital care: Cross sectional surveys of nurses and patients in 12 countries in Europe and the United States. BMJ (Clinical Research Ed.), 344, e1717, http: //dx.doi.org/10.1136/bmj.e1717. Beckstead, J. W. (2009). Content validity is naught. International Journal of Nursing Studies, 46(9), 1274–1283, http://dx.doi.org/10.1016/j.ijnurstu.2009.04.014. Bonneterre, V., Liaudy, S., Chatellier, G., Lang, T., & de Gaudemaris, R. (2008). Reliability, validity, and health issues arising from questionnaires used to measure psychosocial and organizational work factors (POWFs) among hospital nurses: A critical review. Journal of Nursing Measurement, 16(3), 207–230. Chen, H. Y., & Boore, J. R. (2010). Translation and back-translation in qualitative nursing research: Methodological review. Journal of Clinical Nursing, 19(1–2), 234–239, http://dx.doi.org/10.1111/j.1365-2702.2009.02896.x. Chiang, H. Y., & Lin, S. Y. (2009). Psychometric testing of the Chinese version of nursing practice environment scale. Journal of Clinical Nursing, 18(6), 919–929, http: //dx.doi.org/10.1111/j.1365-2702.2008.02433.x.
e9
De Pedro Gómez, J., Morales Asencio, J., Sesé Abad, S., Bennasar Veny, M., Artigues Vives, G., & Pericàs Beltrán, J. (2009). Validación y adaptación al español de la escala del entorno de práctica enfermera del Nursing Work Index. Metas de Enfermería, 12(7), 65–73. Flaherty, J., Gaviria, F., Pathak, D., Mitchell, T., Wintrob, R., Richman, J., & Birz, S. (1988). Developing instruments for cross-cultural psychiatric research. The Journal of Nervous and Mental Disease, 176(5), 257–263. Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.)New York: John Wiley & Sons, Inc. Fuentelsaz-Gallego, C., Moreno-Casbas, M. T., & Gonzalez-Maria, E. (2013). Validation of the Spanish version of the questionnaire practice environment scale of the nursing work index. International Journal of Nursing Studies, 50(2), 274–280, http: //dx.doi.org/10.1016/j.ijnurstu.2012.08.001. Gasparino, R. C., & Guirardello, E. d. B. (2009). Tradução e adaptação para a cultura brasileira do “nursing work index - revised”. Acta Paulista De Enfermagem, 22(3), 281–287. Gunnarsdóttir, S., Clarke, S. P., Rafferty, A. M., & Nutbeam, D. (2009). Front-line management, staffing and nurse-doctor relationships as predictors of nurse and patient outcomes. A survey of Icelandic hospital nurses. International Journal of Nursing Studies, 46(7), 920–927, http://dx.doi.org/10.1016/j.ijnurstu.2006.11.007. Hyrkäs, K., Appelqvist-Schmidlechner, K., & Oksa, L. (2003). Validating an instrument for clinical supervision using an expert panel. International Journal of Nursing Studies, 40(6), 619–625. Kramer, M., & Hafner, L. (1989). Shared values: Impact on staff nurse job satisfaction and perceived productivity. Nursing Research, 38(3), 172–177. Kramer, M., & Schalenberg, C. (2004). Essentials of a magnetic work environment, part 1. Nursing, 34(6), 50–54. Lake, E. (2002). Development of the practice environment scale of the nursing work index. Research in Nursing & Health, 25(3), 176–188. Liu, K., Squires, A., & You, L. (2011). A pilot study of a systematic method for translating patient satisfaction questionnaires. Journal of Advanced Nursing, 67(5), 1012–1021, http://dx.doi.org/10.1111/j.1365-2648.2010.05569.x. Lynn, M. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. Maneesriwongul, W., & Dixon, J. K. (2004). Instrument translation process: A methods review. Journal of Advanced Nursing, 48(2), 175–186. Martin, J. S., Vincenzi, C., & Spirig, R. (2007). Principles and methods of good practice for the translation process for instruments of nursing research and nursing practice [Prinzipien und Methoden einer wissenschaftlich akkuraten Ubersetzungs- praxis von Instrumenten fur Forschung und direkte Pflege]. Pflege, 20(3), 157–163. McCusker, J., Dendukuri, N., Cardinal, L., Laplante, J., & Bambonye, L. (2004). Nursing work environment and quality of care: Differences between units at the same hospital. International Journal of Health Care Quality Assurance Incorporating Leadership in Health Services, 17(6), 313–322. Middleton, S., Griffiths, R., Fernandez, R., & Smith, B. (2008). Nursing practice environment: How does one australian hospital compare with magnet hospitals? International Journal of Nursing Practice, 14(5), 366–372, http: //dx.doi.org/10.1111/j.1440-172X.2008.00708.x. Orts-Cortés MI. (2011). Validez de contenido del Practice Environment Scale of the Nursing Work Index (PES-NWI) en el ámbito europeo. Universidad de Alicante. Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? critique and recommendations. Research in Nursing & Health, 29(5), 489–497. Polit, D. F., Beck, C., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. Sermeus, W., Aiken, L. H., Van den Heede, K., Rafferty, A. M., Griffiths, P., & MorenoCasbas, M. T.RN4CAST consortium. (2011). Nurse forecasting in Europe (RN4CAST): Rationale, design and methodology. BMC Nursing, 10, http://dx.doi.org/10.1186/ 1472-6955-10-6 [6-6955-10-6]. Squires, A., Aiken, L. H., van den Heede, K., Sermeus, W., Bruyneel, L., Lindqvist, R., & Matthews, A. (2013). A systematic survey instrument translation process for multicountry, comparative health workforce studies. International Journal of Nursing Studies, 50(2), 264–273, http://dx.doi.org/10.1016/j.ijnurstu.2012.02.015. Van Bogaert, P., Clarke, S., Vermeyen, K., Meulemans, H., & Van de Heyning, P. (2009). Practice environments and their associations with nurse-reported outcomes in Belgian hospitals: Development and preliminary validation of a Dutch adaptation of the revised nursing work index. International Journal of Nursing Studies, 46(1), 54–64, http://dx.doi.org/10.1016/j.ijnurstu.2008.07.009. Warshawsky, N. E., & Havens, D. S. (2011). Global use of the practice environment scale of the nursing work index. Nursing Research, 60(1), 17–31, http: //dx.doi.org/10.1097/NNR.0b013e3181ffa79c.
Contents lists available at ScienceDirect
Applied Nursing Research journal homepage: www.elsevier.com/locate/apnr
Clinical Methods
Content validity of the Spanish version of the Practice Environment Scale of the Nursing Work Index María Isabel Orts-Cortés, PHD a,⁎, Teresa Moreno-Casbas, PHD b, Allison Squires, PHD c, Carmen Fuentelsaz-Gallego, PHD d, Loreto Maciá-Soler, PHD e, Esther González-María, MSc f en nombre del consorcio RN4CAST-España and RN4CAST Team 1 a
Department of Nursing, University Jaume I, Castellón, Spain Unidad de Investigación en Cuidados (Investén-isciii), Instituto de Salud Carlos III, Madrid, Spain c New York University College of Nursing-Global Health Division, New York, NY 10003 d Hospital Universitario Vall d'Hebron, Universidad Autónoma de Barcelona, Barcelona, Spain e Department of Nursing, University Jaume I, Castellón, Spain f Unidad de Investigación en Cuidados (Investén-isciii), Instituto de Salud Carlos III, Madrid, Spain b
a r t i c l e
i n f o
Article history: Received 24 May 2013 Revised 19 August 2013 Accepted 28 August 2013 Keywords: Content validity indexing Working environment Translating Language Nursing
a b s t r a c t Objective: The objective of this study is to evaluate the content validity of the Iberian Spanish version of the questionnaire The Practice Environment Scale of the Nursing Work Index (PES-NWI) by using the Content Validity Indexing (CVI). Methods: A descriptive cross-sectional observational study was conducted. The Spanish version of the questionnaire was translated from the American English instrument through forward and back translation processes. Experts evaluated the translated items through content validity indexing. Once the assessments were completed, CVI indicators were calculated: number of agreements, item Content Validity Index and overall content validity and modified kappa coefficient of the instrument. Results: The overall content validity of the instrument was 0.82. The average modified kappa coefficient of the items was 0.80, with a rating of ‘excellent’. Only 4 of the items were rated as weak or poor. Conclusions: The study demonstrates that the content validity of the Spanish version of the PES-NWI is acceptable. Some results indicate that some items have cross-cultural applicability challenges that need to be addressed in future research studies. Use of the instrument in other Spanish language speaking countries should be taken with caution since some words may not reflect the language of the healthcare systems there. © 2013 Elsevier Inc. All rights reserved.
1. Introduction The Practice Environment Scale-Nursing Work Index (PES-NWI) is a scale used to measure the nursing practice environment. It was developed by Lake in 2002 (Lake, 2002) from two previous This article is part of the doctoral thesis content validity of the practice environment scale of the nursing work index (PES-NWI) in Europe, presented in the doctoral program of health sciences, University of Alicante. Finding: By the European Union's Seventh Framework Programme FP7/2007-2013 under grant agreement no. 223468 and the Spanish Ministry of Science and Innovation (FIS PI080599). ⁎ Corresponding author. University Jaume I Department of Nursing Avgda. Sos Baynat, s/n 12071 - Castellón de la Plana Castellón de la Plana Spain. Tel.: +34 964387728. E-mail address: [email protected] (M.I. Orts-Cortés). 1 For the RN4CAST-España Consortium: Carmen Fuentelsaz-Gallego, Teresa MorenoCasbas, Esther González-María, Teresa Gómez-García, Ana María González-Martín, Beatriz Gómez-González, Cristina Quesada-Ramos, Eduardo Candel-Parra, Enrique Oltra-Rodríguez, Enrique Terol-García, Eva Abad-Corpa, Francisco Javier Guerra-Bernal, Francisco Lozano-Olea, José Rodríguez-Escobar, Lara Martínez-Gimeno, Lena FerrúsEstopa, Leticia San Martín-Rodríguez, María Isabel Orts-Cortés, Manuel Rich-Ruiz, Marta Alonso-Carreño, Martín Rodríguez-Álvaro, Mónica Contreras-Moreira, Pablo Uriel-Latorre, Silvia Esteban-Sepúlveda y Tomás Sebastián-Viana. For the RN4CAST team, please visit (http://www.rn4cast.eu/). 0897-1897/$ – see front matter © 2013 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.apnr.2013.08.006
instruments: the NWI, developed in 1989 (Kramer & Hafner, 1989) and the NWI-R, published in 2000 (Aiken, Havens, & Sloane, 2000). This index seems to be one of the best instruments to measure the nursing practice environment due to its appropriateness, its structure, and its ability to discriminate Magnet-like hospitals. It has also been able to relate patients' health outcomes with the nursing practice by means of cross-sectional studies. The PES-NWI seems to be one of the most promising instruments to cross-culturally measure the work environments of nurses. It appears cross-culturally applicable (content validity), the structure is adequate to measure differences between health systems and hospital organization (construct validity), it has the ability to discriminate Magnet hospitals (MH) (discriminant validity) and it has an association with health outcomes in cross-sectional studies (concurrent validity). More studies, however, are necessary to establish the predictive validity of the instrument outside the United States where it was first developed (Bonneterre, Liaudy, Chatellier, Lang, & de Gaudemaris, 2008). Researchers have used the PES-NWI-R in many different countries and contexts of clinical practice to evaluate the practice environments of nurses (Warshawsky & Havens, 2011).
e6
M.I. Orts-Cortés et al. / Applied Nursing Research 26 (2013) e5–e9
Researchers had previously translated and evaluated a Spanish version of the questionnaire (Fuentelsaz-Gallego, Moreno-Casbas & Gonzalez-Maria, 2013). Content validity of the translated instrument, however, was not well established in the previous study. For the European project, Nurse Forecasting: Human Resources Planning in Nursing (RN4CAST) 2—which aimed to establish how the features of nursing work environment and training impact on productivity, nurse retention and patient outcomes in general acute hospitals from 11 European countries (Spain, Belgium, Germany, Finland, Greece, Ireland, Netherlands, Poland, Switzerland, Sweden and United Kingdom) (Sermeus et al., 2011)—a more rigorous translation was required to ensure that the instrument produced comparable results across the 11 countries involved in the study. A sample of hospitals from the United States was also included for comparison in the study (Aiken et al., 2012). This study describes the process involved with establishing the content validity of the Iberian Spanish version of the PES-NWI for the RN4CAST study. Researchers increasingly translate instruments for cross-cultural research in the field of nursing, but the translation approach for these instruments is not always rigorous. A review study that identified and classified translated instruments used in quantitative research in the field of nursing and indexed in the CINAHL database concluded that widely different translation approaches were used in the included studies and many lacked an appropriate assessment of the quality of the instrument's translation in the target language (Maneesriwongul & Dixon, 2004). The development of cross-culturally relevant and rigorously translated instruments for international research is necessary and has been theoretically and empirically studied by researchers from many disciplines for over 25 years (Chen & Boore, 2010; Maneesriwongul & Dixon, 2004). First, the use of an instrument in different cultures requires an assessment of its validity or the cultural equivalence of the translated instrument. Flaherty and colleagues (Flaherty et al., 1988) proposed five dimensions to establish crosscultural equivalence: content, semantic, technical, criterion and conceptual. In addition to Flaherty and colleagues' recommendations, experts agree that the selection of an instrument for international research must be carried out in accordance with consideration of the following: instruments showing cross-cultural equivalence; instruments with good psychometric properties thoroughly tested in a particular context/culture, but not in others; and instruments with an apparent high validity but which require further psychometric studies in the country of origin where it was developed in order to make a subsequent cross-cultural adaptation. In the absence of such instruments, a new one must be developed by researchers, testing its psychometric properties and its subsequent adaptation process. The quality of the data derived from translated instruments is based on the accuracy of the adaptation process applied (Martin, Vincenzi, & Spirig, 2007). Therefore, the selection of the measuring instrument for international research must be carried out in accordance with the following priorities: instruments showing cross-cultural equivalence; instruments with good psychometric properties thoroughly tested in a particular context/culture, but not in others; instruments with an apparent high validity but which require further psychometric studies in the country of origin where it was developed in order to make a subsequent cross-cultural adaptation; and in the absence of such instruments, a new one must be developed by researchers, testing its psychometric properties and its subsequent adaptation process. Content Validity Indexing (CVI) is one of the most important aspects in the adaptation and validation of measuring instruments, especially if they are to be used in international research with samples from different languages, or are to be applied in different contexts (Squires et al., 2013). There is a general agreement on the definition of content validity, which refers to the degree to which a sample of 2 Funded by the Seventh Framework Programme: Expt: FP7-HEALTH-2007-B: OPI 1448/08; Proposal Number 223468.
items, as a whole, constitutes an adequate operational definition of a construct (Polit & Beck, 2006). Determining the content validity of an instrument depends largely on the investigator's judgement, although two different methodological phases have been described. First, an a priori approach involves the creation of a new instrument that begins with developing a comprehensive conceptualization of the construct of interest so that the measure would adequately capture the entire domain. Such a conceptualization might come from deep first-hand knowledge, but is more likely to come from a thorough literature review or a qualitative research study (Polit & Beck, 2006). The content validity of an instrument, in contrast, is based on the investigator's discretion since there are no totally objective methods for ensuring the adequate content coverage of an instrument. Including panels of experts in a review of the content area adds rigor to the process (Polit & Beck, 2006). The process requires a minimum of five experts. However, a greater number is usually recommended if the construct is complex. The panel members are asked to evaluate the items of the instrument both separately and as a whole. Some researchers use Content Validity Indexing, an indicator based on the assessment made by expert evaluators about the relevance of the instrument items, which measures their quality separately and as a whole. This, together with the modified kappa coefficient calculation, which establishes the validity of each questionnaire item by adjusting the proper correlation to account for chance agreement among rates, would quantitatively assess this psychometric property (Polit & Beck, 2006; Polit, Beck, & Owen, 2007). This indicator has been used in numerous studies published in nursing journals (Orts-Cortés, 2011) to assess the content validity in the field of health sciences. Finally, the CVI process is an important step in the instrument translation and cross-cultural adaptation process when the target language of translation is spoken in multiple countries, like Spanish. Subtle variations in the language mean that not all words from the translation will work in other countries. This is especially true when the language of health systems is involved. For example, the name for “nurse manager” will be different in Spain than it is in Colombia, Mexico, or Argentina because of how their health systems are organized. In the PES-NWI, ensuring that the role of “nurse manager” is correctly translated to reflect the immediate supervisory role that the American English version asks about is critical for the overall reliability and validity of the instrument. 1.1. Objective The objective of the study is to evaluate the content validity of the Iberian Spanish version of the questionnaire The Practice Environment Scale of the Nursing Work Index (PES-NWI) by using Content Validity Indexing (CVI). 2. Methods For the RN4CAST study, certain modifications were made to Lake's original questionnaire—in American English—of 31 items by the group led by European and American researchers (including the creators of the NWI-R and the PES-NWI). The final version used for the study consisted of 32 items divided into five subscales: (1) Nurse participation in hospital affairs. In this, the role and value of nurses within the hospital setting is analyzed. It aims to identify nurses' perception of their contribution to hospital management, their participation in committees, the role of nursing management and the promotion opportunities (eight items: 5, 6, 11, 16, 18, 23, 25 and 29). (2) Nursing foundations for quality of care. This subscale gathers the issues related to the healthcare philosophy, the care model
M.I. Orts-Cortés et al. / Applied Nursing Research 26 (2013) e5–e9
(physician vs. nurse), the care planning within a unit, the quality assurance and the aspects related to nursing continuing education and guidance offered to newly hired nurses (nine items: 4, 15, 19, 20, 24, 27, 28, 31 and 32). (3) Nurse manager ability, leadership and support of nurses. This subscale is focused on the nursing supervisor's role and the support and recognition they receive (four items: 3, 10, 14 and 22). (4) Staffing and resource adequacy. In this, the key elements to provide quality care in relation to the human resources available are investigated. Sufficient number of nursing staff, allocation of sufficient time for patient care and communication among nurses for patient care (four items: 1, 8, 9 and 12). (5) Collegial nurse–physician relationships. In this subscale, nurses' perception of teamwork and nurse–physician collaboration are examined (seven items: 2, 7, 13, 17, 21, 26 and 30). The Spanish version of the questionnaire was translated using a systematic translation process directed by experts in cross-cultural adaptation of instruments (Squires et al., 2013). The process developed to obtain the experts' assessments and, subsequently, the CVI—the Spanish version of the instrument—is described below: - Identification of ten experts in accordance with the following selection criteria: bilingual nursing graduates (English and first language), who had worked as nurses in their home countries for the past 5 years. Two to three additional assessors were identified to replace the failures (those not being able to complete the questionnaire) among those originally selected. - Once assessors were identified for each research team, they were formally invited to participate in the project by means of a participation request letter sent via email. - A computer application was developed for experts to complete the form on-line or in a Word document. - Evaluators had 15 days to complete the assessment form. - Each expert determined whether each of the items was suitable for the study population and whether the question format was appropriate. This required the items to be evaluated using a four-point ordinal scale from 1 (irrelevant) to 4 (extremely important). The model questionnaire with the question 1 is shown in Table 1.
2.1. Data analysis CVI calculations were performed for each of the items of the instrument (I-CVI). The final average of the I-CVI scores produces a scale-level content validity score (S-CVI). According to the methodology proposed by Lynn (1986) and Polit (Polit & Beck, 2006; Polit et al., 2007), as well as Liu, Squires, & You (2011), the scores for the I-CVI could quantify the five types of validity or equivalence described by Flaherty et al. (1988) in order to evaluate the cross-cultural validity of an instrument. Calculations for obtaining the CVI are shown in Fig. 1.
Table 1 Example of the instrument used to assess the relevance of the PES-NWI items. Use the following scale to evaluate the elements of the item: Item in American English Adequate support services allow me to spend time with my patients
Relevancea Translation (1–4)
Comments/Suggestions
Existen servicios de apoyo adecuados que me permiten pasar tiempo con mis pacientes
a 1 = Irrelevant; 2 = somewhat important; 3 = quite important and 4 = extremely important.
e7
Fig. 1. Calculations for the Content Validity Indexing (Polit et al., 2007).
An item with an I-CVI value of ≥ 0.78 is considered acceptable. The S-CVI minimum scores for evaluating the overall validity of the instrument or of each subscale are ≥0.80 and ≥ 0.90. Scores are then converted to a modified kappa value to account for chance agreement among raters. Fleiss (1981) developed a set of scores for the modified kappa coefficient. The scores divide into a four-part rating scale: excellent (≥0.74), good (0.60 to 0.73), moderate (0.40 to 0.59) and poor (≤0.39). Potentially problematic items (PPI) were questions with a kappa coefficient of less than 0.60. That threshold was set to identify items that might produce outlier responses due to cross-cultural instrument adaptation issues. Microsoft Excel 2007 software was used to for all CVI calculations in the study. 2.1.1. Ethical considerations All ethical principles of scientific biomedical research were observed. The consent of all the participants in the study was also requested. All the projects were approved by the various research ethics committees of the institutions from where the key investigators of each participating country came. 3. Results The overall content validity of the Iberian Spanish version of the PES-NWI (S-CVI) was 0.82, with a minimum I-CVI of 0.20 and the maximum of 1.00. The average modified kappa coefficient of the 32 items was 0.80, with a rating of 'excellent'. An impressive 81.3% (n = 26) of the items were rated excellent, 6.3% (2) were rated good, and a 12.5% (n = 4) of the items were viewed as weak or poor. The S-CVI obtained for each of the subscales was: 0.81 for nurse participation in hospital affairs; 0.86 for nursing foundations for quality of care; 0.85 for nurse manager ability leadership and support of nurses; 0.93 for staffing and resource adequacy; and 0.69 for collegial nurse–physician relationships. Four potentially problematic items (with a modified kappa value b0.60) were identified in the process. One was in the subscale “Nurse participation in hospital affairs”: item 18 - There are promotion opportunities and in three items of subscale “collegial nurse–physician relationships”: item 7 Physicians appreciate nurses' observations and judgements; item 13 Physicians recognize the contribution of nurses to patient care; and item 30 - Physicians have high regard for nurses. The results of the content validity indicators of all the items, based on the subscale they belong to, are shown in Table 2. 4. Discussion The content validation of the Iberian Spanish version of the PESNWI proved a useful exercise before conducting data collection in
e8
M.I. Orts-Cortés et al. / Applied Nursing Research 26 (2013) e5–e9
Table 2 Results of the content validity of the Spanish version of the PES-NWI. Subscale
Item
I-CVI
pc
k
Ratinga
Nurse participation in hospital affairs 5 0.80 0.044 0.79 6 0.90 0.010 0.90 11 0.80 0.044 0.79 16 0.70 0.117 0.66 18 0.50 0.246 0.34 23 1.00 0.001 1.00 25 0.90 0.010 0.90 29 0.90 0.010 0.90 S-CVI (subscale 1) = 0,81
Excellent Excellent Excellent Good Poor Excellent Excellent Excellent
4 1.00 0.001 1.00 15 0.80 0.044 0.79 19 0.80 0.044 0.79 20 0.90 0.010 0.90 24 0.80 0.044 0.79 27 0.90 0.010 0.90 28 0.90 0.010 0.90 31 0.70 0.117 0.66 32 0.90 0.010 0.90 S-CVI (subscale 2) = 0,86 Nurse manager ability leadership and support of nurses 3 0.80 0.044 0.79 10 0.90 0.010 0, 90 14 0.80 0.044 0.79 22 0.90 0.010 0, 90 S-CVI (subscale 3) = 0,85 Staffing and resource adequacy 1 0.80 0.044 0.79 8 0.90 0.010 0, 90 9 1.00 0.001 1.00 12 1.00 0.001 1.00 S-CVI (subscale 4) = 0,93 Collegial nurse–physician relationships 2 0.80 0.044 0.79 - Physicians appreciate nurses' 7 0.60 0.205 0.50 observations and judgements - Physicians recognize the contribution 13 0.50 0.246 0.34 of nurses to patient care 17 0.90 0.010 0.90 21 0.80 0.044 0.79 26 1.00 0.001 1.00 - Physicians have high regard for nurses. 30 0.20 0.044 0.16 S-CVI (subscale 5) = 0,69
Excellent Excellent Excellent Excellent Excellent Excellent Excellent Good Excellent
- There are promotion opportunities
Nursing foundations for quality of care
Excellent Excellent Excellent Excellent
Excellent Excellent Excellent Excellent
Excellent Weak Poor Excellent Excellent Excellent Poor
I-CVI: Item Content Validity Index; Pc: probability of random agreement; K: modified kappa coefficient obtained by designing the relevant proportion of agreements: a Evaluation criteria of k: poor ≤0.39, weak = 0.40–0.59; good = 0.60–0.73; excellent ≥0,74 according to Fleiss (Fleiss, 1981); S-CVI: I-CVI average of the items in the subscale.
Spain for the larger RN4CAST study. Almost the 90% of the items of the instrument were rated as excellent/good. We propose several explanations for the PPIs. First, the item “There are promotion opportunities” obtained a low content validity score. This item is included in the subscale “Nurse participation in hospital affairs”, which is strongly related to the clinical management models and the hierarchical structure within the institution. In the Spanish National Health Service (SNHS) hospitals, management positions are usually appointed, and promotion opportunities are very low. Therefore, it is not a surprise that raters might not consider this item as “relevant” to the Spanish nursing context if overall opportunities for advancement are limited. The other subscale, in which almost half of the items were potentially problematic and resulted in a lower overall (but still “good”) S-CVI rating, was “collegial nurse–physician relationships.” This subscale also had one of the highest percentages of PPI in other language versions of the European study (Orts-Cortés, 2011). During the development of this subscale of the original instrument regarding the nurse practice environment, Kramer & Schalenberg (2004) stated
that staff nurses in Magnet hospitals (MH) described five types of relationships between nurses and physicians: collegial, collaborative, student–teacher, friendly stranger and hostile/adversarial. Four-fifths of the MH nurses in the study reported that the relationship between these two professionals should be a collaborative or collegial one in order to improve patient care. Given that this subscale obtained a lower overall content validity, it would be necessary to investigate the types of relationships in SNHS hospitals and determine if these differ from those in MH, which could be one of the possible reasons for these results. Interestingly, the overall content validity (S-CVI) of the instrument was acceptable and higher than the American English version, which scored 0.70 in the European study (Squires et al., 2013). Overall, the content validity of the Spanish version, according to the modified Kappa coefficient, was rated as excellent, like other versions of the instrument: German (Germany), Dutch (Belgium), Finnish, Greek, Irish, Dutch (Netherlands), English (United Kingdom), Swedish, Norwegian, and Italian and German (Switzerland). Swiss and Belgian French versions obtained a good evaluation. The initial assessment of the Polish version was the only one rated as poor (Orts-Cortés, 2011; Squires et al., 2013). The studies found in literature, in which the cross-cultural adaptation of the PES-NWI or the NWI-R into other languages was carried out, were the following: Spanish (De Pedro Gómez et al., 2009); French (McCusker, Dendukuri, Cardinal, Laplante, & Bambonye, 2004), Chinese (Chiang & Lin, 2009), Iceland (Gunnarsdóttir, Clarke, Rafferty, & Nutbeam, 2009), Portuguese (Gasparino & Guirardello, 2009), Australian English (Middleton, Griffiths, Fernandez, & Smith, 2008) and Dutch (Van Bogaert, Clarke, Vermeyen, Meulemans & Van de Heyning, 2009). From these, only the Chinese version analyzed the content validity of the PES-NWI using the CVI, which resulted in an overall score of 0.94. The assessment was conducted by a total of 5 experts. The smaller number of experts used in the study of Chiang & Lin (2009) may explain the excellent rating obtained by the Chinese version given that the smaller the number of experts, the higher the CVI scores and the modified kappa coefficient (Beckstead, 2009; Polit et al., 2007). A small number of experts also means that all raters must produce 100% agreement to receive an “excellent” score for an item. The limitations of the study relate mostly to the limitations of the CVI approach. Its application as a pre-data collection strategy for analyzing an instrument is relatively new and requires additional replication with other instruments. The predictive validity of the PPIs score has also not been established, but is undergoing further analysis. The CVI approach also does not allow for modifying the content of an item to better fit the context, therefore potentially increasing the CVI score, since the utility of the approach described in this study is in establishing the content validity of the instrument in cross cultural research where producing comparable results are part of the study's design. In a forthcoming research, and in order to improve the content validity of the items with low I-CVI scores or the subscales with scores below the minimum acceptable, qualitative methods should be combined with panels of experts to determine whether the removal or reformulation of items, based on the contextual or cultural characteristics of the place where the instrument is to be used, would be appropriate. An example of this process is included in the study in which the translation and validation process of the Manchester Clinical Supervision Scale (MCSS) from English into Finnish (Hyrkäs, Appelqvist-Schmidlechner, & Oksa, 2003) was analyzed. The content validity of the instrument was analysed using both the quantitative approach, by means of the CVI, and the qualitative approach with a panel of eleven experts. This triangulation process determined the items to be removed, which were rated irrelevant and unsuitable by the experts based on the cultural and contextual characteristics of the target population the instrument was intended for.
M.I. Orts-Cortés et al. / Applied Nursing Research 26 (2013) e5–e9
It is also advisable for each country to establish the content validity by using the methods described in this article and by Squires et al. (2013). Management systems and different resources on the development of the nursing role may change the relevance of questions. In addition, each context may have their own terms for the performance of administrative roles which, if not equivalent, may result in what may seem like “odd” participants' responses when compared to other studies. For example, a nursing director may be responsible for the entire hospital or, in large hospitals, for a group of units or services. It is highly important to ensure that the terms are equivalent in order to obtain comparable results. That is why it is advisable that each country, even if it has the same language as the instrument, conducts its own assessment. Spanish speaking countries in Latin America, therefore, would need to conduct individual content validations of the Iberian translation to ensure that the language conforms to local norms. 5. Conclusions Overall, the content validity of the Iberian Spanish version of the Practice Environment Scale of the Nursing Work Index has been proven acceptable. Additional confirmatory factor analyses will provide further support for the reliability and validity of the instrument in the Iberian context. The results of this research paper indicate the need to further improve the instrument without compromising its cross-cultural applicability. This requires reanalyzing the potentially problematic items and those with a low I-CVI from a qualitative perspective in order to examine the possible reasons for why the experts scored the items and further consideration of the items that could be edited to better suit the cultural, professional and training context of the target population. Other analytic alternatives could include whether the overall results of the scale and subscales are adjusted depending on the removal or modification of these items from the instrument as a whole and, at a more micro level, determining whether these items rate differently in relation to the items which obtained an acceptable or high scores. References Aiken, L., Havens, D., & Sloane, D. (2000). The magnet nursing services recognition program. The American Journal of Nursing, 100(3), 26–35 [quiz 35–6]. Aiken, L. H., Sermeus, W., Van den Heede, K., Sloane, D. M., Busse, R., McKee, M., & Kutney-Lee, A. (2012). Patient safety, satisfaction, and quality of hospital care: Cross sectional surveys of nurses and patients in 12 countries in Europe and the United States. BMJ (Clinical Research Ed.), 344, e1717, http: //dx.doi.org/10.1136/bmj.e1717. Beckstead, J. W. (2009). Content validity is naught. International Journal of Nursing Studies, 46(9), 1274–1283, http://dx.doi.org/10.1016/j.ijnurstu.2009.04.014. Bonneterre, V., Liaudy, S., Chatellier, G., Lang, T., & de Gaudemaris, R. (2008). Reliability, validity, and health issues arising from questionnaires used to measure psychosocial and organizational work factors (POWFs) among hospital nurses: A critical review. Journal of Nursing Measurement, 16(3), 207–230. Chen, H. Y., & Boore, J. R. (2010). Translation and back-translation in qualitative nursing research: Methodological review. Journal of Clinical Nursing, 19(1–2), 234–239, http://dx.doi.org/10.1111/j.1365-2702.2009.02896.x. Chiang, H. Y., & Lin, S. Y. (2009). Psychometric testing of the Chinese version of nursing practice environment scale. Journal of Clinical Nursing, 18(6), 919–929, http: //dx.doi.org/10.1111/j.1365-2702.2008.02433.x.
e9
De Pedro Gómez, J., Morales Asencio, J., Sesé Abad, S., Bennasar Veny, M., Artigues Vives, G., & Pericàs Beltrán, J. (2009). Validación y adaptación al español de la escala del entorno de práctica enfermera del Nursing Work Index. Metas de Enfermería, 12(7), 65–73. Flaherty, J., Gaviria, F., Pathak, D., Mitchell, T., Wintrob, R., Richman, J., & Birz, S. (1988). Developing instruments for cross-cultural psychiatric research. The Journal of Nervous and Mental Disease, 176(5), 257–263. Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.)New York: John Wiley & Sons, Inc. Fuentelsaz-Gallego, C., Moreno-Casbas, M. T., & Gonzalez-Maria, E. (2013). Validation of the Spanish version of the questionnaire practice environment scale of the nursing work index. International Journal of Nursing Studies, 50(2), 274–280, http: //dx.doi.org/10.1016/j.ijnurstu.2012.08.001. Gasparino, R. C., & Guirardello, E. d. B. (2009). Tradução e adaptação para a cultura brasileira do “nursing work index - revised”. Acta Paulista De Enfermagem, 22(3), 281–287. Gunnarsdóttir, S., Clarke, S. P., Rafferty, A. M., & Nutbeam, D. (2009). Front-line management, staffing and nurse-doctor relationships as predictors of nurse and patient outcomes. A survey of Icelandic hospital nurses. International Journal of Nursing Studies, 46(7), 920–927, http://dx.doi.org/10.1016/j.ijnurstu.2006.11.007. Hyrkäs, K., Appelqvist-Schmidlechner, K., & Oksa, L. (2003). Validating an instrument for clinical supervision using an expert panel. International Journal of Nursing Studies, 40(6), 619–625. Kramer, M., & Hafner, L. (1989). Shared values: Impact on staff nurse job satisfaction and perceived productivity. Nursing Research, 38(3), 172–177. Kramer, M., & Schalenberg, C. (2004). Essentials of a magnetic work environment, part 1. Nursing, 34(6), 50–54. Lake, E. (2002). Development of the practice environment scale of the nursing work index. Research in Nursing & Health, 25(3), 176–188. Liu, K., Squires, A., & You, L. (2011). A pilot study of a systematic method for translating patient satisfaction questionnaires. Journal of Advanced Nursing, 67(5), 1012–1021, http://dx.doi.org/10.1111/j.1365-2648.2010.05569.x. Lynn, M. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. Maneesriwongul, W., & Dixon, J. K. (2004). Instrument translation process: A methods review. Journal of Advanced Nursing, 48(2), 175–186. Martin, J. S., Vincenzi, C., & Spirig, R. (2007). Principles and methods of good practice for the translation process for instruments of nursing research and nursing practice [Prinzipien und Methoden einer wissenschaftlich akkuraten Ubersetzungs- praxis von Instrumenten fur Forschung und direkte Pflege]. Pflege, 20(3), 157–163. McCusker, J., Dendukuri, N., Cardinal, L., Laplante, J., & Bambonye, L. (2004). Nursing work environment and quality of care: Differences between units at the same hospital. International Journal of Health Care Quality Assurance Incorporating Leadership in Health Services, 17(6), 313–322. Middleton, S., Griffiths, R., Fernandez, R., & Smith, B. (2008). Nursing practice environment: How does one australian hospital compare with magnet hospitals? International Journal of Nursing Practice, 14(5), 366–372, http: //dx.doi.org/10.1111/j.1440-172X.2008.00708.x. Orts-Cortés MI. (2011). Validez de contenido del Practice Environment Scale of the Nursing Work Index (PES-NWI) en el ámbito europeo. Universidad de Alicante. Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? critique and recommendations. Research in Nursing & Health, 29(5), 489–497. Polit, D. F., Beck, C., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. Sermeus, W., Aiken, L. H., Van den Heede, K., Rafferty, A. M., Griffiths, P., & MorenoCasbas, M. T.RN4CAST consortium. (2011). Nurse forecasting in Europe (RN4CAST): Rationale, design and methodology. BMC Nursing, 10, http://dx.doi.org/10.1186/ 1472-6955-10-6 [6-6955-10-6]. Squires, A., Aiken, L. H., van den Heede, K., Sermeus, W., Bruyneel, L., Lindqvist, R., & Matthews, A. (2013). A systematic survey instrument translation process for multicountry, comparative health workforce studies. International Journal of Nursing Studies, 50(2), 264–273, http://dx.doi.org/10.1016/j.ijnurstu.2012.02.015. Van Bogaert, P., Clarke, S., Vermeyen, K., Meulemans, H., & Van de Heyning, P. (2009). Practice environments and their associations with nurse-reported outcomes in Belgian hospitals: Development and preliminary validation of a Dutch adaptation of the revised nursing work index. International Journal of Nursing Studies, 46(1), 54–64, http://dx.doi.org/10.1016/j.ijnurstu.2008.07.009. Warshawsky, N. E., & Havens, D. S. (2011). Global use of the practice environment scale of the nursing work index. Nursing Research, 60(1), 17–31, http: //dx.doi.org/10.1097/NNR.0b013e3181ffa79c.
