CIN: Computers, Informatics, Nursing

& Vol. 32, No. 8, 404–412 & Copyright B 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins

F E A T U R E A R T I C L E

A Comparison of Two Nursing Program Exit Exams That Predict First-Time NCLEX-RN Outcome LISA D. BRODERSEN, EdD, RN ANDREW C. MILLS, PhD, RN

Structured approaches that focus specifically on preparing students to pass the NCLEX-RN have become standard practice in nursing programs.1–3 These approaches to NCLEX-RN preparation typically include the administration of exit examinations during students’ last terms of the nursing program. The purpose of exit examinations is to evaluate students’ readiness to take the licensure examination by quantifying their probability of passing the NCLEX-RN without further preparation. Administered at the end of prelicensure nursing programs, exit examinations provide information about strengths and weaknesses in specific nursing content areas, thereby helping students to focus their remediation efforts. The Health Education Systems, Inc, Exit Exam (E2, Elsevier, Philadelphia, PA) and the RN Comprehensive Predictor (RNCP; Assessment Technologies Institute [ATI], Leawood, KS) are examples of end-of-program exit examinations based on the NCLEX-RN blueprint. Both the E2 and RNCP are computer-based examinations designed to predict students’ probability of passing NCLEX-RN while approximating the NCLEX-RN testing environment. Development of the E2 predates the RNCP by approximately 10 years. Since the fall of 2004, students in one Midwestern baccalaureate nursing program were required to take both the RNCP and E2 as part of an NCLEX-RN preparation course during the final semester of the nursing program. Exit testing with the E2 began the fall of 2000 as part of a progression and remediation program. Exit testing with the RNCP was added in the fall of 2004 when it became part of the ATI Comprehensive Assessment and Review 404

This retrospective descriptive correlational study compared the predictive accuracy of the Health Education Systems, Inc, Exit Exam (Elsevier) and Assessment Technologies Institute’s RN Comprehensive Predictor, both of which were administered to nursing students in an upper-division baccalaureate nursing program during their final semester of study. Using logistic regression analyses, it was determined that the two examinations were statistically significant but weak predictors of success on the RN licensure examination. The RN Comprehensive Predictor had a slightly better odds ratio; however, both examinations had similar sensitivity, specificity, and overall accuracy. Because the RN Comprehensive Predictor was included in the Assessment Technologies Institute’s Comprehensive Assessment and Review Program already being used by the BSN program, based on the results of this study, the nursing faculty decided to use only the RN Comprehensive Predictor during its NCLEX-RN preparation course. KEY WORDS ATI RN Comprehensive predictor & Exit examinations & HESI Exit Exam & NCLEX-RN

Program (CARP), which had been used by the nursing program since 2002. The RNCP was administered as a proctored examination within the first 2 weeks of the students’ final semester to identify content areas in need of remediation before they took the proctored E2 midsemester. The E2 was considered the primary exit examination because of the lack of published literature about the predictive accuracy of the RNCP. Faculty may use a variety of strategies to facilitate nursing graduates’ success on NCLEX-RN, but cost and efficient use of time need to be considered. Because providing NCLEX-RN preparation programs and the use of exit Author Affiliation: School of Nursing, Saint Louis University, St Louis, MO. This study was funded, in part, by the Marion Bender Scholarship Fund at Saint Louis University. The authors have disclosed that they have no significant relationship with, or financial interest in, any commercial companies pertaining to this article. Corresponding author: Lisa D. Brodersen, EdD, RN, School of Nursing, Saint Louis University, 3525 Caroline Mall, St Louis, MO 63104 ([email protected]). DOI: 10.1097/CIN.0000000000000081

CIN: Computers, Informatics, Nursing & August 2014 Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

examinations consume student and faculty time, the cost of education increases. The cost of the RNCP was included in a fee students paid for the CARP upon entering the nursing program. The cost of the E2 was included in a graduation fee that students paid their last semester of study. Additionally, students who did not achieve a score of at least 850 on their first E2 were required to repeat the E2 using an alternate version after a period of remediation and to pay an additional fee. Typically about 50% of students scored below 850 on their first attempt of the E2; therefore, they needed to retest on the alternate form and to pay an additional fee. Given the extra time required of faculty and students and the extra cost to students, the use of two different exit examinations needed to be fully justified. The exit examination with greater predictive accuracy would be preferred, assuming students’ costs and faculty time to administer the examinations were approximately the same. To date, there have been no reports of both the RNCP and E2 being used in an NCLEX-RN preparation program. Although the evidence is limited compared with the E2, the RNCP has shown some promise as an accurate predictor of NCLEX-RN outcome in two of three studies that had sufficiently large samples and used analyses to determine the predictive accuracy of the test.4,5 Additional inquiry is needed to understand the predictive accuracy of the RNCP. Therefore, in order to determine whether it was necessary to continue using both the RNCP and E2 in the nursing NCLEX-RN preparation program, this study was conducted to evaluate the predictive accuracy of the RNCP and the E2 in a population of students who took both exit examinations.

LITERATURE REVIEW A substantial body of literature documents the relationship of E2 scores to first-time NCLEX-RN success.2,3,6–23 The predictive accuracy of the E2 has been studied regularly since 1999.2,3,7,15–19,23 These studies have been conducted in cooperation with the owner of the E2 using large samples selected from the pool of E2 testers during specific time frames. Predictive accuracy in the E2 validity studies has been determined by dividing the number of students who passed NCLEX-RN by the number predicted to pass based on a target cut score (eg, Q900).2 Conversely, the body of literature comparing RNCP scores to NCLEX-RN outcomes is comparatively sparse.3–5,23–30 One published study4 and two doctoral dissertations5,30 report the predictive accuracy of RNCP scores using regression analyses. In one other published study, the relationship of RNCP scores to NCLEX-RN outcomes was evaluated using discriminant analysis.31 Alameida and colleagues4 published the predictive accuracy of the RNCP in two separate large samples of predominantly female, but racially diverse, nursing students in a baccalaureate nursing program. In one

group, there were 367 students who had taken the RNCP (version 3.0) during their final semester. The predictive probabilities associated with students’ RNCP scores best predicted NCLEX-RN success in a model containing several nursing course grades and nursing GPA. In the second group, there were 222 students who had taken version 2007A or B of the RNCP, also during their final semester. In this group, NCLEX-RN success was predicted by the RNCP probabilities and by the students’ grades from two nursing courses, but not GPA. In their unpublished research, Vandenhouten5 and Yates30 separately studied the relationship of RNCP scores (version 3.0) to NCLEX-RN outcomes. Results differed in that Vandenhouten5 found that RNCP scores among 296 predominantly white female students were significant predictors of NCLEX-RN success, whereas Yates30 did not find that same outcome in a large group (n = 298) of racially diverse nursing students. Ukpabi31 also did not find the RNCP to be a significant predictor of NCLEX-RN outcome in a study of 22 BSN students at a historically black university. Unlike the studies by Alameida and colleagues4 and Ukpabi,31 the studies by Vandenhouten5 and Yates30 were conducted among community college students enrolled in associate nursing degree programs. Systematic reviews and recently published studies indicate that NCLEX-RN outcome is multifactorial. Variables found to be significantly related to NCLEX-RN outcome can be broadly categorized as academic, demographic, and psychosocial. Academic variables include high school rank, high school GPA,32,33 ACT and SAT scores,32–35 critical thinking test scores,36 National League for Nursing Achievement Test scores,31–33 ATI Test of Essential Academic Skills and nursing Content Mastery examination scores,31 HESI specialty examination scores,22 nursing theory course grades,22,32–35 and nursing program GPA.32–35 Additionally, grades in natural and social sciences courses, prenursing GPA,33–35,37 and cumulative college GPA33,34 have been associated with NCLEX-RN outcomes. Academic variables also include interventions such as NCLEX-RN preparation and remediation programs,1,38 a support group, personalized instruction, an integrated curriculum,34 and use of the HESI standardized test package, which includes nursing content examinations and the E2.39 Exit examination scores, either alone or combined with other academic variables, have been associated with NCLEX-RN outcomes as well.2,13,22,34,37,40,41 Demographic variables associated with NCLEX-RN outcomes include father’s occupation, parents’ years of formal education, student age,32–34 and racial-ethnic classification.34 Psychosocial variables such as self-esteem34 and test anxiety33,34 have also been associated with licensure examination outcomes. Despite the many variables associated with NCLEX-RN outcomes, it is neither feasible nor desirable to incorporate all of them into a single study. Many of the variables are likely to overlap conceptually on the same latent construct,

CIN: Computers, Informatics, Nursing & August 2014 Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

405

which would create multicollinearity problems with statistical approaches. Moreover, isolating various GPAs in nursing, sciences, and humanities courses achieved in different years of a student’s program of study denies the interrelationship of courses and learning progression throughout a curriculum. Incorporating these variables raises the question of statistical independence from an analytic perspective. It is the culmination of students’ education, including the nursing curriculum, that should predict NCLEX-RN performance as a learning outcome, not the specific parts or content areas of that curriculum. Although high school GPA and college entrance examination scores may provide some indication of students’ readiness for college, once admitted, students achieve a pattern of academic success in order to earn a nursing degree, which ultimately qualifies them to take the licensure examination. Academic success, operationalized as cumulative GPA, represents the students’ ability to apply the general education and nursing knowledge they have gained throughout their program of study and therefore should be a predictor of NCLEX-RN outcomes.

METHODS Design A retrospective, descriptive, correlational study was conducted to explore and compare the accuracy of the RNCP and E2 in predicting NCLEX-RN outcome while controlling for selected demographic and academic variables. Preexisting data were compiled from student records after obtaining approval from the appropriate institutional review boards.

Sampling The sample included 317 prelicensure students who had completed version 2007B or 2010B of the RNCP as well as version 1 of the E2 and had completed the traditional BSN program between May 2008 and May 2012. The sample size was based on the intent to perform logistic regression (LR) analyses on randomly selected halves of the sample. It was calculated that a sample size of 210 subjects would be needed to sufficiently power the study—105 subjects for each half. The anticipated sample size was based on two assumptions: (1) the NCLEX-RN outcome could be predicted with .80 probability given known values of age, gender, race, cumulative GPA, composite ACT score, NCLEX-RN blueprint, and RNCP form, and (2) the addition of RNCP or E2 scores to the predictive model would make it possible to predict NCLEX-RN outcome with .90 probability. Based on these assumptions, it was determined that a sample of 210 subjects would provide .88 power to detect an odds ratio, or Exp(") of 2.25. 406

Data Collection Composite ACT score, cumulative GPA, birth date, race, and gender were obtained from student records. Age was calculated using the date students took NCLEX-RN and their birthdates. When exact NCLEX-RN examination dates were unavailable, the last day of the quarter during which the student tested was used to calculate age. The RNCP score percentages were obtained from a passwordaccessible database (http://www.atitesting.com). Scores from version 1 of the E2 were obtained from another passwordaccessible database (https://evolve.elsevier.com). First-attempt scores from both exit examinations were used in analyses. NCLEX-RN outcomes were obtained from NCLEX-RN outcome reports compiled by the nursing program’s student services department. The different NCLEX-RN blueprint versions were coded for each student because of the periodic changes of the passing standard. For this study, the passing standard was increased from j0.21 logits to j0.16 logits starting April 1, 2010; therefore, in this study, students who took NCLEX-RN prior to April 1, 2010, tested on the 2007 blueprint and those who tested after that date had the 2010 blueprint.42 A binary variable, the RNCP form, was coded because each student tested on either form 2007B or form 2010B, depending on the time frame when the student took the exit examination.

Instruments The RNCP includes 180 items. Possible scores range from 0 to 100 and represent the percentage of correct responses. Scores correspond to a predicted probability of passing the NCLEX-RN. Predicted probabilities of passing and the scores to which they correspond have been determined with LR and vary depending on the form of the examination. Logistic regression classifies students whose scores correspond to probability of passing of .50 or greater as predicted to pass NCLEX-RN.43 The criterion-related validity of the RNCP is demonstrated by accuracy indices calculated for each form of the examination based on predicted and actual NCLEX-RN outcomes. These indices include sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and overall accuracy. Sensitivities, specificity, PPV, and NPV are 16.6%, 98.3%, 60.7%, and 87.9% for version 2007, and 15.5%, 98.1%, 54.3%, and 88.8% for version 2010.43,44 The ATI emphasizes the importance of the overall accuracy, defined as the number of testers correctly predicted by their RNCP scores to either pass or fail NCLEX-RN divided by the number of testers who passed or failed. Overall accuracies for the 2007 and 2010 versions of the test are 86.9% and 87.5%, respectively, indicating that in the samples of testers on which the predictive accuracies were established, 86.9% and 87.5% of testers performed as predicted on NCLEX-RN.43,44

CIN: Computers, Informatics, Nursing & August 2014 Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

The E2 includes 160 items, 10 of which are included on a trial basis and are not figured into the examination score.7 Scores can range from 0 to about 1800.2 E2 scores are determined by applying a proprietary mathematical model to the tester’s raw score.45 A score in the range 850 to 899 is considered an acceptable level of performance by Elsevier, but remediation is recommended in any content areas with scores of less than 850. Most nursing programs prefer that students achieve a minimum score of 850.2 About 97% of all students who score at least 850 on the E2 will pass the NCLEX-RN on the first attempt.2,3 The average internal consistency reliabilities for versions 1, 2, and 3 of the E2 are .91, .98, and .91 respectively.2 Predictive accuracy in the E2 validity studies has been determined by ‘‘tabulating the number of students who scored 900 or greater on the E2 and calculating the percentage of those students who passed the licensure examination on their first attempt.’’2(p57) In other words, E2 predictive accuracy is expressed as the NPV. The number of candidates correctly predicted to pass is divided by the total number predicted to pass. Strong NPVs (96.36–99.16) for version 1 of the E2 have consistently been reported.2,3,7,15–19 The NPV for versions 2 (92.95%–95.57%) and 3 (82.50%– 93.24%) were identified in later E2 validity studies.2,7

Analyses All statistical analyses were conducted using IBM SPSS version 19.0 (IBM, Armonk, NY). Standardized scores (z scores) were computed for both RNCP and E2 scores for use in comparative analyses. Variables measured at the interval or ratio level were described with measures of central tendency and variability. Frequencies and percentages were calculated for categorical variables. Groups were compared on continuous variables by calculating t tests for independence. 2 2 Statistics were calculated to compare groups on categorical variables. To assess associations between all variables for possible inclusion in later regression models, a series of Pearson product moment, Spearman rank order, and point biserial correlations were calculated, depending on each variable’s level of measurement. Statistics indicating P G .05 were considered statistically significant. The sample of 317 students was split randomly into equivalent halves, 159 in half A and 158 in half B. Equivalence of the two halves was determined by comparing them on all demographic and academic study variables and finding no statistically significant differences. To determine a model for predicting success on the NCLEX-RN, backward and hierarchical stepwise LR was run on half A with standardized RNCP score, RNCP form (ie, 2007B or 2010B), NCLEX-RN blueprint (ie, 2007 or 2010), and GPA as the predictor variables. To validate the half A model, the same model was run on half B of the sample using the identical LR procedure. Next, using the same split-half samples, the

entire validation procedure was duplicated substituting standardized E2 scores for the RNCP scores and omitting RNCP form. Because the study was exploratory, a backward stepwise approach was used wherein all predictor variables are entered in the regression equation and iteratively removed if they do not significantly contribute to the model. After determining the validation of both models using standardized RNCP and E2 scores in split samples, LR analyses were run separately using the full sample for actual (unstandardized) RNCP and E2 scores. Accuracy indices of sensitivity, specificity, PPV, NPV, and overall accuracy were calculated for each LR model. Receiver operator characteristic (ROC) curves were calculated using the actual NCLEX-RN outcomes and the predicted probabilities from the final LR models.

RESULTS Most students were female (96.2%) and white (96.5%), ranging in age from 21 to 55 years (mean, 24 [SD, 4.3]). The ACT scores were available for 88.6% of the students. Because of incomplete data, ACT scores were excluded from LR analyses. There were 206 students (65%) who completed RNCP version 2007B and 111 (35%) who completed version 2010B. Students who took version 2007B graduated May 2008 to December 2010. Students who took version 2010B graduated May 2011 to May 2012. Scores ranged from 48% to 86% (mean, 71 [SD, 7]). The E2 scores ranged from 474 to 1141 (mean, 821 [SD, 109]). At the end of the final semester, students’ cumulative program GPAs ranged from 2.77 to 3.99 (mean, 3.3 [SD, 0.3]) on a 4-point scale. Of the 317 students, 295 (93.1%) passed the NCLEX-RN on the first attempt. A series of independent t tests revealed no significant differences (P 9 .05) in success related to gender, race, RNCP form, and NCLEX-RN blueprint. However, students who failed had significantly lower cumulative GPAs, RNCP scores, and E2 scores.

Analysis of Correlations Between Study Variables The NCLEX-RN outcome was correlated with ACT score (r = 0.18, P G .01), GPA (r = 0.22, P G .01), RNCP score (r = 0.34, P G .01), and E2 score (r = 0.36, P G .01). The NCLEX-RN outcome was not significantly correlated with age, NCLEX-RN blueprint, or RNCP form. Despite not being significantly correlated with NCLEX-RN outcome, the NCLEX-RN blueprint was included in the LR analyses of the RNCP and E2 to control for its potential influence on predictive accuracy. Similarly, RNCP form was included in the LR analyses of the RNCP.

CIN: Computers, Informatics, Nursing & August 2014 Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

407

Split Sample Validation Analyses Independent t tests and 2 2 analyses confirmed the equivalence of the two randomly split halves of all students (half A and half B). No significant differences were observed between the two halves for the NCLEX-RN outcome, RNCP scores, RNCP z scores, E2 scores, E2 z scores, age, gender, minority status, GPA, NCLEX-RN blueprint, and RNCP form. Therefore, half A and half B were used in the validation analyses.

RNCP z Score Model Comparisons A two-step, hierarchical, backward stepwise LR was run on half A. To control for the possible effects on the NCLEXRN as the outcome, the GPA, NCLEX-RN blueprint, and RNCP form were entered as predictor variables in the first block. The RNCP z scores were entered in the second block as the predictor variable and NCLEX-RN success as the outcome variable. The same model was run on half B. Both half A and half B models were significant, indicating the variables together in each resulting model predicted NCLEX-RN success better than by chance (Omnibus 2 24 = 29.9, P G .001; 2 22 = 22.0, P G .001). The Cox and Snell R2 tests (sometimes known as pseudo 2 R ) for both half A and half B indicated a low amount of model variance explained by the three variables in each model (0.172 and 0.130, respectively). Hosmer and Lemeshow tests for both half A and half B indicated the goodness of fit between predicted and observed probabilities matched fairly well with the desired low values and statistical nonsignificance (2 28 = 14.2, P = .08; 2 28 = 3.3, P = .911, respectively). Both RNCP models highly predicted students’ success on NCLEX-RN but were less accurate in predicting failure. Of those students who were predicted to pass, 94.8% of them actually passed the examination in half A and 92.3% in half B (NPV). On the other hand, of those who were predicted to fail, 50.0% actually failed in half A, and 0.0% failed in half B (PPV). From another perspective, of those students who passed NCLEX-RN, 98.7% of the students in half A and 98.6% in half B were predicted to do so (specificity). Similarly, of those students who failed the examination, 20% in half A and 0.0% in half B were predicted to do so (sensitivity). In both models, the Wald statistic was significant for RNCP z score, indicating that RNCP z score was a predictor of NCLEX-RN outcome independent of the NCLEXRN blueprint, RNCP form, and GPA. The odds ratios indicated that for each SD increase in RNCP z score, the odds of students being predicted to pass NCLEX-RN were 5.2 times higher (P = .003) for half A and 3.6 times higher (P = .002) for half B. During the backward stepwise process for half B, GPA was also a predictor for success (odds ratio = 74.4, P = .03), but the wide confidence 408

interval (CI) (1.51–3672.17) indicated the measure was unstable and of questionable reliability.

E2 z Score Model Comparisons The same analytic process used for model comparison of RNCP z scores was used with E2 z scores except for omitting RNCP form, a variable unique to the RNCP examination. A two-step, hierarchical, backward stepwise LR was run on half A then on half B controlling for the NCLEXRN blueprint and GPA. Again, half A and half B models were significant, indicating each model’s variables predicted NCLEX-RN success better than by chance (Omnibus 2 22 = 18.4, P G .001; 2 22 = 33.9, P G .001). Furthermore, the Cox and Snell R2 tests for both half A and half B indicated a low amount of model variance explained by the three variables in each model (0.109 and .194, respectively). The goodness of fit between predicted and observed probabilities in half A and half B models also matched fairly well with the desired low values and expected statistical nonsignificance (Hosmer and Lemeshow tests 2 28 = 5.7, P = .675 identical for both models). Overall, both E2 models predicted students’ success on NCLEX-RN better than they predicted failure. The NPV for half A was 94.9% and 93.5% for half B. The PPV for half A was 100.0% and 66.7% for half B. The specificity values for half A and half B were 100.0% and 99.3%, respectively; the sensitivity values were 20.0% and 16.7%, respectively. In both models, the Wald statistic indicated only the E2 z score was a predictor of NCLEX-RN outcome independent of NCLEX-RN blueprint and GPA. For each SD increase in E2 z score, the odds of predicted success on NCLEX-RN were 3.0 times higher (P = .005) for half A and 10.4 times higher (P G .001) for half B.

Final Models on RNCP and E2 Unstandardized Scores After determining that the LR models on the equivalent split halves were highly similar using standardized test scores, two models were compared using the full sample of students’ actual RNCP and E2 unstandardized scores. Final models are presented in Table 1. The RNCP and E2 scores were significant predictors of NCLEX-RN outcome. The final model for RNCP score included NCLEX-RN blueprint (2007 or 2010), GPA, and RNCP form (2007B or 2010B). RN Comprehensive Predictor score was a significant predictor of NCLEX-RN outcome. In the final model for E2 score, GPAwas retained, but NCLEX-RN blueprint was not. The E2 score and GPA were significant predictors of NCLEX-RN outcome. The Exp(") statistics for RNCP and E2 scores indicate that for each one-unit increase in exit examination score, the odds of being predicted to pass

CIN: Computers, Informatics, Nursing & August 2014 Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

T a b l e 1 Final RNCP and E2 Logistic Regression Models Model

B

RNCP score 0.229 Blueprint 1.084 GPA 2.767 RNCP form 0.482 Constant j22.162 E2 score 0.014 GPA 2.79 Constant j16.851

SE

Wald (df = 1)

0.051 0.742 1.437 0.861 4.947 0.003 1.329 4.356

19.863 2.134 3.706 0.313 20.069 20.485 4.41 14.969

P

Exp(")

95% CI

G.001 1.258 1.14, 1.39 .144 2.958 0.69–12.67 .054 15.916 0.95–266.30 .576 1.619 0.30–8.75 G.001 0.000 G.001 1.014 1.01–1.02 .036 16.282 1.21–220.08 G.001 0.000

Omnibus 2 2 (df)

P

C&S R 2 H&L 22 (df)

51.239 (4)

G.001

0.149

2.867 (8) .942

48.833 (2)

G.001

0.143

12.628 (8) .125

P

Abbreviations: C&S R2, Cox and Snell R2; H&L 2 2, Hosmer and Lemeshow 2 2.

NCLEX-RN were 1.3 times higher for the RNCP and 1.01 times higher for the E2. The Wald statistics indicated that RNCP and E2 scores were significant predictors of the NCLEX-RN outcome. Both RNCP and E2 score models had high accuracy in predicting NCLEX-RN success, but low accuracy in predicting failure (Table 2). Specificities were 99%, and NPVs were ; 94% for both RNCP and E2 LR models. For the E2 LR model, the sensitivity was 18.2%, reflecting that 18 students who had failed NCLEX-RN were predicted to pass. The sensitivity for the RNCP LR model was 13.6%, reflecting that 19 students who had failed NCLEXRN were predicted to pass. Positive predictive values were 50% and 57% for the RNCP and E2 LR models, respectively. The ROC curves illustrate the high specificity and low sensitivity of these models. For the RNCP model (Figure 1), the area under the curve was c = 0.896 (SE, 0.026; P G .001; 95% CI, 0.845–0.947). For the E2 model (Figure 2), the area under the curve was c = 0.876 (SE, 0.041; P G .001; 95% CI, 0.796–0.955).

of the RNCP4,5 and E2.8,12,20 Comparing the predictive accuracy of the RNCP and E2 in this study with that reported in other studies is complicated by use of different predictor variables in LR models. Furthermore, in the E2 validity studies, a specific cut score (eg, Q900) has been used to classify students as predicted to pass, and typically, only the NPV has been reported to describe the E2’s predictive accuracy. Despite the high specificities, NPVs, and overall accuracies of both exit examinations in this study (Table 2), low odds ratios, nonsignificant variables, and small Cox and Snell R2 values (Table 1) indicate that neither the E2 nor RNCP LR model strongly predicts NCLEX-RN outcome. Logistic regression models for both exit examinations demonstrated strong specificity (99%), which is the probability of being predicted to pass NCLEX-RN when the candidate

DISCUSSION The use of RNCP and E2 exit examinations yielded similar outcomes for students in this study. The scores they achieved on both examinations were significant predictors of NCLEXRN outcome. These findings are congruent with other studies T a b l e 2 Accuracy Indices Based on Logistic Regression Models LR Model

Sensitivity Specificity PPV NPV Overall accuracy

RNCP Score

E2 Score

13.6 99.0 50.0 93.9 93.1

18.2 99.0 57.1 94.2 93.4

FIGURE 1. The ROC curve for RNCP LR model.

CIN: Computers, Informatics, Nursing & August 2014 Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

409

FIGURE 2. The ROC curve for E2 LR model.

actually passed. Both exit examinations also had strong NPVs (;94%), indicating high probability of passing NCLEX-RN when predicted to do so. Overall accuracy, the probability of a correctly predicted outcome, was also good for both exit examinations (;93%). Sensitivity, the probability of being predicted to fail when the candidate actually failed, was poor (13.6% for RNCP; 18.2% for E2). The RNCP LR model inaccurately predicted 19 students would pass NCLEX-RN. Similarly, the E2 LR model incorrectly predicted that 18 students would pass. In other words, among the 22 students who failed NCLEX-RN, 80% or greater were predicted to pass by the RNCP and E2 LR models. Consider that if NCLEX-RN failure were a disease and the exit examinations were diagnostic tests for the disease, then 80% of the students or more who tested negative for the disease (ie, were predicted to pass) actually ended up having the disease (ie, failed NCLEX-RN or false negative). It is important to recognize that use of the disease detection model has been criticized by some as an inappropriate framework for evaluating the predictive accuracy of exit examinations based on the argument that it falsely dichotomizes testers as predicted to pass or to fail and does not account for considerable uncertainty and error inherent in predicting NCLEX-RN outcome.46 However, others have found the disease detection model to be useful in evaluating the accuracy of predictions made by exit examinations because the resulting indices reflect the ability of the test to accurately detect and predict success or failure.9,20,40 It has been suggested that some indices of the disease detection model may be more useful to nurse educators than 410

others when evaluating the predictive accuracy of exit examinations. According to Spurlock and Hanks,40 ‘‘The predictive ability of a test is best assessed by the PPVand NPV of the test, not the sensitivity and specificity values.’’40(p542) Furthermore, they advise that the PPV is most important to nurse educators because it indicates the test’s ability to predict failure. The PPV is based only on students predicted to fail and the number who actually failed, whereas the NPV is based only on students predicted to pass and the number who actually passed. As shown in Table 2, the E2 had a better PPV than the RNCP, but both tests were less than 60% accurate in predicting failure. The low PPVs of the E2 and RNCP LR models could be attributed to the relatively low prevalence of NCLEX-RN failure in the study sample— only 22 of 317 (6.9%) candidates failed NCLEX-RN. Low prevalence of disease will contribute to the low PPV and higher NPV of a diagnostic test regardless of its specificity and sensitivity, resulting in more false positives.47 What this means for exit examinations with low PPVs is that more students are likely to be incorrectly predicted to fail NCLEX-RN. The ATI also cautions against exclusively using sensitivity and specificity to evaluate the accuracy of exit examinations, arguing that these indices take into account only students who actually passed (specificity) and only those who actually failed (sensitivity).43 However, ATI advises that the overall accuracy, the probability of being correctly predicted to pass or fail NCLEX-RN, is the most valuable index when evaluating the predictive accuracy of any exit examination. Given this advice, both the RNCP and E2 LR models had strong overall accuracies, 93.1% and 93.4%, respectively. Although not surprising, the poor PPV and even poorer sensitivity of both exit examinations in this study are still perplexing. Although recognized for their accuracy in predicting NCLEX success, exit examinations have long been criticized for poor prediction of NCLEX-RN failure because many students go on to pass despite having performed below the desired cut score on an exit examination.13,20,40 The main concern seems to be overprediction of failure (ie, false-positive rate) and the risk of unnecessarily imposing remediation and progression penalties on students who would actually pass NCLEXRN. In this study, the RNCP LR model predicted six students to fail, three of whom actually failed. Similarly, the E2 LR model predicted seven students to fail, four of whom actually failed. However, the fact that so many students who failed NCLEX were predicted to pass indicates the LR models underpredicted failure. Whether failure is overpredicted or underpredicted may depend on how students respond to their exit examination results, as well as on other events that transpire up to the moment they take NCLEX-RN. Overprediction of failure may be easier to explain. Based on exit examination scores, students at risk for failing would remediate voluntarily or because they are required to do so by the nursing program. Remediation strategies could include individual review of nursing content, formal group instruction, or even repetition

CIN: Computers, Informatics, Nursing & August 2014 Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

of nursing courses.3 Additional NCLEX-RN preparation strategies may include taking a commercial review course, coping with anxiety and stress, and honing test-taking skills.1,38 It is possible that any of these strategies could prevent the negative NCLEX-RN outcome predicted by a low exit examination score. Underprediction rather than overprediction of failure was identified by LR models in this study. Students may have been overly confident when their exit examination scores indicated a high probability of passing NCLEX-RN and did not prepare adequately. It is also possible that other factors may have contributed to underperformance on NCLEXRN, such as physical illness and psychosocial stressors. Determining why exit examinations inaccurately predict failure would require a prospective design with measurement of potentially confounding variables from the moment students receive their exit examination results until they take NCLEX-RN. A stronger design would involve controlling the confounding variables, which could be accomplished by having candidates take the exit examination right before they sit for NCLEX-RN. However, getting sufficient student participation for either study approach would probably be difficult.

Limitations This case study used data compiled retrospectively from a single baccalaureate nursing program attended by predominantly white, female students, so the findings cannot be generalized beyond the study setting. According to the program’s policy, students took the RNCP early within the first 2 weeks of their final semester of study and did not take the E2 until midterm. It has been reported that when students retest on different versions of the E2 (eg, version 2 or 3), although their performance on the examination may improve with repeated attempts, it is the first attempt on version 1 that most accurately predicts NCLEX-RN outcome.2,3,7,20 It is not known whether this phenomenon is true when students retest on completely different commercial exit examinations. This study provides no evidence to support that idea because it showed little difference in the predictive accuracy of the RNCP and E2. It is also important to note that although this study provides empirical support for the accuracy of the RNCP and E2 in predicting NCLEX-RN success, it provides no guidance for selecting cut scores that maximize sensitivity, specificity, PPV, NPV, and overall accuracy.

Conclusions Despite differences in the timing of administration, the E2 and RNCP predicted NCLEX-RN outcome with similar NPV, PPV, and overall accuracy; therefore, the benefits of administering both examinations do not outweigh the added cost and workload. RN Comprehensive Predictor

was included in the ATI CARP already being used by the BSN program, so the nursing program made a decision to stop requiring the E2. Students now test with alternate forms of the RNCP at the beginning of their final semester of study and at midterm.

REFERENCES 1. DiBartolo MC, Seldomridge LA. A Review of intervention studies to promote NCLEX-RN success of baccalaureate students. Nurse Educ. 2005;30(4):166–171. 2. Young A, Willson P. Predicting NCLEX-RN success: the seventh validity study HESI Exit Exam. Comput Inform Nurs. 2012;30(1): 55–60. 3. Langford R, Young A. Predicting NCLEX-RN success with the HESI Exit Exam: eighth validity study. J Prof Nurs. 2013;29(2)(suppl 1): 5S–9S. 4. Alameida MD, Prive A, Davis HC, Landry L, Renwanz-Boyle A, Dunham M. Predicting NCLEX-RN success in a diverse student population. J Nurs Educ. 2011;50(5):261–267. 5. Vandenhouten CL. Predictors of Success and Failure on the NCLEX-RN for Baccalaureate Graduates [dissertation]. Milwakee, WI: Marquette University; 2008. 6. Abbott AA, Schwartz MM, Hercinger M, Miller CL, Foyt ME. Predictors of success on National Council Licensure Examination for Registered Nurses for accelerated baccalaureate nursing graduates. Nurse Educ. 2008;33(1):5–6. 7. Adamson C, Britt R. Repeat testing with the HESI Exit Exam— sixth validity study. Comput Inform Nurs. 2009;27(6):393–397. 8. Bentley R. Comparison of traditional and accelerated baccalaureate nursing graduates. Nurse Educ. 2006;31(2):79–83. 9. Daley LK, Kirkpatrick BL, Frazier SK, Chung ML, Moser DK. Predictors of NCLEX-RN success in a baccalaureate nursing program as a foundation for remediation. J Nurs Educ. 2003;42(9): 390–398. 10. De Lima M, London L, Manieri E. Looking at the past to change the future: a retrospective study of associate degree in nursing graduates’ National Council Licensure Examination scores. Teach Learn Nurs. 2011;6(3):119–123. 11. Frith KH, Sewell JP, Clark DJ. Best practices in NCLEX-RN readiness preparation for baccalaureate student success. Comput Inform Nurs. 2005:46S–53s. 12. Hardin J. Predictors of Success on the National Council Licensing Examination Computerized Exam (CAT-NCLEX-RN) in Associate Degree Nursing Programs: A Logistic Regression Analysis [dissertation]. Commerce, TX: Texas A&M University; 2005. 13. Harding M. Predictability associated with exit examinations: a literature review. J Nurs Educ. 2010;49(9):493–497. 14. Higgins B. Strategies for lowering attrition rates and raising NCLEX-RN pass rates. J Nurs Educ. 2005;44(12):541–547. 15. Lauchner KA, Newmann M, Britt RB. Predicting licensure success with a computerized comprehensive nursing exam: the HESI Exit Exam. Comput Nurs. 1999;17(3):120–125. 16. Lewis CC. Predictive Accuracy of the HESI Exit Exam on NCLEX-RN Pass Rates and Effects of Progression Policies on Nursing Student Exit Exam Scores [dissertation]. Houston, TX: Texas Woman’s University; 2005. 17. Newman M, Britt RB, Lauchner KA. Predictive accuracy of the HESI Exit Exam: a follow-up study. Comput Nurs. 2000;18(3): 132–136. 18. Nibert AT, Young A. A third study on predicting NCLEX success with the HESI Exit Exam. Comput Nurs. 2001;19(4):172–178. 19. Nibert AT, Young A, Adamson C. Predicting NCLEX success with the HESI Exit Exam: fourth annual validity study. Comput Inform Nurs. 2002;20(6):261–267. 20. Spurlock DR, Hunt LA. A study of the usefulness of the HESI Exit Exam in predicting NCLEX-RN failure. J Nurs Educ. 2008; 47(4):157–166.

CIN: Computers, Informatics, Nursing & August 2014 Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

411

21. Yoho MJ, Young A, Adamson C, Britt R. The predictive accuracy of Health Education Systems, Inc., examinations for associate degree nursing students. Teach Learn Nurs. 2007;2(3):80–84. 22. Schooley A, Kuhn JRD. Early indicators of NCLEX-RN performance. J Nurs Educ. 2013;52(9):539–542. 23. Zweighaft EL. Impact of HESI Specialty Exams: the ninth HESI Exit Exam validity study. J Prof Nurs. 2013;29(2)(suppl 1):S10–S16. 24. Carl LC. Assessment Technology Institute Test Scores, NCLEXRN Pass-Fail, Nursing Program Evaluation, and Catastrophic Events in Pennsylvania [dissertation]. Phoenix, AZ: University of Phoenix; 2007. 25. Davenport NC. A comprehensive approach to NCLEX-RN success. Nurs Educ Perspect. 2007;28(1):30–33. 26. Humphreys JA. Academic and Nonacademic Predictors of Future Success on the NCLEX-RNA Licensure Examination for Nurses [dissertation]. Omaha, NE: College of Saint Mary; 2008. 27. Jacobs P, Koehn ML. Implementing a standardized testing program: preparing students for the NCLEX-RN. J Prof Nurs. 2006; 22(6):373–379. 28. Norton CK, Relf MV, Cox CW, et al. Ensuring NCLEX-RN success for first-time test-takers. J Prof Nurs. 2006;22(5):322–326. 29. Wray K, Whitehead T, Setter R, Treas L. Use of NCLEX preparation strategies in a hospital orientation program for graduate nurses. Nurs Adm Q. 2006;30(2):162–177. 30. Yates L. The Relationship of Aptitude, Course Grades, and Program Length, to Performance on a Standards Based Test (NCLEX-RN) [dissertation]. Miami, FL: Florida International University; 2007. 31. Ukpabi CV. Predictors of successful nursing education outcomes: a study of the North Carolina Central University’s nursing program. Educ Res Q. 2008;32(2):30–40. 32. Schwirian PM, Baer CL, Basta SM, Larabee JG. Health Resources Administration. Prediction of Successful Nursing Performance: Part I and Part II. Bethesda, MD: US Department of Health and Human Services, Heath Resources Administration; 1976. 33. Carpenter D, Bailey P. Predicting success on the registered nurse licensure examination—1999 update. In: Stevens KR, Cassidy VR, eds. Evidence-Based Teaching: Current Research in Nursing Education. Sudbury, MA: Jones & Bartlett, & NLN Press; 1999:135–170. 34. Campbell AR, Dickson CJ. Predicting student success: a 10-year review using integrative review and meta-analysis. J Prof Nurs. 1996; 12(1):47–59.

412

35. Grossbach A, Kuncel NR. The predictive validity of nursing admission measures for performance on the National Council Licensure Examination: a meta-analysis. J Prof Nurs. 2011;27(2):124–128. 36. Romeo EM. Quantitative research on critical thinking and predicting nursing students’ NCLEX-RN performance. J Nurs Educ. 2010;49(7):378–386. 37. Lavandera R, Whalen DM, Perkel LK, et al. Value-added of HESI Exam as a predictor of timely first-time RN licensure. Int J Nurs Educ Scholarsh. 2011;8(1). 38. Pennington TD, Spurlock D Jr. A systematic review of the effectiveness of remediation interventions to improve NCLEXRN pass rates. J Nurs Educ. 2010;49(9):485–492. 39. Homard CM. Impact of a standardized test package on exit examination scores and NCLEX-RN outcomes. J Nurs Educ. 2013;52(3): 175–178. 40. Spurlock DR Jr, Hanks C. Establishing progression policies with the HESI Exit Examination: a review of the evidence. J Nurs Educ. 2004;43(12):539–545. 41. McGahee TW, Gramling L, Reid TF. NCLEX-RNA success: are there predictors? South Online J Nurs Res. 2010;10(4):208–221. http://www.snrs.org/i4a/pages/index.cfm?pageid=3301. Accessed October 7, 2013. 42. National Council of State Boards of Nursing. Passing standard: setting the passing standard for the NCLEX examinations. https:// www.ncsbn.org. Accessed October 6, 2013. 43. Assessment Technologies, Inc. RN Comprehensive PredictorA 2010 and NCLEX-RNA readiness. https://www.atitesting.com. Accessed December 11, 2011. 44. Assessment Technologies, Inc. RN Comprehensive PredictorA 2007 and NCLEX-RNA readiness. https://www.atitesting.com. Accessed December 11, 2011. 45. Morrison S, Adamson C, Nibert A, Hsia S. HESI exams: an overview of reliability and validity. Comput Inform Nurs. 2004;22(4): 220–226. 46. Nibert AT, Adamson C, Young A, Lauchner KA, Britt RB, Hinds MN. Choosing a theoretical framework to guide HESI Exit Examination research. J Nurs Educ. 2006;45(8):303–307. 47. Altman DG, Bland JM. Diagnostic tests. 1: Sensitivity and specificity. BMJ (Clin Res Ed). 1994;308(6943):1552–1552. http:// www.ncbi.nlm.nih.gov/pmc/articles/PMC2540489/pdf/ bmj00444-0038.pdf. Accessed October 6, 2013.

CIN: Computers, Informatics, Nursing & August 2014 Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

A comparison of two nursing program exit exams that predict first-time NCLEX-RN outcome.

This retrospective descriptive correlational study compared the predictive accuracy of the Health Education Systems, Inc, Exit Exam (Elsevier) and Ass...
807KB Sizes 4 Downloads 4 Views