ORIGINAL RESEARCH: EMPIRICAL RESEARCH – QUANTITATIVE

Applying artificial neural networks to predict communication risks in the emergency department Annamaria Bagnasco, Anna Siri, Giuseppe Aleo, Gennaro Rocco & Loredana Sasso Accepted for publication 13 April 2015

Correspondence to A. Bagnasco: e-mail: [email protected] Annamaria Bagnasco MSC PhD Nursing Researcher Department of Health Sciences, University of Genoa, Italy Anna Siri PhD Researcher School of Medical and Pharmaceutical Sciences, University of Genoa, Italy Giuseppe Aleo MA PhD Student in Epidemiology Department of Health Sciences, University of Genoa, Italy Gennaro Rocco MSC RN NP Director of the Centre of Excellence for Nursing Scholarship Centre of Excellence for Nursing Scholarship, Rome, Italy Loredana Sasso MA MSC RN Associate Professor of Nursing Department of Health Sciences, University of Genoa, Italy

B A G N A S C O A . , S I R I A . , A L E O G . , R O C C O G . & S A S S O L . ( 2 0 1 5 ) Applying artificial neural networks to predict communication risks in the emergency department. Journal of Advanced Nursing 71(10), 2293–2304. doi: 10.1111/jan.12691

Abstract Aims. To describe the utility of artificial neural networks in predicting communication risks. Background. In health care, effective communication reduces the risk of error. Therefore, it is important to identify the predictive factors of effective communication. Non-technical skills are needed to achieve effective communication. This study explores how artificial neural networks can be applied to predict the risk of communication failures in emergency departments. Design. A multicentre observational study. Methods. Data were collected between March–May 2011 by observing the communication interactions of 840 nurses with their patients during their routine activities in emergency departments. The tools used for our observation were a questionnaire to collect personal and descriptive data, level of training and experience and Guilbert’s observation grid, applying the Situation-Background-AssessmentRecommendation technique to communication in emergency departments. Results. A total of 840 observations were made on the nurses working in the emergency departments. Based on Guilbert’s observation grid, the output variables is likely to influence the risk of communication failure were ‘terminology’; ‘listening’; ‘attention’ and ‘clarity’, whereas nurses’ personal characteristics were used as input variables in the artificial neural network model. A model based on the multilayer perceptron topology was developed and trained. The receiver operator characteristic analysis confirmed that the artificial neural network model correctly predicted the performance of more than 80% of the communication failures. Conclusion. The application of the artificial neural network model could offer a valid tool to forecast and prevent harmful communication errors in the emergency department. Keywords: artificial neural network, communication skills, emergency department, nursing, prediction, risk

© 2015 John Wiley & Sons Ltd

2293

A. Bagnasco et al.

Why is this research or review needed?  Communication failures in the emergency department continue to raise serious safety issues. Artificial neural networks can be successfully applied to predict and avoid communication failures.  Artificial neural networks are best suited for cases, where there is a known relationship between inputs and outputs, but their precise nature is not clear.

What are the key findings?  Artificial neural networks can extract patterns and trends from non-linear data sets that are too difficult for conventional computers to calculate.  Artificial neural networks are able to include uncertainty in data sets.  Artificial neural networks can be ‘trained’ using fewer points than any other statistical model.

How should the findings be used to influence policy/ practice/research/education?  Artificial neural networks are demonstrated to be an effective tool for predicting the risk of communication failures, but further research is needed.  Artificial neural networks can be used in general as a risk prediction tool.  Early prediction of risk enables to arrange targeted educational interventions before harmful situations actually occur.

Introduction Several studies on communication relations in Emergency Departments report that approximately 89% of the professional activities require good communication skills (Spencer et al. 2002) and that the use of standardized communication techniques, such as the Situation-Background-Assessment-Recommendation (SBAR) technique (Haig et al. 2006), in clinical situations requiring immediate interventions, is mandatory for patient safety (Meginniss et al. 2012). Clear and complete communication and the constant check that communication is properly understood are instrumental to ensure patient safety and the quality of care (Spencer et al. 2002, Jones et al. 2013). A study, previously conducted by our research group, permitted to identify and categorize the communication failures detected in the nurse–patient communication interactions in the Emergency Department and their coding according to the Joint Commission International (JCI) standards (Bagnasco 2294

et al. 2013). We used the failure mode and effects analysis (FMEA), because it effectively assesses and prioritizes the areas of risk in clinical practice (Day et al. 2006, Lago et al. 2012). The preliminary phase of that study, after a review of the literature, led to the identification of an observation grid on ‘Communication Performance’ by Jean-Jacques Guilbert (Guilbert 1990, 2002) (Figure 1), a validated tool used for the observation of health workers’ communication performance based on ‘Terminology’ (i.e. whether words used were precise and adequate), ‘Listening’ (i.e. listening and correctly understanding a message), ‘Attention’ (i.e. paying verbal and behavioural attention by providing feedback) and Clarity (i.e. communicating in a clear and precise manner so that information is correctly understood. With this observation grid, we observed the nursing staff working in the Emergency Departments and measured the quali-quantitative requirements that needed to be implemented to improve their nontechnical skills (Flin et al. 2003) and in particular internal and external communication (Kilner & Sheppard 2010). ‘Communication Skills Training’ can improve emergency nurses’ communication and empathy skills associated with an increase inpatient satisfaction and a reduction in undesirable events and complaints during nurse–patient interactions (Ak et al. 2011). Therefore, in 2013 our team conducted a study on communication skills training in an advanced simulation centre (Bagnasco et al. 2014) to correct communication failures. In this case Guilbert’s observation grid was used for educational purposes. Instead, in this study, Guilbert’s observation grid (Figure 1) was used with the artificial neural networks (ANNs) to predict nurse–patient communication risks from data collected from three Adult Emergency Departments in Central Italy between March 2011–May 2011.

What are artificial neural networks? ANNs are mathematical models that try to imitate the functional processes of the human brain. ANNs involve nonlinear relationships among different data sets that cannot always be fully identified through the use of conventional linear analyses. ANNs are extensively used in technology and science with applications in chemistry, physics, biology and medicine. As reported by Amato et al. (2013), ANNs manage several types of input data to produce a clinically relevant output, for example, the probability of a certain pathology or classification of biomedical objects. These data are processed in the ‘setting’ of previous training history on a defined sample database. ANNs are known to be a potent tool to simulate several non-linear systems and have been applied to numerous © 2015 John Wiley & Sons Ltd

JAN: ORIGINAL RESEARCH: EMPIRICAL RESEARCH – QUANTITATIVE

–2

Terminology

Listening

Attention

Clarity

Too detailed – not

Hears but not listens

Not pay attention

Communication not

neither verbally nor

clear and information

through behaviour

not precise

Verbal attention is

Communication not

inconsistent with

clear and information

behavioural one

is partially precise

Pays verbal attention

Communication not

adequate

–1

Too detailed –

Listens but not repeats

adequate

0

Risk prediction in ED using ANNs

Adequate – Not precise

Listens but not always repeats correctly

always clear and information not always precise

+1

Precise terminology

Listens and repeats

Pays verbal and

Communication not

but not replies

correctly

behavioural attention

always clear but

immediately to

information is precise

questions +2

Replies immediately to

Checks understanding

questions

Feedback to

Information is correctly

interlocutor

understood

Figure 1 Guilbert’s observation grid on communication performance. The scores in first column on the left measure the level of risk. Source: J.-J. Guilbert. Guide pedagogique pour les personnels de sante. Geneva, World Health Organization, 1990.

problems of statistically significant complexity in many fields, including engineering, agriculture (Rodriguez Galdon et al. 2010), education (Siri 2014), medicinal chemistry (Pandini et al. 2013), diagnostics (Qeethara Kadhim 2011, Irfan Khan et al. 2013) and pharmaceutical research (Wesolowski & Suchacz 2012). In medicine, ANNs have been successfully applied to various areas, such as diagnostic systems (Dey et al. 2011, Hu et al. 2013, Liu & Jiang 2013), biochemical analysis (Catalogna et al. 2012, Fernandez de Canete et al. 2012), image analysis (Barbosa et al. 2012, Saghiri et al. 2012) and drug development (Li et al. 2005, Patel et al. 2012). In diagnostic systems, ANNs are usually exploited to identify cancer and heart diseases. In biochemical analysis, ANNs have been used to examine glucose levels in people affected by pathological conditions such as tuberculosis and diabetes or in the analysis of medical images. In the emergency department, ANNs have predominantly been used to quantitatively forecast patient arrivals and admission rates (Champion et al. 2007, Jones et al. 2008, Schweigler et al. 2009, Xu et al. 2013), or to obtain feedback to support clinical decisions (Hollander et al. 2004). To our knowledge, ANNs have never been used to predict and therefore prevent potentially harmful nurse–patient communication interactions to increase patient safety, which is the purpose of the present article. © 2015 John Wiley & Sons Ltd

Why use artificial neural networks? ANNs are best suited for cases, where there is a known relationship between the variable inputs and outputs, but the particular nature of the connection is not clear. ANNs are helpful when the link between several different variables needs an accurate mathematical model that still has not been developed. The added value of ANNs is that they extract patterns and trends from data sets that are too complex to build with conventional computers. Another advantage of ANNs is their ability to include uncertainty or ‘noise’ in a given data set. Furthermore, ANNs make no assumptions about the statistical nature of the data and can integrate nominal and ordinal data. ANNs can be trained using comparatively fewer points than any other statistical model and it is not necessary to select a data distribution model.

What do we mean by ‘training’ a Neural Network? Biologically, neural networks are made up of large numbers of neuron cells deeply interconnected with one another. In fact, each neuron is a specialized component that sends out specific electrochemical signals and the axons of each neuron are linked to the dendrites of other neurons through synapses. The signals are sent through this intricate mecha2295

A. Bagnasco et al.

2013). During that study, many nurses were found to have poor communication skills, therefore, our research team conducted another study to improve communication skills through educational interventions in an Advanced Simulation Centre (Bagnasco et al. 2014). However, educational interventions on their own are not enough to ensure safe communication in the Emergency Department, but needed to find a tool that could predict communications risks before they actually occurred. Therefore, ANNs were identified as a possible tool that could reliably achieve this purpose.

nism, which is fundamentally regulated by synapses. In his study, Hebb (1949) showed that the ‘learning process’ mainly consists in changing the strength of several synaptic connections. The signals that pass produce a response and the synapses carefully regulate the way it is transferred. Artificial neural networks (ANNs) try to imitate the behaviour of the brain. Similar to the biological nervous system, also ANNs contain layers of simple processors (or nodes) of data that interact through weighted connection lines and generate an outcome. The weight-balance of these lines is done through a ‘training session’ of input data that the network uses to adapt its links. In short, ANNs during training attempt to extract patterns, trends and relationships from data sets starting from various input and output variables. Obviously, the structure of an ANN is much simpler than that of a biological nervous network, with many fewer nodes or interconnections. Researchers have explored artificial intelligence over the last 70 years. McCulloch and Pitts carried out the first studies on ANNs in 1943. In 1959, Rosenblatt formulated the first learning algorithm, known as the perceptron that was only a solution to simple linear problems. In 1974, Werbos reported the first non-linear processing capabilities of ANNs. Thanks to the growing progresses in computer technology, the scientific community discovered the back propagation algorithm – which is a network training or learning algorithm – and its advantages in computational power.

The Artificial Neuron As described by Bardan (2014), ANNs are made up of neurons and each neuron consists of a ‘summation block’ and an ‘activating block’. The input variables are multiplied by the optimized weight parameters (w) and their products are summed. The neuron output is attained through the transformation of the sum using an activation function. Perceptron networks commonly use continuous sigmoidal activation functions that convert the output value into a range between 0-1. From this perspective, the artificial neuron is similar to regression models. In fact, a bias weight (‘constant’ in statistical terminology) is used to optimize the final prediction, similar to regression models. However, regression models have an important weakness that consists in the inability to reveal intricate relationships in given data sets. Moreover, the correlation type between inputs and outputs has to be chosen before starting the analysis (Bardan 2014) (Figure 2). In 2000, Zhang and Gupta described that the most common network structure is the multilayer perceptron (MLP), where the neurons are grouped into layers. No layers

Background Our research team started investigating the important role of communication skills for ensuring safety in the Paediatric Emergency Department setting in 2008 (Bagnasco et al. Inputs

Weights W1

A1 A2

W2

A3

AN

Sum-of-Product

W3 ∑

WN

χ

Summation function

ƒ(χ)

Output γ

Activation function

1 Θ Threshold (bias)

Figure 2 Visual display of an artificial neuron. The inputs are multiplied with the optimized weights and thereafter summed. The sum is transformed by the neuron’s activation function into the output value. Source: Own elaboration. 2296

© 2015 John Wiley & Sons Ltd

JAN: ORIGINAL RESEARCH: EMPIRICAL RESEARCH – QUANTITATIVE

Risk prediction in ED using ANNs

Table 1 Input variables and respective description. The nominal variables are qualitative characteristics that cannot be numerically ordered. Instead, ordinal variable are characteristics that can be numerically ordered, such as level of education and school marks.

Input layer

Hidden layer

Output layer

Synapses

Figure 3 The layers of an artificial neural network. Source: Own elaboration.

(levels) of neurons communicate with each other. The first and last layers are respectively named ‘Input’ and ‘output’, because they represent inputs and outputs of the whole network. The ‘Hidden layers’ are the remaining layers that are situated inside the network, between the input and the output layers. Usually, an MLP neural network contains an input layer, one or more hidden layers and an output layer (Figure 3). Statistically significant parameters for an accurate prediction are the number of hidden layers and the number of neurons on each level. The output layer generates the network response that commonly consists of one or two neurons.

The study This study consists of four phases. In Phase 1, we collected the data through the observations of the nurse–patient communication interactions and measured communication risk with Guilbert’s observation grid (Figure 1). In Phase 2, we trained the ANNs using the input (Table 1) and output variables (Table 2). In Phase 3, we tested the predictive performance of the ANNs (Figure 5). In Phase 4, we used the ROC analysis (Figure 4) to understand whether the predictive performance of the ANNs was correct.

Input variables

Description of the input variables

Type

M/F Age Seniority

Gender Age bracket Years of service in current Emergency Department Title of professional qualification

Nominal Ordinal Ordinal

Professional Qualification Postgraduate qualification Structure

Titles of Postgraduate qualification (masters, specialization, etc.) Hospital where nurses worked

Nominal Nominal Nominal

Source: Own elaboration.

Table 2 Output variables. Output variables

Type

Terminology Listening Attention Clarity

Ordinal Ordinal Ordinal Ordinal

Source: Own elaboration.

naire to collect nurses’ personal characteristics; (2) the Situation-Background-Assessment-Recommendation (SBAR) technique applied to communication in the Emergency Department, which investigated the communication behaviours of the nurses with patients (and their family members) using the four dimensions of Guilbert’s observation grid (i.e. terminology, listening, attention and clarity). Therefore, to test ANNs as a risk prediction tool for communication failures in the ED we asked two research questions, one for each tool: 1 How accurately do nurses’ personal characteristics predict

the risk of communication failures? 2 Which of the four communication dimensions weigh most

in predicting the risk of communication failures?

Aim The aim of this paper was to study the usefulness of ANNs in predicting the risk of communication failures in EDs through selected variables.

Design A multicentre observational study.

Sample Rationale Based on our previous studies (Bagnasco et al. 2013, 2014), two tools were identified to collect data: (1) A question© 2015 John Wiley & Sons Ltd

We selected a convenience sample of 840 nurses working in the emergency departments (EDs) from three different hospitals in Central Italy. After contacting the Nursing 2297

A. Bagnasco et al.

Terminology

1·0

1 2 3

·8

·8

·6

·6

Sensitivity

Sensitivity

1·0

·4

Area under curve 1 ·910 2 ·917 3 ·927

·2

1 2 3

·4

Area under curve 1 ·949 2 ·915 3 ·952

·2

·0

·0 ·0

·2

·4 ·6 1 - Specificity

·8

·0

1·0

Attention

1·0

·8

·8

·6

·6

·4

Area under curve 1 ·911 2 ·886 3 ·924

·2

·2

·4 ·6 1 - Specificity

·8

1·0

Clarity

1·0

1 2 3

Sensitivity

Sensitivity

Listening

1 2 3

·4

Area under curve 1 ·910 2 ·866 3 ·915

·2

·0

·0 ·0

·2

·4 ·6 1 - Specificity

·8

1·0

·0

·2

·4 ·6 1 - Specificity

·8

1·0

Figure 4 ROC curve of the four dimensions of communication. Source: Own elaboration. Directors of the three hospitals and obtaining their approval for conducting this observational study involving the nurse–patient communication interactions, our research team illustrated the project to the head nurse of each emergency department (ED). The head nurses then asked if any nurses working in the ED on a full time basis volunteered to take part in this study. The members of our research team provided preliminary training both to each head nurse and to their group of nurses who had volunteered to take part in this study. Preliminary training was conducted in small groups and consisted of a four-hour theoretical and 2298

practical training session to make sure that the tools (the personal characteristics questionnaire and Guilbert’s observation grid) were used correctly and that observations were conducted in a rigorous and objective manner. This preliminary training session was essential both to produce comparable data and ensure good inter-rater reliability.

Data collection A multicentre observational study database was used for this research. Data were collected by observing nurse–patient © 2015 John Wiley & Sons Ltd

JAN: ORIGINAL RESEARCH: EMPIRICAL RESEARCH – QUANTITATIVE

communication interactions in 840 nurses while carrying out their routine activities in the Emergency Department. Observations were made over a period of 3 months (March–May 2011), until at least 200 communication interactions were collected in each hospital. Therefore, the groups of appropriately trained nurses observed the communication behaviours of their nurse colleagues with patients during their routine work activities in the ED. This strategy to train nurses already working in the ED instead of having a person from outside the hospital (i.e. a member of our research team), to minimize the Hawthorne effect (Landsberger 1958). The tool used for observation consisted of two parts: 1 A questionnaire to collect data on the facility, field of

investigation and characteristics of the nurses included in the sample. 2 Guilbert’s observation grid (Guilbert 1990) based on the four communication dimensions. The purpose of the first part of the tool was to collect the ED staff nurses’ personal and descriptive data and their levels of training and experience (i.e. age, gender, years of service in the current workplace, professional qualification, postgraduate education and type of structure). The second part of the tool investigated the communication behaviours of the ED staff nurses with patients in relation to four dimensions: terminology, listening, attention and clarity (Figure 1). For each of these dimensions, a rating scale (ranging from 2-+2) measured the level of communication risk.

Ethical considerations This study received Research Ethics Committee approval from the University of Genoa. All participants took part on a voluntary basis and were appropriately informed before being included in the study. In addition, all participants spontaneously accepted to reciprocally observe one another.

Data analysis The database of 840 nurses working in three Accident and Emergency Departments in Rome consisted of a total of 10 variables: the four dimensions of the observation grid (i.e. terminology, listening, attention and clarity; these were the output variables) and six regarding age, gender, working seniority, level of education, professional qualifications and the structure (place), where they worked (these were the input variables). Not all the variables included in the model were suitable at the first attempt to train the ANN. © 2015 John Wiley & Sons Ltd

Risk prediction in ED using ANNs

The six main input variables included in this study are shown in Table 1. The output variables were the four dimensions included in Guilbert’s observation grid related to communication in the ED: terminology, listening, attention and clarity (Table 2). The output layer consisted of a variable that indicated communication risk, which in the ANNs was summarized into three units (1 = real risk; 2 = moderate risk; 3 = no risk). To increase ANN predictive performance, a correlation analyses was conducted between all the variables to identify any strong correlations between the different variables included and remove redundant information from the database. This analysis showed that there were strong obvious correlations with age and seniority and weak correlations with gender, education, professional qualifications and the structure where they worked. Microsoft Office Excel, Version 2007 and IBM SPSS Statistics for Windows, Version 220 (IBM Corp. Released 2013, Armonk, NY, USA) were used for data processing and running the ANNs. ANN Topology selection and training The identification of an appropriate topology (i.e. the number of hidden layers and the number of nodes per hidden layer) is one of the most crucial aspects of the application of neural networks. The neural network architecture used in this study is the multi-layered feed-forward network model that is widely and successfully applied for classification, forecasting and problem-solving (Irfan Khan et al. 2013). In general, a network topology is often decided only after a lengthy process of trial-and-error. The approach we adopted considered the training of many different networks and the subsequent selection of the ‘best’ one. All neural networks are trained until the error (sum of squared error) for each training session reached the minimum, before it started to rise again. After topology determination, the data were normalized finding a mean and a standard deviation of each feature and for every cell of each row to subtract its mean and divide it by the standard deviation of the right feature. The resulting data have a zero mean and a standard deviation equal to one. Therefore, the data are separated into training and validation subsets. The ANN is trained with a standard backpropagation learning algorithm. The activation function used for each node in the network is the binary sigmoidal function demarcated (with r = 1) as output = 1/(1 + e x), where x is the sum of the weighted inputs to that specific node. This function restricts the output of all nodes in the network to the range 0-1. Between each training epoch, the error rate is computed for data validation. 2299

A. Bagnasco et al.

Due to the risk of overfitting the neural network too easily, producing an error rate on validation data larger than the error rate on the training data, it is crucial not to overtrain the data. Overtraining is when you train the network for so long that it starts to memorize the training set. The validation error reduces in the early epochs but after some time, it begins to rise. The point of minimum validation error is a valid indicator of the best number of epochs for training. Our results indicated that the minimum error of the validation set could be obtained in approximately 500 epochs. The training results are certainly dependent on the initial values of the weights that are generally random. We examined the performance of the test across the entire range of possible values using the curves of the receiver operator characteristic (ROC) analysis. The ROC curve is a plot of the true positive rate (sensitivity) vs. the false positive rate (1-specificity). A perfect test would show points in the upper-left-corner, with 100% sensitivity and 100% specificity (Woods & Bower 1997). An area under a curve (AUC) of 1 means that the test is perfect, whereas an AUC of 05 means that the test has a probability of 50% to classify (similar to the toss of a coin). A diagnostic test is considered adequate with an area under the curve >80%.

Validity and reliability The validity and reliability of the data sets collected during the communication observations made at the beginning of this was ensured through the observers’ use of Guilbert’s observation grid, by attributing a score between 2-+2 to the four dimensions of communication: terminology; listening; attention and clarity (Figure 1). For this purpose, a 4hour preliminary training session was provided to the ED nurses who were going to observe the nurse–patient communication interactions of their peers.

Results A total of 840 observations were made on the nurses working in the EDs, with a good distribution between males and females. Most of the observations involved nurses aged between 31-40 years. With regard to seniority (years of service in the current ED) most of the nurses had worked between 6-10 years (Table 3). The other three variables (education, professional qualification and structure) are not included because we found that they did not have an impact on the prediction of communication risk. The ANN’s learning ability was checked through the calculation of the mean-squared error, as error measure. We used a learning algorithm called ‘Back Propagation’, which 2300

Table 3 The most significant results of the observations made on the communication interactions of 840 nurses in three different Emergency Departments with their patients between March–May 2011. Seniority (years of service)

Gender

Age

Males = 406% Females = 594%

31-40 years = 413% 41-50 years = 329% 51-60 years = 75% >61 years = 08%

0-5 years = 262% 6-10 years = 37% 11-15 years = 22% 16-20 years = 82% >20 years = 605%

The other three variables (education, professional qualifications and place of work) were excluded because they were found to have no impact on the prediction of communication risk.

modified the synaptic weights to minimize global error made by the ANN, measured with the mean squared error (MSE) between the output of the network and the correct results. The network was trained on 80% of the population included in the study; 20% of the population was used as a pilot test. The network considered to have the highest level of prediction is the multi-perceptron network with 35 units, one hidden layer and 10 units in the hidden layer. The four dimensions of communication (clarity, attention, listening and terminology) were analysed separately to determine the importance of the independent variables for each one and compare them with one another. We examined the performance of the test across the entire range of possible values using the ROC curve analysis (Figure 3). Regarding the clarity of communication, training was performed on 684 cases, causally chosen (814% of the cases) and the test on the remaining 156 (186%). In our case, the values using the ROC curve analysis were higher than 90%. Therefore, the test was very accurate for the risk levels 1 and 3 and moderately accurate for risk level 2. During training, the neural network correctly predicted 825% of the cases and 865% during the test. Concerning the attention to communication, the training was performed on 669 cases, chosen on a casual basis (796% of the cases) and the test was conducted on the remaining 20%, equal to 171 (204%). The values using the ROC curve analysis were >90% and, therefore, corresponded to a probability that was 90% higher than a correct classification. Therefore, the test was very accurate for the risk levels 1 and 3 and moderately accurate for risk level 2. The neural network predicted correctly during training in 81% of the cases and 836% during the test. Pertaining to the terminology dimension, the training was performed on 689 cases, causally chosen (82% of the cases) © 2015 John Wiley & Sons Ltd

JAN: ORIGINAL RESEARCH: EMPIRICAL RESEARCH – QUANTITATIVE

and the test on the remaining 151 (18%). Also in this case, the test was very accurate. The neural network predicted correctly during training in 819% of the cases and 815% during the test. Concerning the listening dimension, the training was performed on 675 cases, causally chosen (804% of the cases) and the test on the remaining 196% was equal to 165. The values were >90% and, therefore, the test was very accurate. During training, the neural network correctly predicted 83% of the cases and 818% during the test. To determine the optimal series of variables to predict the expected result among the ones used in the input, a network sensitivity analysis was conducted. This analysis provided the relative importance of the variables included to determine the output, where 0 indicated a variable with no predictive validity and 1 indicated a variable with the maximum predictive validity. If we consider the importance of the independent variables of the four dimensions (clarity, attention, terminology, listening), we can observe the variables that played the most important role in the four dimensions (Figure 5). The age variable played an important role in all of the four dimensions: the ability to listen, pay attention, use precise terminology and clarity increased with age. Then followed seniority, which is also related to age. Gender and level of education did not have a statistically significant impact on the effectiveness of communication. This was because the data on the level of education did not allow its role in communication to be checked across the years in each

Risk prediction in ED using ANNs

professional. In fact, only education focusing on the communication between health professionals and patients could have had a strong impact on the SBAR scores.

Discussion To our knowledge, ANNs have never been used to predict and prevent harmful nurse–patient communication interactions to increase patient safety. We found that ANNs were able to correctly predict more than 80% of the communication characteristics including the communication risks. Answering to our two research questions, through ANNs we found that the nurses’ personal characteristics that enable to predict the risk of communication failures were age and years of working seniority and experience in the Emergency Department. Gender and education did not have a significant impact on communication. In the case of education, this would have had a significant impact on communication between nurses and patients only if education focused specifically on communication. With regard to the four communication dimensions, clarity and attention weighed most in predicting the risk of communication failures. In summary, all this means that a correct prediction of communication failures needs to be followed by appropriate educational interventions. Notwithstanding, the positive results that emerged from this study, there is still a long way to go in this direction because of the many limits.

Limitations Structure 30% 25% 20% Prof. Qualif.

Age

15% 10% 5% 0%

Gender

Education

Seniority Clarity

Attention

Terminology

Listening

Figure 5 The predictive performance of the independent variable. Source: Own elaboration. © 2015 John Wiley & Sons Ltd

Some of the variables we used were not appropriate to be specifically processed with the neural networks and therefore did not produce any information, so we had to collect other data, to limit subjective evaluations during observations. More specifically, with regard to the observers, we identified the absence of a variable that enabled to understand whether the observer had accessed the personal file of the person observed, and the opportunity to create the observer’s profile (personal data, education, scale rating objectiveness/trustworthiness/level of discrimination or something similar) to cross-check the data collected by the observers. With regard to the population observed, the personal data needed to be more detailed. For instance, the ‘age of the nurse observed’ was reported by the observer using a very wide age bracket. For instance, data on the nurses’ marital status were missing, and the years of education and incentives for continued education in the field of communication, including university courses. 2301

A. Bagnasco et al.

The variable ‘years of service’ did not include how many years of work were spent on a given facility or in general as a nurse. It would be better to state the number of years of nursing practice, the number of years worked in a given clinical facility and in the emergency department where the survey was conducted. The variable ‘context’ could not be included, because it was not stated in all of the observations. We also found that it was important to differentiate the observations made on admission or triage from those made on discharge. Another variable that was missing was the one indicating the nurses’ level of burnout, because it could be important to reduce the level of subjectivity of the survey on the efficacy of communication in the Emergency Department. We also found that it would be important to follow the same patient from admission to discharge, to learn more about the opinions of the patients and of those take care of them, because this would allow to control the variables related to aspects such as their emotional status and concerns. With regard to ANNS, for a specified data set there may be an infinite number of relevant network structures that can learn the characteristics of the data. The question remains: what size of network will be best for a specific data set. Unfortunately, it is not easy to answer this question. The neural network structure that offers the best result for a specific problem needs to be experimentally decided. The size, the characteristics of the training data set and the number of iterations are the other factors that affect the generalization (and therefore predictive) capabilities of a neural network. Generally, the network size influences its complexity and ‘learning time’ but, most importantly, it affects the generalization capabilities of the network. Generalization is the ability of the neural network to interpolate and extrapolate data that it has not seen before. The predictive power of an ANN depends on how well it generalizes from the training data.

Conclusion Nurses can play a very important role in healthcare organizations as safety nets, but very often the environment where they work harbours many latent risks, such as communication failures. The proactive analysis of the factors that contribute to communication failures not only between health professionals and patients but also among health professionals, could be described by the type of organizational culture and in addition stimulates reflection about the health professionals’ communication skills and about the factors linked to the working environments that have an influence on communication errors. 2302

This study has shown the potential of the artificial neural network in predicting the risk of communication failures in Emergency Departments. The model was developed on some selected input variables from the SBAR data. It achieved an accuracy of over 80%, which shows the potential efficacy of an Artificial Neural Network as a prediction tool. A ‘spin-off’ result of this study was the validation of Guilbert’s observation grid as a tool to predict communication failures. Early detection of communication failures in health professionals who have specific social and cultural characteristics would allow health organizations to arrange specific courses only or prevalently for those who have weak communication skills.

Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Conflict of interest No conflict of interest was declared by the authors in relation to the study itself. Note that Editor-in-Chief Roger Watson is a visiting professor at the University of Genoa and has been consulted about this paper but, in line with usual practice, this article was subjected to double-blind peer review and was edited by another editor.

Author contributions All authors have agreed on the final version and meet at least one of the following criteria [recommended by the ICMJE (http://www.icmje.org/recommendations/)]:

• •

substantial contributions to conception and design, acquisition of data or analysis and interpretation of data; drafting the article or revising it critically for important intellectual content.

References Ak M., Cinar O., Sutcigil L., Congologlu E.D., Haciomeroglu B., Canbaz H., Yaprak H., Jay L. & Ozmenler K.N. (2011) Communication skills training for emergency nurses. International Journal of Medical Sciences 8(5), 397–401. Amato F., L opez A., Pe~ na-Mendez E.M., Va nhara P., Hampl A. & Havel J. (2013) Artificial neural networks in medical diagnosis. Journal of Applied Biomedicine 11, 47–58. doi:10.2478/v10136012-0031-x. Bagnasco A., Tubino B., Piccotti E., Rosa F., Aleo G., Di Pietro P. & Sasso L. (2013) Identifying and correcting © 2015 John Wiley & Sons Ltd

JAN: ORIGINAL RESEARCH: EMPIRICAL RESEARCH – QUANTITATIVE communication failures among health professionals working in the Emergency Department. International Emergency Nursing 21, 168–172. Bagnasco A., Pagnucci N., Tolotti A., Rosa F., Torre G. & Sasso L. (2014) The role of simulation in developing communication and gestural skills in medical students. BMC Medical Education 14, 106. doi:10.1186/1472-6920-14-106. Bardan R.T. (2014) Artificial neural networks. Applications in urology. Timisoara Medical Journal 54, 289–296. Barbosa D., Roupar D., Ramos J., Tavares A. & Lima C. (2012) Automatic small bowel tumor diagnosis by using multi-scale wavelet-based analysis in wireless capsule endoscopy images. Biomedical Engineering Online 11, 3. doi:10.1186/1475-925X11-3. Catalogna M., Cohen E., Fishman S., Halpern Z., Nevo U. & BenJacob E. (2012) Artificial neural networks based controller for glucose monitoring during clamp test. PLoS ONE 7(8), e44587. doi:10.1371/journal.pone.0044587. Champion R., Kinsman L.D., Lee G.A., Masman K.A., May E.A., Taylor M.D., Thomas P.R. & Williams R.J. (2007) Forecasting emergency department presentations. Austrian Health Review 31, 83–90. Day S., Dalto J., Fox J. & Turpin M. (2006) Failure mode and effects analysis as a performance improvement tool in trauma. Journal of Trauma Nursing 13(3), 111–117. Dey P., Lamba A., Kumari S. & Marwaha N. (2011) Application of an artificial neural network in the prognosis of chronic myeloid leukemia. Analytical and Quantitative Cytology and Histology 33(6), 335–339. Fernandez de Canete J., Gonzalez-Perez S. & Ramos-Diaz J.C. (2012) Artificial neural networks for closed loop control of in silico and ad hoc type 1 diabetes. Computer Methods and Programs in Biomedicine 106, 55–66. Flin R., Martin L., Goeters K., Hoermann J., Amalberti R., Valot C. & Nijhuis H. (2003) The development of the NOTECHS system for evaluating pilots’ CRM skills. Human Factors and Aerospace Safety 3, 95–117. Guilbert J.-J. (1990) Guide pedagogique pour les personnels de sante. World Health Organization, Geneva, Switzerland. Guilbert J.-J. (2002) The ambiguous and bewitching power of knowledge, skills and attitudes leads to confusing statements of learning objectives. Education for Health 15(3), 362–369. Haig K.M., Sutton S. & Whittington J. (2006) SBAR: a shared mental model for improving communication between clinicians. The Joint Commission Journal on Quality and Patient Safety 32, 167–175. Hebb D.O. (1949) The Organization of Behavior. Wiley, New York. Hollander J.E., Blomkalns A.L., Brogan G.X., Diercks D.B., Field J.M., Garvey J.L., Gibler W.B., Henry T.D., Hoekstra J.W., Holroyd B.R., Hong Y., Kirk J.D., O’Neil B.J., Jackson R.E., Aufderheide T., Blomkalns A.L., Brogan G.X., Christenson J., Collins S., Diercks D.B., Fesmire F.M., Garvey J.L., Green G.B., Lindsell C.J., Peacock W.F., Pollack C.V., Zalenski R.; Multidisciplinary Standardized Reporting Criteria Task Force; Standardized Reporting Criteria Working Group of Emergency Medicine Cardiac Research and Education Group-International (2004) Standardized reporting guidelines for studies evaluating risk stratification of emergency department patients with

© 2015 John Wiley & Sons Ltd

Risk prediction in ED using ANNs

potential acute coronary syndromes. Annals of Emergency Medicine 44(6), 589–598. Hu X., Cammann H., Meyer H.A., Miller K., Jung K. & Stephan C. (2013) Artificial neural networks and prostate cancer-tools for diagnosis and management. Nature Reviews Urology 10(3), 174– 182. doi:10.1038/nrurol.2013.9. Irfan Khan Y., Zope P.H. & Suralkar S.R. (2013) Importance of Artificial Neural Network in Medical Diagnosis disease like acute nephritis disease and heart disease. International Journal of Engineering, Science and Innovative Technology 2(2), 210–217. Jones F., Podila P. & Powers C. (2013) Creating a culture of safety in the emergency department: the value of teamworking training. Journal of Nursing Administration 43(4), 194–200. Jones S.S., Thomas A., Evans R.S., Welch S.J., Haug P.J. & Snow G.L. (2008) Forecasting daily patient volumes in the emergency department. Academic Emergency Medicine 15, 159–170. Kilner E. & Sheppard L.A. (2010) The role of team work and communication in the emergency department: a systematic review. International Emergency Nursing 18(3), 127–137. Lago P., Bizzarri G., Scalzotto F., Parpaiola A., Amigoni A., Putoto G. & Perilongo G. (2012) Use of FMEA analysis to reduce risk of errors in prescribing and administering drugs in paediatric wards: a quality improvement report. BMJ Open 2(6), pii: e001249. doi: 10.1136/bmjopen-2012-001249. Landsberger H.A. (1958) Hawthorne Revisited. Cornell University Press, Ithaca. Li Y., Rauth A.M. & Wu X.Y. (2005) Prediction of kinetics of doxorubicin release from sulfopropyl dextran ion-exchange microspheres using artificial neural networks. European Journal of Pharmaceutical Sciences 24, 401–410. Liu B. & Jiang Y. (2013) A multi-target training method for artificial neural network with application to computer-aided diagnosis. Medical Physics 40, 011908. doi:10.1118/1.4772021. McCulloch W.S. & Pitts W.H. (1943) A logical calculus of the ideas imminent in nervous activity. The Bulletin of Mathematical Biophysics 5, 115–133. Meginniss A., Damian F. & Falvo F. (2012) Time out for patient safety. Journal of Emergency Nursing 38(1), 51–53. Pandini A., Fraccalvieri D. & Bonati L. (2013) Artificial neural networks for efficient clustering of conformational ensembles and their potential for medicinal chemistry. Current Topics in Medical Chemistry 13(5), 642–651. Patel A., Mehta T., Patel M., Patel K. & Patel N. (2012) Design porosity osmotic tablet for delivering low and pH-dependent soluble drug using an Artificial Neural Network. Current Drug Delivery 9, 459–467. Qeethara Kadhim A-S. (2011) Artificial neural network in medical diagnosis. International Journal of Computer Science 8(2), 150– 153. Rodriguez Galdon B., Penna-Mendez E., Havel J., Rodriguez E. & Diaz Romero C. (2010) Cluster analysis and artificial neural networks multivariate classification of onion varieties. Journal of Agricultural Food Chemistry 58, 11435–11440. Rosenblatt F. (1958) The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review 65, 386–408. Saghiri M., Asgar K., Boukani K., Lotfi M., Aghili H., Delvarani A., Karamifar K., Saghiri A., Mehrvarzfar P. & Garcia-Godoy F.

2303

A. Bagnasco et al. (2012) A new approach for locating the minor apical foramen using an artificial neural network. International Endodontic Journal 45, 257–265. Siri A. (2014) Predicting Students’ Academic Dropout Using Artificial Neural Network. Nova Science Publishers, Inc. U.S., Commack, NY. Spencer R., Logan P. & Coiera E. (2002) Supporting Communication in the Emergency Department. Centre for Health Informatics. University of New South Wales, Sidney, NSW. Schweigler L.M., Desmond J.S., McCarthy M.L., Bukowski K.J., Ionides E.L. & Younger J.G. (2009) Forecasting models of emergency department crowding. Academic Emergency Medicine 16, 301–308. Werbos P.J. (1974) Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard

University, Published in the The roots of backpropagation: from ordered derivatives to neural networks and political forecasting (Vol. 1) Werbos, P. J. (1994) John Wiley & Sons. New York. Wesolowski M. & Suchacz B. (2012) Artificial neural networks: theoretical background and pharmaceutical applications: a review. Journal of AOAC International 95, 652–668. Woods K. & Bower K.W. (1997) Generating ROC curves for artificial neural networks. IEEE Transactions on Medical Imaging 16, 329–337. Xu M., Wong T.C. & Chin K.S. (2013) Modeling daily patient arrivals at Emergency Department and quantifying the relative importance of contributing variables using artificial neural network. Decision Support Systems 54, 1488–1498. Zhang Q.-J. & Gupta K.C. (2000) Neural Networks for RF and Microwave Design, Chapter 3. Artech House Inc., Boston, MA, pp. 61–103.

The Journal of Advanced Nursing (JAN) is an international, peer-reviewed, scientific journal. JAN contributes to the advancement of evidence-based nursing, midwifery and health care by disseminating high quality research and scholarship of contemporary relevance and with potential to advance knowledge for practice, education, management or policy. JAN publishes research reviews, original research reports and methodological and theoretical papers. For further information, please visit JAN on the Wiley Online Library website: www.wileyonlinelibrary.com/journal/jan Reasons to publish your work in JAN:

• High-impact forum: the world’s most cited nursing journal, with an Impact Factor of 1·527 – ranked 14/101 in the 2012 ISI Journal Citation Reports © (Nursing (Social Science)).

• Most read nursing journal in the world: over 3 million articles downloaded online per year and accessible in over 10,000 libraries worldwide (including over 3,500 in developing countries with free or low cost access).

• • • •

2304

Fast and easy online submission: online submission at http://mc.manuscriptcentral.com/jan. Positive publishing experience: rapid double-blind peer review with constructive feedback. Rapid online publication in five weeks: average time from final manuscript arriving in production to online publication. Online Open: the option to pay to make your article freely and openly accessible to non-subscribers upon publication on Wiley Online Library, as well as the option to deposit the article in your own or your funding agency’s preferred archive (e.g. PubMed).

© 2015 John Wiley & Sons Ltd

Applying artificial neural networks to predict communication risks in the emergency department.

To describe the utility of artificial neural networks in predicting communication risks...
460KB Sizes 0 Downloads 7 Views