PM R 7 (2015) 555-561
www.pmrjournal.org
Original ResearcheCME
Advanced Joystick Algorithms for Computer Access Tasks Brad E. Dicianno, MD, Harshal Mahajan, PhD, Rory A. Cooper, PhD
Abstract Objective: To compare 2 correction algorithms and 2 joysticks (a conventional movement-sensing joystick and a custom-designed isometric joystick) in computer access tasks. Design: Repeated-measures, within-subject. Setting: National Veterans Wheelchair Games. Participants: Fifteen participants with various diagnoses including multiple sclerosis, spinal cord injury, traumatic brain injury, Wilson disease, and Parkinson disease. Methods: A computer access test scenario was used to evaluate the effects of applying proportional integral derivative (PID)e based and least meansebased algorithms to suppress unintentional cursor motions by users with upper extremity spasticity. Main Outcome Measures: Trial completion time, reaction time, and trajectory-based measures: movement offset, movement variability, and percentage of out-of-path motion on test tracks. Results: The quantitative outcome measures showed a high correlation with clinical measures for spasticity and functional independence. On small test tracks, compared to when no correction algorithms were used, both algorithms performed equally well in suppressing unintentional cursor motions. On longer test tracks, participants navigated most accurately while using the PID algorithm. Participants moved the cursor more accurately using the isometric joystick compared to the movement-sensing joystick, with only a slight increase in the task completion times. Conclusions: The joysticks and the advanced correction algorithms show promise for use in wide-ranging applications as control interfaces.
Introduction Joysticks are commonly used as a control interface for power wheelchairs and video games. However, with the advent of Bluetooth technology, joysticks can also be used for computer access, such as to navigate a computer screen or to perform other mouse operations such as “clicking.” Standard joysticks are also known as “movement-sensing” joysticks (MSJ) because they respond to the tilt applied to the joystick’s post. Isometric joysticks (IJs), on the other hand, have rigid posts and sense the forces that are applied; they do not change position [1]. We have developed an IJ that has shown promise for use in power wheelchair driving [2,3] and computer access tasks [4,5]. We have also developed advanced algorithms that can be applied to various input interfaces such as joysticks and mice to correct for errors that may occur because of involuntary movements or problems with motor control [4]. The purpose of this study is to compare
joysticks and to compare the performance of 2 algorithms and a control condition (no algorithm) while users with disabilities perform computer-based tracking. Our first hypothesis was that the participants would perform better when using the IJ than when using the MSJ. Our second hypothesis was that higher levels of spasticity in users’ operating limbs would be inversely associated with performance, whereas higher levels of independence in activities of daily living (ADLs) would be positively associated with performance. Our third hypothesis was that application of the algorithms would result in improved performance compared to no algorithm, regardless of joystick type used. Methods This study was approved by the VA Pittsburgh Healthcare System Institutional Review Board. Participants were recruited at the National Veterans Wheelchair Games event. Inclusion criteria were age between
1934-1482/$ - see front matter ª 2015 by the American Academy of Physical Medicine and Rehabilitation http://dx.doi.org/10.1016/j.pmrj.2014.12.009
556
Joystick Algorithms for Computer Access Tasks
18 and 80 years, use of manual or power wheelchair for day-to-day mobility, and upper limb spasticity or rigidity. Exclusion criteria were inability to tolerate sitting for 3.5 hours, and active pelvic or thigh wounds (which may be worsened by prolonged sitting). After participants provided informed consent, they underwent a clinical evaluation for upper extremity spasticity using the Modified Ashworth Scale [6,7] (0 indicates no spasticity and 4 indicates complete rigidity). The presence or absence of resting or intentional tremors was also noted. Participants were ranked according to their independence in ADLs using the Barthel Index [8,9]. Participants used a conventional Quickie [10] brand MSJ and our IJ [1,3]. The joysticks were positioned according to the participant’s preferences by using a custom-made mounting device. The IJ and the MSJ were tuned/customized for each participant according to a previously validated protocol [2,11,12] that allowed the researchers to create a dead zone in which unintentional resting movements of the user’s arm created no joystick output. We also biased the directional axes of the joystick according to the participant’s preference, adjusted the maximum force the controller would recognize, and customized the gain. The IJ and MSJ when connected to the computer moved a cursor. Both joysticks were programmed with identical cursor speed and acceleration parameters. Computer access test scenarios were developed in Cþþ [13] using OpenGL [14]. Every participant used both joysticks to trace 3 different paths (Figure 1) of varying levels of difficulty. Mirror images of paths were also used to control for individual bias to handedness and directional preference. The design of paths were based on recommendations in the International Organization for Standardization (ISO) standard 9241-9:2000 [15]. The tasks were 50 pixels wide and simulated commonly used cursor motions such as steering a cursor along straight, circular, and a combination of multiple straight and circular paths. The “ideal” trajectory was defined as the center of the path. Two algorithms were also developed. The proportional integral derivative (PID) algorithm (Equation 1), corrects error in a trajectory by measuring and minimizing the differences between the “ideal path” and cursor location, the degree to which the movements overshoot or
undershoot the ideal path, and the degree of oscillation about the ideal path. Our pilot work [4] showed the PID algorithm significantly improved tracking performance and trial times. Aiming to follow the ideal trajectory Sideal(t), the participant applies a certain force to the joystick. The joystick transduces this force and expects the cursor to follow the desired trajectory SDesired(t). The proportional P(t), integral I(t), and derivative D(t) factors, which are functions of the error e(t) between the ideal and desired trajectories, then get added to the desired trajectory to give the actual cursor trajectory SActual(t). eðtÞ ¼ SDesired ðtÞ SIdeal ðtÞ PðtÞ ¼ Kp eðtÞ IðtÞ ¼ Iðt 1Þ þ Ki eðtÞ Dt ðeðt 1Þ þ eðtÞÞ Dt SActual ðtÞ ¼ SDesired ðtÞ þ Pout ðtÞ þ Iout ðtÞ þ Dout ðtÞ DðtÞ ¼ Kd
Equation 1: Error correction using PID-based algorithm. We also used a least means (LM) algorithm [16,17] that uses the speed error and direction error to correct tracking trajectory (Equation 2). The actual trajectory has a component of trajectory error, controlled by an adaptive gain factor ma.. eðtÞ ¼ SDesired ðtÞ SIdeal ðtÞ SActual ðtÞ ¼ SDesired ðtÞ þ ma eðtÞ Equation 2: Error correction using least meansebased algorithm. Adequate familiarization with the tasks was allowed before starting trials. Participants gave a verbal confirmation when they felt confident in controlling the onscreen cursor. Every trial started with a prompt that read “3, 2, 1, GO!” Participants were instructed to move the cursor along the tracks between regions marked as “Start” and “Finish” as quickly as possible while staying within the borders of the task. Participants were asked to complete each of the 3 tasks and their mirror images 3
Figure 1. The 3 main tasks in the present study: straight, circular, and combined.
B.E. Dicianno et al. / PM R 7 (2015) 555-561
times during which the PID correction algorithm, the LM correction algorithm, and no correction were applied to the cursor. In all, every participant performed a total of 108 (3 2 paths 2 joysticks 3 correction algorithm states 3 repetitions) trials. The sequence of joysticks used by participants was randomized. For each of the 2 joysticks, the order of the displayed tasks and the correction algorithm state were also randomized, and participants did not know whether any correction was applied or which path would be displayed next. Timestamped state variables (sampling rate w60 Hz), such as joystick input and cursor screen location, were recorded for every trial, and the following outcomes measures, which are similar to those collected in other computer access studies [18,19], were obtained: Trial time (TT): Time (in seconds) to complete 1 trial by moving the cursor from “Start” to “End” region. Reaction time (RT): Time (in seconds) taken by the user from the trial start to apply at least 0.125 N force to the joystick. Movement offset (MO): Mean deviation of the cursor path from the desired path in pixels. This variable indicates whether the user is biased toward one side of the path. Movement variability (MV): Standard deviation of movement error. This is an estimate of variation in error values between the desired and actual trajectories. Percentage out-of-path movement (POPM): Percentage of the user’s trajectory that falls outside the track boundaries. Statistical analyses were performed using SPSS software [20]. All a levels were set to 0.05. Because no statistically significant differences were seen between performance measures on the 3 main paths (straight, circular, and combined) and their mirror image paths, all outcome measures were averaged to increase power and to avoid bias resulting from hand preference. Similarly, the performance measures from the 3 repetitions were averaged to obtain representative values. A log transformation was applied to RT, MO, and MV to compensate for their skewed distributions. Within-subject repeatedmeasures multivariate analyses of variance (MANOVAs) were performed to evaluate for differences in performance outcome variables among the within-subjects factors of control algorithm conditions and the 2 joysticks. Because the tasks were very different from each other, separate analyses were performed for each. Post hoc univariate analyses were performed when the main effects were significant. Bonferroni corrections were applied. Bivariate correlations were used to evaluate for an effect on outcomes measures due to previous experience with joysticks, spasticity (measured by Modified Ashworth Scale), and Barthel Index scores (higher score indicates greater functional ability).
557
Results Fifteen participants were recruited. The average age was 52.13 9.26 years. Three participants were female. Diagnoses included multiple sclerosis (n ¼ 7), cervical spinal cord injury (SCI) (n ¼ 6), Wilson disease (n ¼ 1), and Parkinson disease (n ¼ 1). One participant with SCI also had a traumatic brain injury. Ten participants were regular computer users. Four participants used manual wheelchairs as their primary means of mobility, 8 participants used power wheelchairs, and 3 used scooters. Eight participants had prior experience with using joysticks to access computers and/or power wheelchairs. Six participants, in addition to spasticity, had resting tremor or intentional tremor in the limb that they used to operate the control interface. Irrespective of which joystick was used, higher scores on the Modified Ashworth Scale (Table 1) were significantly positively associated with the outcome measures MO, MV, and POPM. While using the IJ, higher Barthel Index was associated with higher trial completion time, lower MO, lower MV, and lower POPM. When using the MSJ, higher Barthel Index was associated with higher RT, but no other associations with outcome variables were found. Having prior experience with a computer or wheelchair joystick was associated with a higher RT (Pearson’s correlation ¼ 0.27, P ¼.001) when participants used an IJ. No associations were significant with any other outcome measure when participants used the MSJ. The MANOVAs for all 3 tracks indicated that the main effects of algorithm (P < .001 for all tracks; partial h2 ¼ 0.70e0.88) and joystick (P < .001 for all tracks; partial h2 ¼ 0.83e0.86) were significant. Post hoc analyses for algorithms (Table 2) showed no differences in TT or RT when no algorithm, PID algorithm, and LM algorithms were used. For all 3 tracks, when no suppression algorithms were used, MO (P < .001 partial h2 ¼ 0.851e0.961) and MV (P < .001 partial h2 ¼ 0.879e0.973) were more than twice, and POPM (P < .001 partial h2 ¼ 0.632e0.752) was more than 3 times the values than when either LM or PID was used. Between PID or LM algorithms, no differences were seen in any outcome variables on straight or circular tracks. The differences between the 2 algorithms were statistically significant only on the combined track. The outcome variables MO, MV, and POPM were smaller when the PID algorithm was used than when the LMN algorithm was used. Post hoc analyses for the joysticks (Table 3) showed that participants had significantly higher TT for the IJ than for the MSJ on all 3 tracks (straight track P ¼ .023, partial h2 ¼ 0.318; circular track P ¼ .021, partial h2 ¼ 0.326; and combined track P ¼ .041, partial h2 ¼ 0.265). Reaction times were higher for the MSJ than for the IJ for the straight track (P ¼ .012, partial h2 ¼ 0.373) and combined track (P ¼ .013, partial h2 ¼ 0.364) and not significant for the circular track. No
558
Joystick Algorithms for Computer Access Tasks
Table 1 Correlations between Modified Ashworth Scale score and Barthel Index with the outcome variables Modified Ashworth Scale
Barthel Index (IJ)
Barthel Index (MSJ)
Outcomes
Pearson’s Correlation Coefficient
P value
Pearson’s Correlation Coefficient
P Value
Pearson’s Correlation Coefficient
P Value
Trial time Reaction time Movement offset Movement variability % Out-of-path motion
0.018 0.023 0.21* 0.18* 0.2*
.767 .702
www.pmrjournal.org
Original ResearcheCME
Advanced Joystick Algorithms for Computer Access Tasks Brad E. Dicianno, MD, Harshal Mahajan, PhD, Rory A. Cooper, PhD
Abstract Objective: To compare 2 correction algorithms and 2 joysticks (a conventional movement-sensing joystick and a custom-designed isometric joystick) in computer access tasks. Design: Repeated-measures, within-subject. Setting: National Veterans Wheelchair Games. Participants: Fifteen participants with various diagnoses including multiple sclerosis, spinal cord injury, traumatic brain injury, Wilson disease, and Parkinson disease. Methods: A computer access test scenario was used to evaluate the effects of applying proportional integral derivative (PID)e based and least meansebased algorithms to suppress unintentional cursor motions by users with upper extremity spasticity. Main Outcome Measures: Trial completion time, reaction time, and trajectory-based measures: movement offset, movement variability, and percentage of out-of-path motion on test tracks. Results: The quantitative outcome measures showed a high correlation with clinical measures for spasticity and functional independence. On small test tracks, compared to when no correction algorithms were used, both algorithms performed equally well in suppressing unintentional cursor motions. On longer test tracks, participants navigated most accurately while using the PID algorithm. Participants moved the cursor more accurately using the isometric joystick compared to the movement-sensing joystick, with only a slight increase in the task completion times. Conclusions: The joysticks and the advanced correction algorithms show promise for use in wide-ranging applications as control interfaces.
Introduction Joysticks are commonly used as a control interface for power wheelchairs and video games. However, with the advent of Bluetooth technology, joysticks can also be used for computer access, such as to navigate a computer screen or to perform other mouse operations such as “clicking.” Standard joysticks are also known as “movement-sensing” joysticks (MSJ) because they respond to the tilt applied to the joystick’s post. Isometric joysticks (IJs), on the other hand, have rigid posts and sense the forces that are applied; they do not change position [1]. We have developed an IJ that has shown promise for use in power wheelchair driving [2,3] and computer access tasks [4,5]. We have also developed advanced algorithms that can be applied to various input interfaces such as joysticks and mice to correct for errors that may occur because of involuntary movements or problems with motor control [4]. The purpose of this study is to compare
joysticks and to compare the performance of 2 algorithms and a control condition (no algorithm) while users with disabilities perform computer-based tracking. Our first hypothesis was that the participants would perform better when using the IJ than when using the MSJ. Our second hypothesis was that higher levels of spasticity in users’ operating limbs would be inversely associated with performance, whereas higher levels of independence in activities of daily living (ADLs) would be positively associated with performance. Our third hypothesis was that application of the algorithms would result in improved performance compared to no algorithm, regardless of joystick type used. Methods This study was approved by the VA Pittsburgh Healthcare System Institutional Review Board. Participants were recruited at the National Veterans Wheelchair Games event. Inclusion criteria were age between
1934-1482/$ - see front matter ª 2015 by the American Academy of Physical Medicine and Rehabilitation http://dx.doi.org/10.1016/j.pmrj.2014.12.009
556
Joystick Algorithms for Computer Access Tasks
18 and 80 years, use of manual or power wheelchair for day-to-day mobility, and upper limb spasticity or rigidity. Exclusion criteria were inability to tolerate sitting for 3.5 hours, and active pelvic or thigh wounds (which may be worsened by prolonged sitting). After participants provided informed consent, they underwent a clinical evaluation for upper extremity spasticity using the Modified Ashworth Scale [6,7] (0 indicates no spasticity and 4 indicates complete rigidity). The presence or absence of resting or intentional tremors was also noted. Participants were ranked according to their independence in ADLs using the Barthel Index [8,9]. Participants used a conventional Quickie [10] brand MSJ and our IJ [1,3]. The joysticks were positioned according to the participant’s preferences by using a custom-made mounting device. The IJ and the MSJ were tuned/customized for each participant according to a previously validated protocol [2,11,12] that allowed the researchers to create a dead zone in which unintentional resting movements of the user’s arm created no joystick output. We also biased the directional axes of the joystick according to the participant’s preference, adjusted the maximum force the controller would recognize, and customized the gain. The IJ and MSJ when connected to the computer moved a cursor. Both joysticks were programmed with identical cursor speed and acceleration parameters. Computer access test scenarios were developed in Cþþ [13] using OpenGL [14]. Every participant used both joysticks to trace 3 different paths (Figure 1) of varying levels of difficulty. Mirror images of paths were also used to control for individual bias to handedness and directional preference. The design of paths were based on recommendations in the International Organization for Standardization (ISO) standard 9241-9:2000 [15]. The tasks were 50 pixels wide and simulated commonly used cursor motions such as steering a cursor along straight, circular, and a combination of multiple straight and circular paths. The “ideal” trajectory was defined as the center of the path. Two algorithms were also developed. The proportional integral derivative (PID) algorithm (Equation 1), corrects error in a trajectory by measuring and minimizing the differences between the “ideal path” and cursor location, the degree to which the movements overshoot or
undershoot the ideal path, and the degree of oscillation about the ideal path. Our pilot work [4] showed the PID algorithm significantly improved tracking performance and trial times. Aiming to follow the ideal trajectory Sideal(t), the participant applies a certain force to the joystick. The joystick transduces this force and expects the cursor to follow the desired trajectory SDesired(t). The proportional P(t), integral I(t), and derivative D(t) factors, which are functions of the error e(t) between the ideal and desired trajectories, then get added to the desired trajectory to give the actual cursor trajectory SActual(t). eðtÞ ¼ SDesired ðtÞ SIdeal ðtÞ PðtÞ ¼ Kp eðtÞ IðtÞ ¼ Iðt 1Þ þ Ki eðtÞ Dt ðeðt 1Þ þ eðtÞÞ Dt SActual ðtÞ ¼ SDesired ðtÞ þ Pout ðtÞ þ Iout ðtÞ þ Dout ðtÞ DðtÞ ¼ Kd
Equation 1: Error correction using PID-based algorithm. We also used a least means (LM) algorithm [16,17] that uses the speed error and direction error to correct tracking trajectory (Equation 2). The actual trajectory has a component of trajectory error, controlled by an adaptive gain factor ma.. eðtÞ ¼ SDesired ðtÞ SIdeal ðtÞ SActual ðtÞ ¼ SDesired ðtÞ þ ma eðtÞ Equation 2: Error correction using least meansebased algorithm. Adequate familiarization with the tasks was allowed before starting trials. Participants gave a verbal confirmation when they felt confident in controlling the onscreen cursor. Every trial started with a prompt that read “3, 2, 1, GO!” Participants were instructed to move the cursor along the tracks between regions marked as “Start” and “Finish” as quickly as possible while staying within the borders of the task. Participants were asked to complete each of the 3 tasks and their mirror images 3
Figure 1. The 3 main tasks in the present study: straight, circular, and combined.
B.E. Dicianno et al. / PM R 7 (2015) 555-561
times during which the PID correction algorithm, the LM correction algorithm, and no correction were applied to the cursor. In all, every participant performed a total of 108 (3 2 paths 2 joysticks 3 correction algorithm states 3 repetitions) trials. The sequence of joysticks used by participants was randomized. For each of the 2 joysticks, the order of the displayed tasks and the correction algorithm state were also randomized, and participants did not know whether any correction was applied or which path would be displayed next. Timestamped state variables (sampling rate w60 Hz), such as joystick input and cursor screen location, were recorded for every trial, and the following outcomes measures, which are similar to those collected in other computer access studies [18,19], were obtained: Trial time (TT): Time (in seconds) to complete 1 trial by moving the cursor from “Start” to “End” region. Reaction time (RT): Time (in seconds) taken by the user from the trial start to apply at least 0.125 N force to the joystick. Movement offset (MO): Mean deviation of the cursor path from the desired path in pixels. This variable indicates whether the user is biased toward one side of the path. Movement variability (MV): Standard deviation of movement error. This is an estimate of variation in error values between the desired and actual trajectories. Percentage out-of-path movement (POPM): Percentage of the user’s trajectory that falls outside the track boundaries. Statistical analyses were performed using SPSS software [20]. All a levels were set to 0.05. Because no statistically significant differences were seen between performance measures on the 3 main paths (straight, circular, and combined) and their mirror image paths, all outcome measures were averaged to increase power and to avoid bias resulting from hand preference. Similarly, the performance measures from the 3 repetitions were averaged to obtain representative values. A log transformation was applied to RT, MO, and MV to compensate for their skewed distributions. Within-subject repeatedmeasures multivariate analyses of variance (MANOVAs) were performed to evaluate for differences in performance outcome variables among the within-subjects factors of control algorithm conditions and the 2 joysticks. Because the tasks were very different from each other, separate analyses were performed for each. Post hoc univariate analyses were performed when the main effects were significant. Bonferroni corrections were applied. Bivariate correlations were used to evaluate for an effect on outcomes measures due to previous experience with joysticks, spasticity (measured by Modified Ashworth Scale), and Barthel Index scores (higher score indicates greater functional ability).
557
Results Fifteen participants were recruited. The average age was 52.13 9.26 years. Three participants were female. Diagnoses included multiple sclerosis (n ¼ 7), cervical spinal cord injury (SCI) (n ¼ 6), Wilson disease (n ¼ 1), and Parkinson disease (n ¼ 1). One participant with SCI also had a traumatic brain injury. Ten participants were regular computer users. Four participants used manual wheelchairs as their primary means of mobility, 8 participants used power wheelchairs, and 3 used scooters. Eight participants had prior experience with using joysticks to access computers and/or power wheelchairs. Six participants, in addition to spasticity, had resting tremor or intentional tremor in the limb that they used to operate the control interface. Irrespective of which joystick was used, higher scores on the Modified Ashworth Scale (Table 1) were significantly positively associated with the outcome measures MO, MV, and POPM. While using the IJ, higher Barthel Index was associated with higher trial completion time, lower MO, lower MV, and lower POPM. When using the MSJ, higher Barthel Index was associated with higher RT, but no other associations with outcome variables were found. Having prior experience with a computer or wheelchair joystick was associated with a higher RT (Pearson’s correlation ¼ 0.27, P ¼.001) when participants used an IJ. No associations were significant with any other outcome measure when participants used the MSJ. The MANOVAs for all 3 tracks indicated that the main effects of algorithm (P < .001 for all tracks; partial h2 ¼ 0.70e0.88) and joystick (P < .001 for all tracks; partial h2 ¼ 0.83e0.86) were significant. Post hoc analyses for algorithms (Table 2) showed no differences in TT or RT when no algorithm, PID algorithm, and LM algorithms were used. For all 3 tracks, when no suppression algorithms were used, MO (P < .001 partial h2 ¼ 0.851e0.961) and MV (P < .001 partial h2 ¼ 0.879e0.973) were more than twice, and POPM (P < .001 partial h2 ¼ 0.632e0.752) was more than 3 times the values than when either LM or PID was used. Between PID or LM algorithms, no differences were seen in any outcome variables on straight or circular tracks. The differences between the 2 algorithms were statistically significant only on the combined track. The outcome variables MO, MV, and POPM were smaller when the PID algorithm was used than when the LMN algorithm was used. Post hoc analyses for the joysticks (Table 3) showed that participants had significantly higher TT for the IJ than for the MSJ on all 3 tracks (straight track P ¼ .023, partial h2 ¼ 0.318; circular track P ¼ .021, partial h2 ¼ 0.326; and combined track P ¼ .041, partial h2 ¼ 0.265). Reaction times were higher for the MSJ than for the IJ for the straight track (P ¼ .012, partial h2 ¼ 0.373) and combined track (P ¼ .013, partial h2 ¼ 0.364) and not significant for the circular track. No
558
Joystick Algorithms for Computer Access Tasks
Table 1 Correlations between Modified Ashworth Scale score and Barthel Index with the outcome variables Modified Ashworth Scale
Barthel Index (IJ)
Barthel Index (MSJ)
Outcomes
Pearson’s Correlation Coefficient
P value
Pearson’s Correlation Coefficient
P Value
Pearson’s Correlation Coefficient
P Value
Trial time Reaction time Movement offset Movement variability % Out-of-path motion
0.018 0.023 0.21* 0.18* 0.2*
.767 .702
