334
Original article
Influence of complementing a robotic upper limb rehabilitation system with video games on the engagement of the participants: a study focusing on muscle activities Chong Lia,b, Zoltán Rusáka, Imre Horvátha and Linhong Jib Efficacious stroke rehabilitation depends not only on patients’ medical treatment but also on their motivation and engagement during rehabilitation exercises. Although traditional rehabilitation exercises are often mundane, technology-assisted upper-limb robotic training can provide engaging and task-oriented training in a natural environment. The factors that influence engagement, however, are not fully understood. This paper therefore studies the relationship between engagement and muscle activities as well as the influencing factors of engagement. To this end, an experiment was conducted using a robotic upper limb rehabilitation system with healthy individuals in three training exercises: (a) a traditional exercise, which is typically used for training the grasping function, (b) a tracking exercise, currently used in robot-assisted stroke patient rehabilitation for fine motor movement, and (c) a video game exercise, which is a proliferating approach of robot-assisted rehabilitation enabling high-level active engagement of stroke patients. These exercises differ not only in the characteristics of the motion that they use but also in their method of triggering engagement. To measure the level of engagement, we used facial expressions, motion analysis of the arm movements, and electromyography. The results show that (a) the video game exercise could engage the participants for a longer period than the other two
Introduction Motivation and engagement are important factors in rehabilitation and are used frequently as determinants of rehabilitation outcomes (Tupper and Henley, 1987; Grant et al., 2004; Holden, 2005; Colombo et al., 2007). A positive rehabilitation outcome is associated strongly with high patient motivation and engagement (Langhorne et al., 2011). Engagement in rehabilitation exercises has been defined as a construct that is driven by motivation and executed through active, effortful participation (Lequerica and Kortte, 2010). It extends the definition of participation beyond exercise attendance and motivation (Kortte et al., 2007). Research has shown that active participation during rehabilitation promotes cortical plasticity through cortical map reorganization (Lynskey et al., 2008) and that active participation level is influenced by the level of engagement (Lequerica and Kortte, 2010). Therefore, increased engagement during rehabilitation exercise is essential to good rehabilitation outcomes. 0342-5282 © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins
exercises, (b) the engagement level decreased when the participants became too familiar with the exercises, and (c) analysis of normalized root mean square in electromyographic data indicated that muscle activities were more intense when the participants are engaged. This study shows that several sub-factors on engagement, such as versatility of feedback, cognitive tasks, and competitiveness, may influence engagement more than the others. To maintain a high level of engagement, the rehabilitation system needs to be adaptive, providing different exercises to engage the participants. International Journal of Rehabilitation Research 37:334–342 © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. International Journal of Rehabilitation Research 2014, 37:334–342 Keywords: engagement level, exercise integrated with video game, influencing factors of engagement, muscle activity, upper limb rehabilitation a Faculty of Industrial Design Engineering, Delft University of Technology, Delft, The Netherlands and bDivision of Intelligent and Bio-mimetic Machinery, The State Key Laboratory of Tribology, Department of Mechanical Engineering, Tsinghua University, Beijing, China
Correspondence to Chong Li, Landbergstraat 15, 2628 CE, Delft, The Netherlands Tel: +31 (0)15 27 83629; fax: +31 (0)15 27 81839; e-mail: [email protected] Received 6 May 2014 Accepted 28 July 2014
Traditional rehabilitation tasks, however, are often mundane and can lead to a lack of patient motivation, resulting in little or no independent patient exercise taking place (Burke et al., 2008). Technology-assisted upper-limb training can provide engaging and taskoriented training in a natural environment using patient-tailored feedback to support learning of motor skills (Timmermans et al., 2009). However, there is little understanding of which factors influence engagement. Although it has been suggested that an individual’s willingness and capacity as well as social and physical aspects of the environment are the influencing factors of engagement in medical rehabilitation (Lequerica and Kortte, 2010), the causalities of these factors are not yet fully understood. According to our review (Li et al., 2014a), these factors can be categorized into five aspects: motor, perceptual, cognitive, emotional, and social factors. The first objective of this paper is to identify subfactors that contribute toward engagement and analyze their causalities. DOI: 10.1097/MRR.0000000000000076
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Study of engagement and muscle activity Li et al. 335
Table 1
Sub-factors of engagement
Engaging methods Motor
Perceptual Cognitive Emotional Social
Sub-factors
Video game exercise
Versatility of motion Motion envelope Velocity of motion Versatility of feedback Cognitive tasks
Random
Competiveness
Tracking exercise
Regular Analyzed by motion characteristics Analyzed by motion characteristics Interaction with video game Continuous feedback Attention and problem solving Attention and adjustment Not analyzed in this experiment Play against the computer Track more precisely
Another objective of our paper is to further study the relationship between engagement and muscle activities to investigate whether there is any reflection when the participants’ engagements vary. A recent attempt aimed at exploring the relationship of the level of engagement and muscle activity (Zimmerli et al., 2013). The authors argue that electromyography (EMG) can be used to measure the level of engagement as they found that increased level of engagement represents increased muscle activities. This study, however, neglected the effect of movement velocity on the muscle activities, which, according to other studies, also has a significant influence on the amplitude of measured EMG signals (Somasundaram, 1974; US Department of Health and Human Services, 1992). To realize these two objectives, an experiment was designed and conducted, which involved different training exercises, a video game exercise, a tracking exercise, and a traditional exercise. These three exercises involve different factors influencing the level of engagement. The Methods section explains how the experiment was conducted and how the measurements were acquired. The next section analyzes the causality of the influencing factors of engagement by comparing the results of engaging performance in each exercise, and processes the EMG data at different engagement levels to investigate the relationship between the two variables. The last section presents discussions of the results and the conclusion.
Traditional exercise Regular
Discontinuous feedback Attention No
As shown in Table 1, it lists the differences in the subfactors of engagement that each exercise could address. The motion envelope and velocity of motion will be analyzed using the motion characteristics. In the third section, the influence of these sub-factors on engagement will be analyzed on the basis of the performance of the participants in each exercise. Video game exercise
In the video game exercise, an upper limb rehabilitation robot (Fig. 1) (Li et al., 2014b) was used. There are passive and active modes in this robot system. In the passive mode, the users do not have to move while the robotic arms are driven by motors, whereas in the active mode, users are required to move the robotic arms voluntarily. In the video game exercise, the participant was required to complete the game tasks in the active mode. The active mode is operated by a human computer interface, which enables the user to interact with the game. The interface is located on the handle grip, which consists of several force sensors that sense the force exerted by the hand. When the force is larger than the threshold set in the system, it will generate a click of the left button of the system mouse. Similarly, when the force is smaller, it Fig. 1
Methods Experiment design
One objective of this study is to understand the causalities of influencing factors on engagement. As mentioned above, there are five aspects of influencing factors, namely, motor, perceptual, cognitive, emotional, and social. These factors could be further divided into subfactors. To test the influence of these factors on engagement level, this experiment was designed, in which a video game exercise, a tracking exercise, and a traditional exercise were used, as these exercises addressed different influencing factors. The sub-factors studied in this paper are versatility of motion, motion envelope, velocity of motion, versatility of feedback, cognitive tasks, and competitiveness.
Upper limb rehabilitation robot.
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
336 International Journal of Rehabilitation Research 2014, Vol 37 No 4
will generate a release, and when it is larger continuously, it will simulate a drag of the mouse. At the same time, when the hand moves, the position of the handle grip can be calculated by two encoders in two servo motors. Then, the position of the cursor will be set to the same as the handle grip, which means that the cursor will move as the hand moves. This cursor is shown on both the embedded screen and the screen on the platform, which are designed to provide visual feedbacks to the users. The other screen on the side is used for physical therapists. In this way, users can play with any game that is operated by a mouse using this upper limb robot. The participants could choose a game between ice hockey and the cooking game. In the ice hockey, the participants play against the computer, whereas in the cooking game, the participants are required to follow the game tasks and complete a dish in a fixed time. Thus, the participants have to move randomly with human computer interaction. Completing the tasks not only requires participants’ physical movement but also cognitive reactions, such as paying attention, understanding the tasks, and solving the problems. Therefore, versatility of motion, versatility of feedback, and tasks are changing, and it is also combined with the feature of competitiveness in this exercise (Fig. 2). Tracking exercise
During the tracking exercise, a circle was shown on the screens. Then, the participant was required to grasp and move the handle grip to follow the circle precisely. As it was the same circle, versatility of motion was relatively regular in the tracking exercise. As for the versatility of feedback, the tracking exercise provided continuous feedback to the participants with the position of the handle grip. The distance between the handle grip and Fig. 2
the circle could make the participants aware of the error so as to adjust their movement to follow the circle more precisely, which required attention and adjustment as cognitive activities. With respect to the aspect of competitiveness, the participants were required to track the circle precisely (Fig. 3). Traditional exercise
In the traditional exercise, the participant was required to grasp objects with different shapes and masses and move into and out of the box, and repeat this regular movement. Different from the other two exercises, traditional exercise did not require the participants’ continuous attention. That is because the participants only needed to notice when they grasped and put down the object, but the path of movement was not required so that no attention was needed from the start position to the end position along the moving path. This means that the traditional exercise could only provide discontinuous feedback to the participants and attention as a cognitive task. There was no competiveness feature in this exercise (Fig. 4). The second objective of this study is to analyze the relationship between engagement level and muscle activities. During the experiment, EMG data on the arm were measured, which represented the muscle activities of the participants. Root mean square (RMS) of EMG, which represents the amplitude of EMG, was calculated and compared in different engagement levels to validate the following hypothesis: there is a relationship between the engagement level and EMG signals. For this analysis, muscle activities during the tracking exercise will be used for the reason that the movements in this exercise are fairly regular with a constant motion envelope. As there is no objective measurement to quantify engagement for each individual, a within-subject design was used in this experiment. All participants were required to complete three exercises with their right hands without stop, whereas the order of these three Fig. 3
Video game exercise.
Tracking exercise.
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Study of engagement and muscle activity Li et al. 337
performing the exercise with a normal facial expression. When the participants are engaged, it could be identified from their facial expression usually with smiles and they are more focused and concentrated all their attention on the exercises. The participants seem to be more careful with the exercise than they are in the normal level. In the bored level, the participants lose interest in the exercises, just repeat the tasks, and look around sometimes, usually with a dull expression on their faces. However, there are small periods that are difficult to judge. Therefore, we have introduced transition levels when the participants’ facial expressions are between two levels.
Fig. 4
Traditional exercise.
exercises was random for each participant. This is because engagement in different exercises could also be related to the sequence of the exercises. Each exercise lasted for 5 min. Participants
In this study, 15 healthy individuals, eight men and seven women, between 23 and 55 years of age (mean, 33.4 ± 8.8 years), were involved. We assume that the effects of influencing factors on engagement are similar between healthy individuals and stroke survivors, and there will be similar results with respect to the relationship between the level of engagement and the level of muscle activities between the two groups. Measurements and research data aggregation
As it is difficult to evaluate the participants’ engagement level, a web cam was utilized to capture the facial expression of the participants to evaluate the engagement level. That is because it is generally accepted that emotion can be analyzed by facial expressions (Ekman, 1993; Keltner et al., 2003); thus, facial expressions could also reflect the level of emotional engagement of the participants (Bardzell et al., 2008; Teixeira et al., 2012). The engagement level is identified according to different expressions of the participants shown on the video. Three levels of engagement could be identified in the video, which are engaged level, normal level, and bored level. The characteristics of each level are summarized in Table 2. In the normal level, the participants are Indicators for engagement on the basis of facial expressions
We used another web cam fixed above the platform to record the arm movements of the participants to characterize the motion by the motion envelope and motion velocity. A motion analysis software, Tracker (https: //www.cabrillo.edu/ ~ dbrown/tracker/), was used to analyze the motion from the videos. In addition, EMG signals were recorded from extensor carpi radialis longus, flexor carpi radialis, first dorsal interosseous muscle, and extensor digitorum to measure the level of activation of the muscles. EMG signals were sampled at 1000 Hz, and filtered by a band pass filter at 20–500 Hz and a band stop filter at 50 Hz. We introduced the normalized EMG signal, which is defined as the RMS of the filtered signal divided by the average velocity in a window of 0.33 s. With this normalization step, the effect of the movement velocity was removed from the signal, which would otherwise alter the EMG amplitude (Somasundaram, 1974; US Department of Health and Human Services, 1992). Finally, postevent questionnaires (Brockmyer et al., 2009) were used to indicate the engagement level. This questionnaire was originally intended to measure the engagement in violent video game playing. As a result, some items in the questionnaires were adopted or deleted. For example, questions such as ‘I feel scared’, ‘I get wound up’, ‘Things seem to happen automatically’, and ‘playing seems automatic’ were deleted from the questionnaire. Terms such as ‘play’ and ‘game’ were changed to ‘do the exercise’ and ‘exercise’, respectively. Each participant was required to fill in this same questionnaire after each exercise. There were three answers for each question: yes, maybe, or no. Different answers yield different scores (Brockmyer et al., 2009), and then the total score will be used to indicate the engagement level of the participants during the exercise. The questions of the adopted questionnaires are listed in Table 3.
Table 2
Engagement levels Engaged Normal Bored
Indicators Smile, focus on the exercise Normal face, looking at the exercise Dull, looking around, or looking at the exercise but not concentrating, holding the chin with hand
Analyses and results Analysis of facial expressions
According to the criteria of identification of the engagement level, a typical pattern of facial expression in each exercise is shown in Fig. 5. 1, 0.5, 0, − 0.5, and − 1 refer to engaged, transition between engaged and normal,
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
338 International Journal of Rehabilitation Research 2014, Vol 37 No 4
Engagement questionnaire items and scores
Items 1. I lose track of time 2. I feel different 3. The exercise feels real 4. If someone talks to me, I do not hear them 5. Time seem to kind of standstill or stop 6. I feel spaced out 7. I do not answer when someone talks to me 8. I cannot tell that I am getting tired 9. My thoughts go fast 10. I lose track of where I am 11. I do the exercise without thinking about how to do it 12. The exercise makes me feel calm 13. I do the exercise longer than I meant to 14. I really get into the exercise 15. I feel like I just cannot stop doing the exercise
Fig. 6
No
Maybe
Yes
− 2.82 − 0.82 −2 − 1.82 −1 − 1.16 − 1.32 − 1.5 − 2.16 0 −2
−1 0.82 − 0.32 0 0.66 0.5 0.32 0.16 − 0.5 1.66 − 0.5
0.82 2.5 1.32 1.82 2.32 2.16 2 1.82 1.16 3.32 1.16
−2 − 2.32 − 3.5 − 1.82
− 0.5 − 0.66 − 1.82 − 0.16
1.16 1 − 0.16 1.5
300
Engaged Transition Normal Transition Bored
250 Duration (s)
Table 3
200 150 100 50 0 Video game
Tracking
Traditional
Different exercise Durations of engagement level in different exercises. In the box, the central line is the median, the circle is the mean, and the edges of the box are the 25th and 75th percentiles.
Fig. 5
1
P = 2 × 10 − 5), between the video game exercise and the traditional exercise (46 ± 17/s; P = 3 × 10 − 9), and between the traditional exercise and the tracking exercise (P = 2 × 10 − 4). Therefore, we could conclude that the video game exercise leads to a longer period of engaged level during the training exercise than the other two exercises.
Engagement level
0.5
0
−0.5
−1 0
Game exercise Tracking exercise Traditional exercise 25 50 75 100 125 150 175 200 225 250 275 300 Time (s)
Typical pattern of engagement level on the basis of facial expression.
normal, transition between normal and bored, and bored, respectively. According to the typical pattern of facial expression in each exercise, the participants are interested in each exercise at first. As the exercise proceeds and they become familiar with the exercise, the level of engagement changes from engaged level to normal level, and then to bored level. In the video game exercise, engagement level may change from engaged level to transition and change back to engaged level from transition, which could be interpreted as a new stimulation in the video game, whereas in the other two exercise, there are no such stimulations. The mean and SD of time duration of each engagement level in different exercise are calculated and shown in Fig. 6. Post-hoc analysis of the durations in the engaged level showed significant differences between the video game exercise (175 ± 32/s) and the tracking exercise (85 ± 30/s;
Post-hoc analysis of the questionnaires also showed significant differences in the scores that indicated engagement between the video game exercise (3.98 ± 1.2) and the tracking exercise (− 2.64 ± 1.8; P = 0.01) and between the video game exercise and the traditional exercise (− 6.28 ± 2.1; P = 0.006). However, no significant differences were found in the scores between the tracking exercise and the traditional exercise (P = 0.11). These results could support the reliability of the engagement level on the basis of facial expressions. Therefore, in the fourth section, engagement level-based facial expressions are used to evaluate the engagement level during each exercise.
Analysis of motion characteristics in different exercises
In this part, the motion characteristics, such as elbow angle and hand velocity, will be analyzed and compared between different exercises (Figs 7 and 8). In the tracking exercise, the elbow makes a regular motion as the participant tracks the same circle. In the other two exercises, elbow angles change arbitrarily because of different tasks. This is the same for all participants. The average motion range of the elbow angle (rad) in each exercise is calculated: the video game exercise (2.12 ± 0.38–2.89 ± 0.46), the tracking exercise (1.51 ± 0.03– 2.86 ± 0.06), and the traditional exercise (1.90 ± 0.12–3.03 ± 0.07). The average hand velocity (236 ± 12 mm/s) in the traditional exercise is larger than that in the video game
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
π/2
Competitiveness
Elbow angle (rad)
π
Game Tracking Traditional 0 0
50
100
150
200
250
300
Time (s)
Cognitive tasks
Typical pattern of elbow angle changes in different exercises.
Fig. 8
500 450 Versatility of feedback
350 300 250 200
Factors influencing engagement
As is clear from the analysis above, the time spent in the engaged level in the video game exercise is longer than the other two exercises, and the differences are statistically significant. The differences may probably be because of the influence of the sub-factors identified. The elbow range in the tracking exercise is the largest, but the mean duration of the engagement level in the
77 ± 25 54 ± 14 236 ± 12
exercise (77 ± 25 mm/s) and the tracking exercise (54 ± 14 mm/s) for all participants. Post-hoc analysis of the average motion range and average hand velocity both showed significant differences in these three exercises (Table 4).
Hand velocity (mm/s)
Average hand velocity in different exercises for one participant.
Elbow range (rad)
Traditional
Random Regular Regular
Tracking exercise
Differences in the sub-factors for engagement
Video game
Table 4
0
Versatility of motion
50
Video game exercise Tracking exercise Traditional exercise
100
2.12 ± 0.38–2.89 ± 0.46 1.51 ± 0.03–2.86 ± 0.06 1.90 ± 0.12–3.03 ± 0.07
150
Engaging methods
Hand velocity (mm/s)
400
175 ± 32 85 ± 30 46 ± 17
Fig. 7
Interaction with video game Attention and problem solving Play against the computer Continuous visual feedback Attention and adjustment Track more precisely Discontinuous visual feedback Attention No
Mean duration in engaged level (/s)
Study of engagement and muscle activity Li et al. 339
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
340 International Journal of Rehabilitation Research 2014, Vol 37 No 4
Analysis of motion characteristics in different engagement levels
To understand the relationship between engagement and muscle activities, this section analyzes EMG data in different levels. In this part, muscle activities during the tracking exercise will be used for the reason that the movements in this exercise are fairly regular with a constant motion envelope. In the tracking exercise, the same movement is selected from each engagement level, which is completing one circle. Then, hand velocity and tracking accuracy in each level are calculated and analyzed (Figs 9 and 10). Post-hoc analysis of the average hand velocity showed significant differences between the engaged level (36 ± 7.9 mm/s) and the normal level (54 ± 6.3 mm/s; P = 0.01), between the normal level and the bored level (66 ± 11.4 mm/s; P = 0.008), and between the engaged level and the bored level (P = 0.0004). Post-hoc analysis of the average distance also showed significant differences between the engaged level (9 ± 2.3 mm) and the normal level (18 ± 4.4 mm; P = 0.006), between the normal level and the bored level (25 ± 4.9 mm; P = 0.01), and between the engaged level and the bored level (P = 0.0007). This maybe because when the participants
50
40
30
20
10
0
Engaged
Normal
Average distance from the handle grip to the circle in different levels for one participant.
were concentrating on the exercise, they tracked the circle more carefully with more accurate, but slower movements.
Analysis of electromyographic data
In this part, normalized RMS of EMG will be analyzed and compared in different engagement levels (Fig. 11). As there is relationship between EMG and the velocity of the movement, EMG data in each engagement level are normalized as in the following equation: RMS : Average velocity
According to the results of this participant, the average of normalized EMG in the engaged level is larger than the other two levels, which means that the muscle activity in the engaged level is more intense than the other two. Then, the mean normalized RMS in each engagement level of all participants is calculated (Fig. 12).
100
80
60
40
20
0
Bored
engagement level
Normalized EMG ¼
Fig. 9
Hand velocity (mm/s)
Fig. 10
Distance (mm)
tracking exercise is smaller than that in the video game exercise and larger than the traditional exercise. Therefore, we could infer that elbow range does not influence engagement as much as the other sub-factors. In the same way, versatility of motion and hand velocity could also be canceled from the main sub-factors. As a result, versatility of feedback, cognitive tasks, and competitiveness remain the main sub-factors that influence engagement.
Engaged
Normal
Bored
engagement level Average hand velocity in different levels for one participant.
Post-hoc analysis of the normalized RMS for the extensor carpi radialis longus showed significant differences between the engaged level and the normal level (P = 0.001), between the normal level and the bored level (P = 0.04), and between the engaged level and the bored level (P = 4 × 10 − 6). Similarly, significant differences were found in the normalized RMS of the flexor carpi radialis between the engaged level and the normal level (P = 0.002), between the normal level and the bored level (P = 0.0001), and between the engaged level and the bored level (P = 2 × 10 − 6). No significant differences were found in the normalized RMS of the first dorsal interosseous between the engaged level and the normal
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Study of engagement and muscle activity Li et al. 341
results of the first dorsal interosseous and extensor digitorum do not show this. This could be because of the fact that the movement of tracking the circle does not involve the recruitments of these two muscles, which means that there is little finger motion during the tracking exercise.
Fig. 11
Normalized RMS [mV/(mm/s)]
2
× 10−3
Participant 1
1.5
Discussions of the results and conclusions Influencing factors on engagement
1
0.5
0
Engaged
Normal
Bored
engagement level Root mean square (RMS) of electromyography in different engagement levels for one participant.
Fig. 12
Normalized RMS [mV/(mm/s)]
1.5
× 10−3 Engaged Normal Bored
1
0.5
0 Extensor cr
Flexor cr
First dorsal i
Extensor d
Muscles Mean of normalized root mean square (RMS) for the four muscles in different engagement levels. In the box, the central line is the median, the circle is the mean, and the edges of the box are the 25th and 75th percentiles. Extensor d, extensor digitorum; extensor cr, extensor carpi radialis longus; first dorsal i, first dorsal interosseous; flexor cr, flexor carpi radialis.
level, and between the normal level and the bored level, whereas it showed significant differences between the engaged level and the bored level (P =0.01). For the extensor digitorum, no significant differences were found between the three levels in the normalized RMS. Engagement and muscle activities
The results indicate that the muscle activities in the extensor carpi radialis longus and flexor carpi radialis are more intense when individuals are engaged. However, the
Our study has shown that versatility of feedback, cognitive tasks, and competiveness are the main sub-factors that contribute toward engagement. Specifically, with respect to the versatility of the feedback, the video game exercise can provide human computer interaction, thereby giving the participants a feeling of presence. During the tracking exercise, the screen provides continuous feedback, showing the distance between the circle on the screen and the handle grip. The traditional exercise only involves discontinuous feedbacks. As the traditional exercise engaged the participants the shortest, we can infer that engagement induced by discontinuous feedback is insufficient. The next main sub-factor involved cognitive tasks. We found that when the participants were playing against the computer, they had to understand the feedback and then respond. During the tracking exercise, as they are required to track the circle precisely, the participants should adjust their movements according to the distance between the target positions and the real positions. These two exercises require continuous attention and synchronizing motor coordination. The traditional exercise, in contrast, does not require a deeper understanding of the tasks, and the task itself can quickly become routine. This means that it cannot engage the participants to the awareness level, which leads to the shortest time in the engaged level as well. Moreover, social engagement that is expressed by the competitiveness sub-factor of the exercises also has a significant effect on the overall engagement of the participants. Both the video game and the tracking exercises contain more competitive elements compared with the traditional exercise. The nature of competitiveness could be seen as a challenge. Another interesting result is that the level of engagement always decreases as the exercise proceeds, which means that the participants are interested in exercises that are new to them. As they become more familiar with the exercise, they seem to lose interest, even in the video game exercise. However, the video game exercise and the tracking exercise can provide the participants with a certain level of challenge, which means something new to them. This is also the reason why these two exercises could engage the participants for a longer time than the traditional exercise, which cannot provide challenge. Therefore, to maintain a high level of engagement during rehabilitation exercises, new tasks should be provided when the exercise becomes too familiar to the participants. In addition, new tasks are supposed to fit the participants’ capabilities because either too easy or too hard
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
342 International Journal of Rehabilitation Research 2014, Vol 37 No 4
tasks cannot engage the participants. This indicates that system adaptiveness is required to deliver proper exercise to different stroke survivors or to the same stroke patients, but at different stages of recovery. Relationship between engagement and muscle activities
Normalized RMS in EMG data indicates that muscle activities are more intense when the participants are engaged than the other two levels. As intensity of muscle activity is important to stroke rehabilitation, engagement should be emphasized as one of the most important factors in the future rehabilitation. The relationship between engagement and muscle activities could enable evaluation of the engagement level during training exercise. As discussed above, the participants tend to be less engaged when they take the exercise as routine, without thinking. Therefore, the system could measure muscle activities of the participants to monitor the engagement level. If the engagement level decreases as the exercise proceeds, the system could provide a different exercise for the participants to maintain the high level of engagement. Conclusion
To analyze the influencing factors of engagement, three different exercises, the video game exercise, the tracking exercise, and the traditional exercise, which addressed different sub-factors that influence the level of engagement (Table 1), were applied in this experiment. According to the difference in the mean duration of time that the participants spent on each exercise, motor engaging factors, such as the versatility, velocity of motion, and motion envelope, were found not to influence engagement as much as other factors. However, perceptual, cognitive, and social engaging factors were identified to be the main factors that contribute toward engagement. Correspondingly, engaging methods, such as increasing the versatility of the feedbacks, involving cognitive tasks, and integrating competitive features during rehabilitation exercise, have the potential to engage the participants to a higher level. The analysis in the tracking exercise also shows interesting results: (a) when the participants are engaged, on the basis of the analysis of normalized EMG, muscle activities are more intense, which may lead to better rehabilitation outcomes for stroke survivors; (b) the motion velocity was inversely correlated to the level of engagement, which may because of the fact that when the participants were more engaged, they tracked the circle more carefully with more accurate but slower movements. Future research opportunities
The results have shown that high level of engagement may lead to more intense muscle activities, and that the participants are less interested in the exercises that are too familiar to them. Therefore, different exercises could be delivered to the participants with an adaptive system. Cyber physical systems have the potential to provide
different engaging exercise to the participants because of high adaptiveness, hybrid structures, automated problem solving, and situated learning. Cyber physical solutions for assisted technology rehabilitation could enable monitoring of the level of engagement of the participants and apply different proper training exercises to the participants to maintain the high level of engagement during rehabilitation exercises.
Acknowledgements Chong Li was supported by the China Scholarship Council. Conflicts of interest
There are no conflicts of interest.
References Bardzell J, Bardzell S, Pace T (2008). Emotion, engagement, and internet video. Emotion http://67.222.24.243/wp-content/uploads/2009/03/affect-studyscreen-view.pdf. Brockmyer JH, Fox CM, Curtiss KA, McBroom E, Burkhart KM, Pidruzny JN (2009). The development of the Game Engagement Questionnaire: a measure of engagement in video game-playing. J Exp Soc Psychol 45:624–634. Burke JW, Morrow PJ, McNeill MDJ, McDonough SM, Charles DK (2008). Vision based games for upper-limb stroke rehabilitation. Machine Vision and Image Processing Conference, 2008. IMVIP’08. Portrush: IEEE. pp. 159–164. Colombo R, Pisano F, Mazzone A, Delconte C, Micera S, Chiara (2007). Design strategies to improve patient motivation during robot-aided rehabilitation. J Neuroengineering Rehabil 4:3. Ekman P (1993). Facial expression and emotion. Am Psychol 48:384–392. Grant JS, Glandon GL, Elliott TR, Giger JN, Weaver M (2004). Caregiving problems and feelings experienced by family caregivers of stroke survivors the first month after discharge. Int J Rehabil Res 27:105–111. Holden MK (2005). Virtual environments for motor rehabilitation: review. Cyberpsychol Behav 8:187–211. Keltner D, Ekman P, Gonzaga GC, Beer J (2003). Facial expression of emotion. In: Handbook of affective sciences. Series in affective science. New York: Oxford University Press, xvii, 1999 pp. 415–432. Kortte KB, Falk LD, Castillo RC, Johnson-Greene D, Wegener ST (2007). The Hopkins Rehabilitation Engagement Rating Scale: development and psychometric properties. Arch Phys Med Rehabil 88:877–884. Langhorne P, Bernhardt J, Kwakkel G (2011). Stroke rehabilitation. Lancet 377:1693–1702. Lequerica AH, Kortte K (2010). Therapeutic engagement: a proposed model of engagement in medical rehabilitation. Am J Phys Med Rehabil 89:415–422. Li C, Rusák Z, Horváth I, Ji L, Hou Y (2014a). Current status of robotic stroke rehabilitation and opportunities for a cyber-physically assisted upper limb stroke rehabilitation. Proceedings of TMCE 2014; 19–23 May; Budapest, Hungary; 2014. Li C, Rusák Z, Hou Y, Young C, Ji L (2014b). Upper limb motor rehabilitation integrated with video games focusing on training the fine movement of the fingers. Int J Robot Autom (in press). Lynskey JV, Belanger A, Jung R (2008). Activity-dependent plasticity in spinal cord injury. J Rehabil Res Dev 45:229–240. Somasundaram M (1974). New developments in electromyography and clinical neurophysiology. Arch Neurol 30:340–340. Teixeira T, Wedel M, Pieters R (2012). Emotion-induced engagement in Internet video advertisements. J Marketing Res 49:144–159. Timmermans AA, Seelen HA, Willmann RD, Kingma H (2009). Technologyassisted training of arm-hand skills in stroke: concepts on reacquisition of motor control and therapist guidelines for rehabilitation technology design. J Neuroengineering Rehabil 6:1. Tupper A, Henley S (1987). Predictive factors in stroke outcome and implications for intervention. Int J Rehabil Res 10:119–121. US Department of Health and Human Services (1992). Selected topics in surface electromyography for use in the occupational setting: expert perspectives. Washington, D.C.: Centers for Disease Control (CDC). Zimmerli L, Jacky M, Lünenburger L, Riener R, Bolliger M (2013). Increasing patient engagement during virtual reality-based motor rehabilitation. Arch Phys Med Rehabil 94:1737–1746.
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Original article
Influence of complementing a robotic upper limb rehabilitation system with video games on the engagement of the participants: a study focusing on muscle activities Chong Lia,b, Zoltán Rusáka, Imre Horvátha and Linhong Jib Efficacious stroke rehabilitation depends not only on patients’ medical treatment but also on their motivation and engagement during rehabilitation exercises. Although traditional rehabilitation exercises are often mundane, technology-assisted upper-limb robotic training can provide engaging and task-oriented training in a natural environment. The factors that influence engagement, however, are not fully understood. This paper therefore studies the relationship between engagement and muscle activities as well as the influencing factors of engagement. To this end, an experiment was conducted using a robotic upper limb rehabilitation system with healthy individuals in three training exercises: (a) a traditional exercise, which is typically used for training the grasping function, (b) a tracking exercise, currently used in robot-assisted stroke patient rehabilitation for fine motor movement, and (c) a video game exercise, which is a proliferating approach of robot-assisted rehabilitation enabling high-level active engagement of stroke patients. These exercises differ not only in the characteristics of the motion that they use but also in their method of triggering engagement. To measure the level of engagement, we used facial expressions, motion analysis of the arm movements, and electromyography. The results show that (a) the video game exercise could engage the participants for a longer period than the other two
Introduction Motivation and engagement are important factors in rehabilitation and are used frequently as determinants of rehabilitation outcomes (Tupper and Henley, 1987; Grant et al., 2004; Holden, 2005; Colombo et al., 2007). A positive rehabilitation outcome is associated strongly with high patient motivation and engagement (Langhorne et al., 2011). Engagement in rehabilitation exercises has been defined as a construct that is driven by motivation and executed through active, effortful participation (Lequerica and Kortte, 2010). It extends the definition of participation beyond exercise attendance and motivation (Kortte et al., 2007). Research has shown that active participation during rehabilitation promotes cortical plasticity through cortical map reorganization (Lynskey et al., 2008) and that active participation level is influenced by the level of engagement (Lequerica and Kortte, 2010). Therefore, increased engagement during rehabilitation exercise is essential to good rehabilitation outcomes. 0342-5282 © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins
exercises, (b) the engagement level decreased when the participants became too familiar with the exercises, and (c) analysis of normalized root mean square in electromyographic data indicated that muscle activities were more intense when the participants are engaged. This study shows that several sub-factors on engagement, such as versatility of feedback, cognitive tasks, and competitiveness, may influence engagement more than the others. To maintain a high level of engagement, the rehabilitation system needs to be adaptive, providing different exercises to engage the participants. International Journal of Rehabilitation Research 37:334–342 © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. International Journal of Rehabilitation Research 2014, 37:334–342 Keywords: engagement level, exercise integrated with video game, influencing factors of engagement, muscle activity, upper limb rehabilitation a Faculty of Industrial Design Engineering, Delft University of Technology, Delft, The Netherlands and bDivision of Intelligent and Bio-mimetic Machinery, The State Key Laboratory of Tribology, Department of Mechanical Engineering, Tsinghua University, Beijing, China
Correspondence to Chong Li, Landbergstraat 15, 2628 CE, Delft, The Netherlands Tel: +31 (0)15 27 83629; fax: +31 (0)15 27 81839; e-mail: [email protected] Received 6 May 2014 Accepted 28 July 2014
Traditional rehabilitation tasks, however, are often mundane and can lead to a lack of patient motivation, resulting in little or no independent patient exercise taking place (Burke et al., 2008). Technology-assisted upper-limb training can provide engaging and taskoriented training in a natural environment using patient-tailored feedback to support learning of motor skills (Timmermans et al., 2009). However, there is little understanding of which factors influence engagement. Although it has been suggested that an individual’s willingness and capacity as well as social and physical aspects of the environment are the influencing factors of engagement in medical rehabilitation (Lequerica and Kortte, 2010), the causalities of these factors are not yet fully understood. According to our review (Li et al., 2014a), these factors can be categorized into five aspects: motor, perceptual, cognitive, emotional, and social factors. The first objective of this paper is to identify subfactors that contribute toward engagement and analyze their causalities. DOI: 10.1097/MRR.0000000000000076
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Study of engagement and muscle activity Li et al. 335
Table 1
Sub-factors of engagement
Engaging methods Motor
Perceptual Cognitive Emotional Social
Sub-factors
Video game exercise
Versatility of motion Motion envelope Velocity of motion Versatility of feedback Cognitive tasks
Random
Competiveness
Tracking exercise
Regular Analyzed by motion characteristics Analyzed by motion characteristics Interaction with video game Continuous feedback Attention and problem solving Attention and adjustment Not analyzed in this experiment Play against the computer Track more precisely
Another objective of our paper is to further study the relationship between engagement and muscle activities to investigate whether there is any reflection when the participants’ engagements vary. A recent attempt aimed at exploring the relationship of the level of engagement and muscle activity (Zimmerli et al., 2013). The authors argue that electromyography (EMG) can be used to measure the level of engagement as they found that increased level of engagement represents increased muscle activities. This study, however, neglected the effect of movement velocity on the muscle activities, which, according to other studies, also has a significant influence on the amplitude of measured EMG signals (Somasundaram, 1974; US Department of Health and Human Services, 1992). To realize these two objectives, an experiment was designed and conducted, which involved different training exercises, a video game exercise, a tracking exercise, and a traditional exercise. These three exercises involve different factors influencing the level of engagement. The Methods section explains how the experiment was conducted and how the measurements were acquired. The next section analyzes the causality of the influencing factors of engagement by comparing the results of engaging performance in each exercise, and processes the EMG data at different engagement levels to investigate the relationship between the two variables. The last section presents discussions of the results and the conclusion.
Traditional exercise Regular
Discontinuous feedback Attention No
As shown in Table 1, it lists the differences in the subfactors of engagement that each exercise could address. The motion envelope and velocity of motion will be analyzed using the motion characteristics. In the third section, the influence of these sub-factors on engagement will be analyzed on the basis of the performance of the participants in each exercise. Video game exercise
In the video game exercise, an upper limb rehabilitation robot (Fig. 1) (Li et al., 2014b) was used. There are passive and active modes in this robot system. In the passive mode, the users do not have to move while the robotic arms are driven by motors, whereas in the active mode, users are required to move the robotic arms voluntarily. In the video game exercise, the participant was required to complete the game tasks in the active mode. The active mode is operated by a human computer interface, which enables the user to interact with the game. The interface is located on the handle grip, which consists of several force sensors that sense the force exerted by the hand. When the force is larger than the threshold set in the system, it will generate a click of the left button of the system mouse. Similarly, when the force is smaller, it Fig. 1
Methods Experiment design
One objective of this study is to understand the causalities of influencing factors on engagement. As mentioned above, there are five aspects of influencing factors, namely, motor, perceptual, cognitive, emotional, and social. These factors could be further divided into subfactors. To test the influence of these factors on engagement level, this experiment was designed, in which a video game exercise, a tracking exercise, and a traditional exercise were used, as these exercises addressed different influencing factors. The sub-factors studied in this paper are versatility of motion, motion envelope, velocity of motion, versatility of feedback, cognitive tasks, and competitiveness.
Upper limb rehabilitation robot.
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
336 International Journal of Rehabilitation Research 2014, Vol 37 No 4
will generate a release, and when it is larger continuously, it will simulate a drag of the mouse. At the same time, when the hand moves, the position of the handle grip can be calculated by two encoders in two servo motors. Then, the position of the cursor will be set to the same as the handle grip, which means that the cursor will move as the hand moves. This cursor is shown on both the embedded screen and the screen on the platform, which are designed to provide visual feedbacks to the users. The other screen on the side is used for physical therapists. In this way, users can play with any game that is operated by a mouse using this upper limb robot. The participants could choose a game between ice hockey and the cooking game. In the ice hockey, the participants play against the computer, whereas in the cooking game, the participants are required to follow the game tasks and complete a dish in a fixed time. Thus, the participants have to move randomly with human computer interaction. Completing the tasks not only requires participants’ physical movement but also cognitive reactions, such as paying attention, understanding the tasks, and solving the problems. Therefore, versatility of motion, versatility of feedback, and tasks are changing, and it is also combined with the feature of competitiveness in this exercise (Fig. 2). Tracking exercise
During the tracking exercise, a circle was shown on the screens. Then, the participant was required to grasp and move the handle grip to follow the circle precisely. As it was the same circle, versatility of motion was relatively regular in the tracking exercise. As for the versatility of feedback, the tracking exercise provided continuous feedback to the participants with the position of the handle grip. The distance between the handle grip and Fig. 2
the circle could make the participants aware of the error so as to adjust their movement to follow the circle more precisely, which required attention and adjustment as cognitive activities. With respect to the aspect of competitiveness, the participants were required to track the circle precisely (Fig. 3). Traditional exercise
In the traditional exercise, the participant was required to grasp objects with different shapes and masses and move into and out of the box, and repeat this regular movement. Different from the other two exercises, traditional exercise did not require the participants’ continuous attention. That is because the participants only needed to notice when they grasped and put down the object, but the path of movement was not required so that no attention was needed from the start position to the end position along the moving path. This means that the traditional exercise could only provide discontinuous feedback to the participants and attention as a cognitive task. There was no competiveness feature in this exercise (Fig. 4). The second objective of this study is to analyze the relationship between engagement level and muscle activities. During the experiment, EMG data on the arm were measured, which represented the muscle activities of the participants. Root mean square (RMS) of EMG, which represents the amplitude of EMG, was calculated and compared in different engagement levels to validate the following hypothesis: there is a relationship between the engagement level and EMG signals. For this analysis, muscle activities during the tracking exercise will be used for the reason that the movements in this exercise are fairly regular with a constant motion envelope. As there is no objective measurement to quantify engagement for each individual, a within-subject design was used in this experiment. All participants were required to complete three exercises with their right hands without stop, whereas the order of these three Fig. 3
Video game exercise.
Tracking exercise.
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Study of engagement and muscle activity Li et al. 337
performing the exercise with a normal facial expression. When the participants are engaged, it could be identified from their facial expression usually with smiles and they are more focused and concentrated all their attention on the exercises. The participants seem to be more careful with the exercise than they are in the normal level. In the bored level, the participants lose interest in the exercises, just repeat the tasks, and look around sometimes, usually with a dull expression on their faces. However, there are small periods that are difficult to judge. Therefore, we have introduced transition levels when the participants’ facial expressions are between two levels.
Fig. 4
Traditional exercise.
exercises was random for each participant. This is because engagement in different exercises could also be related to the sequence of the exercises. Each exercise lasted for 5 min. Participants
In this study, 15 healthy individuals, eight men and seven women, between 23 and 55 years of age (mean, 33.4 ± 8.8 years), were involved. We assume that the effects of influencing factors on engagement are similar between healthy individuals and stroke survivors, and there will be similar results with respect to the relationship between the level of engagement and the level of muscle activities between the two groups. Measurements and research data aggregation
As it is difficult to evaluate the participants’ engagement level, a web cam was utilized to capture the facial expression of the participants to evaluate the engagement level. That is because it is generally accepted that emotion can be analyzed by facial expressions (Ekman, 1993; Keltner et al., 2003); thus, facial expressions could also reflect the level of emotional engagement of the participants (Bardzell et al., 2008; Teixeira et al., 2012). The engagement level is identified according to different expressions of the participants shown on the video. Three levels of engagement could be identified in the video, which are engaged level, normal level, and bored level. The characteristics of each level are summarized in Table 2. In the normal level, the participants are Indicators for engagement on the basis of facial expressions
We used another web cam fixed above the platform to record the arm movements of the participants to characterize the motion by the motion envelope and motion velocity. A motion analysis software, Tracker (https: //www.cabrillo.edu/ ~ dbrown/tracker/), was used to analyze the motion from the videos. In addition, EMG signals were recorded from extensor carpi radialis longus, flexor carpi radialis, first dorsal interosseous muscle, and extensor digitorum to measure the level of activation of the muscles. EMG signals were sampled at 1000 Hz, and filtered by a band pass filter at 20–500 Hz and a band stop filter at 50 Hz. We introduced the normalized EMG signal, which is defined as the RMS of the filtered signal divided by the average velocity in a window of 0.33 s. With this normalization step, the effect of the movement velocity was removed from the signal, which would otherwise alter the EMG amplitude (Somasundaram, 1974; US Department of Health and Human Services, 1992). Finally, postevent questionnaires (Brockmyer et al., 2009) were used to indicate the engagement level. This questionnaire was originally intended to measure the engagement in violent video game playing. As a result, some items in the questionnaires were adopted or deleted. For example, questions such as ‘I feel scared’, ‘I get wound up’, ‘Things seem to happen automatically’, and ‘playing seems automatic’ were deleted from the questionnaire. Terms such as ‘play’ and ‘game’ were changed to ‘do the exercise’ and ‘exercise’, respectively. Each participant was required to fill in this same questionnaire after each exercise. There were three answers for each question: yes, maybe, or no. Different answers yield different scores (Brockmyer et al., 2009), and then the total score will be used to indicate the engagement level of the participants during the exercise. The questions of the adopted questionnaires are listed in Table 3.
Table 2
Engagement levels Engaged Normal Bored
Indicators Smile, focus on the exercise Normal face, looking at the exercise Dull, looking around, or looking at the exercise but not concentrating, holding the chin with hand
Analyses and results Analysis of facial expressions
According to the criteria of identification of the engagement level, a typical pattern of facial expression in each exercise is shown in Fig. 5. 1, 0.5, 0, − 0.5, and − 1 refer to engaged, transition between engaged and normal,
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
338 International Journal of Rehabilitation Research 2014, Vol 37 No 4
Engagement questionnaire items and scores
Items 1. I lose track of time 2. I feel different 3. The exercise feels real 4. If someone talks to me, I do not hear them 5. Time seem to kind of standstill or stop 6. I feel spaced out 7. I do not answer when someone talks to me 8. I cannot tell that I am getting tired 9. My thoughts go fast 10. I lose track of where I am 11. I do the exercise without thinking about how to do it 12. The exercise makes me feel calm 13. I do the exercise longer than I meant to 14. I really get into the exercise 15. I feel like I just cannot stop doing the exercise
Fig. 6
No
Maybe
Yes
− 2.82 − 0.82 −2 − 1.82 −1 − 1.16 − 1.32 − 1.5 − 2.16 0 −2
−1 0.82 − 0.32 0 0.66 0.5 0.32 0.16 − 0.5 1.66 − 0.5
0.82 2.5 1.32 1.82 2.32 2.16 2 1.82 1.16 3.32 1.16
−2 − 2.32 − 3.5 − 1.82
− 0.5 − 0.66 − 1.82 − 0.16
1.16 1 − 0.16 1.5
300
Engaged Transition Normal Transition Bored
250 Duration (s)
Table 3
200 150 100 50 0 Video game
Tracking
Traditional
Different exercise Durations of engagement level in different exercises. In the box, the central line is the median, the circle is the mean, and the edges of the box are the 25th and 75th percentiles.
Fig. 5
1
P = 2 × 10 − 5), between the video game exercise and the traditional exercise (46 ± 17/s; P = 3 × 10 − 9), and between the traditional exercise and the tracking exercise (P = 2 × 10 − 4). Therefore, we could conclude that the video game exercise leads to a longer period of engaged level during the training exercise than the other two exercises.
Engagement level
0.5
0
−0.5
−1 0
Game exercise Tracking exercise Traditional exercise 25 50 75 100 125 150 175 200 225 250 275 300 Time (s)
Typical pattern of engagement level on the basis of facial expression.
normal, transition between normal and bored, and bored, respectively. According to the typical pattern of facial expression in each exercise, the participants are interested in each exercise at first. As the exercise proceeds and they become familiar with the exercise, the level of engagement changes from engaged level to normal level, and then to bored level. In the video game exercise, engagement level may change from engaged level to transition and change back to engaged level from transition, which could be interpreted as a new stimulation in the video game, whereas in the other two exercise, there are no such stimulations. The mean and SD of time duration of each engagement level in different exercise are calculated and shown in Fig. 6. Post-hoc analysis of the durations in the engaged level showed significant differences between the video game exercise (175 ± 32/s) and the tracking exercise (85 ± 30/s;
Post-hoc analysis of the questionnaires also showed significant differences in the scores that indicated engagement between the video game exercise (3.98 ± 1.2) and the tracking exercise (− 2.64 ± 1.8; P = 0.01) and between the video game exercise and the traditional exercise (− 6.28 ± 2.1; P = 0.006). However, no significant differences were found in the scores between the tracking exercise and the traditional exercise (P = 0.11). These results could support the reliability of the engagement level on the basis of facial expressions. Therefore, in the fourth section, engagement level-based facial expressions are used to evaluate the engagement level during each exercise.
Analysis of motion characteristics in different exercises
In this part, the motion characteristics, such as elbow angle and hand velocity, will be analyzed and compared between different exercises (Figs 7 and 8). In the tracking exercise, the elbow makes a regular motion as the participant tracks the same circle. In the other two exercises, elbow angles change arbitrarily because of different tasks. This is the same for all participants. The average motion range of the elbow angle (rad) in each exercise is calculated: the video game exercise (2.12 ± 0.38–2.89 ± 0.46), the tracking exercise (1.51 ± 0.03– 2.86 ± 0.06), and the traditional exercise (1.90 ± 0.12–3.03 ± 0.07). The average hand velocity (236 ± 12 mm/s) in the traditional exercise is larger than that in the video game
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
π/2
Competitiveness
Elbow angle (rad)
π
Game Tracking Traditional 0 0
50
100
150
200
250
300
Time (s)
Cognitive tasks
Typical pattern of elbow angle changes in different exercises.
Fig. 8
500 450 Versatility of feedback
350 300 250 200
Factors influencing engagement
As is clear from the analysis above, the time spent in the engaged level in the video game exercise is longer than the other two exercises, and the differences are statistically significant. The differences may probably be because of the influence of the sub-factors identified. The elbow range in the tracking exercise is the largest, but the mean duration of the engagement level in the
77 ± 25 54 ± 14 236 ± 12
exercise (77 ± 25 mm/s) and the tracking exercise (54 ± 14 mm/s) for all participants. Post-hoc analysis of the average motion range and average hand velocity both showed significant differences in these three exercises (Table 4).
Hand velocity (mm/s)
Average hand velocity in different exercises for one participant.
Elbow range (rad)
Traditional
Random Regular Regular
Tracking exercise
Differences in the sub-factors for engagement
Video game
Table 4
0
Versatility of motion
50
Video game exercise Tracking exercise Traditional exercise
100
2.12 ± 0.38–2.89 ± 0.46 1.51 ± 0.03–2.86 ± 0.06 1.90 ± 0.12–3.03 ± 0.07
150
Engaging methods
Hand velocity (mm/s)
400
175 ± 32 85 ± 30 46 ± 17
Fig. 7
Interaction with video game Attention and problem solving Play against the computer Continuous visual feedback Attention and adjustment Track more precisely Discontinuous visual feedback Attention No
Mean duration in engaged level (/s)
Study of engagement and muscle activity Li et al. 339
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
340 International Journal of Rehabilitation Research 2014, Vol 37 No 4
Analysis of motion characteristics in different engagement levels
To understand the relationship between engagement and muscle activities, this section analyzes EMG data in different levels. In this part, muscle activities during the tracking exercise will be used for the reason that the movements in this exercise are fairly regular with a constant motion envelope. In the tracking exercise, the same movement is selected from each engagement level, which is completing one circle. Then, hand velocity and tracking accuracy in each level are calculated and analyzed (Figs 9 and 10). Post-hoc analysis of the average hand velocity showed significant differences between the engaged level (36 ± 7.9 mm/s) and the normal level (54 ± 6.3 mm/s; P = 0.01), between the normal level and the bored level (66 ± 11.4 mm/s; P = 0.008), and between the engaged level and the bored level (P = 0.0004). Post-hoc analysis of the average distance also showed significant differences between the engaged level (9 ± 2.3 mm) and the normal level (18 ± 4.4 mm; P = 0.006), between the normal level and the bored level (25 ± 4.9 mm; P = 0.01), and between the engaged level and the bored level (P = 0.0007). This maybe because when the participants
50
40
30
20
10
0
Engaged
Normal
Average distance from the handle grip to the circle in different levels for one participant.
were concentrating on the exercise, they tracked the circle more carefully with more accurate, but slower movements.
Analysis of electromyographic data
In this part, normalized RMS of EMG will be analyzed and compared in different engagement levels (Fig. 11). As there is relationship between EMG and the velocity of the movement, EMG data in each engagement level are normalized as in the following equation: RMS : Average velocity
According to the results of this participant, the average of normalized EMG in the engaged level is larger than the other two levels, which means that the muscle activity in the engaged level is more intense than the other two. Then, the mean normalized RMS in each engagement level of all participants is calculated (Fig. 12).
100
80
60
40
20
0
Bored
engagement level
Normalized EMG ¼
Fig. 9
Hand velocity (mm/s)
Fig. 10
Distance (mm)
tracking exercise is smaller than that in the video game exercise and larger than the traditional exercise. Therefore, we could infer that elbow range does not influence engagement as much as the other sub-factors. In the same way, versatility of motion and hand velocity could also be canceled from the main sub-factors. As a result, versatility of feedback, cognitive tasks, and competitiveness remain the main sub-factors that influence engagement.
Engaged
Normal
Bored
engagement level Average hand velocity in different levels for one participant.
Post-hoc analysis of the normalized RMS for the extensor carpi radialis longus showed significant differences between the engaged level and the normal level (P = 0.001), between the normal level and the bored level (P = 0.04), and between the engaged level and the bored level (P = 4 × 10 − 6). Similarly, significant differences were found in the normalized RMS of the flexor carpi radialis between the engaged level and the normal level (P = 0.002), between the normal level and the bored level (P = 0.0001), and between the engaged level and the bored level (P = 2 × 10 − 6). No significant differences were found in the normalized RMS of the first dorsal interosseous between the engaged level and the normal
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Study of engagement and muscle activity Li et al. 341
results of the first dorsal interosseous and extensor digitorum do not show this. This could be because of the fact that the movement of tracking the circle does not involve the recruitments of these two muscles, which means that there is little finger motion during the tracking exercise.
Fig. 11
Normalized RMS [mV/(mm/s)]
2
× 10−3
Participant 1
1.5
Discussions of the results and conclusions Influencing factors on engagement
1
0.5
0
Engaged
Normal
Bored
engagement level Root mean square (RMS) of electromyography in different engagement levels for one participant.
Fig. 12
Normalized RMS [mV/(mm/s)]
1.5
× 10−3 Engaged Normal Bored
1
0.5
0 Extensor cr
Flexor cr
First dorsal i
Extensor d
Muscles Mean of normalized root mean square (RMS) for the four muscles in different engagement levels. In the box, the central line is the median, the circle is the mean, and the edges of the box are the 25th and 75th percentiles. Extensor d, extensor digitorum; extensor cr, extensor carpi radialis longus; first dorsal i, first dorsal interosseous; flexor cr, flexor carpi radialis.
level, and between the normal level and the bored level, whereas it showed significant differences between the engaged level and the bored level (P =0.01). For the extensor digitorum, no significant differences were found between the three levels in the normalized RMS. Engagement and muscle activities
The results indicate that the muscle activities in the extensor carpi radialis longus and flexor carpi radialis are more intense when individuals are engaged. However, the
Our study has shown that versatility of feedback, cognitive tasks, and competiveness are the main sub-factors that contribute toward engagement. Specifically, with respect to the versatility of the feedback, the video game exercise can provide human computer interaction, thereby giving the participants a feeling of presence. During the tracking exercise, the screen provides continuous feedback, showing the distance between the circle on the screen and the handle grip. The traditional exercise only involves discontinuous feedbacks. As the traditional exercise engaged the participants the shortest, we can infer that engagement induced by discontinuous feedback is insufficient. The next main sub-factor involved cognitive tasks. We found that when the participants were playing against the computer, they had to understand the feedback and then respond. During the tracking exercise, as they are required to track the circle precisely, the participants should adjust their movements according to the distance between the target positions and the real positions. These two exercises require continuous attention and synchronizing motor coordination. The traditional exercise, in contrast, does not require a deeper understanding of the tasks, and the task itself can quickly become routine. This means that it cannot engage the participants to the awareness level, which leads to the shortest time in the engaged level as well. Moreover, social engagement that is expressed by the competitiveness sub-factor of the exercises also has a significant effect on the overall engagement of the participants. Both the video game and the tracking exercises contain more competitive elements compared with the traditional exercise. The nature of competitiveness could be seen as a challenge. Another interesting result is that the level of engagement always decreases as the exercise proceeds, which means that the participants are interested in exercises that are new to them. As they become more familiar with the exercise, they seem to lose interest, even in the video game exercise. However, the video game exercise and the tracking exercise can provide the participants with a certain level of challenge, which means something new to them. This is also the reason why these two exercises could engage the participants for a longer time than the traditional exercise, which cannot provide challenge. Therefore, to maintain a high level of engagement during rehabilitation exercises, new tasks should be provided when the exercise becomes too familiar to the participants. In addition, new tasks are supposed to fit the participants’ capabilities because either too easy or too hard
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
342 International Journal of Rehabilitation Research 2014, Vol 37 No 4
tasks cannot engage the participants. This indicates that system adaptiveness is required to deliver proper exercise to different stroke survivors or to the same stroke patients, but at different stages of recovery. Relationship between engagement and muscle activities
Normalized RMS in EMG data indicates that muscle activities are more intense when the participants are engaged than the other two levels. As intensity of muscle activity is important to stroke rehabilitation, engagement should be emphasized as one of the most important factors in the future rehabilitation. The relationship between engagement and muscle activities could enable evaluation of the engagement level during training exercise. As discussed above, the participants tend to be less engaged when they take the exercise as routine, without thinking. Therefore, the system could measure muscle activities of the participants to monitor the engagement level. If the engagement level decreases as the exercise proceeds, the system could provide a different exercise for the participants to maintain the high level of engagement. Conclusion
To analyze the influencing factors of engagement, three different exercises, the video game exercise, the tracking exercise, and the traditional exercise, which addressed different sub-factors that influence the level of engagement (Table 1), were applied in this experiment. According to the difference in the mean duration of time that the participants spent on each exercise, motor engaging factors, such as the versatility, velocity of motion, and motion envelope, were found not to influence engagement as much as other factors. However, perceptual, cognitive, and social engaging factors were identified to be the main factors that contribute toward engagement. Correspondingly, engaging methods, such as increasing the versatility of the feedbacks, involving cognitive tasks, and integrating competitive features during rehabilitation exercise, have the potential to engage the participants to a higher level. The analysis in the tracking exercise also shows interesting results: (a) when the participants are engaged, on the basis of the analysis of normalized EMG, muscle activities are more intense, which may lead to better rehabilitation outcomes for stroke survivors; (b) the motion velocity was inversely correlated to the level of engagement, which may because of the fact that when the participants were more engaged, they tracked the circle more carefully with more accurate but slower movements. Future research opportunities
The results have shown that high level of engagement may lead to more intense muscle activities, and that the participants are less interested in the exercises that are too familiar to them. Therefore, different exercises could be delivered to the participants with an adaptive system. Cyber physical systems have the potential to provide
different engaging exercise to the participants because of high adaptiveness, hybrid structures, automated problem solving, and situated learning. Cyber physical solutions for assisted technology rehabilitation could enable monitoring of the level of engagement of the participants and apply different proper training exercises to the participants to maintain the high level of engagement during rehabilitation exercises.
Acknowledgements Chong Li was supported by the China Scholarship Council. Conflicts of interest
There are no conflicts of interest.
References Bardzell J, Bardzell S, Pace T (2008). Emotion, engagement, and internet video. Emotion http://67.222.24.243/wp-content/uploads/2009/03/affect-studyscreen-view.pdf. Brockmyer JH, Fox CM, Curtiss KA, McBroom E, Burkhart KM, Pidruzny JN (2009). The development of the Game Engagement Questionnaire: a measure of engagement in video game-playing. J Exp Soc Psychol 45:624–634. Burke JW, Morrow PJ, McNeill MDJ, McDonough SM, Charles DK (2008). Vision based games for upper-limb stroke rehabilitation. Machine Vision and Image Processing Conference, 2008. IMVIP’08. Portrush: IEEE. pp. 159–164. Colombo R, Pisano F, Mazzone A, Delconte C, Micera S, Chiara (2007). Design strategies to improve patient motivation during robot-aided rehabilitation. J Neuroengineering Rehabil 4:3. Ekman P (1993). Facial expression and emotion. Am Psychol 48:384–392. Grant JS, Glandon GL, Elliott TR, Giger JN, Weaver M (2004). Caregiving problems and feelings experienced by family caregivers of stroke survivors the first month after discharge. Int J Rehabil Res 27:105–111. Holden MK (2005). Virtual environments for motor rehabilitation: review. Cyberpsychol Behav 8:187–211. Keltner D, Ekman P, Gonzaga GC, Beer J (2003). Facial expression of emotion. In: Handbook of affective sciences. Series in affective science. New York: Oxford University Press, xvii, 1999 pp. 415–432. Kortte KB, Falk LD, Castillo RC, Johnson-Greene D, Wegener ST (2007). The Hopkins Rehabilitation Engagement Rating Scale: development and psychometric properties. Arch Phys Med Rehabil 88:877–884. Langhorne P, Bernhardt J, Kwakkel G (2011). Stroke rehabilitation. Lancet 377:1693–1702. Lequerica AH, Kortte K (2010). Therapeutic engagement: a proposed model of engagement in medical rehabilitation. Am J Phys Med Rehabil 89:415–422. Li C, Rusák Z, Horváth I, Ji L, Hou Y (2014a). Current status of robotic stroke rehabilitation and opportunities for a cyber-physically assisted upper limb stroke rehabilitation. Proceedings of TMCE 2014; 19–23 May; Budapest, Hungary; 2014. Li C, Rusák Z, Hou Y, Young C, Ji L (2014b). Upper limb motor rehabilitation integrated with video games focusing on training the fine movement of the fingers. Int J Robot Autom (in press). Lynskey JV, Belanger A, Jung R (2008). Activity-dependent plasticity in spinal cord injury. J Rehabil Res Dev 45:229–240. Somasundaram M (1974). New developments in electromyography and clinical neurophysiology. Arch Neurol 30:340–340. Teixeira T, Wedel M, Pieters R (2012). Emotion-induced engagement in Internet video advertisements. J Marketing Res 49:144–159. Timmermans AA, Seelen HA, Willmann RD, Kingma H (2009). Technologyassisted training of arm-hand skills in stroke: concepts on reacquisition of motor control and therapist guidelines for rehabilitation technology design. J Neuroengineering Rehabil 6:1. Tupper A, Henley S (1987). Predictive factors in stroke outcome and implications for intervention. Int J Rehabil Res 10:119–121. US Department of Health and Human Services (1992). Selected topics in surface electromyography for use in the occupational setting: expert perspectives. Washington, D.C.: Centers for Disease Control (CDC). Zimmerli L, Jacky M, Lünenburger L, Riener R, Bolliger M (2013). Increasing patient engagement during virtual reality-based motor rehabilitation. Arch Phys Med Rehabil 94:1737–1746.
Copyright © Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
