This article was downloaded by: [Columbia University] On: 01 March 2015, At: 21:35 Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Ergonomics Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/terg20
The influence of combined visual and tactile information on finger and eye movements during shape tracing MOTOYUKI AKAMATSU
a
a
Industrial Products Research Institute , 1-1-4, Tsukuba Science City, Ibaraki, 305, Japan Published online: 31 May 2007.
To cite this article: MOTOYUKI AKAMATSU (1992) The influence of combined visual and tactile information on finger and eye movements during shape tracing, Ergonomics, 35:5-6, 647-660, DOI: 10.1080/00140139208967844 To link to this article: http://dx.doi.org/10.1080/00140139208967844
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http:// www.tandfonline.com/page/terms-and-conditions
ERGONOMICS,
1992, VOL 35,
NOS
5/6, 647-660
The influence of combined visual and tactile information on finger and eye movements during shape tracing MOTOYUKI AKAMATSU
Industrial Products Research Institute, 1-1-4, Tsukuba Science City, 305, Ibaraki, Japan
Downloaded by [Columbia University] at 21:35 01 March 2015
Keywords: Vision;
Tactile sensation; Finger movement; Eye movement; Visuo-tactile integration.
Adaptation experiments in shape tracing were conducted to investigate finger and eye movements in various conditions of visual and tactile information. Maximum velocity, mean velocity, maximum acceleration and reacceleration point were calculated from finger movements. Number of eye fixations and lead time of eye fixation to finger position were calculated from eye movements. The results showed that for the finger movement the values of the indices studied were higher in the combined visual and tactile condition than in the visual only condition. The number of eye fixations decreased when subjects repeated the tracing and was more marked in the combined visual and tactile condition than in the visual only condition. The results suggest that finger movements become faster and use of vision is reduced when both visual and tactile information are given.
1. Introduction Information processing in humans is generally separated into sensory functions, central processing and motor functions. Sensory functions can then be divided into modalities, e.g. vision, audition and touch. However, in daily life, we obtain various modalities of sensory information simultaneously. Perception of extrapersonal space depends' on the integration of this information. Sensory function and motor behaviour do not work separately but rather interactively and integratively. In the interface between man and the extrapersonal space, as mentioned above, multimodal sensory information is used and operations with many degrees of freedom are performed. In contrast, at the man-computer interface, information is presented using only a single modality ofsensory information, i.e. visual information on a Visual Display Terminal, and the number of degrees of freedom for operation is limited. However, the number of degrees of freedom for operation in computers is increasing from one-dimensional key presses on the keyboard to two-dimensional movements using an input device such as a mouse. Furthermore, recent development of the glove type input device makes it possible to input data to the computer using various finger movements. In addition to this development of input devices, development of a display device using multimodal input can be expected to create a more natural and friendlier interface. Some systems of display of multirnodal display have been developed which give auditory information in addition to visual information. Auditory signals in such systems inform us about having touched an object. However, auditory signals are not a natural mode of sensation for touching. To overcome this problem, several systems which give tactile and force sensation are under development. However, the effect of multimodal input with visual and tactile sensory information on the efficiency of our behaviour has not been studied in detail. Thus, the aim of this study is to investigate the effects of combined visual and tactile sensory input on movements. 0014-0139/92 $3·00 01992 Taylor & Francis Ltd.
Downloaded by [Columbia University] at 21:35 01 March 2015
648
M. Akamatsu
In studies of hand-eye coordination, the relation between visual information and movements has been studied. The well-known study of Fitts proposed the law of difficulty in movement (Fitts 1954). Recent studies on the patterns of finger movement have been made by several authors (Carlton 1980, 1981, Schmidt et al. 1979, Crossman and Goodeve 1983, Prablanc et al. 1979a, I 979b, and Jeannerod 1986). Fischer and Rogal investigated the relation between reaching movements and eye movements by analysing reaction times (Fischer and Rogal 1986, Rogal and Fischer 1986). I n these studies, only visual information was under consideration as a source of sensory information for movement. Studies of the relation between multimodal sensation and movement have mainly focused on the effects of active movements on sensory functions (Held and Gottlieb 1958, Singer and Day 1966, Mather and Lackner 1980). It has been shown that the amount of adaptation to a deformed relationship between visual and tactile shape information was larger in active touch than in passive touch (Akamatsu 1990). In this paper, finger and eye movements during shape tracing with visual and tactile information have been investigated. Finger and eye movements were measured under conditions of visual information only and with combined visual and tactile information. 2. Method Adaptation experiments were performed in which the subjects adapted to a deformation of the relationship between visual and tactile shape information. In the experiment, subjects were asked to repeatedly trace the edges of a shape whilst finger and eye movements were measured to observe the characteristics of movement during and after adaptation. 2.1. Stimulus Twenty-eight rectangles were used as test shapes for tracing. The distance of an edge of these rectangles was in the range of 60 mm to 150 mm. The range of the length/width ratio was 1/5 to 5/1. An outline ofthe rectangle was presented on a CRT display as the visual shape stimulus. Each tactile shape stimulus was presented as an acrylic rectangular board which was 4 mm high. Subjects obtained tactile shape information by tracing the edges of the acrylic rectangles with the tip of the right index finger. The visual stimulus was presented on a surface mirror in front of the subject. This mirror was inclined at 45 0 so as to reflect the image of a CRT display placed to the left of the subject (figure I). The head of the subject was supported and held by a chin and forehead support. The distance from the eyes to the virtual image of the surface of the visual display was SO em. The cen tre of the visual display and the eyes were in the same horizontal plane. The acrylic rectangle for the tactile stimulus was positioned horizontally at the height of the elbow. The distance between the centre of the tactile stimulus and the body of the subject was 35 em. In order to measure finger movement during shape tracing a Position Sensitive Detector (PSD) System (Sel Spot System, SEL SPOT Corp.) was used. The target LED was attached to the dorsal side of the distal phalanx of the index finger and the position of this target was detected by the PSD camera which was situated 75 cm above the centre of the tactile stimulus. In order to measure eye movements during shape tracing, an Eye Position Detective System (Eye Mark Corder Mark IV, NAC Corp.) was used. The relation between visual shape and tactile shape was deformed in the length/width ratio by a micro-computer system. The ratio of deformation was 1·22 (i.e. enlarged lengthwise by 11/9) or 0·82 (i.e. enlarged in width by 9/11), and was
TJzc inflrrcnce q f ~ i s u aand l ractile information an shape rracing
649
Downloaded by [Columbia University] at 21:35 01 March 2015
constant during an experimental session. Subjects obtained visual feedback of the position of the distal phalanx of the index as an open circle, diameter 10 mm, on the xisual display. The visualized finger position was deformed identically with the \isu:tl shape. Therefore. discrepancy was not within each sensory modality but between sensory modalities. Subjects did not receive prior information about the deformation and, thus, they adapted to the discrepancy between visual and tactile information according to the visual and somatosensory information they obtained.
Figure I .
Experimental arrangement.
2.2. Slitn1(1li~condirions Experiments were performed in a dark room and visual information was obtained only from the CRT display. The condition with combined visual and tactile information was compared with the condition with visual information only. These two conditions were as follows:
Condirion I : The subject traced edges of the acrylic rectangle with his finger whilst a visualized outline ofthe rectangle was on the visual display. The position of the distal phalanx of the index finger was presented on the visual display and overlaid on the image of the rectangle. In this condition, the subject could use both visual and tactile information (combined visual and tactile condition). Corldirion 2: The subject drew a rectangle with his finger tip using the overlaid visualized finger position and an outline of the test rectangle. He moved his finger on a horizontally positioned board which was the same height as the rectangles. In this condition. the subject could only use visual information (visual only condition). 2.3. E-rprrimcntai procedure The subject was ~nstructedto trace the outline of each rectangle three times in a clockwise direction and as fast as possible. The index finger was brought by the experimenter to the starting point for tracing, which was the lower right corner. Then, an auditory pre-cue for the tracing was presented. Two seconds after the precue signal, a visual shape was presented on the screen a s the 'go' signal, and the subject started to trace. After the subject finished tracing, he took his finger off the
650
M. Akamatsu
Downloaded by [Columbia University] at 21:35 01 March 2015
rectangle. A sample of 45 rectangles were selected pseudo-randomly from the 28 test rectangles and were presented one after another. Two different deformations of }.22 and 0·82 were examined for both condition I and 2. Subjects were 22- to 32-year-old adult males and females. All of them were right-handed. Four subjects were studied in each condition.
2.4. Analysis 2.4.1. Analysis of the finger movements: The position of the distal phalanx of the index finger as measured by the PSD camera was separated into ventral-dorsal direction and right-left direction, and was sampled by 125 Hz. Data were taken from the part of the tracing movement which corresponded to the movement right to left from the start point to the corner (i.e. tracing length) and, then, from the corner away from the body (i.e. tracing width). A fourth-order Butterworth digital filter, cutoff frequency 2·5 Hz, was applied to the position data in order to reduce high frequency noise. Velocity was calculated by time differentiation of the position data. As an index at velocity, maximum velocity (figure 2, the downward-pointing, filled triangle) and mean velocity were used. The acceleration was obtained by time differentiation of velocity data. The acceleration pattern of the finger movement showed that the early stage was an acceleration phase and the later stage was mainly a deceleration phase. As an index of the acceleration phase, maximum acceleration was used (figure 2 a, the downward-pointing, open triangle). In the deceleration phase, several acceleration peaks and deceleration peaks were observed. This pattern of deceleration and acceleration suggests that the finger movement was controlled by visual and tactile feedback so as to reach the comer of the ractangle (Carlton 1980, Jeannerod 1986). The time of occurrence of the first deceleration peak of this phase was considered to be an index of the time of change from open loop (feedback) control to closed loop control (figure 2 a, the upward-pointing, open triangle). This time has been called reacceleration. In this paper, the relative position of the reacceleration along the edge was called the reacceleration point, and was obtained as another index from the acceleration pattern. 2.4.2. Analysis of eye movements: Data for eye posrnon were divided into an upward-downward component and a right-left component. These components were digitized at 125 Hz. Eye fixation was regarded as being when the eyes stayed in the same position for more than 100 ms. Eye movements were studied when the finger was near a comer and the number of eye fixations were counted when the finger was at a distance from a corner corresponding to 20 per cent of an edge of the rectangle (figure 2 b). The time difference between the beginning of eye fixation and when the finger came to this point indicated how early the eyes were fixed before the finger reached this point. This difference in time was called lead time (figure 2 C, t). Finger and eye movements were studied both during an adaptation process and during a period of stabilized performance. The adaptation process was followed by taking data from the observation zone for only the first of the three tracing movements and expressing them as the mean of each index for every five consecutive rectangles. Visual examination of results of the adaptation process indicated that the evolution of all the indices studied became stable after presentation of ten rectangles. Performance during the stabilized period was studied by the indices obtained in the observation zone for first, second and third tracing of rectangles II to 45. In addition, data were studied by calculating the means of data obtained from edges of
The influence of visual and tactile information on shape tracing
-11 ·.m~/f3: j\_-
1.0----reaccelc:ration point
:
It
jlOcm
"I
velocity
~
65 t
!Iac.v. 2
/\ \ A r0.5m1s wrl=bOn~
Downloaded by [Columbia University] at 21:35 01 March 2015
.t:.
O.5sec.
time
Figure 2. An example of measured finger and eye movement. (a) Displacement, velocity and acceleration of finger movement. '9': maximum velocity, 9: maximum acceleration. 6: reacceleration. (b) Displacement of eye and finger movement. Number of eye fixations was counted when finger was within 20 per cent of the distance of the edge from the corner. (e) 't' denotes the lead time of eye fixation to finger movement.
J 00 mm only so as to overcome the effects of distance of finger movement (Fitts 1954). Edges of 100 rom occurred most frequently in the test rectangles. Data were pooled for the two degrees of deformation.
3. Results 3.I. Finger movements Maximum velocity was taken as an index of speed of movement and its adaptation during the experiment is shown in figure 3 a. There was no apparent trend in the change of maximum velocity in the combined visual and tactile condition. However, there was a trend for an increase in maximum velocity in the visual only condition. The maximum velocity reached a steady state after the eleventh rectangle. This indicated that it took several trials until the maximum velocity became stable when visual information only was presented. To compare the maximum velocity in a steady state, maximum velocities on and after the eleventh rectangle were averaged (figure 3 b). The asterisks denote the statistical significances. The asterisks above the point denote that there was a significant difference between the combined visual and tactile condition and the visual only condition at this round of the repetition. The asterisks on the right-hand side of the points denote that there was a significant difference for the three tracings. In the visual only condition, the averages of maximum velocity were around 2 I cm/s, and there was no difference of maximum velocity for the three tracings. In the combined visual and tactile condition, the averages of maximum velocity were around 25 emls which were significantly faster than those in the visual only condition for each tracing. With repetition, the maximum velocities increased significantly. This indicated that the maximum velocity increased when a rectangle was traced repeatedly, if both visual and tactile information were available.
M. Akamatsu
652
-.
28
~ g
24
---.-
a:visual and tactile {n-40}
--0-
b:vlsuaJ {n.40}
a
]
1st tracing
MEAN:tSE
"i)
>
§
20
.§ ~
E
16
1 I
Downloaded by [Columbia University] at 21:35 01 March 2015
5
6
,
11 16 21 26 31 36 41
I I 1 I I f 10 15 20 25 30 35 40 45 I
stimulusorder
------
--0--
~
.3 ~
s
25
u
>
e:::s
e:vlsuaJand taalle (n.123) b:visual (n.130)
...
~
b
...
edge length: 1Oem MEAN%SE
.5
~
20
1st
2nd tracing
3rd
Figure 3. Comparison between maximum velocity of finger movements in the combined visual and tactile condition and that in the visual only condition. (a) Changes of maximum velocity of the first tracing during the experiment. (b) Comparison ofthe three tracings on and after the eleventh rectangle. Asterisks denote statistical significance levels; *:5%, ··:1%, ···0-5%.
The tendency for mean velocity during the adaptation period was similar to that of maximum velocity (figure 4 a). Averages on and after the eleventh rectangle were 7 to 8 cm/s in the visual only condition and 9 to 10 cmls in the combined visual and tactile; condition (figure 4 b). There were significant differences in each round of repetition. In both conditions; the mean velocities increased significantly. There was no noticeable change in maximum acceleration during the experiment for the combined visual and tactile condition (figure 5 a). In the visual only condition, the maximum acceleration increased at the beginning of the experiment, as did maximum and mean velocity. In the combined visual and tactile condition, averages in a steady state were about 0·9 cm/s which were significantly higher than these in the visual only condition (figure 5 b). There was no significant difference for maximum acceleration between the three tracing in either condition I or 2. This indicated that the maximum acceleration did not change if the rectangle was traced repeatedly.
The influence of visual and tactile information on shape tracing
----.~
l
a: visual and tactile (n-40) b: visual (n-40)
653
a
10
'-"
lst tracing
~
'u 0
MEAN±SE
'0 > ~ II}
E 5
Downloaded by [Columbia University] at 21:35 01 March 2015
1
I 5
11 16 21 26 31 36 41 I I I I I I I J 10 15 20 25 30 35 40 45 6
stimulus order
----.-
-
~
0
•••
10
"g 0
> ~ 4.)
b
12
~
~
e:visual and tactile (n-123) b:vlsual (n-130)
8
E 6
•••
•••
.~ .... ~ 15t
2nd
edge length: lOan MEAN±SE
3rd
tracing Figure 4. Comparison between mean velocity of finger movements in the combined visual and tactile condition and that in the visual only condition (a) and (b): as for figure 3.
The change of the reacceleration point was not observed in the combined visual and tactile condition (figure 6 a). However, changes were observed in the visual only condition. At the beginning of the experiment, reacceleration points were further from the corner of the edge than during the steady state. When both visual and tactile information were given, averages of the reacceleration points on and after the eleventh rectangle in the first tracing were about 0·91, and those in the second and third were about 0·95 (figure 6 b). There were significant differences for these three tracings. This indicated that the rcacceleration point came closer to the comer of the edge by repetition of the tracing. There were no significant differences among repetition in the visual only condition. There was no difference in the reacceleration point between the two conditions in the first tracing, however, the reacceleration point was closer to the comer in the combined visual and tactile condition than in the visual only condition with the second and third tracings. In the second tracing, there was a significant difference of the reacceleration point between the two conditions.
3.2. Eye movements The number of eye fixations near the end of the edge was an index of the degree to
M. Akamatsu
654 --..-
avlsual and taCllle (n.40)
---0--
b:vlsual (n-40)
a
1.0 1st tnlCing
MEAN±SE
0.5
Downloaded by [Columbia University] at 21:35 01 March 2015
1
I 5
11 16 21 26 31 36 41 I I I I I ·1 I 10 15 20 25 30 35 40 45
6 I
stimulus order
---~
1.0
••• 0.9
t
a:vlsual and tactile (n.123)
b
b:visual (n.130)
•••
r
•••
-1
MEAN±SE
0.8
~
0.7 0.6
edge length: 1Oem
1s t
2nd tracing
3rd
Figure 5. Comparison between maximum acceleration of finger movements in the combined visual and tactile condition and that in the visual only condition. (0) and (b) as for figure 3.
which vision was used for positioning the finger at the corner of a rectangle. The number of eye fixations during the experiment is shown in figure 7 a.There was no noticeable tendency in the combined visual and tactile condition although there was some fluctuation in values. In the visual only condition, the number of the eye fixations was about 1·7 on average at the beginning of the experiment. This average decreased to the same level as the number of fixations in the combined visual and tactile condition as the .experirnent proceeded. Averages on and after the eleventh rectangle are shown in figure 7 b. In both conditions, a decrease in the number ofeye fixations on repetition of tracing was observed but the decrease was significant only in the combined visual and tactile condition. The number of eye fixations in the second and third tracing in the combined visual and tactile condition was lower than in the visual only condition. In the second tracing, the difference between the two conditions was significant. These results indicated that the use of vision decreased on repetition of the tracing when visual and tactile information were available. There was a tendency for the lead time of the eye fixation to finger position to decrease in the visual only condition (figure 8 a). When only visual information was available, the eye movements initially led the finger movement by 400 ms. Then, this
The influence of visual and tactile information on shape tracing
655
a Ist tracing
Downloaded by [Columbia University] at 21:35 01 March 2015
MEAN:tSE
b edge length: 1Oem MEAN±SE
Figure 6. Comparison between reacceleration point of finger movements in the combined visual and tactile condition and that in the visual only condition (a) and (b) as for figure 3.
lead time decreased to less than 300 ms after the eleventh rectangle. In the steady state, the average of the lead time of the first tracing was about 290 ms in the combined visual and tactile condition and was about 325 ms in the visual only condition, though, there was no significant difference (figure 8 b). In both conditions, the lead time tended to be shorter in the second and third tracings. However, statistical significance was only observed in the visual only condition. This indicated that repetition of tracing with visual information reduced the lead time and, in condition 2, the lead time became close to that of condition 1.
4. Discussion The effects of combined visual and tactile information on finger and eye movements have been investigated by adaptation experiments with a deformed relationship between visual and tactile information. As experimental conditions, a visual only condition and a combined visual and tactile condition were used. In the visual only condition, the subject may integrate information from vision, sensations from muscle and joint receptors and efferent copy of the motor command. In the
M. Akamatsu
656
-..-
a: visual and tactile (n.40)
---0--
b: visual (n.40)
a
2.0 e
.2 ~
x t.:::
lst tracing
()
>,. 0
'-
1.5
MEAN±SE
c
....
.8 E c='
1.0
, I
Downloaded by [Columbia University] at 21:35 01 March 2015
5
11 16 21 26 31 36 41 1 I I I' I I I I 10 15 20 25 30 35 40 45
6
stimulus order
------0--
1.5
a:visuaJand lactlla (n.131)
b
bvtsuat (n.130)
.gc:: l"l:l lo(
l:
edge length: lOcm
0
>,. IU
'-
MEAN:t:$E
1.0
't.
0
~
E ::l
t::
0.5
Figure 7.
1s t
2nd tracing
3rd
Comparison between the number of eye fixations in the combined visual and tactile condition and that in the visual only condition. (a) and (b) as for figure 3.
combined visual and tactile condition, the subject may integrate information from vision, cutaneous sensation, sensation from muscle and joint receptors and efferent copy. For the purpose of investigating the adaptation process in each condition, the pattern of change of various indices was followed during the experiment. To investigate movement after adaptation, comparisons were made by averaging data on and after the eleventh rectangle. The relation between movement distance and velocity of the positioning movement have been studied by Fitts and others (Fitts 1954). Although the tracing movement of an edge was not the same as that which Fitts and others examined, the distance of movement may have had an influence on the velocity and the acceleration of the tracing movement we studied. Therefore, only the results from edges which had the same length were analysed. The effects of the length of edge on the indices of movement has not been examined in this study because oflack of data. The finger movement of the first tracing of a rectangle with the combined visual and tactile information had a higher maximum velocity and maximum acceleration than-that with visual information only. This implies that when tactile information is
The influence of visual and tactile information on shape tracing -+--
a:visual and tactile (n-44-29)
----0--
b:visual (n-37·26)
657
a
400 Ist tracing
MEAN±SE
300
200 1 I
Downloaded by [Columbia University] at 21:35 01 March 2015
5
11 16 21 26 31 36 41 I I I I I I I 10 15 20 25 30 35 40 45 6 I
stimulus order
-+--
a:visual and tactile{ 1st:n-13S,2nd:n-117.3rd:n c97)
---0--
bvlsuat (1stn..139.2nd:n.141.3rd:n.117)
b
350 ...-. u cu
.§ "'0
~
,.
MEAN±SE
250 200
1st
2nd
3rd
tracing Figure 8. Comparison between lead time of eye fixation to finger movement in the combined visual and tactile condition and that in the visual only condition. (a) and (b) as for figure 3.
added to visual information, we can perform rapid tracing movements by integrating visual and tactile information. Up to the reacceleration point, the movement can be considered to be executed under open loop control. If it is assumed that maximum acceleration indicates the amount of output from the control system, the results of maximum acceleration suggest that the gain under open loop control is greater when both visual and tactile information are obtained. The mean velocity in the combined visual and tactile condition was also higher than that in the visual only condition. According to Fitts's law, there is a relation between difficulty of positioning and movement time, i.e. an inverse of the mean velocity. This suggests that the positioning of the finger at the corner of the edge was less difficult with both visual and tactile information than with visual information alone. . The reacceleration point was considered to be an index of the change point from open loop control to closed loop control. However, the point of reacceleration may not be a good index of this change point when there is an important difference in the velocity. If there was a difference in the velocity of the open control movement, the change point from open loop to close loop control should not be at the same point even though the reacceleration points were the same. If the movement in the open
Downloaded by [Columbia University] at 21:35 01 March 2015
658
M. Akamatsu
loop control phase was very fast, the reacceleration point would be far from the corner because it will take time to reduce the velocity enough to change the control of the finger by sensory feedback. In our results, there was no noticeable difference in reacceleration point for the two conditions in the first tracing. However, as the mean velocity in the visual and tactile condition was about 1·3 times faster than that in the visual only condition, the change point from open loop control to closed loop control was considered to be closer to the corner when both visual and tactile information were given. From this discussion, we can deduce that the finger movement was faster when both visual and tactile information were available than when only visual information was available because of the high gain and the high proportion of open loop control for the movement. Comparisons between the first, second and third tracing of the rectangle showed that maximum velocity and mean velocity were increased at the second and third tracing when both visual and tactile information were given. As no increase was observed in maximum acceleration, the increase in velocity was not by the increase in gain of open loop control but rather by the increase in the proportion of open loop control. This was confirmed by the change in position of the reacceleration point, i.e. nearer to the corner. The movement by open loop control is, in other words, a preprogrammed movement. The effects of repetition of tracing suggested that the preprogrammed movement was enhanced by the stored somatosensory information obtained during the first tracing. As clear enhancement was not observed in the visual only condition, such enhancement must be the effect of a combined presentation of visual and tactile information or combined tactile information and information from muscles and joints. An increase in mean velocity at the second and third tracing was observed in the visual only condition. This indicates that there were some changes in the phase of deceleration because there was no increase in the other indices related to the acceleration phase. ActualJy, though it has not been presented here, the increase in the amount of deceleration was observed from the results ofthe experiments. From Fitts's law, it can be said that the positioning of the finger at the corner of the edge became easier on repetition of tracing even when only visual information was available. The eventual modification of the indices of the tracing movement during the experiment was observed by pooling data for every consecutive five rectangles. There ~as no change in the finger movement when both visual and tactile information were given. However, adaptation did occur in the visual only condition. The observation that the indices in the combined visual and tactile condition did not change suggests that the adaptation had occurred within the first five stimuli. This implies that the adaptation to the display-operation system is quicker if combined visual and tactile information is available than when only visual information is available. Visual function in shape tracing was investigated by using the number of eye fixations and the lead time at the corner of the rectangles. In the first tracing of a rectangle, there was no difference in the number of eye fixations for the two conditions. Although no statistical significance was observed, the lead time in the visual only condition was longer than that in the combined visual and tactile condition. The positioning of the finger at the corner of an edge would be performed using feedback of visual and tactile information in the combined visual and tactile condition. In the visual only condition, however, tactile information was not available for feedback. Therefore. the eyes would be fixed near the corner of the edge earlier than in the combined condition.
Downloaded by [Columbia University] at 21:35 01 March 2015
The influence of visual and tactile information on shape tracing
659
The comparison of repetitions of tracing showed that there were decreases in number of the eye fixations. As discussed above, it was considered that the repetitive tracing movement was performed with the help of stored somatosensory information. Thus, the contribution of visual feedback to positioning may be decreased by repetition. This decrease was more marked in the combined visual and tactile condition than in the visual only condition although somatosensory information from muscles and joint receptors would be the same in the two conditions. This suggests that combined tactile information with vision and other somaosensory information are able to reduce the contribution of visual function to repetitive tracing. The observation of eye movements during the experiment showed that there was a decrease in both the number of eye fixations and the lead time at the beginning of the experiment in the visual only condition such that these indices almost reached the same level as in the combined visual and tactile condition. This implies that subjects made more use of vision for the positioning movement at the beginning of the experiment. These results suggest that the load on vision during the adaptation process may be reduced if both visutal and tactile information are available. 5. Conclusion Adaptation experiments were performed by deforming the relationship between visual and tactile information in order to investigate finger and eye movements during shape tracing when combined visual and tactile information were avaialble. Characteristics of finger and eye movements remained stable when combined visual and tactile information was available. However, when only visual information was available. these characteristics were initially modified and then became stable during the experiment. The velocity and acceleration of the finger movement were higher in the combined visual and tactile condition than in the visual only condition once the steady state was reached. With repetition of the tracing, the finger movement became quicker and the contribution of visual function was reduced in the combined condition. From these results, the following conclusions can be made. When combined visual and tactile information is available, there is a rapid adaptation to a display-operation system as measured by certain movement characteristics and eye fixation. After adaptation, the operational movement become faster. If the operational movement is repeated, the movement will become even faster and the use of vision will reduced. Acknowledgement The author would like to thank Dr John Seal for helping with the English text. References AKAMATSU, M. 1990, Study on visuo-tactile sensory integration in shape perception, Biomechanism 10 (Tokyo University Press, Tokyo), 23-32 (in Japanese). CARLTON, L. G. 1980, Movement control characteristics of aiming responses, Ergonomics, 23,1019-1032. CARLTON, L. G. 1981, Processing visual feedback information for movement control, Journal of Experimental Psychology, 7, 1019-1030. CROSSMAN, E. R. F. W. and GOODEVE, P. J. 1983, Feedback control of hand-movement and Fitts' law, Quarterly Journal of Experimental Psychology, 35A, 251-278. FISCHER, B. and ROGAL, L. 1986, Eye-hand coordination in man-a reaction time study, Biological Cybernetics, 55, 253-261.
Downloaded by [Columbia University] at 21:35 01 March 2015
660
The influence
0/ visual and tactile information on shape tracing
FlITS, P. M. 1954, The information capacity of the human motor system in controlling the amplitude of movement, Journal ofExperimental Psychology, 47, 381-391. HELD, R. and GOTTLIEB, N. 1958, Technique for studying adaptation to disarranged handeye coordination, Perception and Motor Skill, 8, 83-86. JEANNEROD, M. 1986, Mechanisms of visuornoior coordination-a study in normal and brain-damaged subjects, Neuropsychologia, 24, 41-78. MATHER, J. A. and LACKNER, J. R. 1980, Adaptation to visual displacement with active and passive limb movement: effect of movement frequency and predictability of movement, Quarterly Journal of Experimental Psychology, 32, 317-323. PRABLANC, c., ECHALLlER, J. E., J EANNEROD, M. and KOMILlS, E. 1979, Opt imal response of eye and hand motor systems in pointing at a visual target. II. Static and dynamic visual cues in the can trol of ha nd movement, Biological Cybernet ics, 35, 183-187. PRABLANC, C. ECHALLIER, J. F., KOMILlS, E. and JEANNEROD, M. 1979, Optimal response of eye and hand motor systems in pointing at a visual target I. Spatio-ternporal characteristics of eye and hand movements and their relationships when varying the amount of visual information, Biological Cybernetics, 35, 113-124. ROGAL, L. and FISCHER, B. 1986, Eye-hand-coordination-a model for computing reaction times in a visually guided reach task, Biological Cybernetics, 55, 263-273. SCHMIDT, R. A., ZEl.AZNIK, H., HAWKINS, B., FRANK, J. S. and QUINN, J. T., Jr. J 979, Motor-output variability-a theory for the accuracy of rapid motor acts, Psychological Review, 86, 415-451. SINGER, G. and DAY, R. H. 1966, Spatial adaptation and aftereffect with optically transformed vision: effects of active and passive responding and the relationship between test and exposure responses, Journal of Experimental Psychology, 7, 725-731. Manuscript received 4 July 1990.. Manuscript accepted 10 December 1990.
• ~mm~~.m~#4~;h~.~m~~~~m.~~~~~~~~~,B~~~~~
mtJf'F~n·~.~fT-:> t:; m~ib~ tp C:" _*i!1.1\ ~~H, o *1Jo" 13 .t {fiij:1J1ljf t.U: "?1t~"(t1it-t~fT-:> t:; fI1l~illbi)"c:' U:, m.~w{j@Iti3 J: um~J(b ~ ':M-r .0Wi' O);tfi~flU~~1M"~fi?r:; -t-~MHR:, tl1t cMjt~J'iij~~:~~T-b~-f4:T~~m~tb~ m1i:~}j.~~~~'f4:Q);tA~,:lt"'-r:* ~ ~ ~ c: C '/Jf1ttJ,,-:> tz; mt9.Q)[email protected]±.~;g-ilf~ ~ t) ~lt t) il!t" ,: -o n, "'C~~ L. t: tJf, c ttl:i:t!jl ~ M1l 0) rirJlij:~7FQ)J:i.g-(J)1J ~fmYt(J)h. ':It'''~16 ~ ~ -:> t:» c rt, t;, (J)Uf *-IJ" ~, mit ~ M1t C. iJf~':4~ ~ tLt::.af': Ii, t1jQ)1b ~ ilfj! < ~ -b C. ~,:tJt1t(1){VJh~{«tTT ~ ~ C ilf~~ ~
tJ" C:,~ ~ ht::.~:mt'.t~~t1:,
ttt:. o
Ergonomics Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/terg20
The influence of combined visual and tactile information on finger and eye movements during shape tracing MOTOYUKI AKAMATSU
a
a
Industrial Products Research Institute , 1-1-4, Tsukuba Science City, Ibaraki, 305, Japan Published online: 31 May 2007.
To cite this article: MOTOYUKI AKAMATSU (1992) The influence of combined visual and tactile information on finger and eye movements during shape tracing, Ergonomics, 35:5-6, 647-660, DOI: 10.1080/00140139208967844 To link to this article: http://dx.doi.org/10.1080/00140139208967844
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http:// www.tandfonline.com/page/terms-and-conditions
ERGONOMICS,
1992, VOL 35,
NOS
5/6, 647-660
The influence of combined visual and tactile information on finger and eye movements during shape tracing MOTOYUKI AKAMATSU
Industrial Products Research Institute, 1-1-4, Tsukuba Science City, 305, Ibaraki, Japan
Downloaded by [Columbia University] at 21:35 01 March 2015
Keywords: Vision;
Tactile sensation; Finger movement; Eye movement; Visuo-tactile integration.
Adaptation experiments in shape tracing were conducted to investigate finger and eye movements in various conditions of visual and tactile information. Maximum velocity, mean velocity, maximum acceleration and reacceleration point were calculated from finger movements. Number of eye fixations and lead time of eye fixation to finger position were calculated from eye movements. The results showed that for the finger movement the values of the indices studied were higher in the combined visual and tactile condition than in the visual only condition. The number of eye fixations decreased when subjects repeated the tracing and was more marked in the combined visual and tactile condition than in the visual only condition. The results suggest that finger movements become faster and use of vision is reduced when both visual and tactile information are given.
1. Introduction Information processing in humans is generally separated into sensory functions, central processing and motor functions. Sensory functions can then be divided into modalities, e.g. vision, audition and touch. However, in daily life, we obtain various modalities of sensory information simultaneously. Perception of extrapersonal space depends' on the integration of this information. Sensory function and motor behaviour do not work separately but rather interactively and integratively. In the interface between man and the extrapersonal space, as mentioned above, multimodal sensory information is used and operations with many degrees of freedom are performed. In contrast, at the man-computer interface, information is presented using only a single modality ofsensory information, i.e. visual information on a Visual Display Terminal, and the number of degrees of freedom for operation is limited. However, the number of degrees of freedom for operation in computers is increasing from one-dimensional key presses on the keyboard to two-dimensional movements using an input device such as a mouse. Furthermore, recent development of the glove type input device makes it possible to input data to the computer using various finger movements. In addition to this development of input devices, development of a display device using multimodal input can be expected to create a more natural and friendlier interface. Some systems of display of multirnodal display have been developed which give auditory information in addition to visual information. Auditory signals in such systems inform us about having touched an object. However, auditory signals are not a natural mode of sensation for touching. To overcome this problem, several systems which give tactile and force sensation are under development. However, the effect of multimodal input with visual and tactile sensory information on the efficiency of our behaviour has not been studied in detail. Thus, the aim of this study is to investigate the effects of combined visual and tactile sensory input on movements. 0014-0139/92 $3·00 01992 Taylor & Francis Ltd.
Downloaded by [Columbia University] at 21:35 01 March 2015
648
M. Akamatsu
In studies of hand-eye coordination, the relation between visual information and movements has been studied. The well-known study of Fitts proposed the law of difficulty in movement (Fitts 1954). Recent studies on the patterns of finger movement have been made by several authors (Carlton 1980, 1981, Schmidt et al. 1979, Crossman and Goodeve 1983, Prablanc et al. 1979a, I 979b, and Jeannerod 1986). Fischer and Rogal investigated the relation between reaching movements and eye movements by analysing reaction times (Fischer and Rogal 1986, Rogal and Fischer 1986). I n these studies, only visual information was under consideration as a source of sensory information for movement. Studies of the relation between multimodal sensation and movement have mainly focused on the effects of active movements on sensory functions (Held and Gottlieb 1958, Singer and Day 1966, Mather and Lackner 1980). It has been shown that the amount of adaptation to a deformed relationship between visual and tactile shape information was larger in active touch than in passive touch (Akamatsu 1990). In this paper, finger and eye movements during shape tracing with visual and tactile information have been investigated. Finger and eye movements were measured under conditions of visual information only and with combined visual and tactile information. 2. Method Adaptation experiments were performed in which the subjects adapted to a deformation of the relationship between visual and tactile shape information. In the experiment, subjects were asked to repeatedly trace the edges of a shape whilst finger and eye movements were measured to observe the characteristics of movement during and after adaptation. 2.1. Stimulus Twenty-eight rectangles were used as test shapes for tracing. The distance of an edge of these rectangles was in the range of 60 mm to 150 mm. The range of the length/width ratio was 1/5 to 5/1. An outline ofthe rectangle was presented on a CRT display as the visual shape stimulus. Each tactile shape stimulus was presented as an acrylic rectangular board which was 4 mm high. Subjects obtained tactile shape information by tracing the edges of the acrylic rectangles with the tip of the right index finger. The visual stimulus was presented on a surface mirror in front of the subject. This mirror was inclined at 45 0 so as to reflect the image of a CRT display placed to the left of the subject (figure I). The head of the subject was supported and held by a chin and forehead support. The distance from the eyes to the virtual image of the surface of the visual display was SO em. The cen tre of the visual display and the eyes were in the same horizontal plane. The acrylic rectangle for the tactile stimulus was positioned horizontally at the height of the elbow. The distance between the centre of the tactile stimulus and the body of the subject was 35 em. In order to measure finger movement during shape tracing a Position Sensitive Detector (PSD) System (Sel Spot System, SEL SPOT Corp.) was used. The target LED was attached to the dorsal side of the distal phalanx of the index finger and the position of this target was detected by the PSD camera which was situated 75 cm above the centre of the tactile stimulus. In order to measure eye movements during shape tracing, an Eye Position Detective System (Eye Mark Corder Mark IV, NAC Corp.) was used. The relation between visual shape and tactile shape was deformed in the length/width ratio by a micro-computer system. The ratio of deformation was 1·22 (i.e. enlarged lengthwise by 11/9) or 0·82 (i.e. enlarged in width by 9/11), and was
TJzc inflrrcnce q f ~ i s u aand l ractile information an shape rracing
649
Downloaded by [Columbia University] at 21:35 01 March 2015
constant during an experimental session. Subjects obtained visual feedback of the position of the distal phalanx of the index as an open circle, diameter 10 mm, on the xisual display. The visualized finger position was deformed identically with the \isu:tl shape. Therefore. discrepancy was not within each sensory modality but between sensory modalities. Subjects did not receive prior information about the deformation and, thus, they adapted to the discrepancy between visual and tactile information according to the visual and somatosensory information they obtained.
Figure I .
Experimental arrangement.
2.2. Slitn1(1li~condirions Experiments were performed in a dark room and visual information was obtained only from the CRT display. The condition with combined visual and tactile information was compared with the condition with visual information only. These two conditions were as follows:
Condirion I : The subject traced edges of the acrylic rectangle with his finger whilst a visualized outline ofthe rectangle was on the visual display. The position of the distal phalanx of the index finger was presented on the visual display and overlaid on the image of the rectangle. In this condition, the subject could use both visual and tactile information (combined visual and tactile condition). Corldirion 2: The subject drew a rectangle with his finger tip using the overlaid visualized finger position and an outline of the test rectangle. He moved his finger on a horizontally positioned board which was the same height as the rectangles. In this condition. the subject could only use visual information (visual only condition). 2.3. E-rprrimcntai procedure The subject was ~nstructedto trace the outline of each rectangle three times in a clockwise direction and as fast as possible. The index finger was brought by the experimenter to the starting point for tracing, which was the lower right corner. Then, an auditory pre-cue for the tracing was presented. Two seconds after the precue signal, a visual shape was presented on the screen a s the 'go' signal, and the subject started to trace. After the subject finished tracing, he took his finger off the
650
M. Akamatsu
Downloaded by [Columbia University] at 21:35 01 March 2015
rectangle. A sample of 45 rectangles were selected pseudo-randomly from the 28 test rectangles and were presented one after another. Two different deformations of }.22 and 0·82 were examined for both condition I and 2. Subjects were 22- to 32-year-old adult males and females. All of them were right-handed. Four subjects were studied in each condition.
2.4. Analysis 2.4.1. Analysis of the finger movements: The position of the distal phalanx of the index finger as measured by the PSD camera was separated into ventral-dorsal direction and right-left direction, and was sampled by 125 Hz. Data were taken from the part of the tracing movement which corresponded to the movement right to left from the start point to the corner (i.e. tracing length) and, then, from the corner away from the body (i.e. tracing width). A fourth-order Butterworth digital filter, cutoff frequency 2·5 Hz, was applied to the position data in order to reduce high frequency noise. Velocity was calculated by time differentiation of the position data. As an index at velocity, maximum velocity (figure 2, the downward-pointing, filled triangle) and mean velocity were used. The acceleration was obtained by time differentiation of velocity data. The acceleration pattern of the finger movement showed that the early stage was an acceleration phase and the later stage was mainly a deceleration phase. As an index of the acceleration phase, maximum acceleration was used (figure 2 a, the downward-pointing, open triangle). In the deceleration phase, several acceleration peaks and deceleration peaks were observed. This pattern of deceleration and acceleration suggests that the finger movement was controlled by visual and tactile feedback so as to reach the comer of the ractangle (Carlton 1980, Jeannerod 1986). The time of occurrence of the first deceleration peak of this phase was considered to be an index of the time of change from open loop (feedback) control to closed loop control (figure 2 a, the upward-pointing, open triangle). This time has been called reacceleration. In this paper, the relative position of the reacceleration along the edge was called the reacceleration point, and was obtained as another index from the acceleration pattern. 2.4.2. Analysis of eye movements: Data for eye posrnon were divided into an upward-downward component and a right-left component. These components were digitized at 125 Hz. Eye fixation was regarded as being when the eyes stayed in the same position for more than 100 ms. Eye movements were studied when the finger was near a comer and the number of eye fixations were counted when the finger was at a distance from a corner corresponding to 20 per cent of an edge of the rectangle (figure 2 b). The time difference between the beginning of eye fixation and when the finger came to this point indicated how early the eyes were fixed before the finger reached this point. This difference in time was called lead time (figure 2 C, t). Finger and eye movements were studied both during an adaptation process and during a period of stabilized performance. The adaptation process was followed by taking data from the observation zone for only the first of the three tracing movements and expressing them as the mean of each index for every five consecutive rectangles. Visual examination of results of the adaptation process indicated that the evolution of all the indices studied became stable after presentation of ten rectangles. Performance during the stabilized period was studied by the indices obtained in the observation zone for first, second and third tracing of rectangles II to 45. In addition, data were studied by calculating the means of data obtained from edges of
The influence of visual and tactile information on shape tracing
-11 ·.m~/f3: j\_-
1.0----reaccelc:ration point
:
It
jlOcm
"I
velocity
~
65 t
!Iac.v. 2
/\ \ A r0.5m1s wrl=bOn~
Downloaded by [Columbia University] at 21:35 01 March 2015
.t:.
O.5sec.
time
Figure 2. An example of measured finger and eye movement. (a) Displacement, velocity and acceleration of finger movement. '9': maximum velocity, 9: maximum acceleration. 6: reacceleration. (b) Displacement of eye and finger movement. Number of eye fixations was counted when finger was within 20 per cent of the distance of the edge from the corner. (e) 't' denotes the lead time of eye fixation to finger movement.
J 00 mm only so as to overcome the effects of distance of finger movement (Fitts 1954). Edges of 100 rom occurred most frequently in the test rectangles. Data were pooled for the two degrees of deformation.
3. Results 3.I. Finger movements Maximum velocity was taken as an index of speed of movement and its adaptation during the experiment is shown in figure 3 a. There was no apparent trend in the change of maximum velocity in the combined visual and tactile condition. However, there was a trend for an increase in maximum velocity in the visual only condition. The maximum velocity reached a steady state after the eleventh rectangle. This indicated that it took several trials until the maximum velocity became stable when visual information only was presented. To compare the maximum velocity in a steady state, maximum velocities on and after the eleventh rectangle were averaged (figure 3 b). The asterisks denote the statistical significances. The asterisks above the point denote that there was a significant difference between the combined visual and tactile condition and the visual only condition at this round of the repetition. The asterisks on the right-hand side of the points denote that there was a significant difference for the three tracings. In the visual only condition, the averages of maximum velocity were around 2 I cm/s, and there was no difference of maximum velocity for the three tracings. In the combined visual and tactile condition, the averages of maximum velocity were around 25 emls which were significantly faster than those in the visual only condition for each tracing. With repetition, the maximum velocities increased significantly. This indicated that the maximum velocity increased when a rectangle was traced repeatedly, if both visual and tactile information were available.
M. Akamatsu
652
-.
28
~ g
24
---.-
a:visual and tactile {n-40}
--0-
b:vlsuaJ {n.40}
a
]
1st tracing
MEAN:tSE
"i)
>
§
20
.§ ~
E
16
1 I
Downloaded by [Columbia University] at 21:35 01 March 2015
5
6
,
11 16 21 26 31 36 41
I I 1 I I f 10 15 20 25 30 35 40 45 I
stimulusorder
------
--0--
~
.3 ~
s
25
u
>
e:::s
e:vlsuaJand taalle (n.123) b:visual (n.130)
...
~
b
...
edge length: 1Oem MEAN%SE
.5
~
20
1st
2nd tracing
3rd
Figure 3. Comparison between maximum velocity of finger movements in the combined visual and tactile condition and that in the visual only condition. (a) Changes of maximum velocity of the first tracing during the experiment. (b) Comparison ofthe three tracings on and after the eleventh rectangle. Asterisks denote statistical significance levels; *:5%, ··:1%, ···0-5%.
The tendency for mean velocity during the adaptation period was similar to that of maximum velocity (figure 4 a). Averages on and after the eleventh rectangle were 7 to 8 cm/s in the visual only condition and 9 to 10 cmls in the combined visual and tactile; condition (figure 4 b). There were significant differences in each round of repetition. In both conditions; the mean velocities increased significantly. There was no noticeable change in maximum acceleration during the experiment for the combined visual and tactile condition (figure 5 a). In the visual only condition, the maximum acceleration increased at the beginning of the experiment, as did maximum and mean velocity. In the combined visual and tactile condition, averages in a steady state were about 0·9 cm/s which were significantly higher than these in the visual only condition (figure 5 b). There was no significant difference for maximum acceleration between the three tracing in either condition I or 2. This indicated that the maximum acceleration did not change if the rectangle was traced repeatedly.
The influence of visual and tactile information on shape tracing
----.~
l
a: visual and tactile (n-40) b: visual (n-40)
653
a
10
'-"
lst tracing
~
'u 0
MEAN±SE
'0 > ~ II}
E 5
Downloaded by [Columbia University] at 21:35 01 March 2015
1
I 5
11 16 21 26 31 36 41 I I I I I I I J 10 15 20 25 30 35 40 45 6
stimulus order
----.-
-
~
0
•••
10
"g 0
> ~ 4.)
b
12
~
~
e:visual and tactile (n-123) b:vlsual (n-130)
8
E 6
•••
•••
.~ .... ~ 15t
2nd
edge length: lOan MEAN±SE
3rd
tracing Figure 4. Comparison between mean velocity of finger movements in the combined visual and tactile condition and that in the visual only condition (a) and (b): as for figure 3.
The change of the reacceleration point was not observed in the combined visual and tactile condition (figure 6 a). However, changes were observed in the visual only condition. At the beginning of the experiment, reacceleration points were further from the corner of the edge than during the steady state. When both visual and tactile information were given, averages of the reacceleration points on and after the eleventh rectangle in the first tracing were about 0·91, and those in the second and third were about 0·95 (figure 6 b). There were significant differences for these three tracings. This indicated that the rcacceleration point came closer to the comer of the edge by repetition of the tracing. There were no significant differences among repetition in the visual only condition. There was no difference in the reacceleration point between the two conditions in the first tracing, however, the reacceleration point was closer to the comer in the combined visual and tactile condition than in the visual only condition with the second and third tracings. In the second tracing, there was a significant difference of the reacceleration point between the two conditions.
3.2. Eye movements The number of eye fixations near the end of the edge was an index of the degree to
M. Akamatsu
654 --..-
avlsual and taCllle (n.40)
---0--
b:vlsual (n-40)
a
1.0 1st tnlCing
MEAN±SE
0.5
Downloaded by [Columbia University] at 21:35 01 March 2015
1
I 5
11 16 21 26 31 36 41 I I I I I ·1 I 10 15 20 25 30 35 40 45
6 I
stimulus order
---~
1.0
••• 0.9
t
a:vlsual and tactile (n.123)
b
b:visual (n.130)
•••
r
•••
-1
MEAN±SE
0.8
~
0.7 0.6
edge length: 1Oem
1s t
2nd tracing
3rd
Figure 5. Comparison between maximum acceleration of finger movements in the combined visual and tactile condition and that in the visual only condition. (0) and (b) as for figure 3.
which vision was used for positioning the finger at the corner of a rectangle. The number of eye fixations during the experiment is shown in figure 7 a.There was no noticeable tendency in the combined visual and tactile condition although there was some fluctuation in values. In the visual only condition, the number of the eye fixations was about 1·7 on average at the beginning of the experiment. This average decreased to the same level as the number of fixations in the combined visual and tactile condition as the .experirnent proceeded. Averages on and after the eleventh rectangle are shown in figure 7 b. In both conditions, a decrease in the number ofeye fixations on repetition of tracing was observed but the decrease was significant only in the combined visual and tactile condition. The number of eye fixations in the second and third tracing in the combined visual and tactile condition was lower than in the visual only condition. In the second tracing, the difference between the two conditions was significant. These results indicated that the use of vision decreased on repetition of the tracing when visual and tactile information were available. There was a tendency for the lead time of the eye fixation to finger position to decrease in the visual only condition (figure 8 a). When only visual information was available, the eye movements initially led the finger movement by 400 ms. Then, this
The influence of visual and tactile information on shape tracing
655
a Ist tracing
Downloaded by [Columbia University] at 21:35 01 March 2015
MEAN:tSE
b edge length: 1Oem MEAN±SE
Figure 6. Comparison between reacceleration point of finger movements in the combined visual and tactile condition and that in the visual only condition (a) and (b) as for figure 3.
lead time decreased to less than 300 ms after the eleventh rectangle. In the steady state, the average of the lead time of the first tracing was about 290 ms in the combined visual and tactile condition and was about 325 ms in the visual only condition, though, there was no significant difference (figure 8 b). In both conditions, the lead time tended to be shorter in the second and third tracings. However, statistical significance was only observed in the visual only condition. This indicated that repetition of tracing with visual information reduced the lead time and, in condition 2, the lead time became close to that of condition 1.
4. Discussion The effects of combined visual and tactile information on finger and eye movements have been investigated by adaptation experiments with a deformed relationship between visual and tactile information. As experimental conditions, a visual only condition and a combined visual and tactile condition were used. In the visual only condition, the subject may integrate information from vision, sensations from muscle and joint receptors and efferent copy of the motor command. In the
M. Akamatsu
656
-..-
a: visual and tactile (n.40)
---0--
b: visual (n.40)
a
2.0 e
.2 ~
x t.:::
lst tracing
()
>,. 0
'-
1.5
MEAN±SE
c
....
.8 E c='
1.0
, I
Downloaded by [Columbia University] at 21:35 01 March 2015
5
11 16 21 26 31 36 41 1 I I I' I I I I 10 15 20 25 30 35 40 45
6
stimulus order
------0--
1.5
a:visuaJand lactlla (n.131)
b
bvtsuat (n.130)
.gc:: l"l:l lo(
l:
edge length: lOcm
0
>,. IU
'-
MEAN:t:$E
1.0
't.
0
~
E ::l
t::
0.5
Figure 7.
1s t
2nd tracing
3rd
Comparison between the number of eye fixations in the combined visual and tactile condition and that in the visual only condition. (a) and (b) as for figure 3.
combined visual and tactile condition, the subject may integrate information from vision, cutaneous sensation, sensation from muscle and joint receptors and efferent copy. For the purpose of investigating the adaptation process in each condition, the pattern of change of various indices was followed during the experiment. To investigate movement after adaptation, comparisons were made by averaging data on and after the eleventh rectangle. The relation between movement distance and velocity of the positioning movement have been studied by Fitts and others (Fitts 1954). Although the tracing movement of an edge was not the same as that which Fitts and others examined, the distance of movement may have had an influence on the velocity and the acceleration of the tracing movement we studied. Therefore, only the results from edges which had the same length were analysed. The effects of the length of edge on the indices of movement has not been examined in this study because oflack of data. The finger movement of the first tracing of a rectangle with the combined visual and tactile information had a higher maximum velocity and maximum acceleration than-that with visual information only. This implies that when tactile information is
The influence of visual and tactile information on shape tracing -+--
a:visual and tactile (n-44-29)
----0--
b:visual (n-37·26)
657
a
400 Ist tracing
MEAN±SE
300
200 1 I
Downloaded by [Columbia University] at 21:35 01 March 2015
5
11 16 21 26 31 36 41 I I I I I I I 10 15 20 25 30 35 40 45 6 I
stimulus order
-+--
a:visual and tactile{ 1st:n-13S,2nd:n-117.3rd:n c97)
---0--
bvlsuat (1stn..139.2nd:n.141.3rd:n.117)
b
350 ...-. u cu
.§ "'0
~
,.
MEAN±SE
250 200
1st
2nd
3rd
tracing Figure 8. Comparison between lead time of eye fixation to finger movement in the combined visual and tactile condition and that in the visual only condition. (a) and (b) as for figure 3.
added to visual information, we can perform rapid tracing movements by integrating visual and tactile information. Up to the reacceleration point, the movement can be considered to be executed under open loop control. If it is assumed that maximum acceleration indicates the amount of output from the control system, the results of maximum acceleration suggest that the gain under open loop control is greater when both visual and tactile information are obtained. The mean velocity in the combined visual and tactile condition was also higher than that in the visual only condition. According to Fitts's law, there is a relation between difficulty of positioning and movement time, i.e. an inverse of the mean velocity. This suggests that the positioning of the finger at the corner of the edge was less difficult with both visual and tactile information than with visual information alone. . The reacceleration point was considered to be an index of the change point from open loop control to closed loop control. However, the point of reacceleration may not be a good index of this change point when there is an important difference in the velocity. If there was a difference in the velocity of the open control movement, the change point from open loop to close loop control should not be at the same point even though the reacceleration points were the same. If the movement in the open
Downloaded by [Columbia University] at 21:35 01 March 2015
658
M. Akamatsu
loop control phase was very fast, the reacceleration point would be far from the corner because it will take time to reduce the velocity enough to change the control of the finger by sensory feedback. In our results, there was no noticeable difference in reacceleration point for the two conditions in the first tracing. However, as the mean velocity in the visual and tactile condition was about 1·3 times faster than that in the visual only condition, the change point from open loop control to closed loop control was considered to be closer to the corner when both visual and tactile information were given. From this discussion, we can deduce that the finger movement was faster when both visual and tactile information were available than when only visual information was available because of the high gain and the high proportion of open loop control for the movement. Comparisons between the first, second and third tracing of the rectangle showed that maximum velocity and mean velocity were increased at the second and third tracing when both visual and tactile information were given. As no increase was observed in maximum acceleration, the increase in velocity was not by the increase in gain of open loop control but rather by the increase in the proportion of open loop control. This was confirmed by the change in position of the reacceleration point, i.e. nearer to the corner. The movement by open loop control is, in other words, a preprogrammed movement. The effects of repetition of tracing suggested that the preprogrammed movement was enhanced by the stored somatosensory information obtained during the first tracing. As clear enhancement was not observed in the visual only condition, such enhancement must be the effect of a combined presentation of visual and tactile information or combined tactile information and information from muscles and joints. An increase in mean velocity at the second and third tracing was observed in the visual only condition. This indicates that there were some changes in the phase of deceleration because there was no increase in the other indices related to the acceleration phase. ActualJy, though it has not been presented here, the increase in the amount of deceleration was observed from the results ofthe experiments. From Fitts's law, it can be said that the positioning of the finger at the corner of the edge became easier on repetition of tracing even when only visual information was available. The eventual modification of the indices of the tracing movement during the experiment was observed by pooling data for every consecutive five rectangles. There ~as no change in the finger movement when both visual and tactile information were given. However, adaptation did occur in the visual only condition. The observation that the indices in the combined visual and tactile condition did not change suggests that the adaptation had occurred within the first five stimuli. This implies that the adaptation to the display-operation system is quicker if combined visual and tactile information is available than when only visual information is available. Visual function in shape tracing was investigated by using the number of eye fixations and the lead time at the corner of the rectangles. In the first tracing of a rectangle, there was no difference in the number of eye fixations for the two conditions. Although no statistical significance was observed, the lead time in the visual only condition was longer than that in the combined visual and tactile condition. The positioning of the finger at the corner of an edge would be performed using feedback of visual and tactile information in the combined visual and tactile condition. In the visual only condition, however, tactile information was not available for feedback. Therefore. the eyes would be fixed near the corner of the edge earlier than in the combined condition.
Downloaded by [Columbia University] at 21:35 01 March 2015
The influence of visual and tactile information on shape tracing
659
The comparison of repetitions of tracing showed that there were decreases in number of the eye fixations. As discussed above, it was considered that the repetitive tracing movement was performed with the help of stored somatosensory information. Thus, the contribution of visual feedback to positioning may be decreased by repetition. This decrease was more marked in the combined visual and tactile condition than in the visual only condition although somatosensory information from muscles and joint receptors would be the same in the two conditions. This suggests that combined tactile information with vision and other somaosensory information are able to reduce the contribution of visual function to repetitive tracing. The observation of eye movements during the experiment showed that there was a decrease in both the number of eye fixations and the lead time at the beginning of the experiment in the visual only condition such that these indices almost reached the same level as in the combined visual and tactile condition. This implies that subjects made more use of vision for the positioning movement at the beginning of the experiment. These results suggest that the load on vision during the adaptation process may be reduced if both visutal and tactile information are available. 5. Conclusion Adaptation experiments were performed by deforming the relationship between visual and tactile information in order to investigate finger and eye movements during shape tracing when combined visual and tactile information were avaialble. Characteristics of finger and eye movements remained stable when combined visual and tactile information was available. However, when only visual information was available. these characteristics were initially modified and then became stable during the experiment. The velocity and acceleration of the finger movement were higher in the combined visual and tactile condition than in the visual only condition once the steady state was reached. With repetition of the tracing, the finger movement became quicker and the contribution of visual function was reduced in the combined condition. From these results, the following conclusions can be made. When combined visual and tactile information is available, there is a rapid adaptation to a display-operation system as measured by certain movement characteristics and eye fixation. After adaptation, the operational movement become faster. If the operational movement is repeated, the movement will become even faster and the use of vision will reduced. Acknowledgement The author would like to thank Dr John Seal for helping with the English text. References AKAMATSU, M. 1990, Study on visuo-tactile sensory integration in shape perception, Biomechanism 10 (Tokyo University Press, Tokyo), 23-32 (in Japanese). CARLTON, L. G. 1980, Movement control characteristics of aiming responses, Ergonomics, 23,1019-1032. CARLTON, L. G. 1981, Processing visual feedback information for movement control, Journal of Experimental Psychology, 7, 1019-1030. CROSSMAN, E. R. F. W. and GOODEVE, P. J. 1983, Feedback control of hand-movement and Fitts' law, Quarterly Journal of Experimental Psychology, 35A, 251-278. FISCHER, B. and ROGAL, L. 1986, Eye-hand coordination in man-a reaction time study, Biological Cybernetics, 55, 253-261.
Downloaded by [Columbia University] at 21:35 01 March 2015
660
The influence
0/ visual and tactile information on shape tracing
FlITS, P. M. 1954, The information capacity of the human motor system in controlling the amplitude of movement, Journal ofExperimental Psychology, 47, 381-391. HELD, R. and GOTTLIEB, N. 1958, Technique for studying adaptation to disarranged handeye coordination, Perception and Motor Skill, 8, 83-86. JEANNEROD, M. 1986, Mechanisms of visuornoior coordination-a study in normal and brain-damaged subjects, Neuropsychologia, 24, 41-78. MATHER, J. A. and LACKNER, J. R. 1980, Adaptation to visual displacement with active and passive limb movement: effect of movement frequency and predictability of movement, Quarterly Journal of Experimental Psychology, 32, 317-323. PRABLANC, c., ECHALLlER, J. E., J EANNEROD, M. and KOMILlS, E. 1979, Opt imal response of eye and hand motor systems in pointing at a visual target. II. Static and dynamic visual cues in the can trol of ha nd movement, Biological Cybernet ics, 35, 183-187. PRABLANC, C. ECHALLIER, J. F., KOMILlS, E. and JEANNEROD, M. 1979, Optimal response of eye and hand motor systems in pointing at a visual target I. Spatio-ternporal characteristics of eye and hand movements and their relationships when varying the amount of visual information, Biological Cybernetics, 35, 113-124. ROGAL, L. and FISCHER, B. 1986, Eye-hand-coordination-a model for computing reaction times in a visually guided reach task, Biological Cybernetics, 55, 263-273. SCHMIDT, R. A., ZEl.AZNIK, H., HAWKINS, B., FRANK, J. S. and QUINN, J. T., Jr. J 979, Motor-output variability-a theory for the accuracy of rapid motor acts, Psychological Review, 86, 415-451. SINGER, G. and DAY, R. H. 1966, Spatial adaptation and aftereffect with optically transformed vision: effects of active and passive responding and the relationship between test and exposure responses, Journal of Experimental Psychology, 7, 725-731. Manuscript received 4 July 1990.. Manuscript accepted 10 December 1990.
• ~mm~~.m~#4~;h~.~m~~~~m.~~~~~~~~~,B~~~~~
mtJf'F~n·~.~fT-:> t:; m~ib~ tp C:" _*i!1.1\ ~~H, o *1Jo" 13 .t {fiij:1J1ljf t.U: "?1t~"(t1it-t~fT-:> t:; fI1l~illbi)"c:' U:, m.~w{j@Iti3 J: um~J(b ~ ':M-r .0Wi' O);tfi~flU~~1M"~fi?r:; -t-~MHR:, tl1t cMjt~J'iij~~:~~T-b~-f4:T~~m~tb~ m1i:~}j.~~~~'f4:Q);tA~,:lt"'-r:* ~ ~ ~ c: C '/Jf1ttJ,,-:> tz; mt9.Q)[email protected]±.~;g-ilf~ ~ t) ~lt t) il!t" ,: -o n, "'C~~ L. t: tJf, c ttl:i:t!jl ~ M1l 0) rirJlij:~7FQ)J:i.g-(J)1J ~fmYt(J)h. ':It'''~16 ~ ~ -:> t:» c rt, t;, (J)Uf *-IJ" ~, mit ~ M1t C. iJf~':4~ ~ tLt::.af': Ii, t1jQ)1b ~ ilfj! < ~ -b C. ~,:tJt1t(1){VJh~{«tTT ~ ~ C ilf~~ ~
tJ" C:,~ ~ ht::.~:mt'.t~~t1:,
ttt:. o
