IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 31, NO. 12. DECEMBER 1990
An Eye Movement Communication- Control System for the Disabled JOHN R. LACOURSE,
FRANCIS C. HLUDIK, JR.,
Abstract-The discrete electro-oculographic control system (DECS) has been developed as a communication tool for persons with severe handicaps. The system can be used as a means of adaptive control allowing persons with handicaps, especially those with only eye motor coordination to live more independent lives. This paper provides initial information about the design and capability of the DECS.
INTRODUCTION ERSONS with severe handicaps, especially the nonverbal, have limited methods of adaptive control and means of communication. In many disabilities such as paralysis, eye movement may be one of the few controlled movements available. A variety of adaptive switches and scanning devices are available, but can have slow response times and require some motor coordination. Our system, the discrete electro-oculographic control system (DECS) uses electro-oculographic techniques. From electrodes placed around the eyes the polarization potential or corneal-retinal potential (CRP) of the eyebulb can be recorded. This record is commonly known as an electro-oculogram (EOG). It is this steady-state CRP that is used when measuring eye movements. The CRP ranges from 0.05 to 3.5 mV in humans [l] and is linearly proportional to eye displacement. In addition to displacement, the EOG can be used to measure the velocity and acceleration of the eyeball which is proportional to muscular forces causing the eye movement. An excellent review of EOG literature and techniques is reviewed by Young and Sheena . The DECS may be used for augmentative communication and or control. The analog signals from the oculographic measurements in the DECS have been turned into signals suitable for control purposes. The rotational angle of the eye has been divided into discrete positions (target positions), each position being a data point. The DECS requires only small eye movements in the horizontal and vertical directions. Targets may be placed in a radius of less than 30" from center at a comfortable distance from the operator. A target is selected by staring at it for a preset length of time. The selection of a target activates the output drivers and can control a variety of electrical or mechanical devices. At present nine targets are used
Manuscript received February 27, 1989; revised February 3, 1990. The authors are with the Department of Electrical and Computer Engineering, University of New Hampshire, Durham, NH 03824. IEEE Log Number 9038990.
CORNEA-RETINAL POTENTIAL LEVELS
1% = q y D x N e
Fig. 1. Target plane, electrode placement, and CRP for the DECS.
giving a variety of responses with minimal physical effort. The derivation of the EOG is achieved by means of placing two electrodes on the outerside of the eyes to detect horizontal movement and another pair above and below the right eye to detect vertical movement. A reference electrode is placed on the forehead. The CRP is filtered, amplified, and digitized, then used to determine the approximate position of the eyes in the x-y target plane. The CRP voltages are relatively constant at particular eye positions and are either positive or negative in value depending on the quadrant in the target plane as shown in Fig. 1. The system allows the user eight momentary switch outputs for control and communication purposes. EOG techniques are not the only methods that can be used to detect eye movements. Sullivan and Weltman [ 3 ] have shown that eye movements can be recorded using the impedance method (ZOG). The ZOG records the variation in impedance presumably due to changes in current distribution resulting from movement of the eyeball. Electrodes placed near the eyes connected to a direct-coupled impedance recorder are used to obtain the ZOG measurements. Rinard and Rugg  have designed a system of eye movement detection using infrared (IR) light emitting diodes (LED's) and photoconductive arrays. The IR LED's and photodiode arrays are attached to a pair of glasses, coated with an infrared-reflective material. The
0018-9294/90/1200-1215$01.OO @ 1990 IEEE
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 31, NO. 12. DECEMBER 1990
IR LED is attached on the inside of the glasses and aimed toward the infrared mirror. The infrared light is absorbed by the dark pupil and reflected off the sclera. The photoarray detects the reflected IR light determining the position of the eye. The Eyecom  system has been developed by using similar cornea reflection optical techniques. Eyecom implements a 62 element vocabulary in a nested subdivision format. Eight subdivisions are used and are selected from a target array of nine elements. Optical transducers are not affected by electrode offset drifts, but require longer calibration time and extra measurement equipment. Bour et al.  have described an improved magnetic induction method for measuring eye movements. The double magnetic induction method detects eye position indirectly by determining the strength of the induced secondary magnetic field of a metal eye ring that has been placed on the sclera of the eye. A signal related to eye position is obtained from a magnetic detection coil positioned in front of the eye. Sentient Systems Technology  has a commercial portable communication system, called the Eye Typer, that uses controlled eye gazes. Dual cameras look at your eyes while images at a screen. Reflections off the camera are detected. An exciting new system developed by the Smith Kettlewell Eye Research Foundation  allows the user to control a computer with his eye movements. The system, called brain response interface (BRR), monitors your brain waves that are triggered by one’s eye movements. The system was developed for use with cerebral palsy victims and is not yet sophisticated enough for commercial applications. The DECS may have advantages over some of the existing methods of detecting eye movements, and is a safe, cost effective method of adaptive control. In order to measure the impedance change due to eye movements (ZOG) , a current must be passed between the electrodes raising the possibility of microshock. The DECS is completely isolated from the other electronics thereby eliminating the possibility of microshock. The induction method of eye movement measurements requires the placement of a metal ring on the sclera of the eye  and also the surface of the eye must be anesthetized with a local anesthetic before the ring is placed on the sclera. That procedure would make this unacceptable for adaptive control. The low impedance eye ring could also cause strong induced currents which might be harmful to the eye. This system does have a high resolution at angles below 3.5” degrees but reliable eye movements cannot be measured above this point. The DECS is different than the other systems in that it is totally under software control allowing versatility in output configuration and ease in operation. Each target has a separate output and can be configured for specific applications. Gain and offset voltage adjustments are automatically made by the microprocessor, making the DECS completely self adjusting. The development of DECS was undertaken in the hope that people suffering from physical disabilities can live more independent lives.
HARDWARE AND SOFTWARE Design The DECS is completely self-adjusting, utilizing a 2-80 microprocessor to adjust offsets, gains and performs a selftraining session. The controls available to the user are an on-off power switch and the microprocessor reset button. Inputs to the system are nine preset switches tailoring the system to the individual. These preset switches allow the selection of output mode, target resolution, and time response of the system. After the system has been configured to the individual’s needs, the electrodes are placed on the face, and the reset button is pressed, the system is completely automated under software control from this point.
Electrodes The electrodes used are reusable Ag-AgC1 biopotential skin electrodes ( I n Vivo Metric Systems ). Electrodes are mounted in a pair of Carrera Sport Glasses (Model Viper 11) and the electrolyte used is Sigma Gel. The electrode support holds the electrodes firmly against the face and does not interfere with normal vision. It is realized that some people are dermatologically sensitive to silver, thus making Ag-AgC1 electrodes untenable in those cases. However, gold plated electrodes are now avaiable that are usually hypoallergetic. Analog Instrumentation A block diagram of the total DECS system is shown in Fig. 2. The electrodes are directly connected through a double shielded coaxial cable to an isolation amplifier. The isolation amplifier provides protection from electrical hazards, a fixed gain of 10 and a commn mode rejection ratio that reduces common mode body noise. A second order low pass filter, cutoff at 0.5 Hz, eliminates the EKG, EMG, and EEG signals, and 60 cycle line noise. A 12-b digital to analog converter (DAC) adjusts the offsets due to mismatch in electrode bias voltage, amplifier input offset voltage, and voltages due to electrode alignment. The DAC adjusts the offsets producing zero output when the eyes are positioned straight ahead. The 13-b programmable gain amplifiers which are controlled by the microprocessor provide system gains ranging from 0 to 12 000. These amplifiers compensate for differences in CRP due to light conditions, electrode contact, and differences in potential between individuals. The analog to digital converters (ADC) have 8-b resolution with a dynamic range of 10 V . The ADC’s provide the processor with voltage levels that estimate the eye position in the X-Y target plane with a resolution from 1 / 2 to 1 Two programmable parallel points (Z80a-PIO’s) provide both input and output to the system. Nine targets are presently being used and are not limited to a fixed pattern. Six LED’s help in the training session, indicate which target has been selected, and display the status of the DECS. Nine preset switches allow the system response to be altered taking into consideration the individual’s eye motor O .
LA COURSE AND HLUDIK: EYE MOVEMENT SYSTEM FOR DISABLED
--ISOL AT ION AMPLIFIER
AMPL lFlE RS
HORIZONTAL MOVEMENT DETECTION
Fig. 3. Target test pattern
VERTICAL MOVEMENT DETECTION
Fig. 2. Block diagram of the DECS.
coordination, response time, and vision. Two response criteria are needed as inputs to program the DECS.I ) Response Time: The time the target must be selected by eye position before output is activated. Eight ranges provide response times from 1 / 2 to 4 s. The response time determines the speed of the training session. 2) Target Resolution: The amount of error between the target voltage levels obtained during the training session and those voltage levels needed to activate a target. The present values can be selected to give resolutions of 2, 5 , 10, or 15% of the total target field. Output in parallel mode is standard TTL compatible voltages. These voltages may be used directly to control output drivers or relays as the application warrants. Serial mode is standard RS-232 compatible, operating at 300 Bd. Software Software is divided into three major areas: gain and offset adjustment, calibration procedure, and target detection. LED's and a beeper, under software control, indicate the correct eye position necessary for the microprocessor to adjust offsets and gain levels. Coded error messages indicate that either offsets or gains are unable to be adjusted properly, usually requiring minor repositioning of the electrode support. The calibration procedure allows the microprocessor to store the CRP levels at each target. These levels are used during target detection to determine which target is being selected. Target detection involves slope detection and voltage comparison. If a positive or negative slope is detected, eyes in motion, no action is taken. Only when the slope is small are voltage comparisons made to determine if a target position is being selected. When the target has been selected for the preset amount of time the output is activated. RESULTSAND IMPLEMENTATION The amplitudes of the CRP were measured using the isolation and low-pass filter stages of the DECS and a Princeton Applied Research (PAR) instrumentation amplifier (Model 113). Since the adjustable gain stage of the
TABLE I (TOTALG A I N= 1000) Measure Peak Voltage in mV
Target Distance in Inches
100 300 350 500 700 1000 1200 1250
2.0 3.0 4.0 6.0 8.0 10.0 12.0
Angle of Rotation in Degrees
CorneaRetinal Potential in mV
4.1 9.4 14.0 18.4 26.6 33.7 39.8 45.0
0.10 0.30 0.35 0.50 0.70 1 .oo 1.20 I .25
DECS is not presently programmed to set the gains manually, the PAR amplifier was used. The PAR amplifier's bandwidth was set to dc-100 Hz. The recorded voltage levels were displayed and measured on a storage oscilloscope. A target divided into 1 in divisions was placed 12 in in front of the person. The CRP were measured as the eyes were moved to discrete positions from the target center. The CRP level is the peak voltage obtained as the eyes are moved equal distances to each side of the target center as shown in Fig. 3. Measurements were made on a clear afternoon in a well lighted room near an outside window. A total amplifier gain of 1000 was used. Table I shows the relation between target position, angle, and CRP. The CRP has a maximum value of 1.25 mV at 45" from center and a minimum value of 0 . 1 mV at 4.7". Comfortable eye movement range is 0-35" from center producing average corneal-retinal voltages between = 0.0 and 2: 1.O mV. To produce maximum output voltages of + / - 5 V the DECS gains must be approximately 5000, well within the DECS maximum gain of 12 000. Fig. 4 is an example of the relationship between eye position and CRP. A linear relationship between distance and potential was found as others have reported [ 11, [21. Measurements were obtained from eye movements made in one inch steps above and below the center position. The CRP is slightly less in the vertical position presumably due to the electrodes being a greater distance from the cornea and the large bone mass between the eyes
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 37, NO. 12, DECEMBER 1990
Fig. 4. CRP 1, 2, 3 , 4 in right and left of center. PAR gain = dc to 100 Hz.
1000, BW 5 V/div, 5 s/div
and the electrodes. Measurements were made to see if sudden changes in the lighting environment would affect the operation of the DECS. Measurements were made using the following equipment: disposable electrodes, isolation and low-pass filter stages of the DECS, and a PAR amplifier (gain = 1000, BW = 0 - 100 Hz). The eyes were moved a distance of 20" from center in the horizontal direction. The following lighting environments were used: 1) In a bright room with incandescent lighting ( 1 4 . 9 mL). 2) In a very dimly lighted room ( -0.21 mL). 3) In front of a computer monitor in a very dimly lighted room ( ~ 3 2 . mL). 5 4) In front of a computer monitor in a brightly incandescent lighted room ( = 37.2 mL). Note: Illuminance measurements were made under similar lab conditions. Subsequently, measurements were first made under condition 1 for 20 min. All lights were turned out in the room except a small light in the distance, and the measurements were made for 20 min under condition 2. With the lights remaining off measurements were made for 20 min at a distance of 24 in from a computer monitor, condition 3 . The lights were turned on again, condition 1, and measurements were made for 20 min in front of a computer monitor, condition 4. We observed a slight increase in CRP with an increase in illumination. A decrease in CRP was observed with a decrease in illumination. These lighting changes would have little effect on the operation of the DECS producing amplified potential changes between 0.5 and 0.125 V, when the DECS gain is adjusted to 5000. The EOG signals at each target position were studied and described. These potentials were taken directly from the ADC's inputs after offsets and gains has been automatically adjusted. The electrode support was used with Sigma Gel electrolyte and the offsets allowed to stabilize for 10 min before measurement began. The target used to
make these measurements are shown in Fig. 3 . The target was placed = 24 in in front of the person making an angle of rotation =26". The potentials at targets 0, 1 , 2, 3 , 4, 5 , 6, 7, 8, 0 are shown in Fig. 5 and are either positive, negative, or zero depending on the quadrant in the target plane. EVALUATION OF
The DECS was evaluated as to how well it could recognize target selection without false triggering. The performance of the DECS depends on the experience of the operator and the target arrangement. In all the trials the operator was sitting in a chair in front of a test pattern that was no further than 24 in away. In order to use the DECS the operator must be able to follow the simple directions necessary for the adjustment and training routines: look forward, look up, look down, look right, look left, and look at each target. The operator must also be able to understand that a target is chosen by looking at it for a certain period of time. As the operator gains experience using the DECS, the accuracy improves. In Table I1 the results of ten trials conducted on four different days is shown. The second author was the subject for these trials. The targets were arranged as in Fig. 3. The zero target was on the same level as the eyes and the other targets were 15 in from the center making the angle of eye rotation 32". In Table I1 all the delays were set for 1 s. The accuracy range depicted in Table I1 is at - / 25 b. The failure rate equals the number of times the target was selected and was either not activated or another target was activated. False triggering is the number of times the DECS activated the outputs when targets were not intentionally selected. The - / +5 b accuracy range was also tried, but results showed high failure rates. The DECS was able to detect targets 0, 1, 3 , 5 , and 7 with nearly 100% accuracy, but targets 2, 4 , 6, and 8 have much lower accuracy rates. Lower accuracy rates with targets 2, 4, 6, 8 are considered to be due to head
LA COURSE AND HLUDIK: EYE MOVEMENT SYSTEM FOR DISABLED
TABLE I1 DECS ERROR ANALYSIS Target Number
0 1 2 3 4 5 6 7 8 Halt
so 50 50 50 50
Window = -1+25 b so 6 50 1 41 0 49 2 0 38 50 0 39 0 49 2 20 0 2 0
% DECS Wrong
% False Trigger
1.3 0.0 0.0 0.4 0.0 0.0
0.0 18.0 2.0 24.0 0.0 22.0 2.0 60.0 0.0
0.0 0.4 0.0 0.0
movements. This problem is currently being addressed and is discussed under future improvements. An accuracy range of - / 25 b as compared to - / 15 and -/+40 b produced the best results for this set of trials. Some of the false triggering could have been eliminated if the delay was increased, but this would have slowed the response time. We have made other trials at delay time of 0.5 s, but experienced a high number of false triggering.
Fig. 6 . The complete DECS.
DISCUSSION AND CONCLUSIONS The DECS can be used as a means of adaptive control allowing the handicapped, especially those with only eyemotor coordination, to live more independent lives. Eye movements require minimal physical effort and allow direct selection techniques. This increases the response time and the rate of information flow. Results show that as the operator becomes more familiar with the DECS the accuracy increases. The accuracy analysis presented in the previous section is tabulated from ten testing periods. The first few trials produced low accuracy rates, but false triggering decreased and target detection improved as the operator became familiar with the system. The response time and target window (target resolution) can be adjusted to compensate for the user’s eye motor coordination. Targets may be arranged in any pattern, but should be separated so target windows do not overlap. The output of the DECS has nine programmable parallel ports which can be configured to control electrical and mechanical devices. However, the reader should be reminded that the duration of each output is not under the control of the user. The DECS is shown in Fig. 6 . A severely handicapped person with some eye-motor coordination could use the DECS to control wheelchairs, home environmental control centers, communication devices, and home computers. The word “some” used here is unqualified. Subjects with abnormal eye movements such as gaze palsies, impaired saccades or pursuit, etc. have not been tested. However, the authors’ feel that there is a large group of people who would benefit from this system who have normal eye movements.
Fig. 7. The music synthesizer program.
A music synthesizer program is shown in Fig. 7 as example of application. Serial output can easily be implemented giving the DECS the possibility of interfacing a computer either through a serial or parallel ports. Many software programs have been developed for the handicapped. These programs include video games, educational material, scanning routines and communication systems. Video games would provide entertainment and relaxation while educational material could be utilized for classroom and individual instruction. The DECS could also be used for scientific research, in such areas as eye movement disorders and slip pattern analysis. The DECS could be used, with slight software modifications, as a digital electro-oculogram (EOG). The DECS would measure and process the levels, and either display the results or send them to a host computer. The total cost of parts of the DECS is approximately $750.00. This cost could be used to estimate the production costs of this device. In comparison, an IR modular eye movement monitor produced by Universal Initram Corporation  costs about $12 175.00.
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING. VOL. 37. NO. 12. DECEMBER 1990
FUTURE IMPROVEMENTS The electrode support and head-target positioning are two areas of the DECS that need additional research. The diameter of the reusable electrodes should be increased providing better skin-electrode contact. Different types of soft silver/silver chloride electrode pads or other types of electrodes could also be tried. The use of a flexible electrode support might eliminate the adjustment that must be made between different users. A three-electrode EOG has been developed [lo]. Electrodes are placed on each side of the nose (between the eye and nose) and on the forehead. Both horizontal and vertical CRP could be obtained with this arrangement of electrodes, but vertical channel cross talk was experience with purely horizontal eye movements. Drift voltages were reported to be a problem in this system along with the need for increased signal-tonoise ratio and a better electrode support. However, this technique may be interfaceable with the DECS. Target positions being relative to the head and not the eyes is the greatest cause of error. We chose to solve this problem by increasing the target window and detecting gross eye movements. However, this would only allow persons with minimal head tremor usage of the DECS. The system is currently undergoing modifications to improve accuracy. An “eye mouse” is being implemented to provide visual feedback and, we hope, eliminate alot of the inaccuracies associated with head movements. The “eye mouse” will continuously display the eye position relative to the targets. This will, we hope, increase accuracy and reliability and allow more targets to be displayed. Adaptive control interface standards (set compatibility STD) are being developed to make controls compatible with aids from different manufacturers [ 111. These standards should be incorporated into any future output configurations.
REFERENCES [I] C. Kris, “Vision: Electro-oculography,” in Medical Physics, Vol. 111, 0 . Glasser, Ed. Chicago: Year Book, 1960, pp. 692-700. 121 L. Young and D. Sheena, “Eye movement measurement techniques,” in Encyclopedia of Medical Devices and Instrumentation. Vol. 2 , J. G. Webster, Ed. 1988, pp. 1259-1269.  L. Geddes and L. Baker, Principles of Applied Biomedical Instrumentation. New York: Wiley, 1968.
G. Rinard and D. Rugg, “Communicationlcontrol applications of ocular transducer,” in Proc. Annu. Cant Syst. Devicesfor the Disabled, 1972, pp. 139-142. M. Rosen and W . Durfee, “Preliminary report on eyecom, and eye movement detection and decoding system for nonverbal communication,” in Proc. Annu. Con$ Syst. Devices f o r the Disabled, 1978, pp. 167-171. J . Bour, J . Van Gisberger, J. Bruijns, and F. Ottes, “The double magnetic induction method of measuring eye movement results in monkey and man,” IEEE Trans. Biomed. Eng., vol. BME-31, pp. 419-427, May 1984. Eye Typer Model 200, Sentinent Syst. Technol., Inc. 5001 Baum Bivd. ,-Pithburg, PA. J . Hellwiwell. “Glimuse of the future: It’s in the minds’s eye,” PC Week, vol. 6, p . 17(1). Universal Initram Corporation, written correspondence, Feb. 1985. D. Asche, A. Cook, and H. Van Ness, “A three-electrode EOG for use as a communication interface for the nonvocal, physically handicapped,” in Proc. Annu. Con$ Eng., Med., Bio., vo1.18, 1976, p. 2. TRACE Res. Development Center on Commun., Contr. and Comput. Access for Handicapped Individuals, Bulletin, May 1984.
John R. LaCourse (S’75-M’80) received the B.A. degree in biophysics in 1974 and the M.S. and Ph.D. degrees in biological engineering in 1977 and 1981, respectively, all from the University of Connecticut, Storrs. Currently, he is a Professor of Electrical Engineering in the Department of the Electrical and Computer Engineering at the University of New Hampshire, Durham. He is also Coordinator of the UNH Human Factors Research and Development Center and is a member of the Department’s Biomedical Engineering Center. He has active research and development projects in the areas of noninvasive instrumentation for the detection of atherosclerosis and muscle fatigue, hazard detection during TUR electrosurgery, and rehabilitation engineering especially the quantification of functional mobility. He is a consultant to industry in the areas of electrosurgery and ergonomics.
Francis C. Hludik, Jr. (S’81-M’83) received the B.S. and M.S. degrees in electrical engineering from the Department of Electrical and Computer Engineering at the University of New Hampshire, Durham, in 1982 and 1985, respectively. He is currently an instructor in the Department of Electrical and Computer Engineering at the University of New Hampshire. His research interests focus on the applications of ASIC technologies and microprocessor-based systems in medical instrumentation