THE INTERNATIONAL JOURNAL OF MEDICAL ROBOTICS AND COMPUTER ASSISTED SURGERY Int J Med Robotics Comput Assist Surg (2014) Published online in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/rcs.1606
ORIGINAL ARTICLE
A pneumatic laparoscope holder controlled by head movement
Kotaro Tadano* Kenji Kawashima Precision and Intelligence Laboratory, Tokyo Institute of Technology, 4259 Nagatsuta-cho, Midori-ku, Yokohama, Kanagawa 226-8503, Japan *Correspondence to: Kotaro Tadano, Precision and Intelligence Laboratory, Tokyo Institute of Technology, Kotaro Tadano, 4259 Nagatsuta-cho, Midori-ku, Yokohama, Kanagawa 226–8503, Japan. E-mail: [email protected]
Abstract Background In traditional laparoscopic surgery, the laparoscope is handled by a camera assistant according to verbal instructions from the surgeon. Thus there is a strong need for a laparoscope holder who intuitively provides the appropriate view with excellent stability. Methods A pneumatically driven robotic arm was developed to hold and manipulate a laparoscope. The robotic arm is operated by the user’s head movement, which is measured with gyroscopes attached to the operator’s head and body. Results We confirmed experimentally that head tracking can be performed accurately using the proposed method. The experimental results indicated that the robotic camera holder has sufficient dynamic characteristics to quickly follow the operator’s head movement. Conclusions A laparoscope holder control system has been developed. In this system, a laparoscope is held by a pneumatically driven robotic arm that is controlled to follow the operator’s head movement. The experimental results prove the effectiveness of the system. Copyright © 2014 John Wiley & Sons, Ltd. Keywords laparoscopic surgery; camera holder; penumatically-driven robotic arm; control by head movement
Introduction
Accepted: 8 July 2014
Copyright © 2014 John Wiley & Sons, Ltd.
The number of laparoscopic surgeries and their applications have increased in recent years. Since the operations are performed based on laparoscopic images, clear and vivid vision is an essential element of laparoscopic surgery. In recent years, laparoscopes with 3D hi-vision have been commercialized. 3D vision can provide a sense of depth, which is expected to improve the intuitiveness of surgery. On the other hand, in conventional laparoscopic surgery, a scopist has to hold the laparoscope and change its angle under verbal instruction from the operating surgeon. Therefore, the scopist needs a deep understanding of the surgical procedure and must have excellent dexterity. Camera shake may occur due to fatigue of the scopist, which may cause the surgeon to become nauseous, especially in 3D vision. Therefore, a laparoscope holder is an important and effective advancement for laparoscopic surgery.
K. Tanado and K. Kawashima
Laparoscope holders have been studied for many years, and some are commercially available. AESOP is a holder with a SCARA-type robotic arm with four degrees of freedom (4 DOFs) (1). Voice command was also added in the second version. Voice command has the advantage that the operator’s hands remain free throughout the operation. However, there is concern that voice recognition might fail due to environmental noise. EndoAssist is a camera holder with 3 DOFs driven by stepper motors (2). A foot switch is used for its operation. The arm moves with a constant velocity while the foot switch is on. The direction of movement is switched by a head gesture of the operator. The head gesture is detected by an infrared transmitter attached to the operator’s head and a receiver installed on the monitor. FreeHand, whose arm is very compact, is a successor to EndoAssist (3). It follows the operation method of EndoAssist. The receiver and transmitter must be properly arranged in order to avoid interruption of the infrared signal passing between them. ViKY is a compact scope holder that can be positioned on a patient (4); it has 3 DOFs and is operated by a foot switch or voice commands. Naviot is also a camera holder, and it achieves zoom in and out not with robotic arm movement but optically with a dedicated laparoscope (5). The holder is operated by a hand switch on the system, and it is suitable for safe operation. However,
the holder cannot be used with other laparoscopes. There is also a unique design using a combination of parallelogram mechanisms to achieve compactness and easy setup (6). These systems are operated by a small joystick mounted on the instrument. Operation methods (such as voice control or switches) in current laparoscopic holders seem inferior in that they are not intuitive because the operator must change his/her focus between operation of the camera and the surgical procedure. In this research, a laparoscope control system using a pneumatically driven robotic arm was developed. The developed system is operated by the user’s head movement to provide high intuitiveness and synchronization. We also confirm the tracking accuracy of the system experimentally. Moreover, the effectiveness of the system is demonstrated through in vivo experiments.
Concept of the holder system In the proposed laparoscope holder system, instead of a scopist holding the laparoscope, a pneumatically driven robotic arm holds the camera, and the arm is positioned by the operator’s head movements, as shown in Figure 1. The authors have previously developed a surgical robot system with a pneumatic drive called IBIS (8,9). The
Figure 1. Concept of the proposed system Copyright © 2014 John Wiley & Sons, Ltd.
Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
A Pneumatic laparoscope holder controlled by head movement
concept of the pneumatically driven robotic arm in IBIS is applied to the laparoscope holder system. All the robotic holders described in the Introduction use electrical motors for actuation. However, in this system, pneumatic actuators are used instead. This is because pneumatic actuators have many safety advantages such as low heat generation, compressibility, the ability to control the maximum force by regulating the supply pressure, ease of releasing the acting force by discharging compressed air in the actuator, and the ability to realize an arm that is both compact and lightweight. The robotic arm is controlled by head movement measured using two gyroscopes attached to the operator’s head and body. As shown in Figure 1, the view angles of the laparoscope for up and down, left and right, and rotation synchronously follow the operator’s head movement, and the camera’s zoom in and out synchronously follows the operator’s anteroposterior body movements. The operator’s movements can be measured with external sensors such as optical or magnetic 3D measurement devices. However, some markers must be attached to the operator, and a receiver must be installed nearby. The markers and receiver must be properly arranged in order to avoid interruption of the infrared signal or interference from other devices. Therefore, we use gyroscopes, which are internal sensors, in the proposed system. Rotation speed during head movements is directly detected by the 3axis gyroscope attached to the operator’s head. Zoom in and out can be measured from the translation velocity of the head movement. Velocity can be theoretically obtained by integrating data measured by an acceleration sensor attached to the operator’s head. However, it is difficult to obtain an accurate velocity from the acceleration sensor due to errors caused by gravity compensation, drift of the zero point, and sensor noise. Therefore, we focused on anteroposterior body movements of the operator. We estimated translation movement of the head from anteroposterior body movement based on a gyroscope attached to the operator’s body. The estimated value is used as a control signal for the zooming movement (in and out) of the robot. The laparoscope image can be displayed to a monitor or on a head mounted display (HMD). When the HMD is used as the monitor, the image is always kept in front of the operator’s eyes, even when the operator turns his/ her head. Therefore, using a HMD can provide highly intuitive operation and can avoid the need to consider the monitor layout. Moreover, when using a 3D laparoscope, a highly immersive stereoscopic image can be provided without crosstalk. HMDs have already been introduced in clinical practice, and their effectiveness has been reported (10). However, wearing a HMD for an extended period of time may cause fatigue in the user’s neck or eyes due to the weight and short focal length. The user’s view of his/ her surroundings is also limited to what is immediately Copyright © 2014 John Wiley & Sons, Ltd.
below. Users can choose a suitable display according to their situation or preference.
Materials and method Pneumatically driven robotic holder The developed pneumatically driven robotic arm that holds the laparoscope is shown in Figure 2. The robotic holder has 4 DOFs in total, consisting of 3 rotational DOFs around the inlet of the trocar cannula and 1 translational DOF along the insertion direction. Rotation around the scope axis (q4) allows the user to adjust the roll angle view. It is useful for both straight and angled scopes. When an angled scope is used, a coordinate transformation is applied to maintain the correspondence between the user’s head motion and the scope image. Parallel link mechanisms, as shown in Figure 3, provide a remote center of motion (RCM) (7) so that the pivot point at
Figure 2. Developed pneumatic holding arm
Figure 3. Structural drawing of pneumatic holding arm Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
K. Tanado and K. Kawashima
the trocar cannula is mechanically immovable without direct support. As shown in Figure 3, rotation of q1 and q4 is realized by transmitting the output of vane-type pneumatic motors with timing belts. Rotation in the vertical direction q2 is achieved by converting linear motion of the pneumatic cylinder to rotational motion using a slider-crank mechanism. The linear motion q3 is directly driven by the pneumatic cylinder, which controls the zoom in and out of the laparoscope. Increment-type encoders are used to measure the angles. Rotation angles q1, q2, and q4 are measured using rotary-type encoders, and q3 is measured using a wire-type linear encoder. Table 1 shows the motion ranges and maximum forces for each DOF when the supply pressure is 0.5 MPa (gauge). The angle q2 is defined as 0 degrees, at which the laparoscope becomes horizontal. A positive direction indicates that the tip of the laparoscope is pointing down. The state shown in Figure 3 is defined as the initial position. The motion ranges can easily be limited by implementing mechanical stoppers to the mid-stroke. The maximum force can easily be changed by regulating the supply pressure. The laparoscope is inserted from the rotation center of q4 via a dedicated adapter. The holder is designed so that it can easily be detached passively when an unexpected load is applied to the tip of the laparoscope. The robotic arm is 350 mm in length, 240 mm in height, and 80 mm in width in the initial position. A compact size is realized with the robotic arm, and its mass is only 0.98 kg.
Table 1. Working range and maximum torque of holding arm
q1 q2 q3 q4
Working range
Maximum torque
± 95 47° to 10° ± 75 mm ± 162°
1.39 Nm 2.52 Nm 39.3 N 0.107 Nm
System setup Figure 4 shows the configuration of the developed system. The system mainly consists of the pneumatically driven 4-DOF robotic arm shown above, two motion sensors, a foot switch, four pneumatic servo valves, eight pressure sensors, and electrical power supply devices. The motion sensor (ZMP, IMU-Z2) is equipped with a 3-axis gyroscope, a 3-axis geomagnetic sensor, and a 3-axis acceleration sensor. The resolution of the gyroscope is 3.8×103 (°/s), which is sufficiently small relative to human movement. It is experimentally confirmed that the zero drift of the gyroscope can be ignored. Therefore, the gyroscope is suitable for measuring human movement. Measured signals from the gyroscopes in the motion sensors are transmitted to the control PC via CAN communication. The control signals are sent from the PC to the servo valves through a digital-to-analogue converter. The foot switch is positioned at the feet of the operator. Head movement operation is activated only while the foot switch is pressed. Therefore, the operator can move his/her head freely when the foot switch is released.
Head tracking using gyroscopes In order to control the 4-DOF robotic arm with head movement, it is necessary to measure translational motion in the longitudinal direction as well as rotational speed of the head. Angular velocity of the head ωh can be measured by the gyroscope mounted on the operator’s head. Translational motion is measured by the gyroscope attached to the operator’s body as follows: Translational motion of the head is given by the speed of the neck, as shown in Figure 5. This speed is the cross product of the angular velocity of the hip ωw (obtained by the gyroscope) and the vector from the waist to the neck l in absolute coordinates.
Figure 4. System configuration Copyright © 2014 John Wiley & Sons, Ltd.
Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
A Pneumatic laparoscope holder controlled by head movement
geomagnetic sensor, we approximate them as zero in this paper because the actual angular values are small.
Control system The pneumatically driven 4-DOF robotic arm is controlled by the angular velocity ωh and translational velocity vh. Figure 6 shows a block diagram of the control system. To match the direction of the head coordinate system with that of the laparoscope coordinate system, the angular velocity at the head ωh is transformed into the reference angular velocity at the robotic arm ωref by ωref ¼ Rta ωh
(3)
Where Ra is a matrix indicating the attitude of the robot, which is given by the angular position of each joint. Ra ¼ Eiq1 Ejq2 Ekq4
(4)
Here, E denotes a rotation matrix. Next, the reference angular velocity ωref is transformed into the angular velocity of each joint. 2̇ 3 q1 6̇ 7 (5) 4 q2 5 ¼ J 1 ωref Figure 5. Measurement of translational velocity of head motion
q̇ 4
(1)
Here, J is a Jacobian matrix that maps the displacement at each joint to the attitude of the arm, which is given by J ¼ ex Eiq1 ey Eiq1 Ejq2 ez (6)
By multiplying the translation matrix R by the value obtained from the cross product, velocity vh in the head coordinates is calculated as follows:
The velocity of the Axis q3, which is the axis of the zoom in and out movement, is defined as the anteroposterior direction of vh obtained from Equation (2).
ωw l
vh ¼ Rðωw lÞ
(2)
The longitudinal direction of the vector vh that is in the anteroposterior direction of the head is used as a control signal for zooming in and out. Here, the inclination angles of the upper body and head are needed in order to obtain l and R. Although those values can be calculated by integrating the rotation speed or the output of the
q̇ 3 ¼ vh ey
(7)
The reference velocities shown above are integrated while the foot switch is on to give the reference position vector qref. When the foot switch is off, the reference velocities are set to zero. As shown in Figure 6, the reference torque τ ref to be generated at each joint of the robotic arm is given by a PD controller with feed-forward compensation of the inverse dynamics calculated from the reference
Figure 6. Block diagram of control system Copyright © 2014 John Wiley & Sons, Ltd.
Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
K. Tanado and K. Kawashima
position. Kpp and Kpd in Figure 6 denote the proportional and differential gain for the position, respectively. The reference torque is transformed into the reference force for each pneumatic actuator Fref by the Jacobian Ja, which maps the torque from the force. The reference force Fref is controlled
by a PID controller with a feedback force F calculated from the pressure difference in the actuator. The control signal is sent to the pneumatic servo valve, and the force is generated by charging compressed air into the actuator. This process is conducted in the driving force controller block shown in Figure 6. All calculations are performed in a period of 1 ms in the computer shown in Figure 4.
Results Evaluation of head tracking performance
Figure 7. Experimental results for velocity measurement of translational head motion
First, an experiment was conducted to evaluate the performance of the proposed measurement method. The translational velocity was measured by a 3D optical measurement system (Northern Digital Inc., Polaris Spectra) as a reference and by the proposed method, simultaneously.
Figure 8. Experimental results for trajectory tracking of robotic arm (a) q1 (c) q2 (b) q3 (d) q4 Copyright © 2014 John Wiley & Sons, Ltd.
Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
A Pneumatic laparoscope holder controlled by head movement
The user moved his head back and forth during the measurement. In the measurement with the 3D optical measurement system, a passive marker was mounted on the user’s head, and velocities were calculated by differentiating the output numerically. In the experiment, the length of the vector l in Equation (2) was set to 660 mm. Figure 7 shows the experimental results for the comparison of velocities between the two measurements. It can be seen that velocity measured with the gyroscope corresponds to the reference value obtained with the 3D optical measurement system. The proposed method is therefore proved to measure the motion as accurately as the 3D optical measurement device.
Figure 9. Tracking experiment
Evaluation of the robotic arm’s dynamic performance Next, the tracking performance of the pneumatically driven robotic arm was measured experimentally. Parameters such as friction and center of gravity, which are used to obtain the inverse dynamic model, were determined experimentally in advance and agreed well with the driving torque during free motion. Controller gains were determined by trial and error to produce good tracking performance. A laparoscope with a mass of 300 g (Shinko Optical Co., Ltd, HD-101D) was used in this experiment. The light cable was attached to the base arm so that its mass had little influence on performance. The minimum jerk trajectory is given as a reference position for each joint individually. The experimental results for trajectory tracking are shown in Figure 8. In the figure, q, ˙q, and q¨ denote the position, velocity, and acceleration of each joint, respectively. It is clear from this figure that all joints follow the reference trajectory well. The experimental results show that the robotic arm is capable of generating velocities of at least 1.0 (rad/s), 1.0 (rad/s), 150 (mm/s), and 4.0 (rad/s) for joints q1 to q4, respectively. Also, the accelerations achieved are 5.0 (rad/s2), 5.0 (rad/s2), 0.8 (m/s2), and 40.0 (rad/s2).
Evaluation of tracking capability for head movement The tracking performance of the developed holder system was next evaluated experimentally. As shown in Figure 9, an A4 size sheet of paper with four target points was placed approximately 100 mm from the tip of the laparoscope. In this experiment, a 2D-HD scope with a mass of 420 g (Olympus, LTF-S190-10) was used. An operator wearing a HMD operated the robotic arm by head movement to bring the target to the center of the field of view in the order shown in Figure 9. After two cycles, Copyright © 2014 John Wiley & Sons, Ltd.
Figure 10. Transition of the field of view. The operator turned from the top left target to the top right one (from 1 to 2), diagonally from the top right to the bottom left (from 2 to 3), from the bottom left to the bottom right (from 3 to 4), diagonally from the bottom right to the top left (from 6 to 5), zoomed in (from 5 to 6), zoomed out (from 6 to 7), rolled right (from 7 to 8), rolled left (from 8 to 9), and rolled right (from 9 to 10)
the operator zoomed in and out and then rolled the view to the left and right. The above operation was conducted as quickly as possible. The viewing area projected on the HMD changed as shown in Figure 10. Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
K. Tanado and K. Kawashima
Figure 11 shows the velocity response of each axis during operation. The red lines in the figure show the reference velocities of each joint calculated from the gyroscopes, and the blue lines show velocities measured by the encoders attached to the joints. It is clear from the experimental results that the robotic arm tracks reference velocities in each joint well.
Comparison with manual operation To verify the effectiveness of the developed system compared with manual operation by a scopist, an ex vivo experiment was conducted. A training box for laparoscopy was used, in which five numbered targets were placed as shown in Figure 12. A 2D-HD laparoscope was used, and the image was shown on the display placed in front of the operator. The operator instructed the scopist or operated the robotic arm such that the intended target was in the laparoscope’s field of view. The operator grasped a pin with the forceps in his/her right hand. When the target was in sight, the operator touched it with the pin. Then, the operator switched the pin from the right hand to the left hand. After positioning the laparoscope on the next target, the operator touched the target
Figure 12. Experimental setup for evaluation of the developed system
with the pin. This procedure was repeated five times per set, and the completion time was measured. Two surgeons that were unfamiliar with the developed system participated as the subject operators in this experiment. In the manual operation, one of the surgeons grasped the laparoscope. Ten trials were conducted for each kind of scope operation (20 times in total). Figure 13 shows a comparison of the results. No significant difference can be seen between the two operation methods. However, it is confirmed that the developed system allows the task to be completed slightly faster than with a scopist.
Discussion Tracking performance One feature of the developed system is that the laparoscope holding arm smoothly tracks the operator’s head and body movements. To allow highly intuitive operation,
Figure 11. Velocity responses of robotic arm to head movements during operation Copyright © 2014 John Wiley & Sons, Ltd.
Figure 13. Experimental results for performance comparison between the developed system and conventional manual operation Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
A Pneumatic laparoscope holder controlled by head movement
Conclusions
Figure 14. In vivo experiment
the robotic arm must smoothly track movements without any time delay. Figure 7 demonstrates that the proposed head tracking with the gyroscope has almost the same performance as the 3D optical measurement device. Also, as can be seen in Figure 11, maximum velocity for each axis during operation is 0.4 (rad/s), 0.3 (rad/s), 70 (mm/s), and 0.6 (rad/s). The achieved velocities shown in Figure 8 are sufficiently large compared with these values. Thus, the developed system has satisfactory dynamic performance for operating the laparoscope. Compared with manual operation, the task duration using the developed system was slightly shorter than with a scopist, even though both operators were well-trained surgeons. Therefore, it is demonstrated that the developed system provides the intended view quickly with the same or better performance than with a professional scopist. In the future, we would like to compare the performance of the system with other robotic camera holders more systematically, as referred to in some reports (11–13).
In this research, we developed a holder control system in which a laparoscope was held by a pneumatically driven robotic arm that followed the operator’s head movements. Head movements were measured with gyroscopes attached to the operator’s head and body. View angles of the laparoscope for up and down, left and right, and rotation synchronously followed the head rotations. The zoom in and out of the camera synchronously followed the anteroposterior head movements of the operator. We confirmed the tracking accuracy of the system through experiments. Moreover, the effectiveness of the system was demonstrated through in vivo experiments. For future work, we are improving the system to conform to the medical regulations in several countries. Clinical trials of the system need to be conducted for several surgical procedures to make quantitative and statistical evaluations.
Acknowledgements This project was supported by the Program for Creating Start-ups from Advanced Research and Technology by the Ministry of Education, Culture, Sports, Science and Technology in Japan.
Conflicts of interest The author have stated explicitly that there are no conflict of interest in connection with this article.
Funding In vivo experiment To demonstrate the effectiveness of the developed system, in vivo experiments were performed three times on pigs (gastric resection) by several surgeons. Figure 14 shows a photograph of the experiment. In the experiment, a small problem arose because of the space between the robotic arm and the trocar, but the system setup was completed in 10 min. The scope angle smoothly changed as the operator intended. All the surgeons who used the system thought highly of its smooth and intuitive operability. Especially in gastric resection, where the camera angle is often changed, the developed system is effective. Additionally, no lens cleaning was required during the operations. Copyright © 2014 John Wiley & Sons, Ltd.
No specific funding.
References 1. Sackier JM, Wang Y. Robotically assisted laparoscopic surgery: from concept to development. Surg Endosc 1994; 8: 63–66. 2. Kommu SS, Rimington P, Anderson C, Rané A. Initial experience with the EndoAssist camera-holding robot in laparoscopic urological surgery. J Robot Surg 2007; 1(2): 133–137. 3. Rané A, Kommu SS, Grange P, et al. Initial laboratory experience with the FreeHand camera holding robotic arm in laparoscopic urological surgery. J Endourol 2008; 22: 129. 4. Long JA, Cinquin P, Troccaz J, et al. Development of miniaturized light endoscope-holder robot for laparoscopic surgery. J Endourol 2007; 21(8): 911–914. 5. Tanoue K, Yasunaga T, Kobayashi E, et al. Laparoscopic cholecystectomy using a newly developed laparoscope Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
K. Tanado and K. Kawashima
6. 7. 8.
9.
manipulator for 10 patients with cholelithiasis. Surg Endosc 2006; 20(5): 753–756. Herman B, Dehez B, Duy KT, et al. Design and preliminary in vivo validation of a robotic laparoscope holder for minimally invasive surgery. Int J Med Robot Comput Assist Surg 2009; 5: 319–326. Taylor RH, Stoianovici D. Medical robotics in computerintegrated surgery. IEEE Trans Robotics Automat 2003; 19: 765–781. Tadano K, Kawashima K. Development of a master slave system with force-sensing abilities using pneumatic actuators for laparoscopic surgery. Adv Robot 2010; 24(12): 1763–1783. Tadano K, Kawashima K, Kojima K, Tanaka N. Development of a pneumatic surgical manipulator IBIS IV. J Robot Mechatron 2010; 22(2): 179–187.
Copyright © 2014 John Wiley & Sons, Ltd.
10. Kihara K, Fujii Y, Masuda H, et al. New three-dimensional headmounted display system, TMDU-S-3D system, for minimally invasive surgery application: procedures for gasless single-port radical nephrectomy. Int J Urol 2012; 19: 886–889. 11. Yavuz Y, Ystgaard B, Skogvoll E, Marvik R. A comparative experimental study evaluating the performance of surgical robots AESOP and Endosista. Surg Laparosc Endosc Percutan Techn 2000; 10(3): 163–167. 12. Nebot P, Jain Y, Haylett K, et al. Comparison of task performance of the camera-holder robots EndoAssist and AESOP. Surg Laparosc, Endosc Percutan Techn 2003; 13(5): 334–338. 13. Herman B, Olias Lopez A, Rasse C, Raucent B. Experimental comparison of kinematics and control interfaces for laparoscope positioners. In Proceedings of the 9th National Congress on Theoretical and Applied Mechanics. Brussels, Belgium, 2012.
Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
ORIGINAL ARTICLE
A pneumatic laparoscope holder controlled by head movement
Kotaro Tadano* Kenji Kawashima Precision and Intelligence Laboratory, Tokyo Institute of Technology, 4259 Nagatsuta-cho, Midori-ku, Yokohama, Kanagawa 226-8503, Japan *Correspondence to: Kotaro Tadano, Precision and Intelligence Laboratory, Tokyo Institute of Technology, Kotaro Tadano, 4259 Nagatsuta-cho, Midori-ku, Yokohama, Kanagawa 226–8503, Japan. E-mail: [email protected]
Abstract Background In traditional laparoscopic surgery, the laparoscope is handled by a camera assistant according to verbal instructions from the surgeon. Thus there is a strong need for a laparoscope holder who intuitively provides the appropriate view with excellent stability. Methods A pneumatically driven robotic arm was developed to hold and manipulate a laparoscope. The robotic arm is operated by the user’s head movement, which is measured with gyroscopes attached to the operator’s head and body. Results We confirmed experimentally that head tracking can be performed accurately using the proposed method. The experimental results indicated that the robotic camera holder has sufficient dynamic characteristics to quickly follow the operator’s head movement. Conclusions A laparoscope holder control system has been developed. In this system, a laparoscope is held by a pneumatically driven robotic arm that is controlled to follow the operator’s head movement. The experimental results prove the effectiveness of the system. Copyright © 2014 John Wiley & Sons, Ltd. Keywords laparoscopic surgery; camera holder; penumatically-driven robotic arm; control by head movement
Introduction
Accepted: 8 July 2014
Copyright © 2014 John Wiley & Sons, Ltd.
The number of laparoscopic surgeries and their applications have increased in recent years. Since the operations are performed based on laparoscopic images, clear and vivid vision is an essential element of laparoscopic surgery. In recent years, laparoscopes with 3D hi-vision have been commercialized. 3D vision can provide a sense of depth, which is expected to improve the intuitiveness of surgery. On the other hand, in conventional laparoscopic surgery, a scopist has to hold the laparoscope and change its angle under verbal instruction from the operating surgeon. Therefore, the scopist needs a deep understanding of the surgical procedure and must have excellent dexterity. Camera shake may occur due to fatigue of the scopist, which may cause the surgeon to become nauseous, especially in 3D vision. Therefore, a laparoscope holder is an important and effective advancement for laparoscopic surgery.
K. Tanado and K. Kawashima
Laparoscope holders have been studied for many years, and some are commercially available. AESOP is a holder with a SCARA-type robotic arm with four degrees of freedom (4 DOFs) (1). Voice command was also added in the second version. Voice command has the advantage that the operator’s hands remain free throughout the operation. However, there is concern that voice recognition might fail due to environmental noise. EndoAssist is a camera holder with 3 DOFs driven by stepper motors (2). A foot switch is used for its operation. The arm moves with a constant velocity while the foot switch is on. The direction of movement is switched by a head gesture of the operator. The head gesture is detected by an infrared transmitter attached to the operator’s head and a receiver installed on the monitor. FreeHand, whose arm is very compact, is a successor to EndoAssist (3). It follows the operation method of EndoAssist. The receiver and transmitter must be properly arranged in order to avoid interruption of the infrared signal passing between them. ViKY is a compact scope holder that can be positioned on a patient (4); it has 3 DOFs and is operated by a foot switch or voice commands. Naviot is also a camera holder, and it achieves zoom in and out not with robotic arm movement but optically with a dedicated laparoscope (5). The holder is operated by a hand switch on the system, and it is suitable for safe operation. However,
the holder cannot be used with other laparoscopes. There is also a unique design using a combination of parallelogram mechanisms to achieve compactness and easy setup (6). These systems are operated by a small joystick mounted on the instrument. Operation methods (such as voice control or switches) in current laparoscopic holders seem inferior in that they are not intuitive because the operator must change his/her focus between operation of the camera and the surgical procedure. In this research, a laparoscope control system using a pneumatically driven robotic arm was developed. The developed system is operated by the user’s head movement to provide high intuitiveness and synchronization. We also confirm the tracking accuracy of the system experimentally. Moreover, the effectiveness of the system is demonstrated through in vivo experiments.
Concept of the holder system In the proposed laparoscope holder system, instead of a scopist holding the laparoscope, a pneumatically driven robotic arm holds the camera, and the arm is positioned by the operator’s head movements, as shown in Figure 1. The authors have previously developed a surgical robot system with a pneumatic drive called IBIS (8,9). The
Figure 1. Concept of the proposed system Copyright © 2014 John Wiley & Sons, Ltd.
Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
A Pneumatic laparoscope holder controlled by head movement
concept of the pneumatically driven robotic arm in IBIS is applied to the laparoscope holder system. All the robotic holders described in the Introduction use electrical motors for actuation. However, in this system, pneumatic actuators are used instead. This is because pneumatic actuators have many safety advantages such as low heat generation, compressibility, the ability to control the maximum force by regulating the supply pressure, ease of releasing the acting force by discharging compressed air in the actuator, and the ability to realize an arm that is both compact and lightweight. The robotic arm is controlled by head movement measured using two gyroscopes attached to the operator’s head and body. As shown in Figure 1, the view angles of the laparoscope for up and down, left and right, and rotation synchronously follow the operator’s head movement, and the camera’s zoom in and out synchronously follows the operator’s anteroposterior body movements. The operator’s movements can be measured with external sensors such as optical or magnetic 3D measurement devices. However, some markers must be attached to the operator, and a receiver must be installed nearby. The markers and receiver must be properly arranged in order to avoid interruption of the infrared signal or interference from other devices. Therefore, we use gyroscopes, which are internal sensors, in the proposed system. Rotation speed during head movements is directly detected by the 3axis gyroscope attached to the operator’s head. Zoom in and out can be measured from the translation velocity of the head movement. Velocity can be theoretically obtained by integrating data measured by an acceleration sensor attached to the operator’s head. However, it is difficult to obtain an accurate velocity from the acceleration sensor due to errors caused by gravity compensation, drift of the zero point, and sensor noise. Therefore, we focused on anteroposterior body movements of the operator. We estimated translation movement of the head from anteroposterior body movement based on a gyroscope attached to the operator’s body. The estimated value is used as a control signal for the zooming movement (in and out) of the robot. The laparoscope image can be displayed to a monitor or on a head mounted display (HMD). When the HMD is used as the monitor, the image is always kept in front of the operator’s eyes, even when the operator turns his/ her head. Therefore, using a HMD can provide highly intuitive operation and can avoid the need to consider the monitor layout. Moreover, when using a 3D laparoscope, a highly immersive stereoscopic image can be provided without crosstalk. HMDs have already been introduced in clinical practice, and their effectiveness has been reported (10). However, wearing a HMD for an extended period of time may cause fatigue in the user’s neck or eyes due to the weight and short focal length. The user’s view of his/ her surroundings is also limited to what is immediately Copyright © 2014 John Wiley & Sons, Ltd.
below. Users can choose a suitable display according to their situation or preference.
Materials and method Pneumatically driven robotic holder The developed pneumatically driven robotic arm that holds the laparoscope is shown in Figure 2. The robotic holder has 4 DOFs in total, consisting of 3 rotational DOFs around the inlet of the trocar cannula and 1 translational DOF along the insertion direction. Rotation around the scope axis (q4) allows the user to adjust the roll angle view. It is useful for both straight and angled scopes. When an angled scope is used, a coordinate transformation is applied to maintain the correspondence between the user’s head motion and the scope image. Parallel link mechanisms, as shown in Figure 3, provide a remote center of motion (RCM) (7) so that the pivot point at
Figure 2. Developed pneumatic holding arm
Figure 3. Structural drawing of pneumatic holding arm Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
K. Tanado and K. Kawashima
the trocar cannula is mechanically immovable without direct support. As shown in Figure 3, rotation of q1 and q4 is realized by transmitting the output of vane-type pneumatic motors with timing belts. Rotation in the vertical direction q2 is achieved by converting linear motion of the pneumatic cylinder to rotational motion using a slider-crank mechanism. The linear motion q3 is directly driven by the pneumatic cylinder, which controls the zoom in and out of the laparoscope. Increment-type encoders are used to measure the angles. Rotation angles q1, q2, and q4 are measured using rotary-type encoders, and q3 is measured using a wire-type linear encoder. Table 1 shows the motion ranges and maximum forces for each DOF when the supply pressure is 0.5 MPa (gauge). The angle q2 is defined as 0 degrees, at which the laparoscope becomes horizontal. A positive direction indicates that the tip of the laparoscope is pointing down. The state shown in Figure 3 is defined as the initial position. The motion ranges can easily be limited by implementing mechanical stoppers to the mid-stroke. The maximum force can easily be changed by regulating the supply pressure. The laparoscope is inserted from the rotation center of q4 via a dedicated adapter. The holder is designed so that it can easily be detached passively when an unexpected load is applied to the tip of the laparoscope. The robotic arm is 350 mm in length, 240 mm in height, and 80 mm in width in the initial position. A compact size is realized with the robotic arm, and its mass is only 0.98 kg.
Table 1. Working range and maximum torque of holding arm
q1 q2 q3 q4
Working range
Maximum torque
± 95 47° to 10° ± 75 mm ± 162°
1.39 Nm 2.52 Nm 39.3 N 0.107 Nm
System setup Figure 4 shows the configuration of the developed system. The system mainly consists of the pneumatically driven 4-DOF robotic arm shown above, two motion sensors, a foot switch, four pneumatic servo valves, eight pressure sensors, and electrical power supply devices. The motion sensor (ZMP, IMU-Z2) is equipped with a 3-axis gyroscope, a 3-axis geomagnetic sensor, and a 3-axis acceleration sensor. The resolution of the gyroscope is 3.8×103 (°/s), which is sufficiently small relative to human movement. It is experimentally confirmed that the zero drift of the gyroscope can be ignored. Therefore, the gyroscope is suitable for measuring human movement. Measured signals from the gyroscopes in the motion sensors are transmitted to the control PC via CAN communication. The control signals are sent from the PC to the servo valves through a digital-to-analogue converter. The foot switch is positioned at the feet of the operator. Head movement operation is activated only while the foot switch is pressed. Therefore, the operator can move his/her head freely when the foot switch is released.
Head tracking using gyroscopes In order to control the 4-DOF robotic arm with head movement, it is necessary to measure translational motion in the longitudinal direction as well as rotational speed of the head. Angular velocity of the head ωh can be measured by the gyroscope mounted on the operator’s head. Translational motion is measured by the gyroscope attached to the operator’s body as follows: Translational motion of the head is given by the speed of the neck, as shown in Figure 5. This speed is the cross product of the angular velocity of the hip ωw (obtained by the gyroscope) and the vector from the waist to the neck l in absolute coordinates.
Figure 4. System configuration Copyright © 2014 John Wiley & Sons, Ltd.
Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
A Pneumatic laparoscope holder controlled by head movement
geomagnetic sensor, we approximate them as zero in this paper because the actual angular values are small.
Control system The pneumatically driven 4-DOF robotic arm is controlled by the angular velocity ωh and translational velocity vh. Figure 6 shows a block diagram of the control system. To match the direction of the head coordinate system with that of the laparoscope coordinate system, the angular velocity at the head ωh is transformed into the reference angular velocity at the robotic arm ωref by ωref ¼ Rta ωh
(3)
Where Ra is a matrix indicating the attitude of the robot, which is given by the angular position of each joint. Ra ¼ Eiq1 Ejq2 Ekq4
(4)
Here, E denotes a rotation matrix. Next, the reference angular velocity ωref is transformed into the angular velocity of each joint. 2̇ 3 q1 6̇ 7 (5) 4 q2 5 ¼ J 1 ωref Figure 5. Measurement of translational velocity of head motion
q̇ 4
(1)
Here, J is a Jacobian matrix that maps the displacement at each joint to the attitude of the arm, which is given by J ¼ ex Eiq1 ey Eiq1 Ejq2 ez (6)
By multiplying the translation matrix R by the value obtained from the cross product, velocity vh in the head coordinates is calculated as follows:
The velocity of the Axis q3, which is the axis of the zoom in and out movement, is defined as the anteroposterior direction of vh obtained from Equation (2).
ωw l
vh ¼ Rðωw lÞ
(2)
The longitudinal direction of the vector vh that is in the anteroposterior direction of the head is used as a control signal for zooming in and out. Here, the inclination angles of the upper body and head are needed in order to obtain l and R. Although those values can be calculated by integrating the rotation speed or the output of the
q̇ 3 ¼ vh ey
(7)
The reference velocities shown above are integrated while the foot switch is on to give the reference position vector qref. When the foot switch is off, the reference velocities are set to zero. As shown in Figure 6, the reference torque τ ref to be generated at each joint of the robotic arm is given by a PD controller with feed-forward compensation of the inverse dynamics calculated from the reference
Figure 6. Block diagram of control system Copyright © 2014 John Wiley & Sons, Ltd.
Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
K. Tanado and K. Kawashima
position. Kpp and Kpd in Figure 6 denote the proportional and differential gain for the position, respectively. The reference torque is transformed into the reference force for each pneumatic actuator Fref by the Jacobian Ja, which maps the torque from the force. The reference force Fref is controlled
by a PID controller with a feedback force F calculated from the pressure difference in the actuator. The control signal is sent to the pneumatic servo valve, and the force is generated by charging compressed air into the actuator. This process is conducted in the driving force controller block shown in Figure 6. All calculations are performed in a period of 1 ms in the computer shown in Figure 4.
Results Evaluation of head tracking performance
Figure 7. Experimental results for velocity measurement of translational head motion
First, an experiment was conducted to evaluate the performance of the proposed measurement method. The translational velocity was measured by a 3D optical measurement system (Northern Digital Inc., Polaris Spectra) as a reference and by the proposed method, simultaneously.
Figure 8. Experimental results for trajectory tracking of robotic arm (a) q1 (c) q2 (b) q3 (d) q4 Copyright © 2014 John Wiley & Sons, Ltd.
Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
A Pneumatic laparoscope holder controlled by head movement
The user moved his head back and forth during the measurement. In the measurement with the 3D optical measurement system, a passive marker was mounted on the user’s head, and velocities were calculated by differentiating the output numerically. In the experiment, the length of the vector l in Equation (2) was set to 660 mm. Figure 7 shows the experimental results for the comparison of velocities between the two measurements. It can be seen that velocity measured with the gyroscope corresponds to the reference value obtained with the 3D optical measurement system. The proposed method is therefore proved to measure the motion as accurately as the 3D optical measurement device.
Figure 9. Tracking experiment
Evaluation of the robotic arm’s dynamic performance Next, the tracking performance of the pneumatically driven robotic arm was measured experimentally. Parameters such as friction and center of gravity, which are used to obtain the inverse dynamic model, were determined experimentally in advance and agreed well with the driving torque during free motion. Controller gains were determined by trial and error to produce good tracking performance. A laparoscope with a mass of 300 g (Shinko Optical Co., Ltd, HD-101D) was used in this experiment. The light cable was attached to the base arm so that its mass had little influence on performance. The minimum jerk trajectory is given as a reference position for each joint individually. The experimental results for trajectory tracking are shown in Figure 8. In the figure, q, ˙q, and q¨ denote the position, velocity, and acceleration of each joint, respectively. It is clear from this figure that all joints follow the reference trajectory well. The experimental results show that the robotic arm is capable of generating velocities of at least 1.0 (rad/s), 1.0 (rad/s), 150 (mm/s), and 4.0 (rad/s) for joints q1 to q4, respectively. Also, the accelerations achieved are 5.0 (rad/s2), 5.0 (rad/s2), 0.8 (m/s2), and 40.0 (rad/s2).
Evaluation of tracking capability for head movement The tracking performance of the developed holder system was next evaluated experimentally. As shown in Figure 9, an A4 size sheet of paper with four target points was placed approximately 100 mm from the tip of the laparoscope. In this experiment, a 2D-HD scope with a mass of 420 g (Olympus, LTF-S190-10) was used. An operator wearing a HMD operated the robotic arm by head movement to bring the target to the center of the field of view in the order shown in Figure 9. After two cycles, Copyright © 2014 John Wiley & Sons, Ltd.
Figure 10. Transition of the field of view. The operator turned from the top left target to the top right one (from 1 to 2), diagonally from the top right to the bottom left (from 2 to 3), from the bottom left to the bottom right (from 3 to 4), diagonally from the bottom right to the top left (from 6 to 5), zoomed in (from 5 to 6), zoomed out (from 6 to 7), rolled right (from 7 to 8), rolled left (from 8 to 9), and rolled right (from 9 to 10)
the operator zoomed in and out and then rolled the view to the left and right. The above operation was conducted as quickly as possible. The viewing area projected on the HMD changed as shown in Figure 10. Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
K. Tanado and K. Kawashima
Figure 11 shows the velocity response of each axis during operation. The red lines in the figure show the reference velocities of each joint calculated from the gyroscopes, and the blue lines show velocities measured by the encoders attached to the joints. It is clear from the experimental results that the robotic arm tracks reference velocities in each joint well.
Comparison with manual operation To verify the effectiveness of the developed system compared with manual operation by a scopist, an ex vivo experiment was conducted. A training box for laparoscopy was used, in which five numbered targets were placed as shown in Figure 12. A 2D-HD laparoscope was used, and the image was shown on the display placed in front of the operator. The operator instructed the scopist or operated the robotic arm such that the intended target was in the laparoscope’s field of view. The operator grasped a pin with the forceps in his/her right hand. When the target was in sight, the operator touched it with the pin. Then, the operator switched the pin from the right hand to the left hand. After positioning the laparoscope on the next target, the operator touched the target
Figure 12. Experimental setup for evaluation of the developed system
with the pin. This procedure was repeated five times per set, and the completion time was measured. Two surgeons that were unfamiliar with the developed system participated as the subject operators in this experiment. In the manual operation, one of the surgeons grasped the laparoscope. Ten trials were conducted for each kind of scope operation (20 times in total). Figure 13 shows a comparison of the results. No significant difference can be seen between the two operation methods. However, it is confirmed that the developed system allows the task to be completed slightly faster than with a scopist.
Discussion Tracking performance One feature of the developed system is that the laparoscope holding arm smoothly tracks the operator’s head and body movements. To allow highly intuitive operation,
Figure 11. Velocity responses of robotic arm to head movements during operation Copyright © 2014 John Wiley & Sons, Ltd.
Figure 13. Experimental results for performance comparison between the developed system and conventional manual operation Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
A Pneumatic laparoscope holder controlled by head movement
Conclusions
Figure 14. In vivo experiment
the robotic arm must smoothly track movements without any time delay. Figure 7 demonstrates that the proposed head tracking with the gyroscope has almost the same performance as the 3D optical measurement device. Also, as can be seen in Figure 11, maximum velocity for each axis during operation is 0.4 (rad/s), 0.3 (rad/s), 70 (mm/s), and 0.6 (rad/s). The achieved velocities shown in Figure 8 are sufficiently large compared with these values. Thus, the developed system has satisfactory dynamic performance for operating the laparoscope. Compared with manual operation, the task duration using the developed system was slightly shorter than with a scopist, even though both operators were well-trained surgeons. Therefore, it is demonstrated that the developed system provides the intended view quickly with the same or better performance than with a professional scopist. In the future, we would like to compare the performance of the system with other robotic camera holders more systematically, as referred to in some reports (11–13).
In this research, we developed a holder control system in which a laparoscope was held by a pneumatically driven robotic arm that followed the operator’s head movements. Head movements were measured with gyroscopes attached to the operator’s head and body. View angles of the laparoscope for up and down, left and right, and rotation synchronously followed the head rotations. The zoom in and out of the camera synchronously followed the anteroposterior head movements of the operator. We confirmed the tracking accuracy of the system through experiments. Moreover, the effectiveness of the system was demonstrated through in vivo experiments. For future work, we are improving the system to conform to the medical regulations in several countries. Clinical trials of the system need to be conducted for several surgical procedures to make quantitative and statistical evaluations.
Acknowledgements This project was supported by the Program for Creating Start-ups from Advanced Research and Technology by the Ministry of Education, Culture, Sports, Science and Technology in Japan.
Conflicts of interest The author have stated explicitly that there are no conflict of interest in connection with this article.
Funding In vivo experiment To demonstrate the effectiveness of the developed system, in vivo experiments were performed three times on pigs (gastric resection) by several surgeons. Figure 14 shows a photograph of the experiment. In the experiment, a small problem arose because of the space between the robotic arm and the trocar, but the system setup was completed in 10 min. The scope angle smoothly changed as the operator intended. All the surgeons who used the system thought highly of its smooth and intuitive operability. Especially in gastric resection, where the camera angle is often changed, the developed system is effective. Additionally, no lens cleaning was required during the operations. Copyright © 2014 John Wiley & Sons, Ltd.
No specific funding.
References 1. Sackier JM, Wang Y. Robotically assisted laparoscopic surgery: from concept to development. Surg Endosc 1994; 8: 63–66. 2. Kommu SS, Rimington P, Anderson C, Rané A. Initial experience with the EndoAssist camera-holding robot in laparoscopic urological surgery. J Robot Surg 2007; 1(2): 133–137. 3. Rané A, Kommu SS, Grange P, et al. Initial laboratory experience with the FreeHand camera holding robotic arm in laparoscopic urological surgery. J Endourol 2008; 22: 129. 4. Long JA, Cinquin P, Troccaz J, et al. Development of miniaturized light endoscope-holder robot for laparoscopic surgery. J Endourol 2007; 21(8): 911–914. 5. Tanoue K, Yasunaga T, Kobayashi E, et al. Laparoscopic cholecystectomy using a newly developed laparoscope Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
K. Tanado and K. Kawashima
6. 7. 8.
9.
manipulator for 10 patients with cholelithiasis. Surg Endosc 2006; 20(5): 753–756. Herman B, Dehez B, Duy KT, et al. Design and preliminary in vivo validation of a robotic laparoscope holder for minimally invasive surgery. Int J Med Robot Comput Assist Surg 2009; 5: 319–326. Taylor RH, Stoianovici D. Medical robotics in computerintegrated surgery. IEEE Trans Robotics Automat 2003; 19: 765–781. Tadano K, Kawashima K. Development of a master slave system with force-sensing abilities using pneumatic actuators for laparoscopic surgery. Adv Robot 2010; 24(12): 1763–1783. Tadano K, Kawashima K, Kojima K, Tanaka N. Development of a pneumatic surgical manipulator IBIS IV. J Robot Mechatron 2010; 22(2): 179–187.
Copyright © 2014 John Wiley & Sons, Ltd.
10. Kihara K, Fujii Y, Masuda H, et al. New three-dimensional headmounted display system, TMDU-S-3D system, for minimally invasive surgery application: procedures for gasless single-port radical nephrectomy. Int J Urol 2012; 19: 886–889. 11. Yavuz Y, Ystgaard B, Skogvoll E, Marvik R. A comparative experimental study evaluating the performance of surgical robots AESOP and Endosista. Surg Laparosc Endosc Percutan Techn 2000; 10(3): 163–167. 12. Nebot P, Jain Y, Haylett K, et al. Comparison of task performance of the camera-holder robots EndoAssist and AESOP. Surg Laparosc, Endosc Percutan Techn 2003; 13(5): 334–338. 13. Herman B, Olias Lopez A, Rasse C, Raucent B. Experimental comparison of kinematics and control interfaces for laparoscope positioners. In Proceedings of the 9th National Congress on Theoretical and Applied Mechanics. Brussels, Belgium, 2012.
Int J Med Robotics Comput Assist Surg (2014) DOI: 10.1002/rcs
