IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 4, APRIL 2014

1305

Smart Tissue Anastomosis Robot (STAR): A Vision-Guided Robotics System for Laparoscopic Suturing Simon Leonard, Kyle L. Wu, Yonjae Kim, Axel Krieger*, and Peter C.W. Kim

Abstract—This paper introduces the smart tissue anastomosis robot (STAR). Currently, the STAR is a proof-of-concept for a vision-guided robotic system featuring an actuated laparoscopic suturing tool capable of executing running sutures from imagebased commands. The STAR tool is designed around a commercially available laparoscopic suturing tool that is attached to a custom-made motor stage and the STAR supervisory control architecture that enables a surgeon to select and track incisions and the placement of stitches. The STAR supervisory-control interface provides two modes: A manual mode that enables a surgeon to specify the placement of each stitch and an automatic mode that automatically computes equally-spaced stitches based on an incision contour. Our experiments on planar phantoms demonstrate that the STAR in either mode is more accurate, up to four times more consistent and five times faster than surgeons using stateof-the-art robotic surgical system, four times faster than surgeons R  using manual Endo360◦ , and nine times faster than surgeons using manual laparoscopic tools. Index Terms—Actuated laparoscopic suturing tool, roboticassisted surgery, vision-guided laparoscopic suturing.

I. INTRODUCTION HANGING surgical paradigm from open to laparoscopic approach has created significant mechanical, ergonomic, and visual challenges both in the design of surgical technology and mechanics of surgical techniques. A simple technique in open surgery such as suturing of surgical wound became a time-consuming and inconsistent task depending on a surgeon’s experience in laparoscopic approach [1]. The replication of human dexterity and ergonomics by robotically assisted surgeries (RAS) has not clearly demonstrated to improve efficiency and effectiveness of surgical tasks correlating with improved functional and clinical outcome as compared to laparoscopic technology and techniques [2]–[4]. This paper introduces the smart tissue anastomosis robot (STAR). The STAR aims to improve the quality, consistency,

C

Manuscript received May 9, 2013; revised August 12, 2013 and November 4, 2013; accepted January 12, 2014. Date of publication January 23, 2014; date of current version March 17, 2014. Asterisk indicates corresponding author. S. Leonard,K. L. Wu, Y. Kim, and P. C. W. Kim are with the Sheikh Zayed Institute for Pediatric Surgical Innovation at the Children National Medical Center, Washington DC 20002 USA (e-mail: [email protected]; [email protected]; [email protected]; [email protected]). ∗ Axel Krieger is with the Sheikh Zayed Institute for Pediatric Surgical Innovation at the Children National Medical Center, Washington DC 20002 USA (e-mail: [email protected]). Digital Object Identifier 10.1109/TBME.2014.2302385

Fig. 1. (a) STAR hardware: A 7-DOF LWR mounted with an modified Endo360◦ . (b) The actuated needle of the STAR. (a) STAR system. (b) Circular needle of the Endo360◦ .

and the effectiveness of laparoscopic suturing through the development of a specifically-designed robotic system. The focus on laparoscopic suturing is motivated by the amount of time this procedure consumes, even when using state-of-the-art robotic assistance [5], by the desire to standardize and uniformize the quality of suturing outcome [6], [7] and to enable a higher adoption rate of minimally invasive techniques. The current components of the STAR [see Fig. 1(a)] are: 1) a robot arm equipped with an 2) actuated suturing tool and 3) a monocular colour camera to support vision-guidance and supervisory control. The STAR uses a seven degrees of freedom (DOF) light weight robot (LWR) from KUKA (KUKA AG, Augsburg Germany). The LWR is mounted with an actuated suturing tool that is based on the commercially available Endo360◦ from EndoEvolution (North Chelmsford, MA, USA). We replaced the Endo360◦ handle by two actuators: the first actuator drives a circular needle and the other actuator drives a pitch axis located 46 mm above the tip as illustrated in Fig. 1(b). The STAR aims to substitute human surgical suturing skills much like automation has replaced many other human skills on assembly lines. We combine specialized hardware and software to execute this specific task while a surgeon is able to use his/her knowledge and experience to determine the parameters of the task through a supervisory control interface [8]. This shift in paradigm contrasts with several RAS systems that augment or complement human perception [9] or motor skills [10]. The STAR enables a manual mode and an automatic mode. In

0018-9294 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications standards/publications/rights/index.html for more information.

1306

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 4, APRIL 2014

manual mode, the surgeon selects the placement of each stitch, whereas in automatic mode, he/she outlines the incision area and the STAR determines the placement of each stitch at a specified interval. Our main contributions are twofold; our first contribution consists of the development of an actuated laparoscopic suturing tool that is mounted on a robot manipulator. Second, we propose a vision-based supervisory control interface that enables a surgeon to select and track stitch placements in the camera images. In this paper, we demonstrate 1) suturing efficiency, accuracy, and consistency of STAR when compared to R  surgeons using a da Vinci Surgical System or manual laparoscopic tools for suturing planar phantoms; 2) suturing accuracy and efficiency of the STAR when stitch placement is computed automatically compared to when stitch placement is selected manually. II. MOTIVATION AND PREVIOUS WORK Suturing is an essential element of anatomic reconstruction during all surgical procedures. Each year, over 100 million major surgical procedures are performed in the United States, and it is conservatively estimated that over one million reconstructive anastomoses are performed in the United States alone for visceral surgery such as gastrointestinal, urologic, and gynecologic operations [11]. Recent transition of surgery from open to laparoscopic has clearly demonstrated the benefits of minimally invasive surgical (MIS) approach in reducing collateral tissue injury associated with open surgery [12], [13]. Despite the benefits of MIS, we identify three problems of MIS that we aim to address with the STAR. A. Motivation: Efficiency Time is an important aspect to the success of any surgical procedure and for the efficient management of operating rooms. Among one of the frequent findings of “laparoscopic versus open surgeries” in the literature is that laparoscopic procedures take significantly longer operative times [14], [15] and the range of operating times is also greater [16]. More specifically, suturing inefficiencies is a consideration in procedures such as open kidney transplants where anastomosis time ranges from 25 to 45 min and the risk of delayed graft function is 3.5 times higher for anastomosis lasting longer than 29 min [17]. For laparoscopic anastomoses, the duration of an anastomosis alone can range from 90 min for intestinal [18], [19] to 50 mins for aorta [20] to 30 min for vesicourethral [21]. A study demonstrated that using a manual suturing tool such as the EndoStitchTM reduces total surgical costs and time, but developing proficiency with such tools requires time and practice [22]. Although RAS is credited to reduce the operating times of certain procedures [23], the amount of time required to execute an anastomosis can range from 30 to 90 min for intestinal [18], [24] and 28 min for vascular [25]. B. Motivation: Consistency Although clear advantages of MIS and RAS approach over open surgery has been articulated, technical and clinical chal-

lenges associated with changes in tool design and procedural ergonomics of minimally invasive tools have not been addressed [26]. Intra and intersurgeon variations in procedural techniques, decision making, and experience further complicate any surgical task that requires precision, repetition, and maneuverability, such as suturing and anastomosis [27]–[29]. The validated metrics of anastomosis such as accuracy and biomechanical parameters like bursting strength of anastomosis are inferior and associated with significantly longer learning curve with laparoscopic and RAS than open approaches [30]–[33]. This challenge is magnified in pediatric surgery where surgery time with some MIS approaches is twice as long, associated with higher anastomotic complication rates than an open approach. In some instances, a moratorium has been imposed because the early outcomes of laparoscopic approach have been inferior to open surgery [34], [35]. Today, well-recognized clinical parameters of the complications of gastrointestinal anastomosis such as leakage or stricture remain unchanged following the shift from open to MIS [36]–[39]. A three to ten fold increase in mortality and adverse long-term impact on function and quality of life for affected patients associated with intestinal anastomosis complications are not addressed at all with both laparoscopic and RAS [38], [39].

C. Motivation: Adoption The conversion to and adoption of MIS approach has been slow and remains low in fields such as gastrointestinal surgery only about 30% to date [15], [40]–[43]. Both suturing and mechanical staplers are routinely used in open surgery for anastomosis [6], [7], but unchanged design and the lack of flexibility and dexterity of mechanical staplers have not facilitated the adoption of MIS approach [12], [13]. The use of next generation concept for anastomosis such as gluing is an embryonic stage of development to date [44], [45]. Although staplers have gained in popularity during the last two decades, they have not demonstrated a clear clinical advantage beside reducing operation time (15.3 min for esophagogastric anastomosis [6] and by 7.6 min for colorectal anastomosis [46]). Furthermore, intestinal anastomosis in children remains traditionally performed manually [47]. The slow adoption of laparoscopic surgeries is attributed to a number of critical challenges such as haptics, altered depth perception, nonstereoscopic 2-D vision, counterintuitive movement, eye-hand coordination required for visual-spatial skills, and small field of view and unnatural ergonomics [48]. The culminating clinical impact of these technical challenges has been significantly longer average lengths of surgery, much longer and steeper procedural learning curve, and variable adoption rates of and patient access to complex laparoscopic and RAS. Although RAS addresses some of these challenges by improving dexterity, visualization, and ergonomics, it has not translated into a significant acceptance outside of urology for prostatectomies [4]. The current clinical adoption rates for RAS in gynecologic and gastrointestinal surgery remain less than 10% and 3%, respectively [49].

LEONARD et al.: SMART TISSUE ANASTOMOSIS ROBOT (STAR): A VISION-GUIDED ROBOTICS SYSTEM FOR LAPAROSCOPIC SUTURING

1307

Thus, the critical technologic advances that would address the aforementioned motivations must include innovative smart tool concepts with a superior control interface. D. Previous RAS Suturing Systems The field of RAS includes a broad range of applications including orthopedics, neurology, urology, ocular, and general surgeries. In the case of suturing during MIS, one important challenge is to provide a dexterous workspace within a small volume under the constraints imposed by a tool insertion point. The master–slave paradigm is an efficient solution to this problem and has been adopted by several RAS systems such as the da Vinci surgical system [50], Raven [51], Telelap Alf-X [52], and MiroSurge [53]. These MIS robots provide surgeons with adequate dexterity, improved visualization, and better ergonomy. Although master–slave robots are able to enhance the skills of surgeons, they do not fully or partially replace them. On that front, interesting research has demonstrated the benefits of augmenting master–slave robots with the capability of actively monitoring and modifying surgical motions to assist with specific procedures. One example involves a system capable to segment suturing motions [54] and to synchronize one or several segments to a similar segment performed by an expert. A different approach consists of altering a surgical motion by using virtual fixtures to either guide the motion or prevent the robot from entering forbidden areas [55]. In the former case, the system is designed to guide the motion along a specific path within the workspace [56], [57] whereas, in the latter case, the system enables free motion while preventing the robot from entering specific areas [58], [59]. Another paradigm for RAS consists of shifting a greater portion of autonomy on the robot system, sensor, and controllers. In the area related to suturing, several efforts were made to develop laparoscopic tools for suturing and tying knots. In [60], the authors present a hand-held robot system able to tie a knot and suture in the confined laryngeal space. Other hand-held actuated suturing laparoscopic tools were presented in [61]–[64]. Likewise, some master–slave architectures include helpful features for suturing such as force feedback [65], [66], programmable modes [67], [68], and virtual fixture for tool alignment [56]. None of these systems, however, are able to close a cut without a surgeon in the loop. A pneumatic version of the EndoStitch mounted on a robot arm is reported in [69], but no suturing results are presented. Recently, a monocular camera with a single arm equipped with a standard laparoscopic needle holder demonstrated it could insert a needle through a few markers placed on a planar phantom [70]. Their results, however, do not demonstrate improvements over state of the art RAS or manual methods. In fact, the authors conclude that using a tool such as the Endo360◦ would address their problem of unsafe suture exhaustion and knot tying. Given the evolving surgical approach and current state of laparoscopic and RAS instruments, there is a clear need to demonstrate superior attributes and clinical advantages of laparoscopic and RAS tools. Our hypothesis is that a combination of a new actuated end effector that simplifies suturing tasks combined with

Fig. 2. Computer-aided design of the STAR tool. The STAR tool uses the shaft, pitch, and needle of an Endo360◦ . We designed the motor stage to actuate the pitch and needle.

vision-guided robotics that enables supervised control is more efficient and consistent than manual laparoscopic or master– slave RAS and will enable a greater adoption of laparoscopic surgeries. The STAR proposes a supervised control approach that shifts the burden of planning and executing the motion on the robotic system, while maintaining the role of the surgeon to decide where to place each suture. Although we acknowledge that the STAR is not yet able to perform anastomoses in a surgical environment, we demonstrate that, within its current limitations, it is significantly more efficient and consistent than surgeons using either manual tools or master–slave RAS. III. SMART TISSUE ANASTOMOSIS ROBOT The STAR system integrates robot hardware, force and image sensing, and processing. In this paper, our study of automated suturing is limited to planar surfaces. Although this is a strong assumption for clinical cases, several other RAS systems have demonstrated their benefits by experimenting on planar phantoms [56], [62], [70]–[72]. Further discussions about the other assumptions used in our experiments and proposed solutions to address them are postponed to Sections IV and V. In this section, we describe the design of the STAR system and its hardware and software considerations. For the sake of clarity, we define a running suture, often referred as suture in the text, as the task of closing an incision by using a single suture thread. A suture involves several stitches and each stitch bites both sides of an incision to close. A. STAR Suturing Tool We designed and built a suturing tool featuring an actuated pitch and needle by modifying a commercially available manual suture tool Endo360◦ . Fig. 2 shows a computer aided design drawing of the automated suturing tool design. The commercially available manual Endo360◦ suture tool is able to drive a disposable suture needle with attached 3–0 suture along a circular path, enabling single handed laparoscopic suturing [see Fig.1(b)]. The manual handle of the Endo360◦ was removed and the tool head, tool shaft, and cables to actuate needle drive, and push–pull bar to actuate pitch were then

1308

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 4, APRIL 2014

connected to a custom designed motor stage. The tool head and shaft can be detached from the motor stage for cleaning, but this procedure requires tools and several minutes. Two dc brush motors (Maxon Motors, Sachseln, Switzerland) drive the pitch and needle through two bevel gears activating a cam drive for pitch actuation and a bidirectional cable drive with pulley for needle actuation. Two turnbuckles allow for adjusting the tension of the cables for the needle actuation. Epos 2 (Maxon Motors, Sachseln, Switzerland) controllers are used for motor control of the tool. The suturing tool is mounted on a 7-DOF LWR KUKA robot through a 6-DOF force sensor (ATI Industrial Automation, Apex, NC, USA).

B. Supervisory Control for Image-Based Suturing Whereas several master–slave systems aim to replicate the motion of a surgeon or share the control of a task, the STAR is designed around the supervisory control paradigm which decouples the commands of the surgeon from the implementation of the robot motion. The supervisory control interface displays intraoperative images of the incision and the surgeon is limited to use his knowledge to either select stitch placements individually or to outline the incision. In manual mode, the surgeon selects the placement of each stitch by manually clicking at the desired position in the image. In the automatic mode, the surgeon manually outlines the contour of the incision in the image and the STAR automatically computes the 3-D position of each stitch at a regular interval. Given the interaction with intraoperative images, we outline the geometric model used to convert image-based commands to the robot space. 1) Geometry of Planar Suturing: The STAR vision system is currently limited to a calibrated, static, color, and monocular camera. This limitation restricts the geometry of incisions that the STAR can suture because of depth ambiguities. The experiments we present in this paper focuses on planar incisions. This assumption enables the STAR to transform image coordinates to 3-D coordinates with a homography [73]. Thus, the homography between the incision plane and the image plane must be computed prior to transforming image coordinates to 3-D points in the camera frame. Also, the transformation between the camera and the robot must be known to transform camera coordinates in the robot coordinate frame as described in Fig. 3. Since the incision is assumed to be planar, the mapping between the image coordinates of a stitch and its 3-D coordinates does not constrain the origin of the incision plane and it does not constrain the orientation of the plane about its normal axis. Thus, the orientation between the incision plane and the camera plane only requires rotations about X and Y axes as illustrated in Fig. 3. The selection of each stitch placement defines the homogeneous coordinates of a 2-D image point p = [ x y 1 ]T which is related to the coordinates of a 3-D point I P = [ I X I Y 0 ]T on the incision plane by p ∼ H I P, where H is a 3 × 3 homography that is defined up to scale.

Fig. 3. Schematic drawing of coordinate frames. X axes are in red, Y axes are in green, and Z axes are in blue. R E C represents the rigid transformation between the camera and the robot. The plane normal n corresponds to the Z axis of the plane and d to the distance between the camera and the plane.

Let K define a camera projection matrix ⎤ ⎡ fx 0 cx ⎥ ⎢ K = ⎣ 0 fy cy ⎦ 0

0

(1)

1

where fx = sx f and fy = sy f are the scaled focal lengths and cx and cy are the coordinates of the optical center and let  r1 r2 r3 t C (2) EI = 0 0 0 1 be the rigid transformation between the incision plane and the camera frame as illustrated in Fig. 3, where r1 , r2 , and r3 are the columns of a rotation matrix and t is the translation between the coordinate frames. A point on the incision plane I P is projected to the image coordinates p according to ⎡I ⎤ X ⎢I ⎥ p ∼ H⎣ Y ⎦ (3) 1 where H is defined up to a scale by H = K[ r1

r2

t ].

(4)

If H is nonsingular, the coordinates I X and I Y are ⎡ ⎤ ⎡I ⎤ X x ⎢ ⎥ ⎥ ⎢ w⎣ I Y ⎦ = H −1 ⎣ y ⎦ 1 1

(5)

where w is the homogeneous scale factor. Furthermore, a point in the image is transformed to a 3-D point in the coordinate system of the camera by C

P = C EI [ I X

I

Y

0

1 ]T .

(6)

Finally, we define the incision plane normal by n and the distance between the origin of the camera frame and the incision plane by d. Substituting (4), (5) in (6) (taking care of normalizing the homogeneous coordinates), we obtain that the 3-D camera coordinates of an image point p on the incision plane are given

LEONARD et al.: SMART TISSUE ANASTOMOSIS ROBOT (STAR): A VISION-GUIDED ROBOTICS SYSTEM FOR LAPAROSCOPIC SUTURING

1309

by ⎡

⎤ fy (cx − x) d ⎢ ⎥ C f (c − y) ⎦. P= T ⎣ x y T n [ fy (x − cx ) fx (y − cy ) fx fy ] −fx fy (7) Given that we constrain the orientation of the plane with respect to the camera, we note that n = [sin(α) − sin(α) cos(β) cos(α) cos(β)]T , where α and β are the rotation angles about X and Y , respectively. C. Tracking of Incision and Stitches To automate suturing the STAR is capable of operating in two modes. The manual mode enables a surgeon to individually select the placement of each stitch. The automatic mode enables a surgeon to outline the contour of an incision and to evenly distribute the stitches along the incision. The distance between stitches is specified by a single parameter that either determines the number of stitches to close the incision or the desired distance between stitches. In our previous work [74], we presented results for the manual mode where each placement was selected by a surgeon and tracked with a monocular camera by using appearance tracking based on Kanade–Lucas–Tomasi algorithm [75]. In this paper, we further improve the supervisory control interface by shifting the interaction from selecting stitches to selecting the contour of the incision. Once the contour of the incision is determined in the image, we transform the resulting 2-D contour to the 3-D camera frame according to (7) and distribute the stitches evenly. Furthermore, given that appearance based tracking is unreliable for surfaces exhibiting little texture in an environment exhibiting variations of illumination, deformations and occlusions, we implemented an efficient contour tracking method based on the multimodal tracking algorithm [76]. 1) Incision Selection: The selection of the incision and its contour is based on color segmentation. The method is analogous to the GIMP “Fuzzy Select” or Adobe Photoshop (San Jose, CA, USA) “Magic Wand”. The incision is selected by using a mouse and an image of the incision. First, the user clicks on a pixel within the incision and then, while holding the mouse button, moves the mouse within the boundary of the incision. During the selection, the surgeon samples a set of RGB pixels S with average RGB color μ = [ μR μG μB ]T and RGB covariance matrix Σ. Let F be a set of foreground pixels representing the inside of the incision, then an image of foreground pixel candidates p is defined by √ ⎧ R(p) − μR  < ΣR R ⎪ ⎪ ⎪ ⎨ 1 if G(p) − μ  < √Σ G GG √ (8) C(p) = ⎪ B(p) − μ  < Σ B BB ⎪ ⎪ ⎩ 0 otherwise where R(), G(), and B() represents the RGB channels of a pixel. Then, p ∈ F if it is connected to any pixels in S . Once F is determined, its contour is used to determine the distribution of stitches within the incision.

Fig. 4. Automated placement of stitches for a segmented incision. The outline of the incision is tracked and illustrated by the blue contour. Nine placements are computed according to the algorithm presented in Section III-C2 and illustrated by green circles.

2) Distribution of Stitches: The contour of F defines a polygon in the image space which is used to initialize the placement of each stitch. The first task is to transform the 2-D incision contour to 3-D coordinates using (6). From the 3-D coordinates, the straight skeleton of the polygon is extracted [77]. The role of the skeleton is to extract the topological structure of the incision. Given an undirected graph representation of the skeleton and, assuming a linear incision, we look for the longest path between any two vertices in the graph to represent the skeleton of the incision. This problem is also known as computing the diameter of a graph and is evaluated by computing all the shortest paths using Floyd–Warshall algorithm [78] which runs in O(V 3 ), where V is the set of vertices of the skeleton, and then searching for the longest shortest path. Using the resulting path, the stitches are evenly spaced by linearly interpolating the coordinates along the path. A result of our algorithm computing nine stitch placements on a training pad is illustrated in Fig. 4. 3) Incision Tracking: To track in real-time a contour under illumination variations and deformations, we adapt the algorithm presented in [76]. The problem is formulated as determining if a pixel belongs to the foreground or background. This is determined by combining three observations and evaluating if each pixel should be assigned to the background or the foreground. As in [76], we use a 2-D oriented bounding box (OBB), the color signature of the target and the contour of the target as observations. Instead of formulating the problem as a sequence of Bayesian filters and tracking the distributions by using particle filters and extended Kalman filters, we propose to compute the foreground response f at time t of a pixel p to the current and previous observations. The response ft (p) is based on distance transforms according to ⎡

DOBB (p)



⎢ ⎥ w3 ]⎣ DRGB (p) ⎦ + (1 − λ)ft−1 (p) DEDG (p) (9) where DOBB (p), DRGB (p), and DEDG (p) are the distances between the pixel p and the closest pixel on the boundary of the OBB, color segment, and contour segments, respectively, wi is a weight assigned to each of the three distances and 0 < λ < 1 is a scalar that determines the weight of the previous observations at time t − 1. ft (p) = λ[ w1

w2

1310

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 4, APRIL 2014

Fig. 7. Contour tracking control points and search normals. Note that the sharp extremities of the incision requires a dense distribution of control points.

Fig. 5. Example of distance transforms for each model and resulting contour. (a) OBB distance transform (D O B B ). (b) RGB distance transform (D R G B ). (c) Contour distance transform (D E D G ). Resulting contour.

Fig. 6. Tracking sequence with illumination variations, nonrigid deformations and thread occlusion. (a) Nonrigid deformation and illumination variation. (b) Thread occlusions.

The response ft (p) is locally averaged at each p and then compared to a dynamic threshold that is determined from the distribution of ft−1 (p) such that p ∈ F if |ft (p) − μt−1 | < σt−1 , where μt−1 is the average distance of all foreground pixels at time t − 1 and σt−1 is the standard deviation. Examples for the three distance transforms and the resulting contours are presented in Fig. 5 and the robustness of the tracking algorithm is illustrated by the snapshots in Fig. 6. a) Oriented bounding box: A 2-D OBB is used to bound a coarse area defined by the foreground. Its purpose is to impose a large penality on outlier pixels that are observed at significant distances from the box. The OBB is also used to speed up image processing by defining a region of interest around the target. The parameters of the OBB are defined by o = [ cx cy sx sy θ ]T , where cx and cy are the center of the box, sx and sy are its width and height and θ its orientation. Given a set of foreground pixels F , the OBB is determined by first computing the convex hull of the contour and then estimating the eigenvectors to determine its orientation, width, and height as described in the 3-D case [79]. b) Color Tracking: Given a set of foreground pixels F at time t, a color model c = [μR μG μB σR σG σB ]T , where μ and σ represent the mean and standard deviation of the color distribution. Although several color tracking algorithms have

been proposed for tracking in the past [80], [81], their benefits do not outweigh their computational costs for real-time tracking (30 frames per seconds) of large areas. The STAR uses the color model in (8) to obtain a segmented image from which only blobs that are within the OBB are selected. c) Contour Tracking: Given an initial contour, contour tracking searches for edge displacements at regular intervals along the contour. Our approach is based on classic active contour tracking found elsewhere in the literature [82], [83]. Given a closed contour C defined by a sequence of line segments, we distribute uniformly control points ei along C . The spacing between control points is determined based on the maximum curvature of the contour to ensure that sharp areas of the contours, such as the extremities of an incision, will be covered by at least one control point as illustrated in Fig. 7. This is a key consideration for our application since otherwise the contour does not “attach” itself to the extremities and tends to shrink. At each control point ei , the normal direction of the curve is locally estimated and used to search for the nearest edge feature to ei . D. Motion Regardless of whether STAR operates in manual or automatic mode, the motion of the robot follows the same workflow. The motion sequence consists of executing one knot tying, followed by a sequence of stitches. Initially, the system plans to pull a sufficient amount of thread to execute the suture based on the length of the incision, determined from the skeleton (see Section III-C2) and the number of stitches. When operating in automatic mode, the STAR is able to compute the width of the incision at each placement and determine the bite position on either side as illustrated in Fig. 8. The width of the incision cannot be computed in manual mode and must be specified manually. The motion of the STAR is constrained by a virtual remote center of motion (RCM). Given that the STAR disposes of 8-DOF, the unconstrained nature of the suturing task in Cartesian space (6-DOF), and the extra 2-D constraints imposed by the RCM, the motion of the STAR is determined by numerically computing the solution to the minimum norm problem of a system with eight equations and eight variables. The STAR only disposes of 2-DOF between the RCM and the tool control point (TCP): the last joint of the LWR and Endo360◦ articulation as

LEONARD et al.: SMART TISSUE ANASTOMOSIS ROBOT (STAR): A VISION-GUIDED ROBOTICS SYSTEM FOR LAPAROSCOPIC SUTURING

1311

Fig. 8. In automatic mode, the STAR uses the measured width of the incision at each placement, displayed by the yellow line, to determine the bite position on both sides of the incision.

Fig. 10. Position of the needle and thread during looping around the thread. The circular needle bites around the thread near its entry point into the tissue. The thread must clear the surface beneath to prevent the needle from penetrating other tissues.

Fig. 9. Nonholonomic constraint caused by a RCM. (a) Side view. (b) Front view. The orientation of the TCP cannot be oriented about its Y axis.

illustrated in Fig. 9. Therefore, the TCP can be positioned at an arbitrary XYZ coordinates beneath the RCM, but only rotations about the shaft of the tool and about the pitch axis of the Endo360 are possible [see Fig. 9(b)]. This imposes a nonholonomic constraint on the orientation of the TCP which cannot be oriented about its Y axis and this must be accounted for when computing the STAR trajectories. 1) Knot Tying: At the first stitch placement, the STAR ties a double overhand knot around one edge of the incision. The first step of knot tying consists of biting one side of the incision [see Fig. 11(a)], or both sides if they are sufficiently close, and pulling a sufficient amount of thread to execute the suture. The amount of thread L required for the suture is approximated by  (10) L > 2T N + (N − 1) 2T 2 + d2 where T is the thickness of the tissue, N is the number of stitches, and d is the distance between consecutive stitches. In (10), the first term accounts for the amount of thread used for N stitches composed of two lumens, each of thickness T . The second term accounts for the (N − 1) diagonals between stitches. The first loop of the knot is executed by moving the TCP where the thread enters the tissue and biting around the thread, as illustrated in Figs. 10 and 11(b), followed by tensioning the first loop during which a forcep is used to hold the loop down [see Fig. 11(c)]. The second loop is executed in a similar way as illustrated in Fig. 11(d) to (e). In order to bite around the thread, the STAR assumes that the thread enters the tissue at a position and orientation used for the initial bite of the tissue (see Fig. 10). In automatic mode, this position and orientation is determined from the width of the incision at the first placement and from the thickness T of the tissue. The thread must also be above any

Fig. 11. STAR workflow. The steps for knot tying are from (a) to (e). For each stitch, the STAR repeats step (g) and (h). (a) Bite the first lumen. (b) First loop. (c) Tension of first loop (notice the forcep to hold down the loop). (d) Second loop. (e) Tension of second loop. (f) Bite the second lumen (followed by tension). (g) Bite the first lumen. (h) Bite the second lumen (followed by tension).

surface near its entry point to enable the needle to loop around the thread without penetrating tissues beneath the thread. To avoid this situation, we manually lift the thread near its entry point with a forcep to ensure it clears the surface. Furthermore, when STAR tightens up a loop, the loop must be “pinched” with a forcep near the entry point of the thread to prevent the loop from sliding and to force the loop to snug the tissue. 2) Suture: After tying the knot, the STAR runs the sutures by moving to all placements on both sides of the incision as illustrated in Fig. 11(g) and (h). If the tissue is sufficiently thin and the incision sufficiently narrow, it is possible for the STAR to align the Endo360◦ and bite both sides of the incision simultaneously. Given that the width of the jaw of the Endo360◦

1312

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 4, APRIL 2014

TABLE I STAR EFFICIENCY: SUTURING EXECUTION TIME

only measures 5 mm, the conditions for this to succeed are somewhat limited. In our experiments, we use the more common scenario where the STAR bites each side of the incision separately, which requires the STAR to move a small amount between biting each side [see Fig. 11(g) and (h)]. In automatic mode, this small motion from one side of the incision to the other is computed by using the width of the incision. After biting the second side, the STAR pulls on the thread to tension. We use a force/torque sensor to measure a sufficient amount of tension in the thread to avoid tearing the tissue. We determined that the STAR performed well in practice when using 4 N as a threshold. Although this amount of force depends on the type of tissue, such knowledge can be provided by a surgeon before a procedure. To decrease the time for tensioning, the STAR determines when the thread should tighten up by keeping track of the length of thread available by using (10). If a sufficient amount of thread is loose, the STAR pulls at a higher velocity but reduces its velocity otherwise to measure more accurate forces.

IV. EXPERIMENTS In our previous work [74], we compared the STAR in manual mode to surgeons using manual laparoscopic tools, a da Vinci surgical system and a standard Endo360◦ for a suturing task consisting of placing one knot and nine stitches on a training pad. We demonstrated that the STAR executed the task faster and the sutures were more consistent than the other methods. In this paper, we extend the experiments of [74] by using the automatic mode where the placements are computed automatically and we compare the consistency of the results to those in our previous study. We also evaluate the accuracy of the automatic mode for the task of closing an incision of a known length with a given number of stitches by measuring how evenly distributed the STAR computes the placements of the stitches and how evenly distributed the STAR places them. Our experiment consists of executing a running suture composed of one knot followed by nine stitches on a training pad resting on a turntable. We selected a curved incision with a measured length of 55 mm. After the fourth stitch, the robot moved the tool outside of the camera field of view and the turntable was activated and deactivated with a manual ON/OFF switch such that the pad rotated between 45◦ to 90◦ to mimic an intraoperative motion of the tissue. During the rotation, the STAR visually tracked the incision and resumed the suturing once the rotation stopped. Given that our experiments required nine stitches, the STAR creates ten equally spaced segments as illustrated in Fig. 4. From the measured incision length of 55 mm, we conclude that the STAR should compute and execute stitches that are 5.5 mm appart.

Fig. 12. mode.

Results of suturing with the STAR. (a) Automatic mode. (b) Manual

The STAR executed four sutures in automatic mode and four other sutures in manual mode. In manual mode, the stitches were selected from nine consecutive mouse clicks in the image, whereas the algorithms presented in Section III-C were used to compute and track the placements in automatic mode. Results from our experiments are presented in Tables I through V. As a comparison basis, we also included relevant results from our previous study [74], where four trained surgeons (n = 4) used a da Vinci surgical system to perform the same task. Also, four experienced MIS surgeons performed the same task using manual laparoscopic tools with a Fundamentals of Laparoscopic Surgery Trainer System (VTi medical, Waltham, MA, USA). Then, these four surgeons also executed the task by using a manual Endo360◦ . All experiments used RCM constraints. The results presented in [74] were obtained by performing the same task (one knot and nine stitches with a turntable rotation after the fourth stitch) on the same phantom and by using the measurement methodology presented throughout this section. Throughout the experiments, we demonstrate the statistical significance of our results by reporting the p-values. The p-values are computed by a two-sided Wilcoxon rank sum nonparametric test (MATLAB) that determines if two populations are the same1 . Sample images for the automated and manual results are illustrated in Fig. 12(a) and (b), respectively. A. STAR Efficiency We measure the efficiency of the STAR by the time used to complete the suturing task. The STAR was programmed to execute all suturing tasks in one minute regardless of the mode used. This duration was selected based on the velocity limits of the robot, the time to drive the needle twenty times2 , and

1 Several p-values in our experiments are p = 0.0286. This represents the smallest p-value that Wilcoxon rank sum test can give for n = 4. 2 To drive the Endo360◦ needle, a full 360◦ requires two actuations of the needle mechanism for a total measured time of about 0.75 s. Thus, about 20 s of the total 60 s of suturing time is used to only actuate the needle.

LEONARD et al.: SMART TISSUE ANASTOMOSIS ROBOT (STAR): A VISION-GUIDED ROBOTICS SYSTEM FOR LAPAROSCOPIC SUTURING

1313

TABLE II STAR ACCURACY AND CONSISTENCY: DISTANCES BETWEEN SUTURES COMMANDS

our experience for manually keeping dangling thread out of the suturing path3 . The execution times of the suturing tasks are presented in Table I. As explained in Section III-D2, suture tightness involves a small amount of force sensing at a reduced velocity and that motion segment is mainly responsible for the execution time greater than one minute. In manual mode, the average time to select nine placements was 9.8 s, which is slightly longer than one mouse click per second with a standard deviation of 0.88 s, while the average time to select the placements in automatic mode was 3.2 s with a standard deviation of 0.52 s. Thus, efficiency of the STAR in either mode represents a five to nine folds improvements over the da Vinci and the manual tools. The STAR in automatic mode was also determined to be more efficient than the STAR in manual mode (p = 0.0286) and this result is largely imputable to the selection method. The STAR was nine times faster than minimally invasive surgeons and four times faster than surgeons using an Endo360◦ . B. STAR Accuracy and Consistency One important consideration in surgical suturing is the relationship between the distance between stitches and the distance between a stitch and the edge of the incision (bite size). For example, the Jenkin’s rule states that the length of a suture should be four times the length of an incision with sutures 1-cm apart and with 1-cm bites. Although the STAR does not implement Jenkin’s rule, it aims to provide greater accuracy and consistency for intrasuture spacing. We propose to evaluate the accuracy and consistency by measuring: 1) the commanded distances between stitches specified by supervisory control commands in manual and automatic modes; 2) the measured distances between stitches on the pad; 3) the measured bite sizes; and 4) the measured suture tightness. These measurements have been reported in the literature to affect healing complications such as breaking and infections [84]–[86]. 1) Accuracy and Consistency of Suture Commands: The first evaluation of the suturing accuracy of the STAR stems from measuring the average command distance between consecutive stitches. In manual mode, these commands are derived directly from mouse clicks and reflect on the accuracy of the surgeon to manually select equally distant points along a curve. Results from the placement commands for both modes are presented in Table II.

3 As mentioned in Section III-C, the STAR requires a small amount of manual thread management during the knot tying and the suturing. We believe that this level of assistance is comparable to the one used by surgeons during similar procedures within the contexts of open, laparoscopic, or RAS.

Fig. 13.

Points and labels used to measure performances.

Given the measured incision length of 55 mm, the distance between stitches should be 5.5 mm such that the average error for automatic and manual modes are 0.144 and 0.29 mm, respectively. More importantly, we note that the consistency of the automatic mode is about 33% higher than the manual mode. The accuracy and consistency could be improved in manual mode by selecting placements more carefully, but this would likely increase the selection time which is already three times longer. 2) Accuracy and Consistency of Suture Execution: The previous commands are then converted to paths and trajectories to execute the knot and the stitches at the desired positions. We measured the averagedistances between  stitches on 1 ( 9i=2 li − li−1 2 + 9i=2 ui − the pad according to d¯ = 16 ui−1 2 ), where li , and ui are illustrated in Fig. 13. Results for the manual and automatic mode are presented in Table III. We note that although the average distance between stitches is close to the average commanded distances reported in Table II, the standard deviation almost doubles. As a comparison, in our previous study [74], surgeons were asked to execute the same task (one knot and nine stitches) over the same incision and their performance is also reported in Table III. Since surgeons were not explicitly instructed to close the entire incision, we cannot compare the average distance between stitches and the mean distances are reported as not applicable in Table III. We can, however, compare the standard deviation between stitches which is roughly twice as much as the STAR in automatic mode. Furthermore, the STAR in automatic mode was on average slightly more consistent than the STAR in manual mode to distribute the stitches evenly (p = 0.0286). In either mode, however, the STAR was significantly more consistent than the surgeons. 3) STAR Consistency (Stitch Bite Size): Another measurement to assess the quality of a suture is the bite size, which is determined by the distance between the stitch and the edge of the incision. As aforementioned in Section III-D2, due to the Endo360◦ 5-mm jaw size, the STAR must bite each side of the incision separetly. In automatic mode, the amount of displacement between the edge of the incision and the bite position is determined by the width of the incision (see Fig. 8) to which 2.5 mm is added to compensate for the half width of the Endo360◦ jaw. In manual mode, the

1314

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 4, APRIL 2014

TABLE III STAR ACCURACY AND CONSISTENCY: DISTANCES BETWEEN STITCHES

TABLE IV STAR CONSISTENCY: BITE SIZES

TABLE V STAR ACCURACY AND CONSISTENCY: STITCHES TIGHTNESS

magnitude of the displacement from the edge must be determined manually. Based on Fig. 13, we measured  this bite 1 ( 9i=1 li − mi 2 + 9i=1 ui − distance according to ¯b = 18 mi 2 )). Results are reported in Table IV and we compare them to results of surgeons using a da Vinci surgical system or manual tools. We observe that the STAR in automatic mode was more consistent than the STAR in manual mode and surgeons. We note, however, that the trainingpads are not flat and the incision we selected bulges about 10-mm high by 5-mm wide, which could fit in the 5-mm jaw of the Endo360◦ , larger bites might not be possible due to the relatively small jaw of the Endo360◦ . 4) STAR Accuracy and Consistency (Stitches Tightness): The STAR was instructed to pull on the thread up to measured force of 4 N. This force magnitude was determined experimentally on various phantoms and tissues. We determine the tightness of the stitches by measuring the thread displacement by pulling with a small force on each stitch. We used a pull spring scale and slid its hook under each stitch (points mi in Fig. 13). We pulled on the scale to a measured force of 0.2 N and then measured the gap, if any, between the lifted thread and the tissue. Table V suggests that the STAR performs equally well in either mode and the average deflection is smaller than the surgeons by an order of magnitude.

V. DISCUSSION AND FUTURE WORK Our experiments demonstrate that the STAR outperforms experienced surgeons for executing a laparoscopic suturing task on a planar phantom. The STAR decreases the execution time by about five folds over trained surgeons using a state of the art master–slave surgical robot and about nine folds over experts using manual laparoscopic tools. Likewise, the STAR demonstrated submillimeter relative accuracy for the same suturing

task and demonstrated more consistency in all aspects. Although we acknowledge that our experiments did not involve a surgical environment, the same environment applied to surgeons who performed the experiments. Thus, within its limitations, the STAR was more efficient and more consistent than human surgeons. We believe that the gains demonstrated by the STAR can extrapolate to significant improvements for more challenging tasks such as end-to-end laparoscopic anastomosis in surgical environments. Such procedures are known to require on average 45 to 60 min for manual laparoscopy [20], [87]. Thus, a five to nine folds improvement for anastomoses could result in a significant reduction of surgical time and complications. Despite the positive results of the STAR, the scope of our experiments demonstrates its current limitations. Perhaps, the most obvious one is that the STAR is currently limited to execute suture on planar phantoms. This limitation is a direct result of the imaging used in the current version of the STAR and without 3-D sensing, this limitation cannot be overcome. We note, however, that several suturing systems that are reported in the literature also suffer from the same planar limitation or have presented results that are limited to planar phantoms similar to ours [56], [60], [62], [70]–[72], [88]. Other suturing systems have used one or a combination of 3-D sensors such as stereo cameras [89], time of flight cameras, and structured light scanning [90]. We are currently addressing the problem of 3-D sensing by using a light field camera and we expect that these cameras will enable the STAR to sense 3-D point clouds and execute 3-D suturing in the near future. Light field cameras are able to provide fully focused XYZ–RGB point clouds with several frames per seconds. Furthermore, this technology will enhance the supervisory control interface by enabling the surgeon to interact with 3-D intraoperative data as opposed to the current 2-D images. Another advantage of using this technology is that it is also compatible for laparoscopes.

LEONARD et al.: SMART TISSUE ANASTOMOSIS ROBOT (STAR): A VISION-GUIDED ROBOTICS SYSTEM FOR LAPAROSCOPIC SUTURING

The second main limitation of the STAR relates to tracking the incision under realistic surgical conditions. These include difficult light conditions, occlusions, fast nonrigid deformations, irregular incisions, and patient movements. Our proposed solution to address these problems in our related work consists of marking the incision contour, or part thereof, with a fluorescent dye that can be detected by an infrared camera and easily segmented by our tracking system [91]. The marking is executed by the surgeon by using a small device and will address the problem of irregular incisions. Our current experiments in that area involve indocyanine green (ICG) with near infrared cameras and light sources. Another advantage of this approach is the relatively good tissue penetration of near infrared light (NIR) light (5–10mm) which addresses in part the problem of occlusions [92]. We are currently experimenting with various gluing agents and solvents to make ICG penetrate and stick to the edges of lumens. This will enable stereo NIR cameras to segment and track occluded edges of the lumens and provide estimates of 3-D coordinates. Thus, by combining NIR cameras with a light field camera, we expect to address the remaining sensing challenges related to automated suturing. Another assumption of the STAR involves the force used to tighten the stitches. In our experiments, the STAR was instructed to pull on the thread until it measured a 4N force. This value was determined experimentally with various phantoms by pulling on the thread and measuring the magnitude of the rupturing force. The tear strength varies between tissues and there is no obvious method to determine the amount of pulling force a tissue can sustain other than building a lookup table for various tissues. But even in these circumstances, damages caused to the tissue during the surgical intervention can weaken its structure and affect its resistance. Another possibility is to close a control loop around NIR segmented images and pulling on the thread until the STAR detects that points on opposite sides of the incision are touching each other. Furthermore, laparoscopies present an important challenge for a single arm suturing device. For example, a common practice for surgeons during an anastomosis is to manually “flip” the tubes during the procedure to expose the back side. While our aim is to avoid similar interventions, we anticipate the need to manually adjust the viewpoint of the cameras and to determine the pose of the cameras by involving external tracking devices. Finally, as with current anastomosis techniques, we also anticipate that executing a laparoscopic anastomosis will require an amount of staging which will involve manually installing anchor stitches to stabilize the tissues and restrict their motion. VI. CONCLUSION This paper introduced the STAR system. The STAR is a new surgical robot with the objective to improve the efficiency, accuracy and consistency of laparoscopic suturing. The STAR combines the hardware of an actuated suturing tool, a robot arm, a force sensor, and a camera with a software platform that enables supervisory control, and vision-guided robotics. The supervisory control interface enables a surgeon to either manually select the placement of each stitch or to outline the incision and

1315

let the system determine placements at a desired interval. The incision and placements are tracked in the image enabling the system to adjust its path during the procedure. Our experiments demonstrate that the STAR is efficient by executing a planar suturing task on average five times faster than surgeons using a da Vinci surgical system, nine times faster than experienced surgeons using manual laparoscopic tools, and four times faster than surgeons using a manual Endo360◦ . The experiments also demontrate that the STAR is on average more accurate and more consistent than surgeons for placing tight and regular stitches. We are currently upgrading the STAR with 3-D sensing and tracking, stereoscopic multispectral imaging, and fluorescent markers to improve its visual guidance and tracking. These improvements will enable the STAR to execute more delicate interventions, such as anastomosis, in more realistic conditions. A new stapling tool is also being developed in parallel to add greater flexibility and dexterity. ACKNOWLEDGMENT The authors would like to thank EndoEvolution for their support and the Design Resource Group (DRG) in Toronto, Canada, for their contribution to the design of the suturing tool. We also thank C. Cochenour for her support during the experiments. REFERENCES [1] J. Finkelstein, E. Eckersberger, H. Sadri, S. S. Taneja, H. Lepor, and B. Djavan, “Open versus laparoscopic versus robot-assisted laparoscopic prostatectomy: The European and US experience,” Rev. Urol., vol. 12, no. 1, pp. 35–43, 2010. [2] J. Ruurda, I. Broeders, B. Pulles, F. Kappelhof, and C. van der Werken, “Manual robot assisted endoscopic suturingtime-action analysis in an experimental model,” Surg. Endosc. Other Intervention Techn., vol. 18, no. 8, pp. 1249–1252, Aug. 2004. [3] G. Hubens, H. Coveliers, L. Balliu, M. Ruppert, and W. Vaneerdeweg, “A performance study comparing manual and robotically assisted laparoscopic surgery using the da Vinci system,” Surg. Endosc. Other Intervention Techn., vol. 17, no. 10, pp. 1595–1599, Oct. 2003. [4] J. Wright, C. Ananth, S. Lewin, W. Burke, Y. Lu, A. Neugut, T. Herzog, and D. Hershman, “Robotically assisted versus laparoscopic hysterectomy among women with benign gynecologic disease,” J. Amer. Med. Assoc., vol. 309, no. 7, pp. 689–698, 2013. [5] D. Nio, W. A. Bemelman, R. Balm, and D. A. Legemate, “Laparoscopic vascular anastomoses: Does robotic (Zeus-Aesop) assistance help to overcome the learning curve?,” Surg. Endosc., vol. 19, pp. 1071–1076, 2005. [6] M. Honda, A. Kuriyama, H. Noma, S. Nunobe, and T. A. Furukawa, “Hand-sewn versus mechanical esophagogastric anastomosis after esophagectomy: A systematic review and meta-analysis,” Ann. Surg., vol. 257, no. 2, pp. 238–248, Feb. 2013. [7] D. Korolija, “The current evidence on stapled versus hand-sewn anastomoses in the digestive tract,” Minimally Invas. Ther., vol. 17, no. 3, pp. 151–154, 2008. [8] T. B. Sheridan, Telerobotics, Automation, and Human Supervisory Control. Cambridge, MA, USA: The MIT Press, 1992. [9] K. K. Badani, A. Bhandari, A. Tewari, and M. Menon, “Comparison of two-dimensional and three-dimensional suturing: Is there a difference in a robotic surgery setting?,” J. Endourol., vol. 9, pp. 1212–1215, Dec. 2005. [10] D. Nio, R. Balm, S. Maartense, M. Guijt, and W. A. Bemelman, “The efficacy of robot-assisted versus conventional laparoscopic vascular anastomoses in an experimental model,” Eur. J. Vasc. Endovasc. Surg., vol. 27, pp. 283–286, 2004. [11] T. G. Weiser, S. E. Regenbogen, K. D. Thompson, A. B. Haynes, S. R. Lipsitz, W. R. Berry, and A. A. Gawande, “An estimation of the global volume of surgery: A modelling strategy based on available data,” Lancet, vol. 372, no. 9633, pp. 139–144, Jul. 2008.

1316

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 61, NO. 4, APRIL 2014

[12] T. M. Fullum, J. A. Ladapo, and B. J. B. C. L. Gunnarsson, “Comparison of the clinical and economic outcomes between open and minimally invasive appendectomy and colectomy: Evidence from a large commercial payer database,” Surg. Endosc., vol. 24, no. 4, pp. 845–853, Apr. 2010. [13] J. K. Noel, K. Fahrbach, R. Estok, C. Cella, D. Frame, H. Linz, R. R. Cima, E. J. Dozois, and A. J. Senagore, “Minimally invasive colorectal resection outcomes: Short-term comparison with open procedures,” J. Amer. College Surg., vol. 204, no. 2, pp. 291–307, Feb. 2007. [14] N. Katkhouda, R. J. Mason, S. Towfigh, A. Gevorgyan, and R. Essani, “Laparoscopic versus open appendectomy a prospective randomized doubleblind study,” Ann. Surg., vol. 242, no. 3, pp. 439–450, 2005. [15] T. C. O. of Surgical Therapy Study Group, “A comparison of laparoscopically assisted and open colectomy for colon cancer,” New Engl. J. Med., vol. 350, no. 20, pp. 2050–2059, May 2004. [16] A. Shushan, H. Mohamed, and A. L. Magos, “How long does laparoscopic surgery really take? Lessons learned from 1000 operative laparoscopies,” Human Reproduct., vol. 14, no. 1, pp. 39–43, 1999. [17] K. Marzouk, J. Lawen, I. Alwayn, and B. A. Kiberd, “The impact of vascular anastomosis time on early kidney transplant outcomes,” Transplant. Res., vol. 2, no. 8, 2013. [18] C. M. Hollands, L. N. Dixey, and M. J. Torma, “Technical assessment of porcine enteroenterostomy performed with ZEUS robotic technology,” J. Pediatr. Surg., vol. 35, no. 8, pp. 1231–1233, Aug. 2001. [19] S. Msika, A. Iannelli, A. Marano, G. Zeitoun, G. Deroide, R. Kianmanesh, Y. Flamant, and J. Hay, “Anastomose manuelle intracorporelle sous ˜ vidA©olaparoscopie au cours de la chirurgie colorectale,” Annales de Chirurgie, vol. 125, no. 5, pp. 439–443, 2000. [20] Y.-M. Dion, O. Hartung, C. Gracia, and C. Doillon, “Experimental laparoscopic aortobifemoral bypass with end-to-side aortic anastomosis,” Surg. Laparosc. Endosc., vol. 9, no. 1, pp. 35–38, 1999. [21] A. Hoznek, L. Salomon, R. Rabii, M. B. Slama, A. Cicco, P. Antiphon, and C. Abbou, “Vesicourethral anastomosis during laparoscopic radical prostatectomy: The running suture method,” J. Endourol., vol. 14, no. 9, pp. 749–753, 2000. [22] L. Hashemi, S. Hart, S. Craig, D. Geraci, and E. Shatskih, “Effect of an automated suturing device on cost and operating room time in laparoscopic total abdominal hysterectomies,” J. Minimally Invas. Gynecol., vol. 18, 2011. [23] K. K. Badani, S. Kaul, and M. Menon, “Evolution of robotic radical prostatectomy assessment after 2766 procedures,” Cancer, vol. 110, no. 9, pp. 1951–1958, 2007. [24] J. P. Ruurda and I. A. M. J. Broeders, “Robot-assisted laparoscopic intestinal anastomosis an experimental study in pigs,” Surg. Endosc., vol. 17, pp. 236–241, 2003. ˇ adler, L. Dvoˇra´ cˇ ek, P. Vit´asek, and P. Matouˇs, “The application of [25] P. St´ robotic surgery in vascular medicine,” Innovations, vol. 7, no. 4, pp. 247– 253, 2012. [26] P. S. Kang and Y. Tabbakh, “Bridging the training gap in laparoscopic colorectal surgery,” Brit. Med. J., vol. 345, Oct. 2012. [27] K. L. Harold, B. D. Matthews, C. L. Backus, B. L. Pratt, and B. T. Heniford, “Prospective randomized evaluation of surgical resident proficiency with laparoscopic suturing after course instruction,” Surg. Endosc. Other Intervent. Techn., vol. 16, no. 12, pp. 1729–1731, 2002. [28] P. Joice, G. B. Hanna, and A. Cuschieri, “Ergonomic evaluation of laparoscopic bowel suturing,” Amer. J. Surg., vol. 176, no. 4, pp. 373–378, Oct. 1998. [29] D. V. Martinec, P. Gatta, B. Zheng, P. M. Denk, and L. L. Swanstro¨om, “The trade-off between flexibility and maneuverability: Task performance with articulating laparoscopic instruments,” Surg. Endosc., vol. 23, no. 12, pp. 2697–2701, 2009. [30] D. Masud, S. Undre, and A. Darzi, “Using manual dexterity to predict the quality of the final product in the small bowel anastomosis after a period of training,” Amer. J. Surg., vol. 203, no. 6, pp. 776–781, Jun. 2012. [31] M. Waseda, N. Inaki, J. R. T. Bermudez, G. Manukyan, I. A. Gacek, M. O. Schurr, M. Braun, and G. F. Buess, “Precision in stitches: Radius surgical system,” Surg. Endosc., vol. 21, no. 11, pp. 2056–2062, Nov. 2007. [32] S. Seki, “Accuracy of suture placement,” Brit. J. Surg., vol. 74, no. 3, pp. 195–197, Mar. 1987. [33] Z. Szabo, J. Hunter, G. Berci, J. Sackier, and A. Cuschieri, “Analysis of surgical movements during suturing in laparoscopy,” Endosc. Surg. Allied Technol., vol. 2, no. 1, pp. 55–61, Feb. 1994. [34] A. Yamataka, “Laparoscopic kasai portoenterostomy for biliary atresia,” J. Hepato-Biliary-Pancreat. Sci., vol. 20, no. 5, pp. 481–486, Jun. 2013.

[35] B. M. Ure, J. F. Kuebler, N. Schukfeh, C. Engelmann, J. Dingemann, and C. Petersen, “Survival with the native liver after laparoscopic versus conventional Kasai portoenterostomy in infants with biliary atresia,” Ann. Surg., vol. 253, no. 4, pp. 826–830, Apr. 2011. [36] H. Masoomi, B. Buchberg, B. Nguyen, V. Tung, M. J. Stamos, and S. Mills, “Outcomes of laparoscopic versus open colectomy in elective surgery for diverticulitis,” World J. Surg., vol. 35, no. 9, pp. 2143–2148, Sep. 2011. [37] T. Mbadiwe, A. C. Obirieze, E. E. C. III, P. Turner, and T. M. Fullum, “Surgical management of complicated diverticulitis: A comparison of the laparoscopic and open approaches,” J. Amer. College Surg., vol. 216, no. 4, pp. 782–788, Apr. 2013. [38] J. H. Ashburn, L. Stocchi, R. P. Kira, D. W. Dietz, and F. H. Remzi, “Consequences of anastomotic leak after restorative proctectomy for cancer: Effect on long-term function and quality of life,” Diseases Colon Rectum, vol. 56, no. 3, pp. 275–280, Mar. 2013. [39] J. J. Luj´an, Z. H. N´emeth, P. A. Barratt-Stopper, R. Bustami, V. P. Koshenkov, and R. H. Rolandelli, “Factors influencing the outcome of intestinal anastomosis,” Amer. Surg., vol. 77, no. 9, pp. 1169–1175, Sep. 2011. [40] E. F. Taylor, J. D. Thomas, L. E. Whitehouse, P. Quirke, D. Jayne, P. J. Finan, D. Forman, J. R. Wilkinson, and E. J. A. Morris, “Populationbased study of laparoscopic colorectal cancer surgery 2006–2008,” Brit. J. Surg., vol. 100, no. 4, pp. 553–560, 2013. [41] S. J. Moug, K. McCarthy, and C. Nesbitt, “Bridging the gap: How higher surgical training programmes can produce consultant laparoscopic colorectal surgeons,” Colorectal Disease, vol. 15, no. 7, pp. 911–913, 2013. [42] S. S. Patel, M. S. Patel, S. Mahanti, A. Ortega, G. T. Ault, A. M. Kaiser, and A. J. Senagore, “Laparoscopic versus open colon resections in california: A cross-sectional analysis,” Amer. Surg., vol. 78, no. 10, pp. 1063– 1065, Oct. 2012. [43] B. P. Chan, T. Gomes, R. P. Musselman, R. C. Auer, H. Moloo, M. Mamdani, M. Al-Omran, R. P. Boushey, and O. AlObeed, “Trends in colon cancer surgery in ontario: 2002–2009,” Colorectal Disease, vol. 14, pp. e708–e712, Oct. 2012. [44] K. A. Vakalopoulos, F. Daams, Z. Wu, L. Timmermans, J. J. Jeekel, G.-J. Kleinrensink, A. van der Ham, and J. F. Lange, “Tissue adhesives in gastrointestinal anastomosis: A systematic review,” J. Surg. Res., vol. 180, no. 2, pp. 290–300, Apr. 2013. [45] J. W. Huh, H. R. Kim, and Y. J. Kim, “Anastomotic leakage after laparoscopic resection of rectal cancer: The impact of fibrin glue,” Amer. J. Surg., vol. 199, no. 4, pp. 435–441, Apr. 2010. [46] C. B. Neutzling, S. A. Lustosa, I. M. Proenca, E. M. da Silva, and D. Matos, “Stapled versus handsewn methods for colorectal anastomosis surgery,” Cochrane Library. Cochrane Colorectal Cancer Group, vol. 15, no. 2, Feb. 2012. [47] I. C. Mitchell, R. Barber, A. C. Fischer, and D. T. Schindel, “Experience performing 64 consecutive stapled intestinal anastomoses in small children and infants,” J. Pediatr. Surg., vol. 46, no. 1, pp. 128–130, Jan. 2011. [48] V. K. Bansal, T. Tamang, M. C. Misra, P. Prakash, K. Rajan, H. K. Bhattacharjee, S. Kumar, and A. Goswami, “Laparoscopic suturing skills acquisition: A comparison between laparoscopy-exposed and laparoscopy-naive surgeons,” J. Soc. Laparoendosc. Surg., vol. 16, no. 4, pp. 623–631, 2012. [49] W. J. Halabi, C. Y. Kang, M. D. Jafari, V. Q. Nguyen, J. C. Carmichael, S. Mills, M. J. Stamos, and A. Pigazzi, “Robotic-assisted colorectal surgery in the united states: A nationwide analysis of trends and outcomes,” World J. Surg., vol. 37, no. 12, pp. 2782–2790, Apr. 2013. [50] G. T. Sung and I. S. Gill, “Robotic laparoscopic surgery: A comparison of the Da Vinci and Zeus systems,” Urology, vol. 58, no. 6, pp. 893–898, Dec. 2001. [51] J. Rosen and M. Hannaford, “Doc at a distance,” IEEE Spectr., vol. 43, no. 10, pp. 34–39, Oct. 2006. [52] S. Gidaro, M. Buscarini, E. Ruiz, A. Labruzzo, and M. Stark, “Telelap alf-x: A novel telesurgical system for the 21st century,” Surg. Technol. Int. XXII, pp. 20–25, Dec. 2012. [53] U. Hagn, R. Konietschke, A. Tobergte, M. Nickl, ˜ ˜ ˜ ˜ S. JArg, B. KAbler, G. Passig, M. GrAger, F. FrAhlich, U. Seibold, ˜ L. Le-Tien, A. Albu-SchAffer, A. Nothhelfer, F. Hacker, M. Grebenstein, and G. Hirzinger, “DLR MiroSurge: A versatile system for research in endoscopic telesurgery,” Int. J. Comput. Assist. Radiol. Surg., vol. 5, pp. 183–193, Mar. 2010. [54] H. C. Lin, I. Shafran, D. Yuh, and G. D. Hager, “Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions,” Comput. Aided Surg., vol. 11, no. 5, pp. 220–230, 2006.

LEONARD et al.: SMART TISSUE ANASTOMOSIS ROBOT (STAR): A VISION-GUIDED ROBOTICS SYSTEM FOR LAPAROSCOPIC SUTURING

[55] J. J. Abbott, P. Marayong, and A. M. Okamura, “Haptic virtual fixtures for robot-assisted manipulation,” Robot. Res., vol. 28, pp. 49–64, 2007. [56] A. Kapoor, M. Li, and R. Taylor, “Spatial motion constraints for robot assisted suturing using virtual fixtures,” in Proc. Med. Image Comput. Comput.-Assist. Intervention), 2005, vol. 3750, pp. 89–96. [57] P. Marayong, M. Li, A. Okamura, and G. Hager, “Spatial motion constraints: Theory and demonstrations for robot guidance using virtual fixtures,” in Proc. IEEE Int. Conf. Robot. Autom., 2003, pp. 1954–1959. [58] L. B. Rosenberg, “Virtual fixtures: Perceptual tools for telerobotic manipulation,” in Proc. IEEE Virtual Reality Annu. Int. Symp., 1993, pp. 76–82. [59] T. Xia, C. Baird, G. Jallo, K. Hayes, N. Nakajima, N. Hata, and P. Kazanzides, “An integrated system for planning, navigation and robotic assistance for skull base surgery,” Int. J. Med. Robot. Comput. Assist. Surg., vol. 4, no. 4, pp. 321–330, Dec. 2008. [60] H. Wang, S. Wang, J. Ding, and H. Luo, “Suturing and tying knots assisted by a surgical robot system in laryngeal mis,” Robotica, vol. 28, pp. 241– 252, Mar. 2010. [61] B. Brehmer, C. Moll, A. Markis, R. Kirschner-Hermanis, R. Kn¨uchel, and G. Jakse, “Endosew(tm): A new device for laparoscopic running sutures,” J. Endourol., vol. 22, pp. 307–311, Feb. 2008. [62] T. Go¨opel, F. Ha¨artl, A. Schneider, M. Buss, and H. Feussner, “Automation of a suturing device for minimally invasive surgery,” Surg. Endosc., vol. 25, pp. 2100–2104, Jul. 2011. [63] S. R. Jernigan, A. V. Guillaume Chanoit and, S. B. Owen, M. Hilliard, D. Cormier, B. Laffitte, and G. Buckner, “A laparoscopic knot-tying device for minimally invasive cardiac surgery,” Eur. J. Cardio–Thorac. Surg., vol. 37, pp. 626–630, 2010. [64] J. F. Kuniholm, G. D. Buckner, W. Nifong, and M. Orrico, “Automated knot tying for fixation in minimally invasive, robot-assisted cardiac surgery,” J. Biomech. Eng., vol. 127, pp. 1001–1008, 2005. [65] R. Bauernschmitt, E. U. Schirmbeck, A. Knoll, H. Mayer, I. Nagy, N. Wessel, S. M. Wildhirt, and R. Lange, “Towards robotic heart surgery: Introduction of autonomous procedures into an experimental surgical telemanipulator system,” Int. J. Med. Robot. Comput. Assist. Surg., vol. 1, pp. 74–79, 2005. [66] M. Kitagawa, D. Dokko, A. M. Okamura, and D. D. Yuh, “Effect of sensory substitution on suture-manipulation forces for robotic surgical systems,” J. Thorac. Cardiovasc. Surg., vol. 129, no. 1, pp. 151–158, Jan. 2005. [67] H. Kang and J. T. Wen, “Autonomous suturing using minimally invasive surgerical robots,” in Proc. IEEE Int. Conf. Contr. Appl., 2000, pp. 742– 747. [68] H. Kang and J. T. Wen, “Endobot: A robotic assistant in minimally invasive surgeries,” in Proc. IEEE Int. Conf. Robot. Autom., 2001, pp. 2031–2036. [69] K. Claes and H. Bruyninckx, “Endostitch automation using 2D and 3D vision,”, Katholieke Universiteit Leuven, Leuven, Belgium, Tech. Rep. 06PP160, 2006. [70] S. Iyer, T. Looi, and J. Drake, “A single arm, single camera system for automated suturing,” in Proc. IEEE Int. Conf. Robot. an d Autom., 2013, pp. 239–244. [71] C. Staub, T. Osa, A. Knoll, and R. Bauernschmitt, “Automation of tissue piercing using circular needles and vision guidance for computer aided laparoscopic surgery,” in Proc. IEEE Int. Conf. Robot. and Autom., 2010, pp. 4585–4590. [72] C. E. Reiley, E. Plaku, and G. D. Hager, “Motion generation of robotic surgical tasks: Learning from expert demonstrations,” in Proc. IEEE 32nd Annu. Int. Conf. Eng. Med. Biol. Soc., Sep. 2010, pp. 967–970. [73] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge, U.K.: Cambridge Univ. Press, 2003. [74] K. L. Wu, A. Krieger, S. Leonard, Y. Kim, C. Cochenour, and P. C. W. Kim, “The future concept for surgical robotics: A smart tissue anastomosis robot STAR and a proof-of-concept demonstration,” in Proc. Int. Pediatric Endosurg. Group 22nd Annu. Congr. Endosurg. Children, Jul. 2013, [Online]. Available: http://www.ipeg.org/documents/IPEG2013_finalProgram.pdf.

1317

[75] J. Shi and C. Tomasi, “Good features to track,” in Proc. EEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 1994, pp. 593–600. [76] F. Moreno-Noguer, A. Sanfeliu, and D. Samaras, “Dependent multiple cue integration for robust tracking,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 4, Apr. 2008. [77] O. Aichholzer, F. Aurenhammer, D. Alberts, and B. G¨artner, “A novel type of skeleton for polygons,” J. Universal Comput. Sci., vol. 1, no. 12, 1995. [78] T. H. Cormen, C. E. Leiserson, and R. L. Rivest, Introduction to Algorithms. New York, NY, USA: McGraw-Hill Higher Education, 1990. [79] S. Gottschalk, M. C. Lin, and D. Manocha, “OBBtree: A hierarchical structure for rapid interference detection,” in Proc. 23rd Annu. Conf. Comput. Graph. Interact. Techn., 1996. [80] F. Moreno-Noguer and A. Sanfeliu, “Integration of shape and a multihypotheses fisher color model for figure-ground segmentation in nonstationary environment,” in Proc. 17th Int. Conf. Pattern Recognit, vol. 4, pp. 771–774. [81] Y. Raja, S. J. McKenna, and S. Gong, “Colour model selection and adaptation in dynamic scenes,” in Proc. Lect. Notes Comput. Sci. Comput. Vis., 1998, vol. 1406. [82] A. Blake and M. Isard, Active Contours: The Application of Techniques from Graphics,Vision,Control Theory and Statistics to Visual Tracking of Shapes in Motion. New York, NY, USA: Springer-Verlag, 2000. [83] M. Kass, A. Witkin, and D. Terzopoulos, “Snakes: Active contour models,” Int. J. Comput. Vis., vol. 1, no. 4, pp. 321–331, 1988. [84] Y. Cengiz, P. Blomquist, and L. A. Israelsson, “Small tissue bites and wound strengthan experimental study,” Arch. Surg., vol. 136, pp. 272– 275, Mar. 2001. [85] D. Millbourn, Y. Cengiz, and L. A. Israelsson, “Effect of stitch length on wound complications after closure of midline incisions,” Arch. Surg., vol. 144, no. 11, pp. 1056–1059, Nov. 2009. [86] P. M˚ansson, X. W. Zhang, B. Jeppsson, and H. Thorlacius, “Anastomotic healing in the rat colon: Comparison between a radiological method, breaking strength and bursting pressure,” Int. J. Colorectal Disease, vol. 17, no. 6, pp. 420–425, Nov. 2002. [87] V. Lange, G. Meyer, H. Schardey, A. Holker, R. Lang, A. Nerlich, and F. Schildberg, “Different techniques of laparoscopic end-to-end smallbowel anastomoses,” Surg. Endosc., vol. 9, no. 1, pp. 82–87, Jan. 1995. [88] H. Wakamatsu, A. Tsumaya, E. Arai, and S. Hirai, “Manipulation planning for knotting/unknotting and tightly tying of deformable linear objects,” in Proc. IEEE Int. Conf. Robot. Autom., Apr. 2005, pp. 2505–2510. [89] G. Zong, Y. Hu, D. Li, and X. Sun, “Visually servoed suturing for robotic microsurgical keratoplasty,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., Oct. 2006, pp. 2358–2363. [90] H. M¨onnich, H. W¨orn, and D. Stein, “Op sense—a robotic research platform for telemanipulated and automatic computer assisted surgery,” in Proc. 12th IEEE Int. Worksh. Adv. Motion Contr., Mar. 2012, pp. 1–6. [91] A. Shademan, M. F. Dumont, S. Leonard, A. Krieger, and P. C. W. Kim, “Feasibility of near-infrared markers for guiding surgical robots,” in SPIE Proc. Opt. Model. Perform. Predict. VI, vol. 8840, 2013. [92] S. Kim, Y. T. Lim, E. G. Soltesz, A. M. D. Grand, J. Lee, A. Nakayama, J. A. Parker, T. Mihaljevic, R. G. Laurence, D. M. Dor, L. H. Cohn, M. G. Bawendi, and J. V. Frangioni, “Near-infrared fluorescent type ii quantum dots for sentinel lymph node mapping,” Nature Biotechnol., vol. 22, pp. 93–97, 2004.

Authors’ photographs and biographies not available at the time of publication.

Smart tissue anastomosis robot (STAR): a vision-guided robotics system for laparoscopic suturing.

This paper introduces the smart tissue anastomosis robot (STAR). Currently, the STAR is a proof-of-concept for a vision-guided robotic system featurin...
813KB Sizes 0 Downloads 0 Views