Needle Grasp and Entry Port Selection for Automatic Execution of Suturing Tasks in Robotic Minimally Invasive Surgery.

HHS Public Access Author manuscript Author Manuscript

IEEE Trans Autom Sci Eng. Author manuscript; available in PMC 2017 April 01. Published in final edited form as: IEEE Trans Autom Sci Eng. 2016 April ; 13(2): 552–563. doi:10.1109/TASE.2016.2515161.

Needle Grasp and Entry Port Selection for Automatic Execution of Suturing Tasks in Robotic Minimally Invasive Surgery Taoming Liu, IEEE [Student Member] and M. Cenk Çavuşoğlu, IEEE [Senior Member] Department of Electrical Engineering and Computer Science, Case Western Reserve University, Cleveland, OH, 44106 USA Taoming Liu: [email protected]; M. Cenk Çavuşoğlu: [email protected]

Author Manuscript

Abstract

Author Manuscript

This paper presents algorithms for selection of needle grasp and for selection of entry ports of robotic instruments, for autonomous robotic execution of the minimally invasive surgical suturing task. A critical issue for automatic execution of surgical tasks, such as suturing, is the choice of needle grasp for the robotic system. Inappropriate needle grasp increases operating time requiring multiple regrasps to complete the desired task. In robotic minimally invasive surgery, the entry port that the surgical robot goes through into the patient’s body has a significant role on the performance of the robot. Improper entry port affects the robot’s dexterity, manipulability and reachability. The proposed methods use manipulability, dexterity and torque metrics for needle grasp selection, and employ needle grasp robustness and target location robustness metrics for port selection. The results of a case study simulation in thoracoscopic surgery is also presented to demonstrate the proposed methods.

Author Manuscript

Note to Practitioners—This paper is motivated by the problem of automating low-level surgical tasks in robotic surgery, such as, suturing, retraction, dissection, and providing exposure. Specifically, this paper focuses on needle grasp and entry port selection for automating robotic surgical suturing. Selection of an appropriate way of grasping a needle is critical for successfully and robustly completing autonomous suturing. To the best authors’ knowledge, there are no earlier studies in the literature which focus on the needle grasp selection problem. The proposed approach determines how to grasp the needle by optimizing the surgical system’s manipulation performance. The existing approaches in the literature for selecting entry ports for the robotic surgical tools only consider the teleoperated robotic minimally invasive surgery, in which the surgeons directly control the robotic instruments. However, automated performance of suturing introduces additional challenges due to uncertainties in needle localization and grasping. This paper proposes two new performance metrics on selecting port locations from the perspective of autonomously performing surgical suturing, without direct involvement of the human user. The paper also presents preliminary experiments which demonstrate the effectiveness of the proposed methods.

Index Terms surgical robots; kinematics; port placement; needle grasp selection

Liu and Çavuşoğlu

Page 2

Author Manuscript

I. Introduction

Author Manuscript

In robotic minimally invasive surgery (RMIS), the surgical procedure is performed by a surgeon manipulating teleoperated robotic surgical tools instead of using manual instruments. Existing hardware, like Intuitive Surgical’s da Vinci system (Intuitive Surgical, Inc), is able to help the surgeon to perform intricate, precise surgical procedures. However, RMIS is usually tedious and time consuming, due to the nature of master-slave teleoperation. Intelligent robotic surgical assistants that are capable of autonomously performing low-level surgical manipulation tasks, such as, suturing, retraction, dissection, and providing exposure, have been proposed to improve patient health by enhancing surgeon performance, and reduce patient cost by decreasing operation time. Over the last decade, numerous research projects regarding autonomous and semiautonomous robotic surgical systems are actually in progress [1]. Many surgeons and robotic scientists predict automation of surgery as one of the possible future innovations in the operating room [2, 3]. Automated execution of surgical tasks requires development of new robotic planning, perception, and control algorithms for robustly grasping and manipulating deformable objects, such as soft tissue and surgical threads, under significant uncertainty.

Author Manuscript

The focus of this paper is the problem of optimally selecting needle grasps and entry ports for intelligent robotic surgical assistants that can automatically perform suturing in RMIS. Selection of the needle grasp is critical for automating low-level robotic surgical tasks, such as suturing, since improper needle grasps may result in multiple regrasps during the desired task, leading to increased task completion times and even to task failures. Improper entry port locations can lead to the failure of the suturing task given a desired needle-driving trajectory, due to the robot collisions, the inability to reach the suturing locations, robot singularities and joint limits, etc. Some challenges on finding proper port locations for 146 interventions are discussed in [4]. In general, entry port selection is a major aspect of preoperative planning of RMIS, which requires determining the best access to the target sites identified as the areas of interest in the anatomy as the desired surgical sites.

Author Manuscript

This paper proposes a systematic method to quantitatively compare different needle grasps and entry port placements for autonomously performing the suturing tasks. The proposed techniques employ a feasibility algorithm, determining if performing a given reference needle trajectory is successful, and multiple metrics, quantifying the robustness of the robot’s performance. The feasibility algorithm considers the robot’s joint angle and joint torque limits as well as robot link-to-link, link-to-tissue and link-to-body-inner-surface collisions. The performance metrics assess the quality of needle grasp configurations and entry port locations by evaluating the manipulability, joint angle and joint torque distributions, for execution of the reference needle driving and knot tying trajectories. In this paper, the proposed methods are applied in a typical minimally invasive surgery (MIS) scenario, namely in thoracoscopic surgery, for demonstration. One of the primary contributions of this paper is the proposed method for needle grasp selection for autonomously performing the surgical suturing in RMIS. This is an essential algorithm for the automated performance of the suturing task, since the surgeon would not

IEEE Trans Autom Sci Eng. Author manuscript; available in PMC 2017 April 01.


Page 3

Author Manuscript

be directly controlling the surgical tools through teleoperation. To the best of our knowledge, there are no earlier studies in the literature which focus on this problem.

Author Manuscript

The second main contribution of the present paper is the proposed method for entry port selection. Two new performance metrics, namely, needle grasp and target location robustness metrics are proposed for entry port selection in order to deal with challenges significant for autonomously performing the surgical tasks in RMIS. Uncertainties in needle grasping due to calibration and needle localization errors are significant challenges when autonomously performing suturing, while they are not major concerns for teleoperated RMIS, since the surgeon is in the loop. As it will be discussed in depth in Section II, there are a number of earlier studies in the literature for entry port selection in teleoperated RMIS. But none of the literature considered the automation of the surgical tasks when they considered entry port selection. The metrics they used did not take special requirements of automation into account. The rest of the paper is organized as follows. Related studies in the literature and how the present study differs from the earlier studies are briefly given in Section II. The problem formulation and proposed methods are introduced in Section III. The specific details of the needle grasp and entry port selection algorithms are presented, respectively in sections IV and V. A case study for validation of the proposed techniques is presented in VI. The discussion of the results and conclusions are presented in Section VII.

II. Related Studies

Author Manuscript

In conventional MIS, standardized guidelines exist for entry port selection [5]. Even though these guidelines provide good starting points, they are not directly applicable to RMIS, as the use of robots introduces additional constraints and opportunities. Although there are no earlier studies in the literature which consider port or grasp selection for autonomous robotic execution of surgical tasks, there are a number of related studies which investigated port placement for teleoperated robotic minimally invasive surgical systems. Coste-Maniere et al. [6, 7] presented algorithms that select port placement and robot’s pose for RMIS. They used a 3D anatomical model of the patient to choose a set of admissible entry ports and applied visibility and dexterity of the robot as cost functions to discover the optimal port locations by an exhaustive search. After port locations were defined, an optimal pose of the robot was obtained by minimizing the separation distance between any pair of tool arms’ non-attached links or between link and any obstacle.

Author Manuscript

Selha et al. [8, 9] developed an algorithm that uses preoperative images to provide the surgeon with a set of possible entry ports ranked with respect to tool dexterity, endoscopic viewpoint, and workspace limits. Their kinematic model was represented by a set of angles between the target and the instrument. A set of optimal angles based on the experience of surgeon were set as a criterion in the dexterity function.



Page 4

Author Manuscript

Trejos et al. [10, 11] proposed an algorithm for determining an optimal entry port and a surgical robot configuration by optimizing the dexterity of robot with the global conditioning index, distribution of the inverse of the condition number, and the global isotropy index, a ratio of the smallest and the largest singular values in the entire workspace. The global manipulability of the manipulators were also utilized in [12– 17] to measure the performance of the manipulator. A research team from the German Aerospace Center (DLR), proposed a full planning procedure including preoperative stage, the setup in the operating room and the intraoperative refinement for RMIS [15, 18].

Author Manuscript

Azimian et al. [19, 20] explored the geometric uncertainty of the surgical targets in RMIS by assuming that the configuration of the wrist on the manipulator was uncertain and the geometries were fixed. They utilized a probabilistic approach that maximizes the subset of the wrists with no collisions and violation of the joint limits with respect to these fixed individual target. Kang et al. [21, 22] designed a special suturing instrument (especially for knot-tying) for autonomous suturing in RMIS. Currently there is a reusable suturing device named Endo360° for minimally invasive surgery from EndoEvolution, LLC [23]. Leonard et al. [24] modified the Endo360° device by replacing its handle with two actuators such that it can be automatically controlled. However, such special instruments are not widely used due to size limitations by the needle size and their complex designs. Additionally, compared to da Vinci system’s EndoWrist® instruments, the wrist of Endo360° is less dexterous as it has only 1 degree-of-freedom.

Author Manuscript

To the author’s best knowledge, there are no earlier studies in the literature for optimal needle grasp selection for RMIS. Further, none of the aforementioned studies focused on optimal port selection for the automation of RMIS, they mainly worked for robot-assisted surgical system operated in a master-slave fashion. In master-slave mode, the surgeon selects and changes the needle grasp based on their surgical experiences and preferences. However, in autonomous performance of this task, an algorithm is necessary to determine the proper needle grasp. Since there are inherent random errors in sensory localizations and grasping of the needle, it is important for the algorithm to ensure that the task can still be successfully executed under slight variations in needle grasp configuration. The needle grasp choice is also dependent highly on the entry port location and the approach direction of the surgical tool to the suture site. Therefore, in this study, the needle grasp selection problem is considered together with the entry port selection.

Author Manuscript

Several novel performance metrics for needle grasp and entry port selection are also proposed in this paper. These performance metrics are specifically designed to quantify the kinematic robustness to the uncertainties in the system.



Page 5

Author Manuscript

III. Problem Statement and Formulation This section presents the details of the problem formulation, followed by the descriptions of the reference suturing trajectory, the kinematics of needle grasp and the feasibility algorithm. A. Problem Statement

Author Manuscript

In MIS, the robotic surgical instruments are introduced into the operating site inside the patient, which may be a natural or an artificially created cavity, through small incisions (Fig. 1). Selection of the locations of these entry ports is critical for successful performance of the surgical procedure, as a poor choice of the entry ports may not allow the robotic surgical manipulator to be able to effectively perform key manipulation steps of the procedure due to kinematic constraints. Here, we will specifically evaluate the robotic surgical system’s ability to perform the suturing task, which is composed of needle driving and knot tying, for the selection of entry port location, as suturing is arguably one of the most demanding dexterous manipulation tasks in MIS. The second problem that will be investigated is the needle grasp selection. Specifically, the needle grasp selection algorithm selects the proper grasping point and orientation for the robotic needle holder to grasp the needle, so as to be able to successfully drive the needle at a desired suturing location and direction, subject to the kinematic constraints of the system for a given entry port. B. Desired Reference Trajectory for the Suturing Task

Author Manuscript Author Manuscript

Suturing is a bimanual task comprised of needle driving and knot tying. The algorithm proposed in this paper simultaneously chooses a pair of entry ports for the two robotic surgical manipulators, mimicking two hands of the surgeon. One is equipped with a needle holder (labeled as Robot A) and the other is equipped with forceps (labeled as Robot B). In a surgical procedure, typically 1/2 or 3/8 circular curved needles are used for suturing. For such a needle, the canonical needle trajectory with “good suturing technique” is a circular path that follows the natural curve of the needle [25–27]. This paper uses this canonical needle trajectory as the desired reference needle trajectory that the surgical robotic system will perform. The needle-tissue interaction force estimated from an interaction model proposed by Jackson and Cavusoglu [26] for performing the suturing is also specified along with the motion trajectory.1 It should be noted that the needle trajectory and the interaction forces of the system during the actual performance of the task will not be exactly the same as the reference trajectories due to inhomogeneity and intersubject variability of tissue parameters [31–33]. As such, the proposed algorithms will evaluate the kinematic robustness of the manipulators when executing these reference trajectories, so that the system can robustly perform the tasks under perturbations and variations. The knot tying trajectory used in the autonomous suturing tasks is also different from teleoperatively performed robotic surgery. In the latter, the surgeon manually adjusts the 1Other needle-tissue interaction force models proposed in the literature, [28–30], can also be used as alternatives without change to the algorithm.



Page 6

Author Manuscript

length and tension of the suture, and associated uncertainties. However, in autonomous knot tying, the robotic system applies specialized strategies, such as keeping the suture under mild tension at all times, to manage uncertainties in the shape of the thread, such as twist and sag. In this paper, a rolling arc looping knot tying trajectory specially designed for autonomous robotic surgery proposed by Chow et al. [34, 35] is used. This knot-tying motion planning method proposes a series of coordinated actions for a pair of manipulators for automatically tying a surgeon’s knot. This procedure can address the challenges of autonomous tying, such as keeping the suture in mild tension, reducing uncertainty of the thread, and avoiding suture interference. The force requirements during the knot tying part of the suturing task are ignored, as the force requirements for knot tying are substantially smaller than the force required during the needle driving. The explicit details of the reference suturing trajectory can be found in [26, 27, 34, 35].

Author Manuscript

C. Needle Grasp Parameterization The configuration that the surgical manipulator grasps the needle determines how needle trajectory is converted to manipulator motions. A needle grasp with the surgical end-effector can be parameterized with 4 degrees-of-freedom (DoFs) as shown in Fig. 2(a)2. The gripper coordinate frame G is located on the tip of gripper. Its Z axis is perpendicular to the jaw of the gripper and its X axis points along the gripper. The needle coordinate frame N is placed on the tip of needle, its Z axis is along the tangent at the tip of the needle, and its X axis points to the center of the needle curve.

Author Manuscript

The kinematics of the pose of the needle relative to the grasper can be a translation along the gripper followed by three rotations specifying the needle orientation. Specifically, ϑ0 specifies the insertion translation along the negative X axis of the frame G. ω1, ω2 and ω3 are three rotations that determine the orientation of the needle. ω1 and ω2 are defined to be along the Z and Y axes through the origin of the frame G, and ω3 is defined to be along an axis through the center of the needle. In the zero configuration of needle grasp parameterization, i.e., when d0 = 0 and ϕ1 = ϕ2 = ϕ3 = 0, the needle frame N is aligned with the gripper frame G (shown in Fig. 2(b)). Here, d0 is the insertion translation distance for ϑ0. ϕ1, ϕ2 and ϕ3 are the corresponding rotation angles for ω1, ω2 and ω3, respectively. The forward kinematics map of the needle grasp can then be calculated as : (1)

where

Author Manuscript

(2)

2Please note that the specific image of the gripper shown here is for illustrative purposes only.



Page 7

Author Manuscript

(3)

Here, gGN is a 4×4 matrix representing the position and orientation of Frame N relative to Frame G. c1 and s1 are abbreviations for cos ϕ1 and sin ϕ1, respectively, and similarly for the other terms. rn represents the radius of the needle. D. Feasibility Algorithms

Author Manuscript

For given a reference trajectory of the surgical robot’s end-effector at a desired surgical site and a given port placement, the ability of the manipulator to successfully execute the required motions is evaluated by the feasibility algorithm. In particular, the feasibility algorithm checks throughout the desired trajectory for: 1.

Robot reachability,

2.

Robot joint limits, including angles and torques,

3.

Robot link collisions, including link-to-link, link-to-tissue, and link-to-body-innersurface collisions.

The first criterion simply checks if the manipulator can reach the given target site by comparing the distance from the target to the entry port with the manipulator’s shaft length.3

Author Manuscript

The second criterion determines whether the motion of the manipulator is within the robot’s joint angle limits, and whether the predicted needle-tissue force/torque are within the joint torque limits. The joint angle limits are checked with the required joint angles θi computed from the inverse kinematics of the surgical manipulators over the desired manipulator trajectories. The joint torque limits are evaluated by calculating the required joint torque τi from specified needle force profiles using the manipulator Jacobian. Given the geometries of the robotic manipulators, the tissue configurations and the operation environment, the third criterion checks if safe separation distances between any pair of nonattached links of the manipulator, between the manipulator links and the tissue, and between the manipulator links and patient’s inner body surface are maintained to guarantee the collision-free trajectories.

IV. Needle Grasp Selection Author Manuscript

Needle grasp selection is essential for optimizing the performance of the manipulator when performing a suturing task. With given port locations, a tissue configuration, and a reference needle-driving trajectory, the optimization algorithm selects the needle grasp configuration by maximizing the kinematic robustness of the robotic manipulator. The needle grasp selection is only applied during the needle driving motion. This includes choosing two 3It is important to note that the reachability condition is actually a subset of the condition on the validity of the robot joint limits. However, here, we check the robot reachability separately, as it is usually much simpler to evaluate than the robot joint limits, which requires the full solution of the inverse kinematics of the robotic manipulator. This way, it is possible to avoid the computation of the full inverse kinematics, if reachability condition is violated, reducing the overall computational load.



Page 8

Author Manuscript

needle grasps for the needle insertion and needle extraction. As these two grasp choices are independent, they can be analyzed separately. A. Performance Metrics A collection of three kinematic metrics are selected to quantify the quality of the needle grasp configuration. The first metric that is considered here is the distribution of the manipulability measure of the manipulator along the reference trajectory. Manipulability is defined as the ratio of the smallest singular value to the largest singular value of the manipulator Jacobian4 [36], (4)

Author Manuscript

which quantifies the ability of the manipulator to move freely in all directions in the workspace. The manipulability metric (Wμ) can then be defined as the time average of the cumulative manipulability along the trajectory, (5)

where, T is the duration of the desired trajectory and θ(t) is the desired motion trajectory defined over time interval [0, T]. The manipulability measure, μ(θ), is equal to 0 at singularities of the manipulator and equal to 1 at isotropic configurations. Therefore, maximizing the manipulability metric, Wμ, along the trajectory of the manipulator is desirable to improve the kinematic quality of the needle grasp.

Author Manuscript

The dexterity of a robotic manipulator is significantly decreased when the manipulator is close to its joint angle limits. So it is desirable to stay away from the joint angle limits as much as possible for maintaining dexterity. Accordingly, a performance metric is proposed to quantify the distribution of joint angles of the actuated joints in their range of motions. Specifically, the weighting function for each joint angle θi is defined as follows: (6)

where (7)

Author Manuscript

and θi,max and θi, min are respectively the maximum and minimum values of the joint angle i. The constant 4 in (6) and (10) is used for scaling the values of the metrics to the range [0, 1]. Use of the square is preferred over the absolute value for smoothness. The total cost for joint i along the given desired trajectory θi(t), t ∈ [0, T], is then defined as 4In this study, we use the body Jacobian, as it is independent of the choice of the base frame of the manipulator [36].



Page 9

Author Manuscript

(8)

The dexterity metric (Wθ) can then be defined as the average of the sum of all joints’ total costs:

(9)

where N is the total number of degrees of freedom of the system. Wθ ∈ [0, 1] and, minimizing the dexterity metric Wθ improves the dexterity of the manipulator along the given desired trajectory.

Author Manuscript

As mentioned in the feasibility algorithm, the joint torques can be calculated through the Jacobian of the manipulator, given the needle-tissue interaction forces along the needle trajectory. The third metric, Wτ, focuses on the joint torque levels, quantifying the distribution of each joint torque with respect to its joint torque limit during the performance of the suturing task. Similar to the joint angle case described above, the weighting function for each joint torque τi is defined as: (10)

where

Author Manuscript

(11)

and τi,max and τi,min are the maximum and minimum possible torque values for the joint i. The reason for using the middle of the torque range τi,mid is that sometimes the variable torque range might not be symmetric, for example, due to gravity compensation. The total cost for joint i along the given desired needledriving trajectory τi(t), t ∈ [0, T], is then (12)

Author Manuscript

The torque usage metric (Wτ) is then defined as the average of the sum of all joints’ total costs as:

(13)

Wτ ∈ [0, 1] and minimizing the torque usage metric, Wτ, reduces the joint torques that need to be applied by the manipulator along the given desired trajectory. In other words, this metric increases the capability of the manipulator to handle uncertain force deviations from



Page 10

Author Manuscript

the nominal force due to deviations of the actual needle trajectory away from the nominal trajectory. It is important to note that it is possible to combine these three individual kinematic metrics into a single cumulative performance metric in various ways to quantify the kinematic robustness of the manipulator performing a suturing task with a given needle grasp. In this study, a simple linear combination of the three kinematic metrics is used. Namely, the combined performance metric is defined as (14)

Author Manuscript

where σμ, σθ and στ are the scale ratios of the individual kinematic metrics Wμ, Wθ and Wτ. By maximizing the manipulability metric Wμ and minimizing the dexterity metric Wθ and the torque usage metric Wτ subject to the feasibility conditions, the system will ensure that the robot has sufficient dexterity and avoids singular configurations during the performance of the suturing tasks. B. Needle Grasp Selection Algorithm The algorithm of selecting needle grasp is presented in Algorithm 1. Given the entry port location and the desired task specification, a collection of pre-specified needle grasp candidates are investigated by the selection algorithm. θ is the joint angle vector calculated using the inverse kinematics from the end effector’s motions. τ is the joint torque vector calculated through the manipulator Jacobian from the end-effector wrench. The manipulability index μ(θ) is calculated from (4). In the description of the Algorithm 1, Fea represents the result of the feasibility test which indicates whether the corresponding motion can be completed, as discussed in Section III-D.

Author Manuscript

Algorithm 1

Needle Grasp Selection Algorithm input : Suturing site; Suturing trajectory; Port location; Manipulator’s forward and inverse kinematics output: O: a matrix of Ok

Ok: performance metric values for needle grasp k begin

Author Manuscript IEEE Trans Autom Sci Eng. Author manuscript; available in PMC 2017 April 01.


Page 11

Author Manuscript

for each potential needle grasp do for each step in desired trajectory do

angle θ ← use inverse kinematics torque τ ← use body Jacobian Fea ← check feasibility using θ, τ if Fea fails then

Break if Fea not fail then

manipulability μ(θ) ← use Eq. 4 Wμ ← calculate by Eq. 5

Author Manuscript

Wθ ← calculate by Eq. 9 Wτ ← calculate by Eq. 13 Ok ← calculate by Eq. 14 else

Ok ← Infinity return optimal grasp ← find maximum Ok

The optimization for needle grasp selection typically needs to be performed using a bruteforce search as the metrics considered are not convex and may have local extrema points. The computational performance of the algorithms in a specific case will be discussed in more detail in Section VII.

Author Manuscript

If desired, the computations required for grasp selection can be precomputed and stored as a look-up table. Specifically, this look-up table has 6 parameters as inputs which define the transformation relationship between the entry port coordinate frame and the desired suturing location coordinate frame, denoted by gFV. The look-up table outputs 4 parameters which define the needle grasp configuration, gGN, on basis of the dexterity or manipulability index.

V. Entry Port Selection In this study, two performance metrics, namely, needle grasp robustness and target location robustness metrics, are proposed to quantify the quality of entry port configurations for autonomously performing suturing in RMIS. As the surgical suturing task is bimanual, a pair of entry ports needs to be selected for the left and right hand instruments.

Author Manuscript

A. Needle Grasp Robustness Metric 1) Metric Description—This performance metric quantifies the kinematic robustness of a given entry port location by evaluating the kinematic quality of the suturing motion when subjected to small variations in needle grasp. This is a critical concern for autonomous suturing, as ensuring a precise autonomous needle grasp is not always possible due to random errors in sensory localization and in grasping of the needle. A larger set of feasible needle grasp configurations with which the manipulator can perform the desired needle



Page 12

Author Manuscript

motion trajectory at a given entry port indicates a larger likelihood of success in performing the suturing task from that entry port. Additionally, for each needle grasp allowing the manipulator to complete the desired tasks, the kinematic quality of the needle driving and knot tying motion trajectories, such as singularity avoidance, is also an important consideration. The suturing motion is composed of three phases, namely, needle insertion, needle extraction, and knot tying. The robot motion performed during each of the first two phases depends on the configuration in which the needle is grasped during the corresponding phase. In the proposed method, the needle grasp robustness metric for a given entry port is calculated by evaluating the range of needle grasps that lead to feasible robot motion trajectories, and comparing the kinematic quality of these resulting trajectories. Specifically, the needle grasp robustness metric for a given entry port is defined as:

Author Manuscript

(15)

i = 1 … E and k = 1 … F are respectively the set of needle grasp configurations that lead to a feasible execution of the desired needle insertion and extraction trajectories by the robot. Wμ,(․) is the same kinematic quality measure defined in (5). The subscripts, in, ex, and kt, indicate needle insertion, needle extraction, and knot tying segments of the motion trajectory. As only Robot A performs the needle insertion and extraction, Wμ,in and Wμ,ex only involve Robot A. But during the knot tying task, both robots are involved. Then the ultimate manipulability of both robots during the knot tying motion is proposed to be the geometric average of two robots’ manipulability,

Author Manuscript

(16)

where

and

are respectively the cumulative manipulability of robot A and B,

computed by (5). In (15), and quantify the grasp robustness during the needle insertion and the needle extraction, respectively, through the sum of the manipulability metrics of the feasible needle grasp configurations.

Author Manuscript

The resulting performance metric S quantifies the total kinematic robustness of two manipulators over the range of feasible needle grasps, where larger number of the feasible needle grasp configurations and larger manipulability values during the resulting trajectories lead to higher metric values. Algorithm 2

Port Selection Algorithm Using Needle Grasp Robustness Metric input : Suturing Site & Suturing trajectory



Page 13

Author Manuscript

Manipulator’s forward and inverse kinematics output: S: a matrix of Sn

Sn: cost function values for port n begin for each pair of candidate ports do for each potential needle grasp do for each step in insertion trajectory do

angle θ ← use inverse kinematics torque τ ← use body Jacobian Feain ← check feasibility using θ, τ

Author Manuscript

if Feain fails then

Break if Feain not fail then ← calculate by Eqs: 5 & 4 else

←0

← sum up each

Author Manuscript

Repeat extraction process in the same way

← sum up each for each step in knot tying trajectory do

angle θ ← use inverse kinematics torque τ ← use body Jacobian Feakt ← check feasibility with θ, τ if Feakt fails then

Break if Feakt not fail then

Author Manuscript

Wμ, kt ← calculate by Eq: 4 & 5 & 16 Sn ← calculate with Eq: 15 else

Sn ← 0 return optimal entry port ← find maximum Sn



Page 14

Author Manuscript

2) Algorithm—The process of seeking port placement based on the needle grasp robustness is presented in Algorithm 2. Feain, Feaex, and Feakt represent the results of the feasibility test which indicate whether the corresponding motion can be completed. Following the calculation of the metric, the optimal port locations for both robots are simultaneously determined through a brute-force search over a list of candidate entry port pairs. The optimization is performed with a brute-force search, as the metrics defined above are not convex. Since the port selection can be done pre-operatively off-line, this search method does not affect the practical usability of the method. Algorithm 3

Port Selection Algorithm Using Target Location Robustness Metric

Author Manuscript

input : Suturing trajectory & Manipulator’s kinematics output: T: a matrix of Tm

Tm: the number of feasible targets for port m begin

Author Manuscript Author Manuscript IEEE Trans Autom Sci Eng. Author manuscript; available in PMC 2017 April 01.


Page 15

Author Manuscript

for each pair of candidate ports do for each potential suturing site do for each potential needle grasp do for each step in insertion trajectory do θ ← use inverse kinematics

torque τ ← use body Jacobian Feain ← check feasibility with θ, τ if Feain fails then

Break if Feain fail then

Author Manuscript

Continue to next needle grasp else

Break if Feain fail then

Continue to next suturing target Repeat extraction process in the same way if Feaex fail then

Continue to next suturing target for each step in knot tying trajectory do

Author Manuscript

angle θ ← use inverse kinematics torque τ ← use body Jacobian Feakt ← check feasibility with θ, τ if Feakt fails then

Break if Feakt fail then

Continue to next suturing target else

Author Manuscript

Tm ← Tm + 1 return optimal entry port ← find maximum Tm

B. Target Location Robustness Metric 1) Metric Description—This performance metric optimizes the entry port location according to the range of the surgical site locations where the suturing motion can be successfully performed. Specifically, in this metric, a pre-specified set of the configurations of the surgical sites (positions and orientations) are used. Then for a given entry port, the



Page 16

Author Manuscript

number of the suture sites where the manipulator can perform successful needle suturing with at least one feasible needle grasp is used as the metric. 2) Algorithm—The algorithm for calculating the target location robustness metric is given in Algorithm 3. The structure of this algorithm is similar to Algorithm 2. This algorithm also performs a brute-force search over a list of the candidate entry port pairs to simultaneously optimize both entry ports. The proposed method evaluates the target robustness metric using a given discrete set of the suture target configurations. If a higher resolution coverage of the surgical site is desired, a larger number of the suture target configurations can be used. Since port selection can be done preoperatively off-line, these do not affect the practical usability of the method.

Author Manuscript

VI. Case Study: Minimally Invasive Robotic Thoracoscopic Surgery A. Task Description

Author Manuscript

The case study is a robotic thoracoscopic surgery (minimally invasive thoracoscopic surgery) scenario. Generally, the working volume is small in thoracoscopic surgery as it is constrained by the ribcage, and the surgical robotic instruments need to go through one of the intercostal spaces between the ribs to access the surgical site, e.g., on the heart. Typically, the entry ports are chosen from the 3rd to the 7th spaces. In this case study, the left anterior descending (LAD) artery, a frequent target area for coronary artery bypass graft surgery, is chosen as the suturing site of interest. So the anterior and lateral spaces on the left rib cage are the areas of interest for entry ports. A full size human ribcage skeleton model [37] (shown in Fig. 3), similar to those that could be obtained from preoperative CT or MRI imaging of the patient, is assumed to be provided to the algorithm. Fig. 3 shows the set of potential entry ports (red circles) from the 3rd space to the 7th space on the anterior and lateral areas5. The suturing site on the LAD artery is specified by the suturing location coordinate frame V and the desired suturing line is defined to be along the Y axis of frame V (shown with the red line in Fig. 3).

Author Manuscript

The robot kinematics inside the human body is assumed to be a 2 DoFs wrist (+ gripper) at the end of a tool shaft that goes through a 4 DoFs (3 rotations + 1 insertion translation) entry port.6 As the instrument needs to be inserted through one of the intercostal spaces between the ribs, 2 DoFs of the instrument (2 translations) on the entry port are physically constrained. The range of the wrist joint angles is assumed to be [−90°, 90°]. The virtual joint on the entry port is assumed to be a spherical joint, which has roll, pitch, yaw motions. The ranges of the joint angles for these three motions are assumed to be [Ȓ360°, 360°], [−60°, 60°], and [−60°, 60°], respectively. The range of the joint torque for the spherical joint is assumed to be [−500 Nmm, 500 Nmm] and the range for the wrist is assumed to be [−100 Nmm, 100 Nmm]. The needle used in this case study is a Taper Point SH-1 1/2 Circle

5The choices of a suturing site on the LAD artery and the potential entry ports are by no means fundamental to the proposed method. In an actual surgical application, the potential surgical site would be determined by the surgeon and the set of possible entry port locations would be identified by using preoperative medical imaging. 6Please note that the specific kinematic configuration of the wrist, and the constraint on the possible entry port locations are not fundamental to the presented method, and are chosen for illustration purposes only.



Page 17

Author Manuscript

22 mm needle from Ethicon, which is appropriate for thoracoscopic surgery. Additionally, the left lung is assumed to be manually deflated during the heart surgery to create a larger workspace around the heart. The parietal pericardium and visceral pericardium are ignored when accessing the heart to simplify the task demonstration.

Author Manuscript

In order to simplify the collision calculations, a simplified geometric representation is used, instead of the full geometric models of the anatomy, to define the free workspace. Specifically, the interior of the chest wall is approximated with an elliptic cylinder, and the region occupied by the heart and the left lung is approximated by a plane tangent to the heart at the target of interest (on the LAD artery). The suturing site is assumed to occur on the tangent plane, and the collision detection algorithm for link-to-tissue collision only checks if the robot links collide with this tangent plane instead of the real tissue surface of internal organs. The link-to-bodyinner- surface collision detection only checks if the robot links intersect with the predefined elliptic cylinder. Three simulations with algorithms are performed using MATLAB®. The first two simulations are concerned about the port location selection from two perspectives, (1) the consideration of the needle grasp robustness and (2) the sensitivity to the change of target locations. The third simulation involves the selection of the needle grasp once the entry port and target location are provided. The simulation results are described below. B. Simulation Results for Port Selection

Author Manuscript

1) Needle Grasp Robustness—This metric is specifically appropriate for the case in which the position and orientation of the surgical target is known in advance. Specifically, the target organ/tissue is identified by the surgeon as the desired surgical site using the patient’s tomographic data during preoperative planning. Fig. 4 shows the computed grasp robustness metric for choosing a pair of optimal entry ports on the left side of the rib cage. The circles in the intercostal spaces specify all possible entry ports chosen in the simulation. The colors on the marker faces indicate the values of the grasp robustness metric. A value of 0 indicates that there are no feasible needle grasps existing at the selected port location. Higher number corresponds to better grasp robustness and better manipulability of the manipulators. Here, we are interested in the relative values of the metric, as the absolute value of the metric scales with the number of candidate grasp configurations considered.

Author Manuscript

In the simulation, the entry port with a red hexagram (six-pointed star) in Fig. 4(a) is the best choice for Robot A, given the location of entry port B with a magenta triangle. Fig. 4(b) shows the optimal choice of entry port B with a magenta triangle given the optimal entry port A location in a red hexagram.7

7The grasp (and target location) robustness metrics are simultaneous functions of both port A and port B locations. As such, it is not possible to visualize the metric values in a single spatial plot. Therefore, in Figs. 4(a) and 6(a), the values of the corresponding metrics are plotted as a function of port A location when port B location is fixed to its optimal choice (marked). Similarly, in Figs. 4(b) and 6(b), the values of the corresponding metrics are plotted as a function of port B location, when the port A location is fixed to its optimal choice (marked).



Page 18

Author Manuscript

In the results shown in Fig. 4(b), a multitude of entry ports B are quite good as the optimal choice. But Fig. 4(a) shows that only one good entry port for Robot A. The reason is that the major part of the suturing task is performed by Robot A and Robot B assists only during the knot tying to accomplish this task. So Robot B has less kinematic constraints and more flexibility in the selection of the entry ports than Robot A. The choice of the entry port A is more critical for the successful completion of the suturing task. The results also indicate that the entry ports close to the sternum in the 5th, 6th, and 7th spaces are more preferable for performing the suturing tasks based on the grasping robustness metric. These results agree with the general clinical guidelines (e.g. [5]).

Author Manuscript

Although the selected two entry ports are one intercostal space apart, the distance between them in Fig. 4 is not small. Specifically, one port is located in the anterior of the chest and the other port is located in the lateral part of the chest. Furthermore, if a large robot is used in the future, the actual dimensions of the robot can be used in the computation and the separation distance in the feasibility algorithm can be enlarged. The presented case study only considers the interior part of the surgical manipulators, but this is by no means fundamental to the method. In an actual application, the full geometry of the robot would be used in the feasibility algorithm for collision detection.

Author Manuscript

2) Target Location Robustness—The target location robustness metric aims to enhance the robotic system’s ability to perform the surgical tasks on different preferred surgical sites. In reality, the area of interest on the target tissue is a continuous space. To simplify the representation of this continuous space, a collection of discretely distributed characteristic locations is chosen to represent a continuous space of interest on the target tissue. The capability of the robotic system to perform suturing at different directions for a given site can be evaluated by assigning a number of different directions of the suturing lines at that suturing site. In the simulation, a total number of candidate target locations is 6 and the number of directions of the suturing lines used at each site is 6, as shown in Fig. 5. All of these target locations are chosen along the LAD artery for demonstrating this specific example in thoracoscopic surgery.

Author Manuscript

Fig. 6 shows the computed target robustness metric values for the choices of the entry ports. Similar to the grasp robustness metric case, all the circles distributed in the spaces among the ribs represent the possible entry ports. The colors on the colorbar specify the values of the target location robustness metric. 0 indicates that, for the corresponding entry port, the manipulator can not perform the desired reference needle trajectory at any of the 36 target configurations that are considered. The lighter color indicates that more suturing sites are feasible for robot to perform. They specify the values of the target location robustness metric. The results in Fig. 6(a) show that given the selected port B, the optimal entry port for robot A marked with a red hexagram allows the system to perform suturing at 30 of the selected 36 target configurations. This kind of optimal entry port has 10 choices. Besides, 5 more



Page 19

Author Manuscript

entry ports allow suturing at 28 target configurations. For the other robotic instrument (robot B), there are 4 choices (marked with magenta triangles in Fig. 6(b)) that allow suturing at 30 target configurations, and 2 more entry ports that allow suturing at 28 target configurations, for the selected port A (marked with a red hexagram in Fig. 6(b)). C. Simulation Results of Needle Grasp Selection In Section IV-A, the performance metrics, namely, manipulability (Wμ), dexterity (Wθ), and joint toque usage (Wτ) were proposed for optimizing needle grasp. In this case study, the ultimate value of the combined performance metrics is computed through (14), where weights σμ = 1, σθ = 5, στ = 5 are used.

Author Manuscript

In this example, the same operation setup shown in Fig. 3 is used with a prespecified entry port at the 6th intercostal space and a desired suturing site located on the LAD artery. In the needle grasp selection process, the reference needle trajectory is pre-specified. A total of 3410 possible needle grasp configurations are evaluated. The results of the distribution of all feasible needle grasp configurations and the optimal needle grasps for needle insertion and extraction are shown in Fig. 7. The distribution of all feasible needle grasps are shown in cyan dotted curves. The optimal selection is shown in a black solid curve.

Author Manuscript

For needle insertion, 371 needle grasps out of a total of 3410 candidate grasps are feasible. Additionally, 26 needle grasps have metric values of more than 90% of the optimal metric value. Similarly for needle extraction, there are 426 feasible needle grasps. 43 needle grasps have metric values of more than 90% of the optimal grasp robustness metric. These suboptimal grasp configurations are shown in magenta solid curves in Fig. 7. From the results, one can see that the grasp robustness metric varies smoothly with small changes in grasp configurations, as the optimal needle grasp choice is located among a multitude of secondary choices.

VII. Discussion and Conclusions

Author Manuscript

In this paper, we propose a systematic approach to selecting the needle grasp configuration and port placement for performing the automatic execution of the suturing task in RMIS. The algorithms for selecting needle grasp and entry port use the performance metrics introduced, namely, manipulability, dexterity and force metrics for grasp selection, and kinematic needle grasp and target location robustness metrics for port selection. The results of a case study for minimally invasive thoracoscopic surgery with a realistic robotic surgery platform have been presented to demonstrate the proposed needle grasp and port placement selection algorithms. This work demonstrates the usability of the proposed optimization algorithms to select the needle grasp and port placement. In this paper, the implementation of the algorithm considers a pair of robotic arms to complete the automatic needle driving and knot tying tasks collaboratively. A state-of-the-art teleoperated surgical robot may have three (or more) robotic arms, one for mounting the stereoscope, and two (or more) arms for holding surgical tools. Our approach allows



Page 20

Author Manuscript

additional robotic arms, requiring only the inclusion of additional constraints in the optimization procedure, e.g., arm-to-arm collisions, and stereoscope view of surgical site, etc. They will not affect the fundamentals of the approach.

Author Manuscript

The computation time of the proposed algorithms for the entry port and needle grasp selections are important concerns for clinical applicability of the methods. The computation of the grasp robustness metric values in the case study takes approximately 1.15 hours on a computer equipped with Intel® core i7-960 CPU @ 3.20 GHz and 12.0 GB memory, running 64-bit Microsoft Windows 7 operating system. The algorithms are implemented in Matlab® 7.11.0 (R2012b) with parallel computing. In the same case study, the computation of target location robustness metric values requires 42 hours, as this case has more targets to be analyzed than the former case. In the same case study, the selection of the needle grasp takes 105 seconds on an analysis of 3410 candidate grasps by using the same computer setup. The computation time for port placement is less of a concern, as it can be performed off-line during preoperative planning of the procedure. The algorithms are also trivially parallelizable as they require similar independent computations to be repeated for different entry port or needle grasp configurations. As such, significant speedups can be expected if the algorithms are executed on larger scale parallel computers, such as through cloud computing. Also, as an alternative, it is possible to construct a look-up table for the needle grasp selection as discussed in Section IV, for fast on-line operation.

Author Manuscript

Performing an exhaustive search in the proposed algorithms avoids the local minima issues and provides the optimal solution. The current challenge is that the proposed metrics lead to a non-convex optimization problem, which is difficult to deal with. The guarantee of the completeness from the exhaustive search helps the paper to demonstrate the feasibility and effectiveness of the proposed algorithms. However, this search method needs a large amount of the computation time on selecting entry ports that are discussed above. One of the future avenues of the research is how to avoid the need for exhaustive search. Although the selection of the base position and pose of the surgical manipulator has not been explicitly discussed in this paper, the proposed methods are applicable to this problem without any notable change. In this paper, the grasp robustness and target location robustness metrics are presented as two alternative approaches for entry port selection. It is also possible to combine them into a single metric, in a manner similar to what is done for grasp selection in Section IV. However, they are presented independently due to space and clarity considerations.

Author Manuscript

The proposed method similar to the earlier studies in the literature reported in Section II, relies on pre-operative medical images of the patient for planning. However, the actual intraoperative anatomical configuration would typically vary from the pre-operative configuration due to reasons as simple as changes in positioning of the patient. This is the reason why the proposed methods focus on kinematic robustness of the manipulator. Namely, the needle grasps and the entry ports with the largest kinematic robustness will allow the system to be able to perform the desired manipulations under the largest uncertainties.



Page 21

Author Manuscript

In the real surgical environments, additional constraints, e.g., organs, blood vessels, nerves and so on, may need to be taken into consideration in the optimization procedure. These additional constraints will not fundamentally change the proposed method, primarily increasing the workload of constraint calculations in the feasibility algorithm. The effects of the mechanism’s compliance center and the tissue elastic properties in the suturing task are not included in the methods presented in this paper. Investigation of the effects of these on system performance, and development of new methods that take into account these effects are beyond the scope of the present study, and are complex enough to deserve an independent study.

Author Manuscript

In the proposed method, the system is assumed to perform suturing using general purpose tools, as it is typically performed with the da Vinci system, instead of specialized needle driving tools (e.g., [21–24]). These types of specialized instruments have not been considered since they are not widely used due to their limitations.

Acknowledgments The authors would like to thank Dr. Jamshid H. Karimov for providing expertise on minimally invasive surgery and his valuable suggestions regarding clinical aspects. This work was supported in part by NSF under grants IIS-0905344, and CNS-1035602, and National Institutes of Health under grant R21 HL096941, and R01 EB018108.

References


1. Moustris GP, Hiridis SC, Deliparaschos KM, Konstantinidis KM. Evolution of autonomous and semi-autonomous robotic surgical systems: a review of the literature. Int. J Med. Robot Comp. 2011; 7(4):375–392. 2. Camarillo DB, Krummel TM, Salisbury JK. Robotic technology in surgery: Past, present, and future. Am. J. Surg. 2004; 188(4, Supplement 1):2–15. 3. Lanfranco AR, Castellanos AE, Desai JP, Meyers WC. Robotic surgery: a current perspective. Ann. Surg. 2004 Jan; 239(1):14–21. [PubMed: 14685095] 4. Cadire PG-B MD, Himpens J MD, Germay O, Izizaw R, Degueldre M MD, Vandromme J MD, Capelluto E MD, Bruyns J MD. Feasibility of robotic laparoscopic surgery: 146 cases. World Journal of Surgery. 2001; 25(11):1467–1477. [PubMed: 11760751] 5. Ferzli GS, Fingerhut A. Trocar placement for laparoscopic abdominal procedures: a simple standardized method. J. Am. Coll. Surgeons. 2004 Jan; 198(1):163–173. 6. Coste-Maniere, E.; Adhami, L.; Severac-Bastide, R.; Lobontiu, A.; Salisbury, JK.; Boissonnat, J-D.; Swarup, N.; Guthart, G.; Mousseaux, E.; Carpentier, A. Experimental Robotics VII, ser. Lecture Notes in Control and Information Sciences. Vol. 271. Springer; 2001. Optimized port placement for the totally endoscopic coronary artery bypass grafting using the da vinci robotic system; p. 199-208. 7. Adhami L, Coste-Maniere E. Optimal planning for minimally invasive surgical robots. IEEE Trans Robot. and Autom. 2003 Oct; 19(5):854–863. 8. Selha, SD.; Dupont, PE.; Howe, RD.; Torchiana, DF. Dexterity optimization by port placement in robot-assisted minimally invasive surgery. Int. Symp. on Intell. Syst. and Adv. Manuf.; Oct. 2001; Newton, MA. p. 28-31. 9. Cannon JW, Stoll JA, Selha SD, Dupont PE, Howe RD, Torchiana DF. Port placement planning in robot-assisted coronary artery bypass. IEEE Trans. Robot. Autom. 2003 Oct; 19(5):912–917. 10. Trejos, AL.; Patel, RV. Port placement for endoscopic cardiac surgery based on robot dexterity optimization. Proc. Int. Conf. Robot. Autom.; April 2005; p. 912-917.



Page 22

Author Manuscript Author Manuscript Author Manuscript Author Manuscript

11. Trejos AL, Patel RV, Ross I, Kiaii B. Optimizing port placement for robot-assisted minimally invasive cardiac surgery. Int. J. Med. Robot Comp. 2007; 3(4):355–364. 12. Locke, RCO.; Patel, RV. Optimal remote center-of-motion location for robotics-assisted minimallyinvasive surgery. Proc. Int. Conf. Robot. Autom.; April 2007; p. 1900-1905. 13. Sun, LW.; Yeung, CK. Port placement and pose selection of the da vinci surgical system for collision-free intervention based on performance optimization. Proc. IEEE Int. Conf. Intell. Robots Syst.; Oct 2007; San Diego, CA, USA. p. 1951-1956. 14. Çavuşoğlu, MC.; Villanueva, I.; Tendick, F. Workspace analysis of robotic manipulators for a teleoperated suturing task. Proc. IEEE Int. Conf. Intell. Robots Syst.; Oct. 29–Nov. 3; Maui, Hawaii. 2001. p. 2234-2239. 15. Rainer, K. Ph.D. dissertation, Techn. Univ. Mnchen. 2007. Planning of workplaces with multiple kinematically redundant robots. 16. Azimian, H.; Breetzke, J.; Trejos, AL.; Patel, RV.; Naish, MD.; Peters, T.; Moore, J.; Wedlake, C.; Kiaii, B. Preoperative planning of robotics-assisted minimally invasive coronary artery bypass grafting. Proc. Int. Conf. Robot. Autom.; May 2010; p. 1548-1553. 17. Li, Z.; Glozman, D.; Milutinovic, D.; Rosen, J. Maximizing dexterous workspace and optimal port placement of a multi-arm surgical robot. Proc. Int. Conf. Robot. Autom.; May 2011; p. 3394-3399. 18. Tobergte, A.; Konietschke, R.; Hirzinger, G. Planning and control of a teleoperation system for research in minimally invasive robotic surgery. Proc. Int. Conf. Robot. Autom.; May 2009; p. 4225-4232. 19. Azimian, H.; Patel, RV.; Naish, MD.; Kiaii, B. A framework for preoperative planning of roboticsassisted minimally invasive cardiac surgery (RAMICS) under geometric uncertainty. Proc. Int. Conf. Robot. Autom.; May 2011; p. 5018-5023. 20. Azimian H, Patel RV, Naish MD, Kiaii B. A semi-infinite programming approach to preoperative planning of robotic cardiac surgery under geometric uncertainty. J. Biomed. Health Inform. 2013 Jan; 17(1):172–182. 21. Kang, H.; Wen, J. Autonomous suturing using minimally invasive surgical robots. Proc. Int. Conf. Contr.; Sep. 2000; p. 742-747. 22. Kang, H.; Wen, J. Endobot: a robotic assistant in minimally invasive surgeries. Proc. Int. Conf. Robot. Autom.; 2001. p. 2031-2036. 23. EndoEvolution. http://endoevolution.com. 24. Leonard S, Wu K, Kim Y, Krieger A, Kim P. Smart tissue anastomosis robot (star): A visionguided robotics system for laparoscopic suturing. IEEE Trans. Biomed. Eng. 2014 Apr; 61(4): 1305–1317. [PubMed: 24658254] 25. Sherris, DA.; Kern, EB. Essential Surgical Skills. 2nd. Saunders; 2004. 26. Jackson, RC.; Çavuşoğlu, MC. Modeling of needle-tissue interaction forces during surgical suturing. Proc. Int. Conf. Robot. Autom.; May 2012; p. 4675-4680. 27. Jackson, RC.; Cavusoglu, MC. Needle path planning for autonomous robotic surgical suturing. Proc. Int. Conf. Robot. Autom.; May 2013; p. 1669-1675. 28. Okamura AM, Simone C, O’Leary MD. Force modeling for needle insertion into soft tissue. IEEE Trans. Biomed. Eng. 2004 Oct; 51(10):1707–1716. [PubMed: 15490818] 29. Abolhassani N, Patel RV, Moallem M. Needle insertion into soft tissue: A survey. Med. Eng. Phys. 2007; 29(4):413–431. [PubMed: 16938481] 30. Misra S, Ramesh KT, Okamura AM. Modeling of tool-tissue interactions for computer-based surgical simulation: A literature review. Presence: Teleoper. Virtual Environ. 2008 Oct; 17(5):463– 491. 31. Fung, YC. Biomechanics: Mechanical Properties of Living Tissues, ser. Biomechanics. Springer; 1993. 32. Brouwer, I.; Ustin, J.; Bentley, L.; Sherman, A.; Dhruv, N.; Tendick, F. Stud. Health Technol. Inform. IOS Press; 2001. Measuring in vivo animal soft tissue properties for haptic modeling in surgical simulation; p. 69-74. 33. Boonvisut P, Çavuşoğlu MC. Estimation of soft tissue mechanical parameters from robotic manipulation data. IEEE Trans. Mecha. 2013; 18(5):1602–1611.



Page 23

Author Manuscript

34. Chow, D-L.; Newman, W. Improved knot-tying methods for autonomous robot surgery. Proc. Int. Conf. Autom. Sci. Eng.; Aug. 2013; p. 461-465. 35. Chow, D-L.; Jackson, RC.; Cavusoglu, MC.; Newman, W. A novel vision guided knot-tying method for autonomous robotic surgery. Proc. Int. Conf. Autom. Sci. Eng.; Aug. 2014; p. 504-508. 36. Murray, RM.; Li, Z.; Sastry, SS. A Mathematical Introduction to Robotic Manipulation. 1st. 6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742: CRC Press; 1994. 37. Okubo, K. BodyParts3D. Tokyo, Japan: The Database Center for Life Science;

Biographies

Author Manuscript

Taoming Liu received the B.S. degree in Mechatronics Engineering from South China University of Technology, Guangzhou, China, in 2007, and the M.S. degree in Mechanical Engineering from Case Western Reserve University, Cleveland, OH, in 2011. He is currently working toward the Ph.D. degree with the Department of Electrical Engineering and Computer Science, Case Western Reserve University, Cleveland, OH. His research interests include medical robotics, continuum robotics, mechatronics, and control systems.


M. Cenk Çavuşoğlu (S’93-M’01-SM’06) received the B.S. degree in Electrical and Electronic Engineering from the Middle East Technical University, Ankara, Turkey, in 1995 and the M.S. and Ph.D. degrees in Electrical Engineering and Computer Sciences from the University of California, Berkeley, in 1997 and 2000, respectively. He is currently a Professor of Electrical Engineering and Computer Science, Biomedical Engineering, and Mechanical and Aerospace Engineering with Case Western Reserve University, Cleveland, OH. He was a Visiting Researcher with the INRIA Rhones-Alpes Research Center, Grenoble, France, in 1998, a Postdoctoral Researcher and Lecturer with the University of California, Berkeley (2000–2002), and a Visiting Associate Professor with Bilkent University, Ankara, (2009–2010). His research interests include robotics, systems and control theory, and humanmachine interfaces, with emphasis on medical robotics, haptics, virtual environments, surgical simulation, and biosystem modeling and simulation. His current research involves applications of robotics and control engineering to biomedical and biologically inspired engineered systems. Dr. Cavusoglu has previously served as an Associate Editor of the IEEE TRANSACTIONS ON ROBOTICS, and as a Technical Editor of the IEEE/ASME TRANSACTIONS ON MECHATRONICS.



Page 24

Author Manuscript Author Manuscript Author Manuscript

Fig. 1.

Coordinate frames used in the analysis, including the global coordinate frame, two entry port coordinate frames, two tip coordinate frames and a suturing location coordinate frame. Robot A is equipped with a needle holder, and Robot B is equipped with a pair of forceps. The incision/suturing location coordinate frame (V) is set up such that the Z axis is orthogonal to the surface, and the XY axes are on the target plane. The entry port coordinate frames (F(․)) are set up such that the Y axis points to the center of the hemisphere and the XZ axes are tangent to the surface of the sphere.



Page 25

Author Manuscript Author Manuscript Fig. 2.

Author Manuscript

Needle grasp parameterization. (a) Illustrations of a needle grasp pose and definitions of gripper coordinate frame (G, shown in black) and needle coordinate frame (N, shown in blue). The origin of the Frame G is at the tip of the gripper. The Z axis is perpendicular to the jaw of the gripper. The X axis points along the gripper outwards. The origin of the Frame N is located on the tip of the needle. Its Z axis is along the tangent at the tip of the needle and its X axis points to the center of the needle curve. (b) Zero configuration of the needle grasp parameterization, when d0 = 0 and ϕ1 = ϕ2 = ϕ3 = 0. d0 is the insertion translation distance for ϑ0. ϕ1, ϕ2 and ϕ3 are the rotation angles for ω1, ω2 and ω3, respectively.



Page 26

Author Manuscript Author Manuscript Author Manuscript Fig. 3.

Author Manuscript

The setup of the automatical suturing task and the potential entry ports with red circles for the case study. Possible entry ports are located in the spaces between the ribs. The coordinate frame on the suturing site is the frame V. The Y axis of the frame V is along the direction of the incision. The Z axis of the frame V is perpendicular to the incision and pointing outwards. The coordinate frames on the candidate entry ports for robot A and robot B are the frame A and B. Both of the Y axes of these two coordinate frames are pointing to the origin of the frame V. The other two axes, X and Z, depend on the poses of the robots.



Page 27

Author Manuscript Author Manuscript Fig. 4.

Author Manuscript

The simulation result for optimal port selection based on grasp robustness in the case study. Lighter colors correspond to better grasp robustness metric values. The optimal locations for ports A and B are marked with a red hexagram and a magenta triangle, respectively. Figure (a) shows how the grasp robustness metric changes as the port A location is varied, for the optimal choice of the port B location. Figure (b) shows how the grasp robustness metric changes as the port B location is varied, for the optimal choice of the port A location.



Page 28


Fig. 5.

The distribution of 6 suturing sites along the LAD artery randomly chosen for port selection in terms of the target location robustness metric. At each site, the suturing line has 6 different orientation angles. The red lines represent the selected suturing lines. The black arrows represent the directions of the normal vectors of the corresponding tangent planes.



Page 29


Fig. 6.

The simulation result for optimal port selection based on target location robustness in the case study. Lighter colors correspond to more feasible targets where the suturing can be performed. (a) Target robustness metric values of port A, for the port B (marked with a magenta triangle). The ports with optimal values for robot A are marked with red hexagrams. (b) Target robustness metric values of port B, for the port A (marked with a red hexagram). The ports with optimal values for robot B are marked with magenta triangles.



Page 30


Fig. 7.

The result for optimal needle grasp selections in the case study. The optimal needle grasp for the given needle driving task is shown in black. The needle grasps with at least 90% of the optimal metric value are shown in magenta. Other feasible needle grasp configurations are shown in cyan. The bold black line shows the needle holder. (a) Grasps for needle insertion, (b) Grasps for needle extraction.


Ultra-minimally invasive cardiac surgery: robotic surgery and awake CABG.

Exploring the umbilical and vaginal port during minimally invasive surgery.

Needle Path Planning for Autonomous Robotic Surgical Suturing.

Minimally invasive mitral valve surgery utilizing heart port technology.

Systematic review of robotic minimally invasive mitral valve surgery.

Minimally invasive surgery for gastric cancer: A comparison between robotic, laparoscopic and open surgery.

Endoluminal flexible endoscopic suturing for minimally invasive therapies.

Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery.

Design of a Multitasking Robotic Platform with Flexible Arms and Articulated Head for Minimally Invasive Surgery.

Comfort and learnability assessment of a new soft robotic manipulator for minimally invasive surgery.

Anaesthesia for minimally invasive surgery.

Integrate imaging approach for minimally invasive and robotic procedures.

Robotic transanal minimally invasive surgery for local excision of rectal neoplasms.

Partial splenectomy in the era of minimally invasive surgery: the current laparoscopic and robotic experiences.

Is single port enough in minimally surgery for pneumothorax?

Automatic prebent customized prosthesis for pectus excavatum minimally invasive surgery correction.

Minimally invasive mediastinal surgery.

Minimally Invasive Gastric Surgery.

Minimally Invasive Spine Surgery.

Transanal Minimally Invasive Surgery.

Minimally invasive surgery.

Minimally invasive parathyroid surgery.

Minimally Invasive Spinal Surgery.

Minimally invasive rib resection with preservation of periosteum using 1-port video-assisted thoracoscopic surgery.