HHS Public Access Author manuscript Author Manuscript

Stud Health Technol Inform. Author manuscript; available in PMC 2017 October 26. Published in final edited form as: Stud Health Technol Inform. 2016 ; 220: 19–24.

A Unified Framework for Haptic Interaction in Multimodal Virtual Environments Venkata S. ARIKATLAa,1, Ricardo ORTIZa, DE Suvranub, and Andinet ENQUOBAHRIEa aMedical bCenter

Computing Team, Kitware Inc

for Modeling Simulation and Imaging in Medicine, Rensselaer Polytechnic Institute

Author Manuscript

Abstract In this paper we introduce a Modified Iterative Constraint Anticipation (MICA) method that provides a unified framework for direct and response-based indirect haptic interaction common in many interactive virtual environments. Collision constraints during response based interaction that are modeled using the linear complementarity problem (LCP) framework resolves collision constraints from response-based interactions while allowing for accurate computation of reaction forces. Direct user manipulation is enabled by the linear projection constraints (LPC). A smoothing filter is used to post-process the reaction forces arising from both LCP and LPC to achieve stable interactions in real-time. The effectiveness of MICA is demonstrated using example problems involving deformable bodies.

Author Manuscript

Keywords Haptics; Real-time simulation; Physics based simulation

1. Introduction

Author Manuscript

There is an increasing demand for developing complex multimodal virtual environments (VE) for a variety of application domains including simulation-based surgical training [1]. Such environments require realistic real-time interactions with multiple objects in the scene. This makes physically realistic haptic interactions with deformable objects a computationally daunting task. In this paper, we present Modified Iterative Constraint Anticipation (MICA) - a unified method that can handle both type haptic interactions that are common in interactive VE. Haptic interaction with deformable objects may either be direct manipulation or indirect response-based interaction. Direct manipulation involves the user directly controlling the position/velocities of certain regions of the deformable object through applied boundary conditions. Such interactions are common in organ manipulation during surgical simulation and haptic enabled CAD assembly and planning. During indirect interaction, the user interacts with the deformable object through a virtual handle (avatar). The resulting

1

Corresponding Author: Venkata S Arikatla, Kitware Inc., [email protected].

ARIKATLA et al.

Page 2

Author Manuscript

interaction forces are then felt through the haptic device. Collision response strategies are generally employed in computing the deformation fields and interaction forces with the deformable object. MICA proposed in this work is a unified framework that can incorporate both types of interactions. Various methods have been proposed in literature to address the high update rate requirement for force feedback for scenarios of direct or indirect interactions. The first set of methods aim to reduce run-time computational load by improving the efficiency of various processes involved. Techniques based on stiffness precomputation [2], pre-recording and reproducing the impulse response [3], model order reduction [4] have been proposed. However these methods cannot be extended to practical applications involving complex meshes and topology changes due to the increased complexity.

Author Manuscript

The second set of methods, called the multi-rate methods, decouple the simulation update with the high rate haptic updates. The force for the haptic update cycles in between two simulation updates is derived from intermediate approximations of the force or geometry. One of the main advantages of multi-rate methods is that the deformation and collision response computations can be performed at a slower rate independent of much faster haptic rates.

Author Manuscript

Multi-rate methods in literature are designed for special cases like direct point interactions or response-based interactions but not both. There is a clear need for a single comprehensive multi-rate method that can simulate both multiple direct and response-based interactions under one framework while providing physically realistic visual and haptic responses. This paper introduces modified iterative constraint anticipation (MICA) method for this purpose. In MICA direct user interactions such as picking and articulation are incorporated using linear projection constraints (LPCs) and the collisions in indirect interactions are handled with a mixed linear complementarity problem (MLCP) formulation of contact. We modify the MLCP solver proposed in [5] to incorporate both the type of constraints.

2. Methods 2.1. Deformation and Contact Handling

Author Manuscript

We model the deformation using finite elements. Simulations based on FE method have better physical accuracy and allow for assignment of meaningful material parameters such as Young’s modulus and Poisson’s ratio. The discretized version of weak form of the governing equations of motion obtained by using the above displacement approximation results in a set of linear algebraic equations (subject to appropriate boundary conditions) MÜ +CU̇ +KU = F where U is the displacement vector, F is the load vector, M is the consistent mass matrix, K is the stiffness matrix and C is the damping matrix. We further use co-rotational formulation to remove the artifacts due to large rotations in the linear formulation. In modeling contact between objects using LCP formulation, two positive quantities the gap Ψ and the contact forces (modeled using Lagrange multipliers λ) are maintained complementary to each other. Within the purview of discrete time simulation, the complementarity condition at the present time step (t +Δt) can be stated as 0 ≤ λt+Δt ⊥ Ψt+Δt

Stud Health Technol Inform. Author manuscript; available in PMC 2017 October 26.

ARIKATLA et al.

Page 3

Author Manuscript

≥ 0 where λt+Δt, Ψt+Δt are the discretized Lagrange multipliers and gap vector at time (t +Δt) respectively. Combining the deformation modeling with kinetic complementarity constraints, we get the following MLCP

(1)

Where, G is the Jacobian matrix computed from solution at time t that transforms quantities in local coordinates of contact to global coordinates (see figure 2), vt+Δt, λt+Δt are the unknown global velocities and Lagrange multipliers (λt+Δt) at time (t+Δt).

Author Manuscript

2.2. Linear Projection Constraints Linear projection constraints (LPCs) are commonly used in VE simulations to enable contact response, user interaction and internal Dirichlet boundary conditions such as skeletal constraints. The global LPC for the body can be expressed using an orthogonal projector matrix S(S2= S), S ∈ Rn×n where n is the number of degree of freedom. S is a block diagonal matrix composed of local nodal projector matrices and is written as S = diag{S1,S2..........SN}. Where, N (= n / 3) is the number of nodes of the model and Si (∈ R3×3)is the local nodal projector at node i constructed from the constraint directions. If a node i belonging to a deformable object penetrates the surface of a rigid body along the direction pi by an amount zi. Node i is further prevented from moving along the direction pi during the iterative solution process by forming a local projection constraint of the form Si = I −pi ⊗ pi.

Author Manuscript

2.3. Modified Iterative Constraint Anticipation (MICA) MICA combines ICA [5] which is able to resolve generic collisions, with the MBGS [6] which can resolve linear projection constraints. This is achieved by replacing the GaussSeidel relaxation of velocities in the outer iteration with MBGS. MICA first updates the Lagrange multipliers respecting the LCP constraints followed by solving for velocities respecting the projection constraints. Both the iterations, PGS and MBGS are coupled through force terms as can be seen in algorithm 1. The fact that MICA can handle both LPC and LCP type of constraints is of paramount importance from the practical application point-of-view in VR simulations. The convergence is based on the 2norm of the solution vectors from the consecutive iterations.

Author Manuscript

2.4. Force Computation As mentioned before, there are two kinds of interaction forces: (a) forces from direct user manipulation (b) indirect forces resulting from the interaction of user handle with the deformable objects. At a given time t, we compute the reaction force from interaction directly from solution of MICA as . In the case of direct interaction, we use the fact that sum of external and internal forces should be equal in

Stud Health Technol Inform. Author manuscript; available in PMC 2017 October 26.

ARIKATLA et al.

Page 4

Author Manuscript

order for the dynamic equilibrium to hold. We therefore compute the sum of internal forces at all the nodes that incorporate projection constraints in order to find the interaction force as where

is the externally applied forces at node i. Algorithm 1

Modified Iterative Constraint Anticipation (MICA)

Author Manuscript Author Manuscript

Haptic interaction involves active manipulation by a user as a part of the close loop system resulting in changes in the overall force output since the characteristics of haptic device, biomechanics of hand motion and human sensory mechanisms becomes part of the system. Therefore if the computed force is directly sent to the haptic device one would observe nonsmooth and unrealistic interaction. In order to eliminate the high frequency fluctuations involved during real-time interaction, we use a moving average smoothing filter to smoothen out the force response especially in cases of high stiff user interaction such as compression. If the window size N is high (eg: N= 20 ) at low frame rates, the user can experience artificial forces immediately post-contact. Therefore a correction is made to the weighting window to not account for forces during contact in the calculation of forces immediately post-contact.

3. Results We used a desktop computer equipped with a 3.40GHz Quad core CPU supported by 4GB physical RAM and a NVIDIA Quadro K600 graphics card for the simulations. Geomagic® Touch™ from Geomagic® capable of 3-DOF force feedback with a maximum force of 3.3N was used as a force feedback device. All the computations were performed in a single thread execution of the solver.

Author Manuscript

Figure 1 shows an example of a sphere indenting a deformable cube (see illustration). Figure 1(a) shows a smooth force output from the finite element model when subject is not interacting with the simulation. When the force information from MICA is directly used for feedback a non-smooth forces are felt (Figure 1(b)). It can be seen that the nature of force is non-smooth, especially during compression. When a moving average smoothing filter (with correction) is applied with N=20 during user interaction the force is observed to be smooth as shown in Figure 1(c).

Stud Health Technol Inform. Author manuscript; available in PMC 2017 October 26.

ARIKATLA et al.

Page 5

Author Manuscript

The frame rate of the simulation depends on the convergence of MICA solver. Figure 2 shows the typical convergence of MICA in the case of 3D elasticity with different elastic modulus and Poisson’s ratio of 0.3. The number of iterations (inversely proportional to the frame rate) to converge increases with increasing stiffness. The convergence tolerance for the example shown in the graph is set to10−8. Convergence also depends on the number of elements and LCP type contacts. The greater number of contacts the higher number of iterations to converge.

Author Manuscript

We have used the methods discussed in this paper to arteriovenous malformation (AVM) neurosurgery simulation. Figure 3 shows the surgical forceps controlled by the user interacting with the nidus with force feedback. The interaction of the tool with the nidus is based on a sphere (not displayed) placed at the end of the tool tip on which the collision detection and response are based. The nidus model consists of 9667 tetrahedral elements with 8202 nodal degree of freedom. The MICA was able to update the deformation and the contact response at 10 fps. The force rendering algorithm was able to render smooth force even though the update rates were low.

4. Conclusions In this paper we provided a unified framework for haptic interaction in VE. Our novel solver is able to seamlessly handle two types of user interactions that are common besides solving for contact response of deformable objects. Various examples have been provided that demonstrate our method. As an extension to this work, we would like to (a) integrate the haptics rendering algorithm with the rest of the AVM brain surgery simulator and (b) enhance the speed of MICA solver using parallelization techniques.

Author Manuscript

Acknowledgments The authors gratefully acknowledge the support of this work by the NIH/NIBIB grant #5R01EB010037 and NIH/OD #R44OD018334.

References

Author Manuscript

1. Coles TR, Meglan D, John N. The Role of Haptics in Medical Training Simulators: A Survey of the State of the Art. IEEE Transactions on Haptics. 2011; 4(1):51–66. [PubMed: 26962955] 2. Arikatla VS, Sankaranarayanan G, De S. Cost-efficient suturing simulation with pre-computed models. Stud Health Technol Inform. 2011; 163:31–35. [PubMed: 21335753] 3. Tagawa, K., Hirota, K., Hirose, M. Impulse Response Deformation Model: an Approach to Haptic Interaction with Dynamically Deformable Object. 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems; 2006; p. 209-215. 4. Barbic J, James DL. Six-DoF Haptic Rendering of Contact Between Geometrically Complex Reduced Deformable Models. EEE Trans Haptics. 2008; 1(1):39–52. 5. Otaduy MA, Tamstorf R, Steinemann D, Gross M. Implicit Contact Handling for Deformable Objects. Computer Graphics Forum. 2009; 28(2):559–568. 6. Arikatla VS, De S. A modified multilevel scheme for internal and external constraints in virtual environments. Stud Health Technol Inform. 2013; 184:31–35. [PubMed: 23400125]

Stud Health Technol Inform. Author manuscript; available in PMC 2017 October 26.

ARIKATLA et al.

Page 6

Author Manuscript Author Manuscript Figure 1.

Author Manuscript

Force during compression and release with (a) reaction force without human subject (b) with human subject and no filter (c) with moving average filter (N=20).

Author Manuscript Stud Health Technol Inform. Author manuscript; available in PMC 2017 October 26.

ARIKATLA et al.

Page 7

Author Manuscript Author Manuscript

Figure 2.

Typical convergence of MCIA for 3D elastic object with different elastic modulus. is the difference of the solution vectors.

Author Manuscript Author Manuscript Stud Health Technol Inform. Author manuscript; available in PMC 2017 October 26.

ARIKATLA et al.

Page 8

Author Manuscript Figure 3.

Author Manuscript

(a) Arteriovenous malformation (AVM) (from www.TAAFonline.org) (b) surgical forceps controlled by the user interacting with the finite element model of the nidus.

Author Manuscript Author Manuscript Stud Health Technol Inform. Author manuscript; available in PMC 2017 October 26.

A Unified Framework for Haptic Interaction in Multimodal Virtual Environments.

In this paper we introduce a Modified Iterative Constraint Anticipation (MICA) method that provides a unified framework for direct and response-based ...
506KB Sizes 1 Downloads 10 Views