Biomedical Applications
uPy: A Ubiquitous CG Python API with Biological-Modeling Applications Ludovic Autin and Graham Johnson ■ Scripps Research Institute, La Jolla Johan Hake ■ University of California, San Diego Arthur Olson and Michel Sanner ■ Scripps Research Institute, La Jolla
T
oday’s general-purpose 3D computer graphics software programs (which we call hosts) model, simulate, animate, and render everything from planets to people. Example hosts include Cinema 4D, Maya, and Blender. At the same time, people use scientific and other domaincompetent 3D software to model, simulate, visualize, and analyze data, typically from domain-specific entities. uPy provides a uniform Such entities include molecules, abstraction of the APIs of 3D internal organs, ecosystems, and computer graphics programs human-engineered structures. (hosts). A plug-in written Many recent plug-ins adapt with uPy can run in all uPyhosts to handle such domainsupported hosts. Using uPy, specific data—for example, Bio researchers have created plugBlender (http://bioblender.eu), ins for molecular and cellular Blender for robotics (http:// w i k i.blender.org/i nde x.php/ modeling and visualization. Robotics:Contents), and mMaya uPy can also simplify (Molecular Maya).1 But designprogramming for other types ing a plug-in for each data type is of projects. time-consuming. Furthermore, such plug-ins and scripts are host-dependent because of the differences in host APIs. Although the APIs are similar, porting plug-ins to additional hosts generates duplicated effort and code. To solve this problem, we developed the uPy (ubiquitous Python) module to provide a unified API for modeling geometry and designing UIs across a variety of hosts. (For more on other re50
September/October 2012
search in unified APIs for 3D graphics software, see the sidebar.) Scripts and plug-ins developed using this API can run in all the hosts that uPy supports, thus facilitating their maintenance, broadening their user base, and promoting their reuse. uPy currently supports Blender 2.49, 2.5x, and 2.6x; Cinema 4D 12 and 13; and Maya 2011 and 2012. In addition, it supports DejaVu, our lab’s freely distributed Python- and OpenGL-based 3D visualization engine.2 uPy’s unified API lets programmers and users explore each host’s advantages and drawbacks and apply tools that might be unique to certain hosts. With its simple implementation and architecture, uPy has helped us develop visualization tools covering a range of fields including molecular modeling and mescoscale cellular visualization. Although we use uPy for scientific visualization, it’s completely generic and can be used to extend hosts with functionality of any type, from any application domain.
Working with Python Whereas each type of host offers low-level API access, typically through C++ or host-specific scripting languages, over the past 10 years more and more hosts have been providing high-level API access through Python. This creates the opportunity to develop a Python layer abstracting the host’s APIs and thus presenting a unified API for plug-in developers.
Published by the IEEE Computer Society
0272-1716/12/$31.00 © 2012 IEEE
Python has rapidly gained acceptance in academia and industry, and many scientific tools have been developed in or made callable from it (see, for instance, the Python Papers Monograph; http:// ojs.pythonpapers.org/index.php/tppm). Hosts that embed a Python interpreter can leverage these third-party Python modules for a range of scientific and domain-specific purposes. Nevertheless, integrating Python modules into hosts requires code specific to each host’s Python API, which differs greatly in syntax and design from one host to the next. Because hosts share many core functionalities and the domain-specific modules have a limited number of I/O data types, a single Python adaptor can provide a unified interface for plugging these modules into a variety of hosts at once. With such a connection, domain-specific Python modules can access any host-specific Python-wrapped API. They can thus apply that host’s core functionality to the data to efficiently or natively ■■
■■ ■■ ■■
■■
model and deform shapes, from spheres and boxes to highly complex polyhedra; create materials, light scenes, and cameras; produce keyframe animations; generate realistic and other advanced renderings of scientific and domain-specific data; and apply advanced physics simulations (rigid bodies, soft bodies, fluids, particles, or hair) to the data.
Related Work in Unified APIs for 3D Graphics Software and UI Design
T
he Enthought tool suite (www.enthought.com) includes the TraitsUI package, which provides a unified API for creating UIs based on wxPython or PyQt, and the MayaVI package, which provides some 3D-geometry-modeling capabilities. TraitsUI, however, doesn’t support the custom UI tools of the uPy hosts (3D computer graphics software programs) described in the main article. The Blur-dev project (http://code.google.com/p/blur-dev) extends Autodesk Softimage and 3ds Max with a Python interpreter. It enables scripting in Python and the integration of third-party Python packages, such as PyQt, to create plug-in interfaces in these hosts. However, it’s limited to Windows. The Cortex project (http://code.google.com/p/cortex-vfx) is a low-level development framework of libraries reusable in different hosts for abstracting the APIs of various rendering back ends, including RenderMan, Maya, OpenGL, Houdini, and Nuke. Currently, no unifying API exists for these back ends’ UIs. Moreover, building and installing Cortex isn’t trivial. Because Cortex is a C++ library with associated Python bindings, it’s interoperable with uPy. Compared to similar approaches, uPy is lightweight, OS-agnostic, and easy to install because it’s written in pure Python (that is, it requires no C or C++ extensions to compile). Furthermore, we could easily extend it to support Enthought, Softimage, and 3ds Max as additional hosts, thus enabling the interoperation of all these other software tools.
Cinema 4D
Hosts also provide their own toolkits for developing GUIs for plug-ins. So, employing the same approach, you can also use a single Python adaptor to customize a host’s UI to create module-specific GUIs for every host with no code redundancy.
The uPy Architecture We developed uPy as two independent modules. The modeling-helper class enables access to the hosts’ basic modeling features. The UI-adaptor class provides a unified API for accessing the host UI system. uPy also provides Tkinter and Qt back ends that allow the UIs of plug-ins written using uPy to be instantiated in any Python application. Figure 1 illustrates uPy’s simple architecture, in which the modeling helper and UI adaptor comprise two Python files per host that wrap the host’s overlapping functions. Extending uPy for a new host that already embeds a Python interpreter requires only creating these two files. Each file implements host-specific code for a particular set of functions, which abstracts common host functionalities, such as widget creation and geometry generation.
uPy
PyQt PySide
Maya Modeling helper
UI adaptor Tkinter
Blender
Figure 1. uPy (ubiquitous Python) unites scientific software with various 3D computer graphics software programs (hosts). This diagram represents uPy’s feature coverage across the different Python-wrapped hosts. A uPy modeling-helper class and UI-adaptor class expose each host’s modeling and UI functions. IEEE Computer Graphics and Applications
51
Biomedical Applications uPy-wrapped hosts
(a)
Code snippet for creating a sphere
Blender 2.4
me = Mesh.Primitive.Uvsphere(segments, ring_count, radius) obj = scene.object.new(me, name)
Blender 2.5 and 2.6
bpy.ops.mesh.primitive.uv_sphere_add (segments = segments, ring_count, size = radius) obj = bpy.context.obj obj.name = name
Maya
transform_node, shape = cmds.polySphere (n = name, r = radius, sx = ring_count, sy = segments)
Cinema 4D
obj = c4d.BaseObject(c4d.Osphere) obj[PRIM_SPHERE_RAD] = radius obj[PRIM_SPHERE_SUB] = segments obj.SetName(name)
DejaVu
from DejaVu.Spheres import Spheres obj = Spheres(name, radii = [radius,], quality = segments) vi.AddObject(obj)
uPy
helper = upy.getHelperClass()() obj, mesh = helper.Sphere(name, res = segments, radius = radius)
uPy-wrapped hosts
Code snippet for creating a button
Blender 2.4
button = Blender.Draw.PushButton(“button1”, 0, xpos, ypos, width, height, “tooltip”)
Blender 2.5 and 2.6
class OBJECT_OT_General(bpy.types.Operator): bl_idname = “button.push” bl_label = “pushbutton” def execute(self, context): return{‘FINISHED’} bpy.utils.register_class(OBJECT_OT_General) layout.operator(“button.push”, text = “button1”)
Maya
button = cmds.button(label = “button1”, w = width, h = height)
Cinema 4D
button = self.AddButton(id = 0, flags = c4d.BFH_CENTER | c4d.BFV_MASK, initw = width, inith = height, name = “button1”)
Tk
button = Button(frame, text = “button1”, height = height, width = width)
Qt
button = QtGui.QPushButton(“button1”, self) button.setGeometry(xpos, ypos, width, height)
uPy
button = self._addElement(id = 0, name = “button1”, width = width, height = height, type = “button”, icon = None, variable = self.addVariable(“int”, 0))
(b)
Figure 2. uPy improves coding efficiency. Different hosts require different lines of code (the blue portion) to generate (a) a single sphere and (b) a GUI button via their APIs. A single uPy call (the yellow portion) can replace all the native calls in the blue portions to generate a sphere or button in any or all of the uPy-wrapped hosts. This call produces the same results that the native code would redundantly produce. In this manner, five lines of uPy code effectively replace the 31 LOC a programmer would have to write to make a sphere and button in all five hosts.
Implementation Writing scientific-visualization plug-ins for different applications requires code to ■■ ■■
■■
52
generate the data; create, visualize, and modify the data’s graphical representation; and create a GUI that exposes the plug-in’s usersettable parameters.
September/October 2012
Whereas domain-specific stand-alone packages can generate the data independent of any host API, building a graphical representation and modifying it requires access to each supported hostspecific API (see Figure 2a). To avoid redundancy, the modeling helper wraps the host-specific code snippet (the blue portion of Figure 2a) under a uPy function that has a unique uPy name (the yellow portion). These higher-level uPy functions thus
import upy #recognize host upy.setUIClass() #call the specific UI adaptor class from upy import uiadaptor
Cinema 4D
#confirm host helperClass = upy.getHelperClass() class SphereUI(uiadaptor): def setup(self, **kw): self.helper = helperClass(kw) self.name = “newSphere” self.initWidget() self.setupLayout()
Maya
def CreateLayout(self): self._createLayout() return 1 def Command(self, *args): self._command(args) return 1 def createSphere(self, *args): o = self.helper.getObject(self.name) if o is None: o = self.helper.Sphere(name, res = 12) def scaleSphere(self, *args): #get the state scale = self.getReal(self.sphereSlider) o = self.helper.getObject(self.name) if o is not None: self.helper.scaleObj(o, scale)
Blender 2.49b
(b)
def initWidget(self, id = None): #define the buttons here self.createSphereButton = self._addElemt(name = "Sphere", width = 80, height = 10, type = "button", action = self.createSphere, icon = None, variable = self.addVariable("int", 0)) #this GUI slider adjusts the sphere’s radius self.sphereSlider = self._addElemt(name = "sphere_scale", width = 80, height = 10, type = "sliders", action = self.scaleSphere, variable = self.addVariable("float", 1.0), mini = 0.01, maxi = 5.0, step = 0.01) def setupLayout(self): self._layout = [self.createSphereButton, self.sphereSlider] self._layout.append()
Blender 2.6
DejaVu (c)
mygui = SphereUI(title = "SphereUI") mygui.setup() mygui.display()
(a) Figure 3. Creating a sphere-builder widget for five hosts with a single uPy script file. (a) This uPy code will run in the different hosts and generate a dialog window exposing a button for generating a sphere. (b) uPy’s folder hierarchy represents the Python class hierarchy. Each host has a modeling-helper file and a UI-adaptor file. The modeling-helper and UI-adaptor modules (see the red and blue text, respectively, in Figure 3a) automatically recognize the host software. (c) The code produces homogenous results.
support creating host-specific graphical objects using a unified syntax (the yellow portion). We apply the same approach to create GUI widgets and dialog windows using the UI adaptor. Figure 2b illustrates the different API functions called to create a button in each host and the equivalent single call using uPy. With this approach, one piece of code can execute the same behavior in every supported host. For instance, Figure 3a depicts the plug-in code required to create and scale a sphere. The uPy code will run in the different hosts and generate a GUI dialog window exposing a button for generating the sphere. The window will also display a slider that can scale the generated sphere in real time.
Figure 4 illustrates the geometries currently accessible from the modeling helper. They include text objects, simple primitives (planes, spheres, cones, cylinders, and cubes), platonic solids, pointbased geometry (points, metaballs, and particles), lines (mesh lines, curves, extruded curves, and bones), complex polyhedral meshes, and object instances. These common geometries are the basis for most scientific model visualizations. For example, you can use meshes to represent an isosurface computed from volumetric data or to visualize different molecule representations such as surfaces or ribbon diagrams. For nonscientific applications, meshes are often used to create characters’ skin or generate landscapes. IEEE Computer Graphics and Applications
53
Biomedical Applications
Figure 4. A Cinema 4D screenshot shows the geometries that the modeling helper supports for all hosts. These geometries are the basis for most scientific model visualizations.
The basic geometry objects are common in scientific or domain-competent 3D software. More advanced objects, such as bones and armatures (traditionally used in character animation and robotics to create and control joints using inverse or forward kinematics), particle systems (used for rendering clouds or fire), metaballs (blobby spheres), and instances are less common in scientific or domain-competent 3D software. However, they’re ubiquitous in general professional 3D software and have proven to be useful additions to domain-specific geometry types for building efficient models native to each host. Furthermore, these host features can extend domain-specific functionalities. For example, we used armatures to model and explore protein conformation and flexibility, and particle systems to efficiently display a grid of points or volumetric data. Instances allow the efficient rendering of multiple copies of a given geometry. Instances are defined by a list of 3D transformation matrices or directly translated and rotated onto another object’s model data—for example, onto a mesh’s vertex positions. You can use instances to represent a molecule’s atoms (instances of sphere primitives) or symmetry-related subunits of viral capsids (instances of mesh models). In addition, you can associate any geometry with a material to apply color and texture. For example, a textured plane can display results produced by a graph-plotting Python module such as matplotlib 54
September/October 2012
(http://matplotlib.sourceforge.net) or any plotting library. The modeling helper supports generating geometries, selecting and updating them (modifying vertices, points, edges, and faces), and transformations (position, rotation, and scale). Figure 5 demonstrates the widgets and layouts that the UI adaptor supports, for all hosts. The UI adaptor translates most of the basic widgets commonly used for designing modern GUIs. It currently supports menu bars; check boxes; buttons; labels; string, integer, or floating-point input fields; text area input fields; integer and floating-point sliders; color-chooser fields that trigger system-native color choosers; and pull-down menus. It also provides access to message and file dialog windows that allow for console reporting and native-file browsing. Users can arrange these widgets in rows and group them into collapsible or tabbed subsections. Users can easily extend the list of supported features. To add support for new features, programmers create an empty function in the modeling-helper base class to declare the feature name in uPy. Our approach implements the feature by overriding this empty function in each host-specific modeling-helper subclass with the appropriate snippet of host-specific code. If a feature can’t be implemented in a given host API, the empty function will be called and no operation will occur. Rather than limiting development to the lowest-common-denominator hosts, this architecture lets programmers write many types of scripts for highly functioning or specificfunctioning hosts and automatically culls the scripts for hosts with limited functionality. A created function can be called across all hosts with a single line of code.
Applications We used uPy to develop four plug-ins: ePMV (an embedded version of PMV—Python Molecular Viewer), an extension of GAMer (GeometryPreserving Adaptive Mesher), Tetra, and an extension of AutoFill.
ePMV PMV is a suite of Python packages for visualizing and analyzing molecular structure.2 Its core functionalities include reading, writing, and manipulating molecular data (adding charges, radii, hydrogen atoms, and so on) and generating geometrical objects from that data (CPK coloring, ribbons, surfaces, and so on). Any program providing a Python interpreter can access these functionalities independently of PMV’s native GUI.
Figure 5. The same uPy code generates a common UI across all supported hosts. The dialog windows show the lowest-common-denominator widgets and layouts as provided by the UI adaptor. The widgets are labeled according to their respective type—that is, data input, buttons, and menus. Each dialog window’s title labels the particular host.
The molecular-illustration community has created host-specific extensions for dealing with molecular data, leading to the reimplementation of many long-existing PMV features. We created ePMV (http://epmv.scripps.edu) to provide a richer, more easily extensible set of capabilities than previous molecular plug-ins provided.3 Moreover, ePMV is usable in any uPy-supported host. ePMV uses the modeling helper to translate PMV’s molecular representations into host-native
geometric objects to be displayed in the host’s 3D viewport, leveraging all the host’s rendering capabilities (materials, textures, lights, animations, and so on). Moreover, ePMV retains the correspondence between the host’s geometric object and the underlying atomic and molecular representation. Figure 6a shows the ePMV GUI, which provides an easy, intuitive interface to ePMV’s capabilities. We developed the GUI with the UI adaptor to be usable across all the supported hosts. IEEE Computer Graphics and Applications
55
Biomedical Applications
relative to the receptor. The host provides the method for handling collisions of rigid or soft bodies. Molecules treated as soft bodies can deform for an induced fit. Because ePMV interfaces with our augmented-reality software component,4 users can position the receptor and ligand via handheld markers tracked by a camera (see Figure 6b). Developing ePMV as a uPy plug-in (that is, using uPy to create GUI and geometry objects and to handle user interactions) lets ePMV run in a range of hosts. So, more potential users can access ePMV in their preferred host. uPy also facilitates ePMV maintenance and extension because the code duplication associated with a host-specific plug-in approach doesn’t occur.
GAMer
(a) ∆ Time PyRosetta simulation
Modeler moleculardynamics (MD) simulation
MD driver / Interactive MD protocol
Augmented reality
autoCell mesoscale modeling
∆ Time
Inverse kinematics / bones
Interactive rigid docking
Interactive flexible docking
(b) Figure 6. uPy allows deep plug-ins such as ePMV (an embedded version of Python Molecular Viewer) to support hundreds of functions and GUI options. (a) ePMV UI windows and subwindows inside Cinema 4D. The main window controls the current molecular representation. The subwindows expose options, detailed commands, and extensions. (b) ePMV allows interoperation of a variety of host and external algorithms on the same dataset. The left images show the initial state of an imported or constructed model; the right images show a change in the model state as induced by each labeled method.
When ePMV is instantiated in a host, the host’s computational capabilities become available for manipulating ePMV’s molecular data. For instance, we leveraged Cinema 4D’s bullet-based (http://bulletphysics.org) fast collision detection mechanism to implement interactive manual ligand-receptor docking. Here, ePMV provides the docking mechanism with which the user, guided by an “energy score,” moves the ligand molecule 56
September/October 2012
GAMer is a library that generates high-quality simplex meshes of surfaces and volumes.5 GAMer is used for improving meshes (coarsening and smoothing) and generating quality tetrahedral meshes (using TetGen, http://tetgen.berlios.de), for finite-element simulations. GAMer can serve as a stand-alone command-line tool. However, visual feedback during mesh improvement has proven helpful. Originally, we wrapped GAMer to make it callable from Python and integrated it as a dedicated plug-in for Blender 2.49b. With the new Blender 2.6 arriving and increased demand for GAMer by non-Blender users, we rewrote the plug-in only once using uPy to facilitate migration to Blender 2.5 and 2.6 and to make GAMer available in the other supported hosts. Figure 7 shows how we used GAMer on geometry acquired from manual segmentation of an electron tomography dataset from a mouse heart cell.6 Figure 7a shows portions of two subcellular features from the segmented surface mesh after it has been imported into Blender. On the left is part of the sarcoplasmic reticulum; on the right is part of a Ttubule. In Figure 7b, we applied the mesh improvement algorithms, both coarsening and smoothing the mesh. In Figure 7c, we merged the surface mesh with a surrounding box mesh. Merging a surface is usually a nontrivial task, but it’s easy in this case because Blender and other hosts fully support it. In Figure 7d we used tools exposed by the GAMer plug-in to mark and color regions on the surface mesh. For volumetric meshes, these regions become boundary domains, which you can use to declare boundary conditions in a finite-element computation (see the calcium simulation we describe later). GAMer communicates data (meshes) between the hosts and the GAMer module via the modeling helper. The UI adaptor exposes this process to the
(a)
(b)
(c)
(d)
(e)
Figure 7. GAMer (Geometry-Preserving Adaptive Mesher) improves polyhedral meshes. We (a) took a roughly segmented surface, (b) smoothed it, (c) merged it with a bounding box, and (d) used it to select a boundary region. We also used GAMer to segment different surfaces in (e) Blender 2.6 (top) and Blender 2.49b (bottom).
user to provide a comprehensive GUI that looks the same across all supported hosts. Figure 7e shows GAMer used for mesh improvements and boundary selection in Blender 2.6 and Blender 2.49b.
Tetra Meshes and boundaries generated with GAMer serve as the domain for simulating calcium (Ca 2+) diffusion in cardiac ventricular myocytes. With the recent availability of submicron-resolution myocyte structural data from confocal and electron microscopy,6 researchers have developed Csmol, a tool for modeling reaction-diffusion models for calcium in rat and rabbit myocytes.7 They used it to model half-sarcomere subdomains of ventricular myocytes and to identify inhomogeneities in calcium flux that depend on the cell’s geometric and topological features. Csmol results are usually analyzed using volumetric visualizers such as GMViewer or as projected data via Matlab. These applications provide graphical insight into the Ca2+ concentration as a function of time and position. However, they aren’t directly integrated into a common platform, complicating the workflow for the average scientific user. uPy let us rapidly create Tetra, a stand-alone application to facilitate the visualization of Csmol simulation results. Tetra’s core accesses the raw data from Csmol, produces the tetrahedral mesh,
and colors the mesh according to the compartment and the different Ca2+ concentration and flux values. Tetra’s prototype consists of a Python module that handles the Csmol data (that is, parsing) and communicates it to the modeling helper, which generates the colored tetrahedral representation of the simulated volume (parsed Csmol data). A dialog window developed using the UI adaptor exposes this workflow to users. The window lets users load different tetrahedral meshes, color them according to various properties, and split them on the basis of properties. Using uPy lets us switch between hosts at different stages of the analysis to exploit each host’s strengths. We perform the initial, interactive analysis in DejaVu,2 in which the availability of arbitrary 3D clipping planes and the ability to animate in vertex-based colors greatly facilitates analysis of calcium flux over time in the different compartments (see Figure 8 and the related video at http://youtu.be/wvs-nWE6ypo). Other hosts can then generate high-quality rendering to produce more effective images and advanced animations for publication and communication.
AutoFill AutoFill (http://autofill.scripps.edu) synthesizes 3D models of densely filled volumes and surfaces with arbitrarily shaped geometries (including procedurally IEEE Computer Graphics and Applications
57
Biomedical Applications
(b)
(a)
(c)
Figure 8. Using Tetra with Csmol (a Smoluchowski solver at the subcellular level) to visualize calcium dynamics simulations with tetrahedral meshing from GAMer. A cylindrical membrane feature (a T-tubule) from muscle cells is red to indicate high calcium, compared to the initial concentrations in blue. (a) A view in DejaVu of the cell interior using splitting tools and real-time cutting planes. (b) A view in Cinema 4D of the same cell surface and interior. This view is more usable for publication of a static image thanks to enhanced rendering. (c) A view of two perpendicular plane slices in Maya.
defined structures). AutoCell, a specialization of autoFill, models the biological mesoscale, an intermediate scale (10 –7 to 10 –8 m) between molecular and cellular biology that’s difficult to visualize with experimental methods. AutoCell packs molecular shapes into larger cellular frameworks while satisfying chemical and biological constraints. In general, autoFill positions ingredient objects into, onto, and around larger volumes or compartments, with various methods and degrees of control according to autoFill recipe files. The most common type of ingredient preparation requires users to generate a sphereTree (.sph) file. AutoFill uses this file for efficient collision detection between ingredients and containers. The sphereTree algorithm identifies a set of spheres approximating an object’s shape. We used uPy to create a plug-in that allows use of the sphereTree clustering algorithm on any 3D model type available in any host. This includes, for instance, models ■■ ■■
■■
generated by the host (see Figure 9a), generated by plug-ins such as ePMV (see Figure 9b), or imported from other formats that are highly useful but not traditionally accessible to molecularmodeling software.
With all the ingredients and recipes prepared and defined (shape, concentration, behavior, and so on), autoFill packs the ingredients into a speci58
September/October 2012
fied volume. During packing, autoFill produces a list of transformation matrices for each placed ingredient, which becomes the output file. To efficiently visualize the results, we developed a Python script that calls modeling-helper functions to ■■
■■
generate the master geometry for each unique ingredient and deposit visible instances of each ingredient’s geometry according to the autoFill output transformation matrices.
Ingredient geometries can be molecular meshes, simplified representations (for example, primitive spheres, cylinders, and boxes), curve points for fibrous molecules, and grid points for volumetric data (see Figure 10a). To help the visualization and development of autoFill analysis tools, we implemented uPy scripts for visualization of ■■
■■
■■
runtime packing and different object-coloring modes for debugging; size-dependent available volumes, distance heatmapping, packing order, and distribution graphs; and randomness statistics for the analysis of results.
Figure 10b shows examples. Using uPy to develop autoFill results-visualization tools let us exploit different hosts’ capa-
(a)
(b)
Figure 9. uPy exposes the capabilities of a Python sphereTree generator. (a) A tree of five spheres (one trunk with five branches) generated to approximate a torus geometry clustered stochastically about the coordinates of the torus’s vertices. (b) A tree of three spheres (one trunk with three branches) clustered about the atomic coordinates provided in the protein databank file 1CRN. We generated this tree to approximate the molecule’s shape in a coarse but efficient manner suitable for generating a coarse mesoscale model with autoCell (see Figure 10).
bilities. For instance, the DejaVu host, which is optimized for viewport interactivity, allows for real-time manipulation of a data-heavy model of HIV in full detail (see Figure 10c and the related video at http://youtu.be/vEOybMaRoKc, which shows the rotation of HIV). Other hosts proved useful to set up autoFill recipes—for example, to model certain ingredients—and to produce highquality ray-traced images and animations. High-quality host rendering options produce massive yet detailed images for large, dense volumes such as blood serum. To illustrate such volumes, which are difficult to visualize in real time, we used Cinema 4D to render a high-resolution GigaPan image, viewable at http://gigapan.org/ gigapans/85568. This zoomable Web interface shows a 1-µm2 cuboid filled with molecular detail from a blood serum recipe. Such images enable subjectmatter experts to more easily study, communicate about, and iteratively improve the model without having to repeatedly produce multiscale viewers.
Current Limitations Using uPy has raised some issues that must be considered for any project. All the hosts run into problems when displaying many spheres or polygonal meshes. The computer graphics community is well aware of this problem; solutions include distance-dependent level-of-detail adjustment, field-of-view culling, particles with billboard imposters, and low-polygon objects with normal textures applied. These techniques typically aren’t available in domain-specific modeling and visualization applications. However, the professional hosts already support many of these solutions and will likely be among the first to integrate new
techniques that will reduce these and other issues, minimizing the need for workarounds. The examples we described restricted the uPy API to the set of features common to all supported hosts. The number of common features will likely shrink as the number of supported hosts grows. As we described earlier, uPy currently supports basic types of geometric objects and a finite set of widgets, but we could extend uPy beyond these common primitives. For instance, only Maxon Cinema 4D natively supports Platonic solids. To overcome this limitation, we implemented a Platonic-solid modeler in the modeling-helper base class, thus making it available in all host-specific modeling-helper classes. The Cinema 4D subclass overrides this generic implementation to use its native one. This approach is applicable to a variety of functions missing in one or more supported hosts. Another example is that Blender and Cinema 4D provide noise functions, but the Maya Python API doesn’t. Again, the modeling helper can provide a generic implementation of such a missing function that the native versions can override. These implementations can be coded from scratch or based on third-party packages such as the Python Computer Graphics Kit (http://cgkit.sourceforge. net), which provides a complete noise module. Because the core functionality of some hosts such as Maya are available as third-party Python packages, uPy can help users embed Maya functionality into any other host. Currently, we can’t use uPy to develop plug-ins that require executing OpenGL code or that rely on reacting to mouse interactions in the host’s 3D viewport other than identifying currently selected IEEE Computer Graphics and Applications
59
Biomedical Applications
(a)
(b) Blender
Maya
DejaVu
Cinema 4D
(c) Figure 10. uPy interfaces with autoFill to construct, visualize, and analyze multiscale models of complex 3D volumes including biological cellscapes. (a) Cylinders populate the surfaces of a red-orange mesh and a yelloworange mesh. Sphereoids fill the orange mesh, pyramids fill the green mesh, and cuboids fill the space outside either volume while avoiding existing “P” meshes. AutoFill quickly positions these objects, in this case, with minimal overlap (as indicated by the user) to enhance speed. (b) uPy enables the interactive display of much of autoFill’s analysis output and many of its analysis tools. (c) A lightweight autoFill results viewer builds, hides, reveals, and replaces various components of a multiscale autoFill model (here, the HIV virus), directly from a result file. It uses the most efficient instancing systems available to each host to speed up viewport interaction and to reduce file size.
objects and object positions. However, we could add support for such events because almost all the supported hosts provide the necessary APIs. (The Cinema 4D Python API doesn’t currently support OpenGL code execution.) 60
September/October 2012
A
lthough uPy is written in pure Python, it encapsulates the hosts’ native calls. So, it doesn’t reduce performance compared to using the hosts’ Python APIs directly. uPy plug-ins’ usability in a variety of hosts has
three benefits. First, users can utilize their domainspecific plug-ins in the host that they’re the most familiar with. Second, the same plug-in can run in different hosts at different stages of a project pipeline (for example, for large studios with multiple hosts) or for scientific projects at different analysis stages (for example, one host might work better when interactivity is critical versus when high-quality rendering is the goal). Finally, as we mentioned before, the plug-in can have a broader user base. From the domain-specific methods-development viewpoint, uPy exposes computational methods in more environments without users having to write multiple plug-ins. From the 3D graphics software development viewpoint, uPy extends capabilities across the set of supported hosts (that is, implementing a feature in the uPy base class). From the application user’s viewpoint, uPy offers a choice among the hosts, with the possibility to reproduce work in any of the other supported hosts. We believe uPy can help strengthen and broaden the connections between the computer graphics community and the information visualization, scientific-visualization, visual-analytics, and other scientific communities by facilitating the development of new plug-ins and applications.
Acknowledgments Peter Kekenes-Huskey provided data and helped develop the Tetra plug-in. US National Center for Research Resources grant 5P41RR008605-19, US National Institute of General Medical Sciences grant 8 P41 GM103426-19, and US National Institutes of Health grant 8P41GM103426-19 (through the US National Institute of General Medical Sciences) supported this project. Also, a Research Council of Norway Centre of Excellence grant to the Simula Research Laboratory’s Centre for Biomedical Computing partly supported this research.
References 1. G. McGill, “Molecular Movies ... Coming to a Lecture Near You,” Cell, vol. 133, no. 7, 2008, pp. 1127–1132. 2. M.F. Sanner, “Python: A Programming Language for Software Integration and Development,” J. Molecular Graphics and Modeling, vol. 17, no. 1, 1999, pp. 57–61. 3. G.T. Johnson et al., “ePMV Embeds Molecular Modeling into Professional Animation Software Environments,” Structure, vol. 19, no. 3, 2011, pp. 293–303. 4. A. Gillet et al., “Tangible Interfaces for Structural Molecular Biology,” Structure, vol. 13, no. 3, 2005, pp. 483–491. 5. Z. Yu et al., “Feature-Preserving Adaptive Mesh
Generation for Molecular Shape Modeling and Simulation,” J. Molecular Graphics and Modeling, vol. 26, no. 8, 2008, pp. 1370–1380. 6. T. Hayashi et al., “Three-Dimensional Electron Microscopy Reveals New Details of Membrane Systems for Ca 2+ Signaling in the Heart,” J. Cell Science, vol. 122, no. 7, 2009, pp. 1005–1013. 7. Y.H. Cheng et al., “Finite Element Analysis of the Time-Dependent Smoluchowski Equation for Acetylcholinesterase Reaction Rate Calculations,” Biophysical J., vol. 92, no. 10, 2007, pp. 3397–3406. Ludovic Autin is a postdoc at the Scripps Research Institute’s Molecular Graphics Laboratory. His research employs computer graphics to better model macromolecules across a range of scales. Autin has a PhD in molecular modeling from Denis Diderot University. Contact him at [email protected]. Graham Johnson is a medical illustrator and a qb3@UCSF Fellow at the University of California, San Diego. He specializes in molecular and cellular biology. He illustrated the textbook Cell Biology. Johnson received his PhD in biophysics from the Scripps Research Institute. Contact him at [email protected]. Johan Hake is a postdoc in the Bioengineering Department at the University of California, San Diego. He’s also a research scientist at the Simula Research Laboratory’s Center for Biomedical Computing. His research employs computational methods to better understand the excitation-contraction coupling in healthy and pathological cardiac myocytes. Hake has a PhD in informatics from the University of Oslo. Contact him at [email protected]. Arthur Olson is a professor in the Scripps Research Institute’s Department of Molecular Biology. His research focuses on developing and applying computational methods for visualization and analysis of biomolecular interactions. Olson has a PhD in physical chemistry from the University of California, Berkeley and was a postdoctoral fellow at Harvard University. He’s a member of the IEEE Computer Society, ACM Siggraph, the American Chemical Society, the American Crystallographic Association, and the Protein Society. Contact him at [email protected]. Michel Sanner is an associate professor in the Scripps Research Institute’s Department of Molecular Biology. His research interest is component-based software development, emphasizing 3D visualization. Sanner has a PhD in computer science from the University of Haute-Alsace. He’s a member of the IEEE Computer Society and ACM Siggraph. Contact him at [email protected]. Selected CS articles and columns are also available for free at http://ComputingNow.computer.org. IEEE Computer Graphics and Applications
61
uPy: A Ubiquitous CG Python API with Biological-Modeling Applications Ludovic Autin and Graham Johnson ■ Scripps Research Institute, La Jolla Johan Hake ■ University of California, San Diego Arthur Olson and Michel Sanner ■ Scripps Research Institute, La Jolla
T
oday’s general-purpose 3D computer graphics software programs (which we call hosts) model, simulate, animate, and render everything from planets to people. Example hosts include Cinema 4D, Maya, and Blender. At the same time, people use scientific and other domaincompetent 3D software to model, simulate, visualize, and analyze data, typically from domain-specific entities. uPy provides a uniform Such entities include molecules, abstraction of the APIs of 3D internal organs, ecosystems, and computer graphics programs human-engineered structures. (hosts). A plug-in written Many recent plug-ins adapt with uPy can run in all uPyhosts to handle such domainsupported hosts. Using uPy, specific data—for example, Bio researchers have created plugBlender (http://bioblender.eu), ins for molecular and cellular Blender for robotics (http:// w i k i.blender.org/i nde x.php/ modeling and visualization. Robotics:Contents), and mMaya uPy can also simplify (Molecular Maya).1 But designprogramming for other types ing a plug-in for each data type is of projects. time-consuming. Furthermore, such plug-ins and scripts are host-dependent because of the differences in host APIs. Although the APIs are similar, porting plug-ins to additional hosts generates duplicated effort and code. To solve this problem, we developed the uPy (ubiquitous Python) module to provide a unified API for modeling geometry and designing UIs across a variety of hosts. (For more on other re50
September/October 2012
search in unified APIs for 3D graphics software, see the sidebar.) Scripts and plug-ins developed using this API can run in all the hosts that uPy supports, thus facilitating their maintenance, broadening their user base, and promoting their reuse. uPy currently supports Blender 2.49, 2.5x, and 2.6x; Cinema 4D 12 and 13; and Maya 2011 and 2012. In addition, it supports DejaVu, our lab’s freely distributed Python- and OpenGL-based 3D visualization engine.2 uPy’s unified API lets programmers and users explore each host’s advantages and drawbacks and apply tools that might be unique to certain hosts. With its simple implementation and architecture, uPy has helped us develop visualization tools covering a range of fields including molecular modeling and mescoscale cellular visualization. Although we use uPy for scientific visualization, it’s completely generic and can be used to extend hosts with functionality of any type, from any application domain.
Working with Python Whereas each type of host offers low-level API access, typically through C++ or host-specific scripting languages, over the past 10 years more and more hosts have been providing high-level API access through Python. This creates the opportunity to develop a Python layer abstracting the host’s APIs and thus presenting a unified API for plug-in developers.
Published by the IEEE Computer Society
0272-1716/12/$31.00 © 2012 IEEE
Python has rapidly gained acceptance in academia and industry, and many scientific tools have been developed in or made callable from it (see, for instance, the Python Papers Monograph; http:// ojs.pythonpapers.org/index.php/tppm). Hosts that embed a Python interpreter can leverage these third-party Python modules for a range of scientific and domain-specific purposes. Nevertheless, integrating Python modules into hosts requires code specific to each host’s Python API, which differs greatly in syntax and design from one host to the next. Because hosts share many core functionalities and the domain-specific modules have a limited number of I/O data types, a single Python adaptor can provide a unified interface for plugging these modules into a variety of hosts at once. With such a connection, domain-specific Python modules can access any host-specific Python-wrapped API. They can thus apply that host’s core functionality to the data to efficiently or natively ■■
■■ ■■ ■■
■■
model and deform shapes, from spheres and boxes to highly complex polyhedra; create materials, light scenes, and cameras; produce keyframe animations; generate realistic and other advanced renderings of scientific and domain-specific data; and apply advanced physics simulations (rigid bodies, soft bodies, fluids, particles, or hair) to the data.
Related Work in Unified APIs for 3D Graphics Software and UI Design
T
he Enthought tool suite (www.enthought.com) includes the TraitsUI package, which provides a unified API for creating UIs based on wxPython or PyQt, and the MayaVI package, which provides some 3D-geometry-modeling capabilities. TraitsUI, however, doesn’t support the custom UI tools of the uPy hosts (3D computer graphics software programs) described in the main article. The Blur-dev project (http://code.google.com/p/blur-dev) extends Autodesk Softimage and 3ds Max with a Python interpreter. It enables scripting in Python and the integration of third-party Python packages, such as PyQt, to create plug-in interfaces in these hosts. However, it’s limited to Windows. The Cortex project (http://code.google.com/p/cortex-vfx) is a low-level development framework of libraries reusable in different hosts for abstracting the APIs of various rendering back ends, including RenderMan, Maya, OpenGL, Houdini, and Nuke. Currently, no unifying API exists for these back ends’ UIs. Moreover, building and installing Cortex isn’t trivial. Because Cortex is a C++ library with associated Python bindings, it’s interoperable with uPy. Compared to similar approaches, uPy is lightweight, OS-agnostic, and easy to install because it’s written in pure Python (that is, it requires no C or C++ extensions to compile). Furthermore, we could easily extend it to support Enthought, Softimage, and 3ds Max as additional hosts, thus enabling the interoperation of all these other software tools.
Cinema 4D
Hosts also provide their own toolkits for developing GUIs for plug-ins. So, employing the same approach, you can also use a single Python adaptor to customize a host’s UI to create module-specific GUIs for every host with no code redundancy.
The uPy Architecture We developed uPy as two independent modules. The modeling-helper class enables access to the hosts’ basic modeling features. The UI-adaptor class provides a unified API for accessing the host UI system. uPy also provides Tkinter and Qt back ends that allow the UIs of plug-ins written using uPy to be instantiated in any Python application. Figure 1 illustrates uPy’s simple architecture, in which the modeling helper and UI adaptor comprise two Python files per host that wrap the host’s overlapping functions. Extending uPy for a new host that already embeds a Python interpreter requires only creating these two files. Each file implements host-specific code for a particular set of functions, which abstracts common host functionalities, such as widget creation and geometry generation.
uPy
PyQt PySide
Maya Modeling helper
UI adaptor Tkinter
Blender
Figure 1. uPy (ubiquitous Python) unites scientific software with various 3D computer graphics software programs (hosts). This diagram represents uPy’s feature coverage across the different Python-wrapped hosts. A uPy modeling-helper class and UI-adaptor class expose each host’s modeling and UI functions. IEEE Computer Graphics and Applications
51
Biomedical Applications uPy-wrapped hosts
(a)
Code snippet for creating a sphere
Blender 2.4
me = Mesh.Primitive.Uvsphere(segments, ring_count, radius) obj = scene.object.new(me, name)
Blender 2.5 and 2.6
bpy.ops.mesh.primitive.uv_sphere_add (segments = segments, ring_count, size = radius) obj = bpy.context.obj obj.name = name
Maya
transform_node, shape = cmds.polySphere (n = name, r = radius, sx = ring_count, sy = segments)
Cinema 4D
obj = c4d.BaseObject(c4d.Osphere) obj[PRIM_SPHERE_RAD] = radius obj[PRIM_SPHERE_SUB] = segments obj.SetName(name)
DejaVu
from DejaVu.Spheres import Spheres obj = Spheres(name, radii = [radius,], quality = segments) vi.AddObject(obj)
uPy
helper = upy.getHelperClass()() obj, mesh = helper.Sphere(name, res = segments, radius = radius)
uPy-wrapped hosts
Code snippet for creating a button
Blender 2.4
button = Blender.Draw.PushButton(“button1”, 0, xpos, ypos, width, height, “tooltip”)
Blender 2.5 and 2.6
class OBJECT_OT_General(bpy.types.Operator): bl_idname = “button.push” bl_label = “pushbutton” def execute(self, context): return{‘FINISHED’} bpy.utils.register_class(OBJECT_OT_General) layout.operator(“button.push”, text = “button1”)
Maya
button = cmds.button(label = “button1”, w = width, h = height)
Cinema 4D
button = self.AddButton(id = 0, flags = c4d.BFH_CENTER | c4d.BFV_MASK, initw = width, inith = height, name = “button1”)
Tk
button = Button(frame, text = “button1”, height = height, width = width)
Qt
button = QtGui.QPushButton(“button1”, self) button.setGeometry(xpos, ypos, width, height)
uPy
button = self._addElement(id = 0, name = “button1”, width = width, height = height, type = “button”, icon = None, variable = self.addVariable(“int”, 0))
(b)
Figure 2. uPy improves coding efficiency. Different hosts require different lines of code (the blue portion) to generate (a) a single sphere and (b) a GUI button via their APIs. A single uPy call (the yellow portion) can replace all the native calls in the blue portions to generate a sphere or button in any or all of the uPy-wrapped hosts. This call produces the same results that the native code would redundantly produce. In this manner, five lines of uPy code effectively replace the 31 LOC a programmer would have to write to make a sphere and button in all five hosts.
Implementation Writing scientific-visualization plug-ins for different applications requires code to ■■ ■■
■■
52
generate the data; create, visualize, and modify the data’s graphical representation; and create a GUI that exposes the plug-in’s usersettable parameters.
September/October 2012
Whereas domain-specific stand-alone packages can generate the data independent of any host API, building a graphical representation and modifying it requires access to each supported hostspecific API (see Figure 2a). To avoid redundancy, the modeling helper wraps the host-specific code snippet (the blue portion of Figure 2a) under a uPy function that has a unique uPy name (the yellow portion). These higher-level uPy functions thus
import upy #recognize host upy.setUIClass() #call the specific UI adaptor class from upy import uiadaptor
Cinema 4D
#confirm host helperClass = upy.getHelperClass() class SphereUI(uiadaptor): def setup(self, **kw): self.helper = helperClass(kw) self.name = “newSphere” self.initWidget() self.setupLayout()
Maya
def CreateLayout(self): self._createLayout() return 1 def Command(self, *args): self._command(args) return 1 def createSphere(self, *args): o = self.helper.getObject(self.name) if o is None: o = self.helper.Sphere(name, res = 12) def scaleSphere(self, *args): #get the state scale = self.getReal(self.sphereSlider) o = self.helper.getObject(self.name) if o is not None: self.helper.scaleObj(o, scale)
Blender 2.49b
(b)
def initWidget(self, id = None): #define the buttons here self.createSphereButton = self._addElemt(name = "Sphere", width = 80, height = 10, type = "button", action = self.createSphere, icon = None, variable = self.addVariable("int", 0)) #this GUI slider adjusts the sphere’s radius self.sphereSlider = self._addElemt(name = "sphere_scale", width = 80, height = 10, type = "sliders", action = self.scaleSphere, variable = self.addVariable("float", 1.0), mini = 0.01, maxi = 5.0, step = 0.01) def setupLayout(self): self._layout = [self.createSphereButton, self.sphereSlider] self._layout.append()
Blender 2.6
DejaVu (c)
mygui = SphereUI(title = "SphereUI") mygui.setup() mygui.display()
(a) Figure 3. Creating a sphere-builder widget for five hosts with a single uPy script file. (a) This uPy code will run in the different hosts and generate a dialog window exposing a button for generating a sphere. (b) uPy’s folder hierarchy represents the Python class hierarchy. Each host has a modeling-helper file and a UI-adaptor file. The modeling-helper and UI-adaptor modules (see the red and blue text, respectively, in Figure 3a) automatically recognize the host software. (c) The code produces homogenous results.
support creating host-specific graphical objects using a unified syntax (the yellow portion). We apply the same approach to create GUI widgets and dialog windows using the UI adaptor. Figure 2b illustrates the different API functions called to create a button in each host and the equivalent single call using uPy. With this approach, one piece of code can execute the same behavior in every supported host. For instance, Figure 3a depicts the plug-in code required to create and scale a sphere. The uPy code will run in the different hosts and generate a GUI dialog window exposing a button for generating the sphere. The window will also display a slider that can scale the generated sphere in real time.
Figure 4 illustrates the geometries currently accessible from the modeling helper. They include text objects, simple primitives (planes, spheres, cones, cylinders, and cubes), platonic solids, pointbased geometry (points, metaballs, and particles), lines (mesh lines, curves, extruded curves, and bones), complex polyhedral meshes, and object instances. These common geometries are the basis for most scientific model visualizations. For example, you can use meshes to represent an isosurface computed from volumetric data or to visualize different molecule representations such as surfaces or ribbon diagrams. For nonscientific applications, meshes are often used to create characters’ skin or generate landscapes. IEEE Computer Graphics and Applications
53
Biomedical Applications
Figure 4. A Cinema 4D screenshot shows the geometries that the modeling helper supports for all hosts. These geometries are the basis for most scientific model visualizations.
The basic geometry objects are common in scientific or domain-competent 3D software. More advanced objects, such as bones and armatures (traditionally used in character animation and robotics to create and control joints using inverse or forward kinematics), particle systems (used for rendering clouds or fire), metaballs (blobby spheres), and instances are less common in scientific or domain-competent 3D software. However, they’re ubiquitous in general professional 3D software and have proven to be useful additions to domain-specific geometry types for building efficient models native to each host. Furthermore, these host features can extend domain-specific functionalities. For example, we used armatures to model and explore protein conformation and flexibility, and particle systems to efficiently display a grid of points or volumetric data. Instances allow the efficient rendering of multiple copies of a given geometry. Instances are defined by a list of 3D transformation matrices or directly translated and rotated onto another object’s model data—for example, onto a mesh’s vertex positions. You can use instances to represent a molecule’s atoms (instances of sphere primitives) or symmetry-related subunits of viral capsids (instances of mesh models). In addition, you can associate any geometry with a material to apply color and texture. For example, a textured plane can display results produced by a graph-plotting Python module such as matplotlib 54
September/October 2012
(http://matplotlib.sourceforge.net) or any plotting library. The modeling helper supports generating geometries, selecting and updating them (modifying vertices, points, edges, and faces), and transformations (position, rotation, and scale). Figure 5 demonstrates the widgets and layouts that the UI adaptor supports, for all hosts. The UI adaptor translates most of the basic widgets commonly used for designing modern GUIs. It currently supports menu bars; check boxes; buttons; labels; string, integer, or floating-point input fields; text area input fields; integer and floating-point sliders; color-chooser fields that trigger system-native color choosers; and pull-down menus. It also provides access to message and file dialog windows that allow for console reporting and native-file browsing. Users can arrange these widgets in rows and group them into collapsible or tabbed subsections. Users can easily extend the list of supported features. To add support for new features, programmers create an empty function in the modeling-helper base class to declare the feature name in uPy. Our approach implements the feature by overriding this empty function in each host-specific modeling-helper subclass with the appropriate snippet of host-specific code. If a feature can’t be implemented in a given host API, the empty function will be called and no operation will occur. Rather than limiting development to the lowest-common-denominator hosts, this architecture lets programmers write many types of scripts for highly functioning or specificfunctioning hosts and automatically culls the scripts for hosts with limited functionality. A created function can be called across all hosts with a single line of code.
Applications We used uPy to develop four plug-ins: ePMV (an embedded version of PMV—Python Molecular Viewer), an extension of GAMer (GeometryPreserving Adaptive Mesher), Tetra, and an extension of AutoFill.
ePMV PMV is a suite of Python packages for visualizing and analyzing molecular structure.2 Its core functionalities include reading, writing, and manipulating molecular data (adding charges, radii, hydrogen atoms, and so on) and generating geometrical objects from that data (CPK coloring, ribbons, surfaces, and so on). Any program providing a Python interpreter can access these functionalities independently of PMV’s native GUI.
Figure 5. The same uPy code generates a common UI across all supported hosts. The dialog windows show the lowest-common-denominator widgets and layouts as provided by the UI adaptor. The widgets are labeled according to their respective type—that is, data input, buttons, and menus. Each dialog window’s title labels the particular host.
The molecular-illustration community has created host-specific extensions for dealing with molecular data, leading to the reimplementation of many long-existing PMV features. We created ePMV (http://epmv.scripps.edu) to provide a richer, more easily extensible set of capabilities than previous molecular plug-ins provided.3 Moreover, ePMV is usable in any uPy-supported host. ePMV uses the modeling helper to translate PMV’s molecular representations into host-native
geometric objects to be displayed in the host’s 3D viewport, leveraging all the host’s rendering capabilities (materials, textures, lights, animations, and so on). Moreover, ePMV retains the correspondence between the host’s geometric object and the underlying atomic and molecular representation. Figure 6a shows the ePMV GUI, which provides an easy, intuitive interface to ePMV’s capabilities. We developed the GUI with the UI adaptor to be usable across all the supported hosts. IEEE Computer Graphics and Applications
55
Biomedical Applications
relative to the receptor. The host provides the method for handling collisions of rigid or soft bodies. Molecules treated as soft bodies can deform for an induced fit. Because ePMV interfaces with our augmented-reality software component,4 users can position the receptor and ligand via handheld markers tracked by a camera (see Figure 6b). Developing ePMV as a uPy plug-in (that is, using uPy to create GUI and geometry objects and to handle user interactions) lets ePMV run in a range of hosts. So, more potential users can access ePMV in their preferred host. uPy also facilitates ePMV maintenance and extension because the code duplication associated with a host-specific plug-in approach doesn’t occur.
GAMer
(a) ∆ Time PyRosetta simulation
Modeler moleculardynamics (MD) simulation
MD driver / Interactive MD protocol
Augmented reality
autoCell mesoscale modeling
∆ Time
Inverse kinematics / bones
Interactive rigid docking
Interactive flexible docking
(b) Figure 6. uPy allows deep plug-ins such as ePMV (an embedded version of Python Molecular Viewer) to support hundreds of functions and GUI options. (a) ePMV UI windows and subwindows inside Cinema 4D. The main window controls the current molecular representation. The subwindows expose options, detailed commands, and extensions. (b) ePMV allows interoperation of a variety of host and external algorithms on the same dataset. The left images show the initial state of an imported or constructed model; the right images show a change in the model state as induced by each labeled method.
When ePMV is instantiated in a host, the host’s computational capabilities become available for manipulating ePMV’s molecular data. For instance, we leveraged Cinema 4D’s bullet-based (http://bulletphysics.org) fast collision detection mechanism to implement interactive manual ligand-receptor docking. Here, ePMV provides the docking mechanism with which the user, guided by an “energy score,” moves the ligand molecule 56
September/October 2012
GAMer is a library that generates high-quality simplex meshes of surfaces and volumes.5 GAMer is used for improving meshes (coarsening and smoothing) and generating quality tetrahedral meshes (using TetGen, http://tetgen.berlios.de), for finite-element simulations. GAMer can serve as a stand-alone command-line tool. However, visual feedback during mesh improvement has proven helpful. Originally, we wrapped GAMer to make it callable from Python and integrated it as a dedicated plug-in for Blender 2.49b. With the new Blender 2.6 arriving and increased demand for GAMer by non-Blender users, we rewrote the plug-in only once using uPy to facilitate migration to Blender 2.5 and 2.6 and to make GAMer available in the other supported hosts. Figure 7 shows how we used GAMer on geometry acquired from manual segmentation of an electron tomography dataset from a mouse heart cell.6 Figure 7a shows portions of two subcellular features from the segmented surface mesh after it has been imported into Blender. On the left is part of the sarcoplasmic reticulum; on the right is part of a Ttubule. In Figure 7b, we applied the mesh improvement algorithms, both coarsening and smoothing the mesh. In Figure 7c, we merged the surface mesh with a surrounding box mesh. Merging a surface is usually a nontrivial task, but it’s easy in this case because Blender and other hosts fully support it. In Figure 7d we used tools exposed by the GAMer plug-in to mark and color regions on the surface mesh. For volumetric meshes, these regions become boundary domains, which you can use to declare boundary conditions in a finite-element computation (see the calcium simulation we describe later). GAMer communicates data (meshes) between the hosts and the GAMer module via the modeling helper. The UI adaptor exposes this process to the
(a)
(b)
(c)
(d)
(e)
Figure 7. GAMer (Geometry-Preserving Adaptive Mesher) improves polyhedral meshes. We (a) took a roughly segmented surface, (b) smoothed it, (c) merged it with a bounding box, and (d) used it to select a boundary region. We also used GAMer to segment different surfaces in (e) Blender 2.6 (top) and Blender 2.49b (bottom).
user to provide a comprehensive GUI that looks the same across all supported hosts. Figure 7e shows GAMer used for mesh improvements and boundary selection in Blender 2.6 and Blender 2.49b.
Tetra Meshes and boundaries generated with GAMer serve as the domain for simulating calcium (Ca 2+) diffusion in cardiac ventricular myocytes. With the recent availability of submicron-resolution myocyte structural data from confocal and electron microscopy,6 researchers have developed Csmol, a tool for modeling reaction-diffusion models for calcium in rat and rabbit myocytes.7 They used it to model half-sarcomere subdomains of ventricular myocytes and to identify inhomogeneities in calcium flux that depend on the cell’s geometric and topological features. Csmol results are usually analyzed using volumetric visualizers such as GMViewer or as projected data via Matlab. These applications provide graphical insight into the Ca2+ concentration as a function of time and position. However, they aren’t directly integrated into a common platform, complicating the workflow for the average scientific user. uPy let us rapidly create Tetra, a stand-alone application to facilitate the visualization of Csmol simulation results. Tetra’s core accesses the raw data from Csmol, produces the tetrahedral mesh,
and colors the mesh according to the compartment and the different Ca2+ concentration and flux values. Tetra’s prototype consists of a Python module that handles the Csmol data (that is, parsing) and communicates it to the modeling helper, which generates the colored tetrahedral representation of the simulated volume (parsed Csmol data). A dialog window developed using the UI adaptor exposes this workflow to users. The window lets users load different tetrahedral meshes, color them according to various properties, and split them on the basis of properties. Using uPy lets us switch between hosts at different stages of the analysis to exploit each host’s strengths. We perform the initial, interactive analysis in DejaVu,2 in which the availability of arbitrary 3D clipping planes and the ability to animate in vertex-based colors greatly facilitates analysis of calcium flux over time in the different compartments (see Figure 8 and the related video at http://youtu.be/wvs-nWE6ypo). Other hosts can then generate high-quality rendering to produce more effective images and advanced animations for publication and communication.
AutoFill AutoFill (http://autofill.scripps.edu) synthesizes 3D models of densely filled volumes and surfaces with arbitrarily shaped geometries (including procedurally IEEE Computer Graphics and Applications
57
Biomedical Applications
(b)
(a)
(c)
Figure 8. Using Tetra with Csmol (a Smoluchowski solver at the subcellular level) to visualize calcium dynamics simulations with tetrahedral meshing from GAMer. A cylindrical membrane feature (a T-tubule) from muscle cells is red to indicate high calcium, compared to the initial concentrations in blue. (a) A view in DejaVu of the cell interior using splitting tools and real-time cutting planes. (b) A view in Cinema 4D of the same cell surface and interior. This view is more usable for publication of a static image thanks to enhanced rendering. (c) A view of two perpendicular plane slices in Maya.
defined structures). AutoCell, a specialization of autoFill, models the biological mesoscale, an intermediate scale (10 –7 to 10 –8 m) between molecular and cellular biology that’s difficult to visualize with experimental methods. AutoCell packs molecular shapes into larger cellular frameworks while satisfying chemical and biological constraints. In general, autoFill positions ingredient objects into, onto, and around larger volumes or compartments, with various methods and degrees of control according to autoFill recipe files. The most common type of ingredient preparation requires users to generate a sphereTree (.sph) file. AutoFill uses this file for efficient collision detection between ingredients and containers. The sphereTree algorithm identifies a set of spheres approximating an object’s shape. We used uPy to create a plug-in that allows use of the sphereTree clustering algorithm on any 3D model type available in any host. This includes, for instance, models ■■ ■■
■■
generated by the host (see Figure 9a), generated by plug-ins such as ePMV (see Figure 9b), or imported from other formats that are highly useful but not traditionally accessible to molecularmodeling software.
With all the ingredients and recipes prepared and defined (shape, concentration, behavior, and so on), autoFill packs the ingredients into a speci58
September/October 2012
fied volume. During packing, autoFill produces a list of transformation matrices for each placed ingredient, which becomes the output file. To efficiently visualize the results, we developed a Python script that calls modeling-helper functions to ■■
■■
generate the master geometry for each unique ingredient and deposit visible instances of each ingredient’s geometry according to the autoFill output transformation matrices.
Ingredient geometries can be molecular meshes, simplified representations (for example, primitive spheres, cylinders, and boxes), curve points for fibrous molecules, and grid points for volumetric data (see Figure 10a). To help the visualization and development of autoFill analysis tools, we implemented uPy scripts for visualization of ■■
■■
■■
runtime packing and different object-coloring modes for debugging; size-dependent available volumes, distance heatmapping, packing order, and distribution graphs; and randomness statistics for the analysis of results.
Figure 10b shows examples. Using uPy to develop autoFill results-visualization tools let us exploit different hosts’ capa-
(a)
(b)
Figure 9. uPy exposes the capabilities of a Python sphereTree generator. (a) A tree of five spheres (one trunk with five branches) generated to approximate a torus geometry clustered stochastically about the coordinates of the torus’s vertices. (b) A tree of three spheres (one trunk with three branches) clustered about the atomic coordinates provided in the protein databank file 1CRN. We generated this tree to approximate the molecule’s shape in a coarse but efficient manner suitable for generating a coarse mesoscale model with autoCell (see Figure 10).
bilities. For instance, the DejaVu host, which is optimized for viewport interactivity, allows for real-time manipulation of a data-heavy model of HIV in full detail (see Figure 10c and the related video at http://youtu.be/vEOybMaRoKc, which shows the rotation of HIV). Other hosts proved useful to set up autoFill recipes—for example, to model certain ingredients—and to produce highquality ray-traced images and animations. High-quality host rendering options produce massive yet detailed images for large, dense volumes such as blood serum. To illustrate such volumes, which are difficult to visualize in real time, we used Cinema 4D to render a high-resolution GigaPan image, viewable at http://gigapan.org/ gigapans/85568. This zoomable Web interface shows a 1-µm2 cuboid filled with molecular detail from a blood serum recipe. Such images enable subjectmatter experts to more easily study, communicate about, and iteratively improve the model without having to repeatedly produce multiscale viewers.
Current Limitations Using uPy has raised some issues that must be considered for any project. All the hosts run into problems when displaying many spheres or polygonal meshes. The computer graphics community is well aware of this problem; solutions include distance-dependent level-of-detail adjustment, field-of-view culling, particles with billboard imposters, and low-polygon objects with normal textures applied. These techniques typically aren’t available in domain-specific modeling and visualization applications. However, the professional hosts already support many of these solutions and will likely be among the first to integrate new
techniques that will reduce these and other issues, minimizing the need for workarounds. The examples we described restricted the uPy API to the set of features common to all supported hosts. The number of common features will likely shrink as the number of supported hosts grows. As we described earlier, uPy currently supports basic types of geometric objects and a finite set of widgets, but we could extend uPy beyond these common primitives. For instance, only Maxon Cinema 4D natively supports Platonic solids. To overcome this limitation, we implemented a Platonic-solid modeler in the modeling-helper base class, thus making it available in all host-specific modeling-helper classes. The Cinema 4D subclass overrides this generic implementation to use its native one. This approach is applicable to a variety of functions missing in one or more supported hosts. Another example is that Blender and Cinema 4D provide noise functions, but the Maya Python API doesn’t. Again, the modeling helper can provide a generic implementation of such a missing function that the native versions can override. These implementations can be coded from scratch or based on third-party packages such as the Python Computer Graphics Kit (http://cgkit.sourceforge. net), which provides a complete noise module. Because the core functionality of some hosts such as Maya are available as third-party Python packages, uPy can help users embed Maya functionality into any other host. Currently, we can’t use uPy to develop plug-ins that require executing OpenGL code or that rely on reacting to mouse interactions in the host’s 3D viewport other than identifying currently selected IEEE Computer Graphics and Applications
59
Biomedical Applications
(a)
(b) Blender
Maya
DejaVu
Cinema 4D
(c) Figure 10. uPy interfaces with autoFill to construct, visualize, and analyze multiscale models of complex 3D volumes including biological cellscapes. (a) Cylinders populate the surfaces of a red-orange mesh and a yelloworange mesh. Sphereoids fill the orange mesh, pyramids fill the green mesh, and cuboids fill the space outside either volume while avoiding existing “P” meshes. AutoFill quickly positions these objects, in this case, with minimal overlap (as indicated by the user) to enhance speed. (b) uPy enables the interactive display of much of autoFill’s analysis output and many of its analysis tools. (c) A lightweight autoFill results viewer builds, hides, reveals, and replaces various components of a multiscale autoFill model (here, the HIV virus), directly from a result file. It uses the most efficient instancing systems available to each host to speed up viewport interaction and to reduce file size.
objects and object positions. However, we could add support for such events because almost all the supported hosts provide the necessary APIs. (The Cinema 4D Python API doesn’t currently support OpenGL code execution.) 60
September/October 2012
A
lthough uPy is written in pure Python, it encapsulates the hosts’ native calls. So, it doesn’t reduce performance compared to using the hosts’ Python APIs directly. uPy plug-ins’ usability in a variety of hosts has
three benefits. First, users can utilize their domainspecific plug-ins in the host that they’re the most familiar with. Second, the same plug-in can run in different hosts at different stages of a project pipeline (for example, for large studios with multiple hosts) or for scientific projects at different analysis stages (for example, one host might work better when interactivity is critical versus when high-quality rendering is the goal). Finally, as we mentioned before, the plug-in can have a broader user base. From the domain-specific methods-development viewpoint, uPy exposes computational methods in more environments without users having to write multiple plug-ins. From the 3D graphics software development viewpoint, uPy extends capabilities across the set of supported hosts (that is, implementing a feature in the uPy base class). From the application user’s viewpoint, uPy offers a choice among the hosts, with the possibility to reproduce work in any of the other supported hosts. We believe uPy can help strengthen and broaden the connections between the computer graphics community and the information visualization, scientific-visualization, visual-analytics, and other scientific communities by facilitating the development of new plug-ins and applications.
Acknowledgments Peter Kekenes-Huskey provided data and helped develop the Tetra plug-in. US National Center for Research Resources grant 5P41RR008605-19, US National Institute of General Medical Sciences grant 8 P41 GM103426-19, and US National Institutes of Health grant 8P41GM103426-19 (through the US National Institute of General Medical Sciences) supported this project. Also, a Research Council of Norway Centre of Excellence grant to the Simula Research Laboratory’s Centre for Biomedical Computing partly supported this research.
References 1. G. McGill, “Molecular Movies ... Coming to a Lecture Near You,” Cell, vol. 133, no. 7, 2008, pp. 1127–1132. 2. M.F. Sanner, “Python: A Programming Language for Software Integration and Development,” J. Molecular Graphics and Modeling, vol. 17, no. 1, 1999, pp. 57–61. 3. G.T. Johnson et al., “ePMV Embeds Molecular Modeling into Professional Animation Software Environments,” Structure, vol. 19, no. 3, 2011, pp. 293–303. 4. A. Gillet et al., “Tangible Interfaces for Structural Molecular Biology,” Structure, vol. 13, no. 3, 2005, pp. 483–491. 5. Z. Yu et al., “Feature-Preserving Adaptive Mesh
Generation for Molecular Shape Modeling and Simulation,” J. Molecular Graphics and Modeling, vol. 26, no. 8, 2008, pp. 1370–1380. 6. T. Hayashi et al., “Three-Dimensional Electron Microscopy Reveals New Details of Membrane Systems for Ca 2+ Signaling in the Heart,” J. Cell Science, vol. 122, no. 7, 2009, pp. 1005–1013. 7. Y.H. Cheng et al., “Finite Element Analysis of the Time-Dependent Smoluchowski Equation for Acetylcholinesterase Reaction Rate Calculations,” Biophysical J., vol. 92, no. 10, 2007, pp. 3397–3406. Ludovic Autin is a postdoc at the Scripps Research Institute’s Molecular Graphics Laboratory. His research employs computer graphics to better model macromolecules across a range of scales. Autin has a PhD in molecular modeling from Denis Diderot University. Contact him at [email protected]. Graham Johnson is a medical illustrator and a qb3@UCSF Fellow at the University of California, San Diego. He specializes in molecular and cellular biology. He illustrated the textbook Cell Biology. Johnson received his PhD in biophysics from the Scripps Research Institute. Contact him at [email protected]. Johan Hake is a postdoc in the Bioengineering Department at the University of California, San Diego. He’s also a research scientist at the Simula Research Laboratory’s Center for Biomedical Computing. His research employs computational methods to better understand the excitation-contraction coupling in healthy and pathological cardiac myocytes. Hake has a PhD in informatics from the University of Oslo. Contact him at [email protected]. Arthur Olson is a professor in the Scripps Research Institute’s Department of Molecular Biology. His research focuses on developing and applying computational methods for visualization and analysis of biomolecular interactions. Olson has a PhD in physical chemistry from the University of California, Berkeley and was a postdoctoral fellow at Harvard University. He’s a member of the IEEE Computer Society, ACM Siggraph, the American Chemical Society, the American Crystallographic Association, and the Protein Society. Contact him at [email protected]. Michel Sanner is an associate professor in the Scripps Research Institute’s Department of Molecular Biology. His research interest is component-based software development, emphasizing 3D visualization. Sanner has a PhD in computer science from the University of Haute-Alsace. He’s a member of the IEEE Computer Society and ACM Siggraph. Contact him at [email protected]. Selected CS articles and columns are also available for free at http://ComputingNow.computer.org. IEEE Computer Graphics and Applications
61
