Home
Search
Collections
Journals
About
Contact us
My IOPscience
Refraction corrected calibration for aquatic locomotion research: application of Snell’s law improves spatial accuracy
This content has been downloaded from IOPscience. Please scroll down to see the full text. 2015 Bioinspir. Biomim. 10 046009 (http://iopscience.iop.org/1748-3190/10/4/046009) View the table of contents for this issue, or go to the journal homepage for more
Download details: IP Address: 134.129.120.3 This content was downloaded on 21/07/2015 at 13:23
Please note that terms and conditions apply.
Bioinspir. Biomim. 10 (2015) 046009
doi:10.1088/1748-3190/10/4/046009
PAPER
RECEIVED
6 February 2015 REVISED
1 June 2015
Refraction corrected calibration for aquatic locomotion research: application of Snell’s law improves spatial accuracy
ACCEPTED FOR PUBLICATION
Sebastian Henrion1, Cees W Spoor1, Remco P M Pieters1, Ulrike K Müller2 and Johan L van Leeuwen1
8 June 2015
1
PUBLISHED
7 July 2015
2
Experimental Zoology, Department of Animal Sciences, Wageningen University, The Netherlands Department of Biology, California State University Fresno, USA
E-mail: [email protected] Keywords: refraction calibration, 3D reconstruction, image analysis, biomechanics, seahorse, DLT Supplementary material for this article is available online
Abstract Images of underwater objects are distorted by refraction at the water–glass–air interfaces and these distortions can lead to substantial errors when reconstructing the objects’ position and shape. So far, aquatic locomotion studies have minimized refraction in their experimental setups and used the direct linear transform algorithm (DLT) to reconstruct position information, which does not model refraction explicitly. Here we present a refraction corrected ray-tracing algorithm (RCRT) that reconstructs position information using Snell’s law. We validated this reconstruction by calculating 3D reconstruction error—the difference between actual and reconstructed position of a marker. We found that reconstruction error is small (typically less than 1%). Compared with the DLT algorithm, the RCRT has overall lower reconstruction errors, especially outside the calibration volume, and errors are essentially insensitive to camera position and orientation and the number and position of the calibration points. To demonstrate the effectiveness of the RCRT, we tracked an anatomical marker on a seahorse recorded with four cameras to reconstruct the swimming trajectory for six different camera configurations. The RCRT algorithm is accurate and robust and it allows cameras to be oriented at large angles of incidence and facilitates the development of accurate tracking algorithms to quantify aquatic manoeuvers.
1. Introduction Visual recordings and play-back of organismal movement have always played a large role in biomechanics and behavioral research, and leaps in our understanding were often driven by technological advances in imaging (e.g. reviews by Müller and van Leeuwen 2006, Lauder 2011, Woo and Rieucau 2011). Many organisms move and deform their body in three-dimensions (e.g. Tytell et al 2008, Lauder 2011). To quantify deformation, orientation, and position of an organism’s body while it is moving often requires recording the organism from three to ten viewpoints to avoid losing information due to occlusion—part of the organism being obscured by its own body. Threedimensional reconstruction of body shapes from video recordings is well established for humans (e.g. Carranza et al 2003, De Aguiar et al 2008), and is developing for animals (e.g. Fontaine et al 2009, © 2015 IOP Publishing Ltd
Ristroph et al 2009, Zelman et al 2009, Brainerd et al 2010). Any reconstruction of three-dimensional shapes from two-dimensional images requires mapping of volume elements onto image planes. The more calibration points (points in the object volume with known spatial coordinates) are mapped onto the image planes of all cameras, the larger the fraction of object volume elements whose position is known a priori and therefore does not need to be interpolated or extrapolated based on the position of calibrated volume elements. The need to interpolate can be eliminated altogether once the density of calibration points is greater than the density of the volume elements as defined by the detector pixels projected onto the object planes. Such a complete mapping of the entire object volume onto all pixels in the image planes would yield transfer functions that are agnostic of optical theory and of the experimental set-up.
Bioinspir. Biomim. 10 (2015) 046009
S Henrion et al
However, this process is impractically laborious. So experimentalists map only a few points within the object volume onto all image planes. The transfer functions in this case are based on a small subset of points within the volume. From this set of calibration points, it is possible to reconstruct the position of any point within the volume by using an algorithm that predicts its position based on its position in the available image planes. The calibration and reconstruction algorithm most widely used by biologists is the direct linear transform algorithm (DLT). This algorithm assumes a pinhole camera and collinearity (Kwon and Casebolt 2006): light travels through a single point (pin hole) and in a straight line (collinearity) (Hartey and Zisserman 2004). The DLT algorithm is agnostic about Snell’s law—it simply minimizes an algebraic distance as its cost function (Hartey and Zisserman 2004, Kwon and Casebolt 2006)—and ‘so does not necessarily result in a solution that minimizes a meaningful and statistically well-behaved geometric distance’ (Walker et al 2009). If we assume a pinhole camera, which neglects all lens distortions, then the collinearity principle holds when imaging objects in air, but it breaks down when imaging objects through media that cause refraction. Although DLT is able to reconstruct the position of points with great accuracy in air, it is likely to be considerably less accurate when the imaged object is submerged (e.g. Kwon and Casebolt 2006). Images of submerged objects are distorted by refraction, and these distortions can cause considerable reconstruction errors. Image distortion occurs when oblique light rays are refracted at the interface of two media. Light is refracted more strongly the larger the difference in refractive index between the media. In our experiments, light is refracted at two interfaces, air–glass and glass–water. When imaging a submerged object through a water–glass–air interface onto an image plane, this image misrepresents the object’s position and shape (Luria and Kinney 1970, Chang and Chen 2011, Sedlazeck and Koch 2011a, Kang et al 2012, Treibitz et al 2012), and the positional error (difference between recorded and actual position) can be as much as 20% (Kunz and Singh 2008, Kang et al 2012, Jordt-Sedlazeck and Koch 2012). Refraction errors can either be minimized by hardware or corrected by software. Refraction can be reduced by using water-filled prisms that keep optical axes perpendicular to the tank wall (Prasad 2000, Sakakibara et al 2004) or by filming with a camera submerged in a dome (Walton 1988, reviewed in Kwon and Casebolt, 2006, Sedlazeck and Koch 2011a). Refraction can be corrected after the fact by applying image–correction algorithms (reviewed in Kwon and Casebolt 2006, and Jordt 2014). Many of these algorithms have been developed using 2D and/or virtual image sceneries, and focus on numerical stability and convergence 2
criteria (e.g. Ke et al 2008, Kang et al 2012, Treibitz et al 2012, Belden 2013) rather than on quantifying 3D reconstruction errors (e.g. Agrawal et al 2012, Yau et al 2013, Huang et al 2014, Kawahara et al 2014). Currently available hardware solutions severely limit camera placement (Yekutiele et al 2007, Flammang et al 2013, Sefati et al 2013). Currently available software solutions are not designed with end users in mind who want to study aquatic locomotion and do not provide their algorithms as user-ready software (e.g. Gedge et al 2011, Treibitz et al 2012, Chen and Yang 2014, Dolereit and Kuijper 2014, Haner et al 2015). To track the motion of multiple, highly deformable, low-contrast objects such as fish fins during complex manoeuvers involving changes of body orientation and shape, we developed a refraction corrected ray-tracing (RCRT) calibration algorithm. By explicitly accounting for refraction, this algorithm reconstructs three-dimensional position information from multiple two-dimensional images. This algorithm maintains high spatial accuracy even when cameras are oriented at angles of incidence to tank walls that result in large refractive distortions. To evaluate our algorithm as a tool to reconstruct three-dimensional movements of aquatic organisms, we achieved the following four objectives. (1)We developed a RCRT calibration algorithm for single and stereo camera setups recording underwater movements through a tank wall. Furthermore, we developed a basic reconstruction tool for two calibrated cameras to reconstruct a point in space. (2)We performed a sensitivity analysis of the 3D reconstruction error for different camera configurations (stereo angle and average viewing angle), different numbers of calibration points and positions of calibration points within the measurement volume and different methods of calculating the 3D reconstruction error (on calibration points or non-calibration points, further referred to as object points). (3)We compared the accuracy of the RCRT algorithm with the DLT algorithm, the main algorithm currently used in animal motion studies for 3D reconstructions. We discuss the advantages and shortcomings of both approaches. (4)We compared the tracking results of an anatomical marker on a seahorse of six different stereo setups during a forward swimming manoeuvre. This example demonstrates the effectiveness of our RCRT algorithm for tracking animal motion. We used the DLT algorithm as implemented in Matlab (The Mathworks) by Hedrick (2008).
Bioinspir. Biomim. 10 (2015) 046009
S Henrion et al
Figure 1. Recording setups for the 3D reconstruction error sensitivity analysis (A) and the seahorse tracking (B). (A, main and left) Images of the calibration frame were taken from nine evenly spaced camera positions at viewing angles ranging from −60° over 0° (normal to the glass interface of the tank) to +60° in steps of 15°. (A, insert) Calibration frame used for both recording setups in A and B. (A, bottom) Top view of the image setup showing the camera rotation required to keep the calibration frame in the middle of the field of view (FOV). (B) Four high-speed cameras tracking a swimming seahorse with cameras 1 and 2 (Mikrotrons) (stereo angle 60°) imaging through a glass interface of 4 mm, cameras 3 and 4 (Photron PCI and APX) imaging perpendicular through glass interfaces of 6 mm. Inserts show the tracked marker (yellow) on the seahorse’s dorsal side and the calibration-frame image for each camera.
2. Materials and methods 2.1. Recording setups Two recording setups were used to analyze the sensitivity of the 3D reconstruction error and the tracking of an anatomical marker on a seahorse during a forward swimming manoeuvre. The first setup was used for the sensitivity analysis of the 3D reconstruction error. It comprised a Photron APX-RS (Photron Europe Ltd) (1024*1024 pixels) with a 50 mm Nikkor AIS lens, f/8, figure 1(A), recording a calibration object submerged in a waterfilled tank over a range of camera orientations relative to the tank wall. At a diaphragm of f/8, the complete calibration object was within the depth of field, even when the camera’s optical axis was oriented at steep angles of incidence to the tank wall. To explore the effect of camera orientation on reconstruction error, the camera was placed at nine evenly spaced angles of incidence ranging from −60° over 0° (normal to the glass interface of the tank) to +60° in steps of 15°, such that the optical axes for nine different viewing angles all intersected at the origin of the calibration object, figure 1(A, top view). This confluence of all optical axes in one point was achieved only in setup 1 for the sensitivity analysis and was not a necessary experimental condition for the RCRT algorithm, as evident in setup 2 described below. The images recorded from each camera angle can be seen to the left of the camera 3
setup in figure 1(A). A movie of the nine calibration images that shows the deformation due to refraction can be found in supplementary material A. The second recording setup was used to reconstruct the swimming path of a seahorse from four camera views. Two cameras were placed in a stereo configuration (stereo angle ∼60°) looking through the same interface; two cameras were placed orthogonally to two different glass interfaces. The stereo configuration consisted of two Mikrotron EoSens M—1362 full base cameras (Mikrotron GmbH, Ahrensburg, Germany) equipped with 28 mm Nikkor AIS lenses, diaphragm f/8, figure 1(B, cameras 1 and 2). Each camera was connected to a PixCi E4/E8 framegrabber (Epix, Buffalo Groove, IL, USA). The recording software was XCAP Standard v3.7 (Epix, Buffalo Groove, IL, USA). The cameras were synchronized using the 5 V TTL signal from a pulse generator that was converted into a LVDS signal via a custom made TTL module by engineers of Epix for the Mikrotron cameras. The orthogonally placed cameras were Photron cameras (APX and PCI 1024) equipped with 50 mm Nikkor AIS lenses, diaphragm f/5.6, figure 1(B, cameras 3 and 4). The cameras were synchronized by a Quantum Composer 9600+ Series Digital Pulse Delay Generator (Quantum Composers, Bozeman, MT, USA). The two Mikrotron cameras recorded at 400 fps (resolution 700*1024 pixels) through a 4 mm glass interface of the aquarium. The Photron cameras recorded at 800 fps (resolution
Bioinspir. Biomim. 10 (2015) 046009
S Henrion et al
Figure 2. Light rays refract at the boundaries of different media—air, glass and seawater (A) and (B). (A) Schematic overview of a stereo setup with two cameras looking through the same interface. (B) Light ray paths for two points from 2D image plane pixel coordinate to 3D calibration marker refracting at the boundaries of air, glass and seawater (r1 and r2). P: principal point, D: projection distance, C: camera center.
of 1024*1024 pixels), through a 6 mm thick glass interface. All four cameras were calibrated on the same 28 calibration points. The seahorse was tracked by digitizing an arbitrarily chosen point on its ‘neck’ because this point was visible to all four cameras, figure 1(B, inserts: yellow marker) every 0.0125 s (every five frames for the Mikrotrons, every ten frames for the Photrons). We reconstructed the xyz coordinates of the traveled path from six pairs of cameras. One pair comprised two orthogonal cameras (APX– PCI), four pairs comprised one angled and one orthogonal camera (APX–Mik 1, PCI–Mik 1, APX–Mik 2, PCI–Mik 2) and the last pair was a stereo setup (Mik 1 and 2). For the tracking with the first five camera combinations, we used the individual calibrations of each camera separately. For the tracking with the stereo setup (Mik 1 and 2), we calibrated the two cameras simultaneously. Videos of the swimming seahorse from each camera can be viewed in supplementary material B. The complete source code and user manual for the calibration and tracking of two sequences (APX–PCI and Mik 1 and 2) can be found in supplementary material C. 2.2. Calibration frame design We used a 3D printed calibration frame with 50 known marker locations designed in Solidworks (Dassault Systems) and printed by Shapeways (Eindhoven, The Netherlands), figure 1(A, insert) (Solidworks file provided in supplementary material E). The 4
calibration frame was made from laser-sintered nylon plastic, a low cost, high strength, high accuracy (with respect to our purposes) and water-proof material. To circumvent printing inaccuracies, we scanned our calibration frame with an x-ray CT scanner (XRT Phoenix v[tome]x m, GE) with a voxel size resolution of 1.95 × 10−4 mm3 (cube with vertices of 58 μm) and used the coordinate positions of the calibration markers from the x-ray scan. The average difference between the coordinate positions in the Solidworks file versus the positions according to the CT scan of the actual calibration object was 1 mm, respectively 0.62, 0.45 and 0.42 mm in x, y, z direction. The calibration frame dimensions were 80 by 60 by 60 mm, filling approximately one third of the cameras’ field of view, figure 2(A) in experimental setup 1 (sensitivity analysis) and nearly the complete measurement volume in the second setup to track the seahorse, figure 1(B). 2.3. Ray tracing algorithm This section explains the conceptual design of the RCRT algorithm. The algorithm reconstructs light ray paths for each point in the recorded volume. The algorithm uses geometric optics laws—it approximates light as rays and ignores the wave nature of light. However, the algorithm does not use a paraxial approximation—it does not assume that objects on the optical axis of the camera are small enough that the parts of the object that are off-axis experience no
Bioinspir. Biomim. 10 (2015) 046009
S Henrion et al
distortion; instead, all object points are treated as affected by refraction. The RCRT algorithm reconstructs the camera positions, and then uses those reconstructed camera positions to determine the positions of non-calibration points. To reconstruct camera position, the algorithm requires as input calibration points as well as information about refraction events (Austin and Halikas 1976). This information is currently provided by the user in the form of refractive indices for the media through which light travels in our setup (water, glass, air), figure 2. The RCRT algorithm then uses these inputs to start an optimization procedure that traces the light rays from camera to the object. To trace the best set of light rays, the algorithm minimizes the socalled residual—the shortest distance between a 3D calibration marker and its corresponding light ray— for a number of calibration points. During this minimization process, intrinsic and extrinsic camera parameters and setup parameters are optimized (Treibitz et al 2012). The two intrinsic parameters are: (1) the principal point (P, figure 2(B))—the orthogonal projection of the camera center onto the image sensor (often referred to as optical axis of the camera), and (2) projection distance (D, figure 2(B)) (the distance between the camera center and principal point). The extrinsic parameters are the position of the camera center and the camera orientation with respect to the calibration volume coordinate axes (C, figure 2(B)). The setup parameters are the glass interface position and orientation. The algorithm requires two sets of inputs; first, a known set of 3D calibration points distributed across the measurement volume, and their respective 2D image coordinates to set up a correspondence between 2D images and 3D positional data; second, estimates plus lower and upper bounds of the aforementioned intrinsic and extrinsic parameters to initiate and constrain the optimization process. We use a constrained nonlinear multivariable function solver (fmincon, Matlab) to find the set of parameter values that minimize the residual within the parameters’ upper and lower bounds. In our optimization scheme, we excluded the optimization of the refractive indices of the media because we knew their values (1, 1.5 and 1.349, respectively). However, if the refractive index of water is unknown, the user can adapt the software such that the refractive index is included in the optimization process. The source code, user manual, and a detailed algebraic derivation of the RCRT algorithm are included in supplementary material C. Appropriate instructions on the estimation of initial camera parameters are given in the user manual accompanying the source code and can be found in supplementary material C. The code allows users to skip or automate certain steps, such as finding marker locations, and some initialization steps (e.g. lens distortion parameters). When filming through e.g. wideangle or zoom lenses, the user should correct lens 5
distortion using the Caltech camera calibration toolbox (Bouguet 1999) before applying the RCRT algorithm. 2.4. Sensitivity analysis We recorded images of a calibration frame with 50 known marker positions from nine camera orientations. We paired these nine orientations to define camera pairs, resulting in eight stereo angles and eight viewing angles and a total of 20 camera configurations. The stereo angle is defined as the angle between the camera pair, the viewing angle for the camera pair is defined as the average viewing angle of both cameras. Depending on camera configuration, the number of calibration points visible in both camera views ranged from 26 to 38 points, figure D1, supplementary material D. Our goal was to estimate the prediction error of our algorithm, called the 3D reconstruction error in the context of visual reconstruction. To estimate the 3D reconstruction error, we used between 2 and 24 calibration points, leaving between 2 and 36 calibration points to be used as object or validation points. We then explored the sensitivity of the 3D reconstruction error to viewing angle, stereo angle, as well as the number and position of calibration points. For the sensitivity analysis, we randomly sampled 100 times a given number of calibration points (2 to 24 points). From the remaining object points, we then calculated the mean reconstruction error, ε¯, and its standard deviation, σ (every black dot in figure 3: the mean reconstruction error of 100 calibrations; squares and circles in figure 4: mean or standard deviation of 100 calibrations for DLT and RCRT, respectively). This procedure means that we performed 2000 calibrations per ‘number of calibration points’ (100 calibrations for 20 camera configurations) and 46 000 calibrations for the entire sensitivity analysis (100 calibrations for 20 camera configurations for 23 numbers of calibration points). To compare the performance of the DLT and the RCRT algorithms, we used the same sets of randomly selected points rather than performing separate sets of random selections for each algorithm. This sensitivity analysis allowed us to test the stability and accuracy of our algorithm and to compare its performance to the DLT algorithm. To test stability and accuracy, we used the validation points. Our RCRT algorithm uses a constrained nonlinear multivariable function solver (fmincon, Matlab). This optimization tool is unable to distinguish between local and absolute optima and therefore might find a suboptimal solution, which would yield a smaller reconstruction error for calibration than validation points. So to test acccuracy, we compared the prediction error based on calibration points (commonly used to estimate prediction error) with the actual prediction error (=prediction error based on validation points). To test stability, we determined the standard deviation of the
Bioinspir. Biomim. 10 (2015) 046009
S Henrion et al
Figure 3. RCRT calibration shows a homogenous 3D reconstruction error over a wide range of stereo and viewing angle combinations. (A)– (F) Mean 3D reconstruction error, ε¯, of the RCRT algorithm (A)–(C) and DLT algorithm (D)–(F) for 20 stereo, α, and viewing angle, β, combinations with 6, 15 and 24 calibration points. Panels (A), (B), (C), (E), (F) use the color bar on the right, (D) uses the color bar next to the plot. Each black dot represents an average reconstruction error of 100 calibrations on a fixed number (indicated in gray in every graph) of randomly selected calibration points from the shared available points of the stereo pair of images.
prediction error; a large standard deviation, σ, of the mean prediction error, ε¯, indicates that the prediction error is highly sensitive to the random choice of validation points.
2.5. Animal care Seahorses were acquired from a specialist farm (Seahorse Australia, Tasmania, Australia) under CITES No. 12NL118877/11. Seahorses (Hippocampus abdominalis) were kept in two connected 300 l marine aquaria (32 ppt at 17 °C) on a 12 h:12 h light:dark cycle. Salt water was made from CoralSalt plus deionized water, and was aerated for at least 72 h before use. Each aquarium was equipped with a protein skimmer, fluid bed filter and bio filter in order to maintain a stable bacteria culture. Aquaria were lit by four 24 W AquaBlue special lights mounted in an ATI Sunpower Marine Pendant. Seahorses were fed daily on live mysidae shrimp. The animals were allowed to acclimate for three months before experiments started. The animals were recorded in a special filming section of the holding tank to reduce the stress of experimental conditions and to avoid transportation. The experiments were approved by the Animal Ethics Committee of Wageningen University. 6
3. Results In this section, we show a sensitivity analysis of the 3D reconstruction error obtained from both the RCRT approach and DLT algorithm. We explored the sensitivity of both algorithms to camera orientation and the number of calibration points used by the algorithm for calibration. By using fewer than the available calibration points for calibration, we explored how well each algorithm was able to predict the positions of object points. Finally, we compared the 3D reconstructed swimming path of a seahorse based on six different camera configurations to show the efficacy of our refraction based algorithm. The RCRT algorithm generated consistently low mean 3D reconstruction errors, ε¯, for a wide range of camera orientations and numbers of calibration points. In figure 3, we show the error for 20 different cameras configurations for three different numbers of calibration points (6, 15 and 24 calibration points). In the 20 camera configurations that we examined the error remained below 0.6 mm (
Search
Collections
Journals
About
Contact us
My IOPscience
Refraction corrected calibration for aquatic locomotion research: application of Snell’s law improves spatial accuracy
This content has been downloaded from IOPscience. Please scroll down to see the full text. 2015 Bioinspir. Biomim. 10 046009 (http://iopscience.iop.org/1748-3190/10/4/046009) View the table of contents for this issue, or go to the journal homepage for more
Download details: IP Address: 134.129.120.3 This content was downloaded on 21/07/2015 at 13:23
Please note that terms and conditions apply.
Bioinspir. Biomim. 10 (2015) 046009
doi:10.1088/1748-3190/10/4/046009
PAPER
RECEIVED
6 February 2015 REVISED
1 June 2015
Refraction corrected calibration for aquatic locomotion research: application of Snell’s law improves spatial accuracy
ACCEPTED FOR PUBLICATION
Sebastian Henrion1, Cees W Spoor1, Remco P M Pieters1, Ulrike K Müller2 and Johan L van Leeuwen1
8 June 2015
1
PUBLISHED
7 July 2015
2
Experimental Zoology, Department of Animal Sciences, Wageningen University, The Netherlands Department of Biology, California State University Fresno, USA
E-mail: [email protected] Keywords: refraction calibration, 3D reconstruction, image analysis, biomechanics, seahorse, DLT Supplementary material for this article is available online
Abstract Images of underwater objects are distorted by refraction at the water–glass–air interfaces and these distortions can lead to substantial errors when reconstructing the objects’ position and shape. So far, aquatic locomotion studies have minimized refraction in their experimental setups and used the direct linear transform algorithm (DLT) to reconstruct position information, which does not model refraction explicitly. Here we present a refraction corrected ray-tracing algorithm (RCRT) that reconstructs position information using Snell’s law. We validated this reconstruction by calculating 3D reconstruction error—the difference between actual and reconstructed position of a marker. We found that reconstruction error is small (typically less than 1%). Compared with the DLT algorithm, the RCRT has overall lower reconstruction errors, especially outside the calibration volume, and errors are essentially insensitive to camera position and orientation and the number and position of the calibration points. To demonstrate the effectiveness of the RCRT, we tracked an anatomical marker on a seahorse recorded with four cameras to reconstruct the swimming trajectory for six different camera configurations. The RCRT algorithm is accurate and robust and it allows cameras to be oriented at large angles of incidence and facilitates the development of accurate tracking algorithms to quantify aquatic manoeuvers.
1. Introduction Visual recordings and play-back of organismal movement have always played a large role in biomechanics and behavioral research, and leaps in our understanding were often driven by technological advances in imaging (e.g. reviews by Müller and van Leeuwen 2006, Lauder 2011, Woo and Rieucau 2011). Many organisms move and deform their body in three-dimensions (e.g. Tytell et al 2008, Lauder 2011). To quantify deformation, orientation, and position of an organism’s body while it is moving often requires recording the organism from three to ten viewpoints to avoid losing information due to occlusion—part of the organism being obscured by its own body. Threedimensional reconstruction of body shapes from video recordings is well established for humans (e.g. Carranza et al 2003, De Aguiar et al 2008), and is developing for animals (e.g. Fontaine et al 2009, © 2015 IOP Publishing Ltd
Ristroph et al 2009, Zelman et al 2009, Brainerd et al 2010). Any reconstruction of three-dimensional shapes from two-dimensional images requires mapping of volume elements onto image planes. The more calibration points (points in the object volume with known spatial coordinates) are mapped onto the image planes of all cameras, the larger the fraction of object volume elements whose position is known a priori and therefore does not need to be interpolated or extrapolated based on the position of calibrated volume elements. The need to interpolate can be eliminated altogether once the density of calibration points is greater than the density of the volume elements as defined by the detector pixels projected onto the object planes. Such a complete mapping of the entire object volume onto all pixels in the image planes would yield transfer functions that are agnostic of optical theory and of the experimental set-up.
Bioinspir. Biomim. 10 (2015) 046009
S Henrion et al
However, this process is impractically laborious. So experimentalists map only a few points within the object volume onto all image planes. The transfer functions in this case are based on a small subset of points within the volume. From this set of calibration points, it is possible to reconstruct the position of any point within the volume by using an algorithm that predicts its position based on its position in the available image planes. The calibration and reconstruction algorithm most widely used by biologists is the direct linear transform algorithm (DLT). This algorithm assumes a pinhole camera and collinearity (Kwon and Casebolt 2006): light travels through a single point (pin hole) and in a straight line (collinearity) (Hartey and Zisserman 2004). The DLT algorithm is agnostic about Snell’s law—it simply minimizes an algebraic distance as its cost function (Hartey and Zisserman 2004, Kwon and Casebolt 2006)—and ‘so does not necessarily result in a solution that minimizes a meaningful and statistically well-behaved geometric distance’ (Walker et al 2009). If we assume a pinhole camera, which neglects all lens distortions, then the collinearity principle holds when imaging objects in air, but it breaks down when imaging objects through media that cause refraction. Although DLT is able to reconstruct the position of points with great accuracy in air, it is likely to be considerably less accurate when the imaged object is submerged (e.g. Kwon and Casebolt 2006). Images of submerged objects are distorted by refraction, and these distortions can cause considerable reconstruction errors. Image distortion occurs when oblique light rays are refracted at the interface of two media. Light is refracted more strongly the larger the difference in refractive index between the media. In our experiments, light is refracted at two interfaces, air–glass and glass–water. When imaging a submerged object through a water–glass–air interface onto an image plane, this image misrepresents the object’s position and shape (Luria and Kinney 1970, Chang and Chen 2011, Sedlazeck and Koch 2011a, Kang et al 2012, Treibitz et al 2012), and the positional error (difference between recorded and actual position) can be as much as 20% (Kunz and Singh 2008, Kang et al 2012, Jordt-Sedlazeck and Koch 2012). Refraction errors can either be minimized by hardware or corrected by software. Refraction can be reduced by using water-filled prisms that keep optical axes perpendicular to the tank wall (Prasad 2000, Sakakibara et al 2004) or by filming with a camera submerged in a dome (Walton 1988, reviewed in Kwon and Casebolt, 2006, Sedlazeck and Koch 2011a). Refraction can be corrected after the fact by applying image–correction algorithms (reviewed in Kwon and Casebolt 2006, and Jordt 2014). Many of these algorithms have been developed using 2D and/or virtual image sceneries, and focus on numerical stability and convergence 2
criteria (e.g. Ke et al 2008, Kang et al 2012, Treibitz et al 2012, Belden 2013) rather than on quantifying 3D reconstruction errors (e.g. Agrawal et al 2012, Yau et al 2013, Huang et al 2014, Kawahara et al 2014). Currently available hardware solutions severely limit camera placement (Yekutiele et al 2007, Flammang et al 2013, Sefati et al 2013). Currently available software solutions are not designed with end users in mind who want to study aquatic locomotion and do not provide their algorithms as user-ready software (e.g. Gedge et al 2011, Treibitz et al 2012, Chen and Yang 2014, Dolereit and Kuijper 2014, Haner et al 2015). To track the motion of multiple, highly deformable, low-contrast objects such as fish fins during complex manoeuvers involving changes of body orientation and shape, we developed a refraction corrected ray-tracing (RCRT) calibration algorithm. By explicitly accounting for refraction, this algorithm reconstructs three-dimensional position information from multiple two-dimensional images. This algorithm maintains high spatial accuracy even when cameras are oriented at angles of incidence to tank walls that result in large refractive distortions. To evaluate our algorithm as a tool to reconstruct three-dimensional movements of aquatic organisms, we achieved the following four objectives. (1)We developed a RCRT calibration algorithm for single and stereo camera setups recording underwater movements through a tank wall. Furthermore, we developed a basic reconstruction tool for two calibrated cameras to reconstruct a point in space. (2)We performed a sensitivity analysis of the 3D reconstruction error for different camera configurations (stereo angle and average viewing angle), different numbers of calibration points and positions of calibration points within the measurement volume and different methods of calculating the 3D reconstruction error (on calibration points or non-calibration points, further referred to as object points). (3)We compared the accuracy of the RCRT algorithm with the DLT algorithm, the main algorithm currently used in animal motion studies for 3D reconstructions. We discuss the advantages and shortcomings of both approaches. (4)We compared the tracking results of an anatomical marker on a seahorse of six different stereo setups during a forward swimming manoeuvre. This example demonstrates the effectiveness of our RCRT algorithm for tracking animal motion. We used the DLT algorithm as implemented in Matlab (The Mathworks) by Hedrick (2008).
Bioinspir. Biomim. 10 (2015) 046009
S Henrion et al
Figure 1. Recording setups for the 3D reconstruction error sensitivity analysis (A) and the seahorse tracking (B). (A, main and left) Images of the calibration frame were taken from nine evenly spaced camera positions at viewing angles ranging from −60° over 0° (normal to the glass interface of the tank) to +60° in steps of 15°. (A, insert) Calibration frame used for both recording setups in A and B. (A, bottom) Top view of the image setup showing the camera rotation required to keep the calibration frame in the middle of the field of view (FOV). (B) Four high-speed cameras tracking a swimming seahorse with cameras 1 and 2 (Mikrotrons) (stereo angle 60°) imaging through a glass interface of 4 mm, cameras 3 and 4 (Photron PCI and APX) imaging perpendicular through glass interfaces of 6 mm. Inserts show the tracked marker (yellow) on the seahorse’s dorsal side and the calibration-frame image for each camera.
2. Materials and methods 2.1. Recording setups Two recording setups were used to analyze the sensitivity of the 3D reconstruction error and the tracking of an anatomical marker on a seahorse during a forward swimming manoeuvre. The first setup was used for the sensitivity analysis of the 3D reconstruction error. It comprised a Photron APX-RS (Photron Europe Ltd) (1024*1024 pixels) with a 50 mm Nikkor AIS lens, f/8, figure 1(A), recording a calibration object submerged in a waterfilled tank over a range of camera orientations relative to the tank wall. At a diaphragm of f/8, the complete calibration object was within the depth of field, even when the camera’s optical axis was oriented at steep angles of incidence to the tank wall. To explore the effect of camera orientation on reconstruction error, the camera was placed at nine evenly spaced angles of incidence ranging from −60° over 0° (normal to the glass interface of the tank) to +60° in steps of 15°, such that the optical axes for nine different viewing angles all intersected at the origin of the calibration object, figure 1(A, top view). This confluence of all optical axes in one point was achieved only in setup 1 for the sensitivity analysis and was not a necessary experimental condition for the RCRT algorithm, as evident in setup 2 described below. The images recorded from each camera angle can be seen to the left of the camera 3
setup in figure 1(A). A movie of the nine calibration images that shows the deformation due to refraction can be found in supplementary material A. The second recording setup was used to reconstruct the swimming path of a seahorse from four camera views. Two cameras were placed in a stereo configuration (stereo angle ∼60°) looking through the same interface; two cameras were placed orthogonally to two different glass interfaces. The stereo configuration consisted of two Mikrotron EoSens M—1362 full base cameras (Mikrotron GmbH, Ahrensburg, Germany) equipped with 28 mm Nikkor AIS lenses, diaphragm f/8, figure 1(B, cameras 1 and 2). Each camera was connected to a PixCi E4/E8 framegrabber (Epix, Buffalo Groove, IL, USA). The recording software was XCAP Standard v3.7 (Epix, Buffalo Groove, IL, USA). The cameras were synchronized using the 5 V TTL signal from a pulse generator that was converted into a LVDS signal via a custom made TTL module by engineers of Epix for the Mikrotron cameras. The orthogonally placed cameras were Photron cameras (APX and PCI 1024) equipped with 50 mm Nikkor AIS lenses, diaphragm f/5.6, figure 1(B, cameras 3 and 4). The cameras were synchronized by a Quantum Composer 9600+ Series Digital Pulse Delay Generator (Quantum Composers, Bozeman, MT, USA). The two Mikrotron cameras recorded at 400 fps (resolution 700*1024 pixels) through a 4 mm glass interface of the aquarium. The Photron cameras recorded at 800 fps (resolution
Bioinspir. Biomim. 10 (2015) 046009
S Henrion et al
Figure 2. Light rays refract at the boundaries of different media—air, glass and seawater (A) and (B). (A) Schematic overview of a stereo setup with two cameras looking through the same interface. (B) Light ray paths for two points from 2D image plane pixel coordinate to 3D calibration marker refracting at the boundaries of air, glass and seawater (r1 and r2). P: principal point, D: projection distance, C: camera center.
of 1024*1024 pixels), through a 6 mm thick glass interface. All four cameras were calibrated on the same 28 calibration points. The seahorse was tracked by digitizing an arbitrarily chosen point on its ‘neck’ because this point was visible to all four cameras, figure 1(B, inserts: yellow marker) every 0.0125 s (every five frames for the Mikrotrons, every ten frames for the Photrons). We reconstructed the xyz coordinates of the traveled path from six pairs of cameras. One pair comprised two orthogonal cameras (APX– PCI), four pairs comprised one angled and one orthogonal camera (APX–Mik 1, PCI–Mik 1, APX–Mik 2, PCI–Mik 2) and the last pair was a stereo setup (Mik 1 and 2). For the tracking with the first five camera combinations, we used the individual calibrations of each camera separately. For the tracking with the stereo setup (Mik 1 and 2), we calibrated the two cameras simultaneously. Videos of the swimming seahorse from each camera can be viewed in supplementary material B. The complete source code and user manual for the calibration and tracking of two sequences (APX–PCI and Mik 1 and 2) can be found in supplementary material C. 2.2. Calibration frame design We used a 3D printed calibration frame with 50 known marker locations designed in Solidworks (Dassault Systems) and printed by Shapeways (Eindhoven, The Netherlands), figure 1(A, insert) (Solidworks file provided in supplementary material E). The 4
calibration frame was made from laser-sintered nylon plastic, a low cost, high strength, high accuracy (with respect to our purposes) and water-proof material. To circumvent printing inaccuracies, we scanned our calibration frame with an x-ray CT scanner (XRT Phoenix v[tome]x m, GE) with a voxel size resolution of 1.95 × 10−4 mm3 (cube with vertices of 58 μm) and used the coordinate positions of the calibration markers from the x-ray scan. The average difference between the coordinate positions in the Solidworks file versus the positions according to the CT scan of the actual calibration object was 1 mm, respectively 0.62, 0.45 and 0.42 mm in x, y, z direction. The calibration frame dimensions were 80 by 60 by 60 mm, filling approximately one third of the cameras’ field of view, figure 2(A) in experimental setup 1 (sensitivity analysis) and nearly the complete measurement volume in the second setup to track the seahorse, figure 1(B). 2.3. Ray tracing algorithm This section explains the conceptual design of the RCRT algorithm. The algorithm reconstructs light ray paths for each point in the recorded volume. The algorithm uses geometric optics laws—it approximates light as rays and ignores the wave nature of light. However, the algorithm does not use a paraxial approximation—it does not assume that objects on the optical axis of the camera are small enough that the parts of the object that are off-axis experience no
Bioinspir. Biomim. 10 (2015) 046009
S Henrion et al
distortion; instead, all object points are treated as affected by refraction. The RCRT algorithm reconstructs the camera positions, and then uses those reconstructed camera positions to determine the positions of non-calibration points. To reconstruct camera position, the algorithm requires as input calibration points as well as information about refraction events (Austin and Halikas 1976). This information is currently provided by the user in the form of refractive indices for the media through which light travels in our setup (water, glass, air), figure 2. The RCRT algorithm then uses these inputs to start an optimization procedure that traces the light rays from camera to the object. To trace the best set of light rays, the algorithm minimizes the socalled residual—the shortest distance between a 3D calibration marker and its corresponding light ray— for a number of calibration points. During this minimization process, intrinsic and extrinsic camera parameters and setup parameters are optimized (Treibitz et al 2012). The two intrinsic parameters are: (1) the principal point (P, figure 2(B))—the orthogonal projection of the camera center onto the image sensor (often referred to as optical axis of the camera), and (2) projection distance (D, figure 2(B)) (the distance between the camera center and principal point). The extrinsic parameters are the position of the camera center and the camera orientation with respect to the calibration volume coordinate axes (C, figure 2(B)). The setup parameters are the glass interface position and orientation. The algorithm requires two sets of inputs; first, a known set of 3D calibration points distributed across the measurement volume, and their respective 2D image coordinates to set up a correspondence between 2D images and 3D positional data; second, estimates plus lower and upper bounds of the aforementioned intrinsic and extrinsic parameters to initiate and constrain the optimization process. We use a constrained nonlinear multivariable function solver (fmincon, Matlab) to find the set of parameter values that minimize the residual within the parameters’ upper and lower bounds. In our optimization scheme, we excluded the optimization of the refractive indices of the media because we knew their values (1, 1.5 and 1.349, respectively). However, if the refractive index of water is unknown, the user can adapt the software such that the refractive index is included in the optimization process. The source code, user manual, and a detailed algebraic derivation of the RCRT algorithm are included in supplementary material C. Appropriate instructions on the estimation of initial camera parameters are given in the user manual accompanying the source code and can be found in supplementary material C. The code allows users to skip or automate certain steps, such as finding marker locations, and some initialization steps (e.g. lens distortion parameters). When filming through e.g. wideangle or zoom lenses, the user should correct lens 5
distortion using the Caltech camera calibration toolbox (Bouguet 1999) before applying the RCRT algorithm. 2.4. Sensitivity analysis We recorded images of a calibration frame with 50 known marker positions from nine camera orientations. We paired these nine orientations to define camera pairs, resulting in eight stereo angles and eight viewing angles and a total of 20 camera configurations. The stereo angle is defined as the angle between the camera pair, the viewing angle for the camera pair is defined as the average viewing angle of both cameras. Depending on camera configuration, the number of calibration points visible in both camera views ranged from 26 to 38 points, figure D1, supplementary material D. Our goal was to estimate the prediction error of our algorithm, called the 3D reconstruction error in the context of visual reconstruction. To estimate the 3D reconstruction error, we used between 2 and 24 calibration points, leaving between 2 and 36 calibration points to be used as object or validation points. We then explored the sensitivity of the 3D reconstruction error to viewing angle, stereo angle, as well as the number and position of calibration points. For the sensitivity analysis, we randomly sampled 100 times a given number of calibration points (2 to 24 points). From the remaining object points, we then calculated the mean reconstruction error, ε¯, and its standard deviation, σ (every black dot in figure 3: the mean reconstruction error of 100 calibrations; squares and circles in figure 4: mean or standard deviation of 100 calibrations for DLT and RCRT, respectively). This procedure means that we performed 2000 calibrations per ‘number of calibration points’ (100 calibrations for 20 camera configurations) and 46 000 calibrations for the entire sensitivity analysis (100 calibrations for 20 camera configurations for 23 numbers of calibration points). To compare the performance of the DLT and the RCRT algorithms, we used the same sets of randomly selected points rather than performing separate sets of random selections for each algorithm. This sensitivity analysis allowed us to test the stability and accuracy of our algorithm and to compare its performance to the DLT algorithm. To test stability and accuracy, we used the validation points. Our RCRT algorithm uses a constrained nonlinear multivariable function solver (fmincon, Matlab). This optimization tool is unable to distinguish between local and absolute optima and therefore might find a suboptimal solution, which would yield a smaller reconstruction error for calibration than validation points. So to test acccuracy, we compared the prediction error based on calibration points (commonly used to estimate prediction error) with the actual prediction error (=prediction error based on validation points). To test stability, we determined the standard deviation of the
Bioinspir. Biomim. 10 (2015) 046009
S Henrion et al
Figure 3. RCRT calibration shows a homogenous 3D reconstruction error over a wide range of stereo and viewing angle combinations. (A)– (F) Mean 3D reconstruction error, ε¯, of the RCRT algorithm (A)–(C) and DLT algorithm (D)–(F) for 20 stereo, α, and viewing angle, β, combinations with 6, 15 and 24 calibration points. Panels (A), (B), (C), (E), (F) use the color bar on the right, (D) uses the color bar next to the plot. Each black dot represents an average reconstruction error of 100 calibrations on a fixed number (indicated in gray in every graph) of randomly selected calibration points from the shared available points of the stereo pair of images.
prediction error; a large standard deviation, σ, of the mean prediction error, ε¯, indicates that the prediction error is highly sensitive to the random choice of validation points.
2.5. Animal care Seahorses were acquired from a specialist farm (Seahorse Australia, Tasmania, Australia) under CITES No. 12NL118877/11. Seahorses (Hippocampus abdominalis) were kept in two connected 300 l marine aquaria (32 ppt at 17 °C) on a 12 h:12 h light:dark cycle. Salt water was made from CoralSalt plus deionized water, and was aerated for at least 72 h before use. Each aquarium was equipped with a protein skimmer, fluid bed filter and bio filter in order to maintain a stable bacteria culture. Aquaria were lit by four 24 W AquaBlue special lights mounted in an ATI Sunpower Marine Pendant. Seahorses were fed daily on live mysidae shrimp. The animals were allowed to acclimate for three months before experiments started. The animals were recorded in a special filming section of the holding tank to reduce the stress of experimental conditions and to avoid transportation. The experiments were approved by the Animal Ethics Committee of Wageningen University. 6
3. Results In this section, we show a sensitivity analysis of the 3D reconstruction error obtained from both the RCRT approach and DLT algorithm. We explored the sensitivity of both algorithms to camera orientation and the number of calibration points used by the algorithm for calibration. By using fewer than the available calibration points for calibration, we explored how well each algorithm was able to predict the positions of object points. Finally, we compared the 3D reconstructed swimming path of a seahorse based on six different camera configurations to show the efficacy of our refraction based algorithm. The RCRT algorithm generated consistently low mean 3D reconstruction errors, ε¯, for a wide range of camera orientations and numbers of calibration points. In figure 3, we show the error for 20 different cameras configurations for three different numbers of calibration points (6, 15 and 24 calibration points). In the 20 camera configurations that we examined the error remained below 0.6 mm (
