December 15, 2014 / Vol. 39, No. 24 / OPTICS LETTERS
6855
Three-dimensional integral imaging with flexible sensing Jingang Wang, Xiao Xiao, and Bahram Javidi* Electrical & Computer Engineering Department, University of Connecticut, 371 Fairfield Road Unit 4157, Storrs, Connecticut 06269, USA *Corresponding author: [email protected] Received August 11, 2014; revised October 26, 2014; accepted October 29, 2014; posted October 31, 2014 (Doc. ID 220831); published December 10, 2014 We present to the best of our knowledge the first report on three-dimensional (3D) integral imaging capture and reconstruction method with unknown poses of sensors placed on a flexible surface. Compared to a conventional integral imaging system, where a lenslet or sensor array is commonly located on a planar surface, the flexible sensing integral imaging system allows sensors to be placed on a nonplanar surface that can increase the field of view of the 3D imaging system. To obtain the poses of the sensor array on a flexible surface, an estimation algorithm is developed based on two-view geometry theory and the camera projective model. In addition, a super-resolution image is generated from a sequence of low-resolution 2D images with sub-pixel shifts. Super-resolution 3D reconstruction results at different depths are presented to validate the proposed approach. © 2014 Optical Society of America OCIS codes: (110.6880) Three-dimensional image acquisition; (100.6890) Three-dimensional image processing. http://dx.doi.org/10.1364/OL.39.006855
Integral imaging [1] is a glass-free, passive, multiperspective three-dimensional (3D) imaging that records multiple 2D images (elemental images) from different views of a 3D scene and reconstructs the 3D scene using these 2D images with a lenslet array [2–7]. Integral imaging is able to display full parallax true 3D color scene with continuous viewing angles. Some approaches have been proposed to improve integral imaging resolution [8–14]. In conventional integral imaging, lenslet or sensor arrays are usually placed on a planar surface. There could be applications where the sensors need to be placed on a nonflat surface. A concept for three-dimensional integral imaging with detector arrays on arbitrarily shaped surfaces has been presented [15]. There have been numerous advances in research and development of flexible electronics for the purpose of integration on elastomeric substrates to fabricate deformable electronic devices [16]. Nonplanar detector arrays have been demonstrated [17–19]. There are benefits in using these flexible detector arrays for 3D integral imaging, including increased field of view for many applications such as 3D automated object recognition, 3D endoscopy instruments, automotive vehicular sensing, soldiers’ helmets, and aircraft wings with robust structural constraints. In this report, we will present experimental reconstruction results for the flexible sensing integral imaging system with super-resolution. Our focus will be on the capture and reconstruction. Furthermore, we improve the robustness of the system by extending it to a scenario where the poses of sensors are unknown. Using the estimated poses of the sensors, we implement a superresolution technique based on time division multiplexing to improve the image resolution. The novelty of our approach is that we implement a super-resolution 3D integral imaging system with unknown poses of sensor arrays on a nonflat surface. For flexible sensing integral imaging, the lenses may be quite small, and it may be necessary to implement super-resolution technique to overcome the low-resolution problem caused by the small aperture of the lenses. Figure 1 shows the comparisons between a conventional integral imaging system and a flexible sensing integral imaging system (also see Fig. 2). Since the sensors are on a planar surface, the conventional integral imaging 0146-9592/14/246855-04$15.00/0
system has limited field of view. However, a flexible sensing integral imaging system can create an enlarged field of view by having certain sensors on a nonflat surface. In order to implement computational reconstruction in a flexible sensing integral imaging system, poses of all the cameras are needed as the detectors are not placed on a flat surface with known poses or positions. Here, we develop an algorithm to estimate camera poses in a real 3D scene. The estimation algorithm can facilitate the implementation of a flexing sensing system because no manual calibration is needed when the arrangement of the camera array is changed. The estimation algorithm assumes that the relative pose of the first two cameras are known. Based on this prior information, we estimate the remaining camera poses by combining two view geometry theory and the camera projective model (see Fig. 2). The camera projection equations for camera 1 (C1 ) and camera 2 (C2 ) can be written as [20]: m1i ∝ K 1 R1 t1 M i ;
(1a)
m2i ∝ K 2 R2 t2 M i ;
(1b)
where m1i and m2i are the ith pair of image matching points projected from an identical 3D point, M i . K 1 and K 2 are the known 3 × 3 intrinsic parameter matrices of C1 and C2 . R1 , R2 , and t1 , t2 indicate the orientation
Fig. 1. (a) Illustration of a conventional integral imaging system where sensors are on a planar surface. (b) A flexible sensing integral imaging system where the sensors can be placed on any arbitrary surface. © 2014 Optical Society of America
6856
OPTICS LETTERS / Vol. 39, No. 24 / December 15, 2014
matrices and translation vectors of C1 and C2 , and i 1; …P, where P is the total number of 3D points. Based on our assumptions, R1 , R2 and t1 , t2 are known, which can be obtained by standard camera calibration methods [20] or moving the camera along a highly accurate translation stage. In addition, the intrinsic parameters of the sensors (K) are assumed to be known. In our proposed algorithm, scale-invariant feature transform (SIFT), epipolar geometry, and random sample consensus (RANSAC) algorithms [21] are utilized to acquire and track robust image matching points in image sequences. Once the matching points (m1i and m2i ) are ˆ i ) based on obtained, we can estimate 3D points (M Eq. (1). For the kth camera (C k ), the projection equations is written by mki ∝ K k Rk tk M i ;
(2)
where mki is the projection of 3D point M i for camera k, Rk and tk are the unknown rotation and translation matrices needed to estimate, and k 1; …S, where S is the total number of cameras. To estimate the pose (Rk and tk ) of C k , we minimize the following cost function: P X
ˆ i ‖: ‖mki − K k Rk tk M
(3)
i1
Note that Rk and tk include 6 unknown independent parameters. Hence, we need at least 3 image points (P ≥ 3) to estimate these values. To solve the nonlinear optimization problem given by Eq. (3), we used the Levenberg–Marquardt optimization method. The initial estimates are calculated using a singular value decomposition (SVD) linear least-square method [21]. By using this algorithm, we can estimate the poses of the sensors placed on an arbitrary surface. After developing the algorithm for estimating the poses of sensors, we explore the super-resolution method to enhance the quality of elemental images [14,22]. The cameras used in the experiments have 4-mm aperture lens and 8-mm focal length. The resolution of elemental images in integral imaging is limited by the diffraction effects due to the camera lens aperture and the pixel size of the sensor. Using low-resolution elemental images, it is difficult to achieve high-quality reconstruction results. To overcome the limitation, we use an iterative backprojection method [14] to generate a super-resolution image from a sequence of low-resolution elemental
Fig. 2. Estimation of the poses of sensors on a nonplanar surface.
images with a sub-pixel shift technique. This process is implemented by minimizing the error iteratively. The super-resolution algorithm starts with an arbitrary guess f 0 for the initial super-resolution image. A possible guess is the average of the interpolated low-resolution image sequence. At each iteration step n, the imaging process is simulated to obtain a set of simulated low-resolution images gnk . The kth simulated image at the nth iteration step is given by [14]: gnk T k f n hpsf s↓;
(4)
where T k is the translation operator of the kth low resolution images, hpsf ; is point spread function, s↓ is the decimation operator for down sampling, and denotes the convolution operator. At each iteration step, the difference images fgk − gnk gN k1 are used to improve the previous guess f n by the following update scheme [14]: f n1 f n
N 1X T −1 fgk − gnk s↑g p; N k1 k
(5)
where s↑ is the inverse operator of s↓, and p is a back projection kernel determined by point spread function. gk is the kth low-resolution elemental images. N is the number of low-resolution elemental images used to generate one super-resolution image. The operator T −1 is k the inverse of the geometric transformation operator T k . In our case T k consists only of translation, and T −1 k is a registration operator that aligns properly the difference images by performing shifts that are inversed to that of T k . To validate this super-resolution algorithm, we perform an experiment using a simulated resolution target with a resolution of 100 by 100 pixels, as shown in Fig. 3(a). The original high-resolution image is convolved with a simulated point spread function. After a sequence of 1 pixel horizontal shifting and ¼ down sampling, we can obtain a sequence of low-resolution images with ¼ sub-pixel shifting and image size of 25 by 25 pixels [see Fig. 3(b)]. Compared to the enlarged low-resolution images [Fig. 3(c)], the image with super-resolution technique [Fig. 3(d)] has improved the reconstructed object detail (see the highlighted areas). The peak signal-to-noise ratio (PSNR) of the reconstructed images on a flexible sensing system is defined as
Fig. 3. Generation of a super-resolution elemental image from low-resolution elemental images. (a) Original high-resolution image. (b) Four down-sampled low-resolution images with sub-pixel shift. (c) One enlarged low-resolution images with bilinear interpolation. (d) A super-resolution image reconstructed from four low-resolution images based on the iterative backprojection.
December 15, 2014 / Vol. 39, No. 24 / OPTICS LETTERS
Fig. 4. Experimental setup. (a) A miniaturized 2 × 2 camera array with 752 by 432 pixels for each camera. Each camera lens has a 4-mm diameter and 8-mm focal length. (b) A diagram of our camera arrangement. (c) The 3D scene used in our experiment
6857
Fig. 5. Improvement of the field of view using a flexible sensing integral imaging compared to a conventional integral imaging [see the highlighted red areas]. (a) and (b) are the left-most and the right-most elemental images (EIs) captured from a conventional integral imaging. (c) and (d) are the left-most and the right-most elemental images from a flexible sensing system.
2552 Q L ; (6) PSNRO; R 10 log10 PQ PL 2 j Oi; j − Ri; j i where O and R indicate an object image and an integral imaging reconstructed image of that object. Q and L are the image size in pixels. According to Eq. (6), the PSNR of the red highlighted area of images in Figs. 3(c), 3(d) increases from 10.81 to 15.60 dB. We present optical experiments to show the advantages of the proposed flexible sensing integral imaging with unknown camera pose with super-resolution technique. The configurations of our experimental setup are shown in Fig. 4. In the experiments, a 2 × 2 camera array with 752 by 432 pixels for each camera is used as shown in Fig. 4(a). The pixel size is 6 μm. Four groups of the camera array are placed on a nonflat surface. The camera arrangement is shown in Fig. 4(b). The 3D scene shown in Fig. 4(c) consists of 4 objects: object 1 and object 2 are both resolution targets located at 1.45 and 2.00 m away from the sensors array; object 3 and object 4 are a soccer ball (diameter 20 cm) and a Mickey Mouse doll (height 17 cm) both located at 2.00 m. In order to perform super-resolution, the camera array is horizontally translated 3 times equivalent to a (0.25, 0.5, 0.75) sub-pixel shift. The poses of cameras are estimated with the algorithm discussed earlier. To show the improvement of the field of view using a flexible sensing integral imaging compared to a conventional integral imaging, we present the left-most and the right-most elemental images captured from a conventional integral imaging system [see Figs. 5(a), 5(b)]. Similarly, we show the results from a flexible sensing integral imaging [see Figs. 5(c), 5(d)]. The appearance of the soccer ball in Fig. 5(c) and the Mickey Mouse doll in Fig. 5(d) show the improvement of field of view using the flexible sensing integral imaging. After capturing the low-resolution elemental images, we generate superresolution elemental images from these images with sub-pixel shifts using the iterative back-projection algorithm [22]. Figure 6(a) shows one of the low-resolution elemental images enlarged with bilinear interpolation, in which the highlighted yellow area is blurred. The elemental image generated with super-resolution, as
Fig. 6. Super-resolution elemental image generation [see the highlighted yellow areas]. (a) An enlarged low-resolution elemental image with bilinear interpolation (four times). (b) A super-resolution elemental image reconstructed from four low-resolution elemental images based on the iterative backprojection method.
shown in Fig. 6(b), shows improved image resolution as shown in the highlighted areas. Finally, after estimating the poses of the cameras and generating super-resolution elemental images, we reconstruct the 3D scene with the method described in [15] for flexible sensing integral imaging. 3D reconstruction with or without the super-resolution method at different depths (z) are shown in Fig. 7. To show the improvement of the resolution, the local reconstruction details circled by yellow dash dot in Fig. 7 are enlarged. Compared to the enlarged reconstruction details in Figs. 7(a) and 7(c), the white bars can be distinguished in Figs. 7(b) and 7(d), which illustrates the improved 3D reconstruction results of flexible sensing using the super-resolution technique. Also, to illustrate the improvement of our method, we present 1-D intensity profiles of the local reconstruction areas of flexible sensing (along the blue lines in Fig. 7). Figures 8(a) and 8(c) show intensity profiles without the super-resolution method. Figures 8(b) and 8(d) are those with super-resolution method. We can see that the three intensity peaks of the white bars are distinguishable in Figs. 8(b) and 8(d). The PSNR is calculated to evaluate the improvement of the reconstructed images on a flexible sensing system. The PSNR of the highlighted area of reconstructed images in Figs. 7(a)–7(d) are 27.36, 30.11, 26.94, and 30.48 dB, respectively. In experiments, we obtain improvement of 2.75–3.54 dB with super-resolution technique in flexible sensing integral imaging.
6858
OPTICS LETTERS / Vol. 39, No. 24 / December 15, 2014
Fig. 8. 1-D intensity profiles of the local reconstruction areas (along the blue lines in Fig. 7). (a) and (c) are the profiles without super-resolution; (b) and (d) are with super-resolution.
with the super resolution, and Prof. Viktor Gurev for providing the sensor array.
Fig. 7. Comparison of integral imaging reconstruction results in a flexible sensing system without and with super-resolution technique. [See the highlighted yellow areas.] (a) and (b) are computational reconstruction results without and with superresolution elemental images at a distance of 1.45 m focused on the right resolution target. (c) and (d) are those without and with super-resolution elemental images at a distance of 2.00 m focused on the left resolution target, soccer ball, and Mickey Mouse.
In conclusion, we have presented a 3D integral imaging capture and reconstruction method with unknown poses of sensor arrays placed on a flexible nonflat surface. We developed an algorithm to estimate the unknown poses of the cameras and a technique to improve image resolution in the flexible sensing integral imaging system. Super-resolution techniques were applied to compensate for small lens apertures. Optical experiments have been performed to validate the feasibility of the proposed super-resolution 3D flexible sensing integral imaging system. Future work will consider extension to flexible 3D integral imaging displays [23–25]. This work was supported in part by DARPA and US Army under contract number W911NF-13-1-0485. We thank Prof. Adrian Stern and Chen Yang for their help
References 1. G. Lippmann, C. R. Acad. Sci. 146, 446 (1908). 2. A. Stern and B. Javidi, Proc. IEEE 94, 591 (2006). 3. R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, Proc. IEEE 97, 1067 (2009). 4. C. Myungjin, M. Daneshpanah, I. Moon, and B. Javidi, Proc. IEEE 99, 556 (2011). 5. X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, Appl. Opt. 52, 546 (2013). 6. C. B. Burckhardt, J. Opt. Soc. Am. 58, 71 (1968). 7. H. Hoshino, F. Okano, H. Isono, and I. Yuyama, J. Opt. Soc. Am. A 15, 2059 (1998). 8. K. Wakunami and M. Yamaguchi, Opt. Express 19, 9086 (2011). 9. F. Okano, J. Arai, and M. Kawakita, Opt. Lett. 32, 364 (2007). 10. F. Okano, J. Arai, K. Mitani, and M. Okui, Proc. IEEE 94, 490 (2006). 11. J. S. Jang and B. Javidi, Opt. Lett. 27, 1144 (2002). 12. J. S. Jang and B. Javidi, Opt. Lett. 27, 324 (2002). 13. H. Yoo, Opt. Express 21, 8873 (2013). 14. A. Stern and B. Javidi, Appl. Opt. 42, 7036 (2003). 15. M. Daneshpanah and B. Javidi, Opt. Lett. 36, 600 (2011). 16. J. A. Rogers, T. Someya, and Y. Huang, Science 327, 1603 (2010). 17. X. Xu, M. Davanco, X. Qi, and S. R. Forrest, Org. Electron. 9, 1122 (2008). 18. V. Gruev, Z. Yang, J. Van der Spiegel, and R. EtienneCummings, IEEE Circuits Syst. Mag. 57, 1154(2010). 19. S. B. Rim, P. B. Catrysse, R. Dinyari, K. Huang, and P. Peumans, Opt. Express16, 4965(2008). 20. Z. Zhang, IEEE Trans. Pattern Anal. Mach. Intell. 22, 1330 (2000). 21. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision (Cambridge University, 2003). 22. A. Irani and S. Peleg, CVGIP 53, 231 (1991). 23. D. H. Shin, B. Lee, and E. S. Kim, Appl. Opt. 45, 7375 (2006). 24. J. S. Jang and B. Javidi, Opt. Express 12, 3778 (2004). 25. J. S. Jang and B. Javidi, Opt. Eng. 44, 014001 (2004).
6855
Three-dimensional integral imaging with flexible sensing Jingang Wang, Xiao Xiao, and Bahram Javidi* Electrical & Computer Engineering Department, University of Connecticut, 371 Fairfield Road Unit 4157, Storrs, Connecticut 06269, USA *Corresponding author: [email protected] Received August 11, 2014; revised October 26, 2014; accepted October 29, 2014; posted October 31, 2014 (Doc. ID 220831); published December 10, 2014 We present to the best of our knowledge the first report on three-dimensional (3D) integral imaging capture and reconstruction method with unknown poses of sensors placed on a flexible surface. Compared to a conventional integral imaging system, where a lenslet or sensor array is commonly located on a planar surface, the flexible sensing integral imaging system allows sensors to be placed on a nonplanar surface that can increase the field of view of the 3D imaging system. To obtain the poses of the sensor array on a flexible surface, an estimation algorithm is developed based on two-view geometry theory and the camera projective model. In addition, a super-resolution image is generated from a sequence of low-resolution 2D images with sub-pixel shifts. Super-resolution 3D reconstruction results at different depths are presented to validate the proposed approach. © 2014 Optical Society of America OCIS codes: (110.6880) Three-dimensional image acquisition; (100.6890) Three-dimensional image processing. http://dx.doi.org/10.1364/OL.39.006855
Integral imaging [1] is a glass-free, passive, multiperspective three-dimensional (3D) imaging that records multiple 2D images (elemental images) from different views of a 3D scene and reconstructs the 3D scene using these 2D images with a lenslet array [2–7]. Integral imaging is able to display full parallax true 3D color scene with continuous viewing angles. Some approaches have been proposed to improve integral imaging resolution [8–14]. In conventional integral imaging, lenslet or sensor arrays are usually placed on a planar surface. There could be applications where the sensors need to be placed on a nonflat surface. A concept for three-dimensional integral imaging with detector arrays on arbitrarily shaped surfaces has been presented [15]. There have been numerous advances in research and development of flexible electronics for the purpose of integration on elastomeric substrates to fabricate deformable electronic devices [16]. Nonplanar detector arrays have been demonstrated [17–19]. There are benefits in using these flexible detector arrays for 3D integral imaging, including increased field of view for many applications such as 3D automated object recognition, 3D endoscopy instruments, automotive vehicular sensing, soldiers’ helmets, and aircraft wings with robust structural constraints. In this report, we will present experimental reconstruction results for the flexible sensing integral imaging system with super-resolution. Our focus will be on the capture and reconstruction. Furthermore, we improve the robustness of the system by extending it to a scenario where the poses of sensors are unknown. Using the estimated poses of the sensors, we implement a superresolution technique based on time division multiplexing to improve the image resolution. The novelty of our approach is that we implement a super-resolution 3D integral imaging system with unknown poses of sensor arrays on a nonflat surface. For flexible sensing integral imaging, the lenses may be quite small, and it may be necessary to implement super-resolution technique to overcome the low-resolution problem caused by the small aperture of the lenses. Figure 1 shows the comparisons between a conventional integral imaging system and a flexible sensing integral imaging system (also see Fig. 2). Since the sensors are on a planar surface, the conventional integral imaging 0146-9592/14/246855-04$15.00/0
system has limited field of view. However, a flexible sensing integral imaging system can create an enlarged field of view by having certain sensors on a nonflat surface. In order to implement computational reconstruction in a flexible sensing integral imaging system, poses of all the cameras are needed as the detectors are not placed on a flat surface with known poses or positions. Here, we develop an algorithm to estimate camera poses in a real 3D scene. The estimation algorithm can facilitate the implementation of a flexing sensing system because no manual calibration is needed when the arrangement of the camera array is changed. The estimation algorithm assumes that the relative pose of the first two cameras are known. Based on this prior information, we estimate the remaining camera poses by combining two view geometry theory and the camera projective model (see Fig. 2). The camera projection equations for camera 1 (C1 ) and camera 2 (C2 ) can be written as [20]: m1i ∝ K 1 R1 t1 M i ;
(1a)
m2i ∝ K 2 R2 t2 M i ;
(1b)
where m1i and m2i are the ith pair of image matching points projected from an identical 3D point, M i . K 1 and K 2 are the known 3 × 3 intrinsic parameter matrices of C1 and C2 . R1 , R2 , and t1 , t2 indicate the orientation
Fig. 1. (a) Illustration of a conventional integral imaging system where sensors are on a planar surface. (b) A flexible sensing integral imaging system where the sensors can be placed on any arbitrary surface. © 2014 Optical Society of America
6856
OPTICS LETTERS / Vol. 39, No. 24 / December 15, 2014
matrices and translation vectors of C1 and C2 , and i 1; …P, where P is the total number of 3D points. Based on our assumptions, R1 , R2 and t1 , t2 are known, which can be obtained by standard camera calibration methods [20] or moving the camera along a highly accurate translation stage. In addition, the intrinsic parameters of the sensors (K) are assumed to be known. In our proposed algorithm, scale-invariant feature transform (SIFT), epipolar geometry, and random sample consensus (RANSAC) algorithms [21] are utilized to acquire and track robust image matching points in image sequences. Once the matching points (m1i and m2i ) are ˆ i ) based on obtained, we can estimate 3D points (M Eq. (1). For the kth camera (C k ), the projection equations is written by mki ∝ K k Rk tk M i ;
(2)
where mki is the projection of 3D point M i for camera k, Rk and tk are the unknown rotation and translation matrices needed to estimate, and k 1; …S, where S is the total number of cameras. To estimate the pose (Rk and tk ) of C k , we minimize the following cost function: P X
ˆ i ‖: ‖mki − K k Rk tk M
(3)
i1
Note that Rk and tk include 6 unknown independent parameters. Hence, we need at least 3 image points (P ≥ 3) to estimate these values. To solve the nonlinear optimization problem given by Eq. (3), we used the Levenberg–Marquardt optimization method. The initial estimates are calculated using a singular value decomposition (SVD) linear least-square method [21]. By using this algorithm, we can estimate the poses of the sensors placed on an arbitrary surface. After developing the algorithm for estimating the poses of sensors, we explore the super-resolution method to enhance the quality of elemental images [14,22]. The cameras used in the experiments have 4-mm aperture lens and 8-mm focal length. The resolution of elemental images in integral imaging is limited by the diffraction effects due to the camera lens aperture and the pixel size of the sensor. Using low-resolution elemental images, it is difficult to achieve high-quality reconstruction results. To overcome the limitation, we use an iterative backprojection method [14] to generate a super-resolution image from a sequence of low-resolution elemental
Fig. 2. Estimation of the poses of sensors on a nonplanar surface.
images with a sub-pixel shift technique. This process is implemented by minimizing the error iteratively. The super-resolution algorithm starts with an arbitrary guess f 0 for the initial super-resolution image. A possible guess is the average of the interpolated low-resolution image sequence. At each iteration step n, the imaging process is simulated to obtain a set of simulated low-resolution images gnk . The kth simulated image at the nth iteration step is given by [14]: gnk T k f n hpsf s↓;
(4)
where T k is the translation operator of the kth low resolution images, hpsf ; is point spread function, s↓ is the decimation operator for down sampling, and denotes the convolution operator. At each iteration step, the difference images fgk − gnk gN k1 are used to improve the previous guess f n by the following update scheme [14]: f n1 f n
N 1X T −1 fgk − gnk s↑g p; N k1 k
(5)
where s↑ is the inverse operator of s↓, and p is a back projection kernel determined by point spread function. gk is the kth low-resolution elemental images. N is the number of low-resolution elemental images used to generate one super-resolution image. The operator T −1 is k the inverse of the geometric transformation operator T k . In our case T k consists only of translation, and T −1 k is a registration operator that aligns properly the difference images by performing shifts that are inversed to that of T k . To validate this super-resolution algorithm, we perform an experiment using a simulated resolution target with a resolution of 100 by 100 pixels, as shown in Fig. 3(a). The original high-resolution image is convolved with a simulated point spread function. After a sequence of 1 pixel horizontal shifting and ¼ down sampling, we can obtain a sequence of low-resolution images with ¼ sub-pixel shifting and image size of 25 by 25 pixels [see Fig. 3(b)]. Compared to the enlarged low-resolution images [Fig. 3(c)], the image with super-resolution technique [Fig. 3(d)] has improved the reconstructed object detail (see the highlighted areas). The peak signal-to-noise ratio (PSNR) of the reconstructed images on a flexible sensing system is defined as
Fig. 3. Generation of a super-resolution elemental image from low-resolution elemental images. (a) Original high-resolution image. (b) Four down-sampled low-resolution images with sub-pixel shift. (c) One enlarged low-resolution images with bilinear interpolation. (d) A super-resolution image reconstructed from four low-resolution images based on the iterative backprojection.
December 15, 2014 / Vol. 39, No. 24 / OPTICS LETTERS
Fig. 4. Experimental setup. (a) A miniaturized 2 × 2 camera array with 752 by 432 pixels for each camera. Each camera lens has a 4-mm diameter and 8-mm focal length. (b) A diagram of our camera arrangement. (c) The 3D scene used in our experiment
6857
Fig. 5. Improvement of the field of view using a flexible sensing integral imaging compared to a conventional integral imaging [see the highlighted red areas]. (a) and (b) are the left-most and the right-most elemental images (EIs) captured from a conventional integral imaging. (c) and (d) are the left-most and the right-most elemental images from a flexible sensing system.
2552 Q L ; (6) PSNRO; R 10 log10 PQ PL 2 j Oi; j − Ri; j i where O and R indicate an object image and an integral imaging reconstructed image of that object. Q and L are the image size in pixels. According to Eq. (6), the PSNR of the red highlighted area of images in Figs. 3(c), 3(d) increases from 10.81 to 15.60 dB. We present optical experiments to show the advantages of the proposed flexible sensing integral imaging with unknown camera pose with super-resolution technique. The configurations of our experimental setup are shown in Fig. 4. In the experiments, a 2 × 2 camera array with 752 by 432 pixels for each camera is used as shown in Fig. 4(a). The pixel size is 6 μm. Four groups of the camera array are placed on a nonflat surface. The camera arrangement is shown in Fig. 4(b). The 3D scene shown in Fig. 4(c) consists of 4 objects: object 1 and object 2 are both resolution targets located at 1.45 and 2.00 m away from the sensors array; object 3 and object 4 are a soccer ball (diameter 20 cm) and a Mickey Mouse doll (height 17 cm) both located at 2.00 m. In order to perform super-resolution, the camera array is horizontally translated 3 times equivalent to a (0.25, 0.5, 0.75) sub-pixel shift. The poses of cameras are estimated with the algorithm discussed earlier. To show the improvement of the field of view using a flexible sensing integral imaging compared to a conventional integral imaging, we present the left-most and the right-most elemental images captured from a conventional integral imaging system [see Figs. 5(a), 5(b)]. Similarly, we show the results from a flexible sensing integral imaging [see Figs. 5(c), 5(d)]. The appearance of the soccer ball in Fig. 5(c) and the Mickey Mouse doll in Fig. 5(d) show the improvement of field of view using the flexible sensing integral imaging. After capturing the low-resolution elemental images, we generate superresolution elemental images from these images with sub-pixel shifts using the iterative back-projection algorithm [22]. Figure 6(a) shows one of the low-resolution elemental images enlarged with bilinear interpolation, in which the highlighted yellow area is blurred. The elemental image generated with super-resolution, as
Fig. 6. Super-resolution elemental image generation [see the highlighted yellow areas]. (a) An enlarged low-resolution elemental image with bilinear interpolation (four times). (b) A super-resolution elemental image reconstructed from four low-resolution elemental images based on the iterative backprojection method.
shown in Fig. 6(b), shows improved image resolution as shown in the highlighted areas. Finally, after estimating the poses of the cameras and generating super-resolution elemental images, we reconstruct the 3D scene with the method described in [15] for flexible sensing integral imaging. 3D reconstruction with or without the super-resolution method at different depths (z) are shown in Fig. 7. To show the improvement of the resolution, the local reconstruction details circled by yellow dash dot in Fig. 7 are enlarged. Compared to the enlarged reconstruction details in Figs. 7(a) and 7(c), the white bars can be distinguished in Figs. 7(b) and 7(d), which illustrates the improved 3D reconstruction results of flexible sensing using the super-resolution technique. Also, to illustrate the improvement of our method, we present 1-D intensity profiles of the local reconstruction areas of flexible sensing (along the blue lines in Fig. 7). Figures 8(a) and 8(c) show intensity profiles without the super-resolution method. Figures 8(b) and 8(d) are those with super-resolution method. We can see that the three intensity peaks of the white bars are distinguishable in Figs. 8(b) and 8(d). The PSNR is calculated to evaluate the improvement of the reconstructed images on a flexible sensing system. The PSNR of the highlighted area of reconstructed images in Figs. 7(a)–7(d) are 27.36, 30.11, 26.94, and 30.48 dB, respectively. In experiments, we obtain improvement of 2.75–3.54 dB with super-resolution technique in flexible sensing integral imaging.
6858
OPTICS LETTERS / Vol. 39, No. 24 / December 15, 2014
Fig. 8. 1-D intensity profiles of the local reconstruction areas (along the blue lines in Fig. 7). (a) and (c) are the profiles without super-resolution; (b) and (d) are with super-resolution.
with the super resolution, and Prof. Viktor Gurev for providing the sensor array.
Fig. 7. Comparison of integral imaging reconstruction results in a flexible sensing system without and with super-resolution technique. [See the highlighted yellow areas.] (a) and (b) are computational reconstruction results without and with superresolution elemental images at a distance of 1.45 m focused on the right resolution target. (c) and (d) are those without and with super-resolution elemental images at a distance of 2.00 m focused on the left resolution target, soccer ball, and Mickey Mouse.
In conclusion, we have presented a 3D integral imaging capture and reconstruction method with unknown poses of sensor arrays placed on a flexible nonflat surface. We developed an algorithm to estimate the unknown poses of the cameras and a technique to improve image resolution in the flexible sensing integral imaging system. Super-resolution techniques were applied to compensate for small lens apertures. Optical experiments have been performed to validate the feasibility of the proposed super-resolution 3D flexible sensing integral imaging system. Future work will consider extension to flexible 3D integral imaging displays [23–25]. This work was supported in part by DARPA and US Army under contract number W911NF-13-1-0485. We thank Prof. Adrian Stern and Chen Yang for their help
References 1. G. Lippmann, C. R. Acad. Sci. 146, 446 (1908). 2. A. Stern and B. Javidi, Proc. IEEE 94, 591 (2006). 3. R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, Proc. IEEE 97, 1067 (2009). 4. C. Myungjin, M. Daneshpanah, I. Moon, and B. Javidi, Proc. IEEE 99, 556 (2011). 5. X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, Appl. Opt. 52, 546 (2013). 6. C. B. Burckhardt, J. Opt. Soc. Am. 58, 71 (1968). 7. H. Hoshino, F. Okano, H. Isono, and I. Yuyama, J. Opt. Soc. Am. A 15, 2059 (1998). 8. K. Wakunami and M. Yamaguchi, Opt. Express 19, 9086 (2011). 9. F. Okano, J. Arai, and M. Kawakita, Opt. Lett. 32, 364 (2007). 10. F. Okano, J. Arai, K. Mitani, and M. Okui, Proc. IEEE 94, 490 (2006). 11. J. S. Jang and B. Javidi, Opt. Lett. 27, 1144 (2002). 12. J. S. Jang and B. Javidi, Opt. Lett. 27, 324 (2002). 13. H. Yoo, Opt. Express 21, 8873 (2013). 14. A. Stern and B. Javidi, Appl. Opt. 42, 7036 (2003). 15. M. Daneshpanah and B. Javidi, Opt. Lett. 36, 600 (2011). 16. J. A. Rogers, T. Someya, and Y. Huang, Science 327, 1603 (2010). 17. X. Xu, M. Davanco, X. Qi, and S. R. Forrest, Org. Electron. 9, 1122 (2008). 18. V. Gruev, Z. Yang, J. Van der Spiegel, and R. EtienneCummings, IEEE Circuits Syst. Mag. 57, 1154(2010). 19. S. B. Rim, P. B. Catrysse, R. Dinyari, K. Huang, and P. Peumans, Opt. Express16, 4965(2008). 20. Z. Zhang, IEEE Trans. Pattern Anal. Mach. Intell. 22, 1330 (2000). 21. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision (Cambridge University, 2003). 22. A. Irani and S. Peleg, CVGIP 53, 231 (1991). 23. D. H. Shin, B. Lee, and E. S. Kim, Appl. Opt. 45, 7375 (2006). 24. J. S. Jang and B. Javidi, Opt. Express 12, 3778 (2004). 25. J. S. Jang and B. Javidi, Opt. Eng. 44, 014001 (2004).
