May 15, 2014 / Vol. 39, No. 10 / OPTICS LETTERS
2853
Multi-projection integral imaging by use of a convex mirror array Jae-Young Jang,1 Donghak Shin,2,* Byung-Gook Lee,2 and Eun-Soo Kim3 1
2
iReal Co. Ltd., Dongseo University, Churye-ro, Sasang-Gu, Busan 617-716, South Korea Institute of Ambient Intelligence, Dongseo University, Churye-ro, Sasang-Gu, Busan 617-716, South Korea 3
HoloDigilog Human Media Research Center (HoloDigilog), 3D Display Research Center (3DRC), Kwangwoon University, Wolgye-Dong, Nowon-Gu, Seoul 139-701, South Korea *Corresponding author: [email protected] Received February 24, 2014; revised April 7, 2014; accepted April 8, 2014; posted April 9, 2014 (Doc. ID 206417); published May 6, 2014
We propose a new multi-projection integral imaging scheme using a convex mirror array. In the proposed scheme, to overcome the resolution limitation of the conventional method due to observing the single aperture imaging point (AIP) from each convex mirror, we introduce the multi-projection to obtain multiple AIPs per convex mirror so that the viewer observes the resolution-improved 3D reconstructed images. We validate the theoretical analysis of the proposed scheme and confirm its feasibility through the optical experiments. To our best knowledge, this is the first report to generate multiple AIPs per convex mirror in a projection integral imaging system. © 2014 Optical Society of America OCIS codes: (100.6890) Three-dimensional image processing; (110.6880) Three-dimensional image acquisition; (110.2990) Image formation theory. http://dx.doi.org/10.1364/OL.39.002853
The three-dimensional (3D) integral imaging technique is capable of producing full parallax 3D images in space using two-dimensional (2D) display devices under incoherent light [1–7]. Basically, the integral imaging system is composed of pickup and display parts. In the pickup part, the 3D objects are recorded into the elemental images through the lenslet array. And 3D images are generated by crossing rays coming from the elemental images through the lenslet array in the display part. Recently, various projection-type integral imaging (PII) systems have been studied for large television or movie applications [8–14]. Among them, Jang and Javidi proposed a single-projection-type integral imaging (SPII) system using a convex mirror array (CMA) as a screen [8]. This system could provide a dramatically improved viewing angle, no depth reverse process, and flippingfree observations of 3D images. Even though this CMAbased SPII system has many advantages, the resolution of the 3D image is dependent on the total number of elemental convex mirrors in the CMA used for optical display. That is, the viewer observes a single imaging point through each convex mirror in the CMA. In addition, the CMA-based PII method may be limited to using commercial projectors directly, because relay optics with low magnification is additionally needed for pixel matching between elemental images and the CMA within the small area. In this Letter, we propose a novel multi-projection integral imaging (MPII) scheme using a CMA. Compared with the conventional SPII, multiple projectors at different positions are projected to the same CMA at the same time. In so doing, we can improve the resolution of the 3D reconstructed image in PII. Figure 1 shows a schematic diagram of the proposed MPII system using CMA. In the proposed method, multiprojection is performed into the same CMA at the same time. Compared with the conventional SPII with a microconvex mirror array [8], we use a CMA with a large diameter. This enables us to directly use the commercial 0146-9592/14/102853-04$15.00/0
projectors without any modification or additional relay optics. The corresponding elemental images for each projector are projected into the CMA as shown in Fig. 1. Then, the viewer observes the displayed 3D object image through the CMA. We first analyze the geometrical relationship between a single-projection system and a CMA. As shown in Fig. 2, the elemental image array is projected onto the CMA through the projection system. In our analysis, the aperture of the projection system is assumed to be a pinhole. Then, the geometrical relationship between the aperture of the projector and its corresponding aperture imaging point (AIP) by the nth elemental convex mirror in the CMA can be given by xIpn xp
zp zp − f
n−
1 P − xp : 2
(1)
Here, the origin of the coordinates is assumed to be the edge of the elemental convex mirror located at the
Fig. 1.
Schematic diagram of the proposed MPII scheme.
© 2014 Optical Society of America
2854
OPTICS LETTERS / Vol. 39, No. 10 / May 15, 2014
calculated by the common diverging angles from elemental convex mirrors. Assuming that the aperture of the projection system is far enough from CMA, then xIpn in Eq. (1) and zIp in Eq. (2) approach the optical axis of each elemental convex mirror and focal length, respectively, as shown in Fig. 3(a). In this case, the viewer can observe only a single AIP through each elemental convex mirror within the viewing zone. Based on the analysis of Fig. 3(a), we want to analyze the proposed MPII method. Let us consider the two projection systems as shown in Fig. 3(b) for simplicity. This can be easily extended to the general case of the MPII. We assume that two projection systems are Δxp away from each other, as shown in Fig. 3(b). In this case, the viewer can observe two AIPs through each elemental convex mirror due to the separation between two projection systems. The relation between Δxp and ΔxIp as shown in Fig. 3(b) can be calculated by using Eq. (1) and given by zp : (3) ΔxIp Δxp 1 − zp − f Here, Δxp represents the distance between the projection systems and ΔxIp represents the distance between the AIPs of the projection systems within a single elemental convex mirror. From Eq. (3), we can see that the Fig. 2. Geometrical relationship between the aperture of the projection system and its AIPs in the CMA: (a) integrated point located in the virtual field and (b) integrated point located in the real field.
bottom of the mirror array. The point (zp , xp ) is a position of the aperture of the projection system along the x and z axes. P represents the pitch of an elemental convex mirror in the CMA. f denotes the focal length of an elemental convex mirror, and xIpn represents the x coordinate of the AIP by the nth elemental convex mirror, in which the valid xIpn is restricted by n − 1P ≤ xIpn ≤ nP, where n is a natural number. The imaging distance of an aperture of the projector system measured from the CMA can be given by the following equation based on Gaussian formula: zIp
zp f : zp − f
(2)
From Eqs. (1) and (2), we can see that the aperture of the projection system located at (zp , xp ) is imaged at the AIP located at (zIp , xIpn ) for the nth elemental convex mirror. In this case, the aperture of the projection system and its AIP can be considered as the entrance pupil and the exit pupil, respectively. Each elemental convex mirror itself acts as both the entrance window and the exit window. Using the calculated AIPs, the diverging angle can be defined as the angle made with the exit window by the chief ray starting at the center of the exit pupil, as shown in Fig. 3(a). Then, the viewing zone is
Fig. 3. Relationship between projection system and AIPs: (a) conventional SPII method and (b) proposed MPII method.
May 15, 2014 / Vol. 39, No. 10 / OPTICS LETTERS
Fig. 4.
Graph of the distance between AIPs.
distance between the AIPs is linearly proportional to the distance between the projection systems in our MPII method. We plotted the change in the distance between the AIPs, ΔxIp , according to the change in the distance between the projection systems, Δxp , and in the distance between the CMA and the projection systems, zp , in Fig. 4, where the focal length of the elemental convex mirror is f −7.47 mm. For example, ΔxIp becomes 1.85 mm when Δxp is 200 mm and zp is 800 mm. This means that the viewer observes two AIPs separated by 1.85 mm through the elemental convex mirror. To verify the theoretical analysis and to demonstrate the practical feasibility of the proposed MPII method, the optical experiments were performed. Figure 5(a) shows the system structure in our experiment. In the experiment of the MPII method, 2 × 2 projectors located 800 mm from the CMA were used to project the elemental image arrays to reconstruct the 3D object images. The
Fig. 5. (a) 2 × 2 multi-projection system, (b) CMA, (c) elemental image array, and (d) example of the reconstructed 3D object image.
2855
projectors were 200 mm away from each other, as shown in Fig. 5(a). The total size of the CMA, which is composed of 90 × 50 elemental convex mirrors, is 373.5 mm× 680 mm, as shown in Fig. 5(b). Here, 50 × 50 elemental convex mirrors throughout the CMA were used for reconstruction of the 3D objects. The focal length and the diameter of an elemental convex mirror are −7.47 and 7.47 mm, respectively. The elemental image arrays for 2 × 2 projections were generated using a graphical tool [15]. Each generated elemental image array was composed of 50 × 50 elemental images. An example of the elemental image array used in our experiment is shown in Fig. 5(c). A 3D volumetric object, “Dice,” and three plane alphabet letters, D, S, and U, are used as the test objects. And the “Dice” object is set to be rotated 30° against the CMA in the clockwise direction. Figure 5(d) shows an example of the reconstruction image, which was taken under white light. Figures 6(a) and 6(b) show optically reconstructed 3D object images by use of SPII and MPII, respectively. We can observe the AIPs of the projection systems from the enlarged images at the right side of Fig. 6. In the above discussion, the resolution of the reconstructed image from the conventional SPII method is found to be the same as the number of the CMA, as shown in Fig. 6(a), from observing a single AIP by a single-projection system. Therefore, the resolution of SPII is 50 × 50 pixels. However, the reconstructed images from the proposed MPII method are shown in Fig. 6(b). Here we can observe the increased number of AIPs from using multiple projection systems. In fact, the AIP for each convex mirror becomes a single pixel observed by a viewer. In the proposed MPII, we can generate multiple AIPs through each convex mirror. This can increase the resolution of the 3D images. Therefore, the resolution of the MPII increases in proportion to the number of projection systems, as shown in Fig. 6(b). In our experiment, the resolution of MPII is 100 × 100 pixels, because the 2 × 2 projectors were used. In addition, Fig. 7 shows the optically reconstructed 3D object images from MPII, which are captured from the three different lateral directions of the left,
Fig. 6. Reconstructed 3D object images: (a) the conventional SPII method and (b) the proposed MPII method.
2856
OPTICS LETTERS / Vol. 39, No. 10 / May 15, 2014
Research Foundation of Korea (NRF) grant funded by the Korean government (Ministry of Science, ICT and Future Planning) (No. 2013-067321).
Fig. 7. 3D object images reconstructed at the different viewing directions in the proposed MPII.
center, and right to confirm the prospective variations of the reconstructed object images. In conclusion, we have proposed a novel (to our knowledge) PII method based on multi-projection. Multi-projection can increase the resolution of a 3D reconstructed image by increasing the number of AIPs observed by the viewer. The proposed MPII was analyzed using the geometrical relationship. To show the feasibility of the proposed MPII, the optical experiments were performed successfully. The experimental results verify that our MPII method can improve the resolution of 3D images using commercial projectors without any modification. We believe that our method may serve as a new method for large-scale 3D display or movie applications based on integral imaging. This work was supported in part by the IT R&D program of MKE/KEIT. [10041682, Development of highdefinition 3D image processing technologies using advanced integral imaging with improved depth range], and this work was supported in part by the National
References 1. A. Stern and B. Javidi, Proc. IEEE 94, 591 (2006). 2. F. Okano, J. Arai, H. Hoshino, and I. Yuyama, Opt. Eng. 38, 1072 (1999). 3. R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, Proc. IEEE 97, 1067 (2009). 4. M. Cho, M. Daneshpanah, I. Moon, and B. Javidi, Proc. IEEE 99, 556 (2011). 5. K. Wakunami, M. Yamaguchi, and B. Javidi, Opt. Lett. 37, 5103 (2012). 6. X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, Appl. Opt. 52, 546 (2013). 7. M. Cho and D. Shin, J. Opt. Soc. Korea 17, 410 (2013). 8. J. S. Jang and B. Javidi, Opt. Express 12, 1077 (2004). 9. J. S. Jang, Y. S. Oh, and B. Javidi, Opt. Express 12, 557 (2004). 10. H. Liao, M. Iwahara, N. Hata, and T. Dohi, Opt. Express 12, 1067 (2004). 11. M. Okui, J. Arai, Y. Nojiri, and F. Okano, Appl. Opt. 45, 9132 (2006). 12. Y. Kim, S. G. Park, S.-W. Min, and B. Lee, Appl. Opt. 48, H71 (2009). 13. Y. Kim, S. G. Park, S.-W. Min, and B. Lee, Appl. Opt. 50, B18 (2011). 14. L. Zhang, Y. Yang, X. Zhao, Z. Fang, and X. Yuan, Opt. Express 20, 26021 (2012). 15. J.-Y. Jang, D. Shin, and E.-S. Kim, Opt. Express 22, 1533 (2014).
2853
Multi-projection integral imaging by use of a convex mirror array Jae-Young Jang,1 Donghak Shin,2,* Byung-Gook Lee,2 and Eun-Soo Kim3 1
2
iReal Co. Ltd., Dongseo University, Churye-ro, Sasang-Gu, Busan 617-716, South Korea Institute of Ambient Intelligence, Dongseo University, Churye-ro, Sasang-Gu, Busan 617-716, South Korea 3
HoloDigilog Human Media Research Center (HoloDigilog), 3D Display Research Center (3DRC), Kwangwoon University, Wolgye-Dong, Nowon-Gu, Seoul 139-701, South Korea *Corresponding author: [email protected] Received February 24, 2014; revised April 7, 2014; accepted April 8, 2014; posted April 9, 2014 (Doc. ID 206417); published May 6, 2014
We propose a new multi-projection integral imaging scheme using a convex mirror array. In the proposed scheme, to overcome the resolution limitation of the conventional method due to observing the single aperture imaging point (AIP) from each convex mirror, we introduce the multi-projection to obtain multiple AIPs per convex mirror so that the viewer observes the resolution-improved 3D reconstructed images. We validate the theoretical analysis of the proposed scheme and confirm its feasibility through the optical experiments. To our best knowledge, this is the first report to generate multiple AIPs per convex mirror in a projection integral imaging system. © 2014 Optical Society of America OCIS codes: (100.6890) Three-dimensional image processing; (110.6880) Three-dimensional image acquisition; (110.2990) Image formation theory. http://dx.doi.org/10.1364/OL.39.002853
The three-dimensional (3D) integral imaging technique is capable of producing full parallax 3D images in space using two-dimensional (2D) display devices under incoherent light [1–7]. Basically, the integral imaging system is composed of pickup and display parts. In the pickup part, the 3D objects are recorded into the elemental images through the lenslet array. And 3D images are generated by crossing rays coming from the elemental images through the lenslet array in the display part. Recently, various projection-type integral imaging (PII) systems have been studied for large television or movie applications [8–14]. Among them, Jang and Javidi proposed a single-projection-type integral imaging (SPII) system using a convex mirror array (CMA) as a screen [8]. This system could provide a dramatically improved viewing angle, no depth reverse process, and flippingfree observations of 3D images. Even though this CMAbased SPII system has many advantages, the resolution of the 3D image is dependent on the total number of elemental convex mirrors in the CMA used for optical display. That is, the viewer observes a single imaging point through each convex mirror in the CMA. In addition, the CMA-based PII method may be limited to using commercial projectors directly, because relay optics with low magnification is additionally needed for pixel matching between elemental images and the CMA within the small area. In this Letter, we propose a novel multi-projection integral imaging (MPII) scheme using a CMA. Compared with the conventional SPII, multiple projectors at different positions are projected to the same CMA at the same time. In so doing, we can improve the resolution of the 3D reconstructed image in PII. Figure 1 shows a schematic diagram of the proposed MPII system using CMA. In the proposed method, multiprojection is performed into the same CMA at the same time. Compared with the conventional SPII with a microconvex mirror array [8], we use a CMA with a large diameter. This enables us to directly use the commercial 0146-9592/14/102853-04$15.00/0
projectors without any modification or additional relay optics. The corresponding elemental images for each projector are projected into the CMA as shown in Fig. 1. Then, the viewer observes the displayed 3D object image through the CMA. We first analyze the geometrical relationship between a single-projection system and a CMA. As shown in Fig. 2, the elemental image array is projected onto the CMA through the projection system. In our analysis, the aperture of the projection system is assumed to be a pinhole. Then, the geometrical relationship between the aperture of the projector and its corresponding aperture imaging point (AIP) by the nth elemental convex mirror in the CMA can be given by xIpn xp
zp zp − f
n−
1 P − xp : 2
(1)
Here, the origin of the coordinates is assumed to be the edge of the elemental convex mirror located at the
Fig. 1.
Schematic diagram of the proposed MPII scheme.
© 2014 Optical Society of America
2854
OPTICS LETTERS / Vol. 39, No. 10 / May 15, 2014
calculated by the common diverging angles from elemental convex mirrors. Assuming that the aperture of the projection system is far enough from CMA, then xIpn in Eq. (1) and zIp in Eq. (2) approach the optical axis of each elemental convex mirror and focal length, respectively, as shown in Fig. 3(a). In this case, the viewer can observe only a single AIP through each elemental convex mirror within the viewing zone. Based on the analysis of Fig. 3(a), we want to analyze the proposed MPII method. Let us consider the two projection systems as shown in Fig. 3(b) for simplicity. This can be easily extended to the general case of the MPII. We assume that two projection systems are Δxp away from each other, as shown in Fig. 3(b). In this case, the viewer can observe two AIPs through each elemental convex mirror due to the separation between two projection systems. The relation between Δxp and ΔxIp as shown in Fig. 3(b) can be calculated by using Eq. (1) and given by zp : (3) ΔxIp Δxp 1 − zp − f Here, Δxp represents the distance between the projection systems and ΔxIp represents the distance between the AIPs of the projection systems within a single elemental convex mirror. From Eq. (3), we can see that the Fig. 2. Geometrical relationship between the aperture of the projection system and its AIPs in the CMA: (a) integrated point located in the virtual field and (b) integrated point located in the real field.
bottom of the mirror array. The point (zp , xp ) is a position of the aperture of the projection system along the x and z axes. P represents the pitch of an elemental convex mirror in the CMA. f denotes the focal length of an elemental convex mirror, and xIpn represents the x coordinate of the AIP by the nth elemental convex mirror, in which the valid xIpn is restricted by n − 1P ≤ xIpn ≤ nP, where n is a natural number. The imaging distance of an aperture of the projector system measured from the CMA can be given by the following equation based on Gaussian formula: zIp
zp f : zp − f
(2)
From Eqs. (1) and (2), we can see that the aperture of the projection system located at (zp , xp ) is imaged at the AIP located at (zIp , xIpn ) for the nth elemental convex mirror. In this case, the aperture of the projection system and its AIP can be considered as the entrance pupil and the exit pupil, respectively. Each elemental convex mirror itself acts as both the entrance window and the exit window. Using the calculated AIPs, the diverging angle can be defined as the angle made with the exit window by the chief ray starting at the center of the exit pupil, as shown in Fig. 3(a). Then, the viewing zone is
Fig. 3. Relationship between projection system and AIPs: (a) conventional SPII method and (b) proposed MPII method.
May 15, 2014 / Vol. 39, No. 10 / OPTICS LETTERS
Fig. 4.
Graph of the distance between AIPs.
distance between the AIPs is linearly proportional to the distance between the projection systems in our MPII method. We plotted the change in the distance between the AIPs, ΔxIp , according to the change in the distance between the projection systems, Δxp , and in the distance between the CMA and the projection systems, zp , in Fig. 4, where the focal length of the elemental convex mirror is f −7.47 mm. For example, ΔxIp becomes 1.85 mm when Δxp is 200 mm and zp is 800 mm. This means that the viewer observes two AIPs separated by 1.85 mm through the elemental convex mirror. To verify the theoretical analysis and to demonstrate the practical feasibility of the proposed MPII method, the optical experiments were performed. Figure 5(a) shows the system structure in our experiment. In the experiment of the MPII method, 2 × 2 projectors located 800 mm from the CMA were used to project the elemental image arrays to reconstruct the 3D object images. The
Fig. 5. (a) 2 × 2 multi-projection system, (b) CMA, (c) elemental image array, and (d) example of the reconstructed 3D object image.
2855
projectors were 200 mm away from each other, as shown in Fig. 5(a). The total size of the CMA, which is composed of 90 × 50 elemental convex mirrors, is 373.5 mm× 680 mm, as shown in Fig. 5(b). Here, 50 × 50 elemental convex mirrors throughout the CMA were used for reconstruction of the 3D objects. The focal length and the diameter of an elemental convex mirror are −7.47 and 7.47 mm, respectively. The elemental image arrays for 2 × 2 projections were generated using a graphical tool [15]. Each generated elemental image array was composed of 50 × 50 elemental images. An example of the elemental image array used in our experiment is shown in Fig. 5(c). A 3D volumetric object, “Dice,” and three plane alphabet letters, D, S, and U, are used as the test objects. And the “Dice” object is set to be rotated 30° against the CMA in the clockwise direction. Figure 5(d) shows an example of the reconstruction image, which was taken under white light. Figures 6(a) and 6(b) show optically reconstructed 3D object images by use of SPII and MPII, respectively. We can observe the AIPs of the projection systems from the enlarged images at the right side of Fig. 6. In the above discussion, the resolution of the reconstructed image from the conventional SPII method is found to be the same as the number of the CMA, as shown in Fig. 6(a), from observing a single AIP by a single-projection system. Therefore, the resolution of SPII is 50 × 50 pixels. However, the reconstructed images from the proposed MPII method are shown in Fig. 6(b). Here we can observe the increased number of AIPs from using multiple projection systems. In fact, the AIP for each convex mirror becomes a single pixel observed by a viewer. In the proposed MPII, we can generate multiple AIPs through each convex mirror. This can increase the resolution of the 3D images. Therefore, the resolution of the MPII increases in proportion to the number of projection systems, as shown in Fig. 6(b). In our experiment, the resolution of MPII is 100 × 100 pixels, because the 2 × 2 projectors were used. In addition, Fig. 7 shows the optically reconstructed 3D object images from MPII, which are captured from the three different lateral directions of the left,
Fig. 6. Reconstructed 3D object images: (a) the conventional SPII method and (b) the proposed MPII method.
2856
OPTICS LETTERS / Vol. 39, No. 10 / May 15, 2014
Research Foundation of Korea (NRF) grant funded by the Korean government (Ministry of Science, ICT and Future Planning) (No. 2013-067321).
Fig. 7. 3D object images reconstructed at the different viewing directions in the proposed MPII.
center, and right to confirm the prospective variations of the reconstructed object images. In conclusion, we have proposed a novel (to our knowledge) PII method based on multi-projection. Multi-projection can increase the resolution of a 3D reconstructed image by increasing the number of AIPs observed by the viewer. The proposed MPII was analyzed using the geometrical relationship. To show the feasibility of the proposed MPII, the optical experiments were performed successfully. The experimental results verify that our MPII method can improve the resolution of 3D images using commercial projectors without any modification. We believe that our method may serve as a new method for large-scale 3D display or movie applications based on integral imaging. This work was supported in part by the IT R&D program of MKE/KEIT. [10041682, Development of highdefinition 3D image processing technologies using advanced integral imaging with improved depth range], and this work was supported in part by the National
References 1. A. Stern and B. Javidi, Proc. IEEE 94, 591 (2006). 2. F. Okano, J. Arai, H. Hoshino, and I. Yuyama, Opt. Eng. 38, 1072 (1999). 3. R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, Proc. IEEE 97, 1067 (2009). 4. M. Cho, M. Daneshpanah, I. Moon, and B. Javidi, Proc. IEEE 99, 556 (2011). 5. K. Wakunami, M. Yamaguchi, and B. Javidi, Opt. Lett. 37, 5103 (2012). 6. X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, Appl. Opt. 52, 546 (2013). 7. M. Cho and D. Shin, J. Opt. Soc. Korea 17, 410 (2013). 8. J. S. Jang and B. Javidi, Opt. Express 12, 1077 (2004). 9. J. S. Jang, Y. S. Oh, and B. Javidi, Opt. Express 12, 557 (2004). 10. H. Liao, M. Iwahara, N. Hata, and T. Dohi, Opt. Express 12, 1067 (2004). 11. M. Okui, J. Arai, Y. Nojiri, and F. Okano, Appl. Opt. 45, 9132 (2006). 12. Y. Kim, S. G. Park, S.-W. Min, and B. Lee, Appl. Opt. 48, H71 (2009). 13. Y. Kim, S. G. Park, S.-W. Min, and B. Lee, Appl. Opt. 50, B18 (2011). 14. L. Zhang, Y. Yang, X. Zhao, Z. Fang, and X. Yuan, Opt. Express 20, 26021 (2012). 15. J.-Y. Jang, D. Shin, and E.-S. Kim, Opt. Express 22, 1533 (2014).
