Improvement method of integral imaging quality based on an aperture-tunable lens array JianLei Zhang, XiaoRui Wang,* YuJiao Chen, Shuo Yu, QiPing Zhang, and ZhaoHui Li School of Physics and Optoelectronic Engineering, Xidian University, Xi’an Shaanxi 710071, China *Corresponding author: [email protected] Received 5 June 2014; revised 17 July 2014; accepted 17 July 2014; posted 18 July 2014 (Doc. ID 213570); published 25 August 2014
A method for improving the viewing quality of integral imaging (II) is proposed, based on using an aperture-tunable lens array (LA). The proposed method uses a liquid crystal (LC) panel without a backlighting unit to tune the aperture of an LA dynamically. The shape and transmittance of the aperture can be controlled arbitrarily by programming the state of the pixels on the LC panel. Adding the temporal multiplexing technique, the viewing quality can be improved by the after-image effect of the human eye. Moreover, the relationships between the lateral resolution and the aperture tuning pattern and the depth of field and the aperture tuning pattern are derived, respectively. The product of the depth of field, the lateral resolution squared, and the lateral viewing range is proposed as a new figure of merit for an II system. Experimental results show the validity of the proposed method. © 2014 Optical Society of America OCIS codes: (100.6890) Three-dimensional image processing; (110.4190) Multiple imaging; (120.2040) Displays. http://dx.doi.org/10.1364/AO.53.005654
1. Introduction
Integral imaging (II) is one of the most feasible threedimensional (3D) capture and display techniques. It has attracted much attention due to its various advantages such as quasi-continuous, full-color viewpoints within a viewing angle, full parallax, and ability to work with incoherent light [1,2]. Despite its many advantages, the viewing quality, such as the lateral resolution, depth of field, and viewing angle of the reconstructed 3D image, is still poor. Various approaches have been proposed to enhance the viewing angle [3–8], improve the viewing resolution [9–13], and enlarge the depth of field [14–18]. A trade-off relationship exists among the viewing parameters of the II system. Thus, one of them must be sacrificed to improve the others. Moreover, the key element of an II system, the lens array (LA), is usually a fixed component with fixed physical parameters. So the viewing parameters cannot be 1559-128X/14/255654-06$15.00/0 © 2014 Optical Society of America 5654
APPLIED OPTICS / Vol. 53, No. 25 / 1 September 2014
changed after the system architecture is determined. A digital II system using a spatial light modulator as a Fresnel LA is proposed [19]. In this way, the physical parameters of the LA can be programmed. However, because of pixelation and quantization of the Fresnel LA pattern, higher diffraction orders and multiple focal points emerge. In addition to aberration and lens deviation, the pixel pitch of the display device and the elemental lens pitch are the main factors affecting the reconstructed image quality. For the current display panel, the pixel size is limited to no less than dozens of micrometers. Moreover, each elemental lens should be large enough to cover a number of pixels dozens of times more than the number of views to provide a focal accommodation cue off the screen [20]. Thus, the pitch of the elemental lens in the LA should be no less than a few millimeters. Therefore, the size and pitch of the LA become the fundamental factors affecting the viewing quality. In this paper, we propose an II display system with an aperture-tunable LA to improve the viewing quality. In the proposed method, the aperture of each
elemental lens in the LA is controlled by a liquid crystal (LC) panel. The LC panel acts as a mask to adjust the aperture shape and transmittance of each elemental lens. Therefore, the generation and changing of the mask are done electrically, without any mechanical movement. Adding the temporal multiplexing technique, the viewing quality can be improved and the trade-off relationships among the lateral viewing resolution, depth of field, and viewing range can be relieved. 2. Integral Imaging with an Aperture-Tunable Lens Array
Figure 1 depicts the principle of the conventional II display system. Each elemental lens samples the ray information at its location. A 3D image is reconstructed by displaying the recorded 2D elemental images on the focal plane of the LA with a 2D display device. In conventional II, the aperture of the LA is fixed, so the sampling rate of the LA remains constant. Figure 2 shows the schematic diagram of the proposed method. In the proposed method, an LC panel without a backlighting unit is placed in front of the LA and is used to tune the aperture of each elemental lens in the LA. The aperture shape of the LA can be tuned by programming the state of the pixels of the LC panel. That is to say, the filling factor of the LA can be controlled arbitrarily. In addition, an 8 bit LC panel can tune the transmittance of the LA among 256 levels from 0 to 1. The aperture-tunable LA can be applied either in the pickup stage or in the display stage. When used in the pickup stage, the LC panel can be used as a tunable amplitude modulator or a mask to reduce the filling factor, which can improve the depth of field [16,17]. However, the optimal aperture pattern of the LA can be quite different from one application to another. Unlike the existing methods, the aperture of the LA can be tuned freely in the proposed method for different application purposes, which adds the to flexibility of an II system. In this paper, we focus on the application of the aperture-tunable LA in the display stage. The aperture of each elemental lens can be segmented into many parts by controlling the state of the pixels on the LC panel. As shown in Figs. 2(a) and 2(b), each
Fig. 1. Schematic diagram of conventional II display system.
Fig. 2. Schematic diagram of the II display system with an aperture-tunable LA: (a) aperture tuning pattern 1, (b) aperture tuning pattern 2, and (c) improved viewing quality based on time multiplexing of patterns 1 and 2.
aperture of the elemental lens in the LA is segmented into two parts. The upper half of each elemental lens in the LA is made transparent with mask 1 displayed on the LC panel. The lower half of each elemental lens in the LA is made transparent with mask 2 displayed on the LC panel. Within the response time of the human eye, the lower half and upper half of the aperture of each elemental lens are tuned to the “transmittance” state sequentially and the corresponding elemental images are displayed synchronously. Thus the viewing quality can be improved by the after-image effect of human eye. It is worth noting that the elemental lens becomes a decentered elemental lens [21] when the upper half or lower half of each elemental lens in the LA is made transparent with the other half blocked. So the position of the elemental images corresponding to a different aperture tuning pattern does not change. In general, it is assumed that the aperture of each elemental lens is segmented into K × K parts. We only consider square microlenses in this paper, but it is not difficult to expand our proposed method to other microlens shapes. Figure 3 shows an example of the aperture tuning pattern. We consider one aperture of the LA for simplification. At the time of t t1, the upper left part of the aperture is tuned to the “transmittance” state, with the other parts blocked. Next, at the time of t t2, the upper right part of the aperture is tuned to the “transmittance” state. At the time of t t3, the lower left part of the aperture is tuned to the “transmittance” state. At the time of t t4 , the lower right part of the aperture is tuned to the “transmittance” state. Then, we repeat this process over and over. In addition, according to the aperture tuning pattern, the corresponding elemental images should be displayed synchronously. If the aperture tuning patterns change fast enough to raise 1 September 2014 / Vol. 53, No. 25 / APPLIED OPTICS
5655
acceptable blur is produced. In general, the acceptable blur is determined by the pixel size of the image sensor. In a similar way, the depth of field for II is defined as the range in which the lateral spot size of the reconstructed 3D image should not exceed D0 , which is given by the acceptable blur for human eyes. So the depth of field for II with an aperturetunable LA can be expressed as Δzf 2Nf D0 ∕p − 1∕K:
Fig. 3. Aperture tuning pattern with 2 × 2 segmentation at (a) t t1 , (b) t t2 , (c) t t3 , and (d) t t4 .
the after-image effect of human eye, the reconstructed 3D scene with a different aperture tuning pattern can be observed simultaneously, as shown in Fig. 2(c). Therefore, the viewing quality can be improved. Next, we analyze the effect of a different tuning pattern on the lateral resolution and depth of field. In the following sections, we only take one dimension into account for simplicity. As mentioned, it is assumed that the aperture of each elemental lens is segmented into K parts in one dimension. The lateral resolution of the reconstructed 3D image in II is determined by many system parameters, for example, the size and the pitch of the elemental lens and the resolution of the 2D display device. However, one of the most fundamental factors that limit the lateral resolution of the reconstructed 3D image is the pitch of the elemental lens, which determines the sampling rate in the spatial dimension. Since the resolution of the 2D display device is finite, the rays emitted from one pixel diverge rather than propagate in parallel after traveling through the LA. Now, suppose that each elemental image covers N × N pixels. When the LA is positioned as z 0, the spot size s at z > 0 becomes approximately 1 z∕Nf p for the conventional II without considering the diffraction effect [22]. With an aperture-tunable LA, the aperture of each elemental lens is segmented into K parts sequentially, and thus the spot size s at z > 0 becomes 1∕K z∕Nf p. So the lateral resolution for an II system with an aperture-tunable LA becomes Rz;p;K
KNf : Nf zKp
(1)
For a given 2D optical system, the depth of field is defined as the amount that the object may be shifted with respect to some reference plane before the 5656
APPLIED OPTICS / Vol. 53, No. 25 / 1 September 2014
(2)
A trade-off relationship exists among the viewing parameters of the II system. For example, a simple way to enhance the viewing angle is to increase the elemental lens pitch with the focal length of LA constant. However, the viewing resolution and depth of field of the reconstructed 3D image are degraded due to the large elemental lens pitch. The viewing angle, the depth of field, and the viewing resolution of the reconstructed 3D image are closely related to one another, and one of them must be sacrificed to improve the others. To study the relationship between the lateral resolution and the depth of field of the reconstructed 3D image, the product of depth of field and resolution squared (PDRS) is proposed [18]. Unfortunately, the PDRS does not include the viewing angle, which is also an important viewing parameter. A characteristic equation of an II system, which is the product of the square of the image resolution, the image depth, and the tangent of one-half of the viewing angle, is induced to show the relationship among the viewing parameters [23]. However, in order to characterize the quality of an II system objectively, the figure of merit for an II system should be nondimensional. So we propose the product of the depth of field, the lateral resolution squared, and the lateral viewing range (PDRV) as the figure of merit for an II system. The lateral viewing range refers to the range within which an observer can move laterally to see the correct reconstructed 3D image with the flipping effect. Thus the lateral viewing range is p∕Lf , where L is the distance between the LA and the observer. For an II system with an aperturetunable LA, the PDRV can be expressed as PDRV
2KLKD0 − pN 3 f 2 : Nf zKp2
(3)
For K 1, the PDRV shows the relationships among the depth of field, the lateral resolution squared, and the lateral viewing range for a conventional II system. In addition, a higher value for the PDRV means a better overall performance of an II system. Figure 4(a) shows the relationship of the lateral resolution versus the aperture tuning pattern with K-segmentation and the relationship of the depth of field versus the aperture tuning pattern with K-segmentation. K 1 corresponds to the conventional II case. The aperture and focal length of the elemental lens are assumed to be 10 mm. Each
elemental image covers 30 × 30 pixels. The size of acceptable blur is set to be 10.5 mm. We can see that both the lateral resolution and depth of field can be improved by increasing K. Moreover, the increase in the lateral resolution and depth of field slows down when the value of K increases. This is because the aperture tuning technique can just improve the sampling rate of the LA by time-division multiplexing. The effect of finite pixel size on the lateral resolution and depth of field becomes more obvious when the value of K increases. Figure 4(b) shows the relationship of the PDRV versus the aperture tuning pattern with K-segmentation. The PDRV increases significantly with an increase in K, which means the overall performance of the II display system becomes better. In other words, the trade-off relationships among the lateral viewing resolution, the depth of field, and the lateral viewing range can be relieved by increasing K. Considering that the minimum refresh rate of the display device to induce the after-image effect is 30 frames∕s, the system needs to realize, for an aperture tuning pattern with 5 × 5 segmentation, at least a refresh rate of 750 frames∕s. In addition, to eliminate the flickers, LC panels with a refresh rate of larger than 1000 are needed. In this situation, there may be no commercial LC panels available. New development of high-refresh-rate LC panels is needed.
Fig. 4. (a) Lateral resolution and depth of field versus the aperture tuning pattern with K-segmentation and (b) PDRV versus the aperture tuning pattern with K-segmentation.
3. Experimental Results and Discussions
Preliminary experiments have been performed to confirm the validity of the proposed method. An experimental II display setup with an aperture-tunable LA is shown in Fig. 5. Due to the limitations of the experimental condition, we do not have an LC panel with a refresh rate larger than 60 frames∕s. The LC panel is disassembled from a View Sonic VA721 LCD monitor, which acts as a mask to tune the aperture of the LA. The LC panel size is 431.8 mm in the diagonal direction, and the resolution is 1280 (horizontal) by 1024 (vertical). The pixel size of the LC panel is 0.264 mm with an RGB-subpixel structure. It is worth noting that the RGB-subpixel structure is not necessary; a monochrome LC panel is enough to tune the aperture of the LA. The maximum frequency of the LC panel is 60 Hz. The LA we use is composed of 10 × 13 elemental lenses. Each elemental lens is rectangle-shaped and has a uniform base size of 4 mm × 3 mm with negligible separation, and the focal length is 38.1 mm. Therefore, only a small part of the LC panel is used in this experiment, and each elemental lens aperture covers approximately 15 × 11 pixels of the LC panel. Another View Sonic VA721 LCD monitor acts as the 2D display device. Each elemental image is composed of approximately 15 × 11 pixels. The 3D scene is composed of two letters, X and O, positioned at depths of 40 and 120 mm in front of the LA, respectively. The elemental images of the 3D scene corresponding to different aperture tuning patterns are captured by computer based on the ray-tracing theory. Since each elemental lens is rectangle-shaped, the lateral resolution in the horizontal direction is slightly different from that in vertical direction. To investigate the viewing quality enhancement, we compare the reconstructed 3D image of a conventional II display system with that of the proposed method. Figure 6 shows the reconstructed 3D images from different viewpoints for conventional II, i.e., K 1. The viewpoints are captured by a digital camera placed 180 cm in front of the LA. Different viewpoints show different perspectives of the reconstructed 3D scene. Due to the large aperture of the LA, the lateral resolution of the reconstructed 3D images is low.
Fig. 5. Experimental aperture-tunable II display setup. 1 September 2014 / Vol. 53, No. 25 / APPLIED OPTICS
5657
Fig. 6. Reconstructed 3D images from different viewpoints for conventional II.
An aperture tuning pattern with 2 × 2 segmentation is used in this experiment. Each aperture of the elemental lens is segmented into 2 × 2 parts. As is shown in Fig. 3, the upper left part, upper right part, lower left part, and lower right part of each aperture of the LA are tuned to the “transmittance” state in turn. Then, we repeat this process over and over. At the same time, the corresponding elemental images should be displayed synchronously according to the aperture tuning pattern. The frame synchronization between the two LCD monitors can be controlled by a computer. Due to the limitation of
the frame rate of the display panel, flickers can be observed by the human eye. So we set the shutter time of the camera to be longer than the time in which all aperture tuning patterns change sequentially to simulate the after-image effect of human vision. Figure 7 shows the reconstructed 3D images from different viewpoints for II with aperture-tunable LA with 2 × 2 segmentation. The viewpoints are captured by a digital camera placed 180 cm in front of the LA. Compared with Fig. 6, the lateral resolution of the reconstructed 3D images shown in Fig. 7 is clearly improved. Since the lateral spot size of the 3D reconstructed image is reduced, the depth of field is also improved by the aperture tuning technique. However, the price that may be paid to improve the viewing quality by this technique is low light efficiency and the requirement for a high refresh rate of the LC component. In this paper, we just tune the aperture by turning on or off the pixels on the LC panel. It is worth noting that the transmittance of the aperture can be tuned among 256 levels from 0 to 1 for an 8 bit LC panel. Thus we can tune the aperture by a given transmittance function. Another important advantage is that the aperture of the LA is tuned by an LC panel and can be time multiplexed fast without any mechanical movement. However, the aperture-tunable LA based on an LC panel has some problems. The light rays can leak from the gaps between the LC cells and from the cells that cannot be completely turned off. The electronic elements on LCD pixels occlude lights, which causes slight diffraction artifacts. These problems can be relieved by eliminating the color filter structure, increasing the aperture ratio, or using a thinner LC panel. 4. Conclusion
In this paper, a viewing quality improvement method in II is proposed based on using an aperture-tunable LA. By placing an LC panel in front of the LA, the aperture of the LA can be tuned by turning on or off the pixels on the LC panel. By use of the afterimage effect of the human eye, the viewing resolution can be improved by the addition of the temporal multiplexing technique. The effect of the aperture tuning pattern on the improvement of viewing quality is derived. The PDRV is proposed as a new figure of merit for an II system. The experimental results support the feasibility of the proposed method. This work is supported by the National Natural Science Foundation of China (61377007 and 61007014). References
Fig. 7. Reconstructed 3D images from different viewpoints for II with an aperture-tunable LA with 2 × 2 segmentation. 5658
APPLIED OPTICS / Vol. 53, No. 25 / 1 September 2014
1. X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, “Advances in three-dimensional integral imaging—sensing, display, and applications [Invited],” Appl. Opt. 52, 546–560 (2013). 2. J. S. Geng, “Three-dimensional display technologies,” Adv. Opt. Photon. 5, 456–535 (2013).
3. Y. Kim, J. H. Park, S. W. Min, S. Jung, H. J. Choi, and B. Lee, “Wide-viewing-angle integral three-dimensional imaging system by curving a screen and a lens array,” Appl. Opt. 44, 546–552 (2005). 4. J. Y. Jang, H. S. Lee, S. Cha, and S. H. Shin, “Viewing angle enhanced integral imaging display by using a high refractive index medium,” Appl. Opt. 50, B71–B76 (2011). 5. H. Kim, J. Hahn, and B. Lee, “The use of a negative index planoconcave lens array for wide-viewing angle integral imaging,” Opt. Express 16, 21865–21880 (2008). 6. R. Martinez-Cuenca, H. Navarro, G. Saavedra, B. Javidi, and M. Martinez-Corral, “Enhanced viewing-angle integral imaging by multiple-axis telecentric relay system,” Opt. Express 15, 16255–16260 (2007). 7. G. Park, J. H. Jung, K. Hong, Y. Kim, Y. H. Kim, S. W. Min, and B. Lee, “Multi-viewer tracking integral imaging system and its viewing zone analysis,” Opt. Express 17, 17895–17908 (2009). 8. J. S. Jang and B. Javidi, “Improvement of viewing angle in integral imaging by use of moving lenslet arrays with low fill factor,” Appl. Opt. 42, 1996–2002 (2003). 9. J. S. Jang and B. Javidi, “Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics,” Opt. Lett. 27, 324–326 (2002). 10. H. Liao, T. Dohi, and M. Iwahara, “Improved viewing resolution of integral videography by use of rotated prism sheets,” Opt. Express 15, 4814–4822 (2007). 11. Y. Kim, J. Kim, J. M. Kang, J. H. Jung, H. Choi, and B. Lee, “Point light source integral imaging with improved resolution and viewing angle by the use of electrically movable pinhole array,” Opt. Express 15, 18253–18267 (2007). 12. J. H. Park, J. Kim, Y. Kim, and B. Lee, “Resolutionenhanced three-dimension/two-dimension convertible display based on integral imaging,” Opt. Express 13, 1875–1884 (2005).
13. X. R. Wang and H. Hua, “Theoretical analysis for integral imaging performance based on microscanning of a microlens array,” Opt. Lett. 33, 449–451 (2008). 14. A. Castro, Y. Frauel, and B. Javidi, “Integral imaging with large depth of field using an asymmetric phase mask,” Opt. Express 15, 10266–10273 (2007). 15. Y. Kim, J. H. Park, H. Choi, J. Kim, S. W. Cho, and B. Lee, “Depth-enhanced three-dimensional integral imaging by use of multilayered display devices,” Appl. Opt. 45, 4334–4343 (2006). 16. M. Martinez-Corral, B. Javidi, R. Martinez-Cuenca, and G. Saavedra, “Integral imaging with improved depth of field by use of amplitude-modulated microlens arrays,” Appl. Opt. 43, 5806–5813 (2004). 17. R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, “Enhanced depth of field integral imaging with sensor resolution constraints,” Opt. Express 12, 5237–5242 (2004). 18. J. S. Jang, F. Jin, and B. Javidi, “Three-dimensional integral imaging with large depth of focus by use of real and virtual image fields,” Opt. Lett. 28, 1421–1423 (2003). 19. A. O. Yontem and L. Onural, “Integral imaging using phaseonly LCoS spatial light modulators as Fresnel lenslet arrays,” J. Opt. Soc. Am. A 28, 2359–2375 (2011). 20. H. Kakeya and T. Kurokawa, “Energy-efficient integral imaging with suppression of pseudo images,” Opt. Lett. 38, 3227–3229 (2013). 21. S. Sawada and H. Kakeya, “Integral volumetric imaging using decentered elemental lenses,” Opt. Express 20, 25902–25913 (2012). 22. F. Jin, J.-S. Jang, and B. Javidi, “Effects of device resolution on three-dimensional integral imaging,” Opt. Lett. 29, 1345–1347 (2004). 23. S. W. Min, J. Kim, and B. Lee, “New characteristic equation of three-dimensional integral imaging system and its applications,” Jpn. J. Appl. Phys. 44, L71–L74 (2005).
1 September 2014 / Vol. 53, No. 25 / APPLIED OPTICS
5659
A method for improving the viewing quality of integral imaging (II) is proposed, based on using an aperture-tunable lens array (LA). The proposed method uses a liquid crystal (LC) panel without a backlighting unit to tune the aperture of an LA dynamically. The shape and transmittance of the aperture can be controlled arbitrarily by programming the state of the pixels on the LC panel. Adding the temporal multiplexing technique, the viewing quality can be improved by the after-image effect of the human eye. Moreover, the relationships between the lateral resolution and the aperture tuning pattern and the depth of field and the aperture tuning pattern are derived, respectively. The product of the depth of field, the lateral resolution squared, and the lateral viewing range is proposed as a new figure of merit for an II system. Experimental results show the validity of the proposed method. © 2014 Optical Society of America OCIS codes: (100.6890) Three-dimensional image processing; (110.4190) Multiple imaging; (120.2040) Displays. http://dx.doi.org/10.1364/AO.53.005654
1. Introduction
Integral imaging (II) is one of the most feasible threedimensional (3D) capture and display techniques. It has attracted much attention due to its various advantages such as quasi-continuous, full-color viewpoints within a viewing angle, full parallax, and ability to work with incoherent light [1,2]. Despite its many advantages, the viewing quality, such as the lateral resolution, depth of field, and viewing angle of the reconstructed 3D image, is still poor. Various approaches have been proposed to enhance the viewing angle [3–8], improve the viewing resolution [9–13], and enlarge the depth of field [14–18]. A trade-off relationship exists among the viewing parameters of the II system. Thus, one of them must be sacrificed to improve the others. Moreover, the key element of an II system, the lens array (LA), is usually a fixed component with fixed physical parameters. So the viewing parameters cannot be 1559-128X/14/255654-06$15.00/0 © 2014 Optical Society of America 5654
APPLIED OPTICS / Vol. 53, No. 25 / 1 September 2014
changed after the system architecture is determined. A digital II system using a spatial light modulator as a Fresnel LA is proposed [19]. In this way, the physical parameters of the LA can be programmed. However, because of pixelation and quantization of the Fresnel LA pattern, higher diffraction orders and multiple focal points emerge. In addition to aberration and lens deviation, the pixel pitch of the display device and the elemental lens pitch are the main factors affecting the reconstructed image quality. For the current display panel, the pixel size is limited to no less than dozens of micrometers. Moreover, each elemental lens should be large enough to cover a number of pixels dozens of times more than the number of views to provide a focal accommodation cue off the screen [20]. Thus, the pitch of the elemental lens in the LA should be no less than a few millimeters. Therefore, the size and pitch of the LA become the fundamental factors affecting the viewing quality. In this paper, we propose an II display system with an aperture-tunable LA to improve the viewing quality. In the proposed method, the aperture of each
elemental lens in the LA is controlled by a liquid crystal (LC) panel. The LC panel acts as a mask to adjust the aperture shape and transmittance of each elemental lens. Therefore, the generation and changing of the mask are done electrically, without any mechanical movement. Adding the temporal multiplexing technique, the viewing quality can be improved and the trade-off relationships among the lateral viewing resolution, depth of field, and viewing range can be relieved. 2. Integral Imaging with an Aperture-Tunable Lens Array
Figure 1 depicts the principle of the conventional II display system. Each elemental lens samples the ray information at its location. A 3D image is reconstructed by displaying the recorded 2D elemental images on the focal plane of the LA with a 2D display device. In conventional II, the aperture of the LA is fixed, so the sampling rate of the LA remains constant. Figure 2 shows the schematic diagram of the proposed method. In the proposed method, an LC panel without a backlighting unit is placed in front of the LA and is used to tune the aperture of each elemental lens in the LA. The aperture shape of the LA can be tuned by programming the state of the pixels of the LC panel. That is to say, the filling factor of the LA can be controlled arbitrarily. In addition, an 8 bit LC panel can tune the transmittance of the LA among 256 levels from 0 to 1. The aperture-tunable LA can be applied either in the pickup stage or in the display stage. When used in the pickup stage, the LC panel can be used as a tunable amplitude modulator or a mask to reduce the filling factor, which can improve the depth of field [16,17]. However, the optimal aperture pattern of the LA can be quite different from one application to another. Unlike the existing methods, the aperture of the LA can be tuned freely in the proposed method for different application purposes, which adds the to flexibility of an II system. In this paper, we focus on the application of the aperture-tunable LA in the display stage. The aperture of each elemental lens can be segmented into many parts by controlling the state of the pixels on the LC panel. As shown in Figs. 2(a) and 2(b), each
Fig. 1. Schematic diagram of conventional II display system.
Fig. 2. Schematic diagram of the II display system with an aperture-tunable LA: (a) aperture tuning pattern 1, (b) aperture tuning pattern 2, and (c) improved viewing quality based on time multiplexing of patterns 1 and 2.
aperture of the elemental lens in the LA is segmented into two parts. The upper half of each elemental lens in the LA is made transparent with mask 1 displayed on the LC panel. The lower half of each elemental lens in the LA is made transparent with mask 2 displayed on the LC panel. Within the response time of the human eye, the lower half and upper half of the aperture of each elemental lens are tuned to the “transmittance” state sequentially and the corresponding elemental images are displayed synchronously. Thus the viewing quality can be improved by the after-image effect of human eye. It is worth noting that the elemental lens becomes a decentered elemental lens [21] when the upper half or lower half of each elemental lens in the LA is made transparent with the other half blocked. So the position of the elemental images corresponding to a different aperture tuning pattern does not change. In general, it is assumed that the aperture of each elemental lens is segmented into K × K parts. We only consider square microlenses in this paper, but it is not difficult to expand our proposed method to other microlens shapes. Figure 3 shows an example of the aperture tuning pattern. We consider one aperture of the LA for simplification. At the time of t t1, the upper left part of the aperture is tuned to the “transmittance” state, with the other parts blocked. Next, at the time of t t2, the upper right part of the aperture is tuned to the “transmittance” state. At the time of t t3, the lower left part of the aperture is tuned to the “transmittance” state. At the time of t t4 , the lower right part of the aperture is tuned to the “transmittance” state. Then, we repeat this process over and over. In addition, according to the aperture tuning pattern, the corresponding elemental images should be displayed synchronously. If the aperture tuning patterns change fast enough to raise 1 September 2014 / Vol. 53, No. 25 / APPLIED OPTICS
5655
acceptable blur is produced. In general, the acceptable blur is determined by the pixel size of the image sensor. In a similar way, the depth of field for II is defined as the range in which the lateral spot size of the reconstructed 3D image should not exceed D0 , which is given by the acceptable blur for human eyes. So the depth of field for II with an aperturetunable LA can be expressed as Δzf 2Nf D0 ∕p − 1∕K:
Fig. 3. Aperture tuning pattern with 2 × 2 segmentation at (a) t t1 , (b) t t2 , (c) t t3 , and (d) t t4 .
the after-image effect of human eye, the reconstructed 3D scene with a different aperture tuning pattern can be observed simultaneously, as shown in Fig. 2(c). Therefore, the viewing quality can be improved. Next, we analyze the effect of a different tuning pattern on the lateral resolution and depth of field. In the following sections, we only take one dimension into account for simplicity. As mentioned, it is assumed that the aperture of each elemental lens is segmented into K parts in one dimension. The lateral resolution of the reconstructed 3D image in II is determined by many system parameters, for example, the size and the pitch of the elemental lens and the resolution of the 2D display device. However, one of the most fundamental factors that limit the lateral resolution of the reconstructed 3D image is the pitch of the elemental lens, which determines the sampling rate in the spatial dimension. Since the resolution of the 2D display device is finite, the rays emitted from one pixel diverge rather than propagate in parallel after traveling through the LA. Now, suppose that each elemental image covers N × N pixels. When the LA is positioned as z 0, the spot size s at z > 0 becomes approximately 1 z∕Nf p for the conventional II without considering the diffraction effect [22]. With an aperture-tunable LA, the aperture of each elemental lens is segmented into K parts sequentially, and thus the spot size s at z > 0 becomes 1∕K z∕Nf p. So the lateral resolution for an II system with an aperture-tunable LA becomes Rz;p;K
KNf : Nf zKp
(1)
For a given 2D optical system, the depth of field is defined as the amount that the object may be shifted with respect to some reference plane before the 5656
APPLIED OPTICS / Vol. 53, No. 25 / 1 September 2014
(2)
A trade-off relationship exists among the viewing parameters of the II system. For example, a simple way to enhance the viewing angle is to increase the elemental lens pitch with the focal length of LA constant. However, the viewing resolution and depth of field of the reconstructed 3D image are degraded due to the large elemental lens pitch. The viewing angle, the depth of field, and the viewing resolution of the reconstructed 3D image are closely related to one another, and one of them must be sacrificed to improve the others. To study the relationship between the lateral resolution and the depth of field of the reconstructed 3D image, the product of depth of field and resolution squared (PDRS) is proposed [18]. Unfortunately, the PDRS does not include the viewing angle, which is also an important viewing parameter. A characteristic equation of an II system, which is the product of the square of the image resolution, the image depth, and the tangent of one-half of the viewing angle, is induced to show the relationship among the viewing parameters [23]. However, in order to characterize the quality of an II system objectively, the figure of merit for an II system should be nondimensional. So we propose the product of the depth of field, the lateral resolution squared, and the lateral viewing range (PDRV) as the figure of merit for an II system. The lateral viewing range refers to the range within which an observer can move laterally to see the correct reconstructed 3D image with the flipping effect. Thus the lateral viewing range is p∕Lf , where L is the distance between the LA and the observer. For an II system with an aperturetunable LA, the PDRV can be expressed as PDRV
2KLKD0 − pN 3 f 2 : Nf zKp2
(3)
For K 1, the PDRV shows the relationships among the depth of field, the lateral resolution squared, and the lateral viewing range for a conventional II system. In addition, a higher value for the PDRV means a better overall performance of an II system. Figure 4(a) shows the relationship of the lateral resolution versus the aperture tuning pattern with K-segmentation and the relationship of the depth of field versus the aperture tuning pattern with K-segmentation. K 1 corresponds to the conventional II case. The aperture and focal length of the elemental lens are assumed to be 10 mm. Each
elemental image covers 30 × 30 pixels. The size of acceptable blur is set to be 10.5 mm. We can see that both the lateral resolution and depth of field can be improved by increasing K. Moreover, the increase in the lateral resolution and depth of field slows down when the value of K increases. This is because the aperture tuning technique can just improve the sampling rate of the LA by time-division multiplexing. The effect of finite pixel size on the lateral resolution and depth of field becomes more obvious when the value of K increases. Figure 4(b) shows the relationship of the PDRV versus the aperture tuning pattern with K-segmentation. The PDRV increases significantly with an increase in K, which means the overall performance of the II display system becomes better. In other words, the trade-off relationships among the lateral viewing resolution, the depth of field, and the lateral viewing range can be relieved by increasing K. Considering that the minimum refresh rate of the display device to induce the after-image effect is 30 frames∕s, the system needs to realize, for an aperture tuning pattern with 5 × 5 segmentation, at least a refresh rate of 750 frames∕s. In addition, to eliminate the flickers, LC panels with a refresh rate of larger than 1000 are needed. In this situation, there may be no commercial LC panels available. New development of high-refresh-rate LC panels is needed.
Fig. 4. (a) Lateral resolution and depth of field versus the aperture tuning pattern with K-segmentation and (b) PDRV versus the aperture tuning pattern with K-segmentation.
3. Experimental Results and Discussions
Preliminary experiments have been performed to confirm the validity of the proposed method. An experimental II display setup with an aperture-tunable LA is shown in Fig. 5. Due to the limitations of the experimental condition, we do not have an LC panel with a refresh rate larger than 60 frames∕s. The LC panel is disassembled from a View Sonic VA721 LCD monitor, which acts as a mask to tune the aperture of the LA. The LC panel size is 431.8 mm in the diagonal direction, and the resolution is 1280 (horizontal) by 1024 (vertical). The pixel size of the LC panel is 0.264 mm with an RGB-subpixel structure. It is worth noting that the RGB-subpixel structure is not necessary; a monochrome LC panel is enough to tune the aperture of the LA. The maximum frequency of the LC panel is 60 Hz. The LA we use is composed of 10 × 13 elemental lenses. Each elemental lens is rectangle-shaped and has a uniform base size of 4 mm × 3 mm with negligible separation, and the focal length is 38.1 mm. Therefore, only a small part of the LC panel is used in this experiment, and each elemental lens aperture covers approximately 15 × 11 pixels of the LC panel. Another View Sonic VA721 LCD monitor acts as the 2D display device. Each elemental image is composed of approximately 15 × 11 pixels. The 3D scene is composed of two letters, X and O, positioned at depths of 40 and 120 mm in front of the LA, respectively. The elemental images of the 3D scene corresponding to different aperture tuning patterns are captured by computer based on the ray-tracing theory. Since each elemental lens is rectangle-shaped, the lateral resolution in the horizontal direction is slightly different from that in vertical direction. To investigate the viewing quality enhancement, we compare the reconstructed 3D image of a conventional II display system with that of the proposed method. Figure 6 shows the reconstructed 3D images from different viewpoints for conventional II, i.e., K 1. The viewpoints are captured by a digital camera placed 180 cm in front of the LA. Different viewpoints show different perspectives of the reconstructed 3D scene. Due to the large aperture of the LA, the lateral resolution of the reconstructed 3D images is low.
Fig. 5. Experimental aperture-tunable II display setup. 1 September 2014 / Vol. 53, No. 25 / APPLIED OPTICS
5657
Fig. 6. Reconstructed 3D images from different viewpoints for conventional II.
An aperture tuning pattern with 2 × 2 segmentation is used in this experiment. Each aperture of the elemental lens is segmented into 2 × 2 parts. As is shown in Fig. 3, the upper left part, upper right part, lower left part, and lower right part of each aperture of the LA are tuned to the “transmittance” state in turn. Then, we repeat this process over and over. At the same time, the corresponding elemental images should be displayed synchronously according to the aperture tuning pattern. The frame synchronization between the two LCD monitors can be controlled by a computer. Due to the limitation of
the frame rate of the display panel, flickers can be observed by the human eye. So we set the shutter time of the camera to be longer than the time in which all aperture tuning patterns change sequentially to simulate the after-image effect of human vision. Figure 7 shows the reconstructed 3D images from different viewpoints for II with aperture-tunable LA with 2 × 2 segmentation. The viewpoints are captured by a digital camera placed 180 cm in front of the LA. Compared with Fig. 6, the lateral resolution of the reconstructed 3D images shown in Fig. 7 is clearly improved. Since the lateral spot size of the 3D reconstructed image is reduced, the depth of field is also improved by the aperture tuning technique. However, the price that may be paid to improve the viewing quality by this technique is low light efficiency and the requirement for a high refresh rate of the LC component. In this paper, we just tune the aperture by turning on or off the pixels on the LC panel. It is worth noting that the transmittance of the aperture can be tuned among 256 levels from 0 to 1 for an 8 bit LC panel. Thus we can tune the aperture by a given transmittance function. Another important advantage is that the aperture of the LA is tuned by an LC panel and can be time multiplexed fast without any mechanical movement. However, the aperture-tunable LA based on an LC panel has some problems. The light rays can leak from the gaps between the LC cells and from the cells that cannot be completely turned off. The electronic elements on LCD pixels occlude lights, which causes slight diffraction artifacts. These problems can be relieved by eliminating the color filter structure, increasing the aperture ratio, or using a thinner LC panel. 4. Conclusion
In this paper, a viewing quality improvement method in II is proposed based on using an aperture-tunable LA. By placing an LC panel in front of the LA, the aperture of the LA can be tuned by turning on or off the pixels on the LC panel. By use of the afterimage effect of the human eye, the viewing resolution can be improved by the addition of the temporal multiplexing technique. The effect of the aperture tuning pattern on the improvement of viewing quality is derived. The PDRV is proposed as a new figure of merit for an II system. The experimental results support the feasibility of the proposed method. This work is supported by the National Natural Science Foundation of China (61377007 and 61007014). References
Fig. 7. Reconstructed 3D images from different viewpoints for II with an aperture-tunable LA with 2 × 2 segmentation. 5658
APPLIED OPTICS / Vol. 53, No. 25 / 1 September 2014
1. X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, “Advances in three-dimensional integral imaging—sensing, display, and applications [Invited],” Appl. Opt. 52, 546–560 (2013). 2. J. S. Geng, “Three-dimensional display technologies,” Adv. Opt. Photon. 5, 456–535 (2013).
3. Y. Kim, J. H. Park, S. W. Min, S. Jung, H. J. Choi, and B. Lee, “Wide-viewing-angle integral three-dimensional imaging system by curving a screen and a lens array,” Appl. Opt. 44, 546–552 (2005). 4. J. Y. Jang, H. S. Lee, S. Cha, and S. H. Shin, “Viewing angle enhanced integral imaging display by using a high refractive index medium,” Appl. Opt. 50, B71–B76 (2011). 5. H. Kim, J. Hahn, and B. Lee, “The use of a negative index planoconcave lens array for wide-viewing angle integral imaging,” Opt. Express 16, 21865–21880 (2008). 6. R. Martinez-Cuenca, H. Navarro, G. Saavedra, B. Javidi, and M. Martinez-Corral, “Enhanced viewing-angle integral imaging by multiple-axis telecentric relay system,” Opt. Express 15, 16255–16260 (2007). 7. G. Park, J. H. Jung, K. Hong, Y. Kim, Y. H. Kim, S. W. Min, and B. Lee, “Multi-viewer tracking integral imaging system and its viewing zone analysis,” Opt. Express 17, 17895–17908 (2009). 8. J. S. Jang and B. Javidi, “Improvement of viewing angle in integral imaging by use of moving lenslet arrays with low fill factor,” Appl. Opt. 42, 1996–2002 (2003). 9. J. S. Jang and B. Javidi, “Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics,” Opt. Lett. 27, 324–326 (2002). 10. H. Liao, T. Dohi, and M. Iwahara, “Improved viewing resolution of integral videography by use of rotated prism sheets,” Opt. Express 15, 4814–4822 (2007). 11. Y. Kim, J. Kim, J. M. Kang, J. H. Jung, H. Choi, and B. Lee, “Point light source integral imaging with improved resolution and viewing angle by the use of electrically movable pinhole array,” Opt. Express 15, 18253–18267 (2007). 12. J. H. Park, J. Kim, Y. Kim, and B. Lee, “Resolutionenhanced three-dimension/two-dimension convertible display based on integral imaging,” Opt. Express 13, 1875–1884 (2005).
13. X. R. Wang and H. Hua, “Theoretical analysis for integral imaging performance based on microscanning of a microlens array,” Opt. Lett. 33, 449–451 (2008). 14. A. Castro, Y. Frauel, and B. Javidi, “Integral imaging with large depth of field using an asymmetric phase mask,” Opt. Express 15, 10266–10273 (2007). 15. Y. Kim, J. H. Park, H. Choi, J. Kim, S. W. Cho, and B. Lee, “Depth-enhanced three-dimensional integral imaging by use of multilayered display devices,” Appl. Opt. 45, 4334–4343 (2006). 16. M. Martinez-Corral, B. Javidi, R. Martinez-Cuenca, and G. Saavedra, “Integral imaging with improved depth of field by use of amplitude-modulated microlens arrays,” Appl. Opt. 43, 5806–5813 (2004). 17. R. Martinez-Cuenca, G. Saavedra, M. Martinez-Corral, and B. Javidi, “Enhanced depth of field integral imaging with sensor resolution constraints,” Opt. Express 12, 5237–5242 (2004). 18. J. S. Jang, F. Jin, and B. Javidi, “Three-dimensional integral imaging with large depth of focus by use of real and virtual image fields,” Opt. Lett. 28, 1421–1423 (2003). 19. A. O. Yontem and L. Onural, “Integral imaging using phaseonly LCoS spatial light modulators as Fresnel lenslet arrays,” J. Opt. Soc. Am. A 28, 2359–2375 (2011). 20. H. Kakeya and T. Kurokawa, “Energy-efficient integral imaging with suppression of pseudo images,” Opt. Lett. 38, 3227–3229 (2013). 21. S. Sawada and H. Kakeya, “Integral volumetric imaging using decentered elemental lenses,” Opt. Express 20, 25902–25913 (2012). 22. F. Jin, J.-S. Jang, and B. Javidi, “Effects of device resolution on three-dimensional integral imaging,” Opt. Lett. 29, 1345–1347 (2004). 23. S. W. Min, J. Kim, and B. Lee, “New characteristic equation of three-dimensional integral imaging system and its applications,” Jpn. J. Appl. Phys. 44, L71–L74 (2005).
1 September 2014 / Vol. 53, No. 25 / APPLIED OPTICS
5659
