Resolution-enhanced integral imaging in focal mode with a time-multiplexed electrical mask array Yongseok Oh,1 Donghak Shin,2 Byung-Gook Lee,2 Shin-Il Jeong, 1 and Hee-Jin Choi3,* 1
Department of Information and Communication Engineering, Pukyung National University, 45 Yongso-ro, Nam-Gu, Busan, 608-737, South Korea 2 Institute of Ambient Intelligence, Dongseo University,47 Jurye-ro, Sasang-Gu, Busan, 617-716, South Korea 3 Department of Physics, Sejong University, 209 Neungdong-ro, Gwangjin-Gu, Seoul, 143-747, South Korea * [email protected]
Abstract: The inferior resolution of the three-dimensional (3D) image is one of the main problems to be resolved for realizing a commercial autosteresosopic 3D display device. In this paper, a time-multiplexing technique using electrically moving masks is proposed to enhance the resolution of the 3D image realized by integral imaging in a focal mode while preserving the viewing angle of it. ©2014 Optical Society of America OCIS codes: (010.1080) Active or adaptive optics; (110.2990) Image formation theory; (100.6890) Three-dimensional image processing.
References and links 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17.
G. Lippmann, “La photographie integrale,” C. R. Acad. Sci. 146, 446–451 (1908). F. Okano, H. Hoshino, J. Arai, and I. Yuyama, “Real-time pickup method for a three-dimensional image based on integral photography,” Appl. Opt. 36(7), 1598–1603 (1997). T. Okoshi, “Optimum design and depth resolution of lens-sheet and projection-type three-dimensional displays,” Appl. Opt. 10(10), 2284–2291 (1971). J.-H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt. 48(34), H77–H94 (2009). M. Cho and B. Javidi, “Optimization of 3D integral imaging system parameters,” J. Displ. Technol. 8(6), 357– 360 (2012). M. Martínez-Corral, H. Navarro, R. Martínez-Cuenca, G. Saavedra, and B. Javidi, “Full parallax 3-D TV with programmable display parameters,” Opt. Photon. News 22(12), 50 (2011). J.-Y. Jang, D. Shin, and E.-S. Kim, “Optical three-dimensional refocusing from elemental images based on a sifting property of the periodic δ-function array in integral-imaging,” Opt. Express 22(2), 1533–1550 (2014). C.-W. Chen, M. Cho, Y.-P. Huang, and B. Javidi, “Improved viewing zones for projection type integral imaging 3D display using adaptive liquid crystal prism array,” J. Displ. Technol. 10(3), 198–203 (2014). J.-S. Jang, F. Jin, and B. Javidi, “Three-dimensional integral imaging with large depth of focus by use of real and virtual image fields,” Opt. Lett. 28(16), 1421–1423 (2003). J.-S. Jang and B. Javidi, “Large depth-of-focus time-multiplexed three-dimensional integral imaging by use of lenslets with nonuniform focal lengths and aperture sizes,” Opt. Lett. 28(20), 1924–1926 (2003). H. Choi, Y. Kim, J.-H. Park, J. Kim, S.-W. Cho, and B. Lee, “Layered-panel integral imaging without the translucent problem,” Opt. Express 13(15), 5769–5776 (2005). J. Hong, J.-H. Park, S. Jung, and B. Lee, “Depth-enhanced integral imaging by use of optical path control,” Opt. Lett. 29(15), 1790–1792 (2004). D.-H. Shin, B. Lee, and E.-S. Kim, “Multidirectional curved integral imaging with large depth by additional use of a large-aperture lens,” Appl. Opt. 45(28), 7375–7381 (2006). J.-S. Jang and B. Javidi, “Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics,” Opt. Lett. 27(5), 324–326 (2002). Y. Kim, J. Kim, J.-M. Kang, J.-H. Jung, H. Choi, and B. Lee, “Point light source integral imaging with improved resolution and viewing angle by the use of electrically movable pinhole array,” Opt. Express 15(26), 18253– 18267 (2007). H.-J. Choi, Y. M. Kim, J. Jung, K.-M. Jung, and S.-W. Min, “Tiling integral floating display system with optimized viewing window,” Appl. Opt. 51(22), 5453–5457 (2012). A. J. Woods and A. Sehic, “The compatibility of LCD TVs with time-sequential stereoscopic 3D visualization,” Proc. SPIE 7237, 72370N (2009).
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17620
18. S. S. Kim, B. H. You, H. Choi, B. H. Berkeley, and N. D. Kim, “World's first 240 Hz TFT-LCD technology for full-HD LCD-TV and its application to 3D display,” in SID International Symposium Digest of Technical Papers (Society for Information Display, 2009), Vol. 40, pp. 424–427. 19. D.-S. Kim, S.-M. Park, J.-H. Jung, and D.-C. Hwang, “New 240Hz driving method for Full HD and high quality 3D LCD TV,” in SID International Symposium Digest of Technical Papers (Society for Information Display, 2010), Vol. 41, pp. 762–765. 20. S.-M. Jung, Y.-B. Lee, H.-J. Park, S.-C. Lee, W.-N. Jeong, J.-K. Shin, and I.-J. Chung, “Improvement of 3-D crosstalk with over-driving method for the active retarder 3-D displays,” in SID International Symposium Digest of Technical Papers (Society for Information Display, 2010), Vol. 41, pp. 1264–1267. 21. F.-H. Chen, K.-C. Huang, L.-C. Lin, Y.-H. Chou, and K. Lee, “System crosstalk measurement of a timesequential 3D display using ideal shutter glasses,” Proc. SPIE 7863, 78632E (2011). 22. M. Park, J. Kim, and H.-J. Choi, “Effect of interlacing methods of stereoscopic displays on perceived image quality,” Appl. Opt. 53(3), 520–527 (2014). 23. A. J. Woods and T. Rourke, “The compatibility of consumer DLP projectors with time-sequential stereoscopic 3D visualization,” Proc. SPIE 6490, 64900V (2007). 24. H. J. Lee, H. Nam, J. D. Lee, H. W. Jang, M. S. Song, B. S. Kim, J. S. Gu, C. Y. Park, and K. H. Choi, “A high resolution autostereoscopic display employing a time division parallax barrier,” in SID International Symposium Digest of Technical Papers (Society for Information Display, 2006), Vol. 37, pp. 81–84. 25. S. Ishizuka, T. Mukai, and H. Kakeya, “Viewing zone of an autostereoscopic display with a directional backlight using a convex lens array,” J. Electron. Imaging 23(1), 011002 (2014). 26. M. Okui, M. Kobayashi, J. Arai, and F. Okano, “Moire fringe reduction by optical filters in integral threedimensional imaging on a color flat-panel display,” Appl. Opt. 44(21), 4475–4483 (2005).
1. Introduction Recently, the stereoscopic three-dimensional (3D) display to use special glasses becomes popular with rapid progresses of two-dimensional (2D) flat panel display (FPD) devices and increasing numbers of 3D contents. However, the customers are still considering the special glasses inconvenient and they are not likely to wear them. Therefore, the autostereoscopic 3D display techniques which can make the observers watch the 3D image without special glasses attracts much attention nowadays. Among those technologies, the integral imaging (InIm), which is proposed in 1908, is regarded one of the promising methods to provide a full parallax volumetric autostereoscopic 3D images to the observer [1–7]. Despite the above advantages and customer’s need, current autostereoscopic 3D display technologies have common bottlenecks such as limited viewpositions and inferior image quality. Considering that the existing stereoscopic 3D display are providing 3D images whose quality is comparable with that of the conventional 2D FPDs, there will be no opposition that the poor quality of current autostereosopic 3D display should be improved [8–13]. Although the image quality is basically a subjective one and hard to be measured, there are some parameters to present the level of it – resolution, brightness, color gamut, and contrast ratio (C/R). Among them, the resolution is the most important key factor to affect the perceived image quality in near distance, but it is also the weakest characteristics in autostereoscopic 3D display including the InIm in spite of efforts to enhance it [14–16]. The most common resolution in 2D and stereoscopic 3D display is a Full-HD (1920 x 1080), while even the most advanced autostereoscopic 3D display can provide a HD (1280 x 800) resolution with horizontal parallax only. Therefore, it is essential to develop a resolutionenhanced autostereoscopic technology to open a new era of 3D display. It is well known that the basic way to improve the resolution is to increase the amount of information to be shown. A commonly used method is to adopt a display device with higher pixel density. However, even though we use a display panel with the highest pixel density such as 4K (about 4000 x 2000) resolution, we can provide only four viewpoints to display a Full-HD 3D images. As a result, it is needed to find another approach. The time-multiplexing technique to use those higher refresh rate can be a good candidate to enhance the resolution of the 3D display. Although not developed to enhance the 3D resolution, there are several FPD devices to provide higher refresh rate such as 120Hz or 240Hz liquid crystal display (LCD) TVs, and what we need is to develop a way to convert that higher temporal resolution (refresh rate) into spatial one (3D resolution). For the
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17621
stereoscopic 3D display using glasses, there have been various attempts to adopt the timesequential technique in realizing a stereoscopic 3D display and they have proved that the time-sequential technique is an effective way to improve the resolution of the 3D images [17– 22]. The digital mirror device (DMD) is also a good candidate for the projection type 3D display with its fast operational speed [23]. In addition, there were also researches implementing a directional backlight or a time-sequential parallax barrier in order to improve the resolution of the multi-view autostereoscopic 3D display system [24, 25]. In this paper, another novel method to adopt a time-multiplexed combination of electrical masks and displayed images is proposed. The proposed scheme can be especially used for InIm 3D display in focal mode and can increase the resolution by four times. 2. Principles The basic principle of the conventional InIm in focal mode is shown in Fig. 1. The basic (conventional) integral imaging system consists of a display device and a lens array. There is no restriction in type of the display device – it can be an LCD, an organic light emitting diode (OLED) display, and even a cathode ray tube (CRT) display. The role of the display device is to show an image array composed of many small images called elemental images with different perspectives. Then, the lens array in front of the display device integrates those elemental images into a single 3D image to provide different parallaxes within the viewing angle. In the focal mode, the distance between the lens array and the display panel is set to be the focal length f of the lens array. Then, only a single point (pixel) can be shown through a single elemental lens and therefore that elemental lens becomes a basic unit of the 3D image. In other words, the volume pixel (voxel) to consist the 3D image is formed through several elemental lenses and the resolution (number of voxel) of the 3D image depends on the number of elemental lenses in a lens array. If the size of the display device is fixed, the size of a single elemental lens determines the number of the elemental lenses – the 3D resolution of InIm in focal mode. Display Device
Elemental Images
Lens Array
3D Voxel
Fig. 1. Basic structure and principles of conventional InIm system in focal mode.
As a result, a fundamental way to increase the 3D resolution of InIm in focal mode is to increase the number of elemental lenses by reducing the size of each elemental lens. However, we should also consider the relationship between the size p of the elemental lens and the viewing angle shown in Fig. 2. As described in Fig. 2, there also exists a trade-off
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17622
relationship between the viewing angle and the 3D resolution and it is needed to overcome it in order to improve the overall image quality.
Elemental Image
Lenslet
Fig. 2. Viewing angle of conventional InIm system in focal mode.
In this paper, an electrical mask array synchronized with the displayed elemental image sets is adopted to improve that trade-off relation and to enhance the 3D resolution without reducing the viewing angle. The structure of the proposed scheme is shown in Fig. 3. The structural difference between the conventional InIm in Fig. 2 and the proposed method in Fig. 3 is that an electrical mask is attached to each elemental lens. The role of the electrical mask is to block the aperture of each elemental lens partially and therefore reduces the size of the elemental lens (size of the voxel), while the size of the elemental image area remains the same p as in Fig. 2. As a result, the integrated 3D image will consist of smaller voxels. Elemental Image 1
Lenslet +Mask 1
Elemental Image 2
Lenslet +Mask 2
1st Sequence
2nd Sequence
Fig. 3. The structure and the operational principles of the proposed InIm system
However, since the number of voxel is fixed while the size of it is reduced, there will be a gap between those smaller voxels and the above condition is not sufficient to increase the 3D resolution. Therefore, a time-multiplexing technique to increase the number the displayed voxel is also needed and it is the reason for the mask to be moved electrically for each sequence. In Fig. 3, there is an example of the proposed scheme to increase the vertical 3D resolution by two times. In order to achieve that goal, the proposed method is composed of two different sequences with an electrical mask of half aperture. Since the electrical mask blocks half of each elemental lens’ aperture vertically, only half of the 3D image can be seen through it and two sequences are needed to display the entire image as shown in Fig. 3. With the time-multiplexing operation through those sequences, the entire 3D image can be displayed with double number of voxels in vertical direction while the vertical viewing angle (θP1 and θP2) of each sequence are preserved and the same as θC in Fig. 2. Although the examples in Fig. 3 shows the 3D resolution enhancement in vertical direction only, it is also
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17623
possible to increase the horizontal 3D resolution using the same principles. In that case, instead, the system needs to be operated with four sequences. In the proposed method described above, the full 3D image can be realized through the overlapping of the displayed images of each sequence and it is needed to switch them with faster speed than the conventional one. Considering that the minimum switching speed to induce the afterimage effect is 30 frames/second, it is needed to realize the proposed principle shown in Fig. 3 with at least a switching speed of 60 frames/second. If we want to increase the horizontal 3D resolution as well, a switching speed of 120 frames/second is needed. Another point to be considered is the forming of the viewing zone where the entire 3D image can be observed. Although the viewing angle of each sequence are the same as that of the conventional method, the viewing zone of the proposed method should be inside the overlapped area of each sequence as shown in Fig. 4. As a result, the viewing zone of the proposed method is located a little bit farther (f/2) from the lens array than the conventional one, but the difference is typically small and regarded to be negligible considering that the focal length f of the lens array InIm is usually short in focal mode. For example, in the preliminary experiments in next chapter, a lens array with focal length f of 8.03mm was used. Therefore, the difference between the viewing zones of the conventional InIm and the proposed method is about 4mm - which could be thought to be a negligible one considering that the viewing distance is over several hundreds of millimeters. Conventional InIm Elemental Image
Proposed Method Elemental Image
Lenslet
θC
Viewing Zone
Lenslet +Masks θP
Viewing Zone
Fig. 4. Comparison of location of the viewing zone between the conventional InIm and the proposed method.
3. Experimental setup and results In order to prove the above time-multiplexing principle, we have built a preliminary experimental setup. Since the most effective way to generate electrical mask is to display a black mask pattern on an LCD panel, we have used two 24-inch LCD panel with a resolution of full-HD (1920 x 1080) and a refresh rate of 120Hz as a display panel and a mask panel. Since those two LCD panels have same specification, it is possible to match the position and the size and the location of the elemental images with those of the mask patterns. Considering that the minimum switching speed to induce the afterimage effect is about 30Hz, the proposed time-multiplexing InIm system which operates with a refresh rate of 120Hz could provide four times higher resolution than the conventional one. Between the two LCD panels, a 24inch lens array composed of 330 x 185 elemental lenses with focal length f of 8.03mm was inserted. The specification of the experimental setup is summarized in Table 1. Table 1. Experimental Parameters of the Proposed Setup Diagonal Size
LCD Panel
24inch
Resolution
1920 x 1080 (Full-HD)
Pixel Pitch Refresh Rate
0.27mm x 0.27mm 120Hz
#208212 - $15.00 USD (C) 2014 OSA
Lens Array Diagonal Size 24inch Number of Elemental 330 x 185 Lenses Elemental Lens Pitch 1.6mm x 1.6mm Focal Length 8.03mm
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17624
Display Panel
Lens Detached Array Mask
Upper Viewpoint
gap Display Panel
Lens Detached Array Mask
Center Viewpoint
gap Display Panel
Lens Detached Array Mask
Lower Viewpoint
gap
Fig. 5. The malfunction of detached mask with a gap between it and the lens array.
Refer to Table 1, it is expected that each lenslet has about 6 x 6 pixels behind and can provide 34~36 different views. Although the number of pixels per each lenslet is the same both in conventional and proposed methods, the 3D resolution can be enhanced through the proposed time-sequential scheme. However, we expect that there still exists a limitation to increase the number of views due to the luminance decreasing by the electrical mask even if we can use infinitely fast LCD panel. Also, since the number of pixel per each lenslet was not an exact integer, there could be a mismatch between each lenslet and its corresponding elemental image. In the experimental setup, we have adjusted the size of each elemental image to minimize the overall mismatch between the lens array and the elemental image array. In addition, since we have used same LCD panels for both display and mask panels, there could be a similar problem of size mismatch between the mask array and the lens array. We expect that the size mismatch problem can be resolved if the lens array and the LCD panels are fabricated together at the first level of manufacture. In implementing the experimental setup, it is also very important to reduce the gap between the lens array and the mask panel as shown in the ideal cases of Fig. 3. If there exists a certain gap between them and the mask to be detached from the lens array, the detached
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17625
mask cannot cover the designated area (part) of the each lenslet as shown in Fig. 5 and it can make the 3D crosstalk worse. However, in real cases, the mask can be partially detached due to the bending of the lens array itself. In order to prevent the above problem, we have used a zig (holder) to hold the lens array and a glass panel and while the mask panel has been placed between them as shown in the top view of Fig. 6. As a result, by keeping a pressure to the lens array and the glass panel using the zig in order to eliminate the unexpected side effects due to bending of the lens array, the gap between the mask panel and the lens array could be minimized (less than 1mm) and also be uniform for the entire lens array. In addition, the Moiré between the display panel and the mask panel could be a problem that should be resolved to realize the proposed scheme. In the experimental setup, by referring the prior art, we have inserted a diffuser film attached at a transparent acrylic plate between the lens array and could remove the Moiré [26]. The pictures of the experimental setup which consists of the above devices are shown in Fig. 6. Since the lens array is located between the display panel and mask panel, it is not shown well in Fig. 6. Instead, the front view of the lens array which was used in the experimental setup is shown in Fig. 7. Side View
Front View
Mask Panel + Lens Array
Display Panel
Top View (Magnified)
Display Panel Diffuser Film + Acrylic Panel Zig to hold Lens Array
Lens Array Mask Panel Glass Panel
Zig to hold Glass Panel
Fig. 6. The experimental setup composed of a lens array and two 120Hz LCD panels for display & mask panel.
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17626
528mm (330ea)
24-inch
Fig. 7. The picture of a 24inch lens array used in the experiment .
Elemental Image (Full)
Elemental Image (Magnified)
Mask Pattern (Magnified)
1st Sequence
2nd Sequence
3rd Sequence
4th Sequence
Fig. 8. Arrays of elemental images and its corresponding mask patterns for each sequence.
The basic 3D resolution of the experimental setup is 330 x 180 to be the same as the number of elemental lenses. On the other hand, if the proposed scheme is adopted, the 3D resolution can be enhanced to be 660 x 360 (four times higher than the basic one). In order to achieve that goal, the electric mask patterns and elemental images for the four sequences are generated and prepared as shown in Fig. 8. The elemental images are generated by the computer generated integral imaging (CGII) method to form three 3D images of ‘SJU’, ‘DSU’ and ‘IAI’ at the location of −80.3mm, 0mm, + 80.3mm from the lens array.
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17627
The experimental results realized using the proposed scheme are shown at Fig. 9 and Fig. 10. In Fig. 9, the 3D images of ‘SJU’, ‘DSU’, and ‘IAI’, which were captured from various view positions, are showing proper perspectives corresponding to the relative positions of each own. The movie in Fig. 10 also shows a smooth motion parallax between those 3D images according to the change of the view positions. Up
Left
Center
Right
Down
Fig. 9. The experimental 3D images of ‘SJU’, ‘DSU’, and ‘IAI’ captured at various view positions.
Fig. 10. The captured movie of the experimental 3D images (Media 1)
In order to prove the resolution enhancing effect of the proposed scheme, we have compared the experimental result with the conventional one. The result of comparison is shown at Fig. 11. At first, the full image is shown at Fig. 11(a)-11(c) also show a magnified image of ‘Area 1’ and ‘Area 2’ to provide detailed information. The results in Fig. 11 clearly show that the 3D images realized by the proposed method consist of more number of voxels with smaller size than the conventional ones although there exist some artifacts due to the size mismatch problem between the pixels of mask panel and each lenslet which has been described above. Therefore, it can be thought that the proposed scheme successfully increases the resolution of the 3D image and is also useful to realize high-performance autostereoscopic 3D display system with enhanced 3D image quality.
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17628
Conventional
Proposed
Area 1 Area 2
Area 1 Area 2
(a) Conventional (Area 1)
Proposed (Area 1)
(b) Conventional (Area 2)
Proposed (Area 2)
(c)
Fig. 11. Comparison of the results of the proposed method and the conventional one: (a) full image at center, (b) magnified image for Area 1, and (c) magnified image for Area 2
4. Conclusion We have proposed a resolution-enhanced integral imaging in focal mode with a timemultiplexed electrical mask array and realized a preliminary setup which could display 3D images with four times higher resolution. Regarding to the current level of FPD technologies to provide a refresh rate over 240Hz, the proposed time-multiplexing scheme is expected to be useful to improve the quality of the 3D images. Acknowledgment This work was supported in part by the IT R&D program of MKE/KEIT. [10041682, Development of high-definition 3D image processing technologies using advanced integral imaging with improved depth range] and this work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (Ministry of Science, ICT and Future Planning) (No. 2013-067321).
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17629
Department of Information and Communication Engineering, Pukyung National University, 45 Yongso-ro, Nam-Gu, Busan, 608-737, South Korea 2 Institute of Ambient Intelligence, Dongseo University,47 Jurye-ro, Sasang-Gu, Busan, 617-716, South Korea 3 Department of Physics, Sejong University, 209 Neungdong-ro, Gwangjin-Gu, Seoul, 143-747, South Korea * [email protected]
Abstract: The inferior resolution of the three-dimensional (3D) image is one of the main problems to be resolved for realizing a commercial autosteresosopic 3D display device. In this paper, a time-multiplexing technique using electrically moving masks is proposed to enhance the resolution of the 3D image realized by integral imaging in a focal mode while preserving the viewing angle of it. ©2014 Optical Society of America OCIS codes: (010.1080) Active or adaptive optics; (110.2990) Image formation theory; (100.6890) Three-dimensional image processing.
References and links 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17.
G. Lippmann, “La photographie integrale,” C. R. Acad. Sci. 146, 446–451 (1908). F. Okano, H. Hoshino, J. Arai, and I. Yuyama, “Real-time pickup method for a three-dimensional image based on integral photography,” Appl. Opt. 36(7), 1598–1603 (1997). T. Okoshi, “Optimum design and depth resolution of lens-sheet and projection-type three-dimensional displays,” Appl. Opt. 10(10), 2284–2291 (1971). J.-H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt. 48(34), H77–H94 (2009). M. Cho and B. Javidi, “Optimization of 3D integral imaging system parameters,” J. Displ. Technol. 8(6), 357– 360 (2012). M. Martínez-Corral, H. Navarro, R. Martínez-Cuenca, G. Saavedra, and B. Javidi, “Full parallax 3-D TV with programmable display parameters,” Opt. Photon. News 22(12), 50 (2011). J.-Y. Jang, D. Shin, and E.-S. Kim, “Optical three-dimensional refocusing from elemental images based on a sifting property of the periodic δ-function array in integral-imaging,” Opt. Express 22(2), 1533–1550 (2014). C.-W. Chen, M. Cho, Y.-P. Huang, and B. Javidi, “Improved viewing zones for projection type integral imaging 3D display using adaptive liquid crystal prism array,” J. Displ. Technol. 10(3), 198–203 (2014). J.-S. Jang, F. Jin, and B. Javidi, “Three-dimensional integral imaging with large depth of focus by use of real and virtual image fields,” Opt. Lett. 28(16), 1421–1423 (2003). J.-S. Jang and B. Javidi, “Large depth-of-focus time-multiplexed three-dimensional integral imaging by use of lenslets with nonuniform focal lengths and aperture sizes,” Opt. Lett. 28(20), 1924–1926 (2003). H. Choi, Y. Kim, J.-H. Park, J. Kim, S.-W. Cho, and B. Lee, “Layered-panel integral imaging without the translucent problem,” Opt. Express 13(15), 5769–5776 (2005). J. Hong, J.-H. Park, S. Jung, and B. Lee, “Depth-enhanced integral imaging by use of optical path control,” Opt. Lett. 29(15), 1790–1792 (2004). D.-H. Shin, B. Lee, and E.-S. Kim, “Multidirectional curved integral imaging with large depth by additional use of a large-aperture lens,” Appl. Opt. 45(28), 7375–7381 (2006). J.-S. Jang and B. Javidi, “Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics,” Opt. Lett. 27(5), 324–326 (2002). Y. Kim, J. Kim, J.-M. Kang, J.-H. Jung, H. Choi, and B. Lee, “Point light source integral imaging with improved resolution and viewing angle by the use of electrically movable pinhole array,” Opt. Express 15(26), 18253– 18267 (2007). H.-J. Choi, Y. M. Kim, J. Jung, K.-M. Jung, and S.-W. Min, “Tiling integral floating display system with optimized viewing window,” Appl. Opt. 51(22), 5453–5457 (2012). A. J. Woods and A. Sehic, “The compatibility of LCD TVs with time-sequential stereoscopic 3D visualization,” Proc. SPIE 7237, 72370N (2009).
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17620
18. S. S. Kim, B. H. You, H. Choi, B. H. Berkeley, and N. D. Kim, “World's first 240 Hz TFT-LCD technology for full-HD LCD-TV and its application to 3D display,” in SID International Symposium Digest of Technical Papers (Society for Information Display, 2009), Vol. 40, pp. 424–427. 19. D.-S. Kim, S.-M. Park, J.-H. Jung, and D.-C. Hwang, “New 240Hz driving method for Full HD and high quality 3D LCD TV,” in SID International Symposium Digest of Technical Papers (Society for Information Display, 2010), Vol. 41, pp. 762–765. 20. S.-M. Jung, Y.-B. Lee, H.-J. Park, S.-C. Lee, W.-N. Jeong, J.-K. Shin, and I.-J. Chung, “Improvement of 3-D crosstalk with over-driving method for the active retarder 3-D displays,” in SID International Symposium Digest of Technical Papers (Society for Information Display, 2010), Vol. 41, pp. 1264–1267. 21. F.-H. Chen, K.-C. Huang, L.-C. Lin, Y.-H. Chou, and K. Lee, “System crosstalk measurement of a timesequential 3D display using ideal shutter glasses,” Proc. SPIE 7863, 78632E (2011). 22. M. Park, J. Kim, and H.-J. Choi, “Effect of interlacing methods of stereoscopic displays on perceived image quality,” Appl. Opt. 53(3), 520–527 (2014). 23. A. J. Woods and T. Rourke, “The compatibility of consumer DLP projectors with time-sequential stereoscopic 3D visualization,” Proc. SPIE 6490, 64900V (2007). 24. H. J. Lee, H. Nam, J. D. Lee, H. W. Jang, M. S. Song, B. S. Kim, J. S. Gu, C. Y. Park, and K. H. Choi, “A high resolution autostereoscopic display employing a time division parallax barrier,” in SID International Symposium Digest of Technical Papers (Society for Information Display, 2006), Vol. 37, pp. 81–84. 25. S. Ishizuka, T. Mukai, and H. Kakeya, “Viewing zone of an autostereoscopic display with a directional backlight using a convex lens array,” J. Electron. Imaging 23(1), 011002 (2014). 26. M. Okui, M. Kobayashi, J. Arai, and F. Okano, “Moire fringe reduction by optical filters in integral threedimensional imaging on a color flat-panel display,” Appl. Opt. 44(21), 4475–4483 (2005).
1. Introduction Recently, the stereoscopic three-dimensional (3D) display to use special glasses becomes popular with rapid progresses of two-dimensional (2D) flat panel display (FPD) devices and increasing numbers of 3D contents. However, the customers are still considering the special glasses inconvenient and they are not likely to wear them. Therefore, the autostereoscopic 3D display techniques which can make the observers watch the 3D image without special glasses attracts much attention nowadays. Among those technologies, the integral imaging (InIm), which is proposed in 1908, is regarded one of the promising methods to provide a full parallax volumetric autostereoscopic 3D images to the observer [1–7]. Despite the above advantages and customer’s need, current autostereoscopic 3D display technologies have common bottlenecks such as limited viewpositions and inferior image quality. Considering that the existing stereoscopic 3D display are providing 3D images whose quality is comparable with that of the conventional 2D FPDs, there will be no opposition that the poor quality of current autostereosopic 3D display should be improved [8–13]. Although the image quality is basically a subjective one and hard to be measured, there are some parameters to present the level of it – resolution, brightness, color gamut, and contrast ratio (C/R). Among them, the resolution is the most important key factor to affect the perceived image quality in near distance, but it is also the weakest characteristics in autostereoscopic 3D display including the InIm in spite of efforts to enhance it [14–16]. The most common resolution in 2D and stereoscopic 3D display is a Full-HD (1920 x 1080), while even the most advanced autostereoscopic 3D display can provide a HD (1280 x 800) resolution with horizontal parallax only. Therefore, it is essential to develop a resolutionenhanced autostereoscopic technology to open a new era of 3D display. It is well known that the basic way to improve the resolution is to increase the amount of information to be shown. A commonly used method is to adopt a display device with higher pixel density. However, even though we use a display panel with the highest pixel density such as 4K (about 4000 x 2000) resolution, we can provide only four viewpoints to display a Full-HD 3D images. As a result, it is needed to find another approach. The time-multiplexing technique to use those higher refresh rate can be a good candidate to enhance the resolution of the 3D display. Although not developed to enhance the 3D resolution, there are several FPD devices to provide higher refresh rate such as 120Hz or 240Hz liquid crystal display (LCD) TVs, and what we need is to develop a way to convert that higher temporal resolution (refresh rate) into spatial one (3D resolution). For the
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17621
stereoscopic 3D display using glasses, there have been various attempts to adopt the timesequential technique in realizing a stereoscopic 3D display and they have proved that the time-sequential technique is an effective way to improve the resolution of the 3D images [17– 22]. The digital mirror device (DMD) is also a good candidate for the projection type 3D display with its fast operational speed [23]. In addition, there were also researches implementing a directional backlight or a time-sequential parallax barrier in order to improve the resolution of the multi-view autostereoscopic 3D display system [24, 25]. In this paper, another novel method to adopt a time-multiplexed combination of electrical masks and displayed images is proposed. The proposed scheme can be especially used for InIm 3D display in focal mode and can increase the resolution by four times. 2. Principles The basic principle of the conventional InIm in focal mode is shown in Fig. 1. The basic (conventional) integral imaging system consists of a display device and a lens array. There is no restriction in type of the display device – it can be an LCD, an organic light emitting diode (OLED) display, and even a cathode ray tube (CRT) display. The role of the display device is to show an image array composed of many small images called elemental images with different perspectives. Then, the lens array in front of the display device integrates those elemental images into a single 3D image to provide different parallaxes within the viewing angle. In the focal mode, the distance between the lens array and the display panel is set to be the focal length f of the lens array. Then, only a single point (pixel) can be shown through a single elemental lens and therefore that elemental lens becomes a basic unit of the 3D image. In other words, the volume pixel (voxel) to consist the 3D image is formed through several elemental lenses and the resolution (number of voxel) of the 3D image depends on the number of elemental lenses in a lens array. If the size of the display device is fixed, the size of a single elemental lens determines the number of the elemental lenses – the 3D resolution of InIm in focal mode. Display Device
Elemental Images
Lens Array
3D Voxel
Fig. 1. Basic structure and principles of conventional InIm system in focal mode.
As a result, a fundamental way to increase the 3D resolution of InIm in focal mode is to increase the number of elemental lenses by reducing the size of each elemental lens. However, we should also consider the relationship between the size p of the elemental lens and the viewing angle shown in Fig. 2. As described in Fig. 2, there also exists a trade-off
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17622
relationship between the viewing angle and the 3D resolution and it is needed to overcome it in order to improve the overall image quality.
Elemental Image
Lenslet
Fig. 2. Viewing angle of conventional InIm system in focal mode.
In this paper, an electrical mask array synchronized with the displayed elemental image sets is adopted to improve that trade-off relation and to enhance the 3D resolution without reducing the viewing angle. The structure of the proposed scheme is shown in Fig. 3. The structural difference between the conventional InIm in Fig. 2 and the proposed method in Fig. 3 is that an electrical mask is attached to each elemental lens. The role of the electrical mask is to block the aperture of each elemental lens partially and therefore reduces the size of the elemental lens (size of the voxel), while the size of the elemental image area remains the same p as in Fig. 2. As a result, the integrated 3D image will consist of smaller voxels. Elemental Image 1
Lenslet +Mask 1
Elemental Image 2
Lenslet +Mask 2
1st Sequence
2nd Sequence
Fig. 3. The structure and the operational principles of the proposed InIm system
However, since the number of voxel is fixed while the size of it is reduced, there will be a gap between those smaller voxels and the above condition is not sufficient to increase the 3D resolution. Therefore, a time-multiplexing technique to increase the number the displayed voxel is also needed and it is the reason for the mask to be moved electrically for each sequence. In Fig. 3, there is an example of the proposed scheme to increase the vertical 3D resolution by two times. In order to achieve that goal, the proposed method is composed of two different sequences with an electrical mask of half aperture. Since the electrical mask blocks half of each elemental lens’ aperture vertically, only half of the 3D image can be seen through it and two sequences are needed to display the entire image as shown in Fig. 3. With the time-multiplexing operation through those sequences, the entire 3D image can be displayed with double number of voxels in vertical direction while the vertical viewing angle (θP1 and θP2) of each sequence are preserved and the same as θC in Fig. 2. Although the examples in Fig. 3 shows the 3D resolution enhancement in vertical direction only, it is also
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17623
possible to increase the horizontal 3D resolution using the same principles. In that case, instead, the system needs to be operated with four sequences. In the proposed method described above, the full 3D image can be realized through the overlapping of the displayed images of each sequence and it is needed to switch them with faster speed than the conventional one. Considering that the minimum switching speed to induce the afterimage effect is 30 frames/second, it is needed to realize the proposed principle shown in Fig. 3 with at least a switching speed of 60 frames/second. If we want to increase the horizontal 3D resolution as well, a switching speed of 120 frames/second is needed. Another point to be considered is the forming of the viewing zone where the entire 3D image can be observed. Although the viewing angle of each sequence are the same as that of the conventional method, the viewing zone of the proposed method should be inside the overlapped area of each sequence as shown in Fig. 4. As a result, the viewing zone of the proposed method is located a little bit farther (f/2) from the lens array than the conventional one, but the difference is typically small and regarded to be negligible considering that the focal length f of the lens array InIm is usually short in focal mode. For example, in the preliminary experiments in next chapter, a lens array with focal length f of 8.03mm was used. Therefore, the difference between the viewing zones of the conventional InIm and the proposed method is about 4mm - which could be thought to be a negligible one considering that the viewing distance is over several hundreds of millimeters. Conventional InIm Elemental Image
Proposed Method Elemental Image
Lenslet
θC
Viewing Zone
Lenslet +Masks θP
Viewing Zone
Fig. 4. Comparison of location of the viewing zone between the conventional InIm and the proposed method.
3. Experimental setup and results In order to prove the above time-multiplexing principle, we have built a preliminary experimental setup. Since the most effective way to generate electrical mask is to display a black mask pattern on an LCD panel, we have used two 24-inch LCD panel with a resolution of full-HD (1920 x 1080) and a refresh rate of 120Hz as a display panel and a mask panel. Since those two LCD panels have same specification, it is possible to match the position and the size and the location of the elemental images with those of the mask patterns. Considering that the minimum switching speed to induce the afterimage effect is about 30Hz, the proposed time-multiplexing InIm system which operates with a refresh rate of 120Hz could provide four times higher resolution than the conventional one. Between the two LCD panels, a 24inch lens array composed of 330 x 185 elemental lenses with focal length f of 8.03mm was inserted. The specification of the experimental setup is summarized in Table 1. Table 1. Experimental Parameters of the Proposed Setup Diagonal Size
LCD Panel
24inch
Resolution
1920 x 1080 (Full-HD)
Pixel Pitch Refresh Rate
0.27mm x 0.27mm 120Hz
#208212 - $15.00 USD (C) 2014 OSA
Lens Array Diagonal Size 24inch Number of Elemental 330 x 185 Lenses Elemental Lens Pitch 1.6mm x 1.6mm Focal Length 8.03mm
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17624
Display Panel
Lens Detached Array Mask
Upper Viewpoint
gap Display Panel
Lens Detached Array Mask
Center Viewpoint
gap Display Panel
Lens Detached Array Mask
Lower Viewpoint
gap
Fig. 5. The malfunction of detached mask with a gap between it and the lens array.
Refer to Table 1, it is expected that each lenslet has about 6 x 6 pixels behind and can provide 34~36 different views. Although the number of pixels per each lenslet is the same both in conventional and proposed methods, the 3D resolution can be enhanced through the proposed time-sequential scheme. However, we expect that there still exists a limitation to increase the number of views due to the luminance decreasing by the electrical mask even if we can use infinitely fast LCD panel. Also, since the number of pixel per each lenslet was not an exact integer, there could be a mismatch between each lenslet and its corresponding elemental image. In the experimental setup, we have adjusted the size of each elemental image to minimize the overall mismatch between the lens array and the elemental image array. In addition, since we have used same LCD panels for both display and mask panels, there could be a similar problem of size mismatch between the mask array and the lens array. We expect that the size mismatch problem can be resolved if the lens array and the LCD panels are fabricated together at the first level of manufacture. In implementing the experimental setup, it is also very important to reduce the gap between the lens array and the mask panel as shown in the ideal cases of Fig. 3. If there exists a certain gap between them and the mask to be detached from the lens array, the detached
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17625
mask cannot cover the designated area (part) of the each lenslet as shown in Fig. 5 and it can make the 3D crosstalk worse. However, in real cases, the mask can be partially detached due to the bending of the lens array itself. In order to prevent the above problem, we have used a zig (holder) to hold the lens array and a glass panel and while the mask panel has been placed between them as shown in the top view of Fig. 6. As a result, by keeping a pressure to the lens array and the glass panel using the zig in order to eliminate the unexpected side effects due to bending of the lens array, the gap between the mask panel and the lens array could be minimized (less than 1mm) and also be uniform for the entire lens array. In addition, the Moiré between the display panel and the mask panel could be a problem that should be resolved to realize the proposed scheme. In the experimental setup, by referring the prior art, we have inserted a diffuser film attached at a transparent acrylic plate between the lens array and could remove the Moiré [26]. The pictures of the experimental setup which consists of the above devices are shown in Fig. 6. Since the lens array is located between the display panel and mask panel, it is not shown well in Fig. 6. Instead, the front view of the lens array which was used in the experimental setup is shown in Fig. 7. Side View
Front View
Mask Panel + Lens Array
Display Panel
Top View (Magnified)
Display Panel Diffuser Film + Acrylic Panel Zig to hold Lens Array
Lens Array Mask Panel Glass Panel
Zig to hold Glass Panel
Fig. 6. The experimental setup composed of a lens array and two 120Hz LCD panels for display & mask panel.
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17626
528mm (330ea)
24-inch
Fig. 7. The picture of a 24inch lens array used in the experiment .
Elemental Image (Full)
Elemental Image (Magnified)
Mask Pattern (Magnified)
1st Sequence
2nd Sequence
3rd Sequence
4th Sequence
Fig. 8. Arrays of elemental images and its corresponding mask patterns for each sequence.
The basic 3D resolution of the experimental setup is 330 x 180 to be the same as the number of elemental lenses. On the other hand, if the proposed scheme is adopted, the 3D resolution can be enhanced to be 660 x 360 (four times higher than the basic one). In order to achieve that goal, the electric mask patterns and elemental images for the four sequences are generated and prepared as shown in Fig. 8. The elemental images are generated by the computer generated integral imaging (CGII) method to form three 3D images of ‘SJU’, ‘DSU’ and ‘IAI’ at the location of −80.3mm, 0mm, + 80.3mm from the lens array.
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17627
The experimental results realized using the proposed scheme are shown at Fig. 9 and Fig. 10. In Fig. 9, the 3D images of ‘SJU’, ‘DSU’, and ‘IAI’, which were captured from various view positions, are showing proper perspectives corresponding to the relative positions of each own. The movie in Fig. 10 also shows a smooth motion parallax between those 3D images according to the change of the view positions. Up
Left
Center
Right
Down
Fig. 9. The experimental 3D images of ‘SJU’, ‘DSU’, and ‘IAI’ captured at various view positions.
Fig. 10. The captured movie of the experimental 3D images (Media 1)
In order to prove the resolution enhancing effect of the proposed scheme, we have compared the experimental result with the conventional one. The result of comparison is shown at Fig. 11. At first, the full image is shown at Fig. 11(a)-11(c) also show a magnified image of ‘Area 1’ and ‘Area 2’ to provide detailed information. The results in Fig. 11 clearly show that the 3D images realized by the proposed method consist of more number of voxels with smaller size than the conventional ones although there exist some artifacts due to the size mismatch problem between the pixels of mask panel and each lenslet which has been described above. Therefore, it can be thought that the proposed scheme successfully increases the resolution of the 3D image and is also useful to realize high-performance autostereoscopic 3D display system with enhanced 3D image quality.
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17628
Conventional
Proposed
Area 1 Area 2
Area 1 Area 2
(a) Conventional (Area 1)
Proposed (Area 1)
(b) Conventional (Area 2)
Proposed (Area 2)
(c)
Fig. 11. Comparison of the results of the proposed method and the conventional one: (a) full image at center, (b) magnified image for Area 1, and (c) magnified image for Area 2
4. Conclusion We have proposed a resolution-enhanced integral imaging in focal mode with a timemultiplexed electrical mask array and realized a preliminary setup which could display 3D images with four times higher resolution. Regarding to the current level of FPD technologies to provide a refresh rate over 240Hz, the proposed time-multiplexing scheme is expected to be useful to improve the quality of the 3D images. Acknowledgment This work was supported in part by the IT R&D program of MKE/KEIT. [10041682, Development of high-definition 3D image processing technologies using advanced integral imaging with improved depth range] and this work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (Ministry of Science, ICT and Future Planning) (No. 2013-067321).
#208212 - $15.00 USD (C) 2014 OSA
Received 14 Mar 2014; revised 21 Jun 2014; accepted 23 Jun 2014; published 14 Jul 2014 28 July 2014 | Vol. 22, No. 15 | DOI:10.1364/OE.22.017620 | OPTICS EXPRESS 17629
