Graphically Speaking
Editor: André Stork
LumiConSense A Transparent, Flexible, and Scalable Thin-Film Sensor Alexander Koppelhuber and Oliver Bimber Johannes Kepler University
I
magine that car windshields can observe passengers and that shopping windows can observe customers. Glasses can track gaze direction, while floors can track crowds. 2D screens no longer have to be touched, and 3D screens can be properly interacted with. Plastic or glass containers can be aware of their content, and our clothing can be aware of our environment. What sounds futuristic could soon become part of our daily life. Both imaging and display technologies are evolving faster than ever. For instance, bendable, transparent screens based on organic light-emitting polymers are close to commercialization, and flexible image sensors are in their infancy. In addition, how we represent scenes with that technology is also evolving. So far, scene representation has relied on capturing digital ras-
ter images with planar sensors, computing them, and presenting them on planar displays. However, simple extensions toward 3D representation with 3D cameras and 3D screens are becoming increasingly successful. More advanced representations, such as 4D light fields or holograms, might hold even more potential. These advances will radically change everything related to computer graphics, visualization, computer vision, and image processing. The mathematical models and underlying algorithms and data structures based on processing 2D raster images are far from being carved in stone. They’ll evolve along with imaging and display technology, as has always been the case. The reward will be a much more seamless integration of visual computing into our everyday lives. As a small step toward that goal, we’re developing LumiConSense, the first image sensor to use a luminescent concentrator (LC), a transparent polycarbonate film doped with fluorescent dyes.1,2 LumiConSense is ■
■ ■
■
transparent (it has no integrated circuits or other structures such as grids of optical fibers or photodiodes), flexible (it makes curved sensor shapes possible), scalable (the sensor can range from small to large at a similar cost, and the pixel size isn’t restricted to the photodiode size), and disposable (the sensing area is low-cost and replaceable).
Figure 1 shows the LC, which allows lensless multifocal imaging and depth estimation at interactive rates. Figure 1. The LumiConSense sensor employs a thin luminescent concentrator (LC) film, which allows lensless multifocal imaging and depth estimation at interactive rates. 98
g5gra.indd 98
September/October 2014
LC Physics and Sensor Optics Our research is motivated by advances in mate-
Published by the IEEE Computer Society
0272-1716/14/$31.00 © 2014 IEEE
8/21/14 4:13 PM
Diffusor (40 µm) Total internal reflection
mm
Blocker
Transmission
Optic
al fib ers
6.
3.25
Optic
25
m
6.7
m
500 µm
5m
Emission
LC film
m
Blocker
Aperture
al fib ers
Blocker
250 µm
300 µm
(a)
Cut-out areas not filled with opaque plastiline lead to undesired stray light
LC foil
x1
LC foil
x2
x3
x4
LC foil
x5
... Strip of reflective paint
φ1.φ2.φ3.φ4.φ5, ... Measured sensor response at each triangle slit position (xi) and each direction (φj ) (b) Figure 2. Two views of the LC film. (a) The basic physics.2 (b) The sensor optics that sample the 2D light field being transported inside the LC.1
rial science, mechanical and optical engineering, and computational methods that have led to novel sensors and camera designs that go beyond conventional CCD (charge-coupled device) or CMOS chips. For example, hemispherical image sensors inspired by the human eye’s curved retina3 or the compound apposition layouts of arthropod eyes4 demonstrate wide field-of-view, low-aberration imaging. Woven polymeric photodetecting fibers5 and organic photodiodes together with ink-jet digital lithography6 enable scalable image sensors with complex geometric shapes. Compressive sensing has made lensless imaging possible.7 In LumiConSense, light of a particular wavelength subband that penetrates the LC is emitted in longer wavelengths; wavelengths outside the
g5gra.indd 99
subband are fully transmitted. The examples in this article absorb blue light and emit green light. As Figure 2 illustrates, the emitted light is mostly trapped inside the film by total internal reflection. It’s transported with reduced multiscattering toward the LC edges while losing energy over the transport distance. The bright film edges indicate decoupling of the light integral transported to each edge point from all directions inside the LC.
Image Reconstruction To reconstruct an image from this complex light integral, we must reversely estimate the brightness of all image points at various distances. We do this by measuring and demultiplexing the light integral that leaves the edges.1 As Figure 2 illustrates, IEEE Computer Graphics and Applications
99
8/21/14 4:14 PM
Graphically Speaking
we cut the LC border regions into a structure of adjacent triangular apertures. The apertures act as an array of 1D pinholes that multiplex the light integral into a 2D light field l(x, f). This light field encodes the fractions of the light transported from a particular direction f to a particular position x on the LC edges. The light field projected at the edges is forwarded through optical fibers to line scan cameras (LSCs) for measurement. Image reconstruction requires an initial (one time) calibration of the sensor. One option is to project a uniform light impulse sequentially to each discrete image point (pixel) pi on the sensor surface while measuring the resulting basis light
Our flexible, low-cost sensor can also be leveraged for devices and objects to sense their interactions with environments and humans in richer ways. field li(x, f).1 These measurements are stored in the columns of a light transport matrix T. They represent the individual transport of light from each pixel, through the apertures and optical fibers, to each photosensor of the LSCs. Because LumiConSense doesn’t consist of physical pixels, its resolution is virtual and depends on the resolution applied to T and on the number of sampled basis light fields. So, the size of T equals the number of virtual pixels (number of columns) times the number of photosensors (number of rows) of the LSCs. We formulate the light transport of a complex monochrome image p focused on the LC as l = Tp + e, where p is the vector-form representation of the image pixels, and l and e are the vectorform representations of the integral light fields measured during online image recording and off line recording of the baseline light level (part of the calibration), respectively. We thus reconstruct the image by solving p = T–1(l – e). Tomographic image reconstruction techniques such as SART (simultaneous algebraic reconstruction technique) are well suited to addressing this problem.1 In practice, the LSCs’ signal-to-noise ratio limits the reconstructed images’ resolution. To achieve a better image quality at a higher resolution, we project several thousand randomly selected Flickr images onto the sensor and apply linear regression to derive T–1 directly. We 100
g5gra.indd 100
can achieve a superresolution reconstruction by combining the lower-resolution reconstruction results of multiple (lower-resolution) transport matrices measured or computed with subpixel shifts on the sensor.1 Figures 3 illustrates the image quality achievable today; Figure 4 shows a prototype that reconstructs images of shadows cast onto the LC surface at up to 12 fps with a resolution of up to 128 × 128 pixels.
Multifocal Imaging and Depth Sensing So far, we’ve explained how to reconstruct images focusing on the LC surface. However, we can shift our sensor’s focus in the axial direction without additional optics such as lenses.2 This allows instant estimation of an entire focal stack from a single measurement. It also enables subsequent depth reconstruction of objects in front of the sensor with a noise-tolerant variation of depth from defocus (see Figure 5). To achieve lensless multifocal imaging and depth sensing, we don’t use the transport matrix that we measured with light impulses focused on the film surface. Instead, we apply transport matrices that we’ve calibrated or computed for the corresponding focal distances.2 These matrices unblur the defocus on the sensor surface due to out-of-focus objects. So, LumiConSense lets us reconstruct an entire focal stack after only one recording. We can use the focal stack to estimate depth from defocus. Stacking sensors lets us simultaneously sample a variety of information, such as color, dynamic range, spatial resolution, and defocus.
M
ost touch sensors are limited mainly to planar shapes and interaction through direct touch. In contrast, our approach could lead to new human–computer interfaces that are unconstrained in shape and sensing distance. Our flexible, low-cost sensor can also be leveraged for devices and objects to sense their interactions with environments and humans in richer ways. We’re constantly improving image quality and reconstruction performance by enhancing the mathematical and optical models. This might lead to entirely new sensing options such as omnidirectional 4D light-field imaging or implicit gesture and object recognition.
Acknowledgments We thank Robert Koeppe of isiQiri Interface Technologies for fruitful discussions and for providing luminescent-concentrator samples. Microsoft Research
September/October 2014
8/21/14 4:14 PM
(a)
2011
2014
(b) Figure 3. Working toward better image quality. (a) An image focused on the LC surface.1 (b) How reconstruction results have improved over the past three project years.
supported this research under contract 2012-030 (DP874903)—LumiConSense.
References 1. A. Koppelhuber and O. Bimber, “Towards a Transparent, Flexible, Scalable and Disposable Image Sensor Using Thin-Film Luminescent Concentrators,” Optics Express, vol. 21, no. 4, 2013, pp. 4796–4810. 2. A. Koppelhuber et al., “A Transparent Thin-Film Sensor for Multi-focal Image Reconstruction and Depth Estimation,” Optics Express, vol. 22, no. 8, 2014, pp. 8928–8942. 3. H.C. Ko et al., “A Hemispherical Electronic Eye Camera Based on Compressible Silicon Optoelectronics,” Nature, vol. 454, no. 7205, 2008, pp. 748–753. 4. Y.M. Song et al., “Digital Cameras with Designs Inspired by the Arthropod Eye,” Nature, vol. 497, no. 7447, 2013, pp. 95–99. 5. A.F. Abouraddy et al., “Large-Scale Optical-Field Measurements with Geometric Fibre Constructs,” Nature Materials, vol. 5, no. 7, 2006, pp. 532–536.
g5gra.indd 101
Figure 4. Real-time reconstruction of shadows cast onto the film surface. This prototype reconstructs the shadows at up to 12 fps with a resolution of up to 128 × 128 pixels. IEEE Computer Graphics and Applications
101
8/21/14 4:14 PM
Graphically Speaking
350 mm (+20 mm)
326 mm (+11 mm)
Focal stack xy slices
y (a)
z
x
(b)
Figure 5. Depth reconstruction of objects in front of the sensor. (a) An experimental setup with two targets at different distances. (b) A reconstructed focal stack and target distances with estimation error.
NEW
STORE Find the latest trends and insights for your • presentations • research • events
6. T.N. Ng et al., “Flexible Image Sensor Array with Bulk Heterojunction Organic Photodiode,” Applied Physics Letters, vol. 92, no. 21, 2008, article 213303. 7. J. Hunt et al., “Metamaterial Apertures for Computational Imaging,” Science, vol. 339, no. 6117, 2013, pp. 310–313. Alexander Koppelhuber is a project assistant at Johannes Kepler University’s Institute of Computer Graphics. Contact him at [email protected]. Oliver Bimber is the head of Johannes Kepler University’s Institute of Computer Graphics. Contact him at oliver. [email protected]. Contact department editor André Stork at andre.stork@igd. fraunhofer.de.
webstore.computer.org Selected CS articles and columns are also available for free at http://ComputingNow.computer.org. 102
g5gra.indd 102
September/October 2014
CSStore_quarter_kc_updated.indd 1
7/2/14 10:47 AM
8/21/14 4:14 PM
Editor: André Stork
LumiConSense A Transparent, Flexible, and Scalable Thin-Film Sensor Alexander Koppelhuber and Oliver Bimber Johannes Kepler University
I
magine that car windshields can observe passengers and that shopping windows can observe customers. Glasses can track gaze direction, while floors can track crowds. 2D screens no longer have to be touched, and 3D screens can be properly interacted with. Plastic or glass containers can be aware of their content, and our clothing can be aware of our environment. What sounds futuristic could soon become part of our daily life. Both imaging and display technologies are evolving faster than ever. For instance, bendable, transparent screens based on organic light-emitting polymers are close to commercialization, and flexible image sensors are in their infancy. In addition, how we represent scenes with that technology is also evolving. So far, scene representation has relied on capturing digital ras-
ter images with planar sensors, computing them, and presenting them on planar displays. However, simple extensions toward 3D representation with 3D cameras and 3D screens are becoming increasingly successful. More advanced representations, such as 4D light fields or holograms, might hold even more potential. These advances will radically change everything related to computer graphics, visualization, computer vision, and image processing. The mathematical models and underlying algorithms and data structures based on processing 2D raster images are far from being carved in stone. They’ll evolve along with imaging and display technology, as has always been the case. The reward will be a much more seamless integration of visual computing into our everyday lives. As a small step toward that goal, we’re developing LumiConSense, the first image sensor to use a luminescent concentrator (LC), a transparent polycarbonate film doped with fluorescent dyes.1,2 LumiConSense is ■
■ ■
■
transparent (it has no integrated circuits or other structures such as grids of optical fibers or photodiodes), flexible (it makes curved sensor shapes possible), scalable (the sensor can range from small to large at a similar cost, and the pixel size isn’t restricted to the photodiode size), and disposable (the sensing area is low-cost and replaceable).
Figure 1 shows the LC, which allows lensless multifocal imaging and depth estimation at interactive rates. Figure 1. The LumiConSense sensor employs a thin luminescent concentrator (LC) film, which allows lensless multifocal imaging and depth estimation at interactive rates. 98
g5gra.indd 98
September/October 2014
LC Physics and Sensor Optics Our research is motivated by advances in mate-
Published by the IEEE Computer Society
0272-1716/14/$31.00 © 2014 IEEE
8/21/14 4:13 PM
Diffusor (40 µm) Total internal reflection
mm
Blocker
Transmission
Optic
al fib ers
6.
3.25
Optic
25
m
6.7
m
500 µm
5m
Emission
LC film
m
Blocker
Aperture
al fib ers
Blocker
250 µm
300 µm
(a)
Cut-out areas not filled with opaque plastiline lead to undesired stray light
LC foil
x1
LC foil
x2
x3
x4
LC foil
x5
... Strip of reflective paint
φ1.φ2.φ3.φ4.φ5, ... Measured sensor response at each triangle slit position (xi) and each direction (φj ) (b) Figure 2. Two views of the LC film. (a) The basic physics.2 (b) The sensor optics that sample the 2D light field being transported inside the LC.1
rial science, mechanical and optical engineering, and computational methods that have led to novel sensors and camera designs that go beyond conventional CCD (charge-coupled device) or CMOS chips. For example, hemispherical image sensors inspired by the human eye’s curved retina3 or the compound apposition layouts of arthropod eyes4 demonstrate wide field-of-view, low-aberration imaging. Woven polymeric photodetecting fibers5 and organic photodiodes together with ink-jet digital lithography6 enable scalable image sensors with complex geometric shapes. Compressive sensing has made lensless imaging possible.7 In LumiConSense, light of a particular wavelength subband that penetrates the LC is emitted in longer wavelengths; wavelengths outside the
g5gra.indd 99
subband are fully transmitted. The examples in this article absorb blue light and emit green light. As Figure 2 illustrates, the emitted light is mostly trapped inside the film by total internal reflection. It’s transported with reduced multiscattering toward the LC edges while losing energy over the transport distance. The bright film edges indicate decoupling of the light integral transported to each edge point from all directions inside the LC.
Image Reconstruction To reconstruct an image from this complex light integral, we must reversely estimate the brightness of all image points at various distances. We do this by measuring and demultiplexing the light integral that leaves the edges.1 As Figure 2 illustrates, IEEE Computer Graphics and Applications
99
8/21/14 4:14 PM
Graphically Speaking
we cut the LC border regions into a structure of adjacent triangular apertures. The apertures act as an array of 1D pinholes that multiplex the light integral into a 2D light field l(x, f). This light field encodes the fractions of the light transported from a particular direction f to a particular position x on the LC edges. The light field projected at the edges is forwarded through optical fibers to line scan cameras (LSCs) for measurement. Image reconstruction requires an initial (one time) calibration of the sensor. One option is to project a uniform light impulse sequentially to each discrete image point (pixel) pi on the sensor surface while measuring the resulting basis light
Our flexible, low-cost sensor can also be leveraged for devices and objects to sense their interactions with environments and humans in richer ways. field li(x, f).1 These measurements are stored in the columns of a light transport matrix T. They represent the individual transport of light from each pixel, through the apertures and optical fibers, to each photosensor of the LSCs. Because LumiConSense doesn’t consist of physical pixels, its resolution is virtual and depends on the resolution applied to T and on the number of sampled basis light fields. So, the size of T equals the number of virtual pixels (number of columns) times the number of photosensors (number of rows) of the LSCs. We formulate the light transport of a complex monochrome image p focused on the LC as l = Tp + e, where p is the vector-form representation of the image pixels, and l and e are the vectorform representations of the integral light fields measured during online image recording and off line recording of the baseline light level (part of the calibration), respectively. We thus reconstruct the image by solving p = T–1(l – e). Tomographic image reconstruction techniques such as SART (simultaneous algebraic reconstruction technique) are well suited to addressing this problem.1 In practice, the LSCs’ signal-to-noise ratio limits the reconstructed images’ resolution. To achieve a better image quality at a higher resolution, we project several thousand randomly selected Flickr images onto the sensor and apply linear regression to derive T–1 directly. We 100
g5gra.indd 100
can achieve a superresolution reconstruction by combining the lower-resolution reconstruction results of multiple (lower-resolution) transport matrices measured or computed with subpixel shifts on the sensor.1 Figures 3 illustrates the image quality achievable today; Figure 4 shows a prototype that reconstructs images of shadows cast onto the LC surface at up to 12 fps with a resolution of up to 128 × 128 pixels.
Multifocal Imaging and Depth Sensing So far, we’ve explained how to reconstruct images focusing on the LC surface. However, we can shift our sensor’s focus in the axial direction without additional optics such as lenses.2 This allows instant estimation of an entire focal stack from a single measurement. It also enables subsequent depth reconstruction of objects in front of the sensor with a noise-tolerant variation of depth from defocus (see Figure 5). To achieve lensless multifocal imaging and depth sensing, we don’t use the transport matrix that we measured with light impulses focused on the film surface. Instead, we apply transport matrices that we’ve calibrated or computed for the corresponding focal distances.2 These matrices unblur the defocus on the sensor surface due to out-of-focus objects. So, LumiConSense lets us reconstruct an entire focal stack after only one recording. We can use the focal stack to estimate depth from defocus. Stacking sensors lets us simultaneously sample a variety of information, such as color, dynamic range, spatial resolution, and defocus.
M
ost touch sensors are limited mainly to planar shapes and interaction through direct touch. In contrast, our approach could lead to new human–computer interfaces that are unconstrained in shape and sensing distance. Our flexible, low-cost sensor can also be leveraged for devices and objects to sense their interactions with environments and humans in richer ways. We’re constantly improving image quality and reconstruction performance by enhancing the mathematical and optical models. This might lead to entirely new sensing options such as omnidirectional 4D light-field imaging or implicit gesture and object recognition.
Acknowledgments We thank Robert Koeppe of isiQiri Interface Technologies for fruitful discussions and for providing luminescent-concentrator samples. Microsoft Research
September/October 2014
8/21/14 4:14 PM
(a)
2011
2014
(b) Figure 3. Working toward better image quality. (a) An image focused on the LC surface.1 (b) How reconstruction results have improved over the past three project years.
supported this research under contract 2012-030 (DP874903)—LumiConSense.
References 1. A. Koppelhuber and O. Bimber, “Towards a Transparent, Flexible, Scalable and Disposable Image Sensor Using Thin-Film Luminescent Concentrators,” Optics Express, vol. 21, no. 4, 2013, pp. 4796–4810. 2. A. Koppelhuber et al., “A Transparent Thin-Film Sensor for Multi-focal Image Reconstruction and Depth Estimation,” Optics Express, vol. 22, no. 8, 2014, pp. 8928–8942. 3. H.C. Ko et al., “A Hemispherical Electronic Eye Camera Based on Compressible Silicon Optoelectronics,” Nature, vol. 454, no. 7205, 2008, pp. 748–753. 4. Y.M. Song et al., “Digital Cameras with Designs Inspired by the Arthropod Eye,” Nature, vol. 497, no. 7447, 2013, pp. 95–99. 5. A.F. Abouraddy et al., “Large-Scale Optical-Field Measurements with Geometric Fibre Constructs,” Nature Materials, vol. 5, no. 7, 2006, pp. 532–536.
g5gra.indd 101
Figure 4. Real-time reconstruction of shadows cast onto the film surface. This prototype reconstructs the shadows at up to 12 fps with a resolution of up to 128 × 128 pixels. IEEE Computer Graphics and Applications
101
8/21/14 4:14 PM
Graphically Speaking
350 mm (+20 mm)
326 mm (+11 mm)
Focal stack xy slices
y (a)
z
x
(b)
Figure 5. Depth reconstruction of objects in front of the sensor. (a) An experimental setup with two targets at different distances. (b) A reconstructed focal stack and target distances with estimation error.
NEW
STORE Find the latest trends and insights for your • presentations • research • events
6. T.N. Ng et al., “Flexible Image Sensor Array with Bulk Heterojunction Organic Photodiode,” Applied Physics Letters, vol. 92, no. 21, 2008, article 213303. 7. J. Hunt et al., “Metamaterial Apertures for Computational Imaging,” Science, vol. 339, no. 6117, 2013, pp. 310–313. Alexander Koppelhuber is a project assistant at Johannes Kepler University’s Institute of Computer Graphics. Contact him at [email protected]. Oliver Bimber is the head of Johannes Kepler University’s Institute of Computer Graphics. Contact him at oliver. [email protected]. Contact department editor André Stork at andre.stork@igd. fraunhofer.de.
webstore.computer.org Selected CS articles and columns are also available for free at http://ComputingNow.computer.org. 102
g5gra.indd 102
September/October 2014
CSStore_quarter_kc_updated.indd 1
7/2/14 10:47 AM
8/21/14 4:14 PM
