SCANNING VOL. 36, 530–539 (2014) © Wiley Periodicals, Inc.

Improvement to the Scanning Electron Microscope Image Adaptive Canny Optimization Colorization by Pseudo-Mapping T. Y. LO, K. S. SIM, C. P. TSO,

AND

M. E. NIA

Faculty of Engineering & Technology, Multimedia University, Melaka, Malaysia

Summary: An improvement to the previously proposed adaptive Canny optimization technique for scanning electron microscope image colorization is reported. The additional feature, called pseudo-mapping technique, is that the grayscale markings are temporarily mapped to a set of pre-defined pseudo-color map as a mean to instill color information for grayscale colors in chrominance channels. This allows the presence of grayscale markings to be identified; hence optimization colorization of grayscale colors is made possible. This additional feature enhances the flexibility of scanning electron microscope image colorization by providing wider range of possible color enhancement. Furthermore, the nature of this technique also allows users to adjust the luminance intensities of selected region from the original image within certain extent. SCANNING 36:530–539, 2014. © 2014 Wiley Periodicals, Inc. Key words: Scanning electron microscope, optimization, colorization, Canny edge detection, adaptive tuning

Introduction Scanning electron microscope (SEM) is an electron microscopy instrument that uses high energy beam of electrons to scan and image the surface of specimen on a very fine scale. During the scanning process, the electrons interact with the atoms of specimen producing signals, which contain information about the properties of the specimen such as topography, morphology, and crystallography as well as material composition of the specimen. In a typical SEM, the specimens are usually

Address for reprints: Dr. Kok-Swee Sim, Faculty of Engineering & Technology, Multimedia University, Melaka 75450, Malaysia E-mail: [email protected] Received 15 April 2014; Accepted with revision 27 June 2014 DOI: 10.1002/sca.21152 Published online 19 August 2014 in Wiley Online Library (wileyonlinelibrary.com).

coated with a thin layer of conductive material and grounded prior to SEM scanning process, in order to increase the resolution of the specimens and prevent charging effect due to accumulation of electrostatic charges. Since SEM images are usually captured as grayscale images, it is of interest to colorize the SEM images as to enhance the visual appeal and information contents of the images. Throughout the history of grayscale image colorization, various colorization techniques had been proposed since the pioneering work by Markle in 1970 (Burns, 2010), yet there are only limited existing techniques that are dedicated for SEM colorization platform. Welsh et al. (2004) introduced a general technique for grayscale image colorizing by transferring color between a source, color image and a designated colorized image. This technique works well on images and videos, provided that texture and luminance are sufficiently distinct. Luan et al. (2007) developed a natural image colorizing method that works in two stages, namely, color labeling and color mapping. Pixels that should roughly share similar colors are grouped into coherent regions in the color labeling stage, and then introduced to further color fine-tuning in each coherent region. Guillaume et al. (2008) developed an automated image colorization technique using multimodal predictions where user-provided color landmarks interactively correct the color propositions. Sung and March (2007) developed variational models to colorize grayscale images using chromaticity color components where edge information is added from the brightness data, while smooth color values are reconstructed for each homogeneous region. Lagodzinski and Smolka (2008) used probabilistic distance transformation for image colorization. This method applies modified morphological distance transform using Gibbs distribution to automatically propagate the color scribbled by a user on the gray scale image. Among the existing techniques which specific for SEM colorization platform are Canny optimization technique (Sim et al., 2008), and its enhanced version, the adaptive Canny optimization technique. For Canny optimization, the technique was based on the

T. Y. Lo et al.: Improvement to the SEM Image Colorization

hybridization of Canny edge detection technique with existing optimization colorization technique. During the colorization process, grayscale SEM image first undergoes Canny edge detection procedure and those detected edges are later overlaid upon the original image prior to the optimization colorization process. The overlaid edges create distinctive boundaries in between the objects and background image, results in better color flow control for the colorized image. This method was further improved by applying adaptive tuning process right after the colorization process in a year later. The enhanced version, called adaptive Canny optimization, incorporates an additional adaptive tuning process to rectify the luminance channel of the colorized image based on the original image intensity level. Compared to its predecessor, the adaptive Canny optimization technique has excellent color flow control and better contrast, which gives satisfactory results for SEM images (Sim et al., 2009). Although adaptive Canny optimization offers users with great ease during SEM colorization process, yet there is still a loophole in this method. As an optimization-based method, colorized results for adaptive Canny optimization are generated through combination of original image luminance channel with the newly generated chrominance channels. This principle works brilliantly for coloring grayscale images, however if users intend to insert grayscale markings instead of color markings, those grayscale markings will have no effect upon the output image. This phenomenon seems trivial as users seldom will apply grayscale markings during colorization, yet it could be rather inconvenient if users do intend to do so. To remove this limitation, the authors have introduced a new procedure, named pseudo-mapping, into the existing colorization process.

Previous Work For conventional optimization technique (Anat et al., 2004), the grayscale images are first converted into luminance-chrominance (YUV) color space, where the luminance channel (Y) contains the grayscale intensity level of original image, while the chrominance channel (UV) carries the remaining color information. Equation (1) shows a conversion formula between the RGB color and YUV (Buchsbaum, ’75): 2 3 2 32 3 Y 0:299 0:587 0:114 R 6 7 6 76 7 4 U 5 ¼ 4 0:595716 0:274453 0:321263 54 G 5: V

0:211456 0:522591

0:311135

B

ð1Þ During the optimization colorization process, for each non-color marked pixel, pixel r, the variance (s 2r ) of its eight-connected neighboring pixels, pixel s, is

531

calculated. The equation is

sr 2 ¼

PN 1 i¼1

ðY ðrÞ  Y ðsi ÞÞ2 ; 9

ð2Þ

where Y(r) and Y(s) represent the intensity of the pixels. The amount of color transferred from color marked pixel to non-color marked pixel is then determined based on a weighting function. The weighting function (vrs) is derived from the squared different between pixel r and pixel s intensities as shown in the following equation: 2

vrs / eðY ðrÞY ðsÞÞ =2sr : 2

ð3Þ

Since the weighting function (vrs) is inversely proportional to difference of pixel r and pixel s intensities, hence the larger the intensity difference between the measured pixels, the lesser amount of color information will be transferred from one to another, and vice versa. The amount of color transferred from one pixel to another is determined using the following equation: X X J ðU Þ ¼ ðU ðrÞ  vrs U ðsÞÞ2 ; ð4Þ r

s2N ðrÞ

where U(r) and U(s) represent the chrominance values of the pixels, while J(U) indicates the amount of color transferred from pixel to pixel. Based on Equation (4), a new color information distribution matrix can be generated. Through the combination of new color information matrix with original image luminance channel, a desired colorized image is obtained. However, if grayscale markings are involved during the colorization process, the conventional optimization technique will fail to generate the desired result. Since the grayscale colors do not contain any color information in chrominance channels; hence there will be no color transferred from grayscale marked pixels to nearby non-marked pixels. Under this circumstance, the presence of grayscale markings is literally being ignored while generating the new color information distribution matrix. This will produce voided areas without any useful color information within the newly generated distribution matrix, thus results in the original grayscale image remaining unchanged or preserved at the end of the colorization process. Figure 1 demonstrates the grayscale colorization limitation of optimization technique. Furthermore, for adaptive Canny optimization technique, color tuning is performed adaptively to rectify the luminance channel of colorized image approximately to its original value. Although this technique greatly reduces the amount of erroneous color assignment during colorization process and enhances the contrast of

532

SCANNING VOL. 36, 5 (2014)

Fig 1. Grayscale colorization limitation: (a) original grayscale image, (b) image with color markings, (c) image after adaptive Canny optimization colorization. Horizontal field width ¼ 25 mm.

the colorized image, it also has the tendency to obscure fine details to be displayed at areas with extreme luminance channel intensity. This will render details around those areas indiscernible by the human eye, thus reducing the amount of information contents observed from the image. Figure 2 illustrates the micro-structures of the image being obscured by the adaptive color tuning technique.

Materials and Methods In the currently proposed technique, we improved the adaptive Canny optimization technique by introducing pseudo-mapping feature. This technique allows a wider color selection, as well as alteration of the luminance intensity level of the original SEM images. Figure 3 demonstrates the pseudo-mapping procedure that is implemented during the colorization process. Once the markings are being placed onto the grayscale SEM image, the presence of markings is first being identified by subtracting the original SEM image

from the “marked” version, followed by classifying the markings into grayscale and non-grayscale categories using the following classification condition: ( M ¼ fðU ¼ 0Þ ðV ¼ 0ÞÞ; M

Grayscale; if m ¼ 1 Non  greyscale; otherwise

;

ð5Þ where U and V represent the chrominance values from the inserted markings. Symbol & denotes the logical AND function used for comparison purpose, while M indicates the result returned by the logical function. Once the locations of grayscale markings are identified, these markings will be temporarily mapped into a pre-defined pseudo colormap. The colormap is defined as two randomly selected columns with each column having a constant value, either 0 or 1. The remaining column consists of values, which gradually increase from 0 up to 1. Figure 4 shows snippet of a possible example of pseudo colormap.

T. Y. Lo et al.: Improvement to the SEM Image Colorization

533

Fig 2. Image micro-structures obscured by adaptive color tuning: (a) original grayscale image, (b) image after Canny optimization colorization, (c) image after adaptive Canny optimization colorization. Horizontal field width ¼ 25 mm.

In order to map the grayscale markings into pseudocolor map, an index is required. The grayscale markings are first converted to defined index values, and based on these values the grayscale markings are later mapped to the pre-define pseudo colormap prior to colorization process. The equation uses for generating index values is   ðG  Gmin Þ index ¼ f ix  ðL  1Þ þ 1; ð6Þ Gmax  Gmin where G represents the intensity value of the grayscale markings, while L uses to indicate the range of index values. The fix function is used to round the elements to the nearest integers towards zero. Once the grayscale markings are indexed, the markings are assigned with their corresponding pseudo-color. While for the rest of the non-grayscale markings, white color are assigned to remove their existing color information. This procedure serves as a mean to provide grayscale markings with useful color information, while at the same time nullify the effect of non-grayscale markings upon image during the colorization process. By introducing color information to the

grayscale markings, optimization colorization can easily recognizes and calculates the color propagation for those grayscale markings. Once the colorization process is completed, regions colorized by the pre-defined pseudo colormap are identified and re-mapped to its original intended colors by matching the colorized pixels with the pseudo colormap; all three color layers of colorized pixels are replaced by nearest index value of pseudo colormap to restore those pixels with their intended grayscale values. Next, the luminance intensity of remapped regions is properly adjusted to achieve optimum result before superimposed onto the luminance channel of the original image. The newly generated luminance layer is then combined with the color information distribution matrix to produce the desired colorized image. Adaptive color tuning procedure is applied to improve the contrast of the image in the end of colorization process. Figure 5 shows the images of the intermediate pseudo-mapping stages. Figure 6 illustrates the overall flow chart for adaptive Canny optimization method with incorporated pseudo-mapping feature.

534

SCANNING VOL. 36, 5 (2014)

Fig 4. Snippet of pseudo colormap (autumn).

Fig 3. Pseudo-mapping flow chart.

Results In this study, we work on four different types of SEM images and compare the results after colorization. In each image, we apply two different methods: adaptive Canny optimization method and the proposed adaptive Canny optimization method with pseudo-mapping feature.

Figure 7(a) shows an example of grayscale SEM image. Figure 7(b) demonstrates the color markings placed by user onto the grayscale image. In Figure 7(c), regions marked by the black color markings retain its original color after the colorization process. For optimization colorization technique, limitation in identifying the presence of grayscale markings creates voided areas in the newly generated color information distribution matrix. Without the proper color information, the combination of luminance and chrominance channels is unable to faithfully produce the desired colorized image. Figure 7(d) illustrates areas successfully colorized by the black color markings with the help of proposed pseudo-mapping method. Figure 8(a) illustrates an example of grayscale SEM image with charging effect. Figure 8(b) shows the color markings placed onto the image. White color markings are deliberately placed at the areas where the charging effect occurred as a mean to allow easily identification of the image artifact after colorization. In Figure 8(c), due to the grayscale colorization limitation, the areas marked with white markings failed to enhance the presence of the image artifact. Figure 8(d) shows that with the help of proposed technique, the areas affected by charging effect had been successfully highlighted with higher brightness compared to the original SEM image. This allows the locations of the charging effect becoming even more outstanding among the rest of the image, which can be rapidly identified by the user. Figure 9(a) shows a SEM image of integrated circuit (IC) wire bonding. Figure 9(b) demonstrates the color markings placed by the user onto the grayscale image. In Figure 9(c), the result after performing adaptive Canny

T. Y. Lo et al.: Improvement to the SEM Image Colorization

535

Fig 5. Images of intermediate pseudo-mapping stages: (a) grayscale color markings, (b) color markings mapped to predefine colormap, (c) image after optimization colorization, (d) combination of new luminance layer with chrominance layer. Horizontal field width ¼ 25 mm.

optimization colorization technique is illustrated. The result shows that color markings placed at the areas with extreme high luminance intensity value or the brighter areas, has not significantly effect upon the image after the colorization process. This is mainly due to the adaptive tuning procedure which rectifies the luminance intensity of colorized image towards its original intensity values. This might cause inconvenience for the user and is undesirable under certain circumstances. Figure 9(d) illustrates the result after colorization where the image is pre-processed with pseudo-mapping technique to alter the luminance channel intensity of specific areas. Result shows that the areas are successfully colorized with blue color as intended by the user. Through the proposed method, the user can adjust the luminance intensity of the image to serve different colorization purposes. Figure 10(a) shows an original SEM image. Figure 10 (b) demonstrates the color markings placed by user onto the grayscale image. Figure 10(c) shows that color markings placed at the areas with extreme low luminance intensity values, or the shaded areas, failed to faithfully represent the microstructure details of the

SEM image at the end of colorization process. The adaptive tuning procedure rectifies the luminance intensity of colorized image, which obscures the information contents and makes it indiscernible to the human eye. Figure 10(d) illustrates the result after colorization where the image is pre-processed with pseudo-mapping technique to alter the luminance intensity of peripheral areas. From the results, one can observe, that details, which were previously obscured in Figure 10(c), can now be clearly observed in Figure 10 (d). Through the proposed method, the area of interest can be emphasized to allow more information contents from the image to be detected by the user. This technique allows compromise between preservation of microstructures at region of interest and better contrast of the image since user can alters the image intensity partially instead of the whole image.

Discussion In this section, we work on several types of SEM images to demonstrate the possible applications of the proposed pseudo-mapping feature.

536

SCANNING VOL. 36, 5 (2014)

Fig 6. Flow chart for adaptive Canny optimization with incorporated pseudo-mapping feature.

Figure 10(a) shows an example of grayscale SEM image. The darker peripheral region of the centered object prevents finer details to be observed from the image. Figure 10(b) demonstrates the color markings placed by the user onto the image. In Figure 10(d), result after performing the pseudo-mapping procedure is illustrated. Prior to optimization colorization process, the peripheral region of the image is first treated by pseudo-mapping procedure to alter its original intensity level until it is sufficient enough to represent the hidden layer of microstructures. From the results, it can be clearly observed that microstructures previously obscured in the original SEM images are faithfully represented in the colorized image. Through the proposed method, user can emphasize or enhance the visibility of microstructures or three dimensional details of the image at region of interest without affecting or degrading the information contents of the rest of image. In Figure 11(a), the original SEM image is captured for industrial material failure analysis purpose. However, due to the low intensity difference, the cracked area cannot be easily identified. Therefore, the original image is pre-processed by pseudo-mapping to increase the intensity values of the suspected region. The postprocessed image is shown in Figure 11(b). Figure 11(c) and (d) illustrates the color-marked image and its corresponding colorized result. One can observe that the

Fig 7. SEM image with black markings: (a) original SEM image, (b) image with color markings, (c) image after adaptive Canny optimization method, (d) image after adaptive Canny optimization with pseudo-mapping feature. Horizontal field width ¼ 25 mm.

T. Y. Lo et al.: Improvement to the SEM Image Colorization

537

Fig 8. SEM image with white markings: (a) original SEM image, (b) image with color markings, (c) image after adaptive Canny optimization method, (d) image after adaptive Canny optimization with pseudo-mapping feature. Horizontal field width ¼ 20mm.

Fig 9. Alteration of luminance channel intensity: (a) original SEM image, (b) image with color markings, (c) image after adaptive Canny optimization method, (d) image after adaptive Canny optimization with pseudo-mapping feature. Horizontal field width ¼ 25 mm.

538

SCANNING VOL. 36, 5 (2014)

Fig 10. Second example for alteration of luminance intensities: (a) original SEM image, (b) image with color markings, (c) image after adaptive Canny optimization method, (d) image after adaptive Canny optimization with pseudo-mapping feature. Horizontal field width ¼ 25 mm.

Fig 11. Cracked chip surface: (a) original SEM image, (b) post-processed SEM image, (c) image with color markings, (d) image after adaptive Canny optimization with pseudo-mapping feature. Horizontal field width ¼ 25 mm.

T. Y. Lo et al.: Improvement to the SEM Image Colorization

presence of cracked area is easier to identify compared to original SEM image without jeopardizing the original intensity values of adjacent regions.

Conclusion In this study, with the implementation of the proposed pseudo-mapping feature, a promising colorized result with wider color enhancement and lesser limitation can be obtained. Through this technique, the intensity of original grayscale image luminance channel can be altered to emphasize the content of interest without significant loss of original information. In future, it may be possible to further enhance this colorization method to allow more precise adjustment of grayscale intensity level by using a more accurate and dynamic classification system.

References Anat L, Dani L, Yair W. 2004. Colorization using Optimisation. ACM T Graphic 23:689–694.

539

Buchsbaum WH. 1975. Color TV servicing. 3rd edition. New Jersey: Prentice Hall. Burns G. 2010. Colorization, Museum of Broadcast Communications. Available Online at: http://www.museum.tv/archives/ etv/C/htmlC/colorization/colorization.htm. [accessed: March 2010]. Guillaume C, Matthias H, Bernhard S. 2008. Automatic image colorization via multimodal predictions, computer vision— ECCV 2008, Lecture Notes in Computer Science. Berlin, Heidelberg: Springer. p 129–139. Lagodzinski P, Smolka B. 2008. Digital image colorization based on probabilistic distance transformation, ELMAR, 2008. 50th International Symposium, Zadar, Croatia 2:495–498. Luan Q, Wen F, Cohen-Or D, et al. 2007. Natural image colorization. In: Kautz J, Pattanaik S, editors. Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR’07). Aire-la-Ville, Switzerland, Switzerland: Eurographics Association. p 309–320. Sim KS, Tso CP, Ting HY. 2008. Canny optimization technique for electron microscope image colorization. J Microsc 232: 313–334. Sim KS, Ting HY, Lai MA, Tso CP. 2009. Improvement to scanning electron microscope image colorization by adaptive tuning. J Microsc 234:243–250. Sung HK, March R. 2007. Variational Models for image colorization via chromaticity and brightness decomposition. IEEE Trans Image Process 16:2251–2261. Welsh T, Ashikhmin M, Mueller K. 2004. Transferring color to greyscale images. ACM T Graphic 21:277–280.

Improvement to the scanning electron microscope image adaptive Canny optimization colorization by pseudo-mapping.

An improvement to the previously proposed adaptive Canny optimization technique for scanning electron microscope image colorization is reported. The a...
3MB Sizes 0 Downloads 5 Views