4838

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 11, NOVEMBER 2014

Video Tonal Stabilization via Color States Smoothing Yinting Wang, Dacheng Tao, Senior Member, IEEE, Xiang Li, Mingli Song, Senior Member, IEEE, Jiajun Bu, Member, IEEE, and Ping Tan, Member, IEEE

Abstract— We address the problem of removing video color tone jitter that is common in amateur videos recorded with handheld devices. To achieve this, we introduce color state to represent the exposure and white balance state of a frame. The color state of each frame can be computed by accumulating the color transformations of neighboring frame pairs. Then, the tonal changes of the video can be represented by a time-varying trajectory in color state space. To remove the tone jitter, we smooth the original color state trajectory by solving an L1 optimization problem with PCA dimensionality reduction. In addition, we propose a novel selective strategy to remove small tone jitter while retaining extreme exposure and white balance changes to avoid serious artifacts. Quantitative evaluation and visual comparison with previous work demonstrate the effectiveness of our tonal stabilization method. This system can also be used as a preprocessing tool for other video editing methods. Index Terms— Tonal stabilization, color state, L1 optimization, selective strategy.

I. I NTRODUCTION

A

VIDEO captured with a hand-held device, such as a cell-phone or a portable camcorder, often suffers from undesirable exposure and white balance changes between successive frames. This is caused mainly by the continuous automatic exposure and white balance control of the device in response to illumination and content changes of the scene. We use “tone jitter” to describe these undesirable exposure and white balance changes. The first row of Fig. 1 shows an example of a video with tone jitter; it can be seen that some surfaces (e.g., leaves, chairs and glass windows) in frames extracted from the video have different exposures and white balances.

Manuscript received September 24, 2013; revised March 23, 2014 and July 23, 2014; accepted September 5, 2014. Date of publication September 17, 2014; date of current version September 30, 2014. This work was supported in part by the National Natural Science Foundation of China under Grant 61170142, in part by the Program of International Science and Technology Cooperation under Grant 2013DFG12840, National High Technology Research and Development Program of China (2013AA040601), and in part by the Australian Research Council under Project Grant FT-130101457 and Project DP-120103730. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Joseph P. Havlicek. (Corresponding author: Mingli Song.) Y. Wang, X. Li, M. Song, and J. Bu are with the College of Computer Science, Zhejiang University, Hangzhou 310027, China (e-mail: [email protected]). D. Tao is with the Centre for Quantum Computation and Intelligent Systems, Faculty of Engineering and Information Technology, University of Technology, Sydney, NSW 2007, Australia (e-mail: [email protected]). P. Tan is with the School of Computing Science, Simon Fraser University, Burnaby, BC V5A 1S6, Canada (e-mail: [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TIP.2014.2358880

It is of great importance to create a tonally stabilized video by removing tone jitter for online sharing or further processing. In this paper therefore, we address this video tonal stabilization problem to remove undesirable tone jitter in a video. Farbman and Lischinski [1] have proposed a method to stabilize the tone of a video. One or more anchors to the frames of the input video are first designated, and then an adjustment map is computed for each frame to make all frames appear to be filmed with the same exposure and white balance settings as the corresponding anchor. The adjustment map is propagated from one frame to its neighbor, based on an assumption that a large number of the pixel grid points from the two neighboring frames will sample the same scene surfaces. However, this assumption is usually erroneous, especially when the camera undergoes sudden motion or the scene has complex textures. In this case, a very small corresponding pixel set will be produced, and erratic color changes will occur in some regions of the final output. The performance of this method also depends on the anchor selection. Therefore, it is tedious for users to carefully examine the entire video and select several frames as anchors following strict rules. If we simply set one anchor to the middle frame, or two anchors to the first and last frames, the result video might suffer from over-exposure artifacts or contrast loss, especially in videos of a scene with high dynamic range. Exposure and white balance changes in an image sequence have been studied in panoramic image construction before the work of Farbman and Lischinski on tonal stabilization. To compensate for these changes, earlier approaches compute a linear model that matches the averages of each channel over the overlapping area in RGB [2] or YCbCr color spaces [3], while Zhang et al. [4] constructed a mapping function between the color histograms in the overlapping area. However, these models are not sufficiently accurate to represent tonal changes between frames and may result in unwanted compensation results. Other methods have been proposed to perform color correction using non-linear models, such as a polynomial mapping function [5] and linear correction for chrominance and gamma correction for luminance [6]. However, these models have the limitations of large accumulation errors and high computational complexities when adapted to video tonal stabilization. If the camera response function is known, the video tone can be stabilized by applying the camera response function inversely to each frame. Several attempts have been made to model the camera response function by utilizing

1057-7149 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

WANG et al.: VIDEO TONAL STABILIZATION VIA COLOR STATES SMOOTHING

4839

Fig. 1. Five still frames extracted from a video with unstable tone and the results of tonal stabilization. Top: the original frames. Middle: the result of removing all tone jitter. Bottom: the result using our tonal stabilization strategy.

a gamma curve [7], polynomial [8], semi-parametric [9] or PCA model [10]. However, most of these methods require perfect pixel-level image alignment, which is unrealistic in practice for amateur videos. The work proposed by Kim et al. [11], [12] jointly tracks features and estimates the radiometric response function of the camera as well as exposure differences between the frames. Grundmann et al. [13] employed the KLT feature tracker to find the pixel correspondences for alignment. After alignment, they locally computed the response curves for key frames and then interpolated these curves to generate the pixel-to-irradiance mapping. These two methods adjust all frames to have the same exposure and white balance according to the estimated response curves, without taking into account any changes in the illumination and content of the scene; this leads to artifacts of over-exposure, contrast loss or erratic color in the results. Color transfer is a topic that is highly related to this paper. It is possible to stabilize a video using a good color transformation model to make all frames have a tone similar to the selected reference frame. Typical global color transformation models are based on a Gaussian distribution [14], [15] or histogram matching [16]. An and Pellacini [17] proposed a joint model that utilizes an affine model for chrominance and a mapping curve for luminance. Chakrabarti et al. [18] extended the six-parameter color model introduced in [19] to three nonlinear models, independent exponentiation, independent polynomial and general polynomial, and proved that the general polynomial model has the smallest RMS errors. Local model-based methods [20]–[23] either segment the images and then compute a local model for each corresponding segment pair or estimate the local mapping between a small region of the source image and the target image and then propagate the adjustment to the whole image. While these global and local models are powerful for color transfer between a pair of images, stabilizing the tone of a video by using frame-to-frame color transfer is still impractical because of error accumulation. Furthermore, they cannot handle the large exposure and white balance changes contained in some video.

Commercial video editing tools, such as Adobe Premiere or After Effect, can be used to remove tone jitter. However, too many user interactions are required to manually select the key frames and edit their exposure and white balance. In summary, there are two major difficulties in stabilizing the tone of a video: • How to represent the tone jitter? A robust model is required to describe the tonal change between frames. Because the video contains camera motion and the exposure and white balance setting are not constant, it is very challenging to model the exposure and white balance change accurately. • How can the tone jitter be removed selectively? A good strategy should be proposed for tonal stabilization. It should be able to remove tonal jitter caused by imperfect automatic exposure and white balance control, while preserving necessary tonal changes due to illumination and content change of the scene. Videos captured in complex light conditions may have a wide exposure and color range, and neighboring frame pairs from such videos may exhibit very sharp color or exposure changes. Removing these sharp changes will produce artifacts of over-exposure, contrast loss or erratic colors (refer to the second row of Fig. 1). A perfect tonal stabilization strategy will eliminate small tone jitter while preserving sharp exposure and white balance changes, as in the result shown in the last row of Fig. 1. To overcome these two difficulties, a novel video tonal stabilization framework is proposed in this paper. We introduce a new concept of color state, which is a parametric representation of the exposure and white balance of a frame. The tone jitter can then be represented by the change of the color states between two successive frames. To remove the tone jitter, a smoothing technique is applied to the original color states to obtain the optimal color states. We then adjust each frame to its new color state and generate the final output video. In this way, our method stabilizes the tone of the input video and increases its visual quality. Additionally, the proposed method can also serve as a pre-processing step for other video processing and

4840

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 11, NOVEMBER 2014

Fig. 2. Flowchart of our tonal stabilization method. (a) The input frames. (b) The aligned frames. (c) The correspondence masks. (d) The original color states St . (e) Mt . (f) The new color states Pt . (g) The update matrices Bt . (h) The output frames.

computer vision applications, such as video segmentation [24], object tracking [25], etc. Inspired by the camera shake removal work of Grundmann et al. [26], in which the camera pose of each frame in an input video is first recovered and then smoothed to produce a stabilized result, our method further extends the framework and applies it to this new video tonal stabilization problem. Specifically, the contributions of our work are as follows: • We use color state, a parametric representation, to describe the exposure and white balance of an image. With this representation, the tone of a frame is described as a point in a high dimensional space, and then the video tonal stabilization problem can be modeled as a smoothing process of the original color states; • For the first time, we propose a selective strategy to remove undesirable tone jitter while preserving exposure and white balance changes due to sharp illumination and scene content changes. This strategy can help to avoid the artifacts of over-exposure, contrast loss or erratic color when processing videos with high dynamic ranges of tone; • To achieve tonal stabilization, we combine PCA dimensionality reduction with linear programming for color state smoothing. This not only significantly improves the stabilization results but also greatly reduces the computational cost. II. OVERVIEW In this paper, we use color state to represent the exposure and white balance of each frame in the input video. With this representation, the tonal changes between successive frames form a time-varying trajectory in the space of the color states. Undesirable tone jitter in the video can then be removed by adaptively smoothing the trajectory. Fig. 2 shows the flowchart of our method. We first conduct a spatial alignment and find the corresponding pixel pairs between successive frames. This helps us estimate the original

color states, denoted as S. The path of S is then smoothed by an L1 optimization with PCA dimensionality reduction to obtain the stabilized color states P. An update matrix Bt is then estimated, and by applying it, each frame t is transferred from the original state St to the new color state Pt to generate the final output video. We propose a selective strategy to implement video tonal stabilization. Because some videos have sharp exposure and white balance changes, transferring all of the frames to have the same tone will result in serious artifacts. Our goal is to keep the color states constant in the sections of the video with little tone jitter and give the color states a smooth transition between the sections with sharp tone changes. Thus, we adopt the idea in [26] to smooth the path of color states so that it contains three types of motion corresponding to different situations of exposure and white balance changes: • Static: A static path means the final color states stay unchanged, i.e., Dt1 (P) = 0, where Dtn (.) is the n-th derivative at t. • Constant speed: A constant rate of changes allows the tone of the video to change uniformly from one color state to another, i.e., Dt2 (P) = 0. • Constant acceleration: The segments of static and constant rate are both stable; constant acceleration in color state space is needed to connect two discrete stable segments. The transition from one stable segment with a constant acceleration to another segment will make the video tone change smoothly, i.e., Dt3 (P) = 0. To obtain the optimal path composed of distinct constant, linear and parabolic segments instead of a superposition of them, we use L1 optimization to minimize the derivatives of the color states. Our main reason for choosing L1 rather than L2 optimization is that the solution induced by the L1 cost function is sparse, i.e., it will attempt to satisfy many of the above motions along the path exactly. The computed path therefore has derivatives that are exactly zero for most segments, which is very suitable for our selective strategy. On the other hand, the L2 optimization will satisfy the above motions

WANG et al.: VIDEO TONAL STABILIZATION VIA COLOR STATES SMOOTHING

4841

on average (in the least-squares sense), which results in small but non-zero gradients. Qualitatively, the L2 optimized color state path always has some small non-zero motion (most likely in the direction of the original color state motion), while the L1 optimized path is composed only of segments resembling static, constant speed and constant acceleration. The rest of this paper is organized as follows. In Section III, we introduce a clear definition of the color state and show how to estimate it. A color state smoothing method is presented in Section IV to stabilize the path of color states. We show our experimental results in Section V and conclude the paper in Section VI. III. D EFINITION OF C OLOR S TATE A. Frame Color State In this paper, we use the term “color state” to represent the exposure and white balance of an image. Let St denote the color state of frame t of the video. The change to color state St from St −1 is considered to be the exposure and white balance changes between these two frames. We use the following affine transformation to model the color state change between two successive frames, ⎤ ⎡ a00 a01 a02 b0 ⎢a10 a11 a12 b1 ⎥ ⎥ (1) A=⎢ ⎣a20 a21 a22 b2 ⎦ . 0 0 0 1 An affine transformation includes a series of linear transformations, such as a translation, scaling, rotation or similarity transformation. These transformations can model the exposure and white balance changes well. An affine model has been successfully applied to user-controllable color transfer [17] and color correction [27]. In practice, although most cameras contain non-linear processing components, we find in our experiment that an affine model can approximate a non-linear transformation well and produce results with negligible errors. Given a pair of images I and J of different tones, a color j can be applied to transfer the transformation function A i pixels in I to have the same exposure and white balance as their corresponding pixels in J . Let x and x  denote a pair of corresponding pixels in I and J , respectively, and Ix = [IxR , IxG , IxB ]T and Jx  = [ JxR , JxG , JxB ]T represent the j (Ix ). colors of these two pixels. Then, Jx  = A i ⎡ ⎤ ⎡ ⎤⎡ R ⎤ ⎡ ⎤ R J  Ix a00 a01 a02 b0 ⎢ xG ⎥ ⎣ ⎢ G⎥ ⎣ ⎦ ⎦ (2) ⎣ Jx  ⎦ = a10 a11 a12 ⎣ Ix ⎦ + b1 a20 a21 a22 b2 J B IxB x

Note that the color transfer process in all our experiments is performed in the log domain due to the gamma correction of input videos. Let Att −1 denote the color transformation between frames t −1 and t. Then, the color state of frame t can be represented by St = Att −1 St −1 . The color transformation between S0 and St can be computed by accumulating all of the transformation matrices from frame 0 to frame t, i.e., St = Att −1 . . . A21 A10 S0 .

(3)

Fig. 3. Color transfer results by our affine model. (a) The source image I . (b) The target image J . (c) The aligned image. (d) The correspondence mask. The white pixels are the corresponding pixels used to estimate A, and the black ones are the expelled outliers. (e) The result without the constraint term. (f) The result with the identity constraint and using a uniform weight ωc = 2 × 103 .

Thus St is a 4 × 4 affine matrix and has 12 degrees of freedom. We can further set the color state of the first frame S0 to be identity matrix. Therefore, to compute the color state of each video frame, we need only to estimate the color transformation matrices for all neighboring frame pairs. In the next subsection, we will present how to estimate the color transformation model. B. Color Transformation Model Estimation For a common scene point projected in two different frames I and J at pixel locations x and x  , respectively, the colors of these two pixels Ix and Jx  should be the same. Therefore, j if the corresponding pixels can be found, the matrix Ai describing the color transformation between frames I and J can be estimated by minimizing j (Ix )2 . Jx  − A (4) i x

To estimate the color transformation matrix, we first need to find the pixel correspondences. However, we cannot use the sparse matching of local feature points directly to estimate the transformation because local feature points are usually located at corners, where the surface color is not well defined. The positive aspect is that local feature descriptors (such as SIFT [28] and LBP [29]) are robust to tone differences. Thus, we can use the sparse matching of local feature points to align frames. To achieve that, we track the local feature points using pyramidal Lucas-Kanade [30] and then compute a homography between two successive frames to align them. Fig. 3 shows a challenging alignment case in which (c) is the aligned result from (a) to (b). After alignment, two pixels having the same coordinates in the two images can be treated as a candidate corresponding pixel pair. To estimate the color transformation more accurately, we further process the candidate correspondence set to remove outliers. Firstly, the video frames may contain noise, which will affect the model computation. To avoid that, we employ a bilateral filter [31] to smooth the video frames. Secondly, the colors of pixels on edges are not well defined and cannot be

4842

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 11, NOVEMBER 2014

used to estimate the color transformation. Therefore, we conduct edge detection and exclude pixels around edges. Thirdly, because under- and over-exposure truncation may affect estimating the model, we discard all under- and over-exposed pixels from the candidate correspondence set. Fourthly, we adopt the RANSAC algorithm to further expel outliers during the model estimation, avoiding the effect caused mainly by noise and dynamic objects (such as moving persons, cars or trees, in the scene). Fig. 3(d) shows the correspondence mask. Note that in our implementation, frames are downsampled to a small size (shorter side of 180 pixels), so that the computational cost is reduced while an accurate color transformation can still be estimated. We notice that if we estimate the color transformation by directly minimizing Equation (4), the affine model tends to over-fit the data and accumulate errors at each stage, especially for scenes that contain large regions of similar color. To avoid the over-fitting problem, we can add a regularization term to Equation (4). Because the color tone of a natural video is unlikely to change much between two successive frames, the color transformation of two successive frames should be very close to an identity matrix. Based on this observation, the estimation of color transformation can be re-formulated as ωc j (Ix )2 + A j − I4×4 2 . Jx  − A (5) i i |X| x ωc Here, I4×4 is a 4 × 4 identity matrix. |X| is the weight used to combine the two terms, where |X| denotes the number of corresponding pixel pairs and ωc was set to 2 × 103 in our experiments. The identity constraint helps to choose the solution closer to the identity matrix only when getting several solutions with similar small errors by minimizing the first term of Equation (5), which reduces the over-fitting problem and improves the estimation accuracy. Taking the scene in Fig. 3 as an example, the estimated model without using the regularizer over-fits to the creamy-white pixels (table, bookshelf and wall) and causes large errors for the highlighted regions. To provide a numerical comparison, we placed a color check chart into this scene. The accuracy of color transfer is measured by an angular error [32] in RGB color space, which is the angle between the mean color of the pixels inside a color quad (c) in the transfer result  their mean (ˆc) in the target image,

cT cˆ and . The average angular errors of each e AN G = arccos c cˆ  color quad from (e) and (f) are 5.943◦ and 1.897◦, respectively. The quad in the color of Red (Row 3, Col 3) in (e) has the highest error, 22.03◦. For comparison, the highest error of the color check chart in (f) is 4.476◦ from Light Blue (Row 3, Col 6). Note that the color check chart is not used to aid model estimation.

IV. C OLOR S TATES S MOOTHING To remove tone jitter, we can transfer all video frames to have a constant color state. However, as shown in Section I, large tone changes need to be retained in videos that contain large illumination or scene content changes; otherwise, artifacts may arise after removing tone jitter, such

as over-exposure, contrast loss or erratic color. In this paper, we propose a selective strategy for video tonal stabilization. Under this selective strategy, we generate a sequence of frames with smooth color states and meanwhile constrain the new color states close to the original. Here, ‘smooth’ means that the color states remain static or are in uniform or uniformly accelerated motion. We utilize an L1 optimization to find the smooth color states by minimizing the energy function, E = ωs E(P, S) + E(P).

(6)

E(P, S) reflects our selective strategy, which is a soft constraint and ensures that the new color states P do not deviate from the original states S, E(P, S) = |P − S|1 .

(7)

E(P) smoothes the frame color states. As mentioned in Section II, the first, second and third derivatives of Pt should be minimized to make the path of Pt consist of segments of static, constant speed or constant acceleration motion. E(P) = ω1 |D 1 (P)|1 + ω2 |D 2 (P)|1 + ω3 |D 3 (P)|1 ,

(8)

where D n (P) represents the n-th derivative of the new color states P. Minimizing |D 1 (P)|1 causes the color states to tend to be static. Likewise, |D 2 (P)|1 constrains the color states to uniform motion, and |D 3 (P)|1 is relative with the acceleration of the color state motion. The weights ω1 , ω2 and ω3 balance these three derivatives. In our experiments, ω1 , ω2 and ω3 were set to 50, 1 and 100, respectively. The weight ωs combining the two terms is the key parameter in our method. It makes the new color states either have an ideal smooth path or remain very close to the original states. We conducted many experiments to analyze ωs and found that [0.01, 0.3] is the tunable range of ωs . When ωs = 0.01, the new color states remain constant. In contrast, if ωs = 0.3, the new paths of color states retain part of the initial motion. A detailed discussion of parameter setting is presented in Section V. Here we discuss the smoothness of color states in different frames. We can optimize Pt similarly to the L1 camera shake removal method [26], using forward differencing to derive |D n (P)|1 and minimizing their residuals. However, this method has limitations. From its optimization objective, the 12 elements of color state are smoothed independently, whose result is that the 12 elements do not change synchronously. Fig. 4 shows the curves of the new states generated by an L1 optimization-based method as in [26]. To relieve this problem in removing camera shake, an inclusion constraint is utilized in [26] that the four corners of the crop rectangle must reside inside the frame rectangle. However, we cannot find the corners or boundaries of color states. So optimizing the new color states by the L1 optimization directly will result in some new color states being outside the clusters of the original color states, and the corresponding output frames will have erratic colors (shown in the middle row of Fig. 5). We therefore seek to improve the smoothing method so that all 12 elements of color state can change in the same way. The path of original color states is a curve in a 12D space; if we

WANG et al.: VIDEO TONAL STABILIZATION VIA COLOR STATES SMOOTHING

4843

Algorithm 1 LP for Color States Optimization

Fig. 4. The curves of the 12 color state elements before and after the L1 optimization without PCA. Green curves: original color state elements. Red curves: new color state elements. From top to bottom, left to right, each curve corresponds to an element of color state. The vertical axis plots the value of the corresponding element, and the horizontal axis plots the frame index.

Fig. 6. An example of the paths of original color states (Red curve) and the linear subspace (Green curve) generated by PCA. Note that the plotted dots are not the real color states but simulation values. We choose a point in the scene and use the color curve over time to simulate the color state path. Fig. 5. The comparison of stabilization results generated by the L1 optimization methods without and with PCA. Top: the input video. Middle: the result generated by the L1 optimization without PCA, which contains erratic color in some output frames. Bottom: the result using the L1 optimization with PCA.

can constrain all of the new color states to be along a straight line, the above problem will be solved. We employ Principal Component Analysis (PCA) [33], [34] to find a linear subspace near the original color state path. Using PCA, the color states ci ci ¯ can be represented by St = S¯ + i βt S , where S denotes ci c i the mean color state over t, and S and βt denote the eigenvector and eigenvalue of the i -th component, respectively. S¯ and S ci are 4 × 4 matrices, and βtci is a scalar. The mean color state S¯ and the eigenvector of the first component S c1 are used to build this linear subspace, and the new color states are encoded as Pt = S¯ + Mt S c1 .

(9)

Here, Mt denotes the new coefficient, which is a scalar. Because S c1 is the first principle component corresponding to the largest eigenvalue βtc1 , the line of the new color states will not deviate much from the original color state path, as shown in Fig. 6. In this way, we limit the degrees of freedom of the solution to a first order approximation. Then, in the smoothing method we only need to take into account minimizing the magnitude of the velocity and acceleration of the color state path in the L1 optimization and do not need consider their direction changes. Our method is different from the dimensionality-reduction-based smoothing methods that directly smooth the coefficients of the first or several major

components with a low-pass filter; we will find a smoothly changing function (a smooth curve) subject to t, and then all 12 elements of the new color states will have the same motion as the curve of Mt . After PCA, our L1 optimization objective is re-derived and minimized based on Equation (9). Minimizing E(P, S): The new formulation of Pt is substituted into Equation (7),

S¯ + Mt S c1 − St . (10) E(P, S) = 1 t

Minimizing E(P): Forward differencing is used to compute the derivatives of P. Then |D 1 (P)| = |Pt +1 − Pt | t

=



|( S¯ + Mt +1 S c1 ) − ( S¯ + Mt S c1 )|

t

≤



|S c1 ||Mt +1 − Mt |.

t

Because |S c1 | is known, we only need minimize |Mt +1 − Mt |, i.e., |D 1 (M)|. Similarly, we can prove that |D 2 (P)| and |D 3 (P)| are equivalent to |D 2 (M)| and |D 3 (M)|, respectively. Then, our goal is to find a curve Mt such that (1) it changes smoothly and (2) after mapping along this curve, the new color states are close to the original states. Algorithm 1 summarizes the entire process of our L1 optimization. To minimize Equation (6), we introduce non-negative slack variables to bound each dimension of the color state derivatives and solve a linear programming problem as described in [26]. Using Mt

4844

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 11, NOVEMBER 2014

Fig. 7. The curves of the 12 color state elements before and after our L1 optimization with PCA. Green curves: original color state elements. Red curves: new color state elements.

to represent the color states, the number of unknown variables to be estimated for each frame becomes 1, and the number of introduced slack variables also declines substantially. This significantly reduces the time and space costs of the linear programming. In our implementation, we employed COIN CLP1 to minimize the energy function and generate a sequence of stable states. The curves of our optimal color states and the original color states are shown in Fig. 7. In contrast to the result from the L1 optimization without PCA, all 12 elements of color state obey the same motion. The last row of Fig. 5 shows examples of the output frames generated by our L1 optimization with PCA; the problem of unusual color has been avoided. After obtaining the new color states, an update matrix Bt is calculated to transfer each frame from the original color state to the new color state. From the definition of color state, Pt = Bt St . The update matrix Bt can then be computed as Bt = Pt St−1 .

(11)

It is applied to the original frame t pixel by pixel to generate the new frame. V. E XPERIMENTS AND E VALUATION

Fig. 8. The curves of Mt with different ωs for the video in Fig. 1. c Red curves: Mt . Green curves: βt 1 . ω1 = 50, ω2 = 1 and ω3 = 100. (a) ωs = 0.01. (b) ωs = 0.1. (c) ωs = 0.3. (d) ωs = 1.0.

Fig. 9. The curves of Mt with different ω1 , ω2 and ω3 for the video in c Fig. 1. Red curves: Mt . Green curves: βt 1 . In this experiment, ωs = 0.3. (a) ω1 = 100, ω2 = ω3 = 1. (b) ω2 = 100, ω1 = ω3 = 1. (b) ω3 = 100, ω1 = ω2 = 1. (d) ω1 = 50, ω2 = 1, ω3 = 100.

The weight that balances the constraint and smoothness terms is the most important parameter in our system. As this weight is varied, the system generates different results. If ωs is small (such as 0.01), Mt tends to be a straight line, which maps all of the frames to the same color state. This keeps the exposure and white balance unchanged always. For videos whose exposure and color ranges are too wide, a straight line mapping will cause some frames to have artifacts, such as over-exposure, contrast loss or erratic color. If ωs is large (such as 1.0), Pt will be very close to St . Then, most color state changes will be retained. The weight ωs balances these two aspects, and it is difficult to find a general value suitable for all types of videos. We leave this parameter to be tuned by users. We suggest that users tune ωs within three levels, 0.01, 0.1 and 0.3, which were widely used in our experiments. Variant weights were tried to stabilize the video in Fig. 1;

the curves of Mt are shown in Fig. 8. The comparison of the output videos with different parameters is shown in our supplementary material.2 Three other parameters affecting color state trajectory are ω1 , ω2 and ω3 . We explored different weights for stabilizing the video in Fig. 1 and plotted the curves of Mt in Fig. 9. If we set one of the three parameters to a large value but depress the other two, it is apparent that the new color state path tends to be (a) constant non-continuous, (b) linear with sudden changes or (c) a smooth parabola, but it is rarely static. A more agreeable viewing experience is produced by setting ω1 and ω3 larger than ω2 because we hope that the optimal path can sustain longer static segments and be absolutely smooth. For this paper, we set ω1 = 50, ω2 = 1 and ω3 = 100; the corresponding curve of Mt is shown in Fig. 9(d). In practical situations, users may prefer the exposure and white balance of some frames and hope to keep the tone of these frames unchanged. Our system can provide this function

1 COIN CLP is an Open Source linear programming solver that can be freely downloaded from http://www.coin-or.org/download/source/Clp/.

2 The supplementary material can be found in our project page, http://eagle.zju.edu.cn/~wangyinting/TonalStabilization/.

A. Parameter Setting

WANG et al.: VIDEO TONAL STABILIZATION VIA COLOR STATES SMOOTHING

Fig. 10. The stabilization results generated by keeping the exposure and white balance of the ‘preferred frame’ unchanged. Top: the original video. Middle: the stabilization result with the first frames fixed. Bottom: the stabilization result with the last frame fixed. ωs,t is set to 100 for the preferred frame and 0.01 for the others.

by using a non-uniform weight ωs,t instead of ωs . We ask the users to point out one or several frames as ‘preferred frames’ and set higher ωs,t for these frames. The weights ωs,t for the other frames are chosen by our selective strategy. Then, the new optimal color states for the ‘preferred frames’ will be very close to the original ones. Fig. 10 gives an example in which the second row is the result generated by setting the first frame as the ‘preferred frame’, and the third row is the result with the last frame fixed. In this example, we set ωs,t equal to 100 for the ‘preferred frame’ and 0.01 for the others. In a similar way, the weight ωc in Equation (5) can be changed to a spatial-variant one, ωc,x , and then the estimated affine model would be more accurate for the pixels with larger weights. We can extract the salient regions [35], [36] of each frame or ask the users to mark their ‘preferred regions’ and track these regions in each frame. In this way, our system will generate more satisfying results for the regions to which users pay more attention. A numeral comparison experiment is described in the next subsection. B. Quantitative Evaluation To illustrate the effectiveness of our tonal stabilization method, we employed a low reflectance grey card to do quantitative evaluation as [1]. We placed an 18% reflectance grey card into a simple indoor scene and recorded a video with obvious tone jitter with an iPhone 4S; five example frames from the video are shown in the first row of Fig. 11. This video was then processed by our tonal stabilization method.

4845

Note that the grey card was not used to aid and improve the stabilization. We compared our results with Farbman and Lischinski’s pixel-wise color alignment method (PWCA) [1]. For PWCA, we set the first frame as the anchor. To reach a similar processing result, we set ωs,t to 100 for the first frame and 0.01 for the other frames, so that the exposure and white balance of the first frame was fixed and propagated to the others. Both uniform weight ωc and non-uniform weight ωc,x were tried in this experiment. For uniform weight, we set ωc = 2 × 103 . For non-uniform weight, we set ωc,x to 104 for all of the corresponding pixel pairs inside the grey card and 2 × 103 for the others. We measured the angular error [32] in RGB color space between the mean color of the pixels inside the grey card of each frame and the first frame. The plot in Fig. 11(a) shows the angular errors from the first frame of the original video (Red curve), the video stabilized by PWCA (Blue curve), the results generated by our method with uniform weight (Green curve) and non-uniform weight (Dark Green curve). The second column of Table I is the average errors over each frame of the original video and the results generated by PWCA and our method. Both PWCA and our method performed well, and our method with non-uniform weight came out slightly ahead. To assess the benefits of tonal stabilization for white balance estimation, we conducted a similar experiment to that presented in [1]; we applied a Grey-Edge family of algorithms [37] to a video and its stabilization results and compared the performance of white balance estimation. The two white balance estimation methods chosen assume that some properties of the scene, such as average reflectance (GreyWorld) or average derivative (Grey-Edge), are achromatic. We computed the angular error [32] of the mean color inside the grey card of all frames to the ground truth; the plots of the estimation error are shown in Figs. 11(b) and (c). The third and fourth columns of Table I are the average angular errors of each frame after white balance estimation by Grey-World and Grey-Edge, respectively. The grey card restricts that the camera motion during video shooting should not be large and the camera should not be very far from the scene; thus, the video used for evaluation will be relatively simple. On the other hand, PWCA performs extremely well for large homogeneous regions (grey card). These two factors are the reasons why our method only led PWCA a little in the quantitative evaluations. C. Comparison From the discussion above, we find that both PWCA and our method can generate good results for simple videos. However, PWCA sometimes does not work well for videos that include scenes with complex textures or which have sudden camera shaking. For these cases, it produces a very small robust set and results in a final output that is not absolutely stable. Our method aligns the successive frames and detects the corresponding pixels in a more robust way. Fig. 12 compares the result of PWCA with an anchor set to the central frame and our result with ωs = 0.01. We can see that our result is more stable. Another advantage of our method is the selective

4846

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 11, NOVEMBER 2014

Fig. 11. Quantitative evaluation of tonal stabilization and consequent white balance correction. The first row is several still frames from the input video. The second row compares the numeral errors of the original video and the results generated by PWCA and our method. (a) The angular errors in RGB color space of average color within the grey card of each frame with respect to the first frame. (b) and (c) The angular errors from the ground truth after white balance estimation by grey-world and grey-edge. Red curves: original video. Blue curves: the result of PWCA. Green curves: the result of our method with uniform ωc = 2 × 103 . Dark Green curves: the result of our method with non-uniform ωc,x , 104 for the pixels within the grey card and 2 × 103 for the other pixels. TABLE I M EAN A NGULAR E RRORS OF E ACH F RAME . T HE S ECOND C OLUMN I S THE E RRORS W ITH R ESPECT TO THE F IRST F RAME OF THE O RIGINAL V IDEO AND I TS

S TABILIZATION R ESULTS . T HE T HIRD AND F OURTH C OLUMNS A RE THE E RRORS F ROM THE G ROUND T RUTH A FTER W HITE BALANCE E STIMATION BY G REY-W ORLD AND G REY-E DGE , R ESPECTIVELY

Fig. 12. Comparison of the stabilization results generated by PWCA and our method. Top: the original video. Middle: the result of PWCA, which still contains tone jitter. Bottom: the result of our method.

stabilization strategy. It allows us to adaptively smooth the path of color states according to the original color states. Users need only tune the parameter ωs to generate results with different degrees of stability and choose the best of them. Even for a video with a large dynamic range, our method is still very convenient to use. Benefitting from the selective strategy, the stabilized result will remove the small tone jitter and retain the sharp tone changes. In contrast, to generate comparable result, PWCA requires that the user choose the anchors carefully. If one or two anchors are set automatically, the result may include artifacts. The second row of Fig. 13 is the result of PWCA with two anchors set to the first and last frames. It is clear that some frames of the output video are over-exposed.

Fig. 13. Another comparison of PWCA and our method. The right two frames of the result generated by PWCA are over-exposed.

D. Challenging Cases A video containing noise or dynamic objects is very challenging for affine model estimation because of the outliers from the noise and dynamic objects. Our robust correspondence selection helps to handle these challenging cases by discarding most of the outliers. Figs. 14 and 15 show two examples of these challenging cases and their stabilization results from our method with a uniform ωc (2 × 103 ) and a small ωs (0.01). Because noise and dynamic objects do not affect PWCA, we choose its stabilization results as borderline; they are shown in the second row of each figure. These two examples demonstrate that our method can perform well for videos containing noise and dynamic objects, and our results are even a little better than those generated

WANG et al.: VIDEO TONAL STABILIZATION VIA COLOR STATES SMOOTHING

4847

Fig. 16. Failure case. These four frames are extracted from our stabilization result. The stone appears in two different sections of the video, and the colors in our result are not coherent.

we plan to accelerate the performance of our method by using GPUs. F. Discussion Fig. 14. A video containing dynamic object and its stabilization results. Top: the original frames from the video. Middle: the result of PWCA. Some tone jitter can be found in the 3rd and 4th columns. Bottom: the result of our method.

Fig. 15. A noisy video and its stabilization results with PWCA and our method. The exposure and color of the 2nd column is a little different from the other columns in the PWCA result.

by PWCA for exposure and color consistency (refer to the third and fourth columns of Fig. 14 and the second column of Fig. 15). E. Computational Overhead The running time of our system depends on the length of the input video. When we compute the transformation matrix for two neighboring frames, the images are resized to a fixed small size, so the size of frame does not significantly affect the running time. For the 540-frame video (1920 × 1080 in size) shown in Fig. 1, it took about 511 seconds to complete the entire stabilization process. Computing the color transformation matrix for each pair of successive frames took approximately 0.88 seconds, and the system spent 7.28 seconds on optimizing the color states. Our system is implemented in C++ without code optimization, running on a Dell Vostro 420 PC with an Intel Core2 Quad 2.40GHz CPU and 8GB of memory. The running time of our method can be shortened by parallelization. Approximately 90% of running time is spent on computing the color transformation matrix between neighboring frames, which is easily parallelizable because the frame registration and affine model estimation for each pair of neighboring frames can be carried out independently. On the other hand, when processing a long video, we can first cut the video into several short sub-videos and set the frame that contacts two sub-videos as a ‘preferred frame’; these subvideos can then be stabilized synchronously. In future work

Our method depends on alignment of consecutive frames, so feature tracking failures will affect our stabilization results. There are several situations that may cause feature tracking failures. a) The neighboring frames are very homogeneous (e.g., wall, sky) and have too few matching feature points. In this situation, we assume these two source frames are aligned. Because the frames are homogeneous, this will not result in large errors from misalignment; our correspondence selection algorithm helps to discard the outliers from noise, edges, etc. Therefore, our method also performs well on this situation. b) Very sharp brightness changes between consecutive frames may cause feature tracking to fail. We can adopt an iteration-based method to solve this problem. The two neighboring frames are denoted as I and J . We first assume j the two images are aligned, i.e., Hi = I3×3 . We estimate the j color transfer model Ai that is applied to frame I to make the exposure and white balance of I and J closer. Then, a new j homography matrix Hi is estimated to align the modified j j frame I and J . We repeat to compute Ai and Hi ; usually two or three iterations are sufficient to generate a good result. Most of the neighboring frames in natural videos will not have brightness changes sharp enough to affect feature tracking. We tested approximately 150 videos and never encountered this problem. c) Most local features are located at non-rigid moving objects. This is the most challenging case, and a vast majority of feature-tracking-based vision algorithms cannot handle it, such as [38]. Because our method needs to track a feature for only two adjacent frames, if the dynamic objects do not move too quickly, serious artifacts will not result. Otherwise, our method will fail. In addition to feature tracking, our method has another limitation. If the camera is moved from one scene to a new scene and then returned to the former, the color of the same surface in different sections of a video stabilized by our method may be not coherent. An example is shown in Fig. 16, in which a large stone appears, is passed by, and then reappears. We can see from the figure that the color of the stone has changed slightly in our output frames. This is caused by the error arising during color state computation. When we estimate the color transformation model for two neighboring images, if a surface with a particular color exists only in one frame, then the computed model may be unsuitable for the region of this surface. Because the color state is the accumulation of color transformation matrices, the error of color transformation for two frames will be propagated to all later images. Another possible reason for this artifact is that our trajectory smoothing method cannot ensure that

4848

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 11, NOVEMBER 2014

two similar original color states remain similar after stabilization. We leave these unsolved problems for future work. VI. C ONCLUSION In this paper, we utilize a 4 × 4 matrix to model the exposure and white balance state of a video frame, which we refer to as the color state. PCA dimensionality reduction is then applied to find a linear subspace of color states, and an L1 optimization-based method is proposed to generate smooth color states in the linear subspace and achieve the goal of video tonal stabilization. Our experimental results and quantitative evaluation show the effectiveness of our stabilization method. Compared to previous work, our method performs better in looking for pixel correspondences in two neighboring frames. In addition, we use a new stabilization strategy that retains some tone changes due to sharp illumination and scene content changes, so our method can handle videos with an extreme dynamic range of exposure and color. Our system can remove tone jitter effectively and thus increase the visual quality of amateur videos. It also can be used as a pre-processing tool for other video editing methods. R EFERENCES [1] Z. Farbman and D. Lischinski, “Tonal stabilization of video,” ACM Trans. Graph., vol. 30, no. 4, pp. 1–89, 2011. [2] G. Y. Tian, D. Gledhill, D. Taylor, and D. Clarke, “Colour correction for panoramic imaging,” in Proc. 6th Int. Conf. Inf. Visualisat., 2002, pp. 483–488. [3] S. J. Ha, H. I. Koo, S. H. Lee, N. I. Cho, and S. K. Kim, “Panorama mosaic optimization for mobile camera systems,” IEEE Trans. Consum. Electron., vol. 53, no. 4, pp. 1217–1225, Nov. 2007. [4] Z. Maojun, X. Jingni, L. Yunhao, and W. Defeng, “Color histogram correction for panoramic images,” in Proc. 7th Int. Conf. Virtual Syst. Multimedia, Oct. 2001, pp. 328–331. [5] B. Pham and G. Pringle, “Color correction for an image sequence,” IEEE Comput. Graph. Appl., vol. 15, no. 3, pp. 38–42, May 1995. [6] Y. Xiong and K. Pulli, “Color matching of image sequences with combined gamma and linear corrections,” in Proc. Int. Conf. ACM Multimedia, 2010, pp. 261–270. [7] S. Mann and R. W. Picard, “On being ‘undigital’ with digital cameras: Extending dynamic range by combining differently exposed pictures,” in Proc. IS&T, 1995, pp. 442–448. [8] T. Mitsunaga and S. K. Nayar, “Radiometric self calibration,” in Proc. Comput. Vis. Pattern Recognit., vol. 1. Jun. 1999, pp. 374–380. [9] F. M. Candocia and D. A. Mandarino, “A semiparametric model for accurate camera response function modeling and exposure estimation from comparametric data,” IEEE Trans. Image Process., vol. 14, no. 8, pp. 1138–1150, Aug. 2005. [10] M. D. Grossberg and S. K. Nayar, “Modeling the space of camera response functions,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 10, pp. 1272–1282, Oct. 2004. [11] S. J. Kim, J.-M. Frahm, and M. Pollefeys, “Joint feature tracking and radiometric calibration from auto-exposure video,” in Proc. IEEE 11th Int. Conf. Comput. Vis., Oct. 2007, pp. 1–8. [12] S. J. Kim and M. Pollefeys, “Robust radiometric calibration and vignetting correction,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 4, pp. 562–576, Apr. 2008. [13] M. Grundmann, C. McClanahan, S. B. Kang, and I. Essa, “Postprocessing approach for radiometric self-calibration of video,” in Proc. IEEE Int. Conf. Comput. Photography, Apr. 2013, pp. 1–9. [14] E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley, “Color transfer between images,” IEEE Comput. Graph. Appl., vol. 21, no. 5, pp. 34–41, Sep./Oct. 2001. [15] F. Pitié, A. C. Kokaram, and R. Dahyot, “Automated colour grading using colour distribution transfer,” Comput. Vis. Image Und., vol. 107, nos. 1–2, pp. 123–137, Jul. 2007. [16] T. Oskam, A. Hornung, R. W. Sumner, and M. Gross, “Fast and stable color balancing for images and augmented reality,” in Proc. 2nd Int. Conf. 3D Imag., Modeling, Process., Visualizat. Transmiss., Oct. 2012, pp. 49–56.

[17] X. An and F. Pellacini, “User-controllable color transfer,” Comput. Graph. Forum, vol. 29, no. 2, pp. 263–271, May 2010. [18] A. Chakrabarti, D. Scharstein, and T. Zickler, “An empirical camera model for internet color vision,” in Proc. Brit. Mach. Vis. Conf., vol. 1. 2009, pp. 1–4. [19] G. Finlayson and R. Xu, “Illuminant and gamma comprehensive normalisation in log RGB space,” Pattern Recognit. Lett., vol. 24, no. 11, pp. 1679–1690, Jul. 2003. [20] S. Kagarlitsky, Y. Moses, and Y. Hel-Or, “Piecewise-consistent color mappings of images acquired under various conditions,” in Proc. IEEE 12th Int. Conf. Comput. Vis., Sep./Oct. 2009, pp. 2311–2318. [21] Y.-W. Tai, J. Jia, and C.-K. Tang, “Local color transfer via probabilistic segmentation by expectation-maximization,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 1. Jun. 2005, pp. 747–754. [22] D. Lischinski, Z. Farbman, M. Uyttendaele, and R. Szeliski, “Interactive local adjustment of tonal values,” ACM Trans. Graph., vol. 25, no. 3, pp. 646–653, Jul. 2006. [23] Y. Li, E. Adelson, and A. Agarwala, “ScribbleBoost: Adding classification to edge-aware interpolation of local image and video adjustments,” Comput. Graph. Forum, vol. 27, no. 4, pp. 1255–1264, 2008. [24] Q. Zhu, Z. Song, Y. Xie, and L. Wang, “A novel recursive Bayesian learning-based method for the efficient and accurate segmentation of video with dynamic background,” IEEE Trans. Image Process., vol. 21, no. 9, pp. 3865–3876, Sep. 2012. [25] S. Das, A. Kale, and N. Vaswani, “Particle filter with a mode tracker for visual tracking across illumination changes,” IEEE Trans. Image Process., vol. 21, no. 4, pp. 2340–2346, Apr. 2012. [26] M. Grundmann, V. Kwatra, and I. Essa, “Auto-directed video stabilization with robust L1 optimal camera paths,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 225–232. [27] H. Siddiqui and C. A. Bouman, “Hierarchical color correction for camera cell phone images,” IEEE Trans. Image Process., vol. 17, no. 11, pp. 2138–2155, Nov. 2008. [28] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, 2004. [29] T. Ojala, M. Pietikainen, and T. Maenpaa, “Multiresolution gray-scale and rotation invariant texture classification with local binary patterns,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 971–987, Jul. 2002. [30] J. Shi and C. Tomasi, “Good features to track,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., Jun. 1994, pp. 593–600. [31] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Proc. 6th Int. Conf. Comput. Vis., Jan. 1998, pp. 839–846. [32] S. D. Hordley, “Scene illuminant estimation: Past, present, and future,” Color Res. Appl., vol. 31, no. 4, pp. 303–314, Aug. 2006. [33] I. T. Jolliffe, Principal Component Analysis. New York, NY, USA: Springer-Verlag, 1986, p. 487. [34] B.-K. Bao, G. Liu, C. Xu, and S. Yan, “Inductive robust principal component analysis,” IEEE Trans. Image Process., vol. 21, no. 8, pp. 3794–3800, Aug. 2012. [35] M.-M. Cheng, G.-X. Zhang, N. J. Mitra, X. Huang, and S.-M. Hu, “Global contrast based salient region detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 409–416. [36] C. Jung and C. Kim, “A unified spectral-domain approach for saliency detection and its application to automatic object segmentation,” IEEE Trans. Image Process., vol. 21, no. 3, pp. 1272–1283, Mar. 2012. [37] J. van de Weijer, T. Gevers, and A. Gijsenij, “Edge-based color constancy,” IEEE Trans. Image Process., vol. 16, no. 9, pp. 2207–2214, Sep. 2007. [38] F. Liu, M. Gleicher, J. Wang, H. Jin, and A. Agarwala, “Subspace video stabilization,” ACM Trans. Graph., vol. 30, no. 1, pp. 1–4, 2011.

Yinting Wang received the B.E. degree in software engineering from Zhejiang University, Hangzhou, China, in 2008, where he is currently pursuing the Ph.D. degree in computer science. His research interests include computer vision and image/video enhancement.

WANG et al.: VIDEO TONAL STABILIZATION VIA COLOR STATES SMOOTHING

Dacheng Tao (M’07–SM’12) is currently a Professor of Computer Science with the Centre for Quantum Computation and Intelligent Systems, and the Faculty of Engineering and Information Technology, University of Technology, Sydney, NSW, Australia. He mainly applies statistics and mathematics for data analysis problems in data mining, computer vision, machine learning, multimedia, and video surveillance. He has authored over 100 scientific articles at top venues, including the IEEE IEEE T RANSACTIONS ON PATTERN A NALYSIS AND M ACHINE I NTELLIGENCE, the IEEE T RANSACTIONS ON N EURAL N ETWORKS AND L EARNING S YSTEMS , the IEEE T RANSACTIONS ON I MAGE P ROCESSING , the IEEE Conference on Neural Information Processing Systems, the International Conference on Machine Learning, the International Conference on Artificial Intelligence and Statistics, the IEEE International Conference on Data Mining series (ICDM), the IEEE Conference on Computer Vision and Pattern Recognition, the International Conference on Computer Vision, the European Conference on Computer Vision, the ACM Transactions on Knowledge Discovery from Data, the ACM Multimedia Conference, and the ACM Conference on Knowledge Discovery and Data Mining, with the Best Theory/Algorithm Paper Runner Up Award in IEEE ICDM’07 and the Best Student Paper Award in IEEE ICDM’13.

Xiang Li received the B.E. degree in computer Science from Zhejiang University, Hangzhou, China, in 2013. He is currently pursuing the M.S. degree in information technology with Carnegie Mellon University, Pittsburgh, PA, USA. His research interests include machine learning and computer vision.

4849

Mingli Song (M’06–SM’13) is currently an Associate Professor with the Microsoft Visual Perception Laboratory, Zhejiang University, Hangzhou, China. He received the Ph.D. degree in computer science from Zhejiang University in 2006. He was a recipient of the Microsoft Research Fellowship in 2004. His research interests include face modeling and facial expression analysis.

Jiajun Bu is currently a Professor with the College of Computer Science, Zhejiang University, Hangzhou, China. His research interests include computer vision, computer graphics, and embedded technology.

Ping Tan is currently an Assistant Professor with the School of Computing Science, Simon Fraser University, Burnaby, BC, Canada. He was an Associate Professor with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore. He received the Ph.D. degree in computer science and engineering from the Hong Kong University of Science and Technology, Hong Kong, in 2007, and the bachelor’s and master’s degrees from Shanghai Jiao Tong University, Shanghai, China, in 2000 and 2003, respectively. He has served as an Editorial Board Member of the International Journal of Computer Vision and the Machine Vision and Applications. He has served on the Program Committees of SIGGRAPH and SIGGRAPH Asia. He was a recipient of the inaugural MIT TR35@Singapore Award in 2012, and the Image and Vision Computing Outstanding Young Researcher Award and the Honorable Mention Award in 2012.