Article

Moving Object Detection on a Vehicle Mounted Back-Up Camera Dong-Sun Kim * and Jinsan Kwon Received: 12 October 2015; Accepted: 22 December 2015; Published: 25 December 2015 Academic Editor: Felipe Jimenez Embedded Software Convergence Research Center, Korea Electronics Technology Institute, Saenari-ro 25, Bundang-gu, Seongnam-si, Gyeonggi-do 13509, Korea; [email protected] * Correspondence: [email protected]; Tel./Fax: +82-31-789-7598

Abstract: In the detection of moving objects from vision sources one usually assumes that the scene has been captured by stationary cameras. In case of backing up a vehicle, however, the camera mounted on the vehicle moves according to the vehicle’s movement, resulting in ego-motions on the background. This results in mixed motion in the scene, and makes it difficult to distinguish between the target objects and background motions. Without further treatments on the mixed motion, traditional fixed-viewpoint object detection methods will lead to many false-positive detection results. In this paper, we suggest a procedure to be used with the traditional moving object detection methods relaxing the stationary cameras restriction, by introducing additional steps before and after the detection. We also decribe the implementation as a FPGA platform along with the algorithm. The target application of this suggestion is use with a road vehicle’s rear-view camera systems. Keywords: moving object detection; mixture of Gaussians; pyramidal Lucas-Kanade optical flow; backup collision intervention

1. Introduction Improvements to the function, reliability, and manufacturing process of various sensors making them small, sensitive, and yet strong enough, have made it possible to use those sensors in automobiles [1–4]. One of its major applications is Backup Collision Intervention (BCI), which informs drivers of obstacles behind the vehicle by giving visual, aural, or tactile feedback. So far this technology has utilized laser, ultrasonic, microwave radar, and vision sensors to measure the distance from the vehicle to obstacles. Although they differs by specific application, those on-board sensors work as the main components for the BCI function independently or in conjunction with others. Drivers of vehicles with extended bodies or carrying oversized loads may have difficulty in seeing behind them directly through the rear-view mirror. As a part of BCI components, on-board monitors connected to a camera mounted on the back of the vehicle provide a visual aid to its user, helping the driver avoid obstacles behind while backing-up. In addition to covering the blind spots, further processing of the video feeds may provide more features such as moving object detection. The processed information then can be integrated back to the original video feed. Although many other sensors detect stationary obstacles well based on the distance between objects and the vehicle, we focus on moving objects that are beyond the active sensor’s range that might still require the user’s attention while backing up. As a practical approach, we do not assume the obstacles to be of any specific type, therefore, in our suggested method the target objects can range from pedestrians to other vehicles. If available, the detection results may also further processed in combination with the results from other sensors to provide on the screen information in integrated form such as distance, speed, and direction.

Sensors 2016, 16, 23; doi:10.3390/s16010023

www.mdpi.com/journal/sensors

Sensors 2016, 16, 23

2 of 10

Meanwhile, vision-based moving object detection is a major theme in the computer vision arena, and vast number of suggestions has been made. However, many of them assume its viewpoint to be stationary [5–8], because the movement introduced to its vision source is also reflected on the background of the taken scene. Without proper distinction, the mixed motion between the background and foreground object is hard to separate out. Since our target application is a vision-based BCI system mounted on the back of a vehicle, movement of the viewpoint is inevitable. Therefore, in our suggestion, we first compensate for the background movement, called ego-motion, for later processing using the traditional object detection methods. This is possible because of the unique characteristics of the slow-moving back-up movement, which introduces mild motions across the scene. The uniform vectors found all over the frame then be extracted, and based on those vectors, a rough compensation of the background will be made. The actual object detection routine is then performed using the difference between a frame and the compensated background frame at the time. This paper first examines the previous methods for object detection. Details about the suggested procedure then follow, and we discuss about the implementation of the suggestion as a hardware platform. 2. Background Works In computer vision, finding target objects without additional information from other sensing sources is a challenging job [9]. Despite its ill-posed characteristics, various object detection algorithms having been introduced for vision-based applications such as surveillance cameras, robotics, intelligent systems, and smart devices. Yilmaz et al. classified those algorithms into four types according to the completeness of the detected target objects: feature point detectors, image segmentation, supervised learning, and background subtraction [10]. Among other methods, we focus on background subtraction methods to extract the moving objects from image frames. Supervised learning methods, on the other hand, use object classifiers in the form of decision trees or networks [11–13]. To build those decision-making networks, analysis of the every type of target object is required. However, due to the fact that any kind of object can obstruct the vehicle’s course, it is impossible to prepare a tree for all types of objects, including unknown ones. Therefore learning-based object detection should not be used in a vehicle backing-up scenario. In this section, we introduce background subtraction methods first, followed by optical flow which widely finds motion vectors between two consecutive frames. In fact, those vectors are key components for deriving the movement of vision source itself. The optical flow will also be used later in detecting, distinguishing, and finding objects which are moving in the source frames. 2.1. Background Subtraction Background subtraction is a method of separating moving foreground objects from stationary background images. In a video sequence taken from a traffic monitoring camera, for example, vehicles can be separated out from stationary objects such as road markings and traffic signs which are found on the same location over frames. This separation is necessary to transform the video data into traffic information since the moving vehicles are of more interest than the road itself in most cases. To separate the foreground and background objects, building the information about background objects is essential. For this purpose, the background image is kept in various forms during the subtraction process. The actual form differs among methods though, keeping the background information as accurate as possible is critical. Among the background subtraction methods, one common background extraction method is to hold pixels relatively consistent across two or more consequent images, while rejecting other pixels whose value rapidly changes. Frame differences [5] were one of earliest suggestions of a background subtraction process. This generates a background image by simple frame differences between two consequent images, making the method depend solely on the previous frame. Due to its simplicity, this method has modest computational loads and the background image can be generated with only two frames. The simplicity also makes the background references highly adaptive, resulting in very fast updating of background

Sensors 2016, 16, 23

3 of 10

changes. The major drawback is however that moving foreground objects may easily become part of the background if the object is stopped for more than one frame. The interior pixels of an object also may not be distinguished correctly if the inside has a uniformly distributed intensity value, leaving no difference between the two frames. Instead of simple subtraction, approximated median [6] keeps an accumulation of approximated pixel values, continuously updating the background image in the result. During the approximation, the generated background converges to a median value of all frames, therefore, background objects which are stationary across several frames may be better imprinted than foreground moving objects which have less opportunity to converge. Although the number of frames used in this convergence process depends on the pixel values of the objects, it is generally much longer than the simple frame difference. Temporarily stopped foreground objects or ones with a plain interior can now be better filtered than with simple frame differences, unless the object becomes part of the background by appearing in the same position longer than the originally occluded background. Gaussian mixture models [7,8], on the other hand, process the input frames in different ways. Unlike the abovementioned methods, background components are accumulated as terms of Gaussian distribution functions, and the terms do not directly depict a background image at a specific moment. Rather, the Gaussian distribution functions decide whether a pixel from the input image is foreground or background. This statistical determinant is effective for minor changes in the background, such as waving leaves, moving clouds, or raindrops. Furthermore, using mixtures of such functions makes the method multimodal. That is, in the moving clouds example, both clouds and occluded sky should be considered as background. Thus, the distribution must be at least bimodal to classify the clouds and sky as background at the same time. Recent suggestions on the background subtraction, on the other hand, use sophisticated techniques in combination with those basic ones. Several methods use advanced statistic models to extract backgrounds from moving viewpoints [14–17], while others suggests the use of neural networks or outlier detection models [18,19]. These subtraction models are further categorized into 17 groups according to their main characteristics [20]. 2.2. Optical Flow Rather than background subtraction, another strategy to find moving objects from consequent video frames is by using the optical flow of the frames. According to its definition, optical flow is a motion of apparent movement of brightness patterns in an image, making it sensitive to light sources [21]. Finding the movement can be described as finding the displacement δ which minimizes ε in the following equation: εpδx , δy q “

u xÿ `w x

u y `w y

ÿ

`

˘ I1 px, yq ´ I2 px ` δx , y ` δy q

(1)

x “u x ´w x y“u y ´w y

with a given point x, y in an image I. Searching for such a value is performed within a small window w. The window should be introduced because it becomes an aperture problem without the window. An adequate window size provides hints on the direction of vectors from searching through neighbouring points. To find displacements between two consequent frames, various optical flow estimation techniques are introduced [21–24]. The derived displacements, or flow vectors, contain both the directions and magnitude of pixels, and these vectors are grouped together to form objects. Besides optical flow methods, further classification processes are needed to separate the vector group, in turn which represents the movement of an object, into foreground or background. For this purpose, various statistical grouping algorithms, such as RANSAC and K-means clustering are used [25,26]. Both background subtraction and optical flow methods have been widely used for applications including intrusion or motion detection from video stream with stationary cameras. Recent studies showed that the use of grouping algorithms with optical flow also alleviates the limitation of using

Sensors 2016, 16, 23 Sensors 2016, 16, 23

4 of 10

motion groups [25,26]. Rather classification methods, in our suggestion, we combine a stationary cameras for some levelusing by further classification of feature points into two different motion background subtraction and classification an optical flow process the we stationary and groups [25,26]. Rather using methods, in to oureliminate suggestion, combinerestriction a background derive backgrounds at theflow same time. to The key benefits of usingrestriction this combinational are subtraction and an optical process eliminate the stationary and deriveapproach backgrounds that thesame background motion, known of as using ego-motion, can be cancelled by its optical and at the time. The key benefits this combinational approach are thatflow the vectors, background the each known detection from optical flow and background subtraction can beand cross-checked later. motion, asresult ego-motion, can be cancelled by its optical flow vectors, the each detection result from optical flow and background subtraction can be cross-checked later. 3. Moving Object Detection 3. Moving Object Detection In this section, we present the proposed procedure for moving object detection. To achieve this section, present proposed procedure moving object detection. To moving achieve better betterIndetection andweless false the positives than previousfor standalone approaches with view detectionour and proposed less false positives than previous standalonesubtraction approachesand withoptical movingflow viewmethods sources, sources, scheme uses both background our proposed scheme uses both background and optical flow methods simultaneously. simultaneously. As shown in the Figure 1,subtraction this combinational scheme is composed with the As shown in thestep Figure 1, this combinational composed withand the object following two stepThe approach: following two approach: background scheme motion iscompensation, detection. actual background motion compensation, and object detection. The actual detection of moving objects is done detection of moving objects is done in the object detection part by using the optical flow between a in the object detection part by using the optical flow between a frame and the derived background frame and the derived background at the moment. In our suggestion, a background frame at is the moment. In difference our suggestion, a background is defined by the difference twowith video defined by the between two video frame frames, thus, object detection canbetween be started a frames, thus, object detection can be started with a sequence having at least two frames. sequence having at least two frames.

Frame 1, 2

Optical flow optical flows

Background movement calc. (histogram) optical flows (background)

Mixture of Gaussians Background movement compensation

Background frame

Frame 2

Optical flow optical flows

Dynamic flow thresholding (flow > 62~95%)

Difference filter (frame#2–background)

object regions

Median filter

Object detection

Detected regions

Figure Figure 1. 1. Overall Overall detection detection procedure procedure of of the the proposed proposed scheme. scheme.

3.1. 3.1. Background Background Motion Motion Compensation Compensation As the image image sequences sequences obtained As the the camera camera on on the the vehicle vehicle moves moves freely, freely, the obtained from from the the camera camera contain motions of both background and target objects. Therefore, when a new frame arrives, contain motions of both background and target objects. Therefore, when a new frame arrives, the the ego-motion of background at that time needs to be eliminated first. In our scheme, the ego-motion ego-motion of background at that time needs to be eliminated first. In our scheme, the ego-motion is is induced induced by by optical optical flow flow estimation estimation between between the the newly newly captured captured frame frame and and previously previously derived derived background components. The background components, on the other hand, are derived background components. The background components, on the other hand, are derived by by Gaussian Gaussian mixture models.Since Sincewewe have movements the background, the multimodal distribution mixture models. have movements in theinbackground, the multimodal distribution functions functions needed for distinguishing the movingobjects. background In discussed discussed previouslypreviously are needed are for distinguishing the moving background In thisobjects. approach, this approach, background frames are stored as Gaussian mixture function terms, and if the background frames are stored as Gaussian mixture function terms, and if the ego-motion vector is ego-motion is induced, the entire background terms can shifted back along the vector, induced, thevector entire background terms can be shifted back along thebevector, resulting in compensated resulting in compensated background terms. background terms. To derive the motion vector disparities of each pixel, we use an optical flow method which is To derive the motion vector disparities of each pixel, we use an optical flow method which is based on the pyramidal Lucas-Kanade optical flow and warp theory [23,27]. In this method, the based on the pyramidal Lucas-Kanade optical flow and warp theory [23,27]. In this method, the overall overall optical flow is derived through several scales. The input images are scaled down into optical flow is derived through several scales. The input images are scaled down into specified sizes specified sizes for each level, similar to the pyramid structure. Optical flow results on each level are then scaled back and summed to form the overall disparities for all pixels between the two frames like one in the Figure 2. After the disparity between previous background and current frame has 4

Sensors 2016, 16, 23

5 of 10

Sensors 2016, 16, 23 for each level, Sensors 2016, 16, 23similar

to the pyramid structure. Optical flow results on each level are then scaled back and summed to form overall the disparities for all pixels the two frames like one in the been derived, it is possiblethe to extract ego-motion from thebetween entire pixel flow. This normalization been derived, is disparity possible to extractofprevious the ego-motion from on the entire pixel flow. This normalization Figure 2.incorporates Afteritthe between background and current frame has been derived, is process the generation histogram statistics all displacements and selection of itthe process incorporates the generation of histogram statistics on all displacements and selection of the possible to extract the ego-motion from the entire pixel flow. This normalization process incorporates most frequent flow as the representative background movement. In our suggestion, the size of each most frequent flow the representative background movement. In ourof suggestion, the size of each the generation of histogram statistics on all displacements andThe selection the ego-motion most frequent flow as histogram section isas adjusted so there exist 100 sections. estimated vector is histogram section is adjusted so there 100sections sections. estimated vector is the representative movement. In our suggestion, the size eachego-motion is composed with thebackground average values of the exist elected on The each axis,ofsuch ashistogram −0.75 andsection −0.02 in composed with the average values of the elected sections on each axis, such as −0.75 and −0.02 in adjusted so there exist 100 sections. The estimated ego-motion vector is composed with the average Figure 3. Figure of 3. the elected sections on each axis, such as ´0.75 and ´0.02 in Figure 3. values

Figure 2. Optical flow between two consecutive frames. In this result, the vectors representing Figure 2. 2. Optical between two consecutive frames. In this vectors representing representing Figure flow two consecutive frames. result, thechanges. vectors background motions staybetween consistent while moving object region shows major background background motions motions stay stay consistent consistent while whilemoving movingobject objectregion regionshows showsmajor majorchanges. changes.

Figure 3. Histograms on the optical flow results. We assume the average value of the most frequent Figure 3. Histograms on the optical flow results. We assume the average value of the most frequent section found on the histogram as the ego-motion. Figure found 3. Histograms on the optical results. We assume the average value of the most frequent section on the histogram as theflow ego-motion. section found on the histogram as the ego-motion.

Before compensating should be be built first. There are Before compensating the theego-motion, ego-motion,the thebackground backgroundcomponent component should built first. There Before compensating the ego-motion, the background component should be built first. There several methods to do this, as we discussed in the previous section, but we use Gaussian mixture are several methods to do this, as we discussed in the previous section, but we use Gaussian are several methods to do this,adaptation aspartial we discussed in movements the previous but Gaussian models for their better adaptation to movements of the background In we this use model, mixture models for their better to partial of section, the itself. background itself. Inpixels this mixture models for their better adaptation to partial movements of the itself. In The this of the background frame are stored as mean forasthe Gaussian The meanfunctions. and deviation model, pixels of the background frame areterms stored mean termsfunctions. for the background Gaussian model, pixels of the background frame are stored as mean terms for the Gaussian functions. The terms are affected by upcoming frames with predefined update factors, therefore, the mean term can mean and deviation terms are affected by upcoming frames with predefined update factors, mean andthe deviation terms are byover upcoming update factors, be changed over frames. At the very beginning of background the mean term is initialized therefore, mean term can beaffected changed frames. frames Atsubtraction, the with very predefined beginning of background therefore, term can be changed overfirst frames. At the very beginning background by the first the frame directly, assuming it is a background image. subtraction, themean mean term is initialized by the frame directly, assuming it is of a background subtraction, the mean term is initialized by the first frame directly, assuming it is a background With the estimated vector and the subtracted background components, now it is possible to image. image. compensate background motion. Thissubtracted action is done by a simple shift of thenow mean deviation With thethe estimated vector and the background components, it and is possible to With the estimated vector and the subtracted background components, now it is possible to terms of the Gaussian Mixture functions along with the estimated vector, but being aware that the compensate the background motion. This action is done by a simple shift of the mean and deviation compensate the background motion. This action is done by a simple shift of the mean and deviation estimated movement is usually less than one, so the shift action should be handled as a convolution terms of the Gaussian Mixture functions along with the estimated vector, but being aware that the terms of the the Gaussian Mixture functions along with the after estimated vector, but being that the between terms andisthe motion vector. The mean the shift action now can seen as the estimated movement usually less than one, so the term shift action should be handled as aware abeconvolution estimated movement is usually less than one, so the shift action should be handled as a convolution compensated background. between the terms and the motion vector. The mean term after the shift action now can be seen as between the terms and the motion vector. The mean term after the shift action now can be seen as the compensated background. 3.2. Object Detection the compensated background.

3.2. Object Detection Although it may seem possible to extract target objects using the frame differences between 3.2. Object Detection the compensated background and the currenttarget frame,objects the simple between those frames Although it may seem possible to extract usingcomparison the frame differences between the contains both false-positives and false-negatives. This leads the detection results to have way too much Although it may seem possible to extract target objects using the frame differences between the compensated background and the current frame, the simple comparison between those frames noise and yet even less detection tendency than the results obtained sing optical flow methods alone. compensated background and the current frame, the simple comparison between those frames contains both false-positives and false-negatives. This leads the detection results to have way too contains bothand false-positives anddetection false-negatives. Thisthan leads detection resultssing to have wayflow too much noise yet even less tendency thetheresults obtained optical much noise and yet even less detection tendency than the results obtained sing optical flow methods alone. To solve this problem, we use optical flow between the background and the current methods alone. To solve this problem, optical between background andwhich the current frame to deduce the motion vectors ofwe alluse pixels. At flow this point, thethe resulting vectors have frame to deduce the motion vectors of all pixels. At this point, the resulting vectors which have 5 5

Sensors 2016, 16, 23

6 of 10

Sensors 2016, 16,problem, 23 To solve this we use optical flow between the background and the current frame to deduce the motion vectors of all pixels. At this point, the resulting vectors which have different magnitude and different than magnitude and direction the background be grouped moving and direction the background can bethan grouped into movingcan regions, and the into regions can beregions, reported as the regions can be reported as the moving objects. the moving objects. To group group the the regions regions from from scattered scattered motion motion vectors, vectors, aa threshold threshold magnitude magnitude value value should should be be To presented. In our suggestion, we added a routine that chooses the threshold which can bisect and presented. In our suggestion, we added a routine that chooses the threshold which can bisect and holds holds most 5%–38% of vectors overall in vectors in This a frame. This is the optical flow between a at mostat5%–38% of overall a frame. is because thebecause optical flow between a background background and a frame with relatively big ego-motion tends to yield many more false-positives. In and a frame with relatively big ego-motion tends to yield many more false-positives. In this case, the this case, the threshold magnitude needs to be big enough to be less sensitive to both the threshold magnitude needs to be big enough to be less sensitive to both the background and foreground backgroundresulting and foreground resulting in reducedmotion false-positives. background movement, in reducedmovement, false-positives. The background magnitudeThe determinant has motion magnitude determinant has to be chosen based on its application, so that the routine can to be chosen based on its application, so that the routine can choose the percentage proportionally choose theto percentage proportionally according to the determinants. according the determinants. Before reporting reporting the the final final detection detection results, results, it it is is also also possible possible to to further further apply apply aa simple simple filtering filtering Before process which can reduce false-positives. Since we derived the detection results solely from the process which can reduce false-positives. Since we derived the detection results solely from the optical optical flow method, the aforementioned frame difference the background and theframe current flow method, the aforementioned frame difference betweenbetween the background and the current is frame is still valid for cross-checking with the results. In this usage, the subset of optical flow results, still valid for cross-checking with the results. In this usage, the subset of optical flow results, shown shown as red andrectangles green rectangles in the left picture of 4, Figure 4, reported can be reported as moving as red and green in the left picture of Figure can be as moving objects objects which which alsobeen have been confirmed as moving objects on the frame difference results. Othermethods filtering also have confirmed as moving objects on the frame difference results. Other filtering methods including median filter on the are also possible. including a median afilter on the results areresults also possible.

Derivedframe frame difference (right) for false-positive filtering. The difference is between derived Figure 4.4.Derived difference (right) for false-positive filtering. The difference is derived abetween frame (left) and(left) its background (middle) (middle) with adequate threshold.threshold. a frame and its background with adequate

4. Experimental 4. ExperimentalResults Results evaluate our our suggestion suggestion and compare the results with with other other methods, methods, we present two To evaluate experimental results in this section. In In most cases, evaluations evaluations on optical flow based methods are usually focused focusedonon terms of precision andrelative recall, distance, relative angular distance, angular error, and thethe terms of precision and recall, error, and computation computation time. Since is the accuracy is what in our targetwe application, wethe mainly stateand the time. Since the accuracy what matters in ourmatters target application, mainly state precision precision and recall terms. Precision and recall are defined as follows: recall terms. Precision and recall are defined as follows: True positive True positive (2) Precision“  (2) Precision True positive Falsepositive positive True positive ` False True positive Recall  (3) True positive Recall “ True positive  False negative (3) True positive ` False negative where true positive is the region first detected by the algorithm that turned out to be actually where true positive is theisregion first detected thethe algorithm thatasturned outwhen to be actually moving. moving. False positive the region reportedbyby algorithm moving it actually only False positive is the region reportedisby algorithm moving when it actually onlyobject contains contains background. False negative thethe region whichas should be reported as a moving but background. False negative is the region which should be reported as a moving object but not detected not detected by the algorithm. Thus, the precision term reflects the accuracy of overall detected by the algorithm. Thus, the the precision term of reflects the accuracy of overall detected results while recall results while recall depicts sensitivity the algorithm. depicts sensitivity the algorithm. Thethefirst sample ofsequence in Figure 5 consists of 200 consecutive frames capturing three The first sample sequence in Figure consists of 200 consecutive frames capturing three the vehicles vehicles moving from right to left [28].5 The camera also moves right to left along with first moving from right to left [28]. The camera also moves right to left along with the first vehicle, leaving ego-motion on its background pixels. In this sample, two rank-constraint vehicle, models leaving on its of background In thisof sample, models [18,26] result [18,26] ego-motion result precision 0.83, 0.95pixels. and recall 0.99, two 0.92rank-constraint are used, respectively, while the precision of 0.83, 0.95 and recall of 0.99, 0.92 are used, respectively, while the suggested procedure suggested procedure gives precision of 0.98 and recall of 0.78 (see in Table 1). Since the scenario of the target application focuses on rather closer obstacles, better precision on those can draw the immediate attention of the drivers by reducing false-alarms.

6

Sensors 2016, 16, 23

7 of 10

gives precision of 0.98 and recall of 0.78 (see in Table 1). Since the scenario of the target application focuses on rather closer obstacles, better precision on those can draw the immediate attention of the Sensors 2016, 23 drivers by 16, reducing false-alarms. Sensors 2016, 16, 23

Carexample examplewith with consecutive frame The rank-constraint models give results better Figure 5.5.Car 200200 consecutive frame sets. sets. The rank-constraint models give better Figure 5. Car example with 200 consecutive frame sets. The rank-constraint models give better results recallhowever, (left), however, the precision ofresults overallisresults better the suggested in termsinofterms recallof(left), the precision of overall better is with the with suggested method results in terms of recall (left), however, the precision of overall results is better with the suggested methoddue (right) due to the additional (right) to the additional filters. filters. method (right) due to the additional filters. Table 1. Precision and and recall recall comparison comparison Table 1. Table 1. Precision and recall comparison

Algorithm Precision Algorithm Precision Algorithm Precision Rank-constraint 1 [26] 0.83 Rank-constraint 1 [26] 1 [26] 0.83 Rank-constraint 0.83 Rank-constraint 2 [18] 0.95 Rank-constraint 0.95 Rank-constraint 2 [18] 2 [18] 0.95 Suggested method 0.98 Suggested method 0.98 Suggested method 0.98

Recall 0.99 0.99 0.99 0.92 0.92 0.92 0.78 0.78 0.78

Recall Recall

To evaluate the proposed scheme in everyday use, we implemented the scheme on a FPGA To evaluate evaluate the the proposed proposed scheme scheme in in everyday everyday use, use, we we implemented implemented the scheme on a FPGA FPGA To the scheme on a platform [29,30] and mounted it on a test vehicle. The platform consists of a camera with resolution platform [29,30] and mounted it on a test vehicle. The platform consists of a camera with resolution platform [29,30] and mounted it on a test vehicle. The platform consists of a camera with resolution of of 800 × 480 pixels and a HDMI recorder for further analysis (Figure 6). of 800 × 480 pixels and a HDMI recorder further analysis (Figure 800 ˆ 480 pixels and a HDMI recorder forfor further analysis (Figure 6).6).

(a) (a)

(b) (b)

Figure 6. The FPGA platform with camera. The (a) platform takes LVDS signal from (b) camera and Figure 6. 6. The FPGA platform with camera. The takes LVDS signal signal from from (b) (b) camera camera and and Figure The FPGA platform with HDMI camera.recorder The (a) (a) platform platform takes outputs as HDMI signal. External was needed forLVDS further analysis as the platform outputs as as HDMI HDMI signal. signal. External HDMI further analysis analysis as as the the platform platform outputs HDMI recorder recorder was was needed needed for for further processes the input framesExternal on-the-fly. processes the input frames on-the-fly. processes the input frames on-the-fly.

For the test, it took 19 videos 150 frames each, at seven frames per second, having both For the test, it took 19 videos 150 frames each, at seven frames per second, having both ego-motion towards the side the each, targetat object’s movement. Figure 7 and Table 2 show For the test, it took 19 rear videos 150and frames seven frames per second, having both ego-motion ego-motion towards the rear side and the target object’s movement. Figure 7 and Table 2 show parts of the false-positive and negative analysis results onTable all 19 2test scenarios. towards thesequences rear side with and the target object’s movement. Figure 7 and show parts of the parts of the sequences with false-positive and negative analysis results on all 19 test scenarios. sequences with false-positive and negative analysis results on all 19 test scenarios.

Sensors 2016, 16, 23

8 of 10

Sensors 2016, 16, 23 Sensors 2016, 2016, 16, 16, 23 23 Sensors Sensors 2016, 2016, 16, 16, 23 23 Sensors Sensors 2016, 16, 23 Sensors 2016, 2016, 16, 16, 23 23 Sensors

(a)

(b)

(a) (a) (a) (a) (a) (a) (a)

(b) (b) (b) (b) (b) (b) (b)

(c)

(d)

(c) (d) (c) (d) (c) (d) (c) (d) Figure 7. Part of detection results from the test vehicle. Tests were conducted in several several places places and and (c) (d) Figure 7. Part of detection results from Tests were wereconducted conducted Figure 7. Part Part of detection detection results fromthe thetest testvehicle. vehicle. Tests Tests were conducted inin several places places and and Figure 7. of results from the test vehicle. in several (c) (d) in (c) (d) Figure 7. Part of detection results from the test vehicle. Tests were conducted in several places and Figure 7. Part of detection results from the test vehicle. Tests were conducted several places and under various conditions such as (b,c) parking lots and (a,d), streets ranging from (a,c) daytime to under various conditions such as from (b,c) parking lots and (a,d), streets rangingfrom from (a,c)daytime daytime under various conditions such as (b,c) parking lots and (a,d), streets ranging from (a,c) daytime to under various conditions such as (b,c) parking lots and (a,d), streets ranging (a,c) to (d) Figure 7. Part of detection results the test vehicle. Tests were conducted in several places and under various conditions such as (b,c) parking lots and (a,d), streets ranging from (a,c) daytime to under various conditions such as (b,c) parking lots and (a,d), streets ranging from (a,c) daytime to (d) dusk. All 19 videos including the above four were taken while the vehicle was in backwards Figure 7.videos Part ofincluding detection results from the test test vehicle. Tests were conducted in several placesmotion. and Figure 7. Part of detection results the vehicle. conducted in several places and (d) dusk. All 19 19 videos including including the four above four were taken while the vehicle was in daytime backwards dusk. All videos the above four were taken while the vehicle in backwards dusk.(d) All 19 the above were taken while the vehicle wasfrom inwas backwards under various conditions such as from (b,c) parking lots andTests (a,d),were streets ranging (a,c) to (d) dusk. All 19 19 videos including including the above above four were taken while the the vehicle was in daytime backwards (d) dusk. All videos the four were while vehicle was in backwards motion. under various conditions such as as (b,c) (b,c) parking lots andtaken (a,d), streets streets ranging from (a,c) daytime to to under various conditions such parking lots and (a,d), ranging from (a,c) motion. motion.

(d) dusk. All 19 videos including the above four were taken while the vehicle was in backwards motion. motion. (d) dusk. dusk. All 19 videos including the above aboveresults four were were taken while the vehicle vehicle was in in backwards (d) All videos including the four taken while the was backwards Table 2.19 False positive and negative on the all 19 videos (150 frames each). motion. Table 2. False False positive and negativeresults results on all 19 videos (150 frames each). Table 2. positive and negative results frames each). on the motion. Table 2. False positive and negative the all all19 19videos videos(150 (150 frames each). motion. Table 2. 2. False False positive positive and and negative negative results results on on the the all all 19 19 videos videos (150 (150 frames frames each). each). Table Table 2. False positive and negative results on the all 19 videos (150 frames each). Table 2. 2. False False positive positive and and negative negative results results on on the the all all 19 19 videos videos (150 (150 frames frames each). each). Table Scenarios Scenarios Scenarios Scenarios

Scenarios Scenarios Scenarios False-Positives False-Positives Scenarios Scenarios False-Positives False-Positives False-Positives False-Positives False-Negatives False-Negatives False-Negatives False-Positives False-Negatives False-Negatives False-Negatives False-Positives False-Positives False-Negatives False-Negatives Scenarios Scenarios False-Negatives Scenarios Scenarios Scenarios Scenarios Scenarios False-Positives False-Positives Scenarios Scenarios False-Positives False-Positives False-Positives False-Negatives False-Negatives False-Positives False-Positives False-Negatives False-Negatives False-Negatives False-Negatives False-Positives False-Positives False-Negatives Scenarios Scenarios False-Negatives False-Negatives Scenarios Scenarios Scenarios Scenarios Scenarios False-Positives False-Positives Scenarios Scenarios False-Positives False-Positives False-Negatives False-Negatives False-Positives False-Positives False-Negatives False-Negatives False-Positives False-Positives False-Positives False-Negatives False-Negatives False-Negatives False-Negatives Scenarios Scenarios False-Negatives Scenarios Scenarios Scenarios Scenarios False-Positives False-Positives Scenarios Scenarios Scenarios False-Positives False-Positives False-Negatives False-Negatives False-Positives False-Negatives False-Negatives False-Positives False-Positives False-Positives False-Negatives False-Positives False-Negatives False-Negatives False-Negatives 5. Conclusions Conclusions 5. False-Negatives

2 222 00 0 0200

222 2 212 11 1 121 2 12 11

000 0404 0 4 404 4 400 44

333 3030 0 030 033 00

11

00

1 1111 1 1 11111

0 0000 0 0 00000

22 20 202 0 020 2 02 00

00 0000 0 00 00 0 000 00

11 1111 11 11 111 11

1 1

11 1010 0110 1 101 0 0 00

00 0000 000 00 00 00

0 0

00 0000 000 0 000 000 0

0 0

000 00 0000

00 000 00

1 11

00 0000

22 022 00

0 00

0 000 0 0000

0 1001 1 0110 11

00 0202 02 200 0220 2 22

00 0000 000 0000 000

00 0101 0 0 1010 10 1 11 1

00 0000 0 000 00 00 0 000

00

0 0

00 0101

3 0

5. Conclusions Conclusions 5. 5. Conclusions In this this paper, paper, we we discussed discussed two two types types of of methods methods for for moving moving object object detection detection and and their their In 5. Conclusions 5. Conclusions Conclusions 5. In this this We paper, we discussed discussed two types types of methods methods for moving moving object detection and their their In paper, we two of for object detection and constraints. We suggested combinational method and false-positive false-positive filters to relieve relieve the stationary stationary constraints. suggested aa combinational method and filters to the 5. Conclusions In this paper, we discussed two types of methods for moving object detection and their constraints. We suggested combinational method and false-positive filters tothe relieve the stationary constraints. We suggested aa combinational method and false-positive filters to relieve In this this paper, we discussed two types of methods methods for moving object detection and their In paper, discussed two types of methods for moving object detection and their In this paper, wewe discussed two types of for moving object detection and their viewpoint constraint. Experimental results show the overall improvement in termthe of stationary precision, viewpoint constraint. Experimental results show the overall improvement in term of precision, constraints. We suggested a combinational method and false-positive filters tothe relieve the stationary In this paper, we discussed two types of methods for moving object detection and their constraints. viewpoint constraint. Experimental results show the overall improvement in the term of precision, viewpoint constraint. Experimental results show the overall improvement in the term of precision, constraints. We suggested a combinational method and false-positive filters to relieve the stationary constraints. We suggested a combinational method and false-positive filters to relieve the stationary which is an important feature for use with BCI applications. which is an important feature for use with BCI applications. constraints. We suggested a combinational method and false-positive filters to relieve the stationary viewpoint constraint. Experimental results show the overall improvement in the term of precision, which is an ana important featuremethod for use useresults with BCI applications. which is important feature for with applications. We suggested combinational andBCI false-positive filters to relievein the stationary viewpoint viewpoint constraint. Experimental results show the overall overall improvement in the term of precision, precision, viewpoint Experimental show the overall improvement the term of viewpoint constraint. Experimental results the improvement the term of precision, which is anconstraint. important feature for use with show BCI Acknowledgments: This work was was supported byapplications. the IT R&D R&D program program of MKE, MKE,inKorea Korea (10041125, High Acknowledgments: This work supported by the IT of (10041125, High constraint. Experimental results show the overall improvement in the term of precision, which which is an important feature for use with BCI applications. which is an important feature for use with BCI applications. Acknowledgments: This work was supported by the IT R&D program of MKE, Korea (10041125, High is Acknowledgments: This work was supported by the IT R&D program of MKE, Korea (10041125, High which is an important feature for use with BCI applications. Resolution SoC SoC Development Development for for Automotive Automotive SXGA SXGA Camera Camera System). System). Resolution Acknowledgments: This work supported by Camera the IT R&D program of MKE, Korea (10041125, High an important for use with BCI applications. Resolutionfeature SoC Development Development forwas Automotive SXGA Camera System). Resolution SoC for Automotive SXGA System). Acknowledgments: This work was supported by the IT R&D program of of MKE, MKE, Korea Korea (10041125, (10041125, High High Acknowledgments: This work was supported by the IT R&D program Resolution SoC Development forconceived Automotive Camera Author Contributes: Contributes: D.S. Kim Kim conceived andSXGA designed the System). experiments; D.S. D.S. Kim Kim and and J.S. J.S. Kwon Kwon wrote wrote the the Author D.S. and designed the experiments;

Acknowledgments: This work was supported by the IT R&D program of MKE, Korea (10041125, High Resolution SoC Development Development for Automotive SXGA Camera System). Resolution SoC for Automotive Camera Author Contributes: D.S. Kim Kim conceived andSXGA designed the System). experiments; D.S. D.S. Kim Kim and and J.S. J.S. Kwon Kwon wrote wrote the the Author D.S. conceived and designed the experiments; paper. Contributes: paper. Resolution Development for Automotive SXGA Camera AuthorSoC Contributes: D.S. Kim conceived and designed the System). experiments; D.S. Kim and J.S. Kwon wrote the paper. paper. Author Contributes: Contributes: D.S. D.S. Kim Kim conceived conceived and and designed designed the the experiments; experiments; D.S. D.S. Kim Kim and and J.S. J.S. Kwon Kwon wrote wrote the the Author

paper. 88 the experiments; D.S. Kim and J.S. Kwon wrote the Author Contributes: D.S. Kim conceived and designed paper. paper. 88 paper. 8 88

8

Sensors 2016, 16, 23

9 of 10

Acknowledgments: This work was supported by the IT R&D program of MKE, Korea (10041125, High Resolution SoC Development for Automotive SXGA Camera System). Author Contributions: D.S. Kim conceived and designed the experiments; D.S. Kim and J.S. Kwon wrote the paper. Conflicts of Interest: The authors declare no conflict of interest.

References 1. 2. 3. 4. 5.

6. 7. 8.

9. 10. 11.

12.

13.

14. 15. 16. 17.

18. 19. 20.

Fleming, B. New automotive electronics technologies. IEEE Veh. Technol. Mag. 2008, 3, 10–12. Jiménez-Pinto, J.; Torres-Torriti, M. Optical flow and driver’s kinematics analysis for state of alert sensing. Sensors 2013, 13, 4225–4257. [CrossRef] [PubMed] Llamazares, A.; Ivan, V.; Molinos, E.; Ocaña, M.; Vijayakumar, S. Dynamic obstacle avoidance using bayesian occupancy filter and approximate inference. Sensors 2013, 13, 2929–2944. [CrossRef] [PubMed] Dogan, ˘ S.; Temiz, M.S.; Külür, S. Real time speed estimation of moving vehicles from side view images from an uncalibrated video camera. Sensors 2010, 10, 4805–4824. [CrossRef] [PubMed] Zhan, C.; Duan, X.; Xu, S.; Song, Z.; Luo, M. An improved moving object detection algorithm based on frame difference and edge detection. In Proceedings of the Fourth International Conference on Image and Graphics, Chengdu, China, 22–24 August 2007. McFarlane, N.J.B.; Schofield, C.P. Segmentation and tracking of piglets in images. Mach. Vis. Appl. 1995, 8, 187–193. [CrossRef] Friedman, N.; Russell, S. Image segmentation in video sequences: a probabilistic approach. In Proceedings of the 13th Conference on Uncertainty in Artificial Intelligence, Providence, RI, USA, 1–3 August 1997. Chris, S.; Grimson, W.E.L. Adaptive background mixture models for real-time tracking. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Fort Collins, CO, USA, 23–25 June 1999. Poggio, T.; Torre, V.; Koch, C. Computational vision and regularization theory. Nature 1985, 317, 314–319. [CrossRef] [PubMed] Yilmaz, A.; Javed, O.; Shah, M. Object tracking: A survey. J. ACM Comput. Surv. 2006, 38, 1–45. [CrossRef] Dalal, N.; Triggs, B. Histograms of oriented gradients for human detection. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA, 25 June 2005. Viola, P.; Jones, M. Rapid object detection using a boosted cascade of simple features. In Proceedings of the IEEE Computer Society Confercence on Computer Vision and Pattern Recognition, Kauai, HI, USA, 8–14 December 2001. Lienhart, R.; Kuranov, A.; Pisarevsky, V. Empirical Analysis Of Detection Cascades of Boosted Classifiers for Rapid Object Detection. In Pattern Recognition—25th DAGM Symposium, Magdeburg, Germany, September 10–12, 2003; Springer Berlin Heidelberg: Berlin, Germay, 2003. Huang, S.C.; Chen, B.H. Highly accurate moving object detection in variable bit rate video-based traffic monitoring systems. Neural Netw. Learn. 2013, 24, 1920–1931. [CrossRef] [PubMed] Huang, S.C.; Chen, B.H. Automatic moving object extraction through a real-world variable-bandwidth network for traffic monitoring systems. IEEE Trans. Ind. Electron. 2013, 61, 2099–2112. [CrossRef] Cheng, F.C.; Huang, S.C.; Chen, B.H. A hybrid background subtraction method with background and foreground candidates detection. ACM Trans. Intell. Syst. Technol. 2015, 7, 1–14. [CrossRef] Guo, J.M.; Hsia, C.H.; Liu, Y.F.; Shih, M.H.; Chang, C.H.; Wu, J.Y. Fast background subtraction based on a multilayer codebook model for moving object detection. IEEE Trans. Circuits Syst. Video Technol. 2013, 23, 1809–1821. [CrossRef] Zhou, X.; Yang, C.; Yu, W. Moving object detection by detecting contiguous outliers in the low-rank representation. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 35, 597–610. [CrossRef] [PubMed] Huang, S.C.; Do, B.H. Radial basis function based neural network for motion detection in dynamic scenes. IEEE Trans. Cybern. 2013, 44, 114–125. [CrossRef] [PubMed] Bouwmans, T. Traditional and recent approaches in background modeling for foreground detection: An overview. Comput. Sci. Rev. 2014, 11–12, 31–66. [CrossRef]

Sensors 2016, 16, 23

21. 22. 23.

24. 25. 26. 27. 28. 29.

30.

10 of 10

Sharmin, N.; Brad, R. Optimal filter estimation for Lucas-Kanade optical flow. Sensors 2012, 12, 12694–12709. [CrossRef] Black, B.J.; Anandan, P. Robust dynamic motion estimation over time. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Maui, HI, USA, 3–6 June 1991. Brox, T.; Bruhn, A.; Papenberg, N.; Weickert, J. High Accuracy Optical Flow Estimation Based on a Theory for Warping. In Computer Vision—ECCV 2004: 8th European Conference on Computer Vision, Prague, Czech Republic, May 11–14, 2004. Proceedings, Part IV; Springer Berlin Heidelberg: Berlin, Germany, 2004; pp. 25–36. Jepson, A.; Black, M.J. Mixture models for optical flow computation. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New York, NY, USA, 15–17 June 1993. Kim, J.; Wang, X.; Wang, H.; Zhu, C.; Kim, D. Fast moving object detection with non-stationary background. Multimedia Tools Appl. 2013, 67, 311–335. [CrossRef] Sheikh, Y.; Javed, O.; Kanade, T. Background subtraction for freely moving cameras. In Proceedings of the IEEE 12th International Conference on Computer Vision, Kyoto, Japan, 29 September–2 October 2009. Black, M.J.; Anandan, P. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 1996, 63, 75–104. [CrossRef] Sand, P.; Teller, S. Particle Video: Long-Range Motion Estimation Using Point Trajectories. Int. J. Comput. Vis. 2008, 80, 72–91. [CrossRef] Guo, G.; Kaye, M.E.; Zhang, Y. Enhancement of Gaussian background modelling algorithm for moving object detection & its implementation on FPGA. In Proceedings of the IEEE 28th Canadian Conference on Electrical and Computer Engineering, Halifax, NS, Canada, 3–6 May 2015. Botella, G.; Martín, J.A.; Santos, M.; Meyer-Baese, U. FPGA-based multimodal embedded sensor system integrating low- and mid-level vision. Sensors 2011, 11, 8164–8179. [CrossRef] [PubMed] © 2015 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons by Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).