Journal of Microscopy, Vol. 253, Issue 1 2014, pp. 65–78
doi: 10.1111/jmi.12098
Received 29 April 2013; accepted 8 October 2013
Automatic detection and analysis of cell motility in phase-contrast time-lapse images using a combination of maximally stable extremal regions and Kalman filter approaches ¨ K I §, J . H E I K K I L A ¨‡ M . K A A K I N E N ∗ , †, S . H U T T U N E N †, ‡, L . P A A V O L A I N E N §, , V . M A R J O M A & L . E K L U N D ∗, # ∗ Biocenter Oulu, University of Oulu, Finland
†Mika Kaakinen and Sami Huttunen contributed equally to this work ‡Department of Computer Science and Engineering, University of Oulu, Finland §Department of Biological and Environmental Science, Nanoscience Center, University of Jyv¨askyl¨a, Finland Department of Mathematical Information Technology, University of Jyv¨askyl¨a, Finland #Oulu Center for Cell-Matrix Research, Department of Medical Biochemistry and Molecular Biology, Institute of Biomedicine, University of Oulu, Finland
Key words. Automatic cell segmentation, cell migration, Kalman filter, MSER, phase-contrast microscopy, tracking.
Summary Phase-contrast illumination is simple and most commonly used microscopic method to observe nonstained living cells. Automatic cell segmentation and motion analysis provide tools to analyze single cell motility in large cell populations. However, the challenge is to find a sophisticated method that is sufficiently accurate to generate reliable results, robust to function under the wide range of illumination conditions encountered in phase-contrast microscopy, and also computationally light for efficient analysis of large number of cells and image frames. To develop better automatic tools for analysis of low magnification phase-contrast images in time-lapse cell migration movies, we investigated the performance of cell segmentation method that is based on the intrinsic properties of maximally stable extremal regions (MSER). MSER was found to be reliable and effective in a wide range of experimental conditions. When compared to the commonly used segmentation approaches, MSER required negligible preoptimization steps thus dramatically reducing the computation time. To analyze cell migration characteristics in time-lapse movies, the MSER-based automatic cell detection was accompanied by a Kalman filter multiobject tracker that efficiently tracked individual cells even in confluent cell populations. This allowed quantitative cell motion analysis resulting in accurate measurements of the migration magnitude and direction of individual cells, as well as characteristics of collective migration Correspondence to: Lauri Eklund, Oulu Center for Cell-Matrix Research, Department of Medical Biochemistry and Molecular Biology, Institute of Biomedicine, University of Oulu, Finland P.O. Box 5000, FI-90014 Oulu, Finland. Tel: +358 294 486073; fax: +358 8 537 6115; e-mail: [email protected] C 2013 The Authors C 2013 Royal Microscopical Society Journal of Microscopy
of cell groups. Our results demonstrate that MSER accompanied by temporal data association is a powerful tool for accurate and reliable analysis of the dynamic behaviour of cells in phase-contrast image sequences. These techniques tolerate varying and nonoptimal imaging conditions and due to their relatively light computational requirements they should help to resolve problems in computationally demanding and often time-consuming large-scale dynamical analysis of cultured cells.
Introduction Phase-contrast microscopy is the most commonly used contrasting method to visualize living cells. This is due to the relatively simple configuration of the microscopy instruments, the resulting low costs, and the ability to observe cells without staining and the phototoxicity of the short wavelength light used in fluorescent microscopy. Manual tracking of thousands of cells over time, however, is not feasible, and automatic computer based cell segmentation and tracking approaches are necessary for quantitative studies. In contrast to relative ease of image production, automatic cell segmentation of phasecontrast images has been more challenging. Phase-contrast images of cells suffer from low contrast with respect to the background and there is variation in the pixel intensity distribution within the cells due to varying thicknesses, making identification of single cells difficult. Phase-contrasting also produces an artificial halo effect surrounding the cells that obscures cell interfaces. Several approaches have been developed for cell segmentation that are based on the separation of connected
66
M. KAAKINEN ET AL.
pixels belonging to the cell from those belonging to the background. Such methods include thresholding, the watershed method and texture analysis (Wu et al., 1995; Koyuncu et al., 2012; Korzynska et al., 2007). Alternatively, active contour algorithms that capture cell boundaries have been used (Tscherepanow et al., 2008; Wang, He & Metaxas, 2007; Ali et al., 2007, Seroussi et al., 2012). The shortcomings in commonly used techniques is the requirement for image preprocessing or the necessity to use accompanying techniques due to nonuniformities of pixel intensities inside the segmented cells, similarities in the background and specimen pixel intensities, or because the cell boundaries cannot be clearly resolved (Wu et al., 1995; Tse et al., 2009; Debeir et al., 2005; Ali ¨ et al., 2012). Manet al., 2007; Yin et al., 2012; Ambuhl ual adjustments and preprocessing techniques require expertise and their use significantly increases computational time. Therefore, reliable automatic analyzing techniques that are more feasible to use in varying conditions are urgently needed. In 2004 Matas et al. described a technique for detecting regions in an image that remain stable over a range of threshold values, called maximally stable extremal regions or MSERs (Matas et al., 2004). The MSER method has important characteristics, which are useful for the segmentation of objects from complex images. First, the segmented regions are preserved under geometric and monotonic intensity transformations. Secondly, MSER is not sensitive to pixel intensity changes and nonuniformities in background intensities as it is dependent only on the ordering of pixel intensities within the MSER and its outer boundary. The feasibility of MSER in detecting cultured cells was recently recognized in a machine learning based approach in which the detector was used to find candidate regions that represented putative cells (Arteta et al., 2012). In our study, we extended MSER approach to detect cells using a wide range of phase-contrast images and test the feasibility and the intrinsic properties of MSER for the automatic detection of cells under varying and challenging imaging conditions. In addition to automatic cell segmentation of single still images, we combined MSER with a Kalman filter based tracker modified for multiple objects (Huttunen & Heikkil¨a, 2008). This tracking approach uses segmentation masks as a source of measurements and utilizes soft assignment to associate the observations to the objects being tracked. The combination of MSER and a Kalman filter based tracking enabled accurate and reliable cell segmentation and migration analysis even in demanding dense cell populations. Because the computational requirements of these approaches were low and the required user interaction was minimal, the methods developed should be useful in computationally more demanding assays such as experiments performed in multiwell plate formats, analysis of living cells in high densities, and as modules in image analysis software.
Methods MSER MSER is a blob detector identifying regions in an image that remain stable over a certain number of thresholds. MSER was originally developed for stereo matching purposes, and later it has been widely used for object recognition. In MSER, maximum region size and stability range parameter (a delta value, ) control the segmentation sensitivity and accuracy. Maximum region size defines the upper limit for the size of the valid objects, in this work representing cell nuclei. The adjustment of maximum region size was done first by performing segmentation with default settings and then using a parameter value that corresponds to the mean region size multiplied by 1.25. The -value can be adjusted to set the range of threshold values within which the regions should remain stable (i.e. the region does not grow significantly over the selected threshold range); the higher the -value the broader is the range of threshold values within which the regions should remain stable. In phase-contrast images, the pixel intensity profile characterizing the cells often resembles that of background, which makes cell segmentation a challenging problem. Because cells and in particular cell nuclei are slightly darker than the background MSER could be considered as a potential method for this problem. Those cells that exhibit pixel “leaking” to background or to adjacent cells can be discarded and the risk of false detections can be reduced. In addition to the two abovementioned parameters of MSER, minimum region size, maximum variation and minimum diversity of the detected regions can also be adjusted. In our experiments we found that these parameters were not critical and the default values were used. The MSER was obtained from a VLFeat open source library (http://www.vlfeat.org/). Kalman filter Kalman filter is a common approach for object tracking as it can efficiently predict and filter the target locations based on previous observations and a dynamic state model. The main limitation of the Kalman filter is that it assumes Gaussian distribution for the observation noise. It would be possible to utilize the MSER regions directly as a source of location measurements in a Kalman filter based tracker but because of frequently occurring detection errors that do not usually follow Gaussian distribution this would easily lead to tracking failures. In order to alleviate this problem we have applied the multiobject tracking approach proposed by Huttunen & Heikkil¨a (2008) that is based on probabilistic data association where soft assignments of the measurements are used instead of hard assignments. The basic idea in this method is to detect the objects several times from the same frame with varying detector settings and compute soft assignments for each output. This will increase the robustness of tracking against
C 2013 The Authors C 2013 Royal Microscopical Society, 253, 65–78 Journal of Microscopy
AUTOMATIC DETECTION AND ANALYSIS OF CELL MOTILITY
67
Fig. 1. Measurement extraction for tracker. (A) An input image representing cultured squamous cell carcinoma cells. Six individual cells are numbered (1–6). (B) Cells segmented with MSER. Red contours outline the segmented areas and the green dots indicate the centroids of each detection. Note closely adjacent cells that were erroneously merged as a single detection (1–2 and 3–4–5). (C) The corresponding primary masks of the segmented cells. (D) Primary masks after a morphological erosion operation. Red crosses represent new centroids of the primary masks. Note that masks that erroneously merged cells 1–2 and 3–4–5 in initial MSER segmentation were correctly separated. However, the masks corresponding cells 5 and 6 were split in two after erosion operation. (E) After several morphological operations the primary masks may contain many centroids (measurements). The location of centroids is fitted into Gaussian mixture model. The centroids that locate far apart from the others (yellow arrowheads) less likely represent a true location and are down-weighted. (F) The resulting centroids are indicated by red crosses that accurately represent original cell nuclei.
detection errors. The method was originally developed for human tracking, but it can be modified for cell tracking purposes. In our approach we process the binary MSER regions with a filter bank that contains a set of morphological erosions. With this approach we can, for example, separate a binary region of two cells that have been erroneously merged by the MSER detector. As a result we get a large set of measurements (centroids of the binary regions) for each cell, and we assume that these measurements together form a Gaussian mixture, and its modes represent the correct cell locations. A Gaussian mixture model is used for deriving weights for the measurements that indicate the probability of belonging to an individual cell. The extraction for obtaining the measurements is demonstrated in Fig. 1. A probabilistic data association scheme is embedded to the Kalman filter framework to enable multiobject tracking. More details of the data association algorithm can be found from Huttunen & Heikkil¨a (2008).
Cell culture and microscopy Madin-Darby Canine Kidney (MDCK) cells were grown on sixwell plates (Costar, Corning Incorporated, New York, U.S.A.) in MEM+GlutaMAX media (Gibco, Life Technologies Corporation, U.S.A.) supplemented with 5% fetal bovine serum (FBS) and 1% penicillin-streptomycin. Squamous cell carcinoma cells (HSC-3) were kindly provided by M.D. Jyri Moilanen and grown in Dulbecco’s minimum essential medium (DMEM C 2013 The Authors C 2013 Royal Microscopical Society, 253, 65–78 Journal of Microscopy
F-12, Gibco) supplemented with 10% heat inactivated FBS and 1% penicillin-streptomycin. Human umbilical vein endothelial cells (HUVECs) were grown in endothelial cell basal medium supplemented with endothelial cell growth supplement (Cell Applications, San Diego, CA, U.S.A.) and 10% FBS. For the cell motility assay, a scratch was made on the middle of the confluent culture and cells were imaged overnight in a temperature and CO2 controlled microscope stage incubator (Okolab, Italy) mounted onto Olympus IX81 inverted microscope. Images consisting of 1376×1032 pixels were captured with a grey-scale camera (Olympus XM10, Germany) ¨ controlled by CellˆP software (Soft Imaging System, Munster, Germany).
Validation of the segmentation The reliability of MSER segmentation was verified manually by comparing the cells in the microscopic fields and the objects detected by MSER. Before manual annotation the connected component analysis was performed to judge whether closely located regions represent single or separate segmentation. Thereafter the contours of the binary masks of detected regions were overlaid on the original input image to distinguish correctly detected cells from other segmented regions (false positives). The manual counting was performed with Image J (National Institutes of Health, Bethesda, Maryland, U.S.A.) cell counter plug-in (Kurt De Vos, University of Sheffield, Academic Neurology, England). We denote the detection recall
68
M. KAAKINEN ET AL.
as the total number of true positive cells divided by the total number of cells in a frame. As detection precision, we denote the number of true positive cells divided by the total number of segmented regions. Other methods used in comparison The segmentation was performed using the commonly available segmentation approaches: Otsu thresholding, watershed and active contours. The morphological watershed segmentation was performed according to Beucher & Meyer (1993). Images were first preprocessed with a median filter to reduce noise and then with a morphological erosion operation to remove roughness inside the cells and to enhance contrast between the cells and the background. Next a gradient magnitude filter was applied. Finally, local minima close to local background were suppressed to prevent over-segmentation. After applying the watershed transform, all objects less than 30 pixels in size were removed. The active contours method was applied according to Caselles et al. (1997). The image was first smoothed with an anisotropic diffusion filter followed by calculation of gradient magnitude. Then the geodesic active contour level set was initialized by four user-selected seed points from which a circle of radius 375 pixels defined the initial zero level set. The front was defined to contract during 7000 iterations that led to splitting of four initial contours into many individual contours defining individual cells. The speed term for the front propagation was calculated from the gradient magnitude image using sigmoid function. Optionally, prior to applying the abovementioned segmentation approaches, images were first processed with an image preconditioning technique (http://www.kangli.org/page6.html#pcnddemo) described in Li & Kanade (2009). The image preconditioning aims to facilitate object segmentation. To obtain suitable images for watershed transform, the preconditioned images were preprocessed by setting all the pixel values less than 100 to a maximum intensity value (255) and the intensities of all pixels were inverted. This step was necessary to achieve local minima inside of the objects and to avoid overexpansion of the detected regions. Tracking data were also compared to the MTrack2 approach (Stuurman, University of California and the Howard Hughes Medical Institute), a plug-in of freely downloadable Fiji software (Schindelin et al., 2012). Segmentation of cells was first performed by using MSER. MTrack2 was then applied to the primary mask of segmented cells by defining a maximum velocity of tracked objects to correspond to 20 pixels/frame and minimum track length to 1/frame. In manual verification of the tracking data each track was followed until termination. The causes of lost tracks (the tracks that failed to proceed until the end of image sequence) were categorized in four groups: 1. Track failure in which the tracker loses the target due to segmentation failure of MSER or the tracker shifts to an adjacent object (i.e. cell) during the tracking process.
2. The cell leaves the image border. 3. Cell roundup due to death or mitosis. Both cell death and mitosis are characterized by an initial increase of circularity and pixel intensity and decrease in cell area. In cell division the initial features are reversed and daughter cells emerge. 4. Situations in which the identity (ID) of an object is changed during the tracking process.
Implementations The segmentation and tracking approaches were implemented using a PC with Intel Xeon CPU operating at 2.50 GHz and with 128 GB RAM. MATLAB was used for MSER, Otsu and Kalman filtering. BioImageXD (Kankaanp¨aa¨ et al., 2012) was used for watershed and active contours.
Results Segmentation of single cells in dense cell cultures based on MSER approach We first tested the effect of the -value on MSER detection recall and precision (Fig. 2 and Table 1) using challenging phase-contrast images obtained from scratched confluent cell cultures that represent the classic “wound closure” cell migration assay. The image sequences 1 and 2 were acquired by different microscope users representing different experimental conditions, cell densities and cell types. In the images analyzed, the pixel intensity inside the nuclei showed the most prominent contrast in relation to the background and was therefore mainly detected by MSER (Fig. 2). Notably, when the default region size was used, -value adjustment markedly contributed to detection outcome; the smallest values resulted in a higher number of false positive detections as well as artificial merging of single cells (Supporting Fig. S1, Table 1), whereas the detection improved with increasing -values. Merging of several cells within a single detection (Fig. 1 and Supporting Fig. S1) was the most common failure, which decreased the detection result. To reduce the incidence of merged regions we next limited the maximum region size. As a maximum value we used the average region size at -value 3 that mostly represented the cell nuclei (red contours in Fig. 2). As shown in Table 1, limiting the maximum region size improved the detection precision in both sequences, more profoundly in sequence 2. These observations could be explained by the fact that with increasing -values the segmentation becomes less sensitive and, if the pixel intensity variation is small within cells and the regions between the cells, the region size is expanded until it fulfils the criteria of maximally stable region. Adjustment of additional three parameters available (minimum region size, maximum variation and minimum diversity) did not improve the segmentation result of MSER. C 2013 The Authors C 2013 Royal Microscopical Society, 253, 65–78 Journal of Microscopy
AUTOMATIC DETECTION AND ANALYSIS OF CELL MOTILITY
69
Table 2. Comparison of MSER performance with three common segmentation approaches. The MSER 3 and limited maximum region size were used for comparison. The techniques indicated in the table were applied to the original grey-scale image representing the first frame of sequence 1 and 2. Alternatively, an image preconditioning (preconditioned, Li & Kanade, 2009) technique was used to improve contrast in images before applying a segmentation technique. Active contours and Otsu threshold could not be applied to the original input image to result in a reasonable outcome (-). Sequence 1 Segmentation approach MSER 3 Watershed (original) Watershed (preconditioned) Active contours (original) Active contours (preconditioned) Otsu threshold (original) Otsu threshold (preconditioned) Sequence 2 Segmentation approach MSER 3 Watershed (original) Watershed (preconditioned) Active contours (original) Active contours (preconditioned) Otsu threshold (original) Otsu threshold (preconditioned) Fig. 2. First frames of image sequences obtained from wounded confluent squamous cell carcinoma cell cultures (sequence 1, panel A) and MDCK cells (sequence 2, panel B) were used to test the effect of different -values on MSER segmentation with limited maximum region size. Red contours outline the segmented areas most often representing cell nuclei. Green dots indicate the centroids of each detection to distinguish contours that belong to the individual cells. Examples of segmented regions are magnified in the insets at the top right corner of each MSER segmented image. Note that with smaller -values more noncell objects (white arrowheads, exemplified also with higher magnification in the Supporting Fig. S1) are detected in the wound area.
Recall%
Precision%
88 51 88 − 48 − 66
92 74 97 − 91 − 97
Recall%
Precision%
81 87 89 − 50 − 73
88 52 96 − 93 − 60
Comparison of MSER with commonly used segmentation approaches To compare the cell detection recall and precision of MSER to the commonly used segmentation approaches, the first frames of sequence 1 and sequence 2 were segmented using the Otsu threshold, active contours, and watershed, the techniques that are commonly accessible and which have been applied to phase-contrast images (Table 2). Notably, in contrast to MSER, none of the other techniques could be applied directly to the original grey-scale images, but required manual adjustments in variable degrees.
Table 1. The effect of -value on detection recall and precision of cells in confluent cultures by using default region size settings/limited maximum region size. The annotation was performed on the first frame of sequence 1 and 2. The total number of cells (verified manually) in a frame is shown in parenthesis. Detection recall, total number of true positive cells divided by the total number of cells. Detection precision, the number of true positive cells divided by the total number of segmented regions. Experiment
Segmented regions
Segmented cells
Regions of merged cells
Detection recall%
Detection precision%
Sequence 1 (n = 1536)
1 3 5 1 3 5
1298/1712 1401/1459 1264/1283 1898/2800 1962/2614 2173/2579
864/1344 1181/1333 1127/1212 1075/2112 1410/2273 1729/2299
227/83 161/53 111/50 537/303 459/225 379/199
60/90 77/88 73/79 38/78 50/81 62/82
71/81 84/92 89/94 57/78 72/88 80/90
Sequence 2 (n = 2816)
C 2013 The Authors C 2013 Royal Microscopical Society, 253, 65–78 Journal of Microscopy
70
M. KAAKINEN ET AL.
Fig. 3. The comparison of MSER segmentation to other segmentation approaches. (A and I) The original input images were the first frames of sequence 1 and sequence 2. (B and J) Segmentation of the input image was performed with MSER (3 and limited maximum region size), (C and K) watershed with morphological erosion and median filtering and (D and L) Otsu threshold. Each detection is labelled with a different colour to help distinguishing the individual segmented objects. (E and M) Processed images from the first frame of sequence 1 and the first frame of sequence 2, respectively, using the image preconditioning technique. Segmentation of the processed image was performed with active contours (F and N), watershed with invert threshold (G and O) and Otsu threshold (H and P).
After the preprocessing steps, the morphological watershed segmentation worked moderately well (Fig. 3C and K, Table 2). Similar to MSER watershed transform practically segmented cell nuclei (Supporting Fig. S2). In terms of detection recall, watershed even outperformed MSER after preprocessing (81% and 87% for MSER and watershed, respectively). Simultaneously, however, the watershed segmented image became heavily over-segmented resulting in a poor detection precision when compared to MSER, 52% versus 88%. Active contours and Otsu threshold methods failed to segment cells reliably in the original phase-contrast input images. Otsu thresholding classified the halo artefacts surrounding the cells as a foreground and cells and the surrounding background as background (Fig. 3D and L), a shortcoming recognized also by Yin, Kanade & Chen (2012). This reduced the reliability as the extent of the halo varied between the individual cells and in different parts of cell border, and did not persisted in moving cells. It was shown previously that grey-scale bright field images produced using a differential interference contrast technique can be processed with the preconditioning technique to significantly improve the segmentation outcome (Li & Kanade, 2009). To improve the outcome of watershed, active contours and Otsu threshold segmentation approaches, we next
preprocessed the images using Li & Kanade’s technique. This technique produced an image in which cells were represented by bright pixels and the background appeared as uniformly black, thus implying that segmentation of cells will be easier. Accordingly, image preconditioning improved the detection recall and precision of the abovementioned segmentation approaches significantly (Table 2). The processing times of the techniques used at given image resolution (1376×1032) varied from
doi: 10.1111/jmi.12098
Received 29 April 2013; accepted 8 October 2013
Automatic detection and analysis of cell motility in phase-contrast time-lapse images using a combination of maximally stable extremal regions and Kalman filter approaches ¨ K I §, J . H E I K K I L A ¨‡ M . K A A K I N E N ∗ , †, S . H U T T U N E N †, ‡, L . P A A V O L A I N E N §, , V . M A R J O M A & L . E K L U N D ∗, # ∗ Biocenter Oulu, University of Oulu, Finland
†Mika Kaakinen and Sami Huttunen contributed equally to this work ‡Department of Computer Science and Engineering, University of Oulu, Finland §Department of Biological and Environmental Science, Nanoscience Center, University of Jyv¨askyl¨a, Finland Department of Mathematical Information Technology, University of Jyv¨askyl¨a, Finland #Oulu Center for Cell-Matrix Research, Department of Medical Biochemistry and Molecular Biology, Institute of Biomedicine, University of Oulu, Finland
Key words. Automatic cell segmentation, cell migration, Kalman filter, MSER, phase-contrast microscopy, tracking.
Summary Phase-contrast illumination is simple and most commonly used microscopic method to observe nonstained living cells. Automatic cell segmentation and motion analysis provide tools to analyze single cell motility in large cell populations. However, the challenge is to find a sophisticated method that is sufficiently accurate to generate reliable results, robust to function under the wide range of illumination conditions encountered in phase-contrast microscopy, and also computationally light for efficient analysis of large number of cells and image frames. To develop better automatic tools for analysis of low magnification phase-contrast images in time-lapse cell migration movies, we investigated the performance of cell segmentation method that is based on the intrinsic properties of maximally stable extremal regions (MSER). MSER was found to be reliable and effective in a wide range of experimental conditions. When compared to the commonly used segmentation approaches, MSER required negligible preoptimization steps thus dramatically reducing the computation time. To analyze cell migration characteristics in time-lapse movies, the MSER-based automatic cell detection was accompanied by a Kalman filter multiobject tracker that efficiently tracked individual cells even in confluent cell populations. This allowed quantitative cell motion analysis resulting in accurate measurements of the migration magnitude and direction of individual cells, as well as characteristics of collective migration Correspondence to: Lauri Eklund, Oulu Center for Cell-Matrix Research, Department of Medical Biochemistry and Molecular Biology, Institute of Biomedicine, University of Oulu, Finland P.O. Box 5000, FI-90014 Oulu, Finland. Tel: +358 294 486073; fax: +358 8 537 6115; e-mail: [email protected] C 2013 The Authors C 2013 Royal Microscopical Society Journal of Microscopy
of cell groups. Our results demonstrate that MSER accompanied by temporal data association is a powerful tool for accurate and reliable analysis of the dynamic behaviour of cells in phase-contrast image sequences. These techniques tolerate varying and nonoptimal imaging conditions and due to their relatively light computational requirements they should help to resolve problems in computationally demanding and often time-consuming large-scale dynamical analysis of cultured cells.
Introduction Phase-contrast microscopy is the most commonly used contrasting method to visualize living cells. This is due to the relatively simple configuration of the microscopy instruments, the resulting low costs, and the ability to observe cells without staining and the phototoxicity of the short wavelength light used in fluorescent microscopy. Manual tracking of thousands of cells over time, however, is not feasible, and automatic computer based cell segmentation and tracking approaches are necessary for quantitative studies. In contrast to relative ease of image production, automatic cell segmentation of phasecontrast images has been more challenging. Phase-contrast images of cells suffer from low contrast with respect to the background and there is variation in the pixel intensity distribution within the cells due to varying thicknesses, making identification of single cells difficult. Phase-contrasting also produces an artificial halo effect surrounding the cells that obscures cell interfaces. Several approaches have been developed for cell segmentation that are based on the separation of connected
66
M. KAAKINEN ET AL.
pixels belonging to the cell from those belonging to the background. Such methods include thresholding, the watershed method and texture analysis (Wu et al., 1995; Koyuncu et al., 2012; Korzynska et al., 2007). Alternatively, active contour algorithms that capture cell boundaries have been used (Tscherepanow et al., 2008; Wang, He & Metaxas, 2007; Ali et al., 2007, Seroussi et al., 2012). The shortcomings in commonly used techniques is the requirement for image preprocessing or the necessity to use accompanying techniques due to nonuniformities of pixel intensities inside the segmented cells, similarities in the background and specimen pixel intensities, or because the cell boundaries cannot be clearly resolved (Wu et al., 1995; Tse et al., 2009; Debeir et al., 2005; Ali ¨ et al., 2012). Manet al., 2007; Yin et al., 2012; Ambuhl ual adjustments and preprocessing techniques require expertise and their use significantly increases computational time. Therefore, reliable automatic analyzing techniques that are more feasible to use in varying conditions are urgently needed. In 2004 Matas et al. described a technique for detecting regions in an image that remain stable over a range of threshold values, called maximally stable extremal regions or MSERs (Matas et al., 2004). The MSER method has important characteristics, which are useful for the segmentation of objects from complex images. First, the segmented regions are preserved under geometric and monotonic intensity transformations. Secondly, MSER is not sensitive to pixel intensity changes and nonuniformities in background intensities as it is dependent only on the ordering of pixel intensities within the MSER and its outer boundary. The feasibility of MSER in detecting cultured cells was recently recognized in a machine learning based approach in which the detector was used to find candidate regions that represented putative cells (Arteta et al., 2012). In our study, we extended MSER approach to detect cells using a wide range of phase-contrast images and test the feasibility and the intrinsic properties of MSER for the automatic detection of cells under varying and challenging imaging conditions. In addition to automatic cell segmentation of single still images, we combined MSER with a Kalman filter based tracker modified for multiple objects (Huttunen & Heikkil¨a, 2008). This tracking approach uses segmentation masks as a source of measurements and utilizes soft assignment to associate the observations to the objects being tracked. The combination of MSER and a Kalman filter based tracking enabled accurate and reliable cell segmentation and migration analysis even in demanding dense cell populations. Because the computational requirements of these approaches were low and the required user interaction was minimal, the methods developed should be useful in computationally more demanding assays such as experiments performed in multiwell plate formats, analysis of living cells in high densities, and as modules in image analysis software.
Methods MSER MSER is a blob detector identifying regions in an image that remain stable over a certain number of thresholds. MSER was originally developed for stereo matching purposes, and later it has been widely used for object recognition. In MSER, maximum region size and stability range parameter (a delta value, ) control the segmentation sensitivity and accuracy. Maximum region size defines the upper limit for the size of the valid objects, in this work representing cell nuclei. The adjustment of maximum region size was done first by performing segmentation with default settings and then using a parameter value that corresponds to the mean region size multiplied by 1.25. The -value can be adjusted to set the range of threshold values within which the regions should remain stable (i.e. the region does not grow significantly over the selected threshold range); the higher the -value the broader is the range of threshold values within which the regions should remain stable. In phase-contrast images, the pixel intensity profile characterizing the cells often resembles that of background, which makes cell segmentation a challenging problem. Because cells and in particular cell nuclei are slightly darker than the background MSER could be considered as a potential method for this problem. Those cells that exhibit pixel “leaking” to background or to adjacent cells can be discarded and the risk of false detections can be reduced. In addition to the two abovementioned parameters of MSER, minimum region size, maximum variation and minimum diversity of the detected regions can also be adjusted. In our experiments we found that these parameters were not critical and the default values were used. The MSER was obtained from a VLFeat open source library (http://www.vlfeat.org/). Kalman filter Kalman filter is a common approach for object tracking as it can efficiently predict and filter the target locations based on previous observations and a dynamic state model. The main limitation of the Kalman filter is that it assumes Gaussian distribution for the observation noise. It would be possible to utilize the MSER regions directly as a source of location measurements in a Kalman filter based tracker but because of frequently occurring detection errors that do not usually follow Gaussian distribution this would easily lead to tracking failures. In order to alleviate this problem we have applied the multiobject tracking approach proposed by Huttunen & Heikkil¨a (2008) that is based on probabilistic data association where soft assignments of the measurements are used instead of hard assignments. The basic idea in this method is to detect the objects several times from the same frame with varying detector settings and compute soft assignments for each output. This will increase the robustness of tracking against
C 2013 The Authors C 2013 Royal Microscopical Society, 253, 65–78 Journal of Microscopy
AUTOMATIC DETECTION AND ANALYSIS OF CELL MOTILITY
67
Fig. 1. Measurement extraction for tracker. (A) An input image representing cultured squamous cell carcinoma cells. Six individual cells are numbered (1–6). (B) Cells segmented with MSER. Red contours outline the segmented areas and the green dots indicate the centroids of each detection. Note closely adjacent cells that were erroneously merged as a single detection (1–2 and 3–4–5). (C) The corresponding primary masks of the segmented cells. (D) Primary masks after a morphological erosion operation. Red crosses represent new centroids of the primary masks. Note that masks that erroneously merged cells 1–2 and 3–4–5 in initial MSER segmentation were correctly separated. However, the masks corresponding cells 5 and 6 were split in two after erosion operation. (E) After several morphological operations the primary masks may contain many centroids (measurements). The location of centroids is fitted into Gaussian mixture model. The centroids that locate far apart from the others (yellow arrowheads) less likely represent a true location and are down-weighted. (F) The resulting centroids are indicated by red crosses that accurately represent original cell nuclei.
detection errors. The method was originally developed for human tracking, but it can be modified for cell tracking purposes. In our approach we process the binary MSER regions with a filter bank that contains a set of morphological erosions. With this approach we can, for example, separate a binary region of two cells that have been erroneously merged by the MSER detector. As a result we get a large set of measurements (centroids of the binary regions) for each cell, and we assume that these measurements together form a Gaussian mixture, and its modes represent the correct cell locations. A Gaussian mixture model is used for deriving weights for the measurements that indicate the probability of belonging to an individual cell. The extraction for obtaining the measurements is demonstrated in Fig. 1. A probabilistic data association scheme is embedded to the Kalman filter framework to enable multiobject tracking. More details of the data association algorithm can be found from Huttunen & Heikkil¨a (2008).
Cell culture and microscopy Madin-Darby Canine Kidney (MDCK) cells were grown on sixwell plates (Costar, Corning Incorporated, New York, U.S.A.) in MEM+GlutaMAX media (Gibco, Life Technologies Corporation, U.S.A.) supplemented with 5% fetal bovine serum (FBS) and 1% penicillin-streptomycin. Squamous cell carcinoma cells (HSC-3) were kindly provided by M.D. Jyri Moilanen and grown in Dulbecco’s minimum essential medium (DMEM C 2013 The Authors C 2013 Royal Microscopical Society, 253, 65–78 Journal of Microscopy
F-12, Gibco) supplemented with 10% heat inactivated FBS and 1% penicillin-streptomycin. Human umbilical vein endothelial cells (HUVECs) were grown in endothelial cell basal medium supplemented with endothelial cell growth supplement (Cell Applications, San Diego, CA, U.S.A.) and 10% FBS. For the cell motility assay, a scratch was made on the middle of the confluent culture and cells were imaged overnight in a temperature and CO2 controlled microscope stage incubator (Okolab, Italy) mounted onto Olympus IX81 inverted microscope. Images consisting of 1376×1032 pixels were captured with a grey-scale camera (Olympus XM10, Germany) ¨ controlled by CellˆP software (Soft Imaging System, Munster, Germany).
Validation of the segmentation The reliability of MSER segmentation was verified manually by comparing the cells in the microscopic fields and the objects detected by MSER. Before manual annotation the connected component analysis was performed to judge whether closely located regions represent single or separate segmentation. Thereafter the contours of the binary masks of detected regions were overlaid on the original input image to distinguish correctly detected cells from other segmented regions (false positives). The manual counting was performed with Image J (National Institutes of Health, Bethesda, Maryland, U.S.A.) cell counter plug-in (Kurt De Vos, University of Sheffield, Academic Neurology, England). We denote the detection recall
68
M. KAAKINEN ET AL.
as the total number of true positive cells divided by the total number of cells in a frame. As detection precision, we denote the number of true positive cells divided by the total number of segmented regions. Other methods used in comparison The segmentation was performed using the commonly available segmentation approaches: Otsu thresholding, watershed and active contours. The morphological watershed segmentation was performed according to Beucher & Meyer (1993). Images were first preprocessed with a median filter to reduce noise and then with a morphological erosion operation to remove roughness inside the cells and to enhance contrast between the cells and the background. Next a gradient magnitude filter was applied. Finally, local minima close to local background were suppressed to prevent over-segmentation. After applying the watershed transform, all objects less than 30 pixels in size were removed. The active contours method was applied according to Caselles et al. (1997). The image was first smoothed with an anisotropic diffusion filter followed by calculation of gradient magnitude. Then the geodesic active contour level set was initialized by four user-selected seed points from which a circle of radius 375 pixels defined the initial zero level set. The front was defined to contract during 7000 iterations that led to splitting of four initial contours into many individual contours defining individual cells. The speed term for the front propagation was calculated from the gradient magnitude image using sigmoid function. Optionally, prior to applying the abovementioned segmentation approaches, images were first processed with an image preconditioning technique (http://www.kangli.org/page6.html#pcnddemo) described in Li & Kanade (2009). The image preconditioning aims to facilitate object segmentation. To obtain suitable images for watershed transform, the preconditioned images were preprocessed by setting all the pixel values less than 100 to a maximum intensity value (255) and the intensities of all pixels were inverted. This step was necessary to achieve local minima inside of the objects and to avoid overexpansion of the detected regions. Tracking data were also compared to the MTrack2 approach (Stuurman, University of California and the Howard Hughes Medical Institute), a plug-in of freely downloadable Fiji software (Schindelin et al., 2012). Segmentation of cells was first performed by using MSER. MTrack2 was then applied to the primary mask of segmented cells by defining a maximum velocity of tracked objects to correspond to 20 pixels/frame and minimum track length to 1/frame. In manual verification of the tracking data each track was followed until termination. The causes of lost tracks (the tracks that failed to proceed until the end of image sequence) were categorized in four groups: 1. Track failure in which the tracker loses the target due to segmentation failure of MSER or the tracker shifts to an adjacent object (i.e. cell) during the tracking process.
2. The cell leaves the image border. 3. Cell roundup due to death or mitosis. Both cell death and mitosis are characterized by an initial increase of circularity and pixel intensity and decrease in cell area. In cell division the initial features are reversed and daughter cells emerge. 4. Situations in which the identity (ID) of an object is changed during the tracking process.
Implementations The segmentation and tracking approaches were implemented using a PC with Intel Xeon CPU operating at 2.50 GHz and with 128 GB RAM. MATLAB was used for MSER, Otsu and Kalman filtering. BioImageXD (Kankaanp¨aa¨ et al., 2012) was used for watershed and active contours.
Results Segmentation of single cells in dense cell cultures based on MSER approach We first tested the effect of the -value on MSER detection recall and precision (Fig. 2 and Table 1) using challenging phase-contrast images obtained from scratched confluent cell cultures that represent the classic “wound closure” cell migration assay. The image sequences 1 and 2 were acquired by different microscope users representing different experimental conditions, cell densities and cell types. In the images analyzed, the pixel intensity inside the nuclei showed the most prominent contrast in relation to the background and was therefore mainly detected by MSER (Fig. 2). Notably, when the default region size was used, -value adjustment markedly contributed to detection outcome; the smallest values resulted in a higher number of false positive detections as well as artificial merging of single cells (Supporting Fig. S1, Table 1), whereas the detection improved with increasing -values. Merging of several cells within a single detection (Fig. 1 and Supporting Fig. S1) was the most common failure, which decreased the detection result. To reduce the incidence of merged regions we next limited the maximum region size. As a maximum value we used the average region size at -value 3 that mostly represented the cell nuclei (red contours in Fig. 2). As shown in Table 1, limiting the maximum region size improved the detection precision in both sequences, more profoundly in sequence 2. These observations could be explained by the fact that with increasing -values the segmentation becomes less sensitive and, if the pixel intensity variation is small within cells and the regions between the cells, the region size is expanded until it fulfils the criteria of maximally stable region. Adjustment of additional three parameters available (minimum region size, maximum variation and minimum diversity) did not improve the segmentation result of MSER. C 2013 The Authors C 2013 Royal Microscopical Society, 253, 65–78 Journal of Microscopy
AUTOMATIC DETECTION AND ANALYSIS OF CELL MOTILITY
69
Table 2. Comparison of MSER performance with three common segmentation approaches. The MSER 3 and limited maximum region size were used for comparison. The techniques indicated in the table were applied to the original grey-scale image representing the first frame of sequence 1 and 2. Alternatively, an image preconditioning (preconditioned, Li & Kanade, 2009) technique was used to improve contrast in images before applying a segmentation technique. Active contours and Otsu threshold could not be applied to the original input image to result in a reasonable outcome (-). Sequence 1 Segmentation approach MSER 3 Watershed (original) Watershed (preconditioned) Active contours (original) Active contours (preconditioned) Otsu threshold (original) Otsu threshold (preconditioned) Sequence 2 Segmentation approach MSER 3 Watershed (original) Watershed (preconditioned) Active contours (original) Active contours (preconditioned) Otsu threshold (original) Otsu threshold (preconditioned) Fig. 2. First frames of image sequences obtained from wounded confluent squamous cell carcinoma cell cultures (sequence 1, panel A) and MDCK cells (sequence 2, panel B) were used to test the effect of different -values on MSER segmentation with limited maximum region size. Red contours outline the segmented areas most often representing cell nuclei. Green dots indicate the centroids of each detection to distinguish contours that belong to the individual cells. Examples of segmented regions are magnified in the insets at the top right corner of each MSER segmented image. Note that with smaller -values more noncell objects (white arrowheads, exemplified also with higher magnification in the Supporting Fig. S1) are detected in the wound area.
Recall%
Precision%
88 51 88 − 48 − 66
92 74 97 − 91 − 97
Recall%
Precision%
81 87 89 − 50 − 73
88 52 96 − 93 − 60
Comparison of MSER with commonly used segmentation approaches To compare the cell detection recall and precision of MSER to the commonly used segmentation approaches, the first frames of sequence 1 and sequence 2 were segmented using the Otsu threshold, active contours, and watershed, the techniques that are commonly accessible and which have been applied to phase-contrast images (Table 2). Notably, in contrast to MSER, none of the other techniques could be applied directly to the original grey-scale images, but required manual adjustments in variable degrees.
Table 1. The effect of -value on detection recall and precision of cells in confluent cultures by using default region size settings/limited maximum region size. The annotation was performed on the first frame of sequence 1 and 2. The total number of cells (verified manually) in a frame is shown in parenthesis. Detection recall, total number of true positive cells divided by the total number of cells. Detection precision, the number of true positive cells divided by the total number of segmented regions. Experiment
Segmented regions
Segmented cells
Regions of merged cells
Detection recall%
Detection precision%
Sequence 1 (n = 1536)
1 3 5 1 3 5
1298/1712 1401/1459 1264/1283 1898/2800 1962/2614 2173/2579
864/1344 1181/1333 1127/1212 1075/2112 1410/2273 1729/2299
227/83 161/53 111/50 537/303 459/225 379/199
60/90 77/88 73/79 38/78 50/81 62/82
71/81 84/92 89/94 57/78 72/88 80/90
Sequence 2 (n = 2816)
C 2013 The Authors C 2013 Royal Microscopical Society, 253, 65–78 Journal of Microscopy
70
M. KAAKINEN ET AL.
Fig. 3. The comparison of MSER segmentation to other segmentation approaches. (A and I) The original input images were the first frames of sequence 1 and sequence 2. (B and J) Segmentation of the input image was performed with MSER (3 and limited maximum region size), (C and K) watershed with morphological erosion and median filtering and (D and L) Otsu threshold. Each detection is labelled with a different colour to help distinguishing the individual segmented objects. (E and M) Processed images from the first frame of sequence 1 and the first frame of sequence 2, respectively, using the image preconditioning technique. Segmentation of the processed image was performed with active contours (F and N), watershed with invert threshold (G and O) and Otsu threshold (H and P).
After the preprocessing steps, the morphological watershed segmentation worked moderately well (Fig. 3C and K, Table 2). Similar to MSER watershed transform practically segmented cell nuclei (Supporting Fig. S2). In terms of detection recall, watershed even outperformed MSER after preprocessing (81% and 87% for MSER and watershed, respectively). Simultaneously, however, the watershed segmented image became heavily over-segmented resulting in a poor detection precision when compared to MSER, 52% versus 88%. Active contours and Otsu threshold methods failed to segment cells reliably in the original phase-contrast input images. Otsu thresholding classified the halo artefacts surrounding the cells as a foreground and cells and the surrounding background as background (Fig. 3D and L), a shortcoming recognized also by Yin, Kanade & Chen (2012). This reduced the reliability as the extent of the halo varied between the individual cells and in different parts of cell border, and did not persisted in moving cells. It was shown previously that grey-scale bright field images produced using a differential interference contrast technique can be processed with the preconditioning technique to significantly improve the segmentation outcome (Li & Kanade, 2009). To improve the outcome of watershed, active contours and Otsu threshold segmentation approaches, we next
preprocessed the images using Li & Kanade’s technique. This technique produced an image in which cells were represented by bright pixels and the background appeared as uniformly black, thus implying that segmentation of cells will be easier. Accordingly, image preconditioning improved the detection recall and precision of the abovementioned segmentation approaches significantly (Table 2). The processing times of the techniques used at given image resolution (1376×1032) varied from
