Automatic Detection of Cell Divisions (Mitosis) in Live-Imaging Microscopy Images using Convolutional Neural Networks Anat Shkolyar1 , Amit Gefen2 , Dafna Benayahu3 and Hayit Greenspan1 Abstract— We propose a semi-automated pipeline for the detection of possible cell divisions in live-imaging microscopy and the classification of these mitosis candidates using a Convolutional Neural Network (CNN). We use time-lapse images of NIH3T3 scratch assay cultures, extract patches around bright candidate regions that then undergo segmentation and binarization, followed by a classification of the binary patches into either containing or not containing cell division. The classification is performed by training a Convolutional Neural Network on a specially constructed database. We show strong results of AUC = 0.91 and F-score = 0.89, competitive with state-of-the-art methods in this field.
I. INTRODUCTION The detection and tracking of cell proliferation (Mitosis) is important in both cancer and stem cell research. In cancer research, recent methods [1], [2], [3] use Convolutional Neural Networks (CNN) to detect mitotic nuclei in Hematoxylin and Eosin (H&E) stained tissue images with great success. However, H&E staining involves the binding of dyes to the tissue [4], preventing further tissue development, and thus not allowing for continuous live-imaging. In stem cell research cell divisions are detected and tracked using probabilistic models in phase-contrast microscopy images [5], [6]. Phasecontrast and light microscopy do not require any dyes or other possibly cytotoxic compounds, thus allowing for liveimaging of cell cultures for hours at a time. One of the simplest experimental setups is the scratch assay [7], where cells are cultivated in a high-confluence monolayer, some of the cells are removed by a physical intervention (creating a ’scratch’, ’gap’ or ’wound’), and the culture is then left to proliferate under a microscope while the cells migrate and proliferate, closing the artificial ’wound’. Automatic methods for tracking cell migration in scratch assays have been developed recently [8], however, we are not aware of any automatic method for quantifying cell proliferation in scratch assays. In the current work, we propose the use of CNNs for automatic detection and quantification of Mitosis events occurring during a scratch assay. Our proposed method relies on the changes in cell morphology and brightness as it prepares for mitosis. These changes help in the detection of mitosis candidates, around which we extract grayscale patches (section II-B). The *This work was supported by The Israel Science Foundation (ISF). **Corresponding Author: Prof. Hayit Greenspan, [email protected] 1 Medical Image Processing Lab, Department of Biomedical Engineering, Faculty of Engineering, Tel Aviv University, Israel 2 Musculoskeletal Biomechanics Lab, Department of Biomedical Engineering, Faculty of Engineering, Tel Aviv University, Israel 3 Department of Cell And Developmental Biology, School of Medicine, Tel Aviv University, Israel
978-1-4244-9270-1/15/$31.00 ©2015 IEEE
patches undergo segmentation and binarization, resulting in binary patches with a typical shape for cells undergoing mitosis (section II-D). These binary patches are used to train the CNN and create an automatic classifier. II. M ETHODS A. Data Collection We used time-lapse microscopy images of NIH3T3 scratch assay cultures. The cells are cultivated to reach high confluence, then, a ’wound’ is created at a localized area in these cultures by crushing cells with a micro-indenter. The ’wound’ area is tracked until full closure, which takes about a day. The full method is described in [8]. The ’wound’ closure process happens due to both cell migration (movement) and proliferation (mitosis - the creation of new cells by cell division) and can be seen in Fig. 1.
Fig. 1. The progression of ’wound healing’ in scratch assay. (a) The culture after ’wound’ infliction, (b-d) ’Wound’ begins to close as cells proliferate and migrate into the ’wound’ area.
B. Mitosis Candidate Detection In regular conditions, a cell adheres to the surface and appears as a dark elongated object with a bright halo. As a cell prepares for mitosis, it detaches from the surface and becomes rounder and brighter, as seen in Fig. 2.
Fig. 2. An adhered cell (a) is detaching from the surface (b-c), becoming rounder and brighter, until it is non-adhered (d).
The brightness characteristic can be used for cell detection (e.g. [5]). The detection of the bright regions is performed by first highlighting and then isolating the brightest regions in the image. First, the color (RGB) image is converted into the YCB CR color-space. In this color-space, instead of each
743
channel indicating the intensity of a certain pixel in 3 primary colors, the channels indicate the intensity or brightness (Y), the blue-difference chrominance (CB ) and the red-difference chrominance (CR ). This representation assists in highlighting the bright regions in the image by subtracting the CB channel from the Y channel, as seen in Fig. 3(c). The non-zero values highlight both the non-adhered cells and the borders between adjacent cells, creating an image which we label as Bright Border image.
filled, resulting in a binary image (Fig. 3(e)) - a mask of the brightest regions in the original image. The centroids of the bright ’blobs’ in the binary image represent the centroids of the brightest regions of the grayscale image, marked in red circles in (Fig. 3(f)). Following the enhancement process, we have detected the centroids of the brightest regions in the frame. Each centroid is a candidate for a non-adhered cell approaching mitosis. C. Tracking Mitosis Candidates
Fig. 3. Top: Using YCB CR representation to highlight bright regions. (a) RGB (color) image, (b) Grayscale image, (c) Bright Border image (Y-CB ), Bottom: Detection of brightest regions in image. (d) Bright Border image after enhancement, thresholding, filtering; (e) Binary image, following image opening and hole filling; (f) Detected Bright regions (in red) on Grayscale image.
The Bright Border image highlights many bright regions, however, we are only interested in bright cells and not in bright regions that result from cell borders. The non-adhered cells appear brighter than the cell borders, and so, to further isolate the brightest regions, we enhance the Bright Border image. As a first step we apply a sigmoid-shaped function to the intensity levels of the image. This function further diminishes the lower intensity values while amplifying the high intensity values (see Fig. 4, for the equation and the graph of the sigmoid-shaped function).
After detecting the Mitosis candidates (bright region centroids) described in section II-B, each of the Mitosis candidates is tracked between consecutive frames. First, a 32x32 patch is opened around each of the bright ’blob’ centroids. Then, two consecutive patches are examined - if centroid j in frame x+1 falls within the patch of centroid i in frame x, then the patch surrounding centroid j is appended to the patch surrounding centroid i, constructing a time-lapse sequence of patches. As can be seen in Fig. 5, Mitosis sequences include at least one patch where a single non-adhered cell becomes two non-adhered cells (last 3 patches in Fig. 5(a) and Fig. 5(b)). These patches of cell division are unique to Mitosis sequences.
Fig. 5. Examples for sequences: (a-b) Mitosis sequences; (c-d) Non-Mitosis Sequences.
D. Detecting Cell Division Patches containing cell division are preprocessed to highlight the dividing cell, converted into binary frames containing a mask of the dividing cell, then a CNN is trained to detect binary patches that contain cell division. 1) Preprocessing: To isolate the dividing cell from background, each patch is segmented using K-means on the (x,y,I) values of each pixel (K=4). Then, the brightest cluster is isolated, resulting in a binary mask of the dividing cell borders (Fig. 6). The resulting binary frames are then input to a CNN to classify into the 2 categories of Mitosis (cell division) or non-Mitosis. The CNN classification process is described in the following sections.
Fig. 4. Left: Equation. The parameters a and b are assigned the following values: b = average value of non-zero intensity levels in Bright Border image and a = 0.5b; Right: Graph. a and b are marked in red.
After enhancement, the Bright Border image undergoes thresholding (using Otsu’s method) and filtering (3x3 median filter). The resulting binary image (Fig. 3(d)) highlights the brightest regions, resulting both from non-adhered cells and from cell borders. Next, the image undergoes image opening (with a disk-shaped structural element, R=3), removing small non-disk-shaped objects, which discards the majority of cell borders. Then, any holes left in the remaining object are
Fig. 6. Patch segmentation and binarization. Mitosis Sequence (a-c); Non-Mitosis sequence (d-f). Top: Grayscale, Middle: Segmented, Bottom: Binary.
744
2) Training and Testing Databases (Binary Patches): As a first step to a establish Ground Truth (GT) database, an expert labels each patch sequence (Fig. 5) as either Mitosis or Non-Mitosis, establishing a database of labeled sequences. Then, the sequences are divided into separate patches, and all patches undergo segmentation and binarization (Fig. 6). The binary patches are then labeled as either containing cell division, or not, establishing a database of labeled binary patches. From this point on, all the datasets contain patches only. The labeled binary database is then divided into balanced Training and Testing sets - first, all available cell division patches (positive examples) are divided between Training (≈ 80%) and Testing (≈ 20%) sets, then a random sampling of negative examples is added to each of the datasets, in the same amount as the positive examples (thus, the size of the datasets is limited by the number of available positive examples). In order to increase the size of the Training dataset, each one of the original positive example binary patches undergoes mirroring and 45° rotations. Next, all the patches in the augmented dataset are centered by shifting the centroid of the object to the center of the patch (see Fig. 7). Finally, an additional random sampling of negative examples (binary frames not containing cell division) is added in the same amount as the total positive examples.
Unlike the Training set, the Testing set is not augmented by mirroring and rotations, only centered. The Testing set consists of 244 binary patches, half being positive examples and half negative. 3) Classifying using a Convolutional Neural Network (CNN): Each of the Training sets described in section IID.2 is used to train a Convolutional Neural Network (CNN) using the MatConvNet package [10]. Due to the similarity of the binary cell division patches to the MNIST data, the selected CNN architecture was the architecture described in the MatConvNet MNIST example, which is similar to LeNet5 described in [9], see CNN illustration in Fig. 9. In instances where the Training set contains Mitosis patches (as opposed to augmented MNIST data), the patches are resized from 32x32 to 28x28 pixels and the CNN architecture is slightly modified to provide an output layer of 2 neurons (probability of cell division vs. no cell division) instead of 10 neurons (probability of each digit 0-9).
Fig. 9.
Fig. 7. Augmentation of a single binary patch. Grayscale patch (top left); Segmented patch (bottom left); Mirrored and rotated binary patches.
The binary patches of the positive examples (cell division patches) contain two hollow connected circles. This shape is similar to the digit 8 in different orientations (Fig. 8). This similarity led to utilizing an additional Training set, the MNIST dataset [9]. The MNIST dataset was also augmented, creating a total of 5 labeled training sets of different size and contents, as listed in Table I.
Fig. 8. Examples of binary cell division patches. Note the similarity to the digit 8 in different orientations.
TABLE I T RAINING DATASETS S IZES Training Dataset Mitosis MNIST
Balanced 1.1K 70K
Mirroring, 90°rotation 9K 560K
Mirroring, 45°rotation 13K -
As mentioned above, the Testing set consists of ≈ 20% of the available cell division patches (positive examples).
Illustration of CNN architecture.
III. R ESULTS Using the CNN classifier, each test patch is assigned a probability for belonging to each of the classes. For CNN trained on Mitosis data there are 2 possible classes and for CNN trained on MNIST data there are 10 possible classes (as detailed in section II-D.3). To convert this probability vector into a discrete label, each patch is labeled according to the highest assigned probability. For example, patch i with probability vector = [0.01 0.02 0.03 0.04 0.05 0.03 0.02 0.02 0.7 0.08] is assigned the label ’8’ due to the highest probability being in index 9, which corresponds to the digit 8. When training the CNN on Mitosis data, the assigned probabilities vector only contains 2 probabilities. For example, patch i with probability vector = [0.34 0.66] is assigned the label ’1’ due to the highest probability being in index 2, which corresponds to the class containing cell division. The assigned label is then compared to the Ground Truth (GT) label, which allows for quantification of the CNN performance. In Table II, the Sensitivity, Specificity, Area Under Curve (AUC) and F-score are shown for CNNs trained on different datasets. Dataset (1) is the augmented MNIST dataset, while (2)-(5) are Mitosis datasets, created from the cell culture images (as detailed in section II-D.2). The first CNN was trained using the initial, smallest dataset (2) and while the Sensitivity, AUC and F-score were reasonable, the Specificity
745
was very low. The next CNN was trained on a larger, augmented and centered dataset (3), showing a slight improvement in Specificity and F-score. Dataset (4) was the same as (3), however, the Testing dataset was also centered, to match the centering on the Training set, resulting in a noticeable improvement in Specificity, AUC and F-score. The final dataset (5) contained additional augmentation, adding 45° rotations to the previous, 90° rotations. This largest Training set was also tested with a centered Testing set, providing the highest F-score, while keeping Sensitivity, Specificity and AUC roughly the same as dataset (4). TABLE II R ESULTS #
Training Dataset
(1)
MNIST Mirroring, 90° Rotation, Centered (560K) Mitosis (1.1K) Mitosis Mirroring, 90° Rotation, Centered (9K) Mitosis Mirroring, 90° Rotation, Centered (9K) Mitosis Mirroring, 45° Rotation, Centered (13K)
(2) (3)
(4)
(5)
Mitosis Testing Dataset Balanced, Centered
Sens.
Spec.
AUC
FScore
0.68
0.89
0.88
0.76
Balanced Balanced
0.98 1.00
0.23 0.23
0.75 0.64
0.71 0.72
Balanced, Centered
0.81
0.98
0.92
0.88
Balanced, Centered
0.87
0.93
0.91
0.89
The results in Table II are compared to the more classical method of feature extraction from grayscale patches followed by a Support Vector Machine (SVM) classification. First, the grayscale patches are centered and rotated, similar to the centering and augmentation process of the binary patches described in section II-D.2. Then, the grayscale patches undergo feature extraction. The extracted features include mean, median and standard deviation of the gray levels in each patch, mean, median and standard deviation of the magnitude and phase response to Gabor filters in 6 orientations and 4 scales and unique scale gradient histogram features [5] - for a total feature vector of 211 elements. After extracting all features from the Training set, the feature vectors undergo Principal Component Analysis (PCA), and the SVM is trained using the top principal components. The optimal number of top principal components (PC) was found to be 4. The Testing set was then classified using the trained SVM. A comparison between the SVM and CNN classification results can be seen in Table III. The results indicate a significant increase in performance in F-score when using the CNN. TABLE III C OMPARISON T O SVM Classification Method SVM (4 PC) CNN
Sensitivity 0.67 0.86
Specificity 0.94 0.92
F-Score 0.78 0.89
IV. D ISCUSSION & C ONCLUSIONS We have demonstrated a novel method for the automatic detection of cell division in scratch assays. Overall, we show that even with a rather small initial Training set (1.1K patches, before augmentation, 13K after full augmentation), a CNN can be trained to identify binary patches containing cell division with a high success rate. Our results show an AUC of 0.91 and an F-score of 0.89, comparable to stateof-the-art methods in the field. It should be noted than the live-imaging microscopy data used in this work varies greatly from the H&E stained tissue of [1], [2], [3]. In [5], Huh et al. use phase-contrast microscopy data, resulting in similar images, however, our data differs both in the type of cells used and in cell confluence. In [5] and the follow-up work [6] the cell cultures are of a rather low confluence and the analyzed patches often contain only one or two cells. In our data, due to the nature of the scratch assay, the cultures reached a high confluence, resulting in patches with much higher cell density, adding noise and artifacts. Future work entails improvement of the results by implementing a cascaded ensemble of classifiers, as described in [2]. Furthermore, the labeled binary patches will be used to classify the entire patch sequence as Mitosis or non-Mitosis, which will provide an additional layer of information for existing automated methods of scratch assay analysis, giving additional insights into cancer and stem cell research. R EFERENCES [1] M. Veta, P.J. van Diest, S.M Willems, H. Wang, A. Madabhushi, A. Cruz-Roa, F. Gonzalez, A.B.L Larsen, J.S. Vestergaard, A.B. Dahl, and others, Assessment of algorithms for mitosis detection in breast cancer histopathology images, in Medical Image Analysis, 2015, vol. 20(1), pp 237-248. [2] H. Wang, A. Cruz-Roa, A. Basavanhally, H. Gilmore, N. Shih, M. Feldman, J. Tomaszewski, F. Gonzalez, and A. Madabhushi, Cascaded ensemble of convolutional neural networks and handcrafted features for mitosis detection, in SPIE Medical Imaging 2014, pp 90410B90410B. [3] D.C. Cires¸an, A. Giusti, L.M. Gambardella and J. Schmidhuber, Mitosis detection in breast cancer histology images with deep neural networks, in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2013, pp 411-418. [4] G. Avwioro, Histochemical uses of haematoxylin - a review, in J Pharm Clin Sci (JPCS), vol. 1, pp 24-34, 2011. [5] S. Huh, D.F.E Ker, R. Bise, M. Chen, and T. Kanade, Automated mitosis detection of stem cell populations in phase-contrast microscopy images, in IEEE Transactions on Medical Imaging, 2011, vol. 30(3), pp 586-596. [6] S. Huh and M. Chen, Detection of mitosis within a stem cell population of high cell confluence in phase-contrast microscopy images, in 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1033-1040. [7] C. C. Liang, A. Y. Park, and J. L. Guan, In vitro scratch assay: a convenient and inexpensive method for analysis of cell migration in vitro, in Nature protocols 2007, vol 2(2), pp 329-333. [8] G. Topman, O. Sharabani-Yosef, Orna and A. Gefen, A standardized objective method for continuously measuring the kinematics of cultures covering a mechanically damaged site, in Medical engineering & physics, 2012, vol. 34(2), pp 225-232. [9] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition, in Proceedings of the IEEE vol. 83(11), pp 2278-2324, 1998. [10] A. Vedaldi and K. Lenc, MatConvNet-Convolutional Neural Networks for MATLAB, arXiv:1412.4564, 2014.
746
I. INTRODUCTION The detection and tracking of cell proliferation (Mitosis) is important in both cancer and stem cell research. In cancer research, recent methods [1], [2], [3] use Convolutional Neural Networks (CNN) to detect mitotic nuclei in Hematoxylin and Eosin (H&E) stained tissue images with great success. However, H&E staining involves the binding of dyes to the tissue [4], preventing further tissue development, and thus not allowing for continuous live-imaging. In stem cell research cell divisions are detected and tracked using probabilistic models in phase-contrast microscopy images [5], [6]. Phasecontrast and light microscopy do not require any dyes or other possibly cytotoxic compounds, thus allowing for liveimaging of cell cultures for hours at a time. One of the simplest experimental setups is the scratch assay [7], where cells are cultivated in a high-confluence monolayer, some of the cells are removed by a physical intervention (creating a ’scratch’, ’gap’ or ’wound’), and the culture is then left to proliferate under a microscope while the cells migrate and proliferate, closing the artificial ’wound’. Automatic methods for tracking cell migration in scratch assays have been developed recently [8], however, we are not aware of any automatic method for quantifying cell proliferation in scratch assays. In the current work, we propose the use of CNNs for automatic detection and quantification of Mitosis events occurring during a scratch assay. Our proposed method relies on the changes in cell morphology and brightness as it prepares for mitosis. These changes help in the detection of mitosis candidates, around which we extract grayscale patches (section II-B). The *This work was supported by The Israel Science Foundation (ISF). **Corresponding Author: Prof. Hayit Greenspan, [email protected] 1 Medical Image Processing Lab, Department of Biomedical Engineering, Faculty of Engineering, Tel Aviv University, Israel 2 Musculoskeletal Biomechanics Lab, Department of Biomedical Engineering, Faculty of Engineering, Tel Aviv University, Israel 3 Department of Cell And Developmental Biology, School of Medicine, Tel Aviv University, Israel
978-1-4244-9270-1/15/$31.00 ©2015 IEEE
patches undergo segmentation and binarization, resulting in binary patches with a typical shape for cells undergoing mitosis (section II-D). These binary patches are used to train the CNN and create an automatic classifier. II. M ETHODS A. Data Collection We used time-lapse microscopy images of NIH3T3 scratch assay cultures. The cells are cultivated to reach high confluence, then, a ’wound’ is created at a localized area in these cultures by crushing cells with a micro-indenter. The ’wound’ area is tracked until full closure, which takes about a day. The full method is described in [8]. The ’wound’ closure process happens due to both cell migration (movement) and proliferation (mitosis - the creation of new cells by cell division) and can be seen in Fig. 1.
Fig. 1. The progression of ’wound healing’ in scratch assay. (a) The culture after ’wound’ infliction, (b-d) ’Wound’ begins to close as cells proliferate and migrate into the ’wound’ area.
B. Mitosis Candidate Detection In regular conditions, a cell adheres to the surface and appears as a dark elongated object with a bright halo. As a cell prepares for mitosis, it detaches from the surface and becomes rounder and brighter, as seen in Fig. 2.
Fig. 2. An adhered cell (a) is detaching from the surface (b-c), becoming rounder and brighter, until it is non-adhered (d).
The brightness characteristic can be used for cell detection (e.g. [5]). The detection of the bright regions is performed by first highlighting and then isolating the brightest regions in the image. First, the color (RGB) image is converted into the YCB CR color-space. In this color-space, instead of each
743
channel indicating the intensity of a certain pixel in 3 primary colors, the channels indicate the intensity or brightness (Y), the blue-difference chrominance (CB ) and the red-difference chrominance (CR ). This representation assists in highlighting the bright regions in the image by subtracting the CB channel from the Y channel, as seen in Fig. 3(c). The non-zero values highlight both the non-adhered cells and the borders between adjacent cells, creating an image which we label as Bright Border image.
filled, resulting in a binary image (Fig. 3(e)) - a mask of the brightest regions in the original image. The centroids of the bright ’blobs’ in the binary image represent the centroids of the brightest regions of the grayscale image, marked in red circles in (Fig. 3(f)). Following the enhancement process, we have detected the centroids of the brightest regions in the frame. Each centroid is a candidate for a non-adhered cell approaching mitosis. C. Tracking Mitosis Candidates
Fig. 3. Top: Using YCB CR representation to highlight bright regions. (a) RGB (color) image, (b) Grayscale image, (c) Bright Border image (Y-CB ), Bottom: Detection of brightest regions in image. (d) Bright Border image after enhancement, thresholding, filtering; (e) Binary image, following image opening and hole filling; (f) Detected Bright regions (in red) on Grayscale image.
The Bright Border image highlights many bright regions, however, we are only interested in bright cells and not in bright regions that result from cell borders. The non-adhered cells appear brighter than the cell borders, and so, to further isolate the brightest regions, we enhance the Bright Border image. As a first step we apply a sigmoid-shaped function to the intensity levels of the image. This function further diminishes the lower intensity values while amplifying the high intensity values (see Fig. 4, for the equation and the graph of the sigmoid-shaped function).
After detecting the Mitosis candidates (bright region centroids) described in section II-B, each of the Mitosis candidates is tracked between consecutive frames. First, a 32x32 patch is opened around each of the bright ’blob’ centroids. Then, two consecutive patches are examined - if centroid j in frame x+1 falls within the patch of centroid i in frame x, then the patch surrounding centroid j is appended to the patch surrounding centroid i, constructing a time-lapse sequence of patches. As can be seen in Fig. 5, Mitosis sequences include at least one patch where a single non-adhered cell becomes two non-adhered cells (last 3 patches in Fig. 5(a) and Fig. 5(b)). These patches of cell division are unique to Mitosis sequences.
Fig. 5. Examples for sequences: (a-b) Mitosis sequences; (c-d) Non-Mitosis Sequences.
D. Detecting Cell Division Patches containing cell division are preprocessed to highlight the dividing cell, converted into binary frames containing a mask of the dividing cell, then a CNN is trained to detect binary patches that contain cell division. 1) Preprocessing: To isolate the dividing cell from background, each patch is segmented using K-means on the (x,y,I) values of each pixel (K=4). Then, the brightest cluster is isolated, resulting in a binary mask of the dividing cell borders (Fig. 6). The resulting binary frames are then input to a CNN to classify into the 2 categories of Mitosis (cell division) or non-Mitosis. The CNN classification process is described in the following sections.
Fig. 4. Left: Equation. The parameters a and b are assigned the following values: b = average value of non-zero intensity levels in Bright Border image and a = 0.5b; Right: Graph. a and b are marked in red.
After enhancement, the Bright Border image undergoes thresholding (using Otsu’s method) and filtering (3x3 median filter). The resulting binary image (Fig. 3(d)) highlights the brightest regions, resulting both from non-adhered cells and from cell borders. Next, the image undergoes image opening (with a disk-shaped structural element, R=3), removing small non-disk-shaped objects, which discards the majority of cell borders. Then, any holes left in the remaining object are
Fig. 6. Patch segmentation and binarization. Mitosis Sequence (a-c); Non-Mitosis sequence (d-f). Top: Grayscale, Middle: Segmented, Bottom: Binary.
744
2) Training and Testing Databases (Binary Patches): As a first step to a establish Ground Truth (GT) database, an expert labels each patch sequence (Fig. 5) as either Mitosis or Non-Mitosis, establishing a database of labeled sequences. Then, the sequences are divided into separate patches, and all patches undergo segmentation and binarization (Fig. 6). The binary patches are then labeled as either containing cell division, or not, establishing a database of labeled binary patches. From this point on, all the datasets contain patches only. The labeled binary database is then divided into balanced Training and Testing sets - first, all available cell division patches (positive examples) are divided between Training (≈ 80%) and Testing (≈ 20%) sets, then a random sampling of negative examples is added to each of the datasets, in the same amount as the positive examples (thus, the size of the datasets is limited by the number of available positive examples). In order to increase the size of the Training dataset, each one of the original positive example binary patches undergoes mirroring and 45° rotations. Next, all the patches in the augmented dataset are centered by shifting the centroid of the object to the center of the patch (see Fig. 7). Finally, an additional random sampling of negative examples (binary frames not containing cell division) is added in the same amount as the total positive examples.
Unlike the Training set, the Testing set is not augmented by mirroring and rotations, only centered. The Testing set consists of 244 binary patches, half being positive examples and half negative. 3) Classifying using a Convolutional Neural Network (CNN): Each of the Training sets described in section IID.2 is used to train a Convolutional Neural Network (CNN) using the MatConvNet package [10]. Due to the similarity of the binary cell division patches to the MNIST data, the selected CNN architecture was the architecture described in the MatConvNet MNIST example, which is similar to LeNet5 described in [9], see CNN illustration in Fig. 9. In instances where the Training set contains Mitosis patches (as opposed to augmented MNIST data), the patches are resized from 32x32 to 28x28 pixels and the CNN architecture is slightly modified to provide an output layer of 2 neurons (probability of cell division vs. no cell division) instead of 10 neurons (probability of each digit 0-9).
Fig. 9.
Fig. 7. Augmentation of a single binary patch. Grayscale patch (top left); Segmented patch (bottom left); Mirrored and rotated binary patches.
The binary patches of the positive examples (cell division patches) contain two hollow connected circles. This shape is similar to the digit 8 in different orientations (Fig. 8). This similarity led to utilizing an additional Training set, the MNIST dataset [9]. The MNIST dataset was also augmented, creating a total of 5 labeled training sets of different size and contents, as listed in Table I.
Fig. 8. Examples of binary cell division patches. Note the similarity to the digit 8 in different orientations.
TABLE I T RAINING DATASETS S IZES Training Dataset Mitosis MNIST
Balanced 1.1K 70K
Mirroring, 90°rotation 9K 560K
Mirroring, 45°rotation 13K -
As mentioned above, the Testing set consists of ≈ 20% of the available cell division patches (positive examples).
Illustration of CNN architecture.
III. R ESULTS Using the CNN classifier, each test patch is assigned a probability for belonging to each of the classes. For CNN trained on Mitosis data there are 2 possible classes and for CNN trained on MNIST data there are 10 possible classes (as detailed in section II-D.3). To convert this probability vector into a discrete label, each patch is labeled according to the highest assigned probability. For example, patch i with probability vector = [0.01 0.02 0.03 0.04 0.05 0.03 0.02 0.02 0.7 0.08] is assigned the label ’8’ due to the highest probability being in index 9, which corresponds to the digit 8. When training the CNN on Mitosis data, the assigned probabilities vector only contains 2 probabilities. For example, patch i with probability vector = [0.34 0.66] is assigned the label ’1’ due to the highest probability being in index 2, which corresponds to the class containing cell division. The assigned label is then compared to the Ground Truth (GT) label, which allows for quantification of the CNN performance. In Table II, the Sensitivity, Specificity, Area Under Curve (AUC) and F-score are shown for CNNs trained on different datasets. Dataset (1) is the augmented MNIST dataset, while (2)-(5) are Mitosis datasets, created from the cell culture images (as detailed in section II-D.2). The first CNN was trained using the initial, smallest dataset (2) and while the Sensitivity, AUC and F-score were reasonable, the Specificity
745
was very low. The next CNN was trained on a larger, augmented and centered dataset (3), showing a slight improvement in Specificity and F-score. Dataset (4) was the same as (3), however, the Testing dataset was also centered, to match the centering on the Training set, resulting in a noticeable improvement in Specificity, AUC and F-score. The final dataset (5) contained additional augmentation, adding 45° rotations to the previous, 90° rotations. This largest Training set was also tested with a centered Testing set, providing the highest F-score, while keeping Sensitivity, Specificity and AUC roughly the same as dataset (4). TABLE II R ESULTS #
Training Dataset
(1)
MNIST Mirroring, 90° Rotation, Centered (560K) Mitosis (1.1K) Mitosis Mirroring, 90° Rotation, Centered (9K) Mitosis Mirroring, 90° Rotation, Centered (9K) Mitosis Mirroring, 45° Rotation, Centered (13K)
(2) (3)
(4)
(5)
Mitosis Testing Dataset Balanced, Centered
Sens.
Spec.
AUC
FScore
0.68
0.89
0.88
0.76
Balanced Balanced
0.98 1.00
0.23 0.23
0.75 0.64
0.71 0.72
Balanced, Centered
0.81
0.98
0.92
0.88
Balanced, Centered
0.87
0.93
0.91
0.89
The results in Table II are compared to the more classical method of feature extraction from grayscale patches followed by a Support Vector Machine (SVM) classification. First, the grayscale patches are centered and rotated, similar to the centering and augmentation process of the binary patches described in section II-D.2. Then, the grayscale patches undergo feature extraction. The extracted features include mean, median and standard deviation of the gray levels in each patch, mean, median and standard deviation of the magnitude and phase response to Gabor filters in 6 orientations and 4 scales and unique scale gradient histogram features [5] - for a total feature vector of 211 elements. After extracting all features from the Training set, the feature vectors undergo Principal Component Analysis (PCA), and the SVM is trained using the top principal components. The optimal number of top principal components (PC) was found to be 4. The Testing set was then classified using the trained SVM. A comparison between the SVM and CNN classification results can be seen in Table III. The results indicate a significant increase in performance in F-score when using the CNN. TABLE III C OMPARISON T O SVM Classification Method SVM (4 PC) CNN
Sensitivity 0.67 0.86
Specificity 0.94 0.92
F-Score 0.78 0.89
IV. D ISCUSSION & C ONCLUSIONS We have demonstrated a novel method for the automatic detection of cell division in scratch assays. Overall, we show that even with a rather small initial Training set (1.1K patches, before augmentation, 13K after full augmentation), a CNN can be trained to identify binary patches containing cell division with a high success rate. Our results show an AUC of 0.91 and an F-score of 0.89, comparable to stateof-the-art methods in the field. It should be noted than the live-imaging microscopy data used in this work varies greatly from the H&E stained tissue of [1], [2], [3]. In [5], Huh et al. use phase-contrast microscopy data, resulting in similar images, however, our data differs both in the type of cells used and in cell confluence. In [5] and the follow-up work [6] the cell cultures are of a rather low confluence and the analyzed patches often contain only one or two cells. In our data, due to the nature of the scratch assay, the cultures reached a high confluence, resulting in patches with much higher cell density, adding noise and artifacts. Future work entails improvement of the results by implementing a cascaded ensemble of classifiers, as described in [2]. Furthermore, the labeled binary patches will be used to classify the entire patch sequence as Mitosis or non-Mitosis, which will provide an additional layer of information for existing automated methods of scratch assay analysis, giving additional insights into cancer and stem cell research. R EFERENCES [1] M. Veta, P.J. van Diest, S.M Willems, H. Wang, A. Madabhushi, A. Cruz-Roa, F. Gonzalez, A.B.L Larsen, J.S. Vestergaard, A.B. Dahl, and others, Assessment of algorithms for mitosis detection in breast cancer histopathology images, in Medical Image Analysis, 2015, vol. 20(1), pp 237-248. [2] H. Wang, A. Cruz-Roa, A. Basavanhally, H. Gilmore, N. Shih, M. Feldman, J. Tomaszewski, F. Gonzalez, and A. Madabhushi, Cascaded ensemble of convolutional neural networks and handcrafted features for mitosis detection, in SPIE Medical Imaging 2014, pp 90410B90410B. [3] D.C. Cires¸an, A. Giusti, L.M. Gambardella and J. Schmidhuber, Mitosis detection in breast cancer histology images with deep neural networks, in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2013, pp 411-418. [4] G. Avwioro, Histochemical uses of haematoxylin - a review, in J Pharm Clin Sci (JPCS), vol. 1, pp 24-34, 2011. [5] S. Huh, D.F.E Ker, R. Bise, M. Chen, and T. Kanade, Automated mitosis detection of stem cell populations in phase-contrast microscopy images, in IEEE Transactions on Medical Imaging, 2011, vol. 30(3), pp 586-596. [6] S. Huh and M. Chen, Detection of mitosis within a stem cell population of high cell confluence in phase-contrast microscopy images, in 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1033-1040. [7] C. C. Liang, A. Y. Park, and J. L. Guan, In vitro scratch assay: a convenient and inexpensive method for analysis of cell migration in vitro, in Nature protocols 2007, vol 2(2), pp 329-333. [8] G. Topman, O. Sharabani-Yosef, Orna and A. Gefen, A standardized objective method for continuously measuring the kinematics of cultures covering a mechanically damaged site, in Medical engineering & physics, 2012, vol. 34(2), pp 225-232. [9] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition, in Proceedings of the IEEE vol. 83(11), pp 2278-2324, 1998. [10] A. Vedaldi and K. Lenc, MatConvNet-Convolutional Neural Networks for MATLAB, arXiv:1412.4564, 2014.
746
