1 Introduction
A host-tumour immune conflict is a well-known process happening during the tumourigenesis. It is now clear that tumours aim to escape host immune responses by a variety of biological mechanisms (Beatty and Gladney, 2015; Zappasodi et al., 2018; Allard et al., 2018). Thus the importance of tumour-infiltrating lymphocytes (TILs) in pathology diagnosis, prognosis, and treatment increases. Quantification of the immune infiltrate along tumour margins in the tumour microenvironment has gathered researchers’ attention as a reliable prognostic measure for various cancer types (Basavanhally et al., 2010; Galon et al., 2012; Huh et al., 2012; Rasmusson et al., 2020). With the emergence of whole slide imaging (WSI) and recent Federal Drug Administration’s (FDA) approval for WSI usage in clinical practice, various techniques have been proposed to detect lymphocytes in digital pathology images focusing on the algorithms based on colour, texture, and shape feature extraction, morphological operations, region growing, and image classification.
Recent works. In general, prior studies were limited to lymphocyte detection and therefore relied on unsupervised approaches such as in Basavanhally et al. (2010), where lymphocytes were automatically detected by a combination of region growing and Markov random field algorithms. Before detection, applying tissue epithelium-stroma classification reduced the noise irrelevant for the lymphocyte nuclei detection by 18 texture features (Kuse et al., 2010).
As opposed to individual nuclei detection, models proposed in Turkki et al. (2016) and Saltz et al. (2018) have been trained to identify TIL-enriched areas rather than stand-alone lymphocytes. In a study by Saltz et al. (2018), authors have developed a convolutional neural network (CNN) classifier capable of identifying TIL-enriched areas in WSI slides from TCGA (The Cancer Genome Atlas) database. Similarly, in Turkki et al. (2016), lymphocyte-rich areas were identified by training an SVM classifier on a set of features extracted by the VGG-F neural network from CD45 IHC-guided superpixel-level annotations in digitized H&E specimen.
Such a high-level tissue segmentation approach has been widely used for cancer tissue segmentation tasks, such as stroma-epithelium tissue classification (Morkunas et al., 2018). However, lymphocyte infiltration quantification accuracy would benefit from a more granular level analysis using object segmentation models. Convolutional encoder-decoder based model architectures (convolutional autoencoders CAEs) have been established as an efficient method for medical imaging tasks. U-Net autoencoder model, proposed in Ronneberger et al. (2015), has become a golden standard model for medical areas ranging from cell nuclei segmentation to tissue analysis in computed tomography (CT) scans (Ma et al., 2019). The deep, semantic feature maps from the U-Net decoder are combined with shallow, low-level feature maps from the encoder part of the model via skip connections, thus maintaining the fine-grained features of the input image. This renders U-Net applicable in medical image segmentation, where precise detail recreation is of utmost importance. Specifically for lymphocyte detection, approaches utilizing fully convolutional neural networks on the digital H&E slides were published by Chen and Srinivas (2016) and Linder et al. (2019). Both approaches investigate convolutional autoencoders using histology sample patches with annotated lymphocyte nuclei. Detection and classification, but not segmentation of nuclei in H&E images, were done using spatially constrained CNN in Sirinukunwattana et al. (2016). Notably, the classification into four cell types (epithelial, inflammatory, fibroblast, and miscellaneous) was performed on patches centred on nuclei considering their local neighbourhood. A more recent adaptation – the Micro-Net model – incorporates an additional input image downsampling layer that circumvents the max-pooling process, thus maintaining the input features ignored by the max-pooling layer. This way, more detailed contextual information is passed into the output layer, enabling better segmentation of adjacent cell nuclei (Raza et al., 2019).
The Hover-Net model published in Graham et al. (2019) enables simultaneous cell nuclei segmentation and classification by three dedicated branches of the model – segmenting, separating, and classifying. Hover-Net was applied to two datasets and achieved 0.573 and 0.631 classification f-score. In Janowczyk and Madabhushi (2016), AlexNet was employed to identify centres of lymphocyte nuclei. The network was trained on cropped lymphocyte nuclei as a positive class, and the negative class was sampled from the most distant regions with respect to the annotated ground truth. The trained network produces the posterior class membership probabilities for every pixel in the test image; subsequently, potential centres of lymphocyte nuclei are identified by disk kernel convolution and thresholding. In Alom et al. (2019), the same dataset was utilized to evaluate different advanced neural networks for a variety of digital pathology tasks, including lymphocyte detection. The authors proposed a densely connected recurrent convolution network (DCRCNN) to directly regress the density surface with peaks corresponding to lymphocyte centres. When compared to AlexNet, the DCRCNN improves the f-score by 1%, yet it is worth mentioning that both (Janowczyk and Madabhushi, 2016; Alom et al., 2019) do not demonstrate method generalization – in the respective studies, the same dataset was used for training and testing.
Our study focuses on the customization of cell segmentation autoencoder architecture and aims to investigate a two-step cell segmentation and subsequent lymphocyte classification workflow using digital histology images of H&E stained tumour tissues. Robust separation of clumped cell nuclei is a common challenge in whole slide image analysis (Guo et al., 2018). To tackle this nuclei segmentation challenge, our cell nuclei segmentation model renders an additional active contour layer, which increases the segmentation efficiency of adjacent cell nuclei. Apart from overlapping nuclei, image magnification is another critical factor for nuclei segmentation models. Publicly available annotated nuclei datasets contain histological samples scanned at $40\times $ magnification, preserving texture features and facilitating precise feature extraction. In pathology practice, however, samples scanned at $20\times $ magnification are more common. Image analysis at a lower resolution is faster and less memory-exhaustive, yet the precise cell nuclei segmentation becomes a more difficult task. As reported by Cui et al. (2019), the active contour layer improves adjacent nuclei separation – this has been observed in our experiments as well. We report that multiple re-injection of downsampled images to the model – approach initially proposed in Raza et al. (2019) in the Micro-Net model – has significantly boosted nuclei segmentation performance compared to the baseline U-Net model (Raza et al., 2019; Ronneberger et al., 2015). We further observe that our customized model architecture component – two parallel blocks of convolutional layers, referred to as a texture block – increases segmentation quality compared to the original Micro-Net model and reduces model complexity to less than 280 000 parameters. For the lymphocyte classification task, we utilized traditional machine learning approaches – Random Forest classifier, Multilayer perceptron, and a CNN. We have performed minimal hyperparameter tuning of classification models in a grid search procedure. We have used a private dataset to train our models, and a public dataset for final workflow evaluation, thus demonstrating the generalization of proposed models.
The paper is organized as follows. In Section 2.1, we describe the datasets used in the study. In Section 2.2, we introduce the segmentation method based on autoencoder neural network architecture, followed by the classification of segmented nuclei. In Section 3, we present experimental results comparing different cell nuclei segmentation as well as lymphocyte discrimination approaches. In particular, Section 3.3 covers the evaluation of our method on the publicly available annotated data set of breast cancer H&E images. We formulate conclusions in Section 4.
2 Materials and Methods
2.1 The Datasets
Images. In our study, we used 4 whole-slide histology sample images prepared with H&E staining (2 WSI slides from breast cancer patients and 2 WSIs from colorectal cancer). These slides were produced in the National Center of Pathology, Lithuania (NCP), and digitized with the Aperio ScanScope XT Slide Scanner at $20\times $ magnification.
1 WSI slide was obtained from The Cancer Genome Atlas database, tile ID: TCGA_AN_A0AM (Grossman et al., 2016), and used for both segmentation and classification testing.
Two additional public datasets were used for classification testing purposes. The CRCHistoPhenotypes dataset (CRCHP) contains colorectal adenocarcinoma cell nuclei. 1143 nuclei are annotated as inflammatory (used for lymphocyte category in our experiments), and 1040 annotated as epithelial (used for other cell type category) (Sirinukunwattana et al., 2016). The breast cancer dataset (JAN) published by Janowczyk and Madabhushi (2016) consists of 100 images ($100\times 100$ pixel-sized) with lymphocytes annotated. Samples were digitized using $20\times $ magnification and stained with hematoxylin and eosin. An expert pathologist annotated lymphocytes by marking lymphocyte nuclei centres. In contrast to the CRCHP dataset, this image corpus is more suitable for our tasks since the data was prepared specifically for lymphocyte identification. The CRCHP dataset entails broader cell type categories, where lymphocytes are annotated under the inflammatory label and other immune cells such as mast cells and macrophages.
Table 1
Two datasets were used for segmentation and classification tasks. Segmentation experiments were performed on $256\times 256$ pixel-sized image patches. Classification experiments were performed on extracted cell nuclei embedded in blank $32\times 32$ pixel-sized placeholders.
| Segmentation set | Tumour type | Raw set | Final augmented set | Origin |
| BC | 192 | 3648 | NCP | |
| CRC | 82 | 1558 | NCP | |
| Training | total | 274 | 5206 | NCP |
| BC | 54 | 54 | NCP | |
| CRC | 16 | 16 | NCP | |
| Validation | total | 70 | 70 | NCP |
| BC | 96 | 96 | TCGA | |
| Testing | total | 96 | 96 | TCGA |
| Classification set | Nucleus type | Raw set | Final augmented set | Origin |
| lymphocyte nuclei | 11032 | 50950 | NCP | |
| other nuclei | 10922 | 55825 | NCP | |
| Training | total nuclei | 21954 | 106775 | NCP |
| lymphocyte nuclei | 2588 | 2588 | NCP | |
| other nuclei | 2751 | 2751 | NCP | |
| Validation | total nuclei | 5339 | 5339 | NCP |
| BC lymphocytes | 903 | 903 | TCGA | |
| CRC lymphocytes | 1143 | 1143 | CRCHP | |
| total lymphocytes | 2046 | 2046 | ||
| BC other | 1195 | 1195 | TCGA | |
| CRC other | 1040 | 1040 | CRCHP | |
| total other | 2235 | 2235 | ||
| Testing I | total nuclei | 4281 | 4281 | |
| BC lymphocytes | 2949 | 2949 | JAN | |
| BC other | 1921 | 1921 | JAN | |
| Testing II | total nuclei | 4870 | 4870 | JAN |
Segmentation dataset. To train and validate the segmentation model, we randomly selected 344 tiles of $256\times 256$ pixel size. Dataset was split into training and validation sets, respectively. To test the segmentation model, we prepared 96 tiles from the breast cancer TCGA slide. Both tiles generated from the TCGA slide and tiles generated from NCP slides were manually annotated by EB and MM. In the annotation process, each cell nucleus present in an image patch was manually outlined, and 2 pixel-wide active contour borders surrounding each nucleus were added as a second layer to the nuclei segmentation masks. Each outlined nucleus was assigned a class label (a lymphocyte or other). To the training set, we applied various image augmentation methods (rotation, flip, transpose, RGB augmentation, brightness adjustment, CLAHE, Zuiderveld, 1994) to obtain the final training set of 5206 images.
Table 2
Image augmentation techniques and parameters used for training dataset expansion.
| Augmentation | Parameters |
| Transposition, rotation axis flipping | Perpendicular rotation angles |
| CLAHE (Zuiderveld, 1994) | Cliplimit = 2.0, tilegridsize = (8, 8) |
| Brightness adjustment | HSV colourspace, hue layer increased by 30 |
| RGB augmentation | Random pixel value adjustments up to 0.1 |
| RGB2HED colour adjustments (Ruifrok and Johnston, 2001) | Colour values adjusted within range $[0.02,0.001,0.15]$ |
The segmentation dataset is summarized in Table 1, and the techniques used to augment training patches are summarized in Table 2.
Fig. 1
Overall schema of the proposed workflow. On top, a training phase for both segmentation and classification models is shown. The segmentation network is trained on original image patches and manually annotated ground truth images. The classification model is trained on cropped nuclei to discriminate lymphocytes (in the red box) from other nuclei. In the middle, a testing phase is shown. The trained segmentation model accepts new images and produces segmentation masks (for clarity, the active contour layer in the resulting segmentation mask is not shown). Resulting segmentation masks are used to crop out detected cell nuclei that are fed into the classifier model and sorted into lymphocytes and non-lymphocyte nuclei. In the bottom panel, on the left, we have representative segmentation results (lymphocyte nuclei are coloured in red for clarity), and on the right, we have an original image with detected nuclei contours outlined and detected lymphocyte nuclei depicted with red dots. Green dots indicate lymphocyte ground truth.
Classification dataset. To train and validate the classification models, we generated a dataset from the same image patches used to train the segmentation model. Particularly, manually generated segmentation masks were used to crop-out all types of cell nuclei from raw images. Each extracted nucleus was centred in a blank $32\times 32$ pixel-sized patch. Each nucleus-containing patch inherited a class label (assigned manually to the ground truth in an annotation procedure). Nuclei containing patches were further augmented by rotation and axis flipping. The testing set for cell classifier consisted of 2098 TCGA breast cancer cell nuclei and 2183 colorectal adenocarcinoma cell nuclei from the CRCHP dataset (see Table 1).
2.2 The Proposed Method
The overall schema of the proposed workflow is summarized in Fig. 1.
2.2.1 Modified Micro-Net Model
The autoencoder architecture for nuclei segmentation is shown in Fig. 2. The model consists of 3 encoder and 3 decoder blocks consisting of 2 convolution layers ($3\times 3$ convolutional filters with stride 2), dropout (dropout rate 0.2), and max-pooling layers. Our model adopts multiple downsized image input layers after each max-pooling operation, which were originally proposed in the Micro-Net model by Raza et al. We propose additional model enhancement by introducing a texture block after each image input layer. The texture block consists of 2 parallel blocks of 3 convolution layers, which enhance image texture extraction. To ensure robust nuclei separation, we supplement our nuclei annotations with an additional active contour layer. Our experiments indicate that the proposed model architecture is more compact and requires less computational resources than the original Micro-Net structure.
We used elu activation after each convolution layer and sigmoid activation for the output layer. Adam optimizer was used with initial learning rate $lr=0.001$, which was reduced by factor 0.1 if validation loss did not improve for 4 consecutive epochs ($\min lr=1\times {10^{-6}}$) (Kingma and Ba, 2014). Dice coefficient (1) was used to quantify model metrics with binary crossentropy dice loss (3) as custom loss function.
Fig. 3
The performance metrics of segmentation and classifier models. A: training and validation metrics (top-Dice coefficient, below-loss values per epoch) of segmentation autoencoder, B: confusion matrix depicting cell nuclei classifier performance on the testing set (true positive lymphocyte predictions and true negatives marked in grey, false predictions – in red), C: ROC curve obtained from nuclei classifier testing data.
Model converged after 36 epochs (see Fig. 3A) using batch size of 1 (input image dimensions: $256\times 256\times 3$) for training and validation. Input images were normalized by scaling pixel values to the range $[0,1]$.
where $\mathit{TP}$ is true positive, $\mathit{FP}$ is false positive and $\textit{FN}$ is false negative.
where y is binary class indicator and $\hat{y}$ is predicted probability.
(1)
\[ \textit{Dice}=\frac{2\ast \mathit{TP}}{(\mathit{TP}+\mathit{FP})+(\mathit{TP}+\textit{FN})},\]2.2.2 Multilayer Perceptron
The multilayer perceptron model was employed to solve the binary classification problem of lymphocyte identification. Our experiment’s model consists of three dense layers (number of nodes: 4096, 2048, 1024), with softmax as the output layer activation function. For each layer, we used relu activation, followed by batch normalization. The dropout layer (dropout rate 0.4) was used in the middle layer instead of batch normalization to avoid model overfitting. We used Adam optimizer with initial learning rate $lr=0.001$, which was reduced by factor 0.1 if validation loss did not improve for 6 consecutive epochs ($\min lr=1\times {10^{-6}}$). Accuracy was used as metrics with binary cross-entropy as loss function (2). The model was trained until convergence using 64 and 32 batch sizes for training and validation, respectively.
3 Results
3.1 Nuclei Segmentation
3.1.1 Hyperparameter Tuning
The optimal model architecture was experimentally evaluated using a hyperparameter grid search. To test segmentation robustness, we evaluated both pixel-level and object-level metrics. The dice coefficient was used to track pixel-level segmentation performance, while object-level segmentation quality was evaluated by calculating intersection over union (IoU). We treated the predicted nuclei as true positive if at least 50% of the predicted nuclei area overlapped with the ground truth nuclei mask. In order to prevent multiple predicted objects mapping to the same ground truth nucleus, ground truth nucleus mask could only be mapped to a single predicted object. Results of hyperparameter tuning are provided in Table 3. Hyperparameter space was investigated by changing dropout rates, convolution filters per network layer, and activation functions. Due to multiple image down-sampling and concatenation operations in CNN architecture, models with parameter size higher than 500 000 have exceeded memory limitations. Our experiments indicate that expansion of model layer width (tested kernel sizes 16, 32, 48) did not dramatically affect the model prediction metrics – which suggests that texture block component may ensure consistent feature extraction in a wide range of model width.
Table 3
Performance metrics of convolutional autoencoders (CAE) used for the hyperparameter grid search for nuclei segmentation. Dice coefficients (mean Dice coefficient ± standard deviation). Mean and standard deviation values were calculated from stand-alone dice coefficients for each tile from the testing set. DO – drop out rate, BN – batch normalization.
| Act func | Output act func | Kernel size | DO | BN | Dice coefficient | Accuracy | Precision | Recall | f-score |
| U-Net | |||||||||
| elu | sigmoid | 64 | 0.2 | − | $0.78\pm 0.03$ | $0.59\pm 0.08$ | $0.66\pm 0.09$ | $0.84\pm 0.04$ | $0.74\pm 0.06$ |
| Micro-Net model | |||||||||
| tanh | sigmoid | 64 | − | − | $0.79\pm 0.02$ | $0.66\pm 0.06$ | $0.75\pm 0.05$ | $0.85\pm 0.05$ | $0.80\pm 0.04$ |
| Our model | |||||||||
| elu | sigmoid | 16 | 0.2 | − | $0.81\pm 0.02$ | $0.77\pm 0.05$ | $0.86\pm 0.04$ | $0.88\pm 0.04$ | $0.87\pm 0.03$ |
| elu | sigmoid | 32 | 0.2 | − | $0.80\pm 0.02$ | $0.77\pm 0.06$ | $0.85\pm 0.04$ | $0.88\pm 0.04$ | $0.87\pm 0.04$ |
| elu | sigmoid | 48 | 0.2 | − | $0.80\pm 0.02$ | $0.76\pm 0.06$ | $0.85\pm 0.04$ | $0.87\pm 0.04$ | $0.87\pm 0.03$ |
| elu | sigmoid | 16 | 0.3 | − | $0.81\pm 0.02$ | $0.77\pm 0.06$ | $0.86\pm 0.04$ | $0.88\pm 0.05$ | $0.87\pm 0.04$ |
| elu | sigmoid | 32 | 0.3 | − | $0.80\pm 0.02$ | $0.76\pm 0.06$ | $0.85\pm 0.05$ | $0.88\pm 0.05$ | $0.87\pm 0.04$ |
| elu | sigmoid | 48 | 0.3 | − | $0.80\pm 0.02$ | $0.76\pm 0.06$ | $0.86\pm 0.04$ | $0.87\pm 0.04$ | $0.87\pm 0.03$ |
| elu | sigmoid | 32 | − | + | $0.80\pm 0.02$ | $0.74\pm 0.06$ | $0.84\pm 0.05$ | $0.86\pm 0.05$ | $0.85\pm 0.03$ |
| relu | sigmoid | 32 | − | + | $0.80\pm 0.02$ | $0.74\pm 0.06$ | $0.84\pm 0.05$ | $0.87\pm 0.05$ | $0.85\pm 0.03$ |
| elu | softmax | 32 | − | + | $0.73\pm 0.04$ | $0.58\pm 0.08$ | $0.63\pm 0.08$ | $0.87\pm 0.05$ | $0.73\pm 0.06$ |
| relu | softmax | 32 | − | + | $0.77\pm 0.03$ | $0.65\pm 0.07$ | $0.72\pm 0.07$ | $0.87\pm 0.05$ | $0.78\pm 0.0$ |
3.1.2 Model Performance Speed
Instead of basing our optimal model selection rationale solely on the Dice coefficient and object-level testing metrics, we evaluated the gridsearch models based on its loading and image prediction time relative to the original Micro-Net model. Since no significant changes were observed between dropout rates, we chose a custom model of a 0.2 dropout rate, elu activation function, and sigmoid activation function with differing layer widths of 16, 32, and 48 kernels. The testing results provided in Table 4 indicate that the lowest relative image prediction and model loading time was observed for segmentation autoencoder consisting of 32 convolutional kernels per layer, 0.2 dropout rate using elu activation function and sigmoid activation function for output layer with total parameter size lower than 280.000. In comparison to U-Net autoencoder (>1.9 M parameters), which has reached $0.78\pm 0.028$ Dice coefficient for testing dataset, our selected model achieved $0.81\pm 0.018$ Dice coefficient with over 6-fold lower model complexity.
Table 4
A comparison table of autoencoder parameter size and performance speed. Model loading and prediction times were obtained relative to the original Micro-Net model. The best performing model is highlighted in bold.
| Model | Parameters | Relative loading time | Relative prediction time |
| Micro-Net | 73 467 842 | 1 | 1 |
| Custom-16 | 131 746 | 0.212 | 0.314 |
| Custom-32 | 279 506 | 0.212 | 0.288 |
| Custom-48 | 507 138 | 0.268 | 0.359 |
3.1.3 Active Contour Layer
To evaluate the impact of the active contour layer on nuclei separation, we trained convolutional autoencoder using single-layered nuclei masks and compared the results with an identical model trained on two-layered annotations. During this experiment, we used the best-scoring model architecture from the hyperparameter search experiment. Nuclei segmentation using masks supplemented with the active contour layer has outperformed the model with single-layered masks both on pixel-level and object-level measurements, as shown in Table 5. Active-contour increased object segmentation accuracy and f-score by 1 percent ($0.75\pm 0.062$ and $0.85\pm 0.04$, respectively).
Table 5
The active contour layer effect on nuclei segmentation autoencoder performance. Pixel-level Dice coefficients (mean Dice coefficient ± standard deviation) were obtained from a testing set consisting of 96 $256\times 256$ RGB tiles, where mean and standard deviation values were calculated from stand-alone dice coefficients for each tile. Object-level accuracy, precision, recall, and f-score metrics collected if at least 50% overlap between annotated and predicted nuclei masks (mean intersection over union IoU).
| Mask layers | Dice coefficient | Accuracy | Precision | Recall | f-score |
| 2-layered | $0.81\pm 0.02$ | $0.75\pm 0.06$ | $0.85\pm 0.05$ | $0.86\pm 0.04$ | $0.85\pm 0.04$ |
| 1-layered | $0.80\pm 0.02$ | $0.73\pm 0.06$ | $0.84\pm 0.05$ | $0.85\pm 0.04$ | $0.84\pm 0.04$ |
3.2 Nuclei Classification
3.2.1 Hyper Parameter Tuning and Model Comparison
The cell classification problem was approached with several different statistical models. Random Forest was chosen as a baseline machine learning algorithm. We used Python implementation of a random forest classifier from the sklearn machine learning library (Feurer et al., 2015) (using the Gini impurity criterion as split quality measurement and 10 estimators). Random forest classifier was trained on linearized nuclei images ($32\times 32$ RGB-coloured images linearized to 3072-length vector), which achieved 0.77 testing accuracy. In addition, we investigated two deep-learning-based strategies for cell nuclei classification: multilayer perceptron (MLP) consisting of three consecutive dense layers, and convolutional neural network (CNN) consisting of 4 convolutional, 2 max-pooling, and 2 dense layers. Model performance metrics were evaluated for several hyperparameter combinations, including a number of nodes per layer, activation functions, and a number of convolutional kernels. Hyperparameter search is summarized in Table 6. During our experimentations, a multilayer perceptron with three dense layers, softmax for output and relu layer activation functions, 2 batch-normalization layers, and a dropout layer achieved the highest testing accuracy score of 0.78 with 0.82, 0.71, and 0.99 f-score, precision and recall values, respectively.
Table 6
The hyperparameter grid search results for cell nuclei classifier (mean ± standard deviation). The model performance was evaluated on the testing set. Mean and standard deviation values were obtained by running each experiment 5 times.
| Models | Accuracy | Precision | Recall | f-score |
| Random forest | $0.77\pm 0.002$ | $0.69\pm 0.002$ | $0.99\pm 0.002$ | $0.82\pm 0.002$ |
| Multilayer perceptron | ||||
| $2048/1024/512$ | $0.78\pm 0.09$ | $0.71\pm 0.1$ | $0.99\pm 0.004$ | $0.83\pm 0.06$ |
| $4096/2048/1024$ | $0.78\pm 0.003$ | $0.71\pm 0.03$ | $0.99\pm 0.0003$ | $0.82\pm 0.02$ |
| Convolutional neural network | ||||
| Kernels per layer: 16 | $0.76\pm 0.09$ | $0.69\pm 0.1$ | $0.98\pm 0.004$ | $0.80\pm 0.06$ |
| Kernels per layer: 32 | $0.76\pm 0.09$ | $0.70\pm 0.1$ | $0.98\pm 0.004$ | $0.81\pm 0.06$ |
The confusion matrix for our cell classification model demonstrates that out of 2046 labelled lymphocytes, 310 were falsely misclassified as other cell types, while 13 false-positive observations were registered out of 2235 nuclei labelled as other cell types as shown in Fig. 3B. Receiver-operating curve (ROC) shown in Fig. 3C indicates the low false-positive rate of our lymphocyte classifier.
Of note, the proposed two-step lymphocyte detection model can potentially be adapted to detect more cell types by replacing existing lymphocyte classifier with a model trained on several classes.
3.3 Workflow Evaluation
Fig. 4
Exemplary 5 testing images from breast cancer lymphocyte dataset (Janowczyk and Madabhushi, 2016) with corresponding lymphocyte identification model outputs. From left to right: 1st column- original testing image from the lymphocyte dataset. 2nd column: nuclei segmentation masks predicted by autoencoder. 3rd column: Expert pathologist’s annotation supplied in the dataset. 4th column: lymphocyte classifier result (if the nucleus was predicted as a lymphocyte, its centre was labelled with a green dot). 5th column: lymphocyte classifier result after Reinhard stain normalization.
The proposed lymphocyte identification workflow has been tested on the lymphocyte dataset published by Janowczyk and Madabhushi (2016).2 The dataset is composed of 100 breast cancer images stained with hematoxylin and eosin and digitized using $20\times $ magnification. The lymphocyte centres were manually annotated by an experienced pathologist. The same dataset was used in Alom et al. (2019). Since our nuclei segmentation model was trained on $256\times 256$ pixel image patches, each testing image was zero-padded to the desired input size while preserving the original image scale. Each testing slide was first analysed with autoencoder to segment all cell nuclei, followed by nuclei cropping and subsequent classification of each cropped nucleus using a pre-trained multilayer perceptron for lymphocyte identification. If the nucleus was classified as a lymphocyte, the cell centre was marked with a green dot. The classifier’s testing results were evaluated using dataset annotations as a reference.
The first analysis results – nuclei segmentation – are shown in the second column of Fig. 4. Nuclei segmentation masks generated by autoencoder demonstrate consistent cell nuclei detection efficiency regardless of image staining intensity. This can be explained by two factors. Due to robust image colour augmentation during autoencoder training, the CAE model learned to generalize the input image by texture, rather than colour. Secondly, our modified Micro-Net model architecture incorporates texture convolutional blocks shown in Fig. 2, which facilitate relevant feature extraction for the autoencoder.
The confusion matrix in Fig. 5A shows a low false-positive lymphocyte misclassification rate. However, the high false-negative rate suggests that the lymphocyte classification model is sensitive to image stain intensity. This is well reflected in Fig. 4 Unmodified image column, where lymphocyte detection efficiency conspicuously decreases as image staining intensity fades. This is not a surprising result, given that a multilayer perceptron was trained on lymphocytes cropped from histology samples prepared in a different laboratory, where image staining is more consistent across different histology samples. This result illustrates the main limitations of the lymphocyte classification model: cropped nuclei images lose image background information, which otherwise could be leveraged in differentiating nucleus stain intensity versus its background colour intensity.
3.3.1 The Effect of Colour Normalization on Overall Model Performance
To address high staining variability between different histological samples, the lymphocyte testing dataset was normalized using the Reinhard stain normalization method. Reinhard algorithm adjusts the source image’s colour distribution to the colour distribution of the target image by equalizing the mean and standard deviation pixel values in each channel (Reinhard et al., 2001).
where l, α, β are colour channels in LAB colourspace, $\hat{\hspace{2.5pt}}$ means standard deviation, $\bar{\hspace{2.5pt}}$ stands for mean value of all pixel values from channel. Colour normalization algorithm was implemented using openCV (Bradski, 2000) and Numpy (Oliphant, 2006) python libraries using representatively stained image from training dataset as target for stain normalization.
(4)
\[\begin{aligned}{}& {l_{\textit{mapped}}}=\frac{{l_{\textit{original}}}-{\bar{l}_{\textit{original}}}}{{\hat{l}_{\textit{original}}}}{\hat{l}_{\textit{target}}}+{\bar{l}_{\textit{target}}},\end{aligned}\]Fig. 5
Testing metrics for breast cancer lymphocyte dataset. A: confusion matrix for testing images with original sample staining; B: confusion matrix for testing images with Reinhard stain normalization applied on image stain.
Stain normalization effect on cell lymphocyte detection was evaluated by comparing testing metrics before stain normalization and after Reinhard algorithm implementation. The confusion matrix in Fig. 5B indicates a lower false-negative rate for lymphocytes. Stain normalization has increased accuracy, precision, recall, and f-score values by approximately 10%, as shown in Table 7. These results indicate that the stain normalization step is an effective pre-processing part which can mitigate high staining intensity variance between histology samples. Observed improvement of lymphocyte classification accuracy by applied relatively simple Reinhard stain normalization suggests this part of our workflow can be further explored. Structure-preserving image normalization methods (Vahadane et al., 2016; Mahapatra et al., 2020) demonstrate promising results; also, certain medical image denoising techniques (Meiniel et al., 2018; Pham et al., 2020) could appear useful in future work.
Table 7
A comparison table depicting the effect of stain normalization on lymphocyte identification efficiency is presented. For comparison, we give here the results of the studies that utilized the same dataset. It is important to note that we only used this dataset to test our method, while studies referenced in the table used part of this dataset for training as well.
| Accuracy | Precision | Recall | f-score | |
| Proposed method, original staining | 0.71 | 0.76 | 0.75 | 0.70 |
| Proposed method, wt stain normalization | 0.81 | 0.80 | 0.81 | 0.80 |
| Janowczyk and Madabhushi (2016) | – | 0.89 | – | 0.90 |
| Alom et al. (2019) | 0.90 | – | – | 0.91 |
Both Janowczyk and Madabhushi (2016) and Alom et al. (2019) used the same dataset to train and evaluate their proposed models; therefore, to deal with overfitting, authors had to apply some sort of cross-validation. 5-fold cross-validation was used in Janowczyk and Madabhushi (2016), and Alom et al. (2019) reserved 10% of the dataset for testing purposes. In contrast, we used the whole dataset exclusively for the proposed model evaluation, thus completely eliminating the possibility of overfitting. Our result (f-score = 0.80) indicates good model generalization and comparable performance to both the above-mentioned methods.
4 Conclusions
In this paper, we propose an end-to-end deep learning-based algorithm for cell nuclei segmentation and consecutive lymphocyte identification in H&E stained $20\times $ magnified breast and colorectal cancer whole slide images. Our conducted experiments suggest that:
-
• Our proposed autoencoder structure component – convolutional texture blocks – can achieve Dice nuclei segmentation score similar to that of the Micro-Net model (our model achieved 1% higher testing Dice coefficient).
-
• Additional active contour layer in nuclei annotation masks increases nuclei segmentation accuracy by 1.5%.
-
• Lymphocyte classification by multilayer perceptron network achieves $78\pm 0.3$% testing accuracy on the private dataset (NCP), and 0.71 on the public dataset (0.81 with Reinhard stain normalization).
Nuclei segmentation autoencoder architecture investigated in this paper has lower model complexity compared to U-Net and Micro-Net models, which brings the advantage of lower computational resource usage. Our suggested pipeline shows good generalization properties, eliminates overfitting, and can be easily extended for multi-class nuclei identification by replacing the nuclei classification MLP model and re-employing the same pre-trained segmentation autoencoder.
