Deep Learning Model for Cell Nuclei Segmentation and Lymphocyte Identification in Whole Slide Histology Images
Volume 32, Issue 1 (2021), pp. 23–40
Pub. online: 12 January 2021
Type: Research Article
Open Access
Open Access
Received
1 March 2020
1 March 2020
Accepted
1 December 2020
1 December 2020
Published
12 January 2021
12 January 2021
Abstract
Anti-cancer immunotherapy dramatically changes the clinical management of many types of tumours towards less harmful and more personalized treatment plans than conventional chemotherapy or radiation. Precise analysis of the spatial distribution of immune cells in the tumourous tissue is necessary to select patients that would best respond to the treatment. Here, we introduce a deep learning-based workflow for cell nuclei segmentation and subsequent immune cell identification in routine diagnostic images. We applied our workflow on a set of hematoxylin and eosin (H&E) stained breast cancer and colorectal cancer tissue images to detect tumour-infiltrating lymphocytes. Firstly, to segment all nuclei in the tissue, we applied the multiple-image input layer architecture (Micro-Net, Dice coefficient (DC) $0.79\pm 0.02$). We supplemented the Micro-Net with an introduced texture block to increase segmentation accuracy (DC = $0.80\pm 0.02$). We preserved the shallow architecture of the segmentation network with only 280 K trainable parameters (e.g. U-net with ∼1900 K parameters, DC = $0.78\pm 0.03$). Subsequently, we added an active contour layer to the ground truth images to further increase the performance (DC = $0.81\pm 0.02$). Secondly, to discriminate lymphocytes from the set of all segmented nuclei, we explored multilayer perceptron and achieved a 0.70 classification f-score. Remarkably, the binary classification of segmented nuclei was significantly improved (f-score = 0.80) by colour normalization. To inspect model generalization, we have evaluated trained models on a public dataset that was not put to use during training. We conclude that the proposed workflow achieved promising results and, with little effort, can be employed in multi-class nuclei segmentation and identification tasks.
References
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D.G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., Zheng, X. (2016). TensorFlow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, pp. 265–283.
Allard, B., Aspeslagh, S., Garaud, S., Dupont, F., Solinas, C., Kok, M., Routy, B., Sotiriou, C., Stagg, J., Buisseret, L. (2018). Immuno-oncology-101: overview of major concepts and translational perspectives. Seminars in Cancer Biology, 52, 1–11. https://doi.org/10.1016/j.semcancer.2018.02.005.
Alom, Z.Md., Aspiras, T.H., Taha, T.M., Asari, V.K., Bowen, T.J., Billiter, D., Arkell, S. (2019). Advanced deep convolutional neural network approaches for digital pathology image analysis: a comprehensive evaluation with different use cases. CoRR. http://arxiv.org/abs/1904.09075.
Basavanhally, A.N., Ganesan, S., Agner, S., Monaco, J.P., Feldman, M.D., Tomaszewski, J.E., Bhanot, G., Madabhushi, A. (2010). Computerized image-based detection and grading of lymphocytic infiltration in HER2+ breast cancer histopathology. IEEE Transactions on Biomedical Engineering, 57(3). https://doi.org/10.1109/TBME.2009.2035305.
Beatty, G.L., Gladney, W.L. (2015). Immune escape mechanisms as a guide for cancer immunotherapy. Clinical Cancer Research, 21(4), 687–692. https://doi.org/10.1158/1078-0432.CCR-14-1860.
Chen, J., Srinivas, C. (2016). Automatic lymphocyte detection in H&E images with deep neural networks. CoRR. http://arxiv.org/abs/1612.03217.
Cui, Y., Zhang, G., Liu, Z., Xiong, Z., Hu, J. (2019). A deep learning algorithm for one-step contour aware nuclei segmentation of histopathology images. Medical & Biological Engineering & Computing, 57, 2027–2043. https://doi.org/10.1007/s11517-019-02008-8.
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J.T., Blum, M., Hutter, F. (2015). Efficient and robust automated machine learning. In: Advances in Neural Information Processing Systems. https://doi.org/10.1007/978-3-030-05318-5_6.
Galon, J., Pagès, F., Marincola, F.M., Angell, H.K., Thurin, M., Lugli, A., Zlobec, I., Berger, A., Bifulco, C., Botti, G., Tatangelo, F., Britten, C.M., Kreiter, S., Chouchane, L., Delrio, P., Arndt, H., Asslaber, M., Maio, M., Masucci, G.V., Mihm, M., Vidal-Vanaclocha, F., Allison, J.P., Gnjatic, S., Hakansson, L., Huber, C., Singh-Jasuja, H., Ottensmeier, C., Zwierzina, H., Laghi, L., Grizzi, F., Ohashi, P.S., Shaw, P.A., Clarke, B.A., Wouters, B.G., Kawakami, Y., Hazama, S., Okuno, K., Wang, E., O’Donnell-Tormey, J., Lagorce, C., Pawelec, G., Nishimura, M.I., Hawkins, R., Lapointe, R., Lundqvist, A., Khleif, S.N., Ogino, S., Gibbs, P., Waring, P., Sato, N., Torigoe, T., Itoh, K., Patel, P.S., Shukla, S.N., Palmqvist, R., Nagtegaal, I.D., Wang, Y., D’Arrigo, C., Kopetz, S., Sinicrope, F.A., Trinchieri, G., Gajewski, T.F., Ascierto, P.A., Fox, B.A. (2012). Cancer classification using the immunoscore: a worldwide task force. Journal of Translational Medicine, 10, 205. https://doi.org/10.1186/1479-5876-10-205.
Grossman, R.L., Heath, A.P., Ferretti, V., Varmus, H.E., Lowy, D.R., Kibbe, W.A., Staudt, L.M. (2016). Toward a shared vision for cancer genomic data. New England Journal of Medicine, 375(12), 1109–1112. https://doi.org/10.1056/nejmp1607591.
Guo, X., Yu, H., Rossetti, B., Teodoro, G., Brat, D., Kong, J. (2018). Clumped nuclei segmentation with adjacent point match and local shape-based intensity analysis in fluorescence microscopy images. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS. https://doi.org/10.1109/EMBC.2018.8512961.
Huh, J.W., Lee, J.H., Kim, H.R. (2012). Prognostic significance of tumor-infiltrating lymphocytes for patients with colorectal cancer. Archives of Surgery, 147(4), 366–372. https://doi.org/10.1001/archsurg.2012.35.
Janowczyk, A., Madabhushi, A. (2016). Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. Journal of Pathology Informatics, 7, 29. https://doi.org/10.4103/2153-3539.186902.
Linder, N., Taylor, J.C., Colling, R., Pell, R., Alveyn, E., Joseph, J., Protheroe, A., Lundin, M., Lundin, J., Verrill, C. (2019). Deep learning for detecting tumour-infiltrating lymphocytes in testicular germ cell tumours. Journal of Clinical Pathology, 72(2), 157–164. https://doi.org/10.1136/jclinpath-2018-205328.
Mahapatra, D., Bozorgtabar, B., Thiran, J.-P., Shao, L. (2020). Structure preserving stain normalization of histopathology images using self-supervised semantic guidance. In: Martel, A.L. et al. (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020, Lecture Notes in Computer Science, Vol. 12265. Springer, Cham. https://doi-org-443.webvpn.fjmu.edu.cn/10.1007/978-3-030-59722-1_30.
Meiniel, W., Olivo-Marin, J., Angelini, E.D. (2018). Denoising of microscopy images: a review of the state-of-the-art, and a new sparsity-based method. IEEE Transactions on Image Processing, 27(8), 3842–3856. https://doi.org/10.1109/TIP.2018.2819821.
Morkunas, M., Treigys, P., Bernatavičiene, J., Laurinavičius, A., Korvel, G. (2018). Machine learning based classification of colorectal cancer tumour tissue in whole-slide images. Informatica, 29(1), 75–90. https://doi.org/10.15388/Informatica.2018.158.
Pham, C.T., Thao Tran, T.T., Gamard, G. (2020). An efficient total variation minimization method for image restoration. Informatica, 31(3), 539–560. https://doi.org/10.15388/20-INFOR407.
Rasmusson, A., Zilenaite, D., Nestarenkaite, A., Augulis, R., Laurinaviciene, A., Ostapenko, V., Poskus, T., Laurinavicius, A. (2020). Immunogradient indicators for antitumor response assessment by automated tumor-stroma interface zone detection. The American Journal of Pathology, 190(6), 1309–1322. https://doi.org/10.1016/j.ajpath.2020.01.018.
Raza, S.E.A., Cheung, L., Shaban, M., Graham, S., Epstein, D., Pelengaris, S., Khan, M., Rajpoot, N.M. (2019). Micro-Net: a unified model for segmentation of various objects in microscopy images. Medical Image Analysis, 52, 160–173. https://doi.org/10.1016/j.media.2018.12.003.
Reinhard, E., Ashikhmin, M., Shirley, P. (2001). Color transfer between images. IEEE Computer Graphics and Applications, 21(5), 34–41. https://doi.org/10.1109/38.946629.
Ronneberger, O., Fischer, P., Brox, T. (2015). U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W., Frangi, A. (Eds.), Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. MICCAI 2015, Lecture Notes in Computer Science, Vol. 9351. Springer, Cham. https://doi.org/10.1007/978-3-319-24574-4_28.
Saltz, J., Gupta, R., Hou, L., Kurc, T., Singh, P., Nguyen, V., Samaras, D., Shroyer, K.R., Zhao, T., Batiste, R., Van Arnam, J., Shmulevich, I., Rao, A.U.K., Lazar, A.J., Sharma, A., Thorsson, V. (2018). Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Reports, 23(1), 181–193. https://doi.org/10.1016/j.celrep.2018.03.086.
Sirinukunwattana, K., Raza, S.E.A., Tsang, Y.W., Snead, D.R.J., Cree, I.A., Rajpoot, N.M. (2016). Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Transactions on Medical Imaging, 35(5), 1196–1206. https://doi.org/10.1109/TMI.2016.2525803.
Turkki, R., Linder, N., Kovanen, P., Pellinen, T., Lundin, J. (2016). Antibody-supervised deep learning for quantification of tumor-infiltrating immune cells in hematoxylin and eosin stained breast cancer samples. Journal of Pathology Informatics, 7, 38. https://doi.org/10.4103/2153-3539.189703.
Vahadane, A., Peng, T., Sethi, A., Albarqouni, S., Wang, L., Baust, M., Steiger, K., Schlitter, A.M., Esposito, I., Navab, N. (2016). Structure-preserving color normalization and sparse stain separation for histological images. IEEE Transactions on Medical Imaging, 35(8), 1962–1971. https://doi.org/10.1109/TMI.2016.2529665.
Zappasodi, R., Merghoub, T., Wolchok, J.D. (2018). Emerging concepts for immune checkpoint blockade-based combination therapies. Cancer Cell, 33(4), 581–598. https://doi.org/10.1016/j.ccell.2018.03.005.
Zuiderveld, K. (1994). Contrast limited adaptive histogram equalization. In: Graphics Gems, pp. 474–485. https://doi.org/10.1016/b978-0-12-336156-1.50061-6.
Biographies
Budginaitė Elzbieta
E. Budginaitė graduated with master’s degree in system biology from the Vilnius University, Lithuania, in 2019. Interests include machine learning, graph theory, natural language processing, artificial neural networks.
Morkūnas Mindaugas
M. Morkūnas graduated from the Vilnius Gediminas Technical University, Lithuania, in 2002. In 2016 he started PhD studies in informatics engineering at the Institute of Data Science and Digital Technologies, Vilnius University, Lithuania. His interests include bioinformatics, cancer biology, image analysis, machine learning, artificial neural networks.
Laurinavičius Arvydas
A. Laurinavičius MD, PhD, a full-time professor at Vilnius University, Department of Pathology, Forensic Medicine and Pharmacology. Director and consultant pathologist at National Center of Pathology. Chair and board member of multiple international professional societies. Fields of interest: renal pathology, digital pathology image analysis, pathology informatics, health information systems and standards, testing of cancer biomarkers in tissue, multi-resolution analysis of biomarkers.
Treigys Povilas
P. Treigys graduated from the Vilnius Tech (former Vilnius Gediminas Technical University), Lithuania, in 2005. In 2010 he received the doctoral degree in computer science (PhD) from Vilnius University Data Science and Digital Technologies Insitute (former Institute of Mathematics and Informatics) jointly with Vilnius Tech. His interests include deep neural networks application in speech and image analysis. Among his other interests are: automated signal segmentation, medical audio and image analysis, big data, and software engineering.
