A Posteriori Deep Learning Segmentation Quality Estimation Based on Prediction Entropy
DOI:
https://doi.org/10.5566/ias.3024Keywords:
Image Segmentation, Deep Learning, Entropy, Segmentation quality estimationAbstract
Image segmentation is a common intermediate operation in many image processing applications. On automated systems it is important to evaluate how well it, or its subsystems are performing without access to the Ground Truth. In Deep Learning based image segmentation there are very few methods to evaluate the output quality without using a ground truth. Most of them are based on the uncertainty (variance or standard deviation) of the prediction and can be applied to Bayesian Neural Networks, but not to Convolutional Neural Networks. In this research we propose to use the Entropy as a measure of uncertainty applied to the segmented image predicted by the Neural Network and some indicators based on it. The method is tested in a segmentation task of labeled skin images. The entropy based indicators are evaluated without knowing the ground truth and compared with indicators based on the real labels (Jaccard, Dice and Average Symmetrical Surface Distance). This experimentation showed that they are correlated and some Entropy based indicators can predict quite well the ground truth based indicators.
References
Bishop CM (2007). Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, 1st ed.
Bridle JS (1989). Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. 2nd International Conference on Neural Information Processing Systems, NIPS’89. Cambridge, MA, USA: MIT Press.
Canny J (1986). A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI 8:679–98.
Cover TM, Thomas JA (2006). Elements of Information Theory (2. ed.). John Wiley & Sons, Ltd.
DeVries T, Taylor GW (2018a). Learning confidence for out-of-distribution detection in neural
networks. https://arxiv.org/abs/1802.04865.
DeVries T, Taylor GW (2018b). Leveraging Uncertainty Estimates for Predicting Segmentation Quality. https://arxiv.org/abs/1807.00502.
Dice LR (1945). Measures of the amount of ecologic association between species. Ecology 26:297–302. Full publication date: Jul., 1945.
Fawcett T (2006). An introduction to ROC analysis. Pattern Recogn Lett 27:861–74. ROC Analysis in Pattern Recognition.
Gal Y (2016). Uncertainty in Deep Learning. Ph.D. thesis, University of Cambridge.
Gal Y, Ghahramani Z (2016). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. International Conference on Machine Learning ISSN: 1938-7228.
Galil I, Dabbah M, El-Yaniv R (2022). Which models are innately best at uncertainty estimation ? https://arxiv.org/abs/2206.02152.
Hanley JA, McNeil BJ (1982). The meaning and use of the area under a Receiver Operating Characteristic (ROC) curve. Radiology 143:29–36.
Hao J, Shen Y, Xu H, Zou J (2009). A region entropy based objective evaluation method for image segmentation. IEEE IMTC P 373–377
Haralick RM, Shapiro LG (1985). Image Segmentation Techniques. Comput Vision Graph 29:100–32.
Hendrycks D, Gimpel K (2017). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. International Conference on Learning Representations - ICLR 2017 - (poster)
Jaccard P (1912). The distribution of the flora in the alpine zone.1. New Phytol 11:37–50.
Kampffmeyer M, Salberg AB, Jenssen R (2016). Semantic Segmentation of Small Objects and Modeling of Uncertainty in Urban Remote Sensing Images Using Deep Convolutional Neural Networks. Proc CVPR IEEE 680–688
Kendall A, Badrinarayanan V, Cipolla R (2017). Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. British Machine Vision Conference BMVC. 4–7
Kendall A, Gal Y (2017). What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? International Conference on Neural Information Processing Systems, NIPS’17 5580-–5590
Kendall MG (1938). A new measure of rank correlation. Biometrika 30:81–93.
Moon J, Kim J, Shin Y, Hwang S (2020). Confidence-Aware Learning for Deep Neural Networks. https://arxiv.org/abs/2007.01458.
Nair T, Precup D, Arnold DL, Arbel T (2020). Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Med Image Anal 59:101557.
Pal NR, Bhandari D (1993). Image thresholding: Some new techniques. Signal Process 33:139–58.
Pal NR, Pal SK (1993). A review on image segmentation techniques. Pattern Recogn 26:1277–94.
Rill-García R, Dokladalova E, Dokládal P (2022). Syncrack: Improving Pavement and Concrete Crack Detection Through Synthetic Data Generation. Comm Com Inf SC (VISAPP’22).
Serra J (1982). Image Analysis and Mathematical Morphology, vol. 1. Academic Press.
Settles B (2010). Active learning literature survey - Computer Sciences Technical Report 1648 - University of Wisconsin–Madison. http://burrsettles.com/pub/settles.activelearning.pdf.
Shannon CE (1948). A Mathematical Theory of Communication. Bell Syst Tech J 27:379–423.
Spearman C (1904). The proof and measurement of association between two things. Am J Psychol 15:72–101.
Yeghiazaryan V, Voiculescu ID (2018). Family of boundary overlap metrics for the evaluation of medical image segmentation. J Med Imaging 5:015006.
Zhang H, Fritts JE, Goldman SA (2004). An Entropy-based objective evaluation method for image segmentation. Proc. SPIE 5307, Storage and Retrieval Methods and Applications for Multimedia 2004
Zhang H, Fritts JE, Goldman SA (2008). Image segmentation evaluation: A survey of unsupervised methods. Comput Vis Image Und 110:260–80.
Zhang Y (1996). A survey on evaluation methods for image segmentation. Pattern Recogn 29:1335–46.
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Jose-Marcio Martins da Cruz, Mateus Sangalli, Etienne Decencière Ferrandière, Santiago Velasco-Forero, Thérèse Baldeweck
This work is licensed under a Creative Commons Attribution 4.0 International License.
