Deep Learning Driven Breast Cancer Malignancy Prediction in Ultrasound Leveraging Multiscale Feature Fusion and Self Supervised Learning

Abstract

Accurate malignancy prediction in breast ultrasound imaging is challenged by limited annotated data, high inter-observer variability, and inherent noise in sonographic textures. To address these limitations, we propose a deep learning framework that synergistically integrates multiscale feature fusion and self-supervised learning (SSL) to improve diagnostic performance while minimizing reliance on labeled datasets. The architecture employs a hierarchical convolutional backbone with multiscale feature extractors that capture both coarse contextual semantics and fine-grained morphological cues of lesions. Features across multiple receptive fields are fused via a top-down multiscale fusion strategy using bilinear upsampling and channel concatenation, enhancing the model’s ability to localize and characterize malignant regions. We applied a self-supervised contrastive learning approach tailored for medical ultrasound, incorporating spatial transformation invariance and anatomical context preservation to learn domain-relevant representations from unlabeled data. The pretrained encoder is fine-tuned with a supervised classification head using a limited set of annotated images. Extensive experiments on two publicly available breast ultrasound datasets demonstrate that our model achieves higher performance over state-of-the-art baselines, yielding significant improvements in AUC, F1-score, and sensitivity. Ablation studies confirm the individual and combined efficacy of the multiscale fusion and SSL modules. This work establishes a scalable and label-efficient pipeline for ultrasound-based malignancy prediction, with implications for real-time clinical decision support.