Enhancing cross-dataset zero-shot generalization in colorectal polyp detection using vision-language models
image: The framework leverages a frozen Contrastive Language-Image Pretraining (CLIP) backbone adapted via parameter-efficient modules. (a) Training Phase (Source Domain). Utilizing a single annotated dataset, Attribute-Enhanced Prompts guide the Text Encoder to generate discriminative normal and abnormal anchors. The Image Encoder is refined using Gated Visual Adapters and Generalized Mean (GeM) Pooling to capture multi-scale features. Training is guided by a segmentation loss (LSeg) and a pseudo-normal contrastive loss (LPNC), which uses background regions as pseudo-normal samples to distinguish lesions. Only the adapters are updated to preserve pre-trained knowledge. (b) Zero-Shot Inference Phase (Target Domains). The trained model is evaluated directly on unseen datasets without fine-tuning. Anomaly localization is achieved by computing pixel-wise similarity between the adapted visual features and the learned text anchors, generating precise anomaly heatmaps.
Credit: Sichao Tian, Jing Zhang, Weidong Zhang
Background and objectives
Colorectal polyp detection from endoscopic images is critical for the early diagnosis of colorectal cancer. However, traditional deep learning methods often suffer from limited generalization when deployed across datasets containing different polyp morphologies. This work aimed to investigate whether vision-language foundation models can facilitate zero-shot generalization across multiple polyp datasets without target-domain fine-tuning.
Methods
We introduced a zero-shot colorectal polyp detection framework based on Contrastive Language-Image Pretraining (CLIP) to improve cross-dataset detection performance. Key innovations include: (1) a background patch contrastive loss using pseudo-normal tissue patches to teach the model to distinguish normal mucosa from polyps; (2) attribute-enhanced text prompts that incorporate domain-specific descriptors of polyp appearance, improving the model’s semantic generalization to novel polyp morphologies; and (3) an enhanced CLIP visual adapter with per-layer adaptive feature fusion and generalized mean pooling to capture multi-scale features for better polyp localization. During training, we use one annotated colorectal polyp dataset (e.g., CVC-ColonDB) to learn patch-level image-text correspondence. The model is then evaluated in a zero-shot manner on different polyp datasets (CVC-ClinicDB, Kvasir-SEG, and CVC-300), where we evaluate pixel-level anomaly detection performance.
Results
The framework demonstrated robust zero-shot generalization on unseen test cohorts. Without any dataset-specific fine-tuning, the model achieved a mean pixel-level AUROC of 0.94 and a mean average precision of 0.81 across the 12 leave-one-dataset-out zero-shot transfer settings. In the CVC-ColonDB-source benchmark, the model achieved a mean Dice coefficient of 0.84 across CVC-ClinicDB, Kvasir-SEG, and CVC-300. This high level of performance was consistent across datasets with distinct visual characteristics, underscoring the ability of the model to detect diverse polyp morphologies that it had not been explicitly trained to recognize.
Conclusions
This study demonstrates that an anomaly-aware vision-language framework significantly improves zero-shot generalization for colorectal lesion detection without requiring dataset-specific fine-tuning or normal training images. Without any dataset-specific fine-tuning, the model achieved a mean pixel-level AUROC of 0.94 and a mean average precision of 0.81 across the 12 leave-one-dataset-out zero-shot transfer settings. In the CVC-ColonDB-source benchmark, the model achieved a mean Dice coefficient of 0.84 across CVC-ClinicDB, Kvasir-SEG, and CVC-300. These results underscore the model’s ability to accurately localize diverse polyp morphologies across heterogeneous clinical scenarios that it was not explicitly trained to recognize. Although further work is needed to address technical limitations such as real-time inference and inclusion of non-polyp pathologies, this multimodal strategy establishes a solid technical foundation for the deployment of robust AI-based colorectal screening systems.
Full text
https://www.xiahepublishing.com/2996-3427/OnA-2026-00006
The study was recently published in the Oncology Advances.
Oncology Advances is dedicated to improving the diagnosis and treatment of human malignancies, advancing the understanding of molecular mechanisms underlying oncogenesis, and promoting translation from bench to bedside of oncological sciences. The aim of Oncology Advances is to publish peer-reviewed, high-quality articles in all aspects of translational and clinical studies on human cancers, as well as cutting-edge preclinical and clinical research of novel cancer therapies.
Follow us on X: @xiahepublishing
Follow us on LinkedIn: Xia & He Publishing Inc.