Prompt-based medical image segmentation training pipeline integrating BiomedCLIP encoders, SAM with LoRA adapters, and learnable prompt tokens for colonoscopy polyp detection. The architecture branches into text, image, and saliency pathways, with BiomedCLIP processing natural language prompts and colonoscopy images through partially unfrozen encoders, while SAM components receive enriched dense and sparse prompts via projection layers and context vectors. M2IB information bottleneck with learnable gates generates saliency heatmaps refined by differentiable CRF, feeding bounding box extraction into SAM's mask decoder for final segmentation. Training uses DHN-NCE contrastive loss on text embeddings and Dice loss on predicted masks against ground truth, enabling efficient fine-tuning of foundation models for clinical image analysis. Fork and customize this pipeline on Diagrams.so to adapt prompt engineering strategies, adjust LoRA ranks, or integrate alternative vision encoders for your medical imaging tasks.