Agentic prompt optimization with visual contrastive reasoning for fine-grained classification

Fine-grained visual recognition demands attention to subtle, localized differences that current multimodal large language models (MLLMs) often overlook when guided by generic prompts. We propose APO-Pair, a prompt-optimization framework that learns classification rules by contrasting image pairs. A multimodal agent views these pairs, judges whether they depict the same fine-grained class, and iteratively updates a single verification prompt using structured, interpretable YAML-style patches. Each prompt is treated as an arm in a Thompson-sampling bandit whose Beta posterior is updated from pairwise successes and failures, balancing exploration with exploitation without touching model weights. Experiments on seven challenging 3-class fine-grained splits derived from CUB-200, iNaturalist, and VegFru, APO-Pair improves top-1 accuracy over prompt-optimization baselines by up to 17 percentage points. These results suggest that pairwise visual contrast is a useful signal for prompt optimization in controlled fine-grained settings.