Scalable visual attribute recognition in e-commerce products via automated synthetic label generation

E-commerce stores rely on product catalog data, which can be enriched by automated mechanisms like visual attribute extraction, for features like search and filtering. Extracting visual attributes from product images in e-commerce is challenging due to the wide diversity in products and the high cost of manual labeling, making traditional methods that rely on human-annotated data often impractical. In this work, we introduce a scalable approach that uses CLIP models to automatically generate synthetic labeled datasets, significantly reducing the need for manual annotation. To ensure dataset quality, we employ an ensemble of CLIP models for zero-shot classification and selectively incorporate text-based models for choosing high-accuracy examples, such as DeBERTa and BART-large pre-trained for natural language inference to perform zero-shot text classification. For downstream classification model training, a CLIP image encoder is used as the backbone, while only a single vector per class is fine-tuned using the automatically labeled data. We evaluate our models through human assessments, guided by automatically generated annotation guidelines based on high-accuracy examples. We evaluated our approach on a large-scale experimental dataset spanning over a thousand product types and over a hundred visual attributes. To ensure high-quality predictions, we computed rejection thresholds to achieve at least 90% accuracy per class and found that, on average, 60.8% of predictions exceeded the threshold. Our results demonstrate that synthetic labeling with vision-language models could serve as a practical and effective approach for large-scale visual attribute extraction in e-commerce product catalogs.