Evaluating Zero-Shot Object Detection for Fine-Grained Food Ingredient Recognition: A Precision-Recall Analysis

Abstract

This paper evaluates how well a pretrained zero-shot object detector (OWLv2) recognizes food ingredients in photographs, and uses the results to build a recipe recommendation system based on an Indian recipe dataset of 5,938 recipes. Detection was tested against manually labeled ground truth across three evaluation rounds, covering up to 50 ingredient categories. Key findings:- Detection reliability is closely linked to how regular an ingredient's shape is (fruits and eggs performed well; spices and herbs performed poorly).- Ambiguous names cause failures: "drumstick" (a vegetable, but also a chicken leg) produced false detections in nearly every photo.- Precision is not a fixed property of an ingredient: it drops as the detection vocabulary grows larger. These findings led to a narrower, evidence-based vocabulary and a human-in-the-loop correction step, so users can fix detection errors before recipes are recommended.