Instead of following visual salience alone, Eyegent chooses where to look next based on uncertainty about a language-guided task.
Eyegent takes an image and a natural-language query, builds possible interpretations of the relevant object or region, and keeps a probability distribution over them. It then directs simulated fixations toward areas expected to provide the most useful information, updates its beliefs using a vision-language model, and stops when its remaining uncertainty reaches a set risk threshold.
The system uses off-the-shelf components and no gaze supervision or task-specific training. On RefCOCO-Gaze and AiR-D, it performed as well as or better than other zero-shot methods on several measures of semantic, spatial and structural scanpaths, although supervised systems remained ahead on some spatial and timing measures.
How the system chose where to look
Eyegent was evaluated on RefCOCO-Gaze and AiR-D, datasets that pair language-guided visual tasks with human eye-tracking data. On RefCOCO-Gaze, it had the strongest results among zero-shot methods on measures of semantic similarity, fixation-distance similarity and scanpath matching with fixation durations, as well as on one measure of positional similarity. It was consistently among the best-performing zero-shot systems across the reported metrics.
On AiR-D, Eyegent led the zero-shot competitors on both semantic measures and matched the best spatial results. In an example analysis, the system’s probability for the correct interpretation increased while the probability of alternatives fell, and its uncertainty decreased as it made more fixations. These results are consistent with the system’s design, in which each fixation is selected to gather information rather than simply to follow visually prominent areas.
Why uncertainty matters
The work presents a way for an AI vision system to treat looking as a sequence of decisions. That lets it use language, competing interpretations and uncertainty to decide where to inspect next, rather than treating visual attention as only a response to salience.
Because Eyegent does not learn from human gaze or task-specific training, its performance suggests that explicit reasoning about uncertainty can produce scanpaths with meaningful similarities to human viewing. The framework also makes the system’s beliefs and stopping decisions explicit, which could provide testable predictions for future comparisons with human eye movements and brain activity.
Evidence and caveats
The evidence comes from evaluations on the official AiR-D test set and the validation split of RefCOCO-Gaze, using human eye-tracking data as a reference. Eyegent was compared with two zero-shot methods, task-specific supervised systems and inter-human agreement. The system used Qwen3-VL-8B-Instruct and CLIPSeg without fine-tuning or task-specific training.
The results do not show that Eyegent outperforms supervised systems overall. The supervised baselines retained an advantage on some measures, especially spatial and temporal alignment. The reported tests are also limited to the two benchmarks and the particular off-the-shelf components used in this implementation. The framework’s proposed links to human brain networks remain a model-based account; the excerpts do not report direct neural validation.
// Source
International Journal of Computer Vision · 2026 · DOI: 10.1007/s11263-026-03042-4
Authors: Alessandro D’Amelio, Sabrina Patania, Luca Annese, Giuseppe Boccignone
Institutions: University of Milan, University of Milano-Bicocca