MultiSHAP uses a mathematical approach called the Shapley Interaction Index to estimate pairwise effects between image patches and text tokens. It repeatedly masks combinations of visual and textual elements, then uses the model’s score to estimate whether the pair contributes more together than it would separately, or instead reduces the combined contribution.

The framework is designed to work without a model’s internal weights, gradients or attention data. It can produce explanations for an individual prediction as well as summaries of interaction patterns across a dataset, provided the model allows separate image and text perturbations and returns a suitable score.