Closing the Modality Gap via Simplex-Constrained Representations
arXiv cs.AI · 24 Oct 2026, 02:57 AEST
A new geometric approach is proposed to address the modality gap in multimodal contrastive models. By changing the embedding domain to the probability simplex, significant improvements in semantic alignment and retrieval performance are achieved.
Key facts
- While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent modality gap.
- We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere, S^{d-1}, to the probability simplex, Δ^{d-1}.
- By attaching lightweight softmax adapters and using Total Variation similarity, we reduce the centroid ℓ2 gap by 97--99% across five multimodal retrieval benchmarks.
- This confirms that simplicial geometry induces genuine semantic alignment, matching or exceeding the retrieval performance of standard unconstrained models.
Read aloud
A new geometric approach is proposed to address the modality gap in multimodal contrastive models. By changing the embedding domain to the probability simplex, significant improvements in semantic alignment and retrieval performance are achieved. While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent modality gap. We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere, S^ d-one, to the probability simplex, Δ^ d-one. By attaching lightweight softmax adapters and using Total Variation similarity, we reduce the centroid ℓtwo gap by ninety seven--ninety nine percent across five multimodal retrieval benchmarks. This confirms that simplicial geometry induces genuine semantic alignment, matching or exceeding the retrieval performance of standard unconstrained models.
Sources in this cluster
- Closing the Modality Gap via Simplex-Constrained Representations
arXiv cs.AI · 24 Oct 2026, 02:57 AEST