Click here to flash read.
In this paper, we explore the potential of Vision-Language Models (VLMs),
specifically CLIP, in predicting visual object relationships, which involves
interpreting visual features from images into language-based relations. Current
state-of-the-art methods use complex graphical models that utilize language
cues and visual features to address this challenge. We hypothesize that the
strong language priors in CLIP embeddings can simplify these graphical models
paving for a simpler approach. We adopt the UVTransE relation prediction
framework, which learns the relation as a translational embedding with subject,
object, and union box embeddings from a scene. We systematically explore the
design of CLIP-based subject, object, and union-box representations within the
UVTransE framework and propose CREPE (CLIP Representation Enhanced Predicate
Estimation). CREPE utilizes text-based representations for all three bounding
boxes and introduces a novel contrastive training strategy to automatically
infer the text prompt for union-box. Our approach achieves state-of-the-art
performance in predicate estimation, mR@5 27.79, and mR@20 31.95 on the Visual
Genome benchmark, achieving a 15.3\% gain in performance over recent
state-of-the-art at mR@20. This work demonstrates CLIP's effectiveness in
object relation prediction and encourages further research on VLMs in this
challenging domain.
No creative common's license