babylonpolice.com

Despite the excellent performance of vision-language pre-trained models
(VLPs) on conventional VQA task, they still suffer from two problems: First,
VLPs tend to rely on language biases in datasets and fail to generalize to
out-of-distribution (OOD) data. Second, they are inefficient in terms of memory
footprint and computation. Although promising progress has been made in both
problems, most existing works tackle them independently. To facilitate the
application of VLP to VQA tasks, it is imperative to jointly study VLP
compression and OOD robustness, which, however, has not yet been explored. This
paper investigates whether a VLP can be compressed and debiased simultaneously
by searching sparse and robust subnetworks. To this end, we systematically
study the design of a training and compression pipeline to search the
subnetworks, as well as the assignment of sparsity to different
modality-specific modules. Our experiments involve 3 VLPs, 2 compression
methods, 4 training methods, 2 datasets and a range of sparsity levels and
random seeds. Our results show that there indeed exist sparse and robust
subnetworks, which are competitive with the debiased full VLP and clearly
outperform the debiasing SoTAs with fewer parameters on OOD datasets VQA-CP v2
and VQA-VS. The codes can be found at
https://github.com/PhoebusSi/Compress-Robust-VQA.