PyTorch Lightning code for the paper "RMCL: Robust Multimodal Contrastive Learning". Slides of our talk are avialable here.

pip install -r requirements.txt
pip install -e .

We provide five pretrained weights

1. ViLT-B/32 Pretrained with MLM+ITM for 200k steps on GCC+SBU+COCO+VG (ViLT-B/32 200k) link
2. ViLT-B/32 200k finetuned on VQAv2 link
3. ViLT-B/32 200k finetuned on NLVR2 link
4. ViLT-B/32 200k finetuned on COCO IR/TR link
5. ViLT-B/32 200k finetuned on F30K IR/TR link

The synonym selection for the Geometric based attack is computed from the cosine similarity scores between word pairs based on the counter-fitting word embeddings link

See DATA.md

See TRAIN.md

See EVAL.md

If you use any part of this code and pretrained weights for your own purpose, please cite our [paper]

Stan Furrer
Zhao Meng