What is VLM?

What is VLM (Vision Language Model)?

VLM is a model with a multi-modal architecture that learns to associate information from image and text modalities. The focus of the multi-modal learning is to pre-train an model on vision and language task and improve the downstream task performance such as VQA (Vision Question Answering). 

Why VLM? What are the Use Cases?

1. Image Search and Retrieval / 2. Robotics: Integrating VLMs can allow robots to understand visual instructions or describe their surroundings, enabling more complex object ineractions and collaboration.
Read more at: https://viso.ai/deep-learning/vision-language-models/

What are the types of VLMs?

Depending on a fusion mechanism, there are  "Dual Encoder - CLIP", "Fusion Encoer - ViLBERT", and "Hybrid Encoder - ?". The fusion encoding directly combines the image and text embeddings. The dual Encoder is computationally more efficient and performs similarly with the fusion encoder in cross-modality retrieval tasks (image-text retrieval), and the fusion encoder is more sutiable for more complext tasks such as VQA. 

What is ViLBERT (Vision Language BERT)?

It has two streamlines for vision and language input respectively. It integrates co-attention modules that conditions each other's modalitiy. There are two pre-training tasks for ViLBERT. One is  recovering masked tokens and the other is predicting the alignment of the image and text input pair. 

What is simply discretizing the space of visual inputs via clustering?  (ViLBERT page 3)

 

 

Vision Language Models: Exploring Multimodal AI
Read more at: https://viso.ai/deep-learning/vision-language-models/

ViLBERT: https://arxiv.org/pdf/1908.02265v1