Multimodal models: AI systems that combine text, images and audio
Multimodal models are machine learning systems that process and combine data from multiple types, such as text, images, audio, and video. They use separate encoders for each modality and fuse the representations to enable tasks like image captioning and visual question answering. These models can also generate output in one modality from input in another.
What is a multimodal model?
Multimodal AI refers to machine learning models that process and integrate information from multiple modalities, including text, images, audio, and video. According to IBM, these systems combine different data types to achieve a more comprehensive understanding, and they are characterized by heterogeneity, connections, and interactions. Such systems can also maintain performance when one modality is missing by relying on other modalities. IBM's article
A practical example from Google Cloud shows a model that receives a photo of cookies and produces a written recipe, or the reverse. Google notes that multimodal AI expands generative capabilities so that users can prompt with virtually any input to generate virtually any output type, which distinguishes it from generative AI that typically works with a single modality. Google Cloud's multimodal AI overview
How multimodal models are built
A common architecture uses dedicated encoders for each modality, such as convolutional networks for images and transformers for text. The Vision Transformer research demonstrated that a pure transformer applied directly to sequences of image patches can achieve strong image classification results, offering a flexible alternative for visual encoding. Vision Transformer paper
After encoding, the model aligns and combines the different representations, often with attention mechanisms or joint embedding spaces. The main engineering challenges involve representation, alignment, reasoning, and generation across modalities, which require careful design of the fusion step.
Large pre-training datasets that contain interleaved or paired modalities, such as image-text pairs or video with narration, help these models learn cross-modal connections. The choice of pre-training objective strongly influences downstream capabilities.
Notable models and research
The CLIP paper showed that a simple pre-training task, predicting which caption matches an image, can learn transferable visual representations from a dataset of 400 million image-text pairs collected from the internet. The resulting model performs zero-shot classification across many domains. CLIP paper
Flamingo introduced a family of visual language models that bridge pretrained vision-only and language-only models and handle arbitrarily interleaved sequences of images, videos, and text. This design enables few-shot learning on open-ended and closed-ended multimodal tasks. Flamingo paper
LLaVA connects a vision encoder to a large language model and uses language-only GPT-4 to generate multimodal instruction-following data. The model demonstrates impressive chat abilities on unseen images and achieves strong accuracy on visual tasks. LLaVA paper
For practical use, the Hugging Face Transformers library supports image-text-to-text models, also called vision language models, and provides pipelines for inference. The documentation shows how to load a model, prepare chat inputs with images and text, and generate answers. Hugging Face documentation
Benefits and considerations
By combining complementary information, multimodal models can deliver more complete understanding than unimodal systems. They also enable more natural interaction, such as a voice assistant that understands visual cues or a bird identification app that uses both a photo and a song clip.
Building effective multimodal models remains challenging because different modalities have distinct structures and require careful alignment. Issues like missing data, noise, and the need for large labeled or paired datasets are still active areas of research.
Despite these challenges, pre-trained multimodal models and libraries such as Transformers make it easier for developers to integrate such capabilities into applications. The availability of open-domain models and documentation lowers the barrier to entry.
| Model | Inputs | Outputs | Training approach |
|---|---|---|---|
| CLIP | Image and text | Text classification or retrieval | Predict which caption matches which image using 400 million image-text pairs |
| Flamingo | Image or video and text | Text responses, captions | Trained on interleaved multimodal web data for few-shot learning |
| LLaVA | Image and text | Text answers, captions | Instruction tuning on GPT-4 generated multimodal data |
Frequently asked questions
What is the difference between generative AI and multimodal AI?
Generative AI creates new content from a prompt, often for a single type like text or image. Multimodal AI extends this by accepting and producing multiple modalities, such as converting an image to text or vice versa, as noted in Google Cloud's use case overview.
How do multimodal models handle missing data?
By processing several input types, a multimodal model can rely on other modalities when one is absent or noisy. IBM mentions this resilience as a key benefit of combining multiple data sources.
What types of tasks can multimodal models perform?
They support visual question answering, image captioning, speech recognition, and cross-modal generation. The Flamingo paper demonstrates few-shot learning on such tasks, and Hugging Face documents many practical examples.
How can developers get started with multimodal models?
Pre-trained models are available through libraries like Hugging Face Transformers, which provides pipelines for image-text-to-text inference. Developers can load a model and processor, prepare inputs, and generate outputs with a few lines of code.