# Multimodality and Large Multimodal Models (LMMs)

DevFeed: [Multimodality and Large Multimodal Models (LMMs)](<https://devfeed.tech/articles/multimodality-and-large-multimodal-models-lmms-31795.md>)

Original publisher: [Read original article](<https://huyenchip.com//2023/10/10/multimodal.html>)

Author: Chip Huyen

Published: 2023-10-10T00:00:00Z

Content type: tutorial

Language: en

Sources: [Chip Huyen](<https://devfeed.tech/sources/chip-huyen.md>)

Topics: [multimodal](<https://devfeed.tech/topics/multimodal.md>), [AI Research](<https://devfeed.tech/topics/ai-research.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-research](<https://devfeed.tech/tags/ai-research.md>), [models](<https://devfeed.tech/tags/models.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>)

## AI overview

This tutorial explains multimodal systems and Large Multimodal Models (LMMs), including their data modalities, task types, foundational approaches such as CLIP and Flamingo, and research areas such as multimodal outputs and adapter-based training.

## Source excerpt

For a long time, each ML model operated in one data mode - text (translation, language modeling), image (object detection, image classification), or audio (speech recognition). However, natural intelligence is not limited to just a single modality. Humans can read, talk, and see. We listen to music to relax and watch out for strange noises to detect danger. Being able to work with multimodal data is essential for us or any AI to operate in the real world. OpenAI noted in their GPT-4V system card that "incorporating additional modalities (such as image inputs) into LLMs is viewed by some as a key frontier in AI research and development." Incorporating additional modalities to LLMs (Large Language Models) creates LMMs (Large Multimodal Models). Not all multimodal systems are LMMs. For example, text-to-image models like Midjourney, Stable Diffusion, and Dall-E are multimodal but don't have a language model component. Multimodal can mean one or more of the following: Input and output are of different modalities (e.g. text-to-image, image-to-text) Inputs are multimodal (e.g. a system that can process both text and images) Outputs are multimodal (e.g. a system that can generate both text and images) This post covers multimodal systems in general, including LMMs. It consists of 3 parts. Part 1 covers the context for multimodality, including why multimodal, different data modalities, and types of multimodal tasks. Part 2 discusses the fundamentals of a multimodal system, using the examples of CLIP, which lays the foundation for many future multimodal systems, and Flamingo, whose impressive performance gave rise to LMMs. Part 3 discusses some active research areas for LMMs, including generating multimodal outputs and adapters for more efficient multimodal training, covering newer multimodal systems such as BLIP-2, LLaVA, LLaMA-Adapter V2, LAVIN, etc. The post is long. Feel free to skip to the sections most interesting to you. ⚠ Ambiguous terminology ⚠ Multimodal data can als