Definition
A large multimodal model (LMM) is a model trained at substantial scale that works across more than one modality in its inputs, outputs, or both. A modality is a type of data, such as text, images, audio, or video. The model may process several modalities together as input, but its supported inputs and outputs do not have to match.
A text-only large language model works with tokenized language. An LMM needs a way to represent each supported modality and relate those representations during processing or generation. Many current LMMs use a language model as the component that interprets instructions and generates text, but that is one design rather than the definition.
Simple example
Suppose a user uploads a screenshot of a disabled checkout button and asks, “Why can’t I continue?” An LMM can process the screenshot together with the question, notice an unchecked consent box, and answer in text.
This is still multimodal even though the response contains no image. An application using a text-only LLM would need a separate component to convert the screenshot into text or structured data that the LLM can accept. The application is then multimodal, while the LLM remains text-only. The result depends on what the conversion preserves.
Why it matters
Multimodal input lets an application use information that is awkward to express as plain text. Layout, visual relationships, tone of voice, and events across video frames can all affect the result. Converting everything to text first may discard some of that information.
The extra modalities also add engineering constraints. File formats, media size, processing time, cost, and safety rules can differ from those for text. Evaluation should cover the modality combinations the application actually uses. Good image understanding, for example, does not establish good audio performance.
One important nuance
“Multimodal” does not mean that a model accepts and produces every modality. A model that accepts text and images but returns only text is multimodal. Another may generate images from text without accepting audio or video. Check the supported input and output pairs instead of treating multimodal capability as one universal feature.