Multimodal AI

You might say

Can AI understand my screenshot too, not just the text I send it?

AI that can combine information such as text, images, audio, or video in one taskSend a screenshot of a phone checkout page with the question “Why is the payment button missing?” and the AI can consider both the visible layout and your written question.

Know first

AI Application Basics

Multimodality

Separate input modes from output modes

Current example: use a screenshot to find why the payment button disappearedThe theme color marks what this example actually uses

Input modalitiesWhat you give the AI

Text
Image
Speech / audioSpeech is audio
Video

AI model / systemProcesses what it actually supports

Output modalitiesWhat the AI returns

Text
Image
Speech / audioSpeech is audio
Video

This example takes text and images as input and returns text. Supporting image input does not mean it can output images.

What different modes contribute

A mode is a form of information: Text, images, audio, and video are common modes. Support for one does not mean support for all.

The same mode can be on either side: Audio can be input for transcription or output that reads an answer aloud.

Check the specific product: Supported inputs, outputs, and file types vary by model and product.

You can say this to an AI Agent

This screenshot shows the checkout page on a phone. The payment button should be at the bottom, but it is not visible. First, use only the screenshot to identify any visible overlap or layout clues, and say when the image is not enough to confirm something. Then inspect the mobile layout in the project, fix the cause, and verify at the same screen size that the button is visible and clickable.

Learn next

1Context Window→2Structured Output→3Tool Calling→

Recommended tool

Web fine-tuning tool · Selector Pro

Select page elements for precise AI edits

Click a button, text, or card, note the change, then copy its exact location to AI.

Open Selector Pro website

You want AI to explain why the payment button is not visible on a mobile checkout page. What is the best approach?

A. Attach the current screenshot, say where the button should appear, and ask it to separate visible evidence from causes that require checking the codeB. Only say “the page is broken, fix it” without a screenshot or a specific symptomC. Upload only the screenshot and assume the AI can now see all code, hidden state, and runtime behavior