Let a voice AI agent see the caller's camera

AI Agentsmultimodal voice AI with vision

Enable visual input, choose a vision model, and play a filler while the voice AI agent analyzes the caller's camera.

visionvideo

The claim

The bundled schema documents enable_vision as “Enables visual input processing for the AI Agent. When set to true, the AI Agent will be able to utilize visual processing capabilities, while leveraging the get_visual_input function”. Its default is false. vision_model names the model. The schema says “Allowed values are gpt-4o-mini, gpt-4.1-mini, and gpt-4.1-nano”, and its type also admits any other string. You do not define get_visual_input. The schema lists it among the internal fillers, not among the functions you register, and not among the native functions you can switch on. What you can give it is a filler, a phrase filed under its name.

How it works

class EyesAgent(AgentBase):
    def __init__(self):
        super().__init__(name="eyes", route="/eyes")
        self.prompt_add_section("Role", "You are the workshop desk at Ridgeline Cycles, on a video call. ...")
        self.set_params({"enable_vision": True, "vision_model": VISION_MODEL})
        self.set_internal_fillers({"get_visual_input": {"en-US": LOOKING}})

What the platform receives, inside the ai verb:

"params": {"enable_vision": true, "vision_model": "gpt-4o-mini"},
"SWAIG": {"internal_fillers": {"get_visual_input": {"en-US": ["Let me take a look.",
                                                              "One moment while I look at that."]}}}

The prompt tells the agent to look before it answers and to describe what it sees. The SDK warns at registration if you name a filler the schema does not know, which is the check that keeps get_visual_input spelt right.

Limitations

You prove the document and the schema. When the agent looks, what the model sees, and how it describes it are the platform’s side of a live video call.

The schema names three models and admits any string; which one to use is a quality and cost decision this recipe does not make for you.

What to change first

Change VISION_MODEL in .env to gpt-4.1-mini and run the agent. The document renders with that model in ai.params, because the environment is where the name comes from. The verifier sets gpt-4.1-nano for the same reason, and fails if the app ignores the variable.

Where this sits

Before this

Route a call to an AI agent

Give a voice AI agent a video avatar Cover AI agent tool latency with fillers

python/app.py

"""Let an agent see the caller's camera.

Two `ai.params` and one filler. The bundled schema documents `enable_vision`
as "Enables visual input processing for the AI Agent. When set to `true`, the
AI Agent will be able to utilize visual processing capabilities, while
leveraging the `get_visual_input` function." `vision_model` names the model:
"Allowed values are `gpt-4o-mini`, `gpt-4.1-mini`, and `gpt-4.1-nano`." The
`get_visual_input` function is the platform's; you do not define it. What you
can give it is an internal filler, a phrase filed under its name.

Written against signalwire-sdk 3.0.1 (AgentBase.set_params,
set_internal_fillers).
"""
import os

from dotenv import load_dotenv
from signalwire import AgentBase

# the SDK does not read .env for you
load_dotenv()

VISION_MODEL = os.getenv("VISION_MODEL", "gpt-4o-mini")
LOOKING = ["Let me take a look.", "One moment while I look at that."]


class EyesAgent(AgentBase):
    def __init__(self):
        super().__init__(name="eyes", route="/eyes")
        self.prompt_add_section(
            "Role",
            "You are the workshop desk at Ridgeline Cycles, on a video call. When the "
            "caller shows you a part or a problem, look at it before you answer, and "
            "describe what you see before you give advice.")
        # the switch and the model; the platform supplies get_visual_input
        self.set_params({"enable_vision": True, "vision_model": VISION_MODEL})
        # the phrases filed under the platform's function name
        self.set_internal_fillers({"get_visual_input": {"en-US": LOOKING}})


agent = EyesAgent()

if __name__ == "__main__":
    agent.run()

Run it

cd python
pip install -r requirements.txt
cp ../.env.example .env          # then edit .env: the basic-auth pair; VISION_MODEL if you like
python app.py

The webhook needs a public HTTPS URL. For a local run, expose port 3000 with a tunnel such as ngrok and use that hostname. Point a video-capable client or Fabric address at https://<user>:<password>@<your-host>/eyes/, hold a part up to the camera and ask about it.

Verify it

No network, no account.

cd ..                     # back to the recipe folder
python verify.py

The verifier renders the agent’s document, validates it, and asserts the following.

ai.params carries enable_vision: true and the vision_model the environment set, gpt-4.1-nano, which is not the app’s default
SWAIG.internal_fillers is exactly get_visual_input with the two phrases, and no function named get_visual_input is registered
the prompt tells the agent to look before it answers and to describe what it sees
the schema’s enable_vision description names get_visual_input and its default is false
the schema’s vision_model is exactly three named values plus any string
the schema lists get_visual_input among the internal fillers and not among the native functions

Report an issue