Push events from a voice AI agent to the browser

AI Agentsvoice-controlled website navigation

Return a JSON event from a voice AI agent tool and deliver it to the browser connected to the call.

call-fabric

The claim

FunctionResult.swml_user_event(event) wraps a user_event verb in a one-verb SWML document and adds it as a SWML action on the tool result. The bundled schema says user_event “allows the user to set and send events to the connected client on the call”. It notes the verb is “commonly used with the browser-sdk” and that “the event object can be any valid JSON object”. The handler validates the supplied slot id and chooses the event payload. The result carries the event as data and a response string as text, and the verifier asserts both exactly.

How it works

def select_slot(self, args, raw_data):
    slot = args.get("slot")
    if slot not in SLOTS:
        return FunctionResult("INVALID: that slot is not on the page.")
    return FunctionResult(f"Noted {SLOTS[slot]} for the caller.").swml_user_event(
        {"type": "slot_selected", "slot": slot, "label": SLOTS[slot]})

The function result the platform receives:

{"response": "Noted Thursday 2pm for the caller.",
 "action": [{"SWML": {"version": "1.0.0", "sections": {"main": [
   {"user_event": {"event": {"type": "slot_selected", "slot": "thu-14",
                             "label": "Thursday 2pm"}}}]}}}]}

user_event requires event; the schema rejects the verb without one. The slot parameter carries an enum of the ids on the page. The handler returns no action for an id outside SLOTS. A valid id the model supplies wrongly still emits the event: the check bounds the ids, not the model’s accuracy. Nothing here reserves the slot; the event reports a choice.

The swml/ surface places a user_event in a plain document before the ai verb. Your backend can push the same shape mid-call with the REST command calling.user_event. The vendored REST spec’s variant for that command requires command, id and params, and event is the one required key in params. The verifier reads that from the spec.

Limitations

The verifier proves the document. How a page receives the event is the client side of a live call. The Browser SDK’s documentation covers the call a page subscribes with.

The schema gives user_event no result variable. If the page must acknowledge, have it call your backend.

What to change first

Remove "label": SLOTS[slot] from the event and run the verifier. The exact payload assertion fails. On a live call the page would have to look the label up itself, so put in the event everything the page needs to render.

Where this sits

python/app.py

"""Push events from an agent to the browser.

A tool result can carry a SWML `user_event`, whose `event` is any JSON object.
The bundled schema describes it as sending events "to the connected client on
the call". A page running the Browser SDK and subscribed to user events can
receive the structured event. The handler validates the supplied slot id and
chooses the event payload, so the event carries the handler's fields rather
than the model's paraphrase of them.

`FunctionResult.swml_user_event(event)` wraps the verb in a one-verb SWML
document and adds it as a `SWML` action. Your backend can push the same shape
with the REST command `calling.user_event`, whose spec variant requires
`params.event` (tools/openapi/rest.json, Calling.CallRequest).

Written against signalwire-sdk 3.0.1.
"""
import os

from dotenv import load_dotenv
from signalwire import AgentBase, FunctionResult

# the SDK does not read .env for you
load_dotenv()

SLOTS = {"thu-14": "Thursday 2pm", "fri-10": "Friday 10am", "sat-09": "Saturday 9am"}


class BookingAgent(AgentBase):
    def __init__(self):
        super().__init__(name="booking", route="/booking")
        self.prompt_add_section(
            "Role",
            "You book workshop slots for Ridgeline Cycles from a web page that "
            "shows the available times. Use select_slot when the caller picks one.",
        )

    @AgentBase.tool(
        name="select_slot",
        description="Record which workshop slot the caller has chosen.",
        parameters={
            "type": "object",
            "properties": {"slot": {"type": "string", "enum": sorted(SLOTS),
                                    "description": "The slot id shown on the page."}},
            "required": ["slot"],
        },
    )
    def select_slot(self, args, raw_data):
        slot = args.get("slot")
        if slot not in SLOTS:
            return FunctionResult("INVALID: that slot is not on the page.")
        # the event carries the selection as data; the response is a sentence
        return FunctionResult(
            f"Noted {SLOTS[slot]} for the caller."
        ).swml_user_event({"type": "slot_selected", "slot": slot, "label": SLOTS[slot]})


agent = BookingAgent()

if __name__ == "__main__":
    agent.serve(host="0.0.0.0", port=int(os.getenv("PORT", "3000")))

# Push an event to the connected client in plain SWML: `user_event` carries
# any JSON object to the client on the call. This document places it after
# `answer` and before `ai`.
version: "1.0.0"
sections:
  main:
    - answer: {}
    - user_event:
        event:
          type: "call_answered"
          agent: "booking"
    - ai:
        prompt:
          text: >
            You book workshop slots for Ridgeline Cycles from a web page that
            shows the available times.

Run it

This recipe is the agent side. The page side is outside it: a Browser SDK client that dials an address and subscribes to user events. The Browser SDK reference covers that client.

cd python
pip install -r requirements.txt
cp ../.env.example .env          # set SWML_BASIC_AUTH_PASSWORD
python app.py

The webhook needs a public HTTPS URL. For a local run, expose port 3000 with a tunnel such as ngrok and use that hostname. Point a number’s SWML webhook, or a Fabric address’s SWML handler, at https://<user>:<password>@<your-host>/booking/. Dial that number or address from your Browser SDK page, pick a slot by voice, and watch the page.

Verify it

No network, no account.

python verify.py          # from the recipe folder, not python/

It renders and validates the SWML, runs the handler, and asserts the following.

the tool’s slot parameter carries the three slot ids as an enum
a valid slot returns the exact payload: one SWML action holding one user_event with the exact event object, and the exact response
that inline document validates against the bundled schema
a user_event with no event fails the schema
an unknown slot returns INVALID with no action
the REST spec’s calling.user_event variant requires command, id and params, with event the one required key in params
the plain-SWML surface validates and its verbs are answer, user_event, ai, with the exact event

Report an issue

Push events from a voice AI agent to the browser

The claim

How it works

Limitations

What to change first

Where this sits

Before this

Related

Seen in a build

Seen in a build

Seen in a build

Seen in a build