Bottom Linear Gradient  Lines image

Article

6 min

read

Build a Talking Jack-o'-Lantern with a SignalWire AI Agent

A SignalWire AI agent in a jack-o'-lantern

Adam Kalsey

Adam Kalsey

EVP, Product

In this article

Share

Angular Gradient Image

Build it free.

Create a space and ship your first call flow in minutes.

Subscribe

Tags

Voice AI

AI Agents

Developer Tutorials

Three kids walk up the driveway on Halloween night. A scarecrow with a burning jack-o'-lantern for a head watches them come.

"A witch," it says. "How daring. How utterly unprecedented."

The witch laughs. The vampire next to her asks the pumpkin how old it is. "I was carved on a Tuesday," it says, and goes back to ignoring them as attentively as it can.

The pumpkin is real, and it has a secret hidden inside: a SignalWire AI agent on a video call. A computer in the prop keeps the call open all night, with a webcam for eyes and a speaker for a mouth. The agent sees who's in front of it and talks to them in character. When a small child is in the group, it stays as grand as ever, but nobody in front of it is the joke.

This series shows how I built it. The excerpts in each post show the ideas, and the ai-jackolantern repository has the complete, working code.

What the series covers

The series has nine parts, and each one adds a working piece:

  1. The system and the call (this post).

  2. The agent: a personality and three modes, as contexts and steps.

  3. Tools: what the AI calls on my server, and why its material lives in code.

  4. Vision: letting the agent see, and checking the camera before any decision that matters.

  5. The stage: detecting who is standing in front of the pumpkin, in the browser.

  6. The voice: audio effects in the browser.

  7. Pacing and state: the server keeps score.

  8. The operator: a person with a phone who can overrule the agent.

  9. Testing it, and running it for a whole night.

By the end of this post, a browser holds a video call to a SignalWire AI agent.

The pieces

The pumpkin runs in four places:

Piece

Runs on

Job

The pumpkin page

The computer in the prop

Holds the call, sends the camera and mic, plays the voice through audio effects, detects who is on stage

The AI agent

SignalWire

Listens, looks, talks, and calls tools

My server

Any host with a public HTTPS address

Serves the agent's configuration and the page, runs the tools, keeps the state of each visit

The phone remote

The operator's phone

Starts and stops the show, adjusts settings, switches modes

The server is one Python process built on our Python SDK (signalwire-sdk). It serves the agent, the pumpkin page, and the page's APIs from one port.

How a call flows

Every time the page dials, the call goes through the same steps:

  1. The page asks my server for a guest token at /eyes-token.

  2. The page dials the agent's address with that token, over video.

  3. SignalWire fetches the agent's configuration from /pumpkin on my server. The configuration is SWML, our markup for call flows, and the SDK writes it from the Python code.

  4. SignalWire runs the ai verb from that SWML: speech recognition, the model, text to speech, and vision.

  5. When the model calls a tool, SignalWire posts the request to /pumpkin/swaig on my server, and my code answers.

Set up SignalWire from code

The page dials an address that belongs to a resource in my SignalWire space. My server creates that resource when it starts, so a new deployment needs no setup in the dashboard. The resource is an SWML webhook, which means SignalWire fetches the call's instructions from a URL when a call arrives. The setup code finds the resource by name, creates it if it's missing, and corrects its URL if it points somewhere else. Running it twice changes nothing.

resource = next(
    (r for r in self._list("resources/swml_webhooks") if r["swml_webhook"]["name"] == RESOURCE_NAME),
    None,
)
if resource is None:
    resource = self._call("POST", "resources/swml_webhooks", json={
        "name": RESOURCE_NAME,
        "used_for": "calling",
        "primary_request_url": self.agent_url,
        "primary_request_method": "POST",
    })
elif resource["swml_webhook"]["primary_request_url"] != self.agent_url:
    self._call("PATCH", f"resources/swml_webhooks/{resource['id']}", json={
        "primary_request_url": self.agent_url,
        "primary_request_method": "POST",
    })

addresses = list(self._list(f"resources/swml_webhooks/{resource['id']}/addresses"))
resource = next(
    (r for r in self._list("resources/swml_webhooks") if r["swml_webhook"]["name"] == RESOURCE_NAME),
    None,
)
if resource is None:
    resource = self._call("POST", "resources/swml_webhooks", json={
        "name": RESOURCE_NAME,
        "used_for": "calling",
        "primary_request_url": self.agent_url,
        "primary_request_method": "POST",
    })
elif resource["swml_webhook"]["primary_request_url"] != self.agent_url:
    self._call("PATCH", f"resources/swml_webhooks/{resource['id']}", json={
        "primary_request_url": self.agent_url,
        "primary_request_method": "POST",
    })

addresses = list(self._list(f"resources/swml_webhooks/{resource['id']}/addresses"))
resource = next(
    (r for r in self._list("resources/swml_webhooks") if r["swml_webhook"]["name"] == RESOURCE_NAME),
    None,
)
if resource is None:
    resource = self._call("POST", "resources/swml_webhooks", json={
        "name": RESOURCE_NAME,
        "used_for": "calling",
        "primary_request_url": self.agent_url,
        "primary_request_method": "POST",
    })
elif resource["swml_webhook"]["primary_request_url"] != self.agent_url:
    self._call("PATCH", f"resources/swml_webhooks/{resource['id']}", json={
        "primary_request_url": self.agent_url,
        "primary_request_method": "POST",
    })

addresses = list(self._list(f"resources/swml_webhooks/{resource['id']}/addresses"))
resource = next(
    (r for r in self._list("resources/swml_webhooks") if r["swml_webhook"]["name"] == RESOURCE_NAME),
    None,
)
if resource is None:
    resource = self._call("POST", "resources/swml_webhooks", json={
        "name": RESOURCE_NAME,
        "used_for": "calling",
        "primary_request_url": self.agent_url,
        "primary_request_method": "POST",
    })
elif resource["swml_webhook"]["primary_request_url"] != self.agent_url:
    self._call("PATCH", f"resources/swml_webhooks/{resource['id']}", json={
        "primary_request_url": self.agent_url,
        "primary_request_method": "POST",
    })

addresses = list(self._list(f"resources/swml_webhooks/{resource['id']}/addresses"))

The agent's URL carries a basic-auth user and password, so only SignalWire can fetch the configuration. I derive the password from the API token with an HMAC, so it comes out the same after every restart. That means a call in progress keeps working when the server reloads.

os.environ.setdefault("SWML_BASIC_AUTH_USER", "pumpkin")
api_token = os.environ.get("SIGNALWIRE_API_TOKEN")
if api_token and not os.environ.get("SWML_BASIC_AUTH_PASSWORD"):
    os.environ["SWML_BASIC_AUTH_PASSWORD"] = hmac.new(
        api_token.encode(), b"scarecrow swml basic auth", hashlib.sha256
    ).hexdigest()[:32]
os.environ.setdefault("SWML_BASIC_AUTH_USER", "pumpkin")
api_token = os.environ.get("SIGNALWIRE_API_TOKEN")
if api_token and not os.environ.get("SWML_BASIC_AUTH_PASSWORD"):
    os.environ["SWML_BASIC_AUTH_PASSWORD"] = hmac.new(
        api_token.encode(), b"scarecrow swml basic auth", hashlib.sha256
    ).hexdigest()[:32]
os.environ.setdefault("SWML_BASIC_AUTH_USER", "pumpkin")
api_token = os.environ.get("SIGNALWIRE_API_TOKEN")
if api_token and not os.environ.get("SWML_BASIC_AUTH_PASSWORD"):
    os.environ["SWML_BASIC_AUTH_PASSWORD"] = hmac.new(
        api_token.encode(), b"scarecrow swml basic auth", hashlib.sha256
    ).hexdigest()[:32]
os.environ.setdefault("SWML_BASIC_AUTH_USER", "pumpkin")
api_token = os.environ.get("SIGNALWIRE_API_TOKEN")
if api_token and not os.environ.get("SWML_BASIC_AUTH_PASSWORD"):
    os.environ["SWML_BASIC_AUTH_PASSWORD"] = hmac.new(
        api_token.encode(), b"scarecrow swml basic auth", hashlib.sha256
    ).hexdigest()[:32]

The server's .env holds SIGNALWIRE_SPACE, SIGNALWIRE_PROJECT_ID, and SIGNALWIRE_API_TOKEN, plus SWML_PROXY_URL_BASE, the server's public HTTPS address, so SignalWire knows where to fetch the agent.

Mint a guest token

Anyone can read what's in a browser page, so the page never gets the project credentials. My server mints a guest token for it instead, and that token can dial one address and expires in two hours.

def guest_token(self):
    """A short-lived token that can dial the pumpkin and nothing else."""
    address = self.ensure()
    expires = int(time.time()) + TOKEN_LIFETIME_SECONDS
    body = self._call("POST", "guests/tokens", json={
        "allowed_addresses": [address["id"]],
        "expire_at": expires,
    })
    return {"token": body["token"], "expiresAt": expires * 1000, "destination": video_channel(address)}
def guest_token(self):
    """A short-lived token that can dial the pumpkin and nothing else."""
    address = self.ensure()
    expires = int(time.time()) + TOKEN_LIFETIME_SECONDS
    body = self._call("POST", "guests/tokens", json={
        "allowed_addresses": [address["id"]],
        "expire_at": expires,
    })
    return {"token": body["token"], "expiresAt": expires * 1000, "destination": video_channel(address)}
def guest_token(self):
    """A short-lived token that can dial the pumpkin and nothing else."""
    address = self.ensure()
    expires = int(time.time()) + TOKEN_LIFETIME_SECONDS
    body = self._call("POST", "guests/tokens", json={
        "allowed_addresses": [address["id"]],
        "expire_at": expires,
    })
    return {"token": body["token"], "expiresAt": expires * 1000, "destination": video_channel(address)}
def guest_token(self):
    """A short-lived token that can dial the pumpkin and nothing else."""
    address = self.ensure()
    expires = int(time.time()) + TOKEN_LIFETIME_SECONDS
    body = self._call("POST", "guests/tokens", json={
        "allowed_addresses": [address["id"]],
        "expire_at": expires,
    })
    return {"token": body["token"], "expiresAt": expires * 1000, "destination": video_channel(address)}

The response also tells the page to dial the address's video channel, so the agent can see. The token route sits behind a password (EYES_PASSWORD), because each token can start a billed call.

Dial from the browser

The page uses our Browser SDK v4 (@signalwire/js), which takes a token provider, an object with authenticate() and refresh() methods. My provider fetches from /eyes-token, and the SDK calls refresh() before a token expires, so one call can outlive many tokens.

class PumpkinTokens {
  async authenticate() {
    const response = await fetch("/eyes-token", { method: "POST" });
    if (!response.ok) throw new Error(`token request failed: ${response.status}`);
    const { token, expiresAt, destination } = await response.json();
    this.destination = destination;
    return { token, expiry_at: expiresAt };
  }

  refresh() {
    return this.authenticate();
  }
}
class PumpkinTokens {
  async authenticate() {
    const response = await fetch("/eyes-token", { method: "POST" });
    if (!response.ok) throw new Error(`token request failed: ${response.status}`);
    const { token, expiresAt, destination } = await response.json();
    this.destination = destination;
    return { token, expiry_at: expiresAt };
  }

  refresh() {
    return this.authenticate();
  }
}
class PumpkinTokens {
  async authenticate() {
    const response = await fetch("/eyes-token", { method: "POST" });
    if (!response.ok) throw new Error(`token request failed: ${response.status}`);
    const { token, expiresAt, destination } = await response.json();
    this.destination = destination;
    return { token, expiry_at: expiresAt };
  }

  refresh() {
    return this.authenticate();
  }
}
class PumpkinTokens {
  async authenticate() {
    const response = await fetch("/eyes-token", { method: "POST" });
    if (!response.ok) throw new Error(`token request failed: ${response.status}`);
    const { token, expiresAt, destination } = await response.json();
    this.destination = destination;
    return { token, expiry_at: expiresAt };
  }

  refresh() {
    return this.authenticate();
  }
}

The dial sends the page's processed microphone stream and the camera, which I open at 640 by 480 and 10 frames a second. That's enough for the vision model, and it keeps the work light for the hidden computer. The pumpkin has no screen, so it receives audio only.

const options = {
  audio: true,
  video: true,
  receiveAudio: true,
  receiveVideo: false,
  fallbackToReceiveOnly: false,
  inputAudioStream: mic.stream,
  inputVideoStream: camera,
};
const call = await client.dial(tokens.destination, options);
const options = {
  audio: true,
  video: true,
  receiveAudio: true,
  receiveVideo: false,
  fallbackToReceiveOnly: false,
  inputAudioStream: mic.stream,
  inputVideoStream: camera,
};
const call = await client.dial(tokens.destination, options);
const options = {
  audio: true,
  video: true,
  receiveAudio: true,
  receiveVideo: false,
  fallbackToReceiveOnly: false,
  inputAudioStream: mic.stream,
  inputVideoStream: camera,
};
const call = await client.dial(tokens.destination, options);
const options = {
  audio: true,
  video: true,
  receiveAudio: true,
  receiveVideo: false,
  fallbackToReceiveOnly: false,
  inputAudioStream: mic.stream,
  inputVideoStream: camera,
};
const call = await client.dial(tokens.destination, options);

Keep the call up all night

A call that runs for hours will drop now and then, so the page dials in a loop if the call drops. A call that ran at least 30 seconds redials at once. One that failed right away backs off, from 5 seconds up to 60, so a broken agent can't spin the loop. If a dial hasn't started within 30 seconds, the page rebuilds the SDK client in place. A reload would need another click on the page before Chrome plays audio.

Traps to avoid

Don't put project credentials in the page. Anyone can open the page's source. Sure, you're the only one running the page and no one else knows the URL, but have good habits from the start. Mint a short-lived guest token on your server, limited to the one address the page may call, and put a password on the token route.

Don't hand the SDK only your processed audio. Once you pass inputAudioStream, the SDK sends only the streams you give it and ignores device settings. Pass the camera as inputVideoStream too, or the call goes out with no video and the agent is blind.

Next in this series

The call connects, but the agent on the other end has no personality yet. In part 2 I give it one, with a base prompt and three modes built as contexts and steps. All of it can be checked without placing a call.

Bottom Linear Gradient  Lines image

The Communications Stack for What's Next

APIs built for speed. Infrastructure built for scale. AI built in from day one.

The Communications Stack for What's Next

APIs built for speed. Infrastructure built for scale. AI built in from day one.

The Communications Stack for What's Next

APIs built for speed. Infrastructure built for scale. AI built in from day one.

The Communications Stack for What's Next

APIs built for speed. Infrastructure built for scale. AI built in from day one.