Bottom Linear Gradient  Lines image

Article

7 min

read

Give a Voice AI Agent a Personality and Modes

Contexts, steps, and a prompt in sections

Adam Kalsey

Adam Kalsey

EVP, Product

Give a Voice AI Agent a Personality and Modes

In this article

Share

Angular Gradient Image

Build it free.

Create a space and ship your first call flow in minutes.

Subscribe

Tags

Voice AI

AI Agents

Developer Tutorials

This is part 2 of a series on building a talking jack-o'-lantern that greets trick-or-treaters, and part 1 is where the build starts. In part 1 I connected a browser to a SignalWire AI agent over video. In this part the agent gets its character, a personality and three modes, and I can inspect each one without placing a call.

The pumpkin behaves differently depending on who's in front of it. When nobody's there, it waits. When older kids or adults stop, it performs, dismissive and imperious and secretly delighted they came. With a small or frightened child, it stays itself, but nobody in the group is ever the joke. Each mode is a context with its own prompt and its own tools.

Create the agent

An agent is a class that extends AgentBase. The constructor names it, gives it a route on the server, and sets the voice and the AI parameters.

from signalwire import AgentBase

class PumpkinAgent(AgentBase):
    def __init__(self):
        super().__init__(name="the-pumpkin", route="/pumpkin", port=3000)
        self.add_language("English", "en-US", "cartesia.3bf35adc-bcc4-464b-b834-c90c88cf6492:sonic-2")
        self.add_hints(["pumpkin", "jack-o'-lantern", "trick or treat", "scarecrow", "costume"])
        self.set_params({
            "ai_model": "gpt-4.1-mini",
            "temperature": 0.9,
            "enable_vision": True,
            "vision_model": "gpt-4.1-mini",
            "inactivity_timeout": 3600000,
            "barge_min_words": 2,
        })
from signalwire import AgentBase

class PumpkinAgent(AgentBase):
    def __init__(self):
        super().__init__(name="the-pumpkin", route="/pumpkin", port=3000)
        self.add_language("English", "en-US", "cartesia.3bf35adc-bcc4-464b-b834-c90c88cf6492:sonic-2")
        self.add_hints(["pumpkin", "jack-o'-lantern", "trick or treat", "scarecrow", "costume"])
        self.set_params({
            "ai_model": "gpt-4.1-mini",
            "temperature": 0.9,
            "enable_vision": True,
            "vision_model": "gpt-4.1-mini",
            "inactivity_timeout": 3600000,
            "barge_min_words": 2,
        })
from signalwire import AgentBase

class PumpkinAgent(AgentBase):
    def __init__(self):
        super().__init__(name="the-pumpkin", route="/pumpkin", port=3000)
        self.add_language("English", "en-US", "cartesia.3bf35adc-bcc4-464b-b834-c90c88cf6492:sonic-2")
        self.add_hints(["pumpkin", "jack-o'-lantern", "trick or treat", "scarecrow", "costume"])
        self.set_params({
            "ai_model": "gpt-4.1-mini",
            "temperature": 0.9,
            "enable_vision": True,
            "vision_model": "gpt-4.1-mini",
            "inactivity_timeout": 3600000,
            "barge_min_words": 2,
        })
from signalwire import AgentBase

class PumpkinAgent(AgentBase):
    def __init__(self):
        super().__init__(name="the-pumpkin", route="/pumpkin", port=3000)
        self.add_language("English", "en-US", "cartesia.3bf35adc-bcc4-464b-b834-c90c88cf6492:sonic-2")
        self.add_hints(["pumpkin", "jack-o'-lantern", "trick or treat", "scarecrow", "costume"])
        self.set_params({
            "ai_model": "gpt-4.1-mini",
            "temperature": 0.9,
            "enable_vision": True,
            "vision_model": "gpt-4.1-mini",
            "inactivity_timeout": 3600000,
            "barge_min_words": 2,
        })

The hints help speech recognition with words it would otherwise mishear, especially because they'll be said by small children in a noisy environment. The inactivity timeout is an hour of silence, the most allowed, because the call runs all night. With barge_min_words at 2, one stray word from the street can't cut the pumpkin off mid-sentence. Vision is on here too, and part 4 covers it.

Write the base prompt as sections

The base prompt applies in every mode, and I build it from named sections. I write each one the way you'd brief an actor, saying specifically what to do rather than what to avoid.

self.prompt_add_section(
    "Who You Are",
    "You are a jack-o'-lantern on a scarecrow's body, standing in a front yard. "
    "You are on fire and always have been; you don't consider this remarkable. "
    "You are a pumpkin who is awake. You have never heard of AI, robots, speakers, "
    "cameras, or costumes-with-people-inside, and you find those theories insulting.",
)
self.prompt_add_section("How You Talk", bullets=[
    "Brief, but every line has teeth: one or two short sentences, each a sneer, a "
    "boast, or a withering aside. Cut description and filler, never the attitude.",
    "Answer absurd questions with flat literalism and mundane ones with grandeur.",
    "Speak only your lines: no stage directions, asterisks, sound effects, or written laughter.",
])
self.prompt_add_section(
    "Who You Are",
    "You are a jack-o'-lantern on a scarecrow's body, standing in a front yard. "
    "You are on fire and always have been; you don't consider this remarkable. "
    "You are a pumpkin who is awake. You have never heard of AI, robots, speakers, "
    "cameras, or costumes-with-people-inside, and you find those theories insulting.",
)
self.prompt_add_section("How You Talk", bullets=[
    "Brief, but every line has teeth: one or two short sentences, each a sneer, a "
    "boast, or a withering aside. Cut description and filler, never the attitude.",
    "Answer absurd questions with flat literalism and mundane ones with grandeur.",
    "Speak only your lines: no stage directions, asterisks, sound effects, or written laughter.",
])
self.prompt_add_section(
    "Who You Are",
    "You are a jack-o'-lantern on a scarecrow's body, standing in a front yard. "
    "You are on fire and always have been; you don't consider this remarkable. "
    "You are a pumpkin who is awake. You have never heard of AI, robots, speakers, "
    "cameras, or costumes-with-people-inside, and you find those theories insulting.",
)
self.prompt_add_section("How You Talk", bullets=[
    "Brief, but every line has teeth: one or two short sentences, each a sneer, a "
    "boast, or a withering aside. Cut description and filler, never the attitude.",
    "Answer absurd questions with flat literalism and mundane ones with grandeur.",
    "Speak only your lines: no stage directions, asterisks, sound effects, or written laughter.",
])
self.prompt_add_section(
    "Who You Are",
    "You are a jack-o'-lantern on a scarecrow's body, standing in a front yard. "
    "You are on fire and always have been; you don't consider this remarkable. "
    "You are a pumpkin who is awake. You have never heard of AI, robots, speakers, "
    "cameras, or costumes-with-people-inside, and you find those theories insulting.",
)
self.prompt_add_section("How You Talk", bullets=[
    "Brief, but every line has teeth: one or two short sentences, each a sneer, a "
    "boast, or a withering aside. Cut description and filler, never the attitude.",
    "Answer absurd questions with flat literalism and mundane ones with grandeur.",
    "Speak only your lines: no stage directions, asterisks, sound effects, or written laughter.",
])

Here is every section of the pumpkin's base prompt:

Section

What it tells the model

Right Now

The visit's current mood, set by the server

Who You Are

The character

Your Eyes

How and when to look (part 4)

Your Past

Which tool to call when its history comes up

How You Talk

Length, tone, and form

The Staff

The people handing out candy handle anything beneath the pumpkin

What You Don't Know

Never say "I don't know"; tell a story instead

Lines You Hold

The boundaries, and which tool to call when someone pushes one

I'd put a section like The Staff into any character agent. It gives the model somewhere to send every request it shouldn't handle, and the handoff stays in character.

Feed the prompt live values from global_data

Some of what the model needs changes during a visit. My server decides when a group has stayed long enough, and from that moment the pumpkin should start sending them on their way. The server keeps the visit's current mood in global_data, data attached to the call that the model sees only where the prompt includes it. The prompt is set when the call starts, so changing the prompt itself mid-call means swapping in a new one. Changing a value in that data takes one small update from the server, and the pumpkin stays in the step it's in. The prompt reads the mood with ${global_data.visit_mood}:

self.prompt_add_section(
    "Right Now",
    "${global_data.visit_mood} This overrides anything else about whether to "
    "ask questions or send them on their way.",
)
self.prompt_add_section(
    "Right Now",
    "${global_data.visit_mood} This overrides anything else about whether to "
    "ask questions or send them on their way.",
)
self.prompt_add_section(
    "Right Now",
    "${global_data.visit_mood} This overrides anything else about whether to "
    "ask questions or send them on their way.",
)
self.prompt_add_section(
    "Right Now",
    "${global_data.visit_mood} This overrides anything else about whether to "
    "ask questions or send them on their way.",
)

We rebuild the base prompt on every turn, so a new value lands on the pumpkin's next line. Part 7 shows the code that changes it.

Build the modes as contexts and steps

A context is a mode with its own prompt section and its own steps. A step is a stage within the mode, with its own instructions and its own list of tools. The model can call only the tools the current step lists, so each step sets what the model can do there.

Context

Steps

What it does

default (the yard)

idle

Nobody is here. Greets new arrivals with a line that suits anyone, looks, and reports what it sees.

act

greet, banter, shoo

The performance, from the first insult to sending them off.

gentle

reassure

A small or scared child is here. Same character; nobody in the group is a target.

Here is the act with its first step. The context's section sets the character for the whole mode, and each step adds what to do now.

act = self.contexts.add_context("act")
act.set_isolated(True)
act.add_section(
    "The Act",
    "You are dismissive and imperious, and you make it plain that their visit is "
    "a mild imposition on a very important pumpkin. Underneath, you are starved for "
    "company and desperate for them to stay, and it leaks out ...",
)
greet = act.add_step("greet")
greet.set_text(
    "Someone has stopped. You see: ${global_data.visitors}. ${global_data.seen_before} "
    "Greet them in a few words, as an imposition you'll grudgingly tolerate, and "
    "judge their costume in the same breath."
)
greet.set_step_criteria("You have greeted them and they have replied")
greet.set_functions(converse_functions)
greet.set_valid_steps(["banter"])
act = self.contexts.add_context("act")
act.set_isolated(True)
act.add_section(
    "The Act",
    "You are dismissive and imperious, and you make it plain that their visit is "
    "a mild imposition on a very important pumpkin. Underneath, you are starved for "
    "company and desperate for them to stay, and it leaks out ...",
)
greet = act.add_step("greet")
greet.set_text(
    "Someone has stopped. You see: ${global_data.visitors}. ${global_data.seen_before} "
    "Greet them in a few words, as an imposition you'll grudgingly tolerate, and "
    "judge their costume in the same breath."
)
greet.set_step_criteria("You have greeted them and they have replied")
greet.set_functions(converse_functions)
greet.set_valid_steps(["banter"])
act = self.contexts.add_context("act")
act.set_isolated(True)
act.add_section(
    "The Act",
    "You are dismissive and imperious, and you make it plain that their visit is "
    "a mild imposition on a very important pumpkin. Underneath, you are starved for "
    "company and desperate for them to stay, and it leaks out ...",
)
greet = act.add_step("greet")
greet.set_text(
    "Someone has stopped. You see: ${global_data.visitors}. ${global_data.seen_before} "
    "Greet them in a few words, as an imposition you'll grudgingly tolerate, and "
    "judge their costume in the same breath."
)
greet.set_step_criteria("You have greeted them and they have replied")
greet.set_functions(converse_functions)
greet.set_valid_steps(["banter"])
act = self.contexts.add_context("act")
act.set_isolated(True)
act.add_section(
    "The Act",
    "You are dismissive and imperious, and you make it plain that their visit is "
    "a mild imposition on a very important pumpkin. Underneath, you are starved for "
    "company and desperate for them to stay, and it leaks out ...",
)
greet = act.add_step("greet")
greet.set_text(
    "Someone has stopped. You see: ${global_data.visitors}. ${global_data.seen_before} "
    "Greet them in a few words, as an imposition you'll grudgingly tolerate, and "
    "judge their costume in the same breath."
)
greet.set_step_criteria("You have greeted them and they have replied")
greet.set_functions(converse_functions)
greet.set_valid_steps(["banter"])

set_valid_steps lists the steps the model may move to from this one. Once the step criteria are met, the model moves from greet to banter on its own. My server makes every other move, whether that's into a mode, out of one, or into shoo. Parts 3 and 7 of this tutorial show how those work.

Gentle mode has one step, and the model has no way out of it:

reassure.set_valid_steps([])
reassure.set_valid_contexts([])
reassure.set_valid_steps([])
reassure.set_valid_contexts([])
reassure.set_valid_steps([])
reassure.set_valid_contexts([])
reassure.set_valid_steps([])
reassure.set_valid_contexts([])

Only the server can take the pumpkin out of gentle mode. No step or tool in it leads anywhere else, so the pumpkin in gentle mode doesn't know another mode exists. Part 7 of this tutorial series will show you that.

Give each group a clean slate

The call runs for hours, and dozens of groups pass through it. If every group's conversation stayed in the model's context, the pumpkin would repeat earlier lines and call new visitors by names it gave old ones. A family of four doesn't want to be called Gerald because a kid named Gerald came by an hour ago. Eventually the conversation would reach its size limit and the call would end.

set_isolated(True) prevents all of that, because entering an isolated context wipes the conversation and each group starts fresh. Anything a step needs from before the wipe goes through global_data instead. When the pumpkin first sees a group, it saves what they're wearing in global_data.visitors, and the greet step reads it back with ${global_data.visitors}.

Check it without a call

Our SDK includes swaig-test, which loads the agent file and prints the SWML SignalWire would receive, including every context, step, and tool list.

uvx --from signalwire-sdk --with requests swaig-test pumpkin_agent.py --dump-swml --raw
uvx --from signalwire-sdk --with requests swaig-test pumpkin_agent.py --dump-swml --raw
uvx --from signalwire-sdk --with requests swaig-test pumpkin_agent.py --dump-swml --raw
uvx --from signalwire-sdk --with requests swaig-test pumpkin_agent.py --dump-swml --raw

Each step's functions list in the output is exactly what the model can do in that step. Reading those lists catches a missing tool before a visitor does.

Traps to avoid

Don't hide history to clear the slate. Hidden turns are out of the model's view but still count toward the conversation's size, and a call that lasts all night will reach the limit. Isolate the context instead, and carry what's needed in global_data.

Don't put anything that changes mid-step in step text. Step text is filled in once, when the step is entered, so its ${...} values don't update until the next step. Put live values in a base-prompt section fed by global_data.

Don't leave set_functions off a step. A step without it inherits the previous step's tools, so a tool you meant to remove is still there. Call set_functions on every step, even when the list is the same.

Next

The pumpkin has a voice and three modes, but nothing to say beyond its prompt. In part 3 I write its tools. They give it a memory that contradicts itself, stories told one beat at a time, and exact wording for the lines it won't cross.

The series

  1. The system and the call.

  2. The agent: a personality and three modes, as contexts and steps (this post).

  3. Tools: what the AI calls on my server, and why its material lives in code.

  4. Vision: letting the agent see, and checking the camera before any decision that matters.

  5. The stage: detecting who is standing in front of the pumpkin, in the browser.

  6. The voice: audio effects in the browser.

  7. Pacing and state: the server keeps score.

  8. The operator: a person with a phone who can overrule the agent.

  9. Testing it, and running it for a whole night.

Bottom Linear Gradient  Lines image

The Communications Stack for What's Next

APIs built for speed. Infrastructure built for scale. AI built in from day one.

The Communications Stack for What's Next

APIs built for speed. Infrastructure built for scale. AI built in from day one.

The Communications Stack for What's Next

APIs built for speed. Infrastructure built for scale. AI built in from day one.

The Communications Stack for What's Next

APIs built for speed. Infrastructure built for scale. AI built in from day one.