Bottom Linear Gradient  Lines image

Article

6 min

read

Write Server-Side Tools for a SignalWire AI Agent

Tools the AI calls, with the wording kept in code

Adam Kalsey

Adam Kalsey

EVP, Product

Write Server-Side Tools for an  AI Agent

In this article

Share

Angular Gradient Image

Build it free.

Create a space and ship your first call flow in minutes.

Subscribe

Tags

Voice AI

AI Agents

Developer Tutorials

This is part 3 of a series about a jack-o'-lantern I built to talk with trick-or-treaters, and the build starts in part 1. In part 2 the pumpkin got a personality and three modes. In this part it gets tools, which are functions on my server that the model calls in the middle of a conversation. They give the model fresh material for the four-hundredth visitor of the night. They also hold the wording that has to be right, so that wording comes from code rather than from the model's best guess.

Write a tool

A tool is a method on the agent marked with the @AgentBase.tool decorator. The decorator gives it a name, a description, and a JSON schema for its parameters. When the model calls the tool, SignalWire posts the request to my server, our Python SDK runs the method, and the method returns a FunctionResult.

The pumpkin calls consult_memory when someone asks about its past:

@AgentBase.tool(
    name="consult_memory",
    description=(
        "Recall something about yourself. Call when your name, your age, the fire, being awake, "
        "the yard, your body, your head, the house, past Halloweens, or costumes in general "
        "comes up. For the costumes you've seen tonight, call recall_costumes."
    ),
    parameters={
        "type": "object",
        "properties": {
            "topic": {
                "type": "string",
                "enum": list(BACKSTORY),
                "description": "What you're trying to remember about yourself.",
            },
        },
        "required": ["topic"],
    },
)
def consult_memory(self, args, raw_data):
    topic = args.get("topic")
    if topic not in BACKSTORY:
        return FunctionResult("You don't remember, and you resent being asked.")

    fragment, update = pick_unused(raw_data, BACKSTORY[topic])
    return FunctionResult(
        f"{fragment} Allude to this in your own words. Don't explain it or tell the "
        "whole story."
    ).update_global_data(update)
@AgentBase.tool(
    name="consult_memory",
    description=(
        "Recall something about yourself. Call when your name, your age, the fire, being awake, "
        "the yard, your body, your head, the house, past Halloweens, or costumes in general "
        "comes up. For the costumes you've seen tonight, call recall_costumes."
    ),
    parameters={
        "type": "object",
        "properties": {
            "topic": {
                "type": "string",
                "enum": list(BACKSTORY),
                "description": "What you're trying to remember about yourself.",
            },
        },
        "required": ["topic"],
    },
)
def consult_memory(self, args, raw_data):
    topic = args.get("topic")
    if topic not in BACKSTORY:
        return FunctionResult("You don't remember, and you resent being asked.")

    fragment, update = pick_unused(raw_data, BACKSTORY[topic])
    return FunctionResult(
        f"{fragment} Allude to this in your own words. Don't explain it or tell the "
        "whole story."
    ).update_global_data(update)
@AgentBase.tool(
    name="consult_memory",
    description=(
        "Recall something about yourself. Call when your name, your age, the fire, being awake, "
        "the yard, your body, your head, the house, past Halloweens, or costumes in general "
        "comes up. For the costumes you've seen tonight, call recall_costumes."
    ),
    parameters={
        "type": "object",
        "properties": {
            "topic": {
                "type": "string",
                "enum": list(BACKSTORY),
                "description": "What you're trying to remember about yourself.",
            },
        },
        "required": ["topic"],
    },
)
def consult_memory(self, args, raw_data):
    topic = args.get("topic")
    if topic not in BACKSTORY:
        return FunctionResult("You don't remember, and you resent being asked.")

    fragment, update = pick_unused(raw_data, BACKSTORY[topic])
    return FunctionResult(
        f"{fragment} Allude to this in your own words. Don't explain it or tell the "
        "whole story."
    ).update_global_data(update)
@AgentBase.tool(
    name="consult_memory",
    description=(
        "Recall something about yourself. Call when your name, your age, the fire, being awake, "
        "the yard, your body, your head, the house, past Halloweens, or costumes in general "
        "comes up. For the costumes you've seen tonight, call recall_costumes."
    ),
    parameters={
        "type": "object",
        "properties": {
            "topic": {
                "type": "string",
                "enum": list(BACKSTORY),
                "description": "What you're trying to remember about yourself.",
            },
        },
        "required": ["topic"],
    },
)
def consult_memory(self, args, raw_data):
    topic = args.get("topic")
    if topic not in BACKSTORY:
        return FunctionResult("You don't remember, and you resent being asked.")

    fragment, update = pick_unused(raw_data, BACKSTORY[topic])
    return FunctionResult(
        f"{fragment} Allude to this in your own words. Don't explain it or tell the "
        "whole story."
    ).update_global_data(update)

The model reads the description to decide when to call the tool, which makes the description prompt text. It says in plain words when to call and sends the nearest wrong case to another tool. The enum limits topic to the topics the server has material for. The result is prompt text as well, since it carries the fragment along with instructions for how to use it.

Draw lines from a pool, without repeats

The pumpkin's backstory is a dictionary of topics, and each topic has a handful of fragments that contradict each other on purpose:

BACKSTORY = {
    "age": [
        "You were carved on a Tuesday.",
        "You are older than the street.",
        "You predate the house. The house is aware of this.",
        "You are exactly as old as you need to be to win this argument.",
    ],
    # name, fire, yard, ...
}
BACKSTORY = {
    "age": [
        "You were carved on a Tuesday.",
        "You are older than the street.",
        "You predate the house. The house is aware of this.",
        "You are exactly as old as you need to be to win this argument.",
    ],
    # name, fire, yard, ...
}
BACKSTORY = {
    "age": [
        "You were carved on a Tuesday.",
        "You are older than the street.",
        "You predate the house. The house is aware of this.",
        "You are exactly as old as you need to be to win this argument.",
    ],
    # name, fire, yard, ...
}
BACKSTORY = {
    "age": [
        "You were carved on a Tuesday.",
        "You are older than the street.",
        "You predate the house. The house is aware of this.",
        "You are exactly as old as you need to be to win this argument.",
    ],
    # name, fire, yard, ...
}

Asked its age twice, the pumpkin gives two different answers, which suits the character, but giving one group the same answer twice wouldn't. So pick_unused keeps a list in global_data of the fragments this visit has used, and it draws from the rest:

def pick_unused(raw_data, pool):
    """A random item from pool not yet used this visit, and the global_data
    update that records it. Repeats only once the pool runs out."""
    used = group_state(raw_data)["used_fragments"]
    choice = random.choice([item for item in pool if item not in used] or pool)
    return choice, {"used_fragments": used + [choice]}
def pick_unused(raw_data, pool):
    """A random item from pool not yet used this visit, and the global_data
    update that records it. Repeats only once the pool runs out."""
    used = group_state(raw_data)["used_fragments"]
    choice = random.choice([item for item in pool if item not in used] or pool)
    return choice, {"used_fragments": used + [choice]}
def pick_unused(raw_data, pool):
    """A random item from pool not yet used this visit, and the global_data
    update that records it. Repeats only once the pool runs out."""
    used = group_state(raw_data)["used_fragments"]
    choice = random.choice([item for item in pool if item not in used] or pool)
    return choice, {"used_fragments": used + [choice]}
def pick_unused(raw_data, pool):
    """A random item from pool not yet used this visit, and the global_data
    update that records it. Repeats only once the pool runs out."""
    used = group_state(raw_data)["used_fragments"]
    choice = random.choice([item for item in pool if item not in used] or pool)
    return choice, {"used_fragments": used + [choice]}

Every tool request we send carries the call's global_data, so the handler reads the visit's state from it and returns the update with its result. When the next group arrives, the server resets used_fragments, and the whole pool is available again.

Tell stories one beat at a time

The pumpkin never says "I don't know." When a visitor asks something it can't answer, it tells a story that starts on the question and wanders off. tell_a_story builds each story from three absurd turns drawn at random:

STORY_TURNS = [
    "a goose gets involved and refuses to leave",
    "the moon's landlord shows up",
    "there's a lawsuit, and you represented yourself",
    "you briefly become a spoon, and it's relevant",
    # ...
]
STORY_TURNS = [
    "a goose gets involved and refuses to leave",
    "the moon's landlord shows up",
    "there's a lawsuit, and you represented yourself",
    "you briefly become a spoon, and it's relevant",
    # ...
]
STORY_TURNS = [
    "a goose gets involved and refuses to leave",
    "the moon's landlord shows up",
    "there's a lawsuit, and you represented yourself",
    "you briefly become a spoon, and it's relevant",
    # ...
]
STORY_TURNS = [
    "a goose gets involved and refuses to leave",
    "the moon's landlord shows up",
    "there's a lawsuit, and you represented yourself",
    "you briefly become a spoon, and it's relevant",
    # ...
]

Each call returns only the next beat, and the story's place lives in global_data.story. A visitor who says "and then what?" gets the next turn, and one who changes the subject leaves the story behind.

if beat == 0:
    say = (f"Don't answer {question!r}. Tell a story instead, starting plausibly on their "
           "question, as if the answer is coming. One or two sentences, then stop.")
elif beat <= 3:
    say = (f"The next beat of your story: {story['turns'][beat - 1]}. One or two "
           "sentences, stranger than the last.")
else:
    say = "The story is over. Insist it was the answer."
return FunctionResult(
    f"{say} {tone} Never reach the answer or an ending. If they want more, call "
    "this again for the next beat. If they change the subject, drop it without comment."
).update_global_data({"story": {**story, "beat": beat + 1}})
if beat == 0:
    say = (f"Don't answer {question!r}. Tell a story instead, starting plausibly on their "
           "question, as if the answer is coming. One or two sentences, then stop.")
elif beat <= 3:
    say = (f"The next beat of your story: {story['turns'][beat - 1]}. One or two "
           "sentences, stranger than the last.")
else:
    say = "The story is over. Insist it was the answer."
return FunctionResult(
    f"{say} {tone} Never reach the answer or an ending. If they want more, call "
    "this again for the next beat. If they change the subject, drop it without comment."
).update_global_data({"story": {**story, "beat": beat + 1}})
if beat == 0:
    say = (f"Don't answer {question!r}. Tell a story instead, starting plausibly on their "
           "question, as if the answer is coming. One or two sentences, then stop.")
elif beat <= 3:
    say = (f"The next beat of your story: {story['turns'][beat - 1]}. One or two "
           "sentences, stranger than the last.")
else:
    say = "The story is over. Insist it was the answer."
return FunctionResult(
    f"{say} {tone} Never reach the answer or an ending. If they want more, call "
    "this again for the next beat. If they change the subject, drop it without comment."
).update_global_data({"story": {**story, "beat": beat + 1}})
if beat == 0:
    say = (f"Don't answer {question!r}. Tell a story instead, starting plausibly on their "
           "question, as if the answer is coming. One or two sentences, then stop.")
elif beat <= 3:
    say = (f"The next beat of your story: {story['turns'][beat - 1]}. One or two "
           "sentences, stranger than the last.")
else:
    say = "The story is over. Insist it was the answer."
return FunctionResult(
    f"{say} {tone} Never reach the answer or an ending. If they want more, call "
    "this again for the next beat. If they change the subject, drop it without comment."
).update_global_data({"story": {**story, "beat": beat + 1}})

Once the pumpkin should be sending a group on, the tool refuses, because a story is the surest way to keep them standing there.

Return exact wording for boundaries

Some requests need the same answer every time. A visitor asks to touch the pumpkin, wants a photo, swears, shares their address, or says "you're an AI." decline_to_answer takes the kind of request and returns the guidance for it from a table:

BOUNDARY_GUIDANCE = {
    "touch_or_come_closer": (
        "Refuse coldly and briefly, in character. One sentence. You are on fire "
        "and you do not receive visitors past the rope. Do not invite them anywhere."
    ),
    "profanity": (
        'Say exactly "No." and nothing else. No lecture. Then carry on as if '
        "it never happened."
    ),
    "other_boundary": "Decline grandly, and say something splendid about them.",
}
BOUNDARY_GUIDANCE = {
    "touch_or_come_closer": (
        "Refuse coldly and briefly, in character. One sentence. You are on fire "
        "and you do not receive visitors past the rope. Do not invite them anywhere."
    ),
    "profanity": (
        'Say exactly "No." and nothing else. No lecture. Then carry on as if '
        "it never happened."
    ),
    "other_boundary": "Decline grandly, and say something splendid about them.",
}
BOUNDARY_GUIDANCE = {
    "touch_or_come_closer": (
        "Refuse coldly and briefly, in character. One sentence. You are on fire "
        "and you do not receive visitors past the rope. Do not invite them anywhere."
    ),
    "profanity": (
        'Say exactly "No." and nothing else. No lecture. Then carry on as if '
        "it never happened."
    ),
    "other_boundary": "Decline grandly, and say something splendid about them.",
}
BOUNDARY_GUIDANCE = {
    "touch_or_come_closer": (
        "Refuse coldly and briefly, in character. One sentence. You are on fire "
        "and you do not receive visitors past the rope. Do not invite them anywhere."
    ),
    "profanity": (
        'Say exactly "No." and nothing else. No lecture. Then carry on as if '
        "it never happened."
    ),
    "other_boundary": "Decline grandly, and say something splendid about them.",
}

The model recognizes the moment and picks the category, and the response lives in code, where I can review it and test it.

Give each mode its own copy of a tool

Some tools need different wording in gentle mode: a cozy story rather than a grand one, and a warm refusal rather than a cold one. I register a gentle copy of each of those tools and list only the copies in gentle mode's step.

GENTLE_TWINS = {
    "tell_a_story": "tell_a_gentle_story",
    "recall_costumes": "recall_costumes_gently",
    "decline_to_answer": "decline_gently",
    "visitors_leaving": "visitors_leaving_gently",
    "visitors_left": "visitors_left_gently",
    "guess_what_it_is": "guess_what_it_is_gently",
}
GENTLE_TWINS = {
    "tell_a_story": "tell_a_gentle_story",
    "recall_costumes": "recall_costumes_gently",
    "decline_to_answer": "decline_gently",
    "visitors_leaving": "visitors_leaving_gently",
    "visitors_left": "visitors_left_gently",
    "guess_what_it_is": "guess_what_it_is_gently",
}
GENTLE_TWINS = {
    "tell_a_story": "tell_a_gentle_story",
    "recall_costumes": "recall_costumes_gently",
    "decline_to_answer": "decline_gently",
    "visitors_leaving": "visitors_leaving_gently",
    "visitors_left": "visitors_left_gently",
    "guess_what_it_is": "guess_what_it_is_gently",
}
GENTLE_TWINS = {
    "tell_a_story": "tell_a_gentle_story",
    "recall_costumes": "recall_costumes_gently",
    "decline_to_answer": "decline_gently",
    "visitors_leaving": "visitors_leaving_gently",
    "visitors_left": "visitors_left_gently",
    "guess_what_it_is": "guess_what_it_is_gently",
}

Each copy reuses the original's description, parameters, and settings, with a handler fixed to the gentle wording:

self.define_tool(
    name=twin,
    description=description,
    parameters=original.parameters,
    handler=lambda args, raw_data, method=getattr(self, name): method(args, raw_data, gentle=True),
    **original.extra_swaig_fields,
)
self.define_tool(
    name=twin,
    description=description,
    parameters=original.parameters,
    handler=lambda args, raw_data, method=getattr(self, name): method(args, raw_data, gentle=True),
    **original.extra_swaig_fields,
)
self.define_tool(
    name=twin,
    description=description,
    parameters=original.parameters,
    handler=lambda args, raw_data, method=getattr(self, name): method(args, raw_data, gentle=True),
    **original.extra_swaig_fields,
)
self.define_tool(
    name=twin,
    description=description,
    parameters=original.parameters,
    handler=lambda args, raw_data, method=getattr(self, name): method(args, raw_data, gentle=True),
    **original.extra_swaig_fields,
)

Gentle mode's step doesn't list the originals, so the model in gentle mode has no way to reach the act's wording.

Test a tool without a call

swaig-test runs a tool the way SignalWire would, with the arguments you give it and any global_data you want the request to carry:

uvx --from signalwire-sdk --with requests swaig-test pumpkin_agent.py --raw --exec consult_memory --topic age
uvx --from signalwire-sdk --with requests swaig-test pumpkin_agent.py --raw \
  --custom-data '{"global_data":{"used_fragments":["You were carved on a Tuesday."]}}' \
  --exec consult_memory --topic

uvx --from signalwire-sdk --with requests swaig-test pumpkin_agent.py --raw --exec consult_memory --topic age
uvx --from signalwire-sdk --with requests swaig-test pumpkin_agent.py --raw \
  --custom-data '{"global_data":{"used_fragments":["You were carved on a Tuesday."]}}' \
  --exec consult_memory --topic

uvx --from signalwire-sdk --with requests swaig-test pumpkin_agent.py --raw --exec consult_memory --topic age
uvx --from signalwire-sdk --with requests swaig-test pumpkin_agent.py --raw \
  --custom-data '{"global_data":{"used_fragments":["You were carved on a Tuesday."]}}' \
  --exec consult_memory --topic

uvx --from signalwire-sdk --with requests swaig-test pumpkin_agent.py --raw --exec consult_memory --topic age
uvx --from signalwire-sdk --with requests swaig-test pumpkin_agent.py --raw \
  --custom-data '{"global_data":{"used_fragments":["You were carved on a Tuesday."]}}' \
  --exec consult_memory --topic

The second command never returns the Tuesday fragment, however many times you run it. --list-tools shows every registered tool, and the gentle copies are in that list.

Traps to avoid

Don't have one tool read a mode flag to choose its wording. A flag in global_data can drift from the mode the agent is really in, and a tool request doesn't say which context it came from. Register a separate copy for each mode and list it only in that mode's steps, so the context decides.

Don't hand the model a multi-step plan to pace itself. Given all three turns of a story at once, a model tends to tell them in one breath or lose its place. Return one step per call and keep the place on the server.

Next

The pumpkin can talk and has plenty to say, but it can't see who it's talking to. In part 4 it gets eyes, and checks the camera before any decision that matters.

The series

  1. The system and the call.

  2. The agent: a personality and three modes, as contexts and steps.

  3. Tools: what the AI calls on my server, and why its material lives in code (this post).

  4. Vision: letting the agent see, and checking the camera before any decision that matters.

  5. The stage: detecting who is standing in front of the pumpkin, in the browser.

  6. The voice: audio effects in the browser.

  7. Pacing and state: the server keeps score.

  8. The operator: a person with a phone who can overrule the agent.

  9. Testing it, and running it for a whole night.

Bottom Linear Gradient  Lines image

The Communications Stack for What's Next

APIs built for speed. Infrastructure built for scale. AI built in from day one.

The Communications Stack for What's Next

APIs built for speed. Infrastructure built for scale. AI built in from day one.

The Communications Stack for What's Next

APIs built for speed. Infrastructure built for scale. AI built in from day one.

The Communications Stack for What's Next

APIs built for speed. Infrastructure built for scale. AI built in from day one.