Switch language mid-call

AI Agentsmultilingual voice AI

Change the conversation language and matching voice during a live voice AI call.

multilingual

The claim

A caller opens in French, switches to English halfway through, and the agent follows without being asked and without the call going anywhere. It is one AI session throughout: no second agent, no transfer, no routing decision.

Why it holds

Each language has its own voice, so the caller hears a speaker of their language rather than one accent reading three.

How it works

Two things, and the second is the one people miss.

self.add_language("English", "en-US", "rime.spore")
self.add_language("Spanish", "es-ES", "rime.marisol")
self.add_language("French",  "fr-FR", "rime.celeste")

Then turn the feature on:

self.set_params({"languages_enabled": True})

Without languages_enabled, the list is configuration the platform never reads. The agent renders, the call connects, and it answers everything in the first voice. Nothing errors, which is what makes it hard to spot.

The first language registered is the one the agent opens in. Order is a decision, not an accident: put the language most callers use first, because that is what they hear before anyone has said anything.

The prompt asks the model to follow the caller and not to comment on the switch. Announcing “I see you’re speaking French now” is the tell that turns a capability into a party trick.

Limitations

Recognition and synthesis are per language, but the prompt is not. One prompt serves every language, so anything you write in English is being translated by the model on the way out.

A voice per language means a voice you have to pick per language. A missing or wrong voice name does not fail loudly; it falls back, and the caller hears the wrong accent.

What to change first

Delete the languages_enabled parameter and call in Spanish. The agent understands and answers in English, which is the failure this recipe exists to name.

python/app.py

"""Switch language mid-call.

An agent that answers in whatever language the caller is speaking, and changes
partway through if they do, without a transfer and without a second agent.

Two things make it work. Each language is registered with its own TTS voice, so
the agent sounds like a speaker of that language rather than an English voice
reading Spanish. And `languages_enabled` turns the switching on: without it the
list is configuration nobody reads.

The first language registered is the one the agent opens in.

Written against signalwire-sdk 3.0.1.
"""
import os

from dotenv import load_dotenv
from signalwire import AgentBase

# the SDK does not read .env for you
load_dotenv()

# (name, code, voice). A voice per language: the point is that the caller
# hears a native voice, not one accent reading three languages.
LANGUAGES = [
    ("English", "en-US", "rime.spore"),
    ("Spanish", "es-ES", "rime.marisol"),
    ("French", "fr-FR", "rime.celeste"),
]


class FrontDeskAgent(AgentBase):
    def __init__(self):
        super().__init__(name="frontdesk", route="/frontdesk")
        self.prompt_add_section(
            "Role",
            "You are the front desk for a hotel in Montreal. Answer questions "
            "about check-in times, parking and breakfast.",
        )
        self.prompt_add_section(
            "Language",
            "Reply in whatever language the caller is speaking. If they change "
            "language, change with them and do not comment on it.",
        )
        for name, code, voice in LANGUAGES:
            self.add_language(name, code, voice)
        # without this the languages list is ignored
        self.set_params({"languages_enabled": True})


agent = FrontDeskAgent()

if __name__ == "__main__":
    agent.serve(host="0.0.0.0", port=int(os.getenv("PORT", "3000")))

# One agent, three languages, no transfer.
#
# languages_enabled is what makes the languages list mean anything. Without it
# the platform ignores the list and answers everything in the first voice, and
# nothing errors to tell you.
version: "1.0.0"
sections:
  main:
    - answer: {}
    - ai:
        prompt:
          text: |
            ## Role
            You are the front desk for a hotel in Montreal. Answer questions
            about check-in times, parking and breakfast.

            ## Language
            Reply in whatever language the caller is speaking. If they change
            language, change with them and do not comment on it.
        params:
          languages_enabled: true
        # first in the list is what the agent opens in
        languages:
          - name: English
            code: en-US
            voice: rime.spore
          - name: Spanish
            code: es-ES
            voice: rime.marisol
          - name: French
            code: fr-FR
            voice: rime.celeste
    - hangup: {}

Run it

cd python
pip install -r requirements.txt
cp ../.env.example .env          # set SWML_BASIC_AUTH_PASSWORD
python app.py

Point a phone number’s SWML webhook at https://<user>:<password>@<your-host>/frontdesk/, using the credentials you set.

Verify it

No network, no account:

python verify.py          # from the recipe folder, not python/

It renders the SWML and asserts:

languages_enabled is on, so the list is read rather than ignored
all three languages reach the platform, in registration order
each carries its own code and voice, and no two are identical
the document contains no transfer and no connect, so the switch happens inside one session
the prompt mentions language, so the model knows to follow the caller
the hand-written SWML surface satisfies the same checks, in the same order

Report an issue