Translate a call in real time

Voicereal-time call translation

Translate both sides of one bridged call so each participant speaks and hears their chosen language.

translation

The claim

Two people who share no language hold one conversation. The caller speaks Spanish and hears the agent’s English as Spanish. The agent speaks English and hears the caller’s Spanish as English. No third party joins, no interpreter is scheduled, and neither side installs anything.

Why it holds

This is a call-level verb, not an AI agent. There is no prompt to write and no model to govern.

How it works

live_translate opens a translation session, then connect bridges the second party. The order matters. connect blocks until the bridge ends, so a live_translate placed after it never runs while anyone is talking.

- live_translate:
    action:
      start:
        from_lang: en-US
        to_lang: es-ES
        from_voice: Polly.Joanna
        to_voice: Polly.Lucia
        direction: [remote-caller, local-caller]
- connect: { to: "+15550100001" }

The schema requires from_lang, to_lang and direction, and two of those carry the behaviour.

direction decides which legs are translated. Its values are remote-caller and local-caller. Listing only one gives one-way translation, which is a transcript read aloud rather than a conversation.

from_voice and to_voice decide what each side hears: the caller hears to_voice, the agent hears from_voice. Omit one and that direction falls back to the default voice.

webhook is where translation events are POSTed while the call runs, with live_events adding partial results and ai_summary a summary at the end. This recipe sets them and stops there, because the event payload is not something it can prove offline.

Limitations

The action union is start, stop, summarize and inject. There is no update, so changing the language pair mid-call means stopping the session and starting a new one.

Translation is sequential: a side hears the other after the utterance completes, not while it is being spoken. Long turns feel like long pauses, so tune vad_silence_ms before blaming the network.

What to change first

Set direction to [remote-caller] alone and call it. Only the caller is translated, which is the difference between an interpreter and a subtitle.

python/app.py

"""Translate a call in real time.

`live_translate` opens a translation session on the call. Each side speaks its
own language and hears the other rendered by a TTS voice you pick per
direction. The verb starts before `connect`, so the leg the bridge adds is
inside the session.

Written against signalwire-sdk 3.0.1 (SWMLService) and Flask.
"""
import os

from dotenv import load_dotenv
from flask import Flask, jsonify
from signalwire import SWMLService

# the SDK does not read .env for you
load_dotenv()

PUBLIC_URL = os.getenv("PUBLIC_URL", "https://your-host.example.com")
AGENT = os.getenv("AGENT_NUMBER", "+15550100001")
FROM_LANG = os.getenv("FROM_LANG", "en-US")
TO_LANG = os.getenv("TO_LANG", "es-ES")
FROM_VOICE = os.getenv("FROM_VOICE", "Polly.Joanna")
TO_VOICE = os.getenv("TO_VOICE", "Polly.Lucia")

app = Flask(__name__)


def build(service=None):
    service = service or SWMLService(name="translate", route="/translate")
    service.reset_document()
    service.add_verb("answer", {})
    service.add_verb("live_translate", {"action": {"start": {
        "from_lang": FROM_LANG,
        "to_lang": TO_LANG,
        # the voice each side hears the other in
        "from_voice": FROM_VOICE,
        "to_voice": TO_VOICE,
        # both legs: without local-caller the agent is never translated
        "direction": ["remote-caller", "local-caller"],
        # translation events are POSTed here while the call runs
        "webhook": f"{PUBLIC_URL}/translation",
        "live_events": True,
        "ai_summary": True,
        "speech_engine": "deepgram",
    }}})
    service.add_verb("connect", {"to": AGENT})
    service.add_verb("hangup", {})
    return service


@app.route("/translate", methods=["GET", "POST"])
def swml():
    return jsonify(build().get_document())


if __name__ == "__main__":
    app.run(port=int(os.getenv("PORT", "8080")))

# Translate a call in real time. Each side speaks their own language and hears
# the other translated, in a voice you choose per direction.
#
# live_translate starts before connect, so the leg the bridge adds is inside the
# translation session. Started after connect it would only cover the caller.
version: "1.0.0"
sections:
  main:
    - answer: {}
    - live_translate:
        action:
          start:
            from_lang: en-US
            to_lang: es-ES
            # the voice each side hears the other in
            from_voice: Polly.Joanna
            to_voice: Polly.Lucia
            # both legs: without local-caller the agent is never translated
            direction:
              - remote-caller
              - local-caller
            webhook: "https://your-host.example.com/translation"
            live_events: true
            ai_summary: true
            speech_engine: deepgram
    - connect: { to: "+15550100001" }
    - hangup: {}

Run it

cd python
pip install -r requirements.txt
cp ../.env.example .env          # set PUBLIC_URL and AGENT_NUMBER
python app.py

Point a phone number’s SWML webhook at https://<your-host>/translate.

Verify it

No network, no account:

python verify.py          # from the recipe folder, not python/

Both surfaces validate against the SWML schema. The verifier asserts:

live_translate precedes connect, so the bridged leg is inside the session
the schema’s required from_lang, to_lang and direction are all present
both remote-caller and local-caller are translated
each direction names its own voice, and the two differ

Report an issue