Start, steer and stop live translation on a call in progress

Voicereal-time call translation over REST

Switch translation on partway through a call, speak a translated line into it, ask for a summary, and switch it off, each with one REST command.

resttranslation

The claim

The document-level version of this decides before anyone speaks. This one does not. The call is up, the agent has realised what is needed, and a REST command turns translation on for the rest of the conversation.

Why it holds

The vendored REST spec, tools/openapi/rest.json, is the authority. The command requires action, which is a oneOf of four shapes.

start requires from_lang, to_lang and direction. Direction is an array of remote-caller and local-caller, so translating both sides means naming both. It also takes voices per language, a webhook for translation events, a speech_engine of deepgram or google, and voice-activity settings.
inject requires message and direction, and here direction is a single value from that same pair. The message is translated on the way in, which is how a supervisor says something to one side in their own language.
summarize requires nothing, and takes a webhook and a prompt. It is an on-demand AI summary of the translated conversation.
stop is the bare action.

How it works

def start(call_id, from_lang="en-US", to_lang="es-ES", webhook=None):
    action = {"start": {"from_lang": from_lang, "to_lang": to_lang,
                        "direction": list(DIRECTIONS),
                        "speech_engine": "deepgram", "live_events": True}}
    if webhook:
        action["start"]["webhook"] = webhook
    return client.calling.live_translate(call_id, action=action)

def say(call_id, message, direction="remote-caller"):
    if direction not in DIRECTIONS:
        raise ValueError(...)
    return client.calling.live_translate(
        call_id, action={"inject": {"message": message, "direction": direction}})

What the platform receives for say:

{"command": "calling.live_translate",
 "id": "6d3f4a0e-2b1c-4e7a-9f0d-1c2b3a4d5e6f",
 "params": {"action": {"inject": {"message": "A supervisor is joining.",
                                  "direction": "remote-caller"}}}}

One command, four bodies. The action name is the key that selects the variant, so the verifier checks which variant each body matches, not only that its params are documented.

stop sends an empty object rather than nothing at all, because the action is the key that selects the variant.

Limitations

The verifier proves the requests, not the translation. Accuracy, latency and what the far side hears are live behaviour, tuned by the voices and the speech engine you choose.

The summary is generated by a model from the translated conversation. Treat it as a summary, not as a record of what was said.

What to change first

Drop "direction" from start and run the verifier. The exact body comparison fails, which is the point. The spec marks it required alongside the two languages, and translating one side is a different feature.

Where this sits

Before this

Translate a call in real time

Start live transcription and consume the webhook Switch language mid-call End or transfer a live call over REST

python/app.py

"""Start, steer and stop live translation on a call in progress.

`calling.live_translate` carries one `action`, and the spec documents four:
`start`, `inject`, `summarize` and `stop`. Together they turn translation into
something your backend switches on partway through a call, speaks into, and
asks for a summary of, rather than something the document decided up front.

Written against signalwire-sdk 3.0.1 (RestClient.calling).

    python app.py start <call_id> en-US es-ES
    python app.py say <call_id> "A supervisor is joining."
    python app.py summary <call_id> https://your-host/summary
    python app.py stop <call_id>
"""
import sys

from dotenv import load_dotenv
from signalwire.rest import RestClient

# the SDK does not read .env for you
load_dotenv()

# reads SIGNALWIRE_PROJECT_ID / SIGNALWIRE_API_TOKEN / SIGNALWIRE_SPACE from the env
client = RestClient()

# the spec's two directions, which are also the two sides of the call
DIRECTIONS = ("remote-caller", "local-caller")


def start(call_id, from_lang="en-US", to_lang="es-ES", webhook=None):
    """Translate both sides, each hearing the other's language."""
    action = {"start": {"from_lang": from_lang, "to_lang": to_lang,
                        "direction": list(DIRECTIONS),
                        "speech_engine": "deepgram", "live_events": True}}
    if webhook:
        # translation events land here while the call runs
        action["start"]["webhook"] = webhook
    return client.calling.live_translate(call_id, action=action)


def say(call_id, message, direction="remote-caller"):
    """Speak a line into the conversation, translated on the way."""
    if direction not in DIRECTIONS:
        raise ValueError(f"direction must be one of {DIRECTIONS}, not {direction!r}")
    action = {"inject": {"message": message, "direction": direction}}
    return client.calling.live_translate(call_id, action=action)


def summary(call_id, webhook, prompt=None):
    """Ask for a summary of the translated conversation so far."""
    inner = {"webhook": webhook}
    if prompt:
        inner["prompt"] = prompt
    return client.calling.live_translate(call_id, action={"summarize": inner})


def stop(call_id):
    """End translation. The call carries on untranslated."""
    return client.calling.live_translate(call_id, action={"stop": {}})


if __name__ == "__main__":
    args = sys.argv[1:]
    cmd = args[0] if args else "help"
    call_id = args[1] if len(args) > 1 else ""
    rest = args[2:]
    actions = {"start": start, "say": say, "summary": summary, "stop": stop}
    if not call_id or cmd not in actions:
        print(__doc__)
    else:
        print(actions[cmd](call_id, *rest))

/**
 * Start, steer and stop live translation on a call in progress.
 *
 * `calling.live_translate` carries one `action`, and the spec documents four:
 * `start`, `inject`, `summarize` and `stop`. Together they turn translation
 * into something your backend switches on partway through a call, speaks into,
 * and asks for a summary of, rather than something the document decided up
 * front.
 *
 * Written against @signalwire/sdk 2.0.5 (RestClient.calling).
 *
 *     npm start start <call_id> en-US es-ES
 *     npm start stop <call_id>
 */
import "dotenv/config";
import { RestClient } from "@signalwire/sdk";

// reads SIGNALWIRE_PROJECT_ID / SIGNALWIRE_API_TOKEN / SIGNALWIRE_SPACE
export const client = new RestClient();

// the spec's two directions, which are also the two sides of the call
export const DIRECTIONS = ["remote-caller", "local-caller"] as const;
export type Direction = (typeof DIRECTIONS)[number];

/** Translate both sides, each hearing the other's language. */
export function start(callId: string, fromLang = "en-US", toLang = "es-ES",
                      webhook?: string) {
  const inner: Record<string, unknown> = {
    from_lang: fromLang, to_lang: toLang, direction: [...DIRECTIONS],
    speech_engine: "deepgram", live_events: true,
  };
  if (webhook) {
    // translation events land here while the call runs
    inner["webhook"] = webhook;
  }
  return client.calling.liveTranslate(callId, { action: { start: inner } });
}

/** Speak a line into the conversation, translated on the way. */
export function say(callId: string, message: string,
                    direction: Direction = "remote-caller") {
  if (!DIRECTIONS.includes(direction)) {
    throw new Error(
      `direction must be one of ${DIRECTIONS.join(", ")}, not '${direction}'`);
  }
  return client.calling.liveTranslate(callId, {
    action: { inject: { message, direction } },
  });
}

/** Ask for a summary of the translated conversation so far. */
export function summary(callId: string, webhook: string, prompt?: string) {
  const inner: Record<string, unknown> = { webhook };
  if (prompt) {
    inner["prompt"] = prompt;
  }
  return client.calling.liveTranslate(callId, { action: { summarize: inner } });
}

/** End translation. The call carries on untranslated. */
export function stop(callId: string) {
  return client.calling.liveTranslate(callId, { action: { stop: {} } });
}

if (process.argv[1] && import.meta.url.endsWith(process.argv[1].replace(/\\/g, "/"))) {
  const [cmd, callId, a, b] = process.argv.slice(2);
  if (!callId) {
    console.log("usage: npm start <start|say|summary|stop> <call_id> [args]");
  } else if (cmd === "start") {
    console.log(await start(callId, a, b));
  } else if (cmd === "say") {
    console.log(await say(callId, a));
  } else if (cmd === "summary") {
    console.log(await summary(callId, a));
  } else if (cmd === "stop") {
    console.log(await stop(callId));
  }
}

Run it

cd python
pip install -r requirements.txt
cp ../.env.example .env          # project id, API token, space
python app.py start <call_id> en-US es-ES
python app.py say <call_id> "A supervisor is joining."
python app.py summary <call_id> https://your-host/summary
python app.py stop <call_id>

The TypeScript surface is the same four actions on @signalwire/sdk, on Node 20.18.1 or newer:

cd typescript
npm ci
cp ../.env.example .env
npm start start <call_id> en-US es-ES

Take the call id from a status callback or a tool webhook. The webhook you pass to start receives the translation events. The one you pass to summary receives the summary.

Verify it

No network, no account.

python verify.py          # from the recipe folder, not python/

The verifier swaps the SDK’s HTTP layer for a recorder, sends all four actions, and asserts the following.

a direction outside the two documented values raises before any request is made
calling.live_translate requires action, and the spec’s oneOf holds exactly the four documented variants
start requires from_lang, to_lang and direction, and its direction array’s items carry the same enum that inject’s single-value direction does
the speech engine enum is exactly deepgram and google
inject requires message and direction; summarize requires nothing and documents both webhook and prompt
each body equals the expected shape, and every key inside an action is a documented property of that variant
the TypeScript surface sends those same four actions and refuses the same direction

Report an issue