Callingai

multilingual

View as MarkdownOpen in Claude

Use ai.multilingual to let one AI Agent detect the language a caller is speaking and answer in that language, switching whenever the caller does. You give it a voice per language, and the agent selects the matching voice as the conversation moves between them.

Use ai.languages instead when each call stays in one language and you are choosing between a fixed set. Use neither when the agent only ever speaks one language — set the voice and leave both out.

multilingual and languages are mutually exclusive. If both are set, SignalWire uses multilingual and ignores languages entirely. To go back to languages, remove the multilingual key rather than emptying it.

multilingual must be an object. Any other value — including true — fails SWML validation and ends the call, so you cannot switch the mode off by setting it to a boolean. A SWML variable is not accepted here either, because the value is validated before variables are substituted.

Properties

ai.multilingual
object

Configures language detection and the voice used for each detected language.

ai.multilingual.languages
object[]Required

An array of voices, one per language the agent speaks.

At least one entry must supply a voice — either an entry whose language is default, or the first language listed. Without one, multilingual mode does not start and the agent falls back to a single English voice.

ai.multilingual.languages[].language
stringRequired

The language this voice applies to, as a code the ASR engine supports.

Use default instead of a code to mark the catch-all entry. The catch-all speaks any detected language that has no entry of its own, and it is the voice the agent opens the call with.

ai.multilingual.languages[].voice
string

Voice to use for this language, in the format <engine id>.<voice id>, with an optional :<model id> suffix. For example, elevenlabs.rachel or elevenlabs.rachel:eleven_flash_v2_5.

Only the catch-all entry has to carry a voice. An entry that leaves it out is answered in the catch-all voice instead.

Always name the engine. A bare voice id resolves against a default engine that can change. Refer to Supported voices and languages for the voice ids each provider offers.

ai.multilingual.languages[].model
string

The model to use for the specified TTS engine. For example, coda. Set this only if the voice string does not already carry a :<model id> suffix — a suffix on the voice string wins, and this key is then ignored without a warning.

ai.multilingual.languages[].engine
stringDeprecated

The engine to use for this language. For example, "elevenlabs". Deprecated. Set the engine with the voice parameter.

ai.multilingual.languages[].params
object

TTS engine-specific parameters for this language, such as stability and similarity. These are the same parameters accepted by languages[].params.

ai.multilingual.allowed
string[]

Restricts the agent to this set of language codes. The speech recognition engine still transcribes whatever it hears, but the agent will not switch into a language outside the list — it keeps the language it is already speaking. The list also tells the agent, in its prompt, which languages are in play.

Every code must be supported by the ASR engine and model in use, or multilingual mode does not start. Omit to let the agent follow the caller into any language the ASR engine detects.

ai.multilingual.engine
string

The ASR (Automatic Speech Recognition) engine used to detect the spoken language. This is separate from the TTS engine that speaks each voice.

If you leave it unset, deepgram is used.

Do not set this alongside params.openai_asr_engine, which overrides both this engine and model. When both are present, the params value wins.

ai.multilingual.model
stringDefaults to nova-3

The ASR model used to detect the spoken language. The model must support code-switching, or multilingual mode does not start.

ai.multilingual.provider
string

The ASR provider used to detect the spoken language.

If you leave it unset, the provider already configured for the agent is used, falling back to deepgram when the agent has none. Setting it here also applies it to the recognizer.

ai.multilingual.start_language
stringDefaults to en

The language the agent opens the conversation in, before the caller has said anything for the ASR engine to detect.

ai.multilingual.min_switch_words
integerDefaults to 2

How many words a caller must say before the agent will switch languages.

A single word is often the same in several languages — a name, no, ok — so a one-word turn is an unreliable signal. Set to 1 to allow single-word switches. A value below 1 is treated as the default of 2 rather than rejected.

ai.multilingual.fillers
string[] | object

Fillers used to break silence between turns. This is the speech filler map, so it is the counterpart of languages[].speech_fillers rather than of languages[].fillers, which is a deprecated spelling of function_fillers.

Supply an array to have the fillers translated into whichever language the caller is speaking, or an object keyed by language code to set them per language verbatim. An auto key inside the object is translated the same way an array is.

ai.multilingual.function_fillers
string[] | object

Fillers used while a SWAIG function runs.

Supply an array to have the fillers translated into whichever language the caller is speaking, or an object keyed by language code to set them per language verbatim. An auto key inside the object is translated the same way an array is.

ai.multilingual.turn_fillers
string[] | object

Short backchannels played while the caller still has the floor and the agent is deliberately waiting, as when they spell a name or read out a number, to signal it is still listening. Which waits earn one is set by params.turn_filler_sources.

Supply an object keyed by language code, optionally with a default key covering languages you have not listed. Unlike the other two, these are never translated on the fly, so the auto key does not apply here — resolving them has to be fast enough to play immediately. An array is accepted and treated as the default set.

An entry that is an absolute path or a scheme:// URI is played as audio instead of being spoken, which is a way to use a recorded backchannel.

How it differs from languages

Both configure voices, but the shape and the behavior differ:

languagesmultilingual
Chooses a languagePer callPer turn, as the caller switches
Entry keycode, plus a human-readable namelanguage, or default for the catch-all
Catch-all entryNoneThe default entry, or the first listed language
ASR configurationparams.openai_asr_engineengine, model, and provider on this object

The two languages arrays are not interchangeable. An ai.multilingual.languages entry is flat — language and voice, plus an optional model and params — and takes no name, speech_fillers, or function_fillers. Set fillers once for the whole object instead.

Where the two arrays share a property, they behave identically: voice, model, and params mean the same thing in both, and engine is deprecated in both in favor of naming the engine in the voice string.

Examples

Detect and switch between three languages

The default entry supplies the voice for the opening turn and for any detected language without an entry of its own.

1ai:
2  multilingual:
3    start_language: en
4    allowed:
5      - en
6      - es
7      - fr
8    languages:
9      - language: default
10        voice: elevenlabs.rachel
11      - language: es
12        voice: elevenlabs.maria
13      - language: fr
14        voice: gcloud.fr-FR-Neural2-B

Tune switching and per-voice parameters

Allow single-word switches, and set ElevenLabs stability and similarity per voice.

1ai:
2  multilingual:
3    min_switch_words: 1
4    languages:
5      - language: default
6        voice: elevenlabs.josh
7        params:
8          stability: 0.6
9          similarity: 0.8
10      - language: es
11        voice: elevenlabs.maria
12        params:
13          stability: 0.4
14          similarity: 0.9

Set fillers per language

An array of fillers is translated into whichever language the caller is speaking. An object sets them per language verbatim, which is required for turn_fillers.

1ai:
2  multilingual:
3    languages:
4      - language: default
5        voice: elevenlabs.rachel
6      - language: es
7        voice: elevenlabs.maria
8    function_fillers:
9      - one moment please
10      - let me check that
11    turn_fillers:
12      en:
13        - got it
14        - sure
15      es:
16        - entendido
17        - claro
18      default:
19        - mmhm