Frontend Voice Integration Guide

Build a real-time voice assistant into your own frontend using the Syncli WebRTC endpoint.

Docs URL: devoice.syncli.cloud/docs/voice · Gateway: https://devoice.syncli.cloud / https://apidev.syncli.cloud · Agent behaviour & service files ↓ · Full-duplex & barge-in ↓

Prerequisites

1. Get an API Key
   Generate a syncli_... key from the Developer Portal in the Syncli web app, or use a session JWT. All tokens work the same way.

2. Base URL
   The voice gateway runs on the same host as the API. Examples: https://devoice.syncli.cloud, https://apidev.syncli.cloud (development), https://apivoice.syncli.cloud (production).

3. Browser Permissions
   Your page must be served over HTTPS (required for getUserMedia) and the user must grant microphone access.

End-user session identity (API embeds)

When you embed Syncli voice on your website with a syncli_... API key, every visitor must be identified by a stable end_user_session_id that your frontend generates and sends on POST /start. This is separate from the gateway sessionId returned in the response (that ID is per WebRTC call).

Without end_user_session_id, the gateway falls back to the client IP address. Multiple visitors behind the same NAT or corporate network would share one caller profile, chat history, rate-limit bucket, and webhook identity.

What it enables

• Per-browser visitor tracking (matches your Rate Capping → Tracking Mode setting, e.g. browser)
• Stable chat_id across reconnects so the agent remembers prior conversation
• Correct resolved_tracking_key and is_returning_user in post-call webhooks
• Isolated usage limits per end user instead of per IP

Generate and persist a browser session ID

Create a UUID once per browser profile and store it in localStorage (or your own auth/session store). Reuse the same value on every call and reconnect from that browser.

function getBrowserSessionId() {
  const key = 'syncli_browser_session_id';
  let id = localStorage.getItem(key);
  if (!id) {
    id = (crypto.randomUUID && crypto.randomUUID())
      || ('bs_' + Date.now() + '_' + Math.random().toString(36).slice(2));
    localStorage.setItem(key, id);
  }
  return id;
}

If your app already has a logged-in user, you may use your own stable ID instead (e.g. user_abc123), as long as it is unique per end user and consistent across visits.

POST /start request fields

Also send the same end_user_session_id (and persisted chat_id) inside request_data when posting the WebRTC SDP offer.

Connection Flow (6 Steps)

Step 1 — Start a Session

Create a session and receive ICE server configuration before building the WebRTC connection.

const BASE_URL = 'https://apidev.syncli.cloud';
const TOKEN = 'syncli_your_api_key_here';

const startRes = await fetch(`${BASE_URL}/start`, {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${TOKEN}`
  },
  body: JSON.stringify({
    end_user_session_id: getBrowserSessionId(),
    chat_id: localStorage.getItem('syncli_chat_id') || '',
    end_user_email: loggedInUser?.email || null,
    client_timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
    client_timestamp: new Date().toISOString(),
  })
});

if (!startRes.ok) throw new Error('Session start failed');

const startData = await startRes.json();
// Save chat_id for later sessions
if (startData.chat_id) localStorage.setItem('syncli_chat_id', startData.chat_id);

startData fields: | Field | Type | Description | |-------|------|-------------| | sessionId | string | Unique session UUID | | chat_id | string | Conversation thread ID | | iceServers | array | TURN/STUN config for WebRTC | | iceTransportPolicy | string | Usually "relay" |

Step 2 — Create RTCPeerConnection

Use the ICE config returned from /start. This is essential — never hardcode STUN/TURN servers.

const pc = new RTCPeerConnection({
  iceServers: startData.iceServers,
  iceTransportPolicy: startData.iceTransportPolicy || 'relay'
});

Step 3 — Attach Microphone + Data Channel

Full-duplex requirement: the microphone must stay open for the entire call. Syncli runs STT on your uplink even while the bot is speaking so users can interrupt (barge-in). See Full-Duplex & Barge-In below.

// 🎙️ Get microphone — keep constraints simple (see Full-Duplex section)
const localStream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    autoGainControl: true,
    channelCount: 1,
  },
  video: false,
});

// Bidirectional audio — attach BEFORE createOffer()
const audioTransceiver = pc.addTransceiver('audio', { direction: 'sendrecv' });
if (localStream.getAudioTracks()[0]) {
  await audioTransceiver.sender.replaceTrack(localStream.getAudioTracks()[0]);
}

// 🔊 Play remote bot audio (required for browser AEC)
pc.ontrack = (event) => {
  if (event.track.kind === 'audio') {
    const botAudio = document.getElementById('bot-audio') || new Audio();
    botAudio.srcObject = event.streams[0];
    botAudio.autoplay = true;
    botAudio.play().catch(() => {});
  }
};

// 📡 Create data channel for text messages
const dc = pc.createDataChannel('chat', { ordered: true });

// Handle messages FROM the bot
dc.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  handleVoiceMessage(msg);
};

// Heartbeat
let pingInterval = setInterval(() => {
  if (dc.readyState === 'open') dc.send(`ping: ${Date.now()}`);
}, 1000);

Do not use { ideal: true } wrappers, forced sampleRate: 48000, or Chrome-only goog* constraints — they double-cancel echo and can remove the user's voice during barge-in.

Step 4 — Handle Bot Messages

The chat data channel delivers all text/audio lifecycle events.

function handleVoiceMessage(msg) {
  const type = msg.type;
  const text = msg.data?.text;

  switch (type) {
    case 'user-transcript':
      // What the bot heard you say (STT result)
      displayUserMessage(text);
      break;

    case 'bot-output':
      // TTS-expanded text — do NOT render in chat (use bot-output-raw)
      break;

    case 'bot-llm-text':
      // Alternate LLM stream — optional
      break;

    case 'bot-output-raw':
      // Canonical reply text for chat UI
      replaceWithRawText(text);
      break;

    case 'bot-started-speaking':
      // New turn — do NOT mute the microphone here
      startNewBotBubble();
      break;

    case 'bot-stopped-speaking':
      // Turn finished
      finalizeBotBubble();
      break;
  }
}

Chat UI pattern: ignore bot-output and display bot-output-raw only. Track a rawLocked flag so late TTS chunks do not overwrite the final message.

let rawLocked = false;
let currentBotTurnDivs = [];

dc.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === 'bot-output') return;

  if (msg.type === 'bot-output-raw' && msg.data?.text) {
    rawLocked = true;
    currentBotTurnDivs.forEach(el => el.remove());
    currentBotTurnDivs = [];
    appendBotMessage(msg.data.text);
  } else if (msg.type === 'bot-started-speaking') {
    rawLocked = false;
  } else if (msg.type === 'user-transcript' && msg.data?.text) {
    appendUserMessage(msg.data.text);
    currentBotTurnDivs = [];
  }
};

Key nuance: bot-output contains TTS-expanded phrasing (spoken URLs). bot-output-raw is what your chat UI should show.

Step 5 — WebRTC Offer / Answer Exchange

Create your offer, send it to the gateway, and apply the answer.

const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

// Send offer
const offerRes = await fetch(
  `${BASE_URL}/sessions/${startData.sessionId}/api/offer`,
  {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${TOKEN}`
    },
    body: JSON.stringify({
      sdp: pc.localDescription.sdp,
      type: pc.localDescription.type,
      request_data: {
        end_user_session_id: getBrowserSessionId(),
        chat_id: startData.chat_id || localStorage.getItem('syncli_chat_id'),
        client_timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
        client_timestamp: new Date().toISOString()
      }
    })
  }
);

if (!offerRes.ok) throw new Error('Offer rejected');

const answer = await offerRes.json();
pc.pc_id = answer.pc_id;  // Save for ICE candidate routing
await pc.setRemoteDescription(new RTCSessionDescription(answer));

Step 6 — Exchange ICE Candidates

ICE candidates must be sent to the gateway so the bot can reach you through the relay.

const iceBuffer = [];

pc.onicecandidate = async (event) => {
  if (!event.candidate) {
    console.log('ICE gathering complete');
    return;
  }

  const mapped = {
    candidate: event.candidate.candidate,
    sdp_mid: event.candidate.sdpMid,
    sdp_mline_index: event.candidate.sdpMLineIndex
  };

  // Buffer if pc_id isn't known yet
  if (!pc.pc_id) {
    iceBuffer.push(mapped);
    return;
  }

  await sendIceCandidates([mapped]);
};

async function sendIceCandidates(candidates) {
  try {
    await fetch(`${BASE_URL}/sessions/${startData.sessionId}/api/offer`, {
      method: 'PATCH',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${TOKEN}`
      },
      body: JSON.stringify({
        pc_id: pc.pc_id,
        candidates
      })
    });
  } catch (e) {
    console.error('ICE candidate transmission failed:', e);
  }
}

// 🧊 Flush buffered candidates after answer
if (iceBuffer.length > 0) {
  await sendIceCandidates(iceBuffer);
  iceBuffer.length = 0;
}

Complete Minimal Example

<!DOCTYPE html>
<html>
<head>
  <title>Syncli Voice Demo</title>
  <!-- Load the Syncli voice audio helper for mic constraints & AEC warmup -->
  <script src="https://apidev.syncli.cloud/dev/js/voice-audio.js"></script>
</head>
<body>
  <button id="connect">Connect</button>
  <div id="chat"></div>

  <script type="module">
    const BASE = 'https://apidev.syncli.cloud';
    const TOKEN = 'syncli_your_api_key_here';
    let pc, dc, stream, aecWarmup;

    function getBrowserSessionId() {
      const key = 'syncli_browser_session_id';
      let id = localStorage.getItem(key);
      if (!id) {
        id = (crypto.randomUUID && crypto.randomUUID())
          || ('bs_' + Date.now() + '_' + Math.random().toString(36).slice(2));
        localStorage.setItem(key, id);
      }
      return id;
    }

    document.getElementById('connect').onclick = async () => {
      // 1. Start
      const s = await (await fetch(`${BASE}/start`, {
        method: 'POST',
        headers: { 'Authorization': `Bearer ${TOKEN}`, 'Content-Type': 'application/json' },
        body: JSON.stringify({
          end_user_session_id: getBrowserSessionId(),
          client_timezone: Intl.DateTimeFormat().resolvedOptions().timeZone
        })
      })).json();
      if (s.chat_id) localStorage.setItem('syncli_chat_id', s.chat_id);

      // 2. Peer connection
      pc = new RTCPeerConnection({ iceServers: s.iceServers, iceTransportPolicy: 'relay' });

      // 3. Mic — use SyncliVoiceAudio helper for mobile-optimized constraints
      const voiceConstraints = SyncliVoiceAudio.getVoiceAudioConstraints();
      stream = await navigator.mediaDevices.getUserMedia(voiceConstraints);
      stream.getAudioTracks().forEach(t => pc.addTransceiver(t, { direction: 'sendrecv' }));

      // 4. Data channel & AEC warm-up notifier
      dc = pc.createDataChannel('chat');
      aecWarmup = SyncliVoiceAudio.createAecWarmupNotifier(dc, (msg) => append(`sys: ${msg}`));

      dc.onopen = () => {
        if (aecWarmup) aecWarmup.onDataChannelOpen();
      };

      dc.onmessage = (e) => {
        const m = JSON.parse(e.data);
        if (m.type === 'bot-output') return; // skip TTS-expanded formatting
        if (m.data?.text) append(`${m.type}: ${m.data.text}`);
      };

      // 5. Offer
      await pc.setLocalDescription(await pc.createOffer());
      const ans = await (await fetch(`${BASE}/sessions/${s.sessionId}/api/offer`, {
        method: 'POST',
        headers: { 'Authorization': `Bearer ${TOKEN}`, 'Content-Type': 'application/json' },
        body: JSON.stringify({
          sdp: pc.localDescription.sdp,
          type: pc.localDescription.type,
          request_data: {
            end_user_session_id: getBrowserSessionId(),
            chat_id: s.chat_id
          }
        })
      })).json();
      pc.pc_id = ans.pc_id;
      await pc.setRemoteDescription(new RTCSessionDescription(ans));

      // 6. ICE
      const buffer = [];
      pc.onicecandidate = (e) => {
        if (!e.candidate) return;
        const c = { candidate: e.candidate.candidate, sdp_mid: e.candidate.sdpMid, sdp_mline_index: e.candidate.sdpMLineIndex };
        if (!pc.pc_id) buffer.push(c);
        else flush(c);
      };
      async function flush(c) {
        await fetch(`${BASE}/sessions/${s.sessionId}/api/offer`, {
          method: 'PATCH',
          headers: { 'Authorization': `Bearer ${TOKEN}`, 'Content-Type': 'application/json' },
          body: JSON.stringify({ pc_id: pc.pc_id, candidates: [c] })
        });
      }

      // Bot speaking indicator
      pc.ontrack = (e) => {
        const audio = new Audio();
        audio.srcObject = e.streams[0];
        audio.autoplay = true;
        audio.play()
          .then(() => {
            if (aecWarmup) aecWarmup.onRemoteAudioPlaying();
          })
          .catch(err => console.error('Audio play failed:', err));
      };

      append('Connected! Say something...');
    };

    function append(text) {
      const d = document.createElement('div');
      d.textContent = text;
      document.getElementById('chat').appendChild(d);
    }
  </script>
</body>
</html>

Reconnection Pattern

Connection drops are normal in transit. The agent supports fast reconnection.

pc.oniceconnectionstatechange = () => {
  if (pc.iceConnectionState === 'failed' || pc.iceConnectionState === 'disconnected') {
    // Don't panic — try to reconnect
    scheduleReconnectCheck();
  }
};

function scheduleReconnectCheck() {
  setTimeout(() => {
    if (pc.iceConnectionState === 'disconnected') {
      // Re-run the entire connect() flow
      connect();
    }
  }, 2500);
}

Agent Behaviour & Service Files

By default, each voice session uses your Developer Portal settings (Persona tab, Knowledge Base, Plugins). To get full per-call control — personality, which tools the LLM may call, MCP server URLs, LLM parameters — pass an optional agent_config object on POST /start.

Backward compatible: omit agent_config and behaviour is unchanged from the portal defaults.

How config is merged

1. Built-in Syncli defaults (API vs workspace tool sets)
2. Portal resources (default persona, knowledge base, enabled plugins)
3. Named profile from extensions/agent_config_service.py (when you set agent_config_ref)
4. Inline fields on your POST /start body (highest priority)

The gateway resolves config once at /start, stores it on the session, and forwards it to the voice bot on WebRTC offer. You do not need to repeat the full agent_config in offer request_data unless you want to override mid-call.

Inline agent_config on POST /start

const startRes = await fetch(`${BASE_URL}/start`, {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${TOKEN}`
  },
  body: JSON.stringify({
    end_user_session_id: getBrowserSessionId(),
    client_timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
    client_timestamp: new Date().toISOString(),
    agent_config: {
      persona: {
        name: 'Support',
        identity: 'You are Acme Corp support — concise and helpful.',
        rules: 'Escalate billing issues with transfer_to_human.'
      },
      persona_addon: 'Always greet by first name when known.',
      knowledge_base_id: 'kb_xxxxxxxx',       // optional — portal KB id
      tools: {
        mode: 'allowlist',
        allow: ['save_visitor_info', 'transfer_to_human'],
        include_native: true,
        include_plugins: true,
        include_mcp: true
      },
      mcp_servers: [
        {
          server_id: 'crm',
          url: 'http://localhost:3100/sse',
          headers: { Authorization: 'Bearer YOUR_MCP_TOKEN' }
        }
      ],
      plugin_ids: ['scheduler'],              // optional — limit to named plugins
      llm: { temperature: 0.15, max_tokens: 140 },
      web_search: false,
      strict_api_mode: true,                  // false = skip built-in API doc-only rules
      metadata: { page: '/pricing', plan: 'pro' },
      prompt_addon: 'Mention our 30-day trial when relevant.'
    }
  })
});

Field reference

Named profiles — agent_config_service.py

For ChatGPT / Grok-style apps, define reusable agent profiles in server-side Python instead of hardcoding large prompts in the browser. On the gateway host, edit voice_agent/extensions/agent_config_service.py:

AGENT_PROFILES = {
    "support_voice": {
        "persona": {
            "name": "Support",
            "identity": "You are Acme Corp voice support.",
            "rules": "Use transfer_to_human when billing is involved.",
        },
        "tools": {
            "mode": "allowlist",
            "allow": ["save_visitor_info", "transfer_to_human", "check_order_status"],
            "include_plugins": True,
            "include_mcp": True,
        },
        "mcp_servers": [
            {
                "server_id": "orders",
                "url": "http://127.0.0.1:3100/sse",
                "headers": {"Authorization": "Bearer ${ORDERS_MCP_TOKEN}"},
            }
        ],
        "llm": {"temperature": 0.12, "max_tokens": 130},
        "web_search": False,
    },
    "sales_voice": {
        "persona": {
            "name": "Sales",
            "identity": "You are an upbeat sales assistant for Acme Corp.",
        },
        "tools": {"mode": "allowlist", "allow": ["save_visitor_info"]},
    },
}

Reference a profile from your frontend:

agent_config: {
  agent_config_ref: 'support_voice',
  metadata: { page: window.location.pathname, locale: 'en-US' }
}

Dynamic control — resolve_for_session()

Implement last-mile logic in AgentConfigService.resolve_for_session() (same service file). The gateway calls this hook after merging the named profile and inline overrides. Use it to:

• Point MCP URLs at environment-specific ports (MY_MCP_PORT)
• Load persona text or tool lists from your database per developer user
• Attach CRM / cart context from your backend into metadata
• Switch profiles based on session_context (client_ip, chat_id, end_user_email)

@staticmethod
def resolve_for_session(user_id, merged_config, session_context=None):
    import os
    ctx = session_context or {}

    # Example: CRM MCP port from env
    port = os.environ.get("CRM_MCP_PORT", "3100")
    for srv in merged_config.get("mcp_servers", []):
        if srv.get("server_id") == "orders":
            srv["url"] = f"http://127.0.0.1:{port}/sse"

    # Example: page-specific prompt
    page = (merged_config.get("metadata") or {}).get("page", "")
    if page.startswith("/billing"):
        merged_config.setdefault("tools", {})["allow"] = [
            "save_visitor_info", "transfer_to_human"
        ]
        merged_config["persona_addon"] = "You are on the billing page — be extra careful with PII."

    return merged_config

Controller & router layer

Copy template_*.py to your own module names (e.g. acme_service.py, acme_router.py). Any file ending in _router.py under extensions/ is auto-registered at gateway startup.

Preview before going live

Dry-run the same merge logic the gateway uses, without starting WebRTC:

curl -s https://devoice.syncli.cloud/api/agent-config/resolve \
  -H "Authorization: Bearer syncli_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "is_api": true,
    "agent_config": { "agent_config_ref": "support_voice", "metadata": { "page": "/pricing" } },
    "session_context": { "client_ip": "203.0.113.10" }
  }'

List registered profile names:

curl -s https://devoice.syncli.cloud/api/agent-config/profiles \
  -H "Authorization: Bearer syncli_YOUR_KEY"

Custom tools — Plugin system

Register Python tools (with optional persona_addon) via the Developer Portal or REST API. Plugins are scoped to your API key and loaded when tools.include_plugins is true.

# Plugin source_code (stored via POST /api/plugins/{api_key})
from plugins.types import Plugin, Tool

async def check_order_status(params, deps):
    order_id = params.arguments.get("order_id")
    # deps["user_id"], deps["chat_id"], deps["google_actions"], etc.
    return {"status": "shipped", "order_id": order_id}

PLUGIN = Plugin(
    name="orders",
    persona_addon="Confirm order numbers before looking them up.",
    tools=[
        Tool(
            name="check_order_status",
            description="Look up shipment status for an order id",
            parameters={
                "type": "object",
                "properties": {"order_id": {"type": "string"}},
            },
            required=["order_id"],
            handler=check_order_status,
        )
    ],
)

Limit plugins per session with plugin_ids: ["orders"] or expose only plugin tools via tools.allow.

External tools — MCP servers

Connect Model Context Protocol (MCP) servers over SSE for the duration of a single voice session. Tools are discovered at connect time and registered with the LLM as {server_id}_{tool_name}.

"mcp_servers": [
  {
    "server_id": "github",
    "url": "https://your-mcp-host.example/sse",
    "headers": { "Authorization": "Bearer mcp_token" }
  }
]

Set the MCP listen port in your service file (see resolve_for_session above) so dev/staging/prod each point at the correct process.

Built-in API embed tools

When using a syncli_... API key (embedded voice), the default native tool set is:

Workspace (JWT) sessions get the full Google Calendar, Gmail, Tasks, and Contacts tool set unless you restrict via agent_config.tools.

Pre-call metadata sync (optional)

Push page context to the gateway before /start so resolve_for_session or your service can read it:

await fetch(`${BASE_URL}/api/extensions/metadata`, {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${TOKEN}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    metadata: {
      cart_total: 149.99,
      sku: 'PRO-ANNUAL',
      logged_in: true
    }
  })
});

Then pass overlapping keys in agent_config.metadata on /start for prompt injection.

Tool mode examples

// Only visitor capture + human handoff (typical support widget)
"tools": { "mode": "allowlist", "allow": ["save_visitor_info", "transfer_to_human"] }

// Everything except email tools
"tools": { "mode": "denylist", "deny": ["send_email", "read_latest_email"] }

// Plugin + MCP only (no native Google tools)
"tools": { "mode": "default", "include_native": false, "include_plugins": true, "include_mcp": true }

Full-Duplex & Barge-In (Listen While Speaking)

Syncli voice sessions are full-duplex: the bot keeps listening while it speaks, and users can interrupt mid-sentence (barge-in). The server transcribes your uplink continuously and stops TTS when it detects a real user utterance.

Your frontend must keep sending microphone audio for the entire call. Most barge-in failures are caused by frontend mic handling, not the API.

How barge-in works (server)

You do not need to send a special “interrupt” message — keep the mic open and speak clearly.

Frontend checklist (required)

Microphone constraints (critical)

WebRTC already applies AEC against the remote bot track. Extra browser-layer cancellation can strip the user's voice while the bot speaks. In production, use the global helper SyncliVoiceAudio.getVoiceAudioConstraints() which handles browser/device detection, disables Auto-Gain Control on mobile to prevent loopback echo, and enables iOS Voice Isolation if supported.

// ✅ Recommended pattern using SyncliVoiceAudio helper
const voiceConstraints = SyncliVoiceAudio.getVoiceAudioConstraints();
const localStream = await navigator.mediaDevices.getUserMedia(voiceConstraints);

If configuring constraints manually, ensure the following structure to optimize echo cancellation and avoid double-processing:

// Manual production pattern equivalent:
const isIOS = /iPhone|iPad|iPod/i.test(navigator.userAgent);
const isMobile = isIOS || /Android/i.test(navigator.userAgent);

const audioConstraints = {
  channelCount: 1,
  echoCancellation: true,
  noiseSuppression: true,
  autoGainControl: !isMobile, // Disable AGC on mobile to prevent feedback loops
};

// Support native Voice Isolation on iOS devices
if (isIOS && navigator.mediaDevices && navigator.mediaDevices.getSupportedConstraints) {
  const supported = navigator.mediaDevices.getSupportedConstraints();
  if (supported.voiceIsolation) {
    audioConstraints.voiceIsolation = true;
  }
}

const localStream = await navigator.mediaDevices.getUserMedia({
  audio: audioConstraints,
  video: false
});

Avoid these patterns:

// ❌ Over-constrained — can break barge-in
audio: {
  echoCancellation: { ideal: true },
  sampleRate: 48000,
}

// ❌ Chrome-only — double-cancel against WebRTC AEC
audio: {
  googEchoCancellation: true,
  googAutoGainControl: true,
}

AEC Warm-up Notifier (Required for Clean Turn Startup)

When remote bot audio first starts playing, the browser's Acoustic Echo Cancellation (AEC) requires a brief period to adapt and model the acoustic loopback path. If the bot greets the user immediately, the microphone will capture the bot's own greeting and interpret it as user speech before the AEC has built its echo model. This triggers false interruptions (barge-in) and interrupts the greeting.

To prevent this, the gateway holds the initial greeting until the client signals that the AEC model is warm. The client must monitor the remote audio playback, wait for a device-specific warmup delay, and then send an aec-ready message over the data channel.

• Desktop warmup delay: 2500ms
• Mobile warmup delay: 3000ms
• Data channel message: JSON {"type": "aec-ready"} or string "aec-ready"

Use the helper SyncliVoiceAudio.createAecWarmupNotifier(dataChannel, logCallback) to handle this automatically:

const aecWarmup = SyncliVoiceAudio.createAecWarmupNotifier(dc, (msg) => console.log(msg));

// Trigger when the data channel opens
dc.onopen = () => {
  aecWarmup.onDataChannelOpen();
  // ... rest of your dc.onopen handlers ...
};

// Trigger when the remote bot audio begins playing
audioEl.play()
  .then(() => {
    if (aecWarmup) aecWarmup.onRemoteAudioPlaying();
  })
  .catch(e => console.error("Audio play failed:", e));

The backend voice pipeline has a 6.0-second safety fallback timeout. If the client does not send aec-ready within 6.0 seconds, the agent will greet the user anyway.

Migration guide (existing integrations)

Reference: apps/voice-web/src/main.js (portal) · developer-portal/js/webrtc.js (API embed demo) · developer-portal/js/voice-audio.js (helper source)

Common Issues

Authentication Modes

The voice gateway accepts three token types on every endpoint:

All three work identically:

Authorization: Bearer <token>

Data Channel Format Reference

interface VoiceMessage {
  type: 'user-transcript' | 'bot-output' | 'bot-llm-text' | 'bot-output-raw' | 'bot-started-speaking' | 'bot-stopped-speaking';
  data: { text?: string };
}

Endpoint Summary