Voice Session

Voice Session is AIVAX’s low-latency, stateful voice API. It keeps the authenticated AIVAX WebSocket endpoint while connecting to a realtime inference service. Events follow the OpenAI-compatible GA Realtime JSON protocol after the connection is upgraded.

Use Voice Session for natural spoken conversations with streaming audio, assistant speech transcripts, server voice activity detection (VAD), interruptions, and tool calls. Use Audio Transcriptions for transcription-only workloads or Speech Generation when the text to synthesize is already known.

Connect securely #

Open a WebSocket upgrade request to:

wss://inference.aivax.net/api/v1/voice-session

Authenticate the upgrade with a private account API key:

Authorization: Bearer <AIVAX_API_KEY>

The ?api-key=<AIVAX_API_KEY> query parameter is also accepted when the WebSocket client cannot set headers, but headers are preferred because URLs are commonly retained in logs and monitoring systems. Public API keys cannot open Voice Sessions.

Migrate to GA Realtime events #

After the WebSocket upgrade, exchange OpenAI-compatible GA Realtime JSON events directly. Do not send the legacy session_start envelope or depend on the former AIVAX-specific STT, TTS, WAV-segment, or playback-confirmation events.

The main migration changes are:

Use the type property to route every event. Audio remains base64-encoded inside JSON text messages; do not send binary WebSocket frames.

Configure the session #

Send session.update as the first client event. AIVAX supports the GA Realtime session fields and adds two optional selectors:

• gateway: an AI Gateway slug available to the authenticated account;
• model: gpt-realtime-2.1 or gpt-realtime-2.1-mini.

The selector supplied in the event determines the AIVAX configuration used for the session. Configure the output voice at session.audio.output.voice and reasoning effort at session.reasoning.effort.

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "gateway": "<GATEWAY_SLUG>",
    "model": "gpt-realtime-2.1",
    "instructions": "Help the caller complete the requested task.",
    "audio": {
      "input": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "turn_detection": {
          "type": "server_vad"
        }
      },
      "output": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "voice": "marin"
      }
    },
    "reasoning": {
      "effort": "low"
    }
  }
}

Choose the model based on the experience you need:

• gpt-realtime-2.1 for the highest-quality realtime conversations;
• gpt-realtime-2.1-mini for lower-cost, lighter-weight realtime workloads.

You can also provide standard GA Realtime instructions, tools, and turn-detection settings in the same update. Input-transcription sessions are not currently supported, so do not configure audio.input.transcription or depend on conversation.item.input_audio_transcription.* events. Wait for the server’s session confirmation event before treating the configuration as active. A voice cannot be changed after the session has already produced audio; reconnect to select a different voice.

Stream microphone audio #

Capture raw mono signed 16-bit little-endian PCM (s16le) at 24,000 Hz, base64-encode each chunk in chronological byte order, and append it to the input buffer:

{
  "type": "input_audio_buffer.append",
  "audio": "<BASE64_PCM_24KHZ_AUDIO>"
}

Send small chunks continuously for low latency. Do not add WAV, RIFF, or other container headers to each chunk. If automatic turn detection is enabled, server VAD determines when the user starts and stops speaking and creates responses according to the session configuration.

If turn detection is disabled, control the turn explicitly with the native buffer events:

1. Send one or more input_audio_buffer.append events.
2. Send input_audio_buffer.commit when the utterance is complete.
3. Send response.create to request the assistant response.

Use input_audio_buffer.clear to discard buffered audio that should not become a conversation turn.

Receive transcripts and audio #

Handle the native GA Realtime event stream rather than assuming one fixed sequence. In particular:

• input_audio_buffer.speech_started and input_audio_buffer.speech_stopped report server-VAD boundaries;
• conversation.item.input_audio_transcription.* events are not currently emitted because input-transcription sessions are not supported;
• response.output_audio_transcript.delta and response.output_audio_transcript.done provide the assistant’s spoken transcript;
• response.output_audio.delta carries a base64 PCM audio chunk;
• response.output_audio.done marks the end of an output-audio stream;
• response.done reports the final response status and usage;
• error reports a request or session error.

Decode each response.output_audio.delta value as raw mono signed 16-bit little-endian PCM at 24,000 Hz and queue the samples for gapless playback:

{
  "type": "response.output_audio.delta",
  "response_id": "<RESPONSE_ID>",
  "item_id": "<ITEM_ID>",
  "output_index": 0,
  "content_index": 0,
  "delta": "<BASE64_PCM_24KHZ_AUDIO>"
}

Event fields can evolve with the GA Realtime protocol. Route by type, retain the identifiers needed to correlate responses and items, and ignore unknown event types or fields that your client does not use.

Handle interruptions #

When the caller speaks over the assistant:

1. Stop or duck local playback when the confirmed input_audio_buffer.speech_started event arrives.
2. Send response.cancel if a response is still being generated.
3. Send the native conversation-item truncation event with the assistant item identifier and the amount of audio that was actually played.

Cancellation stops generation; truncation keeps the server conversation aligned with what the caller heard. Track played audio duration in the client rather than assuming every received chunk reached the speaker. Treat already-finished response races as normal and make cancel handling idempotent.

Use tools #

Declare client tools through the standard GA Realtime tools session field. Handle response.function_call_arguments.done using its response_id, call_id, name, and JSON-encoded arguments. Execute each requested function, add one native function_call_output conversation item per call_id, then send a single response.create after the originating response.done and every call from that response has an output.

Tools configured as AIVAX server InternalFunctions execute within AIVAX and require no client action. Client-defined tools continue to be returned normally and remain the client’s responsibility.

Your tool handler should validate arguments, preserve call identifiers, enforce timeouts, and return a useful error result when execution fails. Do not wait for a client tool result that is being executed as an AIVAX server function.

Billing and limits #

For current pricing, availability, and account limits, see Pricing and Plans and Limits.

Disconnect gracefully #

A Voice Session lasts only for the active WebSocket. It is not resumable after disconnecting.

For a normal shutdown:

1. Stop microphone capture and stop sending audio.
2. Let any response or required tool-result exchange finish, or cancel it explicitly.
3. Stop and drain local playback as appropriate for the product experience.
4. Close the WebSocket with a normal close code.
5. Release audio devices, playback queues, pending calls, and session state.

Handle peer closure, network loss, authentication failure, model errors, and server shutdown as terminal session outcomes. Reconnecting starts a new session: open a new authenticated WebSocket, send a new session.update, and rebuild any application context your experience requires. Use bounded retry with backoff for transient failures, but do not automatically retry authentication or configuration errors.