models Package

Configuration for the agent.

Configuration for animation outputs including blendshapes and visemes metadata.

An assistant message item within a conversation.

Echo cancellation configuration for server-side audio processing.

Configuration for input audio transcription.

Configuration for input audio noise reduction.

Configuration for avatar streaming and behavior during the session.

Azure custom voice configuration.

Azure personal voice configuration.

Azure semantic end-of-utterance detection (default).

Azure semantic end-of-utterance detection (English-optimized).

Azure semantic end-of-utterance detection (multilingual).

Server Speech Detection (Azure semantic VAD, default variant).

Server Speech Detection (Azure semantic VAD, English-only).

Server Speech Detection (Azure semantic VAD).

Azure standard voice configuration.

Base for Azure voice configurations.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: AzureCustomVoice, AzurePersonalVoice, AzureStandardVoice

Defines a video background, either a solid color or an image URL (mutually exclusive).

Details of output token usage.

A voicelive client event.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: ClientEventConversationItemCreate, ClientEventConversationItemDelete, ClientEventConversationItemRetrieve, ClientEventConversationItemTruncate, ClientEventInputAudioClear, ClientEventInputAudioTurnAppend, ClientEventInputAudioTurnCancel, ClientEventInputAudioTurnEnd, ClientEventInputAudioTurnStart, ClientEventInputAudioBufferAppend, ClientEventInputAudioBufferClear, ClientEventInputAudioBufferCommit, ClientEventResponseCancel, ClientEventResponseCreate, ClientEventSessionAvatarConnect, ClientEventSessionUpdate

Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a "history" of the conversation and to add new items mid-stream, but has the current limitation that it cannot populate assistant audio messages. If successful, the server will respond with a conversation.item.created event, otherwise an error event will be sent.

Send this event when you want to remove any item from the conversation history. The server will respond with a conversation.item.deleted event, unless the item does not exist in the conversation history, in which case the server will respond with an error.

Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD. The server will respond with a conversation.item.retrieved event, unless the item does not exist in the conversation history, in which case the server will respond with an error.

Send this event to truncate a previous assistant message's audio. The server will produce audio faster than voicelive, so this event is useful when the user interrupts to truncate audio that has already been sent to the client but not yet played. This will synchronize the server's understanding of the audio with the client's playback. Truncating audio will delete the server-side text transcript to ensure there is not text in the context that hasn't been heard by the user. If successful, the server will respond with a conversation.item.truncated event.

Send this event to append audio bytes to the input audio buffer. The audio buffer is temporary storage you can write to and later commit. In Server VAD mode, the audio buffer is used to detect speech and the server will decide when to commit. When Server VAD is disabled, you must commit the audio buffer manually. The client may choose how much audio to place in each event up to a maximum of 15 MiB, for example streaming smaller chunks from the client may allow the VAD to be more responsive. Unlike made other client events, the server will not send a confirmation response to this event.

Send this event to clear the audio bytes in the buffer. The server will respond with an input_audio_buffer.cleared event.

Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically. Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an input_audio_buffer.committed event.

Clears all input audio currently being streamed.

Appends audio data to an ongoing input turn.

Cancels an in-progress input audio turn.

Marks the end of an audio input turn.

Indicates the start of a new audio input turn.

Send this event to cancel an in-progress response. The server will respond with a response.cancelled event or an error if there is no response to cancel.

This event instructs the server to create a Response, which means triggering model inference. When in Server VAD mode, the server will create Responses automatically. A Response will include at least one Item, and may have two, in which case the second will be a function call. These Items will be appended to the conversation history. The server will respond with a response.created event, events for Items and content created, and finally a response.done event to indicate the Response is complete. The response.create event includes inference configuration like instructions, and temperature. These fields will override the Session's configuration for this Response only.

Sent when the client connects and provides its SDP (Session Description Protocol) for avatar-related media negotiation.

Send this event to update the session's default configuration. The client may send this event at any time to update any field, except for voice. However, note that once a session has been initialized with a particular model, it can't be changed to another model using session.update. When the server receives a session.update, it will respond with a session.updated event showing the full, effective configuration. Only the fields that are present are updated. To clear a field like instructions, pass an empty string.

Base for any content part; discriminated by type.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: ResponseAudioContentPart, RequestAudioContentPart, RequestTextContentPart, ResponseTextContentPart

The item to add to the conversation.

Base for any response item; discriminated by type.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: FunctionCallItem, FunctionCallOutputItem, MessageItem

Top-level union for end-of-utterance (EOU) semantic detection configuration.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: AzureSemanticDetection, AzureSemanticDetectionEn, AzureSemanticDetectionMultilingual

Standard error response envelope.

A function call item within a conversation.

A function call output item within a conversation.

The definition of a function tool as used by the voicelive endpoint.

ICE server configuration for WebRTC connection negotiation.

Input audio content part.

Input text content part.

Details of input token usage.

A single log probability entry for a token.

Base for any message content part; discriminated by type.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: InputAudioContentPart, InputTextContentPart, OutputTextContentPart

A message item within a conversation.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: AssistantMessageItem, SystemMessageItem, UserMessageItem

OpenAI voice configuration with explicit type field. This provides a unified interface for OpenAI voices, complementing the existing string-based OpenAIVoiceName for backward compatibility.

Output text content part.

Details of output token usage.

An audio content part for a request.

Base for session configuration shared between request and response.

A text content part for a request.

The response resource.

An audio content part for a response.

Details for a cancelled response.

Create a new VoiceLive response with these parameters.

Details for a failed response.

A function call item within a conversation.

A function call output item within a conversation.

Details for an incomplete response.

Base for any response item; discriminated by type.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: ResponseFunctionCallItem, ResponseFunctionCallOutputItem, ResponseMessageItem

Base type for message item within a conversation.

Base for session configuration in the response.

Base for all non-success response details.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: ResponseCancelledDetails, ResponseFailedDetails, ResponseIncompleteDetails

A text content part for a response.

A voicelive server event.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: ServerEventConversationItemCreated, ServerEventConversationItemDeleted, ServerEventConversationItemInputAudioTranscriptionCompleted, ServerEventConversationItemInputAudioTranscriptionDelta, ServerEventConversationItemInputAudioTranscriptionFailed, ServerEventConversationItemRetrieved, ServerEventConversationItemTruncated, ServerEventError, ServerEventInputAudioBufferCleared, ServerEventInputAudioBufferCommitted, ServerEventInputAudioBufferSpeechStarted, ServerEventInputAudioBufferSpeechStopped, ServerEventResponseAnimationBlendshapeDelta, ServerEventResponseAnimationBlendshapeDone, ServerEventResponseAnimationVisemeDelta, ServerEventResponseAnimationVisemeDone, ServerEventResponseAudioDelta, ServerEventResponseAudioDone, ServerEventResponseAudioTimestampDelta, ServerEventResponseAudioTimestampDone, ServerEventResponseAudioTranscriptDelta, ServerEventResponseAudioTranscriptDone, ServerEventResponseContentPartAdded, ServerEventResponseContentPartDone, ServerEventResponseCreated, ServerEventResponseDone, ServerEventResponseFunctionCallArgumentsDelta, ServerEventResponseFunctionCallArgumentsDone, ServerEventResponseOutputItemAdded, ServerEventResponseOutputItemDone, ServerEventResponseTextDelta, ServerEventResponseTextDone, ServerEventSessionAvatarConnecting, ServerEventSessionCreated, ServerEventSessionUpdated

Returned when a conversation item is created. There are several scenarios that produce this event:

The server is generating a Response, which if successful will produce either one or two Items, which will be of type message (role assistant) or type function_call. The input audio buffer has been committed, either by the client or the server (in server_vad mode). The server will take the content of the input audio buffer and add it to a new user message Item. The client has sent a conversation.item.create event to add a new Item to the Conversation.

Returned when an item in the conversation is deleted by the client with a conversation.item.delete event. This event is used to synchronize the server's understanding of the conversation history with the client's view.

This event is the output of audio transcription for user audio written to the user audio buffer. Transcription begins when the input audio buffer is committed by the client or server (in server_vad mode). Transcription runs asynchronously with Response creation, so this event may come before or after the Response events. VoiceLive API models accept audio natively, and thus input transcription is a separate process run on a separate ASR (Automatic Speech Recognition) model. The transcript may diverge somewhat from the model's interpretation, and should be treated as a rough guide.

Returned when the text value of an input audio transcription content part is updated.

Returned when input audio transcription is configured, and a transcription request for a user message failed. These events are separate from other error events so that the client can identify the related Item.

Returned when a conversation item is retrieved with conversation.item.retrieve.

Returned when an earlier assistant audio message item is truncated by the client with a conversation.item.truncate event. This event is used to synchronize the server's understanding of the audio with the client's playback. This action will truncate the audio and remove the server-side text transcript to ensure there is no text in the context that hasn't been heard by the user.

Returned when an error occurs, which could be a client problem or a server problem. Most errors are recoverable and the session will stay open, we recommend to implementors to monitor and log error messages by default.

Details of the error.

Returned when the input audio buffer is cleared by the client with a input_audio_buffer.clear event.

Returned when an input audio buffer is committed, either by the client or automatically in server VAD mode. The item_id property is the ID of the user message item that will be created, thus a conversation.item.created event will also be sent to the client.

Sent by the server when in server_vad mode to indicate that speech has been detected in the audio buffer. This can happen any time audio is added to the buffer (unless speech is already detected). The client may want to use this event to interrupt audio playback or provide visual feedback to the user. The client should expect to receive a input_audio_buffer.speech_stopped event when speech stops. The item_id property is the ID of the user message item that will be created when speech stops and will also be included in the input_audio_buffer.speech_stopped event (unless the client manually commits the audio buffer during VAD activation).

Returned in server_vad mode when the server detects the end of speech in the audio buffer. The server will also send an conversation.item.created event with the user message item that is created from the audio buffer.

Represents a delta update of blendshape animation frames for a specific output of a response.

Indicates the completion of blendshape animation processing for a specific output of a response.

Represents a viseme ID delta update for animation based on audio.

Indicates completion of viseme animation delivery for a response.

Returned when the model-generated audio is updated.

Returned when the model-generated audio is done. Also emitted when a Response is interrupted, incomplete, or cancelled.

Represents a word-level audio timestamp delta for a response.

Indicates completion of audio timestamp delivery for a response.

Returned when the model-generated transcription of audio output is updated.

Returned when the model-generated transcription of audio output is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Returned when a new content part is added to an assistant message item during response generation.

Returned when a content part is done streaming in an assistant message item. Also emitted when a Response is interrupted, incomplete, or cancelled.

Returned when a new Response is created. The first event of response creation, where the response is in an initial state of in_progress.

Returned when a Response is done streaming. Always emitted, no matter the final state. The Response object included in the response.done event will include all output Items in the Response but will omit the raw audio data.

Returned when the model-generated function call arguments are updated.

Returned when the model-generated function call arguments are done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Returned when a new Item is created during Response generation.

Returned when an Item is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Returned when the text value of a "text" content part is updated.

Returned when the text value of a "text" content part is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Sent when the server is in the process of establishing an avatar media connection and provides its SDP answer.

Returned when a Session is created. Emitted automatically when a new connection is established as the first server event. This event will contain the default Session configuration.

Returned when a session is updated with a session.update event, unless there is an error.

Base model for VAD-based turn detection.

VoiceLive session object configuration.

A system message item within a conversation.

Overall usage statistics for a response.

The base representation of a voicelive tool definition.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: FunctionTool

The representation of a voicelive tool_choice selecting a named function tool.

A base representation for a voicelive tool_choice selecting a named tool.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: ToolChoiceFunctionSelection

Top-level union for turn detection configuration.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: AzureSemanticVad, AzureSemanticVadEn, AzureSemanticVadMultilingual, ServerVad

A user message item within a conversation.

Defines a video crop rectangle using top-left and bottom-right coordinates.

Video streaming parameters for avatar.

Resolution of the video feed in pixels.

Error object returned in case of API failure.

Specifies the types of animation data to output.

Output timestamp types supported in audio response content.

Union of all supported Azure voice types.

Client event types used in VoiceLive protocol.

Type of ContentPartType.

Threshold level settings for Azure semantic end-of-utterance detection.

Input audio format types supported.

Indicates the processing status of an item or parameter.

Type of ItemType.

Type of MessageRole.

Supported modalities for the session.

Supported OpenAI voice names (string enum).

Output audio format types supported.

PersonalVoice models.

Indicates the processing status of a response item.

Terminal status of a response.

Server event types used in VoiceLive protocol.

The available set of mode-level, string literal tool_choice options for the voicelive endpoint.

The supported tool type discriminators for voicelive tools. Currently, only 'function' tools are supported.

Type of TurnDetectionType.