I am trying to connect azure voice live web socket api from my frontend and i am integrating ai foundry agent with it

Thanks for the help, authentication token is what I was missing.
But I cannot add custom header on the frontend on WebSocket connection, need to have proxy web socket connection to connect with voice live api with agent.

Any suggestion on implementing proxy for voice live in python django backend?

Hi Jerald Felix

Good day. Could you help with above query.

Thank you.

Hello Vishal Rawat,

You are correct when using the Azure Voice Live API (and the underlying OpenAI Realtime protocol) with Agents, passing the access_token (Entra ID) usually requires the Authorization header, which standard browser WebSocket APIs do not support. While some Azure services allow passing tokens in the query string, it is not always supported or secure for the Agent integration.

A backend proxy is the standard solution. It allows you to hold the credentials securely on the server and bridge the connection between your frontend and Azure.

Since you are using Django, the most robust way to implement this is using Django Channels.

Solution: Django Channels Proxy

Architecture:

1. Frontend: Connects to your Django backend: wss://yourdomain.com/ws/voice-proxy/
2. Django Backend:
   • Accepts the frontend connection.
   • Generates the Entra ID token using DefaultAzureCredential.
   • Opens a separate WebSocket connection to Azure Voice Live (injecting the Authorization header).
   • Pipes messages bi-directionally between Frontend and Azure.

Implementation Guide

1. Install Dependencies

You need channels (for Django WebSockets) and websockets (to act as a client for Azure).

pip install channels daphne websockets azure-identity

2. Django Consumer (consumers.py)

Create an asynchronous consumer that acts as the bridge.

import json

import asyncio

import websockets

from channels.generic.websocket import AsyncWebsocketConsumer

from azure.identity.aio import DefaultAzureCredential

class VoiceProxyConsumer(AsyncWebsocketConsumer):

    async def connect(self):

        # 1. Accept frontend connection

        await self.accept()

        

        # 2. Get Entra ID Token securely on backend

        credential = DefaultAzureCredential()

        token_obj = await credential.get_token("https://ai.azure.com/.default")

        self.access_token = token_obj.token

        

        # 3. Construct Upstream URL (Azure Voice Live)

        # You can fetch these from settings or query params passed from frontend

        agent_id = "your-agent-id"

        project_id = "your-project-id"

        resource_name = "your-resource-name"

        

        upstream_url = (

            f"wss://{resource_name}.services.ai.azure.com/voice-live/realtime"

            f"?api-version=2025-10-01&agent_id={agent_id}&project_id={project_id}"

        )

        # 4. Connect to Azure with Auth Header

        try:

            self.upstream_ws = await websockets.connect(

                upstream_url,

                extra_headers={"Authorization": f"Bearer {self.access_token}"}

            )

            

            # 5. Start background task to listen to Azure and forward to Frontend

            self.forward_task = asyncio.create_task(self.forward_upstream_to_client())

            

        except Exception as e:

            print(f"Upstream connection failed: {e}")

            await self.close(code=4000)

    async def disconnect(self, close_code):

        # Cleanup: Close upstream and cancel task

        if hasattr(self, 'upstream_ws'):

            await self.upstream_ws.close()

        if hasattr(self, 'forward_task'):

            self.forward_task.cancel()

    async def receive(self, text_data=None, bytes_data=None):

        # Frontend -> Azure

        # Pass data exactly as received (whether JSON text or Binary audio)

        if self.upstream_ws.open:

            if text_data:

                await self.upstream_ws.send(text_data)

            if bytes_data:

                await self.upstream_ws.send(bytes_data)

    async def forward_upstream_to_client(self):

        # Azure -> Frontend

        try:

            async for message in self.upstream_ws:

                # Determine if message is text or binary and forward accordingly

                if isinstance(message, str):

                    await self.send(text_data=message)

                else:

                    await self.send(bytes_data=message)

        except asyncio.CancelledError:

            pass

        except Exception as e:

            print(f"Error receiving from Azure: {e}")

            await self.close()

3. Routing (routing.py)

Hook this consumer into your Django Channels routing.

from django.urls import re_path

from . import consumers

websocket_urlpatterns = [

    re_path(r'ws/voice-proxy/$', consumers.VoiceProxyConsumer.as_asgi()),

]

This setup solves your auth header issue while keeping your credentials secure. The forward_upstream_to_client loop ensures that events (like response.audio.delta) from Azure are instantly pushed to your frontend.

Best Regards,

Jerald Felix

Thanks, I got my proxy up and running.
Thanks for the support.