How To Fix The Voice Live API issue so that it can use the agents knowledge

Question

How To Fix The Voice Live API issue so that it can use the agents knowledge

Intikhab Hussain 0

I have an agent in AI Foundry with knowledge sources including uploaded documents and Bing Search. I have created a chat app using ReactJS (TypeScript) and FastAPI, where I can upload documents; these documents are also used as part of the agent's knowledge. I've connected my chat app to the agent in AI Foundry so that it can utilize the agent's knowledge, and this integration works fine.

Now I want to integrate the Azure Voice Live API with this setup. When I test the Azure Voice Live API within Azure AI Foundry, it uses the agent’s knowledge as expected. However, when I try to use the Azure Voice Live API in my chat app, it does not leverage the agent’s knowledge."

1 answer

Your answer

Answer 1

Pavankumar Purilla 10,350 Microsoft External Staff Moderator

Hi Intikhab Hussain,

When using the Azure Voice Live API outside of Azure AI Foundry, the voice session does not automatically connect to the agent runtime or its knowledge sources. This is why testing within AI Foundry works as expected, but your custom application does not leverage the agent’s knowledge. To fix this, you need to explicitly bind the Voice Live API session to the agent by creating a session through the Agent Runtime API and passing the agentId and, if required, a threadId. This ensures that the voice session is routed through the agent runtime, enabling it to access the same knowledge sources that are available in the Foundry environment. Without this binding, the Voice Live API will only interact with the base model and will not include the agent’s knowledge.

Pavankumar Purilla 10,350 Reputation points Microsoft External Staff Moderator

2025-08-06T02:22:48.5666667+00:00

Hi Intikhab Hussain,
Did you get any chance to check the response. Thank you!
Intikhab Hussain 0 Reputation points

2025-08-06T04:37:10.4466667+00:00

I am using ReactJS Typescript as frontend and fastapi as backend.
Can you explain it how to do that. Any sample code.
I already have added my code in text file as attachmentcode.txt

I am getting this error when i try to add the agentid and thread id

Voice error: {'agent': [{'type': 'extra_forbidden', 'msg': 'Extra inputs are not permitted'}], 'thread': [{'type': 'extra_forbidden', 'msg': 'Extra inputs are not permitted'}]}
Intikhab Hussain 0 Reputation points

2025-08-07T04:17:16.1166667+00:00

Hi Pavankumar Purilla
Thanks for response.
Actually i applied that what you said but I am getting error
WebSocket connection to 'wss://resource_name.cognitiveservices.azure.com/voice-agent/realti…qj02pW4CXHOwSW_byUC6cVf9dMZUS4-yLH-DZsfQuUEpOJWzLSzHRbLHoO6aClPsqCLyZ' failed: HTTP Authentication failed; no valid credentials available

This is my code
code.txt
Pavankumar Purilla 10,350 Reputation points Microsoft External Staff Moderator

2025-08-07T07:34:17.09+00:00

Hi Intikhab Hussain,
The error you encountered (Extra inputs are not permitted for agent and thread) is appearing because the Azure Voice Live API does not allow these parameters to be included directly in the request payload. This means you should only pass the fields that are officially accepted in the API documentation. To resolve this, remove any unsupported fields such as agent and thread from your Voice Live API or WebSocket requests. For authentication errors like HTTP Authentication failed; no valid credentials available, ensure that your Speech resource key or access token is valid and added in the headers exactly as documented—either as Ocp-Apim-Subscription-Key or Authorization: Bearer <token>. Also, double-check that your Azure resource region and endpoint URL are correct. The integration between your web app and agent knowledge must be accomplished following the supported methods in the official documentation, usually through endpoint configuration or authentication, not by adding extra fields in the payload.
For your reference:
Quickstart: Create a voice live real-time voice agent (Preview)
Quickstart: Create a voice live real-time voice agent with Azure AI Foundry Agent Service (Preview)
How to troubleshoot Speech SDK issues
Pavankumar Purilla 10,350 Reputation points Microsoft External Staff Moderator

2025-08-08T06:20:33.9033333+00:00

Hi Intikhab Hussain,
Just following up to see if you had a chance to review the above response. Thank you!
Intikhab Hussain 0 Reputation points

2025-08-08T11:41:48.38+00:00
Hi Pavankumar Purilla
Thanks again for response.
I am already using the Authorization: Bearer <token> still i am getting the
HTTP Authentication failed; no valid credentials available
This is my web socket connection url
WebSocket connection to 'wss://resourcename.cognitiveservices.azure.com/voice-agent/realtime?api-version=2025-05-01-preview&x-ms-client-request-id=a624f746-8e2a-4150-998e-36e5e5642404&agent_id=agentid1&agent-project-name=projectname&agent_access_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkpZaEFjVFBNWl9MWDZEQmxPV1E3SG4wTmVYRSIsImtpZCI6IkpZaEFjVFBNWl9MWDZEQmxPV1E3SG4wTmVYRSJ9.eyJhdWQiOiJodHRwczovL2FpLmF6dXJlLmNvbSIsImlzcyI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0L2YzMmI5N2YwLWVmYjgtNGJjMy05MWVlLTE4YTZlNWY2MzVjOS8iLCJpYXQiOjE3NTQ2NTI2MDgsIm5iZiI6MTc1NDY1MjYwOCwiZXhwIjoxNzU0NjU2NjM0LCJhY3IiOiIxIiwiYWlvIjoiQWFRQVcvOFpBQUFBa0lyeXNsL1lUVWZ3eH
Here I am Passing

Resourcename

api version

x-ms-client-request-id

agentId

Project Name

agent_access_token

This is my code
code.txt
Pavankumar Purilla 10,350 Reputation points Microsoft External Staff Moderator

2025-08-11T05:03:37.5466667+00:00

Hi Intikhab Hussain,
The errors occur because the parameters agent and thread are being sent to the base Voice Live API endpoint, which does not support them, resulting in the “extra_forbidden” message. To bind a voice session to your AI Foundry agent so it can access its knowledge sources, you must call the agent-enabled voice session endpoint (https://<region>.api.cognitive.microsoft.com/agents/v1/voice/sessions) from your FastAPI backend, passing the agentId (and optionally threadId) in the request body. This endpoint will return a wsUrl that is already presigned with authentication details. You should then return this wsUrl to your ReactJS frontend and connect directly to it using new WebSocket(wsUrl) without adding any manual authentication headers. Using the correct endpoint and the presigned wsUrl ensures that your voice session is linked to the agent runtime and has full access to your uploaded documents and Bing Search knowledge.
Intikhab Hussain 0 Reputation points

2025-08-12T03:59:02.85+00:00

Hi Pavankumar Purilla
Thanks again
let me try this also.

Share via

How To Fix The Voice Live API issue so that it can use the agents knowledge

1 answer

Your answer