Transcriptions using Azure Communication Services: how to transcribe a speech produced by play_media?

Question

Transcriptions using Azure Communication Services: how to transcribe a speech produced by play_media?

Javier Montes 20

Hi everyone.

Im trying to implement an AI-based assistant that basically has to take a conversation with a real person. I would like the system to be able to deal interruptions, i.e if the assistant is speaking and the person suddenly starts a speech, the assistant has to stop.

In order to deal that I have came with the idea of using "TranscriptionOptions" from the ACS python SDK. The problem here is that all the audio that the person produces is taken by TranscriptionOptions, however, the audio that the assistant produces is achieved by using 'play_media()' method and that audio is not taken by TranscriptionOptions. What could I do?

Thanks all in advance.

Shree Hima Bindu Maganti 5,480 Reputation points Microsoft External Staff Moderator

2025-08-11T07:51:22.9533333+00:00
Hi @Javier Montes
To manage interruptions during conversations in your AI-based assistant using Azure Communication Services (ACS), you can implement a mechanism that detects when the person starts talking and then cancels or truncates the assistant's speech.

While the TranscriptionOptions captures the audio produced by the person, the assistant's audio generated by the play_media() method is not automatically transcribed. To capture the assistant's speech, you may need to ensure that the audio output from the assistant is routed through the transcription service. This can involve using the real-time transcription capabilities of ACS, which allows for capturing both the user's and the assistant's audio.

You can also utilize the response.cancel event to stop the assistant's response when the user begins to speak. This way, you can synchronize the assistant's audio output with the user's input effectively.

References:

How to use the GPT-4o Realtime API for speech and audio (Preview)

Add real-time transcription into your application (programming-language-python)
Let me know if you have any further assistances needed

Your answer

Shree Hima Bindu Maganti 5,480 Reputation points Microsoft External Staff Moderator

2025-08-11T07:51:22.9533333+00:00

Hi @Javier Montes
To manage interruptions during conversations in your AI-based assistant using Azure Communication Services (ACS), you can implement a mechanism that detects when the person starts talking and then cancels or truncates the assistant's speech.

While the TranscriptionOptions captures the audio produced by the person, the assistant's audio generated by the play_media() method is not automatically transcribed. To capture the assistant's speech, you may need to ensure that the audio output from the assistant is routed through the transcription service. This can involve using the real-time transcription capabilities of ACS, which allows for capturing both the user's and the assistant's audio.

You can also utilize the response.cancel event to stop the assistant's response when the user begins to speak. This way, you can synchronize the assistant's audio output with the user's input effectively.

References:

How to use the GPT-4o Realtime API for speech and audio (Preview)

Add real-time transcription into your application (programming-language-python)
Let me know if you have any further assistances needed

Share via

Transcriptions using Azure Communication Services: how to transcribe a speech produced by play_media?

Your answer