Using a Custom AI Avatar on the sample code

Hi It is VMS

Thank you for sharing your observation. It seems you are not able to use your custom avatar on in provided sample code and looking a review on needed modification.

Going through below code and checking with team internally for needed modification.

https://github.com/Azure-Samples/cognitive-services-speech-sdk/tree/master/samples/js/browser/avatar

Shall revert back asap.

Notes:

Custom avatar training is only available in the following service regions: Southeast Asia, West Europe, and West US 2.

You can use a custom avatar model in the following service regions: Southeast Asia, North Europe, West Europe, Sweden Central, South Central US, East US 2, and West US

Thank you

Thank you @Manas Mohanty

I suspect that the "custom avatar" was either not tested, or the form needs some modifications.

For example, on the top of chat.html there is the input for the "Azure Speech Resource". Perhaps this needs to be on the same region as the custom avatar? In which case, which is the API key to be used here? The one provided with the "custom avatar", or the key for the endpoint on that region?

In any case, I think the documentation needs to be improved. I am just keeping my fingers crossed that the sample chat.html does support custom avatar as we had to go through a very long process to get access to it and to get the budget approved to pay for the custom avatar. All that is at risk now for a very interesting project using the custom avatar.

I await your response(s).

Thanks again!

Hi It is VMS

It seems you are trying to interact with your data with OpenAI.

We should run chat.html and input the values of OpenAI , AI Search, Custom Avatar configuration in the text boxes.

Regarding access on custom avatar

Could you confirm that you have added below role on top of owner roles to Speech service to yourself.

Cognitive Services Speech Contributor for your speech resources and Azure OpenAI/AI search resource

Cognitive Service OpenAI contributor for AI search and Speech

Subnet join/read for All cognitive resources and user if virtual network protected, need to add private endpoint url in chat.js

Code modification

We might need to do more modification here in chat.js mentioning any ICE server url and credentials

We might need to add private endpoint url if private endpoint is added.

function setupWebRTC(iceServerUrl, iceServerUsername, iceServerCredential) {
    // Create WebRTC peer connection
    peerConnection = new RTCPeerConnection({
        iceServers: [{
            urls: [ iceServerUrl ],
            username: iceServerUsername,
            credential: iceServerCredential
        }]
    })

Shall revert back once cross checked with FTE's here with a minimal repro. will check with product group post that if needed for any possible doc update.

Thank you

Hi again @Manas Mohanty

• The username that was used to create the custom avatar is "owner" on all the resources
• All resources are set to "all networks"

I don't know where to check/set contributor roles you have mentioned: please advise or point me to the docs to check/set this. With "all networks" & "owner", it should be covered already, right?

Regarding the code change: I shall await your response. This Proof of concept we are doing is critical for a large project.

Thanks again!

@Manas Mohanty : please also check if there is any missing code in https://github.com/Azure-Samples/cognitive-services-speech-sdk/tree/master/samples/js/node/web/avatar for a custom avatar. I think this is a Node implementation of the browser one, so i am guessing that it is missing here as well