Training Custom Vision model fails with unknown error

Question

Training Custom Vision model fails with unknown error

Niclas Schmidt 20

When I try to train a custom model in Vision Studio it always fails with an error message stating: "Training failed unexpectedly. Please contact us for support and investigation."

Things I have already tried without success:

using a different dataset
using different training images
retraining a model with data that has previously trained without issues
using a different resource
using a different server location

santoshkc 15,590 Reputation points Microsoft External Staff Moderator

2025-04-17T14:26:27.7566667+00:00

Hi @Niclas Schmidt,

I have attempted to reproduce the issue on my end, but the training completes successfully, which indicates that the problem may be intermittent or specific to your environment. This could potentially be due to a temporary network or server-side issue. In such cases, we recommend trying again after some time to see if the issue resolves on its own.

Additionally, please try clearing your browser cache and cookies, and consider using a different browser or switching to an incognito/private window, as these steps can often help bypass local browser-related issues. If the problem persists, it may help to create a new Custom Vision project from scratch and train it using a smaller subset of your data. If the issue continues, please share the Region you are working. Thank you.
Niclas Schmidt 20 Reputation points

2025-04-17T17:49:50.1333333+00:00
I have tried the following steps you suggested, but the error message still appears:

I used an inkognito tab in a different browser

I created a new Vision Project and Storage account

I trained it using a smaller subset of our data

It is also not a temporary issue, as this error has been appearing for multiple weeks

I am currently using the europe-west region but I have also tried the region us-east in the past
santoshkc 15,590 Reputation points Microsoft External Staff Moderator

2025-04-18T13:06:20.47+00:00

Hi @Niclas Schmidt,

It’s possible that corrupt data could be causing this. Could you try zipping your dataset and testing it again from your side? If the issue persists, please capture a "HAR trace" while reproducing the error in Vision Studio.

Thank you.
Niclas Schmidt 20 Reputation points

2025-04-18T14:13:48.43+00:00

I don't think its corrupt data, as i have tried it with completely independent datasets

Here is the HAR trace:

portal.vision.cognitive.azure.com.har.txt
santoshkc 15,590 Reputation points Microsoft External Staff Moderator

2025-04-21T13:32:22.1066667+00:00

Hi @Niclas Schmidt,

The issue you're facing with training a Custom Vision model failing unexpectedly due to the image size exceeding the supported limit. If any of the training images are larger than 6MB, it can cause the training to fail with a generic error. To resolve this, please ensure that all images used for training are under 6MB in size. This limitation applies regardless of the dataset, training images, or region being used.

For more details on supported limits and quotas, you can refer to the official documentation here: Custom Vision limits and quotas.

I hope you understand! Thank you.
Niclas Schmidt 20 Reputation points

2025-04-21T17:23:10.0266667+00:00

I have tried compressing all images to be smaller than 6MB but the error still persists
santoshkc 15,590 Reputation points Microsoft External Staff Moderator

2025-04-24T14:50:31.5233333+00:00

Hi @Niclas Schmidt,

Just checking in to see if you had a chance to review the information provided by @Manas Mohanty regarding the deprecation of the custom model feature from portal.vision.cognitive.azure.com. And, if you have any further query do let us know.

Thank you.
santoshkc 15,590 Reputation points Microsoft External Staff Moderator

2025-04-25T09:52:09.8733333+00:00

Hi @Niclas Schmidt,

Following up to see if the above response was helpful. And, if you have any further query do let us know. Thank you.
Niclas Schmidt 20 Reputation points

2025-04-25T10:49:05.3533333+00:00

Yes that seams to be the issue.

Are there any other alternatives to customvision.ai that can better match the performance and workflow of the deprecated feature?
Manas Mohanty 8,150 Reputation points Microsoft External Staff Moderator

2025-04-29T10:14:15.1833333+00:00

Hi Niclas Schmidt

We can also leverage AutoML or TensorFlow keras to label and train an image classification model and host it in Azure machine learning.

But you have to get familiar with AutoML framework / Keras frameworkon image classification.

Customvision.ai should be approachable if you are preferring UI based solution

Thank you.

Accepted answer

1 additional answer

Your answer

santoshkc 15,590 Reputation points Microsoft External Staff Moderator

2025-04-17T14:26:27.7566667+00:00

Hi @Niclas Schmidt,

I have attempted to reproduce the issue on my end, but the training completes successfully, which indicates that the problem may be intermittent or specific to your environment. This could potentially be due to a temporary network or server-side issue. In such cases, we recommend trying again after some time to see if the issue resolves on its own.

Additionally, please try clearing your browser cache and cookies, and consider using a different browser or switching to an incognito/private window, as these steps can often help bypass local browser-related issues. If the problem persists, it may help to create a new Custom Vision project from scratch and train it using a smaller subset of your data. If the issue continues, please share the Region you are working. Thank you.
Niclas Schmidt 20 Reputation points

2025-04-17T17:49:50.1333333+00:00

I have tried the following steps you suggested, but the error message still appears:

I used an inkognito tab in a different browser

I created a new Vision Project and Storage account

I trained it using a smaller subset of our data

It is also not a temporary issue, as this error has been appearing for multiple weeks

I am currently using the europe-west region but I have also tried the region us-east in the past
santoshkc 15,590 Reputation points Microsoft External Staff Moderator

2025-04-18T13:06:20.47+00:00

Hi @Niclas Schmidt,

It’s possible that corrupt data could be causing this. Could you try zipping your dataset and testing it again from your side? If the issue persists, please capture a "HAR trace" while reproducing the error in Vision Studio.

Thank you.
Niclas Schmidt 20 Reputation points

2025-04-18T14:13:48.43+00:00

I don't think its corrupt data, as i have tried it with completely independent datasets

Here is the HAR trace:

portal.vision.cognitive.azure.com.har.txt
santoshkc 15,590 Reputation points Microsoft External Staff Moderator

2025-04-21T13:32:22.1066667+00:00

Hi @Niclas Schmidt,

The issue you're facing with training a Custom Vision model failing unexpectedly due to the image size exceeding the supported limit. If any of the training images are larger than 6MB, it can cause the training to fail with a generic error. To resolve this, please ensure that all images used for training are under 6MB in size. This limitation applies regardless of the dataset, training images, or region being used.

For more details on supported limits and quotas, you can refer to the official documentation here: Custom Vision limits and quotas.

I hope you understand! Thank you.
Niclas Schmidt 20 Reputation points

2025-04-21T17:23:10.0266667+00:00

I have tried compressing all images to be smaller than 6MB but the error still persists
santoshkc 15,590 Reputation points Microsoft External Staff Moderator

2025-04-24T14:50:31.5233333+00:00

Hi @Niclas Schmidt,

Just checking in to see if you had a chance to review the information provided by @Manas Mohanty regarding the deprecation of the custom model feature from portal.vision.cognitive.azure.com. And, if you have any further query do let us know.

Thank you.
santoshkc 15,590 Reputation points Microsoft External Staff Moderator

2025-04-25T09:52:09.8733333+00:00

Hi @Niclas Schmidt,

Following up to see if the above response was helpful. And, if you have any further query do let us know. Thank you.
Niclas Schmidt 20 Reputation points

2025-04-25T10:49:05.3533333+00:00

Yes that seams to be the issue.

Are there any other alternatives to customvision.ai that can better match the performance and workflow of the deprecated feature?
Manas Mohanty 8,150 Reputation points Microsoft External Staff Moderator

2025-04-29T10:14:15.1833333+00:00

Hi Niclas Schmidt

We can also leverage AutoML or TensorFlow keras to label and train an image classification model and host it in Azure machine learning.

But you have to get familiar with AutoML framework / Keras frameworkon image classification.

Customvision.ai should be approachable if you are preferring UI based solution

Thank you.

Answer 1

Hi Niclas Schmidt

Thank you for clarifying that you are using custom model feature from

portal.vision.cognitive.azure.com

Not from

www.customvision.ai

Wanted to quote this from custom model documentation

This Customizing model feature is now deprecated. On March 31, 2025, Azure AI Image Analysis 4.0 Custom Image Classification, Custom Object Detection, and Product Recognition preview API will be retired. After this date, API calls to these services will fail.

Screenshot (109)

Please use custom vision portal only for model training.

Sorry for the inconvenience.

Thank you.

Answer 2

Hi Niclas Schmidt, in particular after you've attempted all the no-brainer solutions swapping out datasets, images, even resources and locations… come on, azure, work with us here

alright, let's break that down. first of all, that error "Training failed unexpectedly" is a bit ambiguous, but it generally appears if the service encounters a hitch that it didn't anticipate. it could be a backend glitch, or possibly there's some kind of weird data formatting issue even if it used to work. lets see what you can do to try to resolve it

check your image tags. at times, the custom vision service can become finicky if the tags are not uniform or are not present. ensure that each image has a tag, and the spelling of the tag is uniform throughout. microsoft's docs discuss that here.

resize those images. if ur uploads are enormous (like 4K+ resolution), the service could slow down. see if scaling them down to under 6MB makes a difference. the docs are recommending keeping them between 256x256 to 1024x1024 pixels here.

region does indeed matter. you mentioned you tested different locations, but make sure ur custom vision resource and ur storage (in case ur utilizing one of those) are in the same region. getting them mixed up can lead to silent fails.

quota limits. Sneaky one if ur subscription crossed over into one of those obscure limits (e.g. api calls or training time), the service could just… give up. take a look at ur azure metrics to exclude that.

occasional glitch. azure is big, but it's not perfect. the service just burps from time to time. wait sixty minutes, retry, and pray to the cloud gods

So, hope that will help, at least i try to help

Best regards,
Alex
P.S. If my answer help to you, please Accept my answer
PPS That is my Answer and not a Comment
https://ctrlaltdel.blog/

Share via

Training Custom Vision model fails with unknown error

1 additional answer

Your answer