Unable to start job runs and start cluster in Azure Databricks

Hello Kian Baula,

Thanks for your question on the Microsoft Q&A portal!

We understand you're facing issues while running jobs or clusters. This may be due to some restrictions on resources or service interruptions.

Here are a few things that typically lead to this kind of issues:

1.Too Many Requests at Once

If too many jobs or cluster requests are triggered in a short time, Databricks might block new ones temporarily to prevent overload.

2.Subscription Limits

Your Azure subscription has limits (like the number of CPUs or public IPs). If these are already in use, new clusters can’t be created.

3.Temporary Platform Issues

Sometimes, Azure or Databricks services might face short-term issues or maintenance in certain regions, which can affect job execution.

>>Could you please check the few following steps:

Check if Azure has any ongoing issues:

• Go to Azure Portal > Help + Support > Service Health
• See if there are any active incidents in your region. If yes, it's best to wait until they're resolved.

Please check if your Azure resource limits have been reached:

• You can do this by going to Azure Portal > Subscriptions > Usage + Quotas and reviewing resources like vCPUs and Public IPs to ensure they’re not fully used.
• If they are, you can request an increase right from the portal, ensure you have Owner or Quota Contributor role on the subscription to request quota increases.

Reduce Job/Cluster Request Frequency:

• If you’re using automated tools (like Data Factory), try not to run too many jobs at the same time. Wait at least 30 seconds between job requests.

Implement Cluster Pools:

• Cluster pools pre-allocate idle VMs, reducing API calls and cluster start time.
• Azure Portal > Workspace > Compute> Pools>Create Pool
• Set min and max idle instances
• Point your jobs to use the pool by selecting “Pool” when configuring a new job cluster
• Pools improve startup times but incur minimal idle-VM costs.

Please refer these Microsoft documents:

https://learn.microsoft.com/en-us/azure/quotas/per-vm-quota-requests

https://learn.microsoft.com/en-us/azure/databricks/compute/pools

Let me know if you have any further questions or need additional assistance. Also, if these answers your query, do click the "Upvote" and click "Accept the answer" of which might be beneficial to other community members reading this thread.

Thanks,

Kalyani

Hello Kian Baula,

Greetings for the day!

Just checking to see if you have a chance to check my previous response and helped, do let me know if you have any further questions on this.

If the provided information answers your query, do click "Upvote" and "Accept Answer", it will help others who might be facing similar challenges.

Thanks

Kalyani

Hi Kalyani,

Thanks for the detailed response! Unfortunately, none of these help towards the issue.

• No ongoing incidents. Actually, to add more details, we had been triggering these daily Databricks jobs from an Azure Durable Function. We had this ongoing for over a year now, but has suddenly stopped working on July 14. We have kept the daily trigger, but we are still receiving the same error. And even starting the job from the UI, we get the exact same error.
• Quotas have extra capacity for both the related vCPU type and public IPs.
• There are no jobs running right now (and there hasn't been for two weeks), and we cannot trigger any new run. It does not seem to be an issue related to too many jobs running.
• We do use cluster pools since we have jobs that happen sequentially. The cluster pool reuse helps with initialization time for each job. We cannot start any cluster (interactive or job) at the moment. We also cannot start any SQL warehouse.

Any further guidance would be much appreciated!

Hello Kian Baula, Hope you are having great day!

Thank you for your response. When you get a chance, could you also kindly check the following points?

Review Service Principal & Token Configuration

Since you’re triggering via Azure Durable Function:

• Check if the Service Principal's secret/token used to authenticate has expired or been revoked.
• Verify the SP still has the "Contributor" or "Owner" role on:
  • The Databricks workspace
    • The underlying resource group
      • VNet/Subnet (if VNet-injected workspace)

Validate Cluster Policies & Workspace Settings

Sometimes new policy enforcement may block cluster creation:

• Go to Admin Console > Compute Policies and check for any recent policy changes or constraints.
• If you use secure cluster connectivity or private networking, check for updates in IP Access List or network security groups (NSG), a change to an NSG rule in your VNet could be blocking the necessary traffic for Databricks to provision resources.

Thanks,

Kalyani

Hi Kalyani,

Thanks for your suggestions!

We tried renewing the token and also upgrading the SP to Owner, but that still returned the same error from the Durable Function. Same error as triggering new job runs or starting clusters from the UI as an admin as well: "Triggering new runs for organization [org_number] is currently disabled temporarily."

As for Compute Policies, there hasn't been any changes to the policies for any of the Compute types (Job, Legacy Shared Compute, Personal Compute).

For the NSG the cluster uses, we are using the default NSG rules with worker-to-worker access allowed. See screenshot. Nothing I see would block Databricks. There also hasn't been any changes in here for the last month.

Appreciate your input and hoping for more ideas.

Thanks,

Kian