Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
Datababricks managed network - can I access firewalled storage account in classic compute?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 43%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDN%3C/text%3E%3C/svg%3E)
Dimitry Nechaev
20
Reputation points
I have a default Databricks setup with managed network and public IP.
I want it to talk to a storage account, that is firewalled. I can do this for serverless but not for a classic compute.
I've added a vnet in between and peered bricks to this network.
this vnet is peered back to bricks vnet
It has private endpoints to the storage account dfs / blob.
My peering vnet is whitelisted in the storage account firewall.
Unfortunately this does not work for classic compute.
dbutils.fs.ls("abfss://...dfs.core.windows.net/")
"This request is not authorized to perform this operation.", 403
Is it possible to peer classic compute and what did I do wrong?
Is there a way to make classic compute to talk to the
Azure Databricks
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 60%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Mahesh Kurva 6,850 Reputation points Microsoft External Staff Moderator2025-08-08T06:19:35.22+00:00 Hi Dimitry Nechaev,
Greetings!!
Could you please give us some time to gather more information regarding your issue?
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 50%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
Venkat Reddy Navari 5,255 Reputation points Microsoft External Staff Moderator2025-08-08T10:02:47.6766667+00:00 Hi Dimitry Nechaev Based on your setup a Databricks workspace with a managed VNet, peered to another VNet that hosts private endpoints for a firewalled storage account it’s clear why your serverless clusters work, but classic compute returns:
"This request is not authorized to perform this operation." (403)Classic compute clusters run inside the Databricks-managed VNet. Even though that VNet is peered with your own (which is whitelisted and has private endpoints), Azure Storage evaluates access based on the originating VNet, not peering relationships. Therefore:
- Requests from classic compute originate from the managed VNet, which is not authorized in the storage firewall.
- If private DNS zones for
dfs.core.windows.net/blob.core.windows.netare not correctly linked, the cluster may resolve to public endpoints, which are also blocked.
To enable classic compute access:
- Add the Databricks-managed VNet to the storage firewall allow list.
- Ensure your Private DNS Zones are linked to the Databricks-managed VNet so name resolution points to private endpoints.
- Optionally, consider:
- Deploying classic compute into your peering VNet, or
- Using a NAT Gateway or forced tunneling from the peered VNet to centralize egress.
Could you please clarify:
- Is the Databricks-managed VNet explicitly added to the storage account firewall?
- Are the Private DNS Zones correctly linked to the managed VNet?
These will help confirm whether it’s a DNS resolution issue or a network origin mismatch.
I hope this information helps. Please do let us know if you have any further queries.
Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
-
Dimitry Nechaev 20 Reputation points
2025-08-11T02:20:55.5+00:00 - Is the Databricks-managed VNet explicitly added to the storage account firewall? This confuses me. You CANNOT add databricks managed network to the firewall.
- Are the Private DNS Zones correctly linked to the managed VNet? I CANNOT link private DNS zone to the databricks managed VNet.
I cannot touch databricks managed vnets in no way I tried.
Example adding to firewall:
In the end databricks classic compute resolves public IP of the storage account ignoring all peering networks etc. So peering is just wasted. What can I do in this situation?
-
Venkat Reddy Navari 5,255 Reputation points Microsoft External Staff Moderator
2025-08-11T10:26:58.34+00:00 Dimitry Nechaev Thanks for clarifying, you’re right that Databricks-managed VNets are fully locked down, so you can’t directly add them to a storage account firewall or link private DNS zones. That’s why your classic compute traffic is still resolving to the public endpoint despite the peering.
Unfortunately, with the managed network model, there’s no way to route classic compute through your peered VNet to hit the private endpoint. This limitation is by design only serverless or workloads in your own VNet can leverage that private connectivity.
If private access from classic compute is a must, your options are:
- Switch to customer-managed VNet (VNet injection) for the workspace, which you control and can whitelist in the storage firewall.
- Use serverless compute for jobs needing firewalled access.
- Move the data to a location accessible from the managed VNet (e.g., public endpoint with SAS/Managed Identity and firewall exceptions).
Right now, with a default managed VNet workspace, there’s no supported way to make classic compute talk to a private endpoint via peering.
I hope this information helps. Please do let us know if you have any further queries.
Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Sign in to comment
Sign in to answer