Grafana can’t access Azure managed Prometheus through private endpoint

Question

Grafana can’t access Azure managed Prometheus through private endpoint

Venkatesh D 0

We have set up Azure Managed Grafana and Azure Managed Prometheus with private endpoints and have disabled public access for both. DNS is also configured to resolve through private link, and both services are in the same VNet.

In Grafana, we’ve added two data sources:

Azure Monitor – works fine and shows data.

Prometheus – not working, and shows this error:

"Dashboards with Grafana: Error – Access through public network is disabled for this resource."

we have created a Managed Private Endpoint from Grafana to Prometheus and approved. Still, Prometheus in Grafana is not connecting. It seems Grafana is trying to use the public network, even though it's disabled.

Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-05-14T07:15:37.9766667+00:00
Hi @Venkatesh D,

Azure Managed Grafana is still trying to reach Prometheus via the public endpoint, despite both being in the same VNet with private endpoints configured.

This usually happens because Grafana is using the default (public) endpoint for Prometheus instead of the private DNS name mapped to the private endpoint.

To resolve this, follow these steps:

Step 1: Get the Private DNS Name for Prometheus You need the correct Private Link endpoint. It will look like:

https://<your-prometheus-name>.privatelink.monitor.azure.com

If you’re unsure:

1.Go to Azure Portal > Azure Monitor Workspace > Endpoints > Private Endpoint connections.

2.Note the DNS name or IP.

3.Ensure your DNS setup resolves this name inside the VNet to the private IP.

Step 2: Update Prometheus Data Source in Grafana

1.In Azure Managed Grafana, go to: Configuration > Data Sources > Prometheus

2.Under URL, replace the existing value with:

https://<your-prometheus-name>.privatelink.monitor.azure.com

3.DO NOT check "Skip TLS verification" unless you're testing (not recommended in production).

4.Save & Test — this should now connect privately.

Reference:

https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/prometheus-grafana?utm_source=chatgpt.com&tabs=azure-managed-grafana

https://learn.microsoft.com/en-us/azure/managed-grafana/how-to-connect-to-data-source-privately

Let us know if you need any help — we're always here to support you whenever you need it!

Please let me know the outcome of it.
Venkatesh D 0 Reputation points

2025-05-14T09:30:50.7966667+00:00

Hi @Ashok Gandhi Kotnana Thank you for your response on this question. We have followed the steps which you provided in the comment and added the private link URL in datasource, but we are facing the issue as below, saying that certificate is not valid for the domain:

are we having the issue with our custom private DNS zone?
Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-05-14T11:50:45.3133333+00:00
Hi @Venkatesh D,

You're encountering a TLS certificate verification error:

x509: certificate is NOT valid for privatelink.eastus2.prometheus.monitor.azure.com Certificate is only valid for: - *.eastus2.data.monitor.azure.com - *.metrics.monitor.azure.com - *.eastus2.prometheus.monitor.azure.com

Azure Monitor’s Prometheus endpoint uses a TLS certificate that’s valid only for its public fully qualified domain name (FQDN), even when accessed through a private IP via Private Link.

This means that although your connection is private, the certificate is only valid for the public hostname, such as

https://<your-workspace-name>.eastus2.prometheus.monitor.azure.com

To resolve this, make sure you configure Grafana to use the public FQDN for the Prometheus

data source: https://<your-monitor-workspace-name>.eastus2.prometheus.monitor.azure.com

However, this public FQDN must resolve to the private IP address of your Private Endpoint — not the public IP.

Azure Private Link secures the network path but does not affect TLS certificate validation — certificates must still match the public domain name.

What needs to happen:

Access https://<your-workspace>.eastus2.prometheus.monitor.azure.com

Ensure DNS resolves that FQDN to your Private Endpoint’s private IP

The TLS certificate will then be valid

Traffic will stay entirely within your VNet

To make the public FQDN resolve to the private IP, do the following:

1.Create or use a Private DNS zone:(Azure might create this automatically)

privatelink.eastus2.prometheus.monitor.azure.com

2.Add a custom A record:

Name: <your-workspace>

Zone: privatelink.eastus2.prometheus.monitor.azure.com

IP Address: Private IP of your Private Endpoint

3. Link the Private DNS zone to the VNet used by Managed Grafana. 4. Test DNS resolution: Run: nslookup <your-workspace>.eastus2.prometheus.monitor.azure.com

This should return the private IP of your Private Endpoint.

Option 2: Skip TLS Verification (Not Recommended for Production)

If you're in a non-production environment and just want to verify connectivity:

In Grafana, go to the Prometheus data source settings

Check "Skip TLS verification"

⚠️ Not recommended unless you're doing temporary testing.

After doing this if you are still facing an issue, please let me know
Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-05-15T07:30:32.4166667+00:00

Hi @Venkatesh D,

Just want to check if the above answer worked for you or else please let us know if any help, we are always here to help whenever you need us.

Thankyou
Venkatesh D 0 Reputation points

2025-05-15T15:22:12.6533333+00:00

Hi Ashok,

Thank you for your response.

I’ve followed the instructions you provided and also added a custom A record pointing to the private IP of our Private Endpoint and confirmed that the DNS resolution is working correctly — nslookup resolves the FQDN to the expected private IP.

However, we are still encountering the same issue as before.
Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-05-16T09:47:09.86+00:00

Hi @Venkatesh D

I have reached out to you in a private message
Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-05-19T11:12:34.8133333+00:00

HI @Venkatesh D

Can you please check your AKS can be able to send the logs to Log analytics/Grafana

Reference:

https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-log-query

https://grafana.com/grafana/dashboards/20811-aks-container-logs/

https://alexisplantin.fr/aks-prometheus-grafana/

Let me check and let me know on this request

Thankyou
Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-05-20T06:33:38.05+00:00

HI @Venkatesh D

Please let me know if you have checked my above response.

Thankyou
Lewis, Adam 0 Reputation points

2025-05-20T18:21:22.63+00:00

I am having the same issue
Anonymous

2025-05-27T12:46:28.7733333+00:00

Hi Venkatesh D

When using azure managed Prometheus with azure managed Grafana and accessing Prometheus through a private endpoint you may hit a TLS error because of certificate mismatch.

The issue happens because Prometheus only provides TLS certificate for its public endpoint domain like > *.eastus2.prometheus.monitor.azure.com but you're accessing it through a private endpoint the actual output with private DNS name like > *.privatelink.monitor.azure.com which cause grafana fail the TLS handshake since the certificate doesn't match the private domain name

To fix:

Use the public endpoint url in grafana even through your using a private network you need to configure the prometheus data source in grafana to use the public FQDN azure will rout this through the private endpoint, so it stays on the private network, no internet access involved.

You can find your prometheus public endpoint by going into azure monitor workspace in the azure portal look for the prometheus endpoint it should look > https://<workspace>.eastus2.prometheus.monitor.azure.com

Later update the grafana data source in your azure managed grafana instance by checking configuration > data sources > prometheus then ste the url to the public endpoint you already found it during prometheus endpoint make sure skip TLS verification is NOT checked keep it as secure then svae & test to verify the connection.

Grafana will use the public name so the TLS certificate is valid, but the traffic will go through the private endpoint if everything is set up correctly in your VNet.

Please let us know if this was helpful. If it was, feel free to upvote it to assist others in the community as well.
Lewis, Adam 0 Reputation points

2025-05-27T13:25:10.64+00:00

I've got the DNS correct and my Prometheus and Grafana screens are working correctly. I would like to just let you know (and I'm not expecting this to work yet as it is still preview) that the Dashboards with Grafana (preview) option from the portal still tries to access Grafana through the public URL as shown in the screenshot below. Please feed this back to your development team as it would be nice to use this new convenient feature from the portal while the Prometheus / Garfana is on private links.
Anonymous

2025-05-28T10:30:49.2666667+00:00

Hi Lewis, Adam

You won't be able to access the Grafana UI directly through the portal dashboards with Grafana (preview) because the button tries to open Grafana using its public URL. instead of this approach access the Grafana UI using the private URL or Ip address directly. the portal shortcut doesn't work with private links yet, but you can still use Grafana normally by going to its private address manually.

Like this:
Anonymous

2025-05-29T04:06:57.52+00:00

Hi Venkatesh D Lewis, Adam

Just checking if the provided workaround resolved your issue. If it did, please let us know. If you have any questions or concerns, please respond, and we will take further steps.

Let us know if this was helpful. If it was, feel free to upvote it to assist others in the community as well.

-Thank you.
Anonymous

2025-06-02T09:17:03.3366667+00:00

Hi Venkatesh D,

We haven’t heard from you on the last response, and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, please respond with more details and we will try to help.

1 answer

Your answer

Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-05-14T07:15:37.9766667+00:00

Hi @Venkatesh D,

Azure Managed Grafana is still trying to reach Prometheus via the public endpoint, despite both being in the same VNet with private endpoints configured.

This usually happens because Grafana is using the default (public) endpoint for Prometheus instead of the private DNS name mapped to the private endpoint.

To resolve this, follow these steps:

Step 1: Get the Private DNS Name for Prometheus You need the correct Private Link endpoint. It will look like:

https://<your-prometheus-name>.privatelink.monitor.azure.com

If you’re unsure:

1.Go to Azure Portal > Azure Monitor Workspace > Endpoints > Private Endpoint connections.

2.Note the DNS name or IP.

3.Ensure your DNS setup resolves this name inside the VNet to the private IP.

Step 2: Update Prometheus Data Source in Grafana

1.In Azure Managed Grafana, go to: Configuration > Data Sources > Prometheus

2.Under URL, replace the existing value with:

https://<your-prometheus-name>.privatelink.monitor.azure.com

3.DO NOT check "Skip TLS verification" unless you're testing (not recommended in production).

4.Save & Test — this should now connect privately.

Reference:

https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/prometheus-grafana?utm_source=chatgpt.com&tabs=azure-managed-grafana

https://learn.microsoft.com/en-us/azure/managed-grafana/how-to-connect-to-data-source-privately

Let us know if you need any help — we're always here to support you whenever you need it!

Please let me know the outcome of it.
Venkatesh D 0 Reputation points

2025-05-14T09:30:50.7966667+00:00

Hi @Ashok Gandhi Kotnana Thank you for your response on this question. We have followed the steps which you provided in the comment and added the private link URL in datasource, but we are facing the issue as below, saying that certificate is not valid for the domain:

are we having the issue with our custom private DNS zone?
Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-05-15T07:30:32.4166667+00:00

Hi @Venkatesh D,

Just want to check if the above answer worked for you or else please let us know if any help, we are always here to help whenever you need us.

Thankyou
Venkatesh D 0 Reputation points

2025-05-15T15:22:12.6533333+00:00

Hi Ashok,

Thank you for your response.

I’ve followed the instructions you provided and also added a custom A record pointing to the private IP of our Private Endpoint and confirmed that the DNS resolution is working correctly — nslookup resolves the FQDN to the expected private IP.

However, we are still encountering the same issue as before.
Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-05-16T09:47:09.86+00:00

Hi @Venkatesh D

I have reached out to you in a private message
Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-05-19T11:12:34.8133333+00:00

HI @Venkatesh D

Can you please check your AKS can be able to send the logs to Log analytics/Grafana

Reference:

https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-log-query

https://grafana.com/grafana/dashboards/20811-aks-container-logs/

https://alexisplantin.fr/aks-prometheus-grafana/

Let me check and let me know on this request

Thankyou
Ashok Gandhi Kotnana 10,430 Reputation points Microsoft External Staff Moderator

2025-05-20T06:33:38.05+00:00

HI @Venkatesh D

Please let me know if you have checked my above response.

Thankyou
Lewis, Adam 0 Reputation points

2025-05-20T18:21:22.63+00:00

I am having the same issue
Anonymous

2025-05-27T12:46:28.7733333+00:00

Hi Venkatesh D

When using azure managed Prometheus with azure managed Grafana and accessing Prometheus through a private endpoint you may hit a TLS error because of certificate mismatch.

The issue happens because Prometheus only provides TLS certificate for its public endpoint domain like > *.eastus2.prometheus.monitor.azure.com but you're accessing it through a private endpoint the actual output with private DNS name like > *.privatelink.monitor.azure.com which cause grafana fail the TLS handshake since the certificate doesn't match the private domain name

To fix:

Use the public endpoint url in grafana even through your using a private network you need to configure the prometheus data source in grafana to use the public FQDN azure will rout this through the private endpoint, so it stays on the private network, no internet access involved.

You can find your prometheus public endpoint by going into azure monitor workspace in the azure portal look for the prometheus endpoint it should look > https://<workspace>.eastus2.prometheus.monitor.azure.com

Later update the grafana data source in your azure managed grafana instance by checking configuration > data sources > prometheus then ste the url to the public endpoint you already found it during prometheus endpoint make sure skip TLS verification is NOT checked keep it as secure then svae & test to verify the connection.

Grafana will use the public name so the TLS certificate is valid, but the traffic will go through the private endpoint if everything is set up correctly in your VNet.

Please let us know if this was helpful. If it was, feel free to upvote it to assist others in the community as well.
Lewis, Adam 0 Reputation points

2025-05-27T13:25:10.64+00:00

I've got the DNS correct and my Prometheus and Grafana screens are working correctly. I would like to just let you know (and I'm not expecting this to work yet as it is still preview) that the Dashboards with Grafana (preview) option from the portal still tries to access Grafana through the public URL as shown in the screenshot below. Please feed this back to your development team as it would be nice to use this new convenient feature from the portal while the Prometheus / Garfana is on private links.
Anonymous

2025-05-28T10:30:49.2666667+00:00

Hi Lewis, Adam

You won't be able to access the Grafana UI directly through the portal dashboards with Grafana (preview) because the button tries to open Grafana using its public URL. instead of this approach access the Grafana UI using the private URL or Ip address directly. the portal shortcut doesn't work with private links yet, but you can still use Grafana normally by going to its private address manually.

Like this:
Anonymous

2025-05-29T04:06:57.52+00:00

Hi Venkatesh D Lewis, Adam

Just checking if the provided workaround resolved your issue. If it did, please let us know. If you have any questions or concerns, please respond, and we will take further steps.

Let us know if this was helpful. If it was, feel free to upvote it to assist others in the community as well.

-Thank you.
Anonymous

2025-06-02T09:17:03.3366667+00:00

Hi Venkatesh D,

We haven’t heard from you on the last response, and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, please respond with more details and we will try to help.

Answer 1

Hi Venkatesh D

The issue arises when Azure Managed Grafana cannot access Azure Managed Prometheus through a private endpoint due to TLS certificate validation failures. This typically occurs when both services are configured with private endpoints and public access is disabled.

Root Cause:

Azure Managed Grafana attempts to connect to Azure Managed Prometheus using the private DNS name > https://<your-prometheus-name>.privatelink.monitor.azure.com. However, the TLS certificate presented by Azure Managed Prometheus is valid only for its public FQDN > *.eastus2.prometheus.monitor.azure.com. This mismatch leads to certificate validation errors.

To fix:

To resolve this issue, configure Grafana to use the public FQDN of Azure Managed Prometheus while ensuring that this FQDN resolves to the private IP address via your private DNS zone.

Update Grafana Data Source:

In Azure Managed Grafana, navigate to > Configuration > Data Sources > Prometheus

Set the URL to the public FQDN of your Azure Monitor workspaces Prometheus endpoint,

Like this: https://<your-monitor-workspace-name>.eastus2.prometheus.monitor.azure.com

Ensure that the "Skip TLS verification" option is not enabled.

Configure Private DNS zone creates or update a private DNS zone to resolve the public FQDN <your-monitor-workspace-name>.eastus2.prometheus.monitor.azure.com to the private IP address of the Azure Managed Prometheus endpoint.

Link this private DNS zone to the virtual network where Azure Managed Grafana resides.

Verify connectivity that the DNS resolution is correctly mapping the public FQDN to the private IP address.

Test the connection in Grafana to confirm that the data source is now accessible.

Additional Considerations:

Disabling TLS verification is not recommended in production environments as it compromises security. and using the private FQDN privatelink.monitor.azure.com directly in Grafana will result in TLS certificate mismatches, as the certificate is valid only for the public FQDN.

Confirm that the managed identity associated with your Azure Managed Grafana workspace has the necessary permissions like Monitoring Data Reader role to access the Azure Monitor workspace.

Doc's:

https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/prometheus-grafana?tabs=azure-managed-grafana

https://learn.microsoft.com/en-us/azure/managed-grafana/how-to-connect-azure-monitor-workspace

https://learn.microsoft.com/en-us/azure/private-link/troubleshoot-private-endpoint-connectivity

By configuring Grafana to use the public FQDN for Azure Managed Prometheus and ensuring that this FQDN resolves to the private IP via your private DNS zone, you can maintain secure, private connectivity without encountering TLS certificate issues.

Please let us know if you feel the answer has not resolved your issue, and we will take further steps to address your concern.

-Thank you.

I hope this has been helpful! If above is unclear and/or you are unsure about something add a comment below.

Please click the answer as original posters help the community find answers faster by identifying the correct answer. User's image

Anonymous

2025-06-03T03:36:58.3133333+00:00

Hi Venkatesh D

Just checking if the provided answer resolved your issue. If you believe it did, please accept the workaround so that the information can benefit the community. If you have any questions or concerns, please let me know, and we will provide the necessary assistance.
Anonymous

2025-06-04T03:41:07.11+00:00

Hi Venkatesh D

We haven’t heard from you on the last response, and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, please respond with more details and we will try to help.

Share via

Grafana can’t access Azure managed Prometheus through private endpoint

1 answer

Your answer