Azure Managed Grafana DNS Resolution Troubleshooting

Hello Rui Silva,

This DNS resolution failure (lookup api.loganalytics.io on 127.0.0.11:53: no such host) is a common issue with Azure Managed Grafana that typically stems from network connectivity or private endpoint configuration problems. Here's a comprehensive troubleshooting approach:

Immediate Diagnostic Steps

1. Check Network Configuration

First, verify your Grafana instance's network setup:

# Check if your Grafana instance is using private endpoints

az grafana show --name <grafana-instance-name> --resource-group <resource-group> --query "properties.publicNetworkAccess"

2. Validate Log Analytics Workspace Connectivity

# Test workspace accessibility

az monitor log-analytics workspace show --workspace-name <workspace-name> --resource-group <resource-group>

# Check workspace network access configuration

az monitor log-analytics workspace show --workspace-name <workspace-name> --resource-group <resource-group> --query "properties.publicNetworkAccessForIngestion"

DNS Resolution Troubleshooting

For Private Endpoint Scenarios:

1. Verify Private Endpoint Configuration:

# List private endpoints for your Log Analytics workspace

az network private-endpoint list --resource-group <resource-group> --query "[?contains(privateLinkServiceConnections[0].privateLinkServiceId, 'Microsoft.OperationalInsights')]"

2. Check Private DNS Zone:

# Verify private DNS zone exists for Log Analytics

az network private-dns zone list --resource-group <resource-group> --query "[?name=='privatelink.ods.opinsights.azure.com']"

3. Validate DNS Resolution:

# From a VM in the same VNet, test DNS resolution

nslookup api.loganalytics.io

nslookup <workspace-id>.ods.opinsights.azure.com

For Public Endpoint Scenarios:

1. Check Public Network Access:

# Ensure public access is enabled for Log Analytics workspace

az monitor log-analytics workspace update --workspace-name <workspace-name> --resource-group <resource-group> --public-network-access-for-ingestion Enabled --public-network-access-for-query Enabled

Step-by-Step Resolution

Option 1: Private Endpoint Fix

1. Recreate Private Endpoint Connection:

# Delete existing private endpoint (if problematic)

az network private-endpoint delete --name <private-endpoint-name> --resource-group <resource-group>

# Create new private endpoint

az network private-endpoint create \

  --name <new-private-endpoint-name> \

  --resource-group <resource-group> \

  --vnet-name <vnet-name> \

  --subnet <subnet-name> \

  --private-connection-resource-id "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.OperationalInsights/workspaces/<workspace-name>" \

  --group-ids opinsights \

  --connection-name <connection-name>

2. Update Private DNS Zone:

# Link private DNS zone to VNet

az network private-dns link vnet create \

  --resource-group <resource-group> \

  --zone-name privatelink.ods.opinsights.azure.com \

  --name <link-name> \

  --virtual-network <vnet-name> \

  --registration-enabled false

Option 2: Switch to Public Endpoint (Temporary Workaround)

# Enable public access as temporary workaround

az monitor log-analytics workspace update \

  --workspace-name <workspace-name> \

  --resource-group <resource-group> \

  --public-network-access-for-ingestion Enabled \

  --public-network-access-for-query Enabled

Azure Managed Grafana Specific Checks

1. Restart Grafana Instance:

az grafana restart --name <grafana-instance-name> --resource-group <resource-group>

2. Check Managed Identity Permissions:

# Verify Grafana has proper permissions to Log Analytics

az role assignment list --assignee <grafana-managed-identity-id> --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.OperationalInsights/workspaces/<workspace-name>"

3. Validate Data Source Configuration:
   • Go to Grafana → Configuration → Data Sources
   • Re-test connection for Azure Monitor data source
   • If needed, delete and recreate the data source

Advanced Diagnostics

Network Trace Analysis:

# Enable network monitoring (if available)

az network watcher packet-capture create \

  --name <capture-name> \

  --resource-group <resource-group> \

  --vm <grafana-vm-name> \

  --storage-account <storage-account>

DNS Cache Flush:

# Clear DNS cache on the underlying infrastructure

# This may require Azure support ticket for managed service

Common Root Causes & Solutions

1. Private Endpoint Misconfiguration:
   • Solution: Recreate private endpoint with correct DNS settings
2. VNet Integration Issues:
   • Solution: Verify subnet delegation and network security groups
3. DNS Zone Conflicts:
   • Solution: Check for overlapping DNS zones or incorrect A records
4. Service Degradation:
   • Solution: Check Azure Service Health for Log Analytics issues

Monitoring & Prevention

1. Set up Alerts:

az monitor metrics alert create \

  --name "Grafana-DNS-Failure" \

  --resource-group <resource-group> \

  --scopes "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Dashboard/grafana/<grafana-name>" \

  --condition "count 'static' > 0" \

  --description "Alert on Grafana connectivity issues"

2. Regular Health Checks:
   • Implement automated testing of data source connectivity
   • Monitor DNS resolution from Grafana instance

If Issue Persists

1. Open Azure support ticket with:
   • Grafana instance resource ID
   • Exact error messages and timestamps
   • Network configuration details
   • Recent changes to environment
2. Check Azure Service Health dashboard for ongoing issues with:
   • Azure Managed Grafana
   • Azure Monitor / Log Analytics

The error you're seeing suggests the DNS resolution is failing at the Grafana infrastructure level, which often indicates private endpoint configuration issues or service-side problems that may require Azure support intervention.

Let me know if you need clarification on any of these steps or if the issue persists after trying these solutions.

Hello Rui Silva,

As you are currently seeing a DNS resolution issue in Azure Managed Grafana (v11) where dashboards using Azure Monitor and Log Analytics datasources failed with the error: dial tcp: lookup api.loganalytics.io on 127.0.0.11:53: no such host.

Grafana instance is not able to resolve the domain api.loganalytics.io, which is used to query Log Analytics data. Microsoft is in the process of transitioning from api.loganalytics.io to api.loganalytics.azure.com, but the .io endpoint remains supported for the foreseeable future.

You can refer the below document: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/api/access-api?tabs=rest

Can you please confirm the error across all dashboards is same?

Try switching to the newer endpoint api.loganalytics.azure.com if your Grafana configuration allows it.

Restart or re-deploy the Grafana workspace via Azure Portal by changing the pricing tier or reapply version upgrade and also use Azure Monitor Logs or Workbooks as a temporary workaround.

As per product team they looked into the recent activities of this vm4z-xxxx instance and noticed that you have-initiated request to create a Managed Private Endpoint (MPE) on 08/26. This action seems to be trying to connect the Azure Monitor DataSource privately, but it has been still in "pending" connection status. The DataSource owner needs to approve the new connection in order for the MPE to work properly.

For the more details around MPE configuration is available in the doc:  How to connect to a data source privately in Azure Managed Grafana | Microsoft Learn. 

Thanks,

Suchitra.