ADX cluster 5 minutes latency on streaming ingestion

Hi @Azure Connect

Thank you for reaching out to Microsoft QA.

Hey there! It sounds like you're experiencing a frustrating 5-minute latency on your ADX cluster's streaming ingestion since November 23rd. This latency can happen for a few reasons, so let's explore what you can check and do to potentially resolve this issue.

Here are some steps you can follow:

Check the Ingestion Batching Policy: Since you're using streaming ingestion, it’s important to ensure that the batching policy is configured properly. The default batching time is set to 5 minutes unless adjusted. You can modify the batching policy at the table or database level to reduce latency:

• Alter Table Ingention Batching Policy
  • Alter Database Ingention Batching Policy
  Monitor CPU and Resource Utilization: High CPU utilization (over 80%) can slow down ingestion. Review the average CPU utilization over the past week:

  - If above 80%, consider scaling up the cluster for better performance: [Scale Up the Cluster](https://learn.microsoft.com/azure/data-explorer/manage-cluster-vertical-scaling)
  
     - If utilization is under 80% but high concurrency is an issue, consider changing query consistency: [Optimize for High Concurrency](https://learn.microsoft.com/azure/data-explorer/high-concurrency).
  
     **Ingestion Workload**: Analyze which commands and queries are dominating resources. You can run the following command to visualize command activity over the last 7 days:
  
     ```sql
     .show commands-and-queries
  

| where StartedOn >= ago(7d) | summarize count() by bin(StartedOn, 1h) | render timechart ```

        **Check for Ingestion Failures**:
        
           - Run the following to inspect any recent ingestion failures:
           
           ```haskell
           .show ingestion failures

| where FailedOn > ago(7d) | project ErrorCode, FailureKind, Details ```

           **Cluster Health**: Make sure your cluster is healthy by checking [Resource Health](https://learn.microsoft.com/azure/data-explorer/check-cluster-health) and monitoring metrics.
           

Follow-Up Questions:

To assist you better, could you provide additional details?

• Have you checked the ingestion batching policy settings recently?
• What is the current average CPU utilization on the cluster?
• Are there specific times or triggers when you noticed the latency spike, or is it consistent?
• Have you noticed any recent changes to your environment or workload that might correlate with the start of this latency issue?

Hope this helps you get to the bottom of your ingestion latency issues! Feel free to ask if you need further assistance or clarification on any of these steps.

References:

• Batching Policy Documentation
• Scaling Azure Data Explorer
• Optimize for High Concurrency
• Ingestion Failures Overview
• Monitor Cluster Health

Let me know if you need anything else!