Edit

Share via

Monitor AI Agents with the Agent Monitoring Dashboard (preview)

Important

Items marked (preview) in this article are currently in public preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

The Agent Monitoring Dashboard in Microsoft Foundry provides real-time insights into the operational health, performance, and compliance of your AI agents. Use this dashboard to track token usage, latency, evaluation metrics, and security posture across multi-agent systems.

Prerequisites

  • A Foundry project. For more information, see Create a Foundry project.
  • At least one deployed agent in your Foundry project.
  • An Application Insights resource connected to your project.
  • Azure role-based access control (RBAC): At minimum, "Reader" role on the Application Insights resource to view monitoring data.

Grant Managed Identity Access

To allow your Azure Foundry project to authenticate and interact with Azure AI resources, you must grant its system-assigned managed identity the Azure AI User role

  1. Open the Foundry Project Resource: In the Azure portal, navigate to the Azure resource associated with your Foundry project (for example, the project’s resource group or a specific service resource).
  2. Go to Access Control (IAM): In the left-hand menu, select Access control (IAM).
  3. Add a New Role Assignment.
  4. Select the Azure AI User Role: In the Role dropdown, search for Azure AI User, then select it.
  5. Choose the Managed Identity
    1. Under Assign access to, choose Managed identity.
    2. Select the system-assigned managed identity for your Foundry project from the list.
  6. Review and Assign.

Once assigned, the project’s managed identity will have the necessary permissions to access and use Foundry Tools by default.

View agent metrics in the portal

To view metrics for your agent in the Foundry portal:

  1. Sign in to Microsoft Foundry. Make sure the New Foundry toggle is on. These steps refer to Foundry (new).

  2. Navigate to the Build page using the top navigation and select the agent you'd like to view data for.

  3. Select the Monitor tab to view operational, evaluation, and red-teaming data for your agent.

View agent metrics

Screenshot of the Agent Monitoring Dashboard in Foundry showing summary cards at the top with high-level metrics and charts below displaying evaluation scores, agent run success rates, and token usage over time.

The Agent Monitoring Dashboard in Foundry is designed for quick insights and deep analysis of your AI agents' performance. It consists of two main areas:

  • Summary Cards at the top for high-level metrics. These cards provide an at-a-glance view of key metrics.

  • Charts and graphs below for granular details. These visualizations reflect data for the selected time range. Use them to view key metrics including evaluation scores by day, agent run success rates, and token usage.

Configure settings

The Monitor Settings panel allows you to enable and customize telemetry, evaluations, and security checks for your AI agents. These settings ensure that the dashboard displays accurate operational and quality metrics.

Screenshot showing the Monitor Settings panel in Foundry with options for operational metrics, continuous evaluation, scheduled evaluations, red team scans, and alerts configuration.

The following table describes the monitoring features available in the Monitor Settings panel:

Setting Purpose Configuration Options
Continuous Evaluation Runs real-time checks on agent responses for intent resolution, coherence, and reconciliation. Enable/Disable toggle
Add evaluators by name or import from past evaluations
Set sample rate (for example, 10 runs/hour)
Scheduled Evaluations Performs periodic evaluations to validate agent performance against benchmarks. Enable/Disable toggle
Select evaluation template
Select evaluation run
Set schedule frequency (weekly recommended)
Red Team Scans Executes adversarial tests to identify vulnerabilities such as sensitive data leakage or prohibited actions. Enable/Disable toggle
Select evaluation template
Select evaluation run
Set schedule frequency (weekly recommended)
Alerts Monitors for performance anomalies, evaluation failures, and security risks. Integrates with Azure Monitor for automated notifications. Configure alerts for:
- Performance anomalies (latency spikes, token overuse)
- Evaluation failures (low coherence scores)
- Security risks detected during red-teaming

Next steps

  • Last updated on