Azure Data Explorer: LightIngest Managed Identity Authentication Issue

Question

Azure Data Explorer: LightIngest Managed Identity Authentication Issue

01725609 105

Hi all

Hoping somebody could help me out here.

We currently are trying to run LightIngest in a containerapp job in order to ingest massive amounts of data into adx, and do this with extra control mechanisms.

however, the authentication via managed identity seems to fail constantly (the ingestion part)

the visual below shows all permissions that have been assigned to the managed identity

User's image

However, for some reason i'm still getting the following error

Ingestion with Managed_Identity '' to database '*****'* is not allowed by policy.

My lightIngest command is like this

LightIngest "https://{ingesturi};Fed=true" -managedIdentity:"{clientId}" -connectToStorageWithManagedIdentity:"{clientId}" -ingestWithManagedIdentity:"{clientId}" -sourcePath:"commpletepath;managed_identity={objectid}" -database:"databasename" -table:"tablename" -format:"csv" -pattern:"*" -ingestionMappingRef:"mappingname" -ignoreFirstRow:True -tag:"" -creationTimePattern:"/'yyyy/MM/dd'/" -dontWait:True -interactive:False -listOnly:False

I've put the even put the managed identity policy at a cluster and database level, but I keep getting the same error.

Am i missing something? I cannot seem to locate the issue. the command works perfectly when running locally using storage account key or azure cli, so it has to do with the MI authentication

Thank you

1 answer

Your answer

Answer 1

Pratyush Vashistha 5,125 Microsoft External Staff Moderator

Hello 01725609,

Thank you for posting your question on the Microsoft Q&A portal and for providing the detailed architecture diagram. It is very helpful for understanding your setup.

I understand you are facing an "Ingestion not allowed by policy" error when using LightIngest with a User-Assigned Managed Identity in an Azure Container App, despite having configured the necessary permissions as shown in your diagram.

The error message Ingestion with Managed_Identity '' to database ... is not allowed by policy is a strong indicator of the root cause. The empty single quotes ('') suggest that the Azure Data Explorer (ADX) ingestion service is not receiving the identity's ID. This typically happens when the authentication token is not correctly requested or presented by the client application (LightIngest).

Based on your diagram and command, the permissions appear to be correctly set. The issue is most likely in the LightIngest command itself, specifically the connection string used to authenticate with ADX.

The most reliable way to authenticate with ADX using a managed identity is to specify the ADX cluster's resource ID in the connection string. This ensures the Kusto client library requests a token for the correct audience.

Action: Modify the connection string to include the mi_res_id parameter.
Find your ADX Cluster Resource ID: You can find this on the Properties page of your ADX cluster in the Azure portal. It will look like this: /subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.Kusto/clusters/{cluster-name}

Your new connection string should be: "https://{ingest-cluster-uri};Fed=true;mi_res_id={your-adx-cluster-resource-id}"

When running in an Azure environment like a Container App where the managed identity is already assigned, you often don't need to specify the identity's client ID in multiple command-line flags. The environment and the mi_res_id in the connection string provide enough context.

Action: Try simplifying your command by removing the explicit -managedIdentity, -connectToStorageWithManagedIdentity, and -ingestWithManagedIdentity flags. The tool should automatically pick up the assigned identity.
The -sourcePath argument must still contain the managed_identity={objectid} parameter, as this is used by the ADX service on the backend to access storage.

Here is a revised version of your command:

# Ensure you replace the placeholders with your actual values
$ingestUri = "https_ingestion_uri_of_your_cluster"
$adxResourceId = "/subscriptions/.../resourceGroups/.../providers/Microsoft.Kusto/clusters/..."
$miObjectId = "object_id_of_your_managed_identity"
$sourcePath = "https://youraccount.blob.core.windows.net/yourcontainer;managed_identity=" + $miObjectId
LightIngest "$ingestUri;Fed=true;mi_res_id=$adxResourceId" `
  -sourcePath:$sourcePath `
  -database:"databasename" `
  -table:"tablename" `
  -format:"csv" `
  -pattern:"*" `
  -ingestionMappingRef:"mappingname"

Your diagram indicates you have set the "ADX Managed Identity Policy". It is crucial to ensure the Object ID in this policy exactly matches the Object ID (also known as Principal ID) of your user-assigned managed identity.

Action: Run the following command in your ADX database to double-check the policy.

.show database YourDatabaseName policy managed_identity

Confirm that the ObjectId in the output matches the Object ID of your identity and that AllowedUsages includes NativeIngestion.

  .alter-merge database YourDatabaseName policy managed_identity ```json
  [
      {
          "ObjectId": "your-managed-identity-object-id",
          "AllowedUsages": "NativeIngestion"
      }
  ]
  ```*

To help us further if the issue persists, could you please confirm the following?

In your -sourcePath argument, are you using the Object (Principal) ID of the Managed Identity for the {objectid} placeholder, or are you using the Client ID? The Object ID is required here.
Could you share the output of the .show database YourDatabaseName policy managed_identity command (with the Object ID redacted if necessary)?

References

Review the different properties, including mi_res_id, for authenticating.
- Kusto connection strings - Azure Data Explorer
This guide provides a complete overview of the process.
- Ingest from storage into Azure Data Explorer using a managed identity
LightIngest Documentation: Use LightIngest to ingest data into Azure Data Explorer

I hope these steps help you resolve the authentication issue. Please let us know how it goes.

Please "Accept as Answer" if the answer provided is useful, so that you can help others in the community looking for remediation for similar issues.

Thanks

Pratyush

01725609 105 Reputation points

2025-10-09T06:11:29.5566667+00:00
Hi

Thank you for responding to my message.

I'm sorry for the confusion, but my managed identity object id was recognized by the service, I just removed it for security purposes.

so it actually was

Ingestion with Managed_Identity 'xxxx-xxxx-xxxx' to database 'xxxxxxx' is not allowed by policy.

however I am wondering on where you find this information?
Could you point me directly to it?

Because I haven't been able to find this in the documentation and when trying to run the command, I also get the following Error.

Key value 'mi_res_id' (indicating 'Keyword') was not found in collection 'KustoConnectionStringBuilder keywords'

Also, when you dig deeper in the Kusto Ingest SDK, Managed identity is not part of the 'KustoConnectionStringBuilder keywords'.

Also, when reading through the documentation, it also states the following:

https://learn.microsoft.com/en-us/kusto/api/connection-strings/kusto?view=azure-data-explorer&preserve-view=true#authenticate-with-a-user-assigned-managed-identity

But this doesn't seem to work either when you try to create it programmatically, it provides you with a kustoconnectionstring that is not recognized by lightingest, (same keyword issue)

Could be that i'm missing something completely here :)

Thanks in advance
Pratyush Vashistha 5,125 Reputation points Microsoft External Staff Moderator

2025-10-09T08:12:43.8966667+00:00
Hello 01725609,

Looking at your screenshots, you've discovered that:

The SDK has managed identity implementation (m_embeddedManagedIdentity property)

The Keywords enum is missing the managed identity keyword

The EmbeddedManagedIdentity property can only be set programmatically, not via connection string.

This means that the

LightIngest is successfully using the Managed Identity to get a token, and the ADX ingestion endpoint is receiving and identifying it. This is great progress.

ADX service, after identifying the identity, is checking its policies and permissions and actively rejecting the request.

This means the problem is not in the LightIngest command's syntax but in the configuration on the Azure Data Explorer side.

Also I appreciate you pointing out the error in my previous suggestion. My apologies for the confusion. I found out that information on this learn document.

https://learn.microsoft.com/en-us/azure/app-service/overview-managed-identity?tabs=portal%2Chttp#rest-endpoint-reference

Meanwhile Workaround Solutions to could be as follows:

Use System-Assigned Managed Identity (if possible): Try modifying your command to use system-assigned managed identity:
-managedIdentity:"system" -connectToStorageWithManagedIdentity:"system" -ingestWithManagedIdentity:"system"

Alternative Ingestion Methods to consider:

Using Azure Data Factory with managed identity authentication

Creating a custom ingestion application using the Kusto SDK programmatically

Using Kusto .NET SDK where you can set EmbeddedManagedIdentity property directly

Temporary Workaround: Use Service Principal authentication instead of managed identity for LightIngest until this gap (I will confirm this with Product Group team) is resolved.

Code Example for Programmatic Approach:

var connectionStringBuilder = new KustoConnectionStringBuilder(ingestUri) .WithAadManagedIdentity(clientId);

Supporting Documentation

The official documentation confirms the use of the -managedIdentity flag. While it doesn't explicitly state Client ID vs. Object ID, this flag is for client-side authentication, which consistently uses the Client ID.

Reference: Use LightIngest to ingest data into Azure Data Explorer

This documentation explicitly states that the policy is configured using the identity's Object ID.

Reference: Managed identities policy overview

This document explains the fundamental difference between an application's Client ID and its Service Principal's Object ID.

Reference: App objects and service principals in Azure AD

The issue you've identified is valid - there's an implementation gap between the SDK capabilities and the LightIngest tool's connection string parsing. This explains why your permissions are correct, but the tool fails to authenticate properly. Allow me to check this with Product team.

Thanks

Pratyush
01725609 105 Reputation points

2025-10-09T08:23:25.7966667+00:00

Thanks for your response and taking this up with the product team.

I get I can use other authentication methods, however, looking at how our complete permission structure is set up, using a user-assigned managed identity would be ideal, and if I recall correctly, using the system assigned managed identity gives me exactly the same issues.

I already have a custom tool present that leverages the SDK, but since the documentation states that in order to backfill a lot of historical data into adx (and I mean billions of records in our case), it is advised to use LightIngest? (that is also why we need it containerized, and ideally authenticating via Managed Identity - if it is indeed the case that this is a big issue, we will use another authentication mechanism)

As far as I read the documentation, for the ingestion parameters for LightIngest we need the ClientId, but for the Storage Connection String we need the ObjectId

https://learn.microsoft.com/en-us/kusto/api/connection-strings/storage-connection-strings?view=azure-data-explorer&preserve-view=true#managed-identity

Thank you in advance for picking this up.
Pratyush Vashistha 5,125 Reputation points Microsoft External Staff Moderator

2025-10-28T04:53:39.31+00:00
Hello 01725609,

I am summarizing our mail communication here on how to make LightIngest work with UAMI in a containerized environment:

Following is the solution that worked on suggested PG response.

Assign Permissions Correctly

For ADX access:

Attach the UAMI to the container app hosting LightIngest.

Grant the MI Ingestor role on the ADX cluster/database.

Ensure ADX managed identity policy includes the Object ID with AllowedUsages = NativeIngestion.

For Storage access:

Grant the MI Storage Blob Data Reader role on the storage account.

Use Correct LightIngest Parameters

-managedIdentity:"<clientId>" → MI for LightIngest to connect to ADX.

-connectToStorageWithManagedIdentity:"<clientId>" → MI for LightIngest to list blobs.

-ingestWithManagedIdentity:"<clientId>" → MI for ADX to pull data from storage.

-sourcePath:"<blobUrl>;managed_identity=<objectId>" → Object ID for storage access.

Run Inside the Container

These flags require the MI to be locally available, so they won’t work from a local machine.

Deploy LightIngest in the container app with the UAMI assigned.

Validate Policies

Run .show cluster policy managed_identity and .show database policy managed_identity in ADX to confirm the MI is allowed for ingestion.

Alternative if Issues Persist

Consider queued ingestion commands (native to ADX) as they don’t require LightIngest to run locally.

LightIngest Command Template

LightIngest "https://<cluster-uri>.westeurope.kusto.windows.net;Fed=true" \ -managedIdentity:"<clientId>" \ -connectToStorageWithManagedIdentity:"<clientId>" \ -ingestWithManagedIdentity:"<ObjectId>" \ -sourcePath:"https://<storage-account>.blob.core.windows.net/<container>/<folder>/.../file.csv;managed_identity=<objectId>" \ -database:"<databaseName>" \ -table:"<tableName>" \ -format:"csv" \ -pattern:"*" \ -ingestionMappingRef:"<mappingName>" \ -ignoreFirstRow:True \ -tag:"<tag>" \ -creationTimePattern:"/'yyyy/MM/dd'/" \ -dontWait:True \ -interactive:False \ -listOnly:False

Key Placeholders

<cluster-uri>: Your ADX cluster ingestion endpoint.

<clientId>: Client ID of the User-Assigned Managed Identity (UAMI).

<objectId>: Object ID of the same UAMI for storage access.

<storage-account>, <container>, <folder>: Blob storage details.

<databaseName>, <tableName>: Target ADX database and table.

<mappingName>: Ingestion mapping reference in ADX.

<tag>: Optional tag for tracking ingestion.

Notes

Ensure the UAMI is assigned to the Azure Container App running LightIngest.

Verify ADX managed identity policy includes the Object ID with AllowedUsages = NativeIngestion.

Grant Ingestor role on ADX and Storage Blob Data Reader on the storage account.

Run this command inside the container, not from local CLI.

Assuming you use the MI for all interactions (tool to Kusto, tool to storage, Kusto to storage), you need all 3. The command looks ok, though make sure your cluster is really in WestEurope or use the URI from the portal page as is. (IIRC you do not need “;fed=true”)

You can complete need to complete rest of the parameters like the storage reference (at minimum). Other than that, consult with lightingest -help

Use the MI client id where you want the tool to make use of the MI to reach out to other services (Kusto, Storage) and use the object id where you want Kusto to access the file using the MI.

Please feel free to respond over the mail thread or over here in order to get your doubts cleared. Kindly 'Accept this as an answer' or click 'Yes' if the provided information was helpful so that others can also get benefit from it.

Thanks, and Happy to help

Pratyush

Share via

Azure Data Explorer: LightIngest Managed Identity Authentication Issue

1 answer

Your answer