Share via

How can i implement Multi-Indexing RAG Chatbot with Agentic Search

Nick Koster 20 Reputation points
2025-09-15T17:37:47.6+00:00

I’m currently developing a RAG-based chatbot using the “azure-search-openai-demo” as a foundation. At the moment, the chatbot works with SharePoint data, but I want to expand it by adding multiple indexes in Azure AI Search for additional data sources. For example, I have CMDB data that contains very different fields compared to SharePoint.

My goal is to make the solution scalable, maintainable, so i was thinking about creating new indexes for different data sources.

Currently, I’m facing challenges with multi-indexing in a single RAG chatbot while using Agentic Search. I’m unsure of the best way to implement this. The idea i want is:

  • Users ask questions about a procedure → the chatbot queries the SharePoint gives that index information.
  • Users ask questions about different “tickets” in the CMDB/EAM → the chatbot queries the CMDB index,

I’ve tried querying both indexes simultaneously for Agentic Search, but this doesn’t seem to be supported yet. If i’m right.

I was thinking about a possible solution: query both indexes separately, retrieve the top N results from each index, and then combine these results into a single set. From this combined set, we can rank or re-score the results to select the best answer for the user query. However, I think this approach might lose the “Agentic Search” aspect, where the Agent LLM applies filtering or decides which index to use dynamically based on the question.

I’m looking for guidance or best practices on how to implement multi-indexing with Agentic Search in a scalable and maintainable way.

Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
0 comments
Answer accepted by question author
  1. Vishvani Jilukara 560 Reputation points Microsoft External Staff Moderator
    2025-09-17T17:43:18.38+00:00

    Hi
    With the Agentic Router pattern to support multiple Azure AI Search indexes in your RAG chatbot, you do not create completely separate agents for each index. Instead, you create "tools" or functions representing each index within a single agent orchestration layer.

    How it works practically:

    • You define each index as a tool your master agent can call

    The master agent (orchestrator) gets the user query and asks the LLM to decide which tool to invoke based on the question context.

    The selected tool executes a search query on its respective index, returns results back to the master agent.

    The master agent then generates the answer using the retrieved results as context.

    Why not create separate agents for each index?

    Agentic retrieval in Azure AI Search is designed to enable an LLM to dynamically select which tools (indexes) to query without fragmenting functionality into disparate agents.

    • Keeping a single agent with multiple tools provides scalability, maintainability, and a coherent query experience.
    • Frameworks like Semantic Kernel or LangChain support this by letting you register multiple tools within one agent orchestration.

    You can mark it 'Accept Answer' if this helped you

                  Regards,

                  Vishvani

    0 comments

2 additional answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Jerald Felix 9,835 Reputation points
    2025-09-16T04:10:16.05+00:00

    Hello Nick Koster,

    Azure's Agentic Retrieval currently only works on a single index.

    To solve this, you need to build an Agentic Router. This is an orchestration layer you create in your application backend.

    Here's the pattern:

    Define "Tools": Create a function for each search index. For example, a sharepoint_search tool and a cmdb_search tool. Give each tool a clear description of what it's for (e.g., "Use this to find company procedures" or "Use this to find IT ticket details").

    Route the Query: When a user asks a question, first send the question and your list of tool descriptions to an LLM (like GPT-4). Ask the LLM to choose the best tool for the job.

    Execute the Chosen Tool: The LLM will respond with a tool name. Your code then calls the corresponding function, querying only the single, most relevant index.

    Generate the Answer: Use the search results from the selected index as context to generate the final response for the user.

    This approach is efficient, scalable, and truly "agentic" because the LLM decides which data source to use. Frameworks like Semantic Kernel or LangChain can help simplify building this router logic.

    Best Regards,

    Jerald Felix

    2 people found this answer helpful.
  2. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

    Comments have been turned off. Learn more

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.