Integrating Azure OpenAI and other AI services

  • 5 minutes

Integrating AI services means hooking up your agent to one or more large language models or cognitive services to give it “intelligence.” In our case, the primary integration is with Azure OpenAI Service (or OpenAI API) for natural language understanding and generation. Here’s how to ensure that’s set up properly, as well as how to incorporate other Azure AI components if needed:

  • Azure/OpenAI Credentials: When you created the project, if you entered your Azure OpenAI key and endpoint, the project’s configuration is already set. Verify this in your .env file (for JS, likely .env.local or .env.playground): it should contain something like AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_KEY, etc., as well as the deployment name of the model. In a C# project, these might be in appsettings.json or in code as part of the AddOpenAI call. Make sure these values are correct. Without a valid key, the bot will not be able to call the model.

  • Selecting Models: Decide which model to use. GPT-4 provides better quality but is slower and costlier, while GPT-3.5 (gpt-35-turbo) is faster/cheaper. For a Teams chat agent, GPT-3.5 is often sufficient for Q&A or simple tasks; for complex reasoning or if the agent should use the Action Planner heavily, GPT-4 might perform better. You can configure the model name in the settings. (Azure OpenAI deployment names often encapsulate which model.)

  • Multiple AI Services: The Microsoft 365 Agents SDK is flexible – you could integrate additional AI services:

    1. Azure AI Foundry or Agent Service: If your organization uses Azure AI Foundry (a platform to host and orchestrate models and tools), you could connect your bot to a Foundry “agent” or model. This is an advanced scenario; essentially your bot becomes a frontend for an existing agent logic running elsewhere.
    2. Cognitive Services: You might use Azure Cognitive Services for specific tasks. Example: Azure Language Understanding (LUIS) or Custom Question Answering for NLP, or Form Recognizer if your bot needs to parse documents. These can be called as APIs from within your bot code. Another example: using Azure Speech service if you plan voice input/output.
    3. Semantic Kernel or LangChain: These are orchestration frameworks that can manage complex sequences of prompts or tool invocations. If your scenario requires complex orchestration (like multi-step conversations or calling many tools), you might integrate Semantic Kernel in your bot. The SDK is compatible – you can invoke SK planners inside your message handler. This is optional; the Teams AI Library’s built-in orchestrator might suffice for many cases.

    • Retrieval-Augmented Generation (RAG): Integrate Retrieval (Azure Cognitive Search) for RAG: a. To make your agent truly useful, you might need it to have knowledge of your enterprise content or specific data. A best practice is to implement Retrieval-Augmented Generation (RAG) – where the bot fetches relevant data from a knowledge source and feeds it into the LLM’s context, rather than relying solely on the model’s training memory. Here’s how to set that up:

      b. Prepare a knowledge source: For enterprise data, a popular choice is Azure Cognitive Search. You would index your documents (PDFs, manuals, SharePoint pages, etc.) into an Azure Cognitive Search index. Alternatively, you might use a vector database (like Pinecone, Redis Embeddings, etc.). But Azure Search is convenient and integrates well with Azure OpenAI (there’s even a built-in “Azure OpenAI on your data” feature).

  • Implement retrieval in the bot: There are two approaches:

    a. In-code retrieval: Use the Azure Search SDK (or REST API) to query the index. For example, when a user asks a question, before calling the LLM you take the user’s query, call the Search service, get top 3 results (snippets of text), and then construct the final prompt as: [System message with instructions]\n[User question]\n[Retrieved content]\n[User question again or prompt asking for answer using data]. Then send that to OpenAI. The Agents SDK doesn’t do this automatically; you’ll write the glue code. But it’s straightforward: treat the search results as additional context. Microsoft’s sample “Weather Agent” or others might have similar retrieval logic to reference.

    b. Using the Toolkit/Planner capabilities: If using Teams AI Library, you can register a Tool that performs a search. For instance, define a function async function performSearch(query) { /* call Azure Search API */ } and then register it as an AI tool: planner.tools.add("PerformSearch", performSearch, “Searches company knowledge base for relevant information.”). The Action Planner can then decide to use the tool when user’s query seems to need external info. It will include the tool’s result in the prompt to itself. This is a more advanced scenario and requires careful prompt engineering (to get the AI to invoke the tool), but it aligns with how OpenAI function calling works.

In our pro-code scenario, most of the AI integration is about configuring the model endpoints and writing the prompt logic. The heavy lifting (calling the OpenAI endpoints, streaming results, etc.) is handled by the SDK libraries. For instance, if using the Teams AI Library, once you provide your OpenAI key and endpoint and specify the model, calling planner.chain() will automatically send the prompt to Azure OpenAI and get the completion for you. If using pure Bot Framework in C#, you might call the OpenAI REST API or use the Azure.AI.OpenAI SDK within your bot logic.