Share via

Copilot Studio Agent: Why Does It Start With the Right Answer and Then Switch Mid-Response?

Amine Mekki 10 Reputation points
2025-11-25T10:01:20.77+00:00

Hi everyone,

I’ve built an agent in Copilot Studio that uses SharePoint and a website as its knowledge base. The agent uses generative orchestration for its flow.

Here’s the issue: Sometimes, when I ask a question, the agent starts answering correctly (I can see the right answer being streamed), but then mid-response it switches and gives a wrong or incomplete final answer. In the test panel, I notice this happens when conversational boosting kicks in.

From what I understand, conversational boosting is a fallback when orchestration fails. But I don’t know why orchestration fails in these cases, especially since it starts off correctly.

My questions:

  1. Why does the agent begin with the correct answer and then switch mid-response?
  2. What causes generative orchestration to fail and trigger conversational boosting?
  3. How can I prevent this fallback and ensure the agent sticks to the orchestrated answer?

For context, here’s an example of my system prompt (company name anonymized as X and document types as Y and Z):

# PURPOSE
Your mission is to answer users’ questions about X using Y and Z documents.

# RESPONSE CONTRACT
- Tone: Professional, clear, and concise.

# RESPONSE FORMAT
1. Answer:
   - Provide a clear answer relevant to the question (do not write “Answer:” as a label).
2. Source:
   - Include excerpts that were used to generate the answer.
3. Disclaimer:
   - Always include:
     - *This response was generated by an AI assistant based solely on X’s official Y and Z documents. Please verify the information provided by reviewing the cited sources, as this content was generated using AI and may require human validation.*

# EXAMPLES TO SIMULATE
User: "Here I give the agent an example of a question"
Your answer: Here I give the agent an example of an answer
Source:
- "here I give an example of the text chunk"

Disclaimer...

Has anyone experienced this behavior? Any ideas on why orchestration fails and how to avoid the fallback?

Thanks!

Microsoft Copilot | Microsoft 365 Copilot | Development

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sayali-MSFT 4,341 Reputation points Microsoft External Staff Moderator
    2025-11-28T09:33:52.47+00:00

    Hello Amine Mekki,
    Generative orchestration in Copilot Studio may start with the correct, grounded answer but then switch mid-response when its confidence drops. When this happens, the system falls back to conversational boosting, which relies on broader model knowledge and can override the grounded content. Orchestration failure is usually caused by low-quality retrieval, ambiguous or complex prompts, irrelevant content chunks, or streaming fallback triggers.
    To prevent this, improve retrieval structure, adjust confidence thresholds, simplify and clarify system prompts, test problematic queries, or disable conversational boosting. You can also instruct the agent to respond with a fallback message instead of generating ungrounded answers.
    Reference Document: -

    1. https://learn.microsoft.com/en-us/microsoft-copilot-studio/knowledge-copilot-studio
    2. https://learn.microsoft.com/en-us/microsoft-copilot-studio/knowledge-add-sharepoint
      3.https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-generative-actions
  2. Gerald Gaston 0 Reputation points
    2025-11-30T19:39:17.8466667+00:00

    I'm seeing this more in agents that were working fine using GPT-4o (now retired but can test to confirm behavior when "Get 30 additional days with your existing model before it becomes unavailable" is enabled) and are now using GPT-4.1 or greater.

    The newer the model the more this issue occurs because:

    1. GPT‑4o uses simpler orchestration logic and doesn’t aggressively override responses.
    2. It respects connector outputs and system instructions more consistently.
    3. It avoids the “second pass” behavior that GPT‑4.1 and GPT‑5x introduced for grounding and confidence scoring.

    Stated another way, GPT‑4o works better in my scenarios because:

    • Deterministic answers: It doesn’t try to “improve” or reformat responses after the fact.
    • Lower orchestration complexity: No mid-response fallback injection.
    • Stable multi-channel behavior: Works in both M365 Copilot and Teams without escalation errors.

    My solutions that use a SharePoint List as the knowledge source are experiencing this the most. Moving from relying heavy on Instructions to utilizing agent flows helps, or for now for simple agents just switching back to classic orchestration with trigger phrases.

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.