Power Platform Community Forum Thread Details

Good morning,

I have built a small RAG solution based on uploading 50 PDF files manually. On 27th of August 2025, the solution was working fine. I tested it again on 9th of September 2025 with the exact same questions and without bringing any change to the agent and now it sytematically triggers the Fallback or Conversational boosting topics. What is really weird is that when looking at the Test UI, I can see it starts writing the right answer but suddenly it stops and reverts to Fallback or Conversational boosting topics.

Has something changed recently (since 27th of August) in Copilot Studio that I missed?

Have you experienced the same behaviour.

Thank you for your help and have a very nice day.

Christophe

Categories:

General topics

Good morning, CU10090540-0,

Thank you for reaching out. I understand your concern about the unexpected behavior change in your Copilot Studio RAG solution. It's indeed perplexing when a previously functional agent starts failing without any apparent changes on your end.

Based on your description, where the agent begins to formulate the correct response and then suddenly reverts to a fallback or conversational boosting topic, it is highly likely that this is due to recent updates within the Copilot Studio platform. Between late August and early September 2025, Microsoft has rolled out several enhancements that could be influencing your agent's behavior.

While you haven't made any changes, the underlying generative AI models and response mechanisms have been updated. Here are some of the key developments during that period and how they might relate to the issue you're observing:

Recent Copilot Studio Updates (Late August - Early September 2025)

Enhanced Generative AI Quality Tracking: New analytics features were released in August 2025 to better track unanswered queries and the quality of generative AI responses. It's possible that stricter internal quality checks are now causing the model to abandon responses it deems potentially low-confidence, even if they appear correct initially.
New Fallback Logic: There have been adjustments to how the system fallback topic is triggered, especially when Generative AI is enabled. A bug was recently resolved where fallback and AI answers could trigger simultaneously. The current behavior is designed to trigger fallback only if the generative AI fails to provide a confident answer. The abrupt stop you are seeing could be the model's confidence score dropping mid-generation, causing it to fail and trigger the fallback.
Introduction of GPT-5: In August 2025, Microsoft began integrating GPT-5 into Copilot services. Copilot now dynamically switches between different models based on the complexity of the query. This shift could mean your prompts are being processed by a different, and possibly more sensitive, underlying model than they were on August 27th, leading to different outcomes.
Updates to Agentic Responses and File Handling: A general availability release in August enhanced the accuracy of agentic responses by introducing file groups and improving how agents analyze uploaded files to generate answers. Changes to the internal processing of your 50 PDF files could be a factor

Why is the Answer Being Cut Off?

The behavior you describe—seeing the correct answer start to generate before being replaced—is a classic sign of the generative AI model losing confidence or hitting an internal boundary. Here's a likely sequence of events:
The agent correctly identifies the relevant information in your PDF files and starts streaming the answer.
Partway through the generation, an internal check (related to content safety, response length, confidence score, or a newly introduced bug) fails.
The generative process is immediately halted.
Because no complete answer was successfully generated, the system triggers the next logical step in your topic flow, which is either the standard "Fallback" topic or your "Conversational Boosting" topic.

Recommendations and Next Steps
You are not alone in experiencing this kind of behavior after a platform update. Here are a few things you can try:

Review Fallback Topic Analytics: Use the new analytics features to check the "Generated Answer Rate and Quality" section. It might provide insight into why the queries are now being classified as unanswered.
Test with a Higher Boost: Temporarily set the "Conversational boosting" topic to a higher priority to see if it can successfully handle the queries instead of going to the system fallback.
Re-index Your Documents: Although you haven't changed the files, it might be beneficial to remove and re-upload your 50 PDFs. This could force a re-indexing of the content with the latest platform version, potentially resolving any data processing discrepancies.
Report the Issue: Since this appears to be a platform-level change, I strongly recommend reporting this behavior on the Power Platform Community forums or through official Microsoft support channels. Include the specific dates and the behavior you've observed, as this information is invaluable for the product team.

It's highly probable that an unannounced fine-tuning of the generative response model is the root cause. While these updates aim to improve performance and safety, they can sometimes lead to temporary regressions for specific use cases like yours.

Thank you for your help and have a very nice day.

Best Regards,
Jerald Felix