Custom Copilot Studio agent defaults to fallback

(2) Share

Report

Posted on by AH-23031939-0

I'm developing a relatively simple Copilot Studio agent, using a SharePoint folder containing 10 DOCX files for knowledge, linked as agent level knowledge (not via a topic). The agent is using generative orchestration, but ungrounded responses and web access is turned OFF (see images below). This is because we only want answers to be generated from the connected knowledge source (the Docx files in SharePoint). I shared the agent to a test group, and we chatted to it for months, iterating and updating the knowledge content etc.

A week ago, it started saying it couldn't help, it kept going directly to fallback, escalate and boosting topics for normal questions we had always asked it, and questions that it definitely has knowledge for. In fact i could ask it the same question and one reply it would fallback, and when asked again it might partially answer, and sometimes it might give a proper answer. But the amount of times it went to fallback and did not answer was significant. For some uses it consistantly goes to fallback and wont answer. I have confirmed all users are shared the agent with necessary chat access, and all shared users have read access to the SharePoint files.

If i create a "General Questions" topic with the knowledge source specificly linked to the topic, to try and "catch all" - it seems to work a bit better, BUT, generative orchestrated agents do not need topics for this type of agent, with one knowledge source.

I created a new environment, new agent, same knowledge source and i get similar very inconsistant results.. Some users have less trouble, others simply get "sorry i can't help" fall back topic..

Any clues?

Categories:

Building Copilot Studio chatbots in Microsoft Teams

Topic creation & management

General topics

I have the same question (0)

All responses (3)

Answers (0)

Sort by

Suggested answer

sumit_artesian 261 on at

Like
a
(1)

Report
Copy link

Link copied!
Hi @AH-23031939-0,

I don’t think your SharePoint knowledge source has actually broken. What has likely changed is how Copilot Studio retrieves content behind the scenes.

Microsoft has been moving tenants over to the newer Microsoft 365 grounding model, where retrieval now happens through Microsoft Graph and respects the permissions of the actual user chatting with the agent.

So earlier, the agent may have been able to answer using the maker’s access to the SharePoint files. Now, if the end user doesn’t have proper access to the documents or the files haven’t been fully indexed yet, the agent basically behaves as if the content doesn’t exist and falls back.

That’s also why you can see inconsistent behavior between users even when the agent itself hasn’t changed.

I’d specifically check:

Whether all users truly have direct access to the SharePoint files/site

Whether the files were recently updated/uploaded and still being indexed

Whether any sensitivity labels, private channels, or item-level permissions are involved

The setup may look identical from your side, but the retrieval model underneath has changed quite a bit recently.

I also found this listed as active issues from 4 weeks ago here in the community:

"We’ve identified that this issue may be related to a recent Copilot service update that changed how agents process knowledge source queries. We’ve disabled a specific component to effectively revert that change in an effort to alleviate this issue. We're reaching out to a subset of impacted users to confirm whether this action restores expected behavior, or if additional actions will need to be taken to fully resolve this impact."

You can also refer to the thread on Reddit where people faced this same issue: Sharepoint knowledge not working with Agent : r/copilotstudio

Please ✅ Does this answer your question if my post helped you solve your issue. This will help others find it more readily. It also closes the item.

If the content was useful in other ways, please consider answering Yes to Was this reply helpful? or give it a Like! 🩷

Was this reply helpful? Yes No
AH-23031939-0 13 on at

Like
a
(1)

Report
Copy link

Link copied!

Thanks Sumit for the reply, it was helpful, but yes my users do truly have access to the SharePoint data and files, also the files are not being indexed, they finished indexing weeks ago.

Based on the Reddit conversation, I am concerned this is just Microsoft doing their normal, changing and breaking stuff under the covers, right before customers launch and go live dates.

Was this reply helpful? Yes No
sumit_artesian 261 on at

Like
a
(0)

Report
Copy link

Link copied!

Hi @AH-23031939-0,

Glad I could be of help. And yes, this does seem like MS just doing their thing.

If my response was helpful, please feel free to mark it as a verified answer. This would close this issue, and also help other community members find the solution as well.

Thanks, and good luck with your implementation.

Was this reply helpful? Yes No