Copilot Studio Agent works well in MS Teams for me, but not others in my company

(0) Share

Report

Posted on by DM-26032358-0

I created an AI Agent with Microsoft Copilot Studio to help answer questions at my job based on all our of policies, procedures, training guides, etc. I created a Sharepoint site to house all of the data and information for the Agent. Then I deployed the Agent in Microsoft Teams, so it works just like a regular chat. It’s like a helpful AI assistant and does a great job at reviewing our docs and answering questions. Very helpful! The answers it gives is very accurate when I tested it out! All good.

The issue is, when I SHARE the bot/agent with someone else (other than me) in my department/company, it doesn't work quite right. It gives nonsensical answers. The other employee can open the Agent in their MS Teams, and they have access to the Sharepoint data source, but it "hallucinates" and doesn't give correct info. I can tell it's using the Sharepoint Library as a source, it's just giving random answers.

Through my research, I think there's is an authority setting or a box I need to check?! My research also mentions something about “grounding.” I’m hoping the fix is something simple, but I haven’t been able to figure it out yet. I also tried setting up an Entra Security group, but still doesn’t work. I think I have all the authorities and permission correct, but what am I missing??? I was hoping someone could help or point me in the right direction.

Thank you,
David

Categories:

Building Copilot Studio chatbots in Microsoft Teams

I have the same question (0)

All responses (2)

Answers (0)

Suggested answer

Sunil Kumar Pashikanti 1,353 Moderator on at

Like (0)

Report
Copy link

Link copied!
Hi @DM-26032358-0,

In the world of Copilot Studio and Retrieval-Augmented Generation (RAG), what often looks like the AI "making things up" is actually a data retrieval failure.

When a Copilot cannot "see" the source documents due to permission layers, it still tries to answer the user's prompt using its general training data. This results in an answer that isn't grounded in your specific SharePoint files.

Why this happens: The "User Context" Rule
In Copilot Studio, when you use SharePoint as a data source, the bot respects Security Trimming. This means the bot can only "read" what the person chatting with it is allowed to see in SharePoint.

Breaking down the causes:

Permissions vs. Visibility: If a user has access to a SharePoint Site but not the specific Document Library or Folder where the files live, the Copilot will return a blank or "hallucinated" result for that specific user.

Inheritance Issues: If you have "Unique Permissions" set on specific folders, the Copilot might successfully answer questions about File A but fail on File B, even if they are in the same site.

The "Maker" Trap: A common mistake is for the developer (the "Maker") to test the bot and see it working perfectly because they have high-level admin access. When a regular employee uses it, it fails because their permissions are more restricted.

Fallback to Generative AI: When the search returns zero results from your SharePoint (due to the permissions mentioned above), the model often uses its internal knowledge to try and be "helpful," which leads to answers that don't align with your company's actual policies.

Pro-Tip for Testing
To verify this, have a user with limited permissions try to search for a specific keyword in that SharePoint library directly. If they can't find the file in the SharePoint search bar, the Copilot won't be able to find it for them either.

✅ If this answer helped resolve your issue, please mark it as Accepted so it can help others with the same problem.
👍 Feel free to Like the post if you found it useful.

Was this reply helpful? Yes No
DM-26032358-0 2 on at

Like (0)

Report
Copy link

Link copied!

Thank you for the reply, that helps me to understand the issue a little better. And I'm starting to think the issue is with Sharepoint, based on your reply. So I tested it with my co-worker, Michael, and he does have access to the Sharepoint Knowledge Library that I have set up. He can look through the folders and see all of the documents I've stored there, and I've given him "Site Owner - full control" permission, but it still hallucinates when he asks the Agent/Bot questions in MS Teams. Any suggestions?

Was this reply helpful? Yes No