Unanswered

Agent behaving inconsistently across different users

(0) Share

Report

Posted on by CH-07011230-0

Hi all,
I built a sales AI agent for my team. The agent is currently hallucinaling or claiming it cannot find the documentation.
The weird part is that it works perfectly for some users but fails for others within the same team.
Any ideas on how to troubleshoot/debug this?

Categories:

Bot analytics

Calling actions from Copilot Studio

Copilot Studio skills development

Copilot Studio pre-built agents/templates

I have the same question (0)

All responses (6)

Answers (0)

André Arnaud de Cal... 249 on at

Like (0)

Report
Copy link

Link copied!

Hi,

You can control a lot by providing instructions to the agent, the chosen language model, and knowledge sources (including documentation pages). Apart from that, the prompt provided by the user can influence the behavior.

Maybe you can share some more information about your agent and the used prompts?

Was this reply helpful? Yes No
CH-07011230-0 4 on at

Like (0)

Report
Copy link

Link copied!

Here is an example of the inconsistent behavior, where the first response is wrong.

Was this reply helpful? Yes No
David_MA 13,091 Super User 2025 Season 2 on at

Like (0)

Report
Copy link

Link copied!

I have limited experience with Copilot Studio, but if I remember correctly, it respects the permissions of the data sources you provide. Do all of the users who use this agent all have the same access permissions to the sources used by the agent? If not, that could account for the different experiences. For it to answer with the permissions you have, you need to configure it to use a service principal or app-level permission.

Was this reply helpful? Yes No
CH-07011230-0 4 on at

Like (0)

Report
Copy link

Link copied!

Hi David,
How can I check if everyone has permission to access the data sources?

Was this reply helpful? Yes No
André Arnaud de Cal... 249 on at

Like (1)

Report
Copy link

Link copied!

Hi,

To be honest, I can't understand the example you shared. I'm not familiar with that language. To help you with the permission question, you can check which knowledge sources are available in the agent and if the users do have access to all these knowledge sources or not. If you added e.g. document available on SharePoint, check for the users with different outcomes if they are able themselves to open the document.

Was this reply helpful? Yes No
David_MA 13,091 Super User 2025 Season 2 on at

Like (0)

Report
Copy link

Link copied!

Unless you are uploading files or using public websites as its only resources, you will need to check with the information owner(s) of the knowledge you are adding to the agent to see how security has been configured. You cannot assume that because you have access to a knowledge source, everyone else does.

If you are only uploading files to the agent and don't have any external sources (except public websites), all of the users of the agent should have access to the content. If there are only uploaded knowledge sources, I would ask your users to send you the prompts they used. Then try these prompts yourself to see what answers you get and compare it to what your users got.

Beyond this guidance, I don't have enough experience to offer any further insights into the issue you are having. We're still just exploring Copilot agents and figuring out how to use them. It's a fun journey into the unknown. :-)

Was this reply helpful? Yes No