Power Platform Community Forum Thread Details

Hello. I am an administrator who built a chatbot using a chat bot agent in Copilot Studio so that internal colleagues can ask questions about things they're curious about.

I built the agent as follows:

1. I uploaded files such as Excel and Word as reference knowledge so that the agent can only provide answers from the original sources.
2. I created topics so that when a user asks a question, the agent only refers to the reference knowledge relevant to that topic.
3. I set the reference knowledge to be accessed only when a topic references it, and I set the agent's AI model to GPT 4.1.

In the current setup, when users ask questions, all answers are output correctly without any hallucination issues. However, I want to use GPT 5.5 Chat for better functionality in the future, but when I select the GPT 5.5 Chat model, the hallucination issues are severe.

I'm actually not sure if "hallucination" is the right term for this. For example, if a user asks, "Who is the person in charge of Product A in Busan?", the original document states "Bae Hyun-deok," but the agent set to GPT 5.5 Chat responds with "Bae Hyeong-deok" along with an unrelated person listed as a co-manager. This is just one example, and there are many similar issues.

I'm wondering whether this is a problem specific to models other than GPT 4.1, or whether I have designed the agent incorrectly. (The same issue occurs with both GPT 5.5 Chat and Claude models.)

I have experimented with providing reference knowledge as Excel files as well as structured Word documents in JSON-like format, but the results remain the same.

I would greatly appreciate any professional advice you could offer.

Categories:

General topics

Topic creation & management

Model context protocol