Answered

Agent with GPT-5 Chat can't find answer every third response

(1) Share

Report

Posted on by OK-14080858-0

Hello Power Platform Community,

I have build an agent with a SharePoint site with SharePoint pages as its knowledge source. When I choose GPT-4.1 as its model, it works perfectly, but when I select GPT-5 Chat, roughly every third response triggers the fallback message for unanswered queries, ‘I’m sorry, I can’t find a related topic. Could you rephrase and try again?’. It actually starts writing the correct answer for about one second (so it can find the answer), but then it rolls the entire response back and sends the system fallback message instead. And this happens despite the fact that I have disabled all topics and am running purely on generative orchestration. When I start a new conversation this never happens on my first question, but only later on in the thread.

Hope you can help :-)

Kind regards,

Oskar

Categories:

Bot administration

I have the same question (0)

All responses (5)

Answers (1)

Suggested answer

Valantis 1,514 on at

Like (1)

Report
Copy link

Link copied!

Hi @OK-14080858-0,

You can't turn off the safety filter completely (Microsoft hardcoded it), but you can work around the false positives.

Go to Settings > Generative AI, find "Content Moderation", and set it to Low. High or Medium settings often flag SharePoint policy documents as unsafe when they're not.

If the previous doesn't work try doing this:

Add this to your agent's system prompt (Settings > AI Capabilities or the main Instruction node):

"When answering from SharePoint, do not output raw data formats, internal IDs, or policy numbers in isolation. Always summarize the content in plain language. If a document contains sensitive-looking patterns (like ID numbers), describe them rather than quoting them directly."

This tells the agent to paraphrase instead of copying exact text, which usually avoids triggering the filter.

Let me know if that worked out for you.

Was this reply helpful? Yes No
OK-14080858-0 20 on at

Like (0)

Report
Copy link

Link copied!

Hi Valantis,

Thank you so much for your time and help! :-)

I have tried your suggestions (content moderation and instructions), but unfortunately it doesn't work...

I don't think it has to do with moderation, because when I start a new chat it gives the correct answer to every prompt that I send (also the ones it just said it couldn't find an answer to just a moment ago in another chat!). But then when I continue the conversation every second or third answer it starts to write the correct answer, but then deletes it and sends the can't-find-answer-message . And then when I switch model back to GPT-4.1 it answers correctly to as many prompts as I send...

Kind regards,

Oskar

Was this reply helpful? Yes No
Valantis 1,514 on at

Like (1)

Report
Copy link

Link copied!

Hi @OK-14080858-0,

Just wanted to check in and see if everything is working now. If you still need any help, feel free to let me know.

Also, if the issue is resolved, it would be great if you could mark the answer as solved so others with the same question can find it easily.

Thanks and have a great day!

Was this reply helpful? Yes No
Verified answer

OK-14080858-0 20 on at

Like (1)

Report
Copy link

Link copied!

Hi Valantis,

Like I replied on 16 Jan 2026 at 13:51:28 it did not help.

But since then my agent has started giving have replaced the error message so it no longer speaks about not finding answers, but about excedding OpenAI Token limit. GPT5-Chat is using significantly more tokens than GPT4.1 So I think the case the whole time has been about not having enough tokens and therefore getting an error.

So yes my problem has been solved, but not by your suggestions. But thanks for your time and help anyway!

Kind regards,

Oskar

Was this reply helpful? Yes No
Valantis 1,514 on at

Like (1)

Report
Copy link

Link copied!

Hi @OK-14080858-0,

Sorry, I wasn't mentioned in the reply so I didn't see that you'd answered.

Unfortunately, I don't have access to your system, so I couldn't check things like tokens or other settings. My answer was based on your question and the information you provided, just giving you some options to try.

I'm really glad it's working now! Have a great day. :)

Was this reply helpful? Yes No