Unanswered

"Create Generative Answers" fail to give an output 50% of the time.

(1) Share

Report

Posted on by Syamil

Need help! This is my first time creating a chatbot on Copilot Studio.

Under Generative AI, I have attached 1 pdf document as the source of data.

Under Topics > Conversational Boosting, there's a "Create Generative Answer" flow that gets triggered when intent of user's message is unknown. Refer to image below. It's supposed to search for data in the PDF and reply the user accordingly. If it fails to provide any output, it'll give a standard apology message that you can see in the image as well.

Screenshot 2024-01-02 at 9.12.48 PM.png

For some reason, it always fails to give an output (i.e. the answer would be a blank and it'll send the apology message.
I looked at the analytics under "Boost Conversations" and noticed that it fails 50% of the time and the main reason is "no search results" (84% of the failures).

Screenshot 2024-01-02 at 9.09.18 PM.png

Anyone knows why this happens? How do I fix it?

Thank you in advance!

Categories:

General topics

I have the same question (0)

All responses (22)

Answers (0)

HenryJammes on at

Like (2)

Report
Hi @Syamil

Generative answers can return blank answers for multiple reasons:
No results (i.e. the data source, e.g. your uploaded document didn't contain the answer to the user question).
Filtered by Azure OpenAI (e.g. the question wasn't appropriate)
Filtered by Content Moderation (e.g., Copilot Studio prefers not to send an answer when there it detects a risk of hallucination, the "High" setting has a number of checks that are applied).

You can easily get access to additional telemetry data on Generative Answers by connecting your bot to Azure Application Insights: Capture telemetry with Application Insights - Microsoft Copilot Studio | Microsoft Learn

For example, the below TQL query returns the questions and generated answers and their status:

customEvents | where name == "GenerativeAnswers" // | where cloud_RoleInstance == "MS Learn Chatbot" | extend cd = todynamic(customDimensions) | extend conversationId = tostring(cd.conversationId) | extend topic = tostring(cd.TopicName) | extend message = tostring(cd.Message) | extend result = tostring(cd.Result) | extend SerializedData = tostring(cd.SerializedData) | extend Summary = tostring(cd.Summary) | extend feedback = tostring(todynamic(replace_string(SerializedData,"$","")).value) | project cloud_RoleInstance, name, timestamp, conversationId, topic, message, result, feedback, Summary | order by timestamp desc

Example in Application Insights:

Was this reply helpful? Yes No
Syamil 31 on at

Like (0)

Report

Thanks for the reply, @HenryJammes.

I can see the summary of the empty replies under Analytics > Boost Conversations. Majority of the errors were caused by "No Search Results".

The thing is, I know the answer to the questions I asked are available in the PDF I've provided. There were also other occasions where I asked the same question that got no results, and it was able to give me answer.

What do you recommend I do to address this issue of gpt not being able to find the answer when it is actually available in the document provided? Or at least find the nearest topic in the PDF related to the question.

For additional context, I've built another bot using OpenAI's Assistant API and gave it the same PDF. It was able to find the answers in the PDF. And in the case where the answer is not available, it would reply saying that the information is not in the PDF it has access to. Never an empty reply.

Was this reply helpful? Yes No
HenryJammes on at

Like (0)

Report

The blank answer is by design in Copilot Studio when there is no confidence in a generated answer.
That way, in the dialog manager, you can precisely choose what happens when this occurs.
Have you tried lowering the content moderation setting on the Conversational Boosting > Create generative answer node properties?

Was this reply helpful? Yes No
Syamil 31 on at

Like (0)

Report

I have changed the content moderation to low and I still face the same issue.

I have also connected the bot to Application insights as you have suggested.

If I'm understanding correctly, empty answers caused by level of content moderation would appear as "Filtered by Content Moderation" right?

80%+ of the empty responses I get from Generative AI are due to "No Search Results".

I only have one PDF file under generative AI.

As you can see in the screenshot, I asked the same question 3 times. 1 got me an answer, the other 2 gave "No Search Results"

What do you suggest I try next?

Was this reply helpful? Yes No
HenryJammes on at

Like (0)

Report

Thank you @Syamil

Is it possible for you to open a support request?
Get Help + Support in Power Platform - Power Platform | Microsoft Learn

Can you also confirm you're using this feature: Use uploaded documents as a generative answers data source (preview) - Microsoft Copilot Studio | Microsoft Learn for your data source?

Henry

Was this reply helpful? Yes No
Syamil 31 on at

Like (0)

Report

Yes, I confirm that that I used that feature mentioned this article: Use uploaded documents as a generative answers data source (preview) - Microsoft Copilot Studio | Microsoft Learn

I have an update on the issue. The Generative AI suddenly were able to give replies after I added the same PDF file to the "Upload a document" section. (i.e. have two of the same PDFs in the upload a document section in the Generative AI page).

I "narrowed" the issue down to the "upload a document" feature after I realise that the AI works fine when I provided it a website to get answers from, but immidiately give no replies when I it has to find answers from the given document.

Weirdly, it didnt work when I added the .doc version of the PDF (i.e. 1 PDF + 1 .doc) but it started working after I added the 2nd PDF (i.e. 2 same PDFs + 1 .doc, all files contain same content).

After it started working, i removed the first PDF and the .doc file, leaving only the 2nd PDF, and it continued working.

I have no idea if this fixed the problem, or if it is just coincidence.

Was this reply helpful? Yes No
jxu on at

Like (0)

Report

I have the same issue for the "Create Generative Answers" node. It will not output anything, even when provided multiple input websites including Wikipedia.

Was this reply helpful? Yes No
MattJimison 577 Most Valuable Professional on at

Like (3)

Report
Hi, @Syamil and @jxu .

I'd echo what @HenryJammes mentioned about:

Lower the moderation threshold to get more answers back

Utilize AppInsights to troubleshoot searches

Beyond that, and this isn't going to be a popular answer, but LLMs are largely non-deterministic, meaning they don't output the same message twice. Due to this, they can at times determine the data found in search wasn't relevant, and the next time, determine that it is. We've struggled with getting anything consistent from websites, SharePoint, or document uploads, but have more success with using Generative Answers and Azure Open AI Bring Your Own Data or calling Azure OpenAI directly in a simple RAG pattern that searches indexes in Azure AI Search. I'm confident Microsoft will continue to improve all of these actions, but right now, they don't feel consistent enough to deliver the experience most users are looking for. (that could also have to do with models used behind the scenes, temperature, top_p, etc. settings)

Cheers,
Matt

Find this post helpful? Please mark it as the solution and/or provide kudos so that it will help others in the future.

Cheers,
Matt

Was this reply helpful? Yes No
jxu on at

Like (1)

Report

I found out through trial and error that Generative Answers will never respond with a data source of jefferson.edu (my org), however it will respond from usnews.com and who.int as examples.
I don't know if our organization has Azure Monitor setup, but the Analytics Boost Conversations tab reports most is "no search results", so it must be getting filtered somewhere (maybe overzealous prevention of internal information from being outputted?)

Was this reply helpful? Yes No
CU05111601-0 1,091 Super User 2024 Season 2 on at

Like (0)

Report
Bumping this thread since I am having a very similar situation. Per @HenryJammes 's suggestion I've also set up the Application insights. I've noticed two things:
A lot of times 'Filtered by OpenAI' is given even though nothing weird is asked.
In some occasions 'Filtered by high content moderation' is shown even though I have set the moderation to low.
Any ways to tackle these two problems?

Was this reply helpful? Yes No