Unanswered

How to improve consistently wrong answers

Like (2) Share

Report

Posted on 2 Nov 2024 18:40:43 by WF-19092047-0

I am building a test bot that should return information from files that are in a Sharepoint site. My use case is Medicare insurance policies. In my test bot, I have one knowledge source, the SharePoint site. In the SharePoint site, I have 10 PDF files (all under 3mb) that contain summaries of each Medicare plan. Each plan has a carrier (i.e. the insurance company), a plan name and a description of the plan. In my sample data set, I have 5 plans from Blue Shield. If I ask a prompt like "Make me a bulleted list of all of the Blue Shield plans." It will only return a list with two plans.

My follow up prompt is "But there are 5 plans from Blue Shield I need a list that shows all of them. Please redo.". The bot will then create a list with 5 plans ... which is better, but the list is wrong.

I am using Generative Topics and High Precision.

How do I improve the accuracy of the answers.

Categories:

General

All responses (3)

Answers (0)

Suggested answer

Vinoth Selvam 1,527 Super User 2025 Season 1 on 03 Nov 2024 at 15:46:52

Like (0)

Report

How to improve consistently wrong answers

Hi WF-19092047-0 ,

Double-check that each document has a consistent format and labeling for the carrier and plan name fields. If fields are labeled differently or if similar terms are used interchangeably (e.g., "HMO" vs. "Health Maintenance Organization"), the bot may miss certain entries.

Generative Topics and High Precision may sometimes cause the bot to focus too narrowly, potentially leading it to summarize rather than list all items. Consider experimenting with lower precision or broader topic scope to see if the bot can retrieve more complete information.

Providing detailed prompts might help it avoid summarizing or misinterpreting the request. Configure your topics with proper details, inputs and descriptions for the LLM to understand the context more.

thanks.
Suggested answer

JF-03111258-0 4 on 03 Nov 2024 at 13:03:36

Like (0)

Report

How to improve consistently wrong answers

Make sure the option to have the LLM use its ‘own knowledge’ is turned off.
ronaldwalcott 3,653 on 02 Nov 2024 at 20:08:32

Like (0)

Report

How to improve consistently wrong answers

If I were you when it comes to using data that can be structured I would structure it as best as I can.

e.g. In your example I would have a SharePoint table which would list the plans, policies, etc.

These tools are ok for interpretating unstructured data but structured data tends to improve results with queries that are of the form list this where so and so is equal to so and so.