How to improve consistently wrong answers

(2) Share

Report

Posted on by WF-19092047-0

I am building a test bot that should return information from files that are in a Sharepoint site. My use case is Medicare insurance policies. In my test bot, I have one knowledge source, the SharePoint site. In the SharePoint site, I have 10 PDF files (all under 3mb) that contain summaries of each Medicare plan. Each plan has a carrier (i.e. the insurance company), a plan name and a description of the plan. In my sample data set, I have 5 plans from Blue Shield. If I ask a prompt like "Make me a bulleted list of all of the Blue Shield plans." It will only return a list with two plans.

My follow up prompt is "But there are 5 plans from Blue Shield I need a list that shows all of them. Please redo.". The bot will then create a list with 5 plans ... which is better, but the list is wrong.

I am using Generative Topics and High Precision.

How do I improve the accuracy of the answers.

Categories:

General topics

I have the same question (0)

All responses (3)

Answers (0)

Sort by

ronaldwalcott 3,866 Moderator on at

Like (0)

Report
Copy link

Link copied!

If I were you when it comes to using data that can be structured I would structure it as best as I can.

e.g. In your example I would have a SharePoint table which would list the plans, policies, etc.

These tools are ok for interpretating unstructured data but structured data tends to improve results with queries that are of the form list this where so and so is equal to so and so.

Was this reply helpful? Yes No
Suggested answer

JF-03111258-0 4 on at

Like (0)

Report
Copy link

Link copied!

Make sure the option to have the LLM use its ‘own knowledge’ is turned off.

Was this reply helpful? Yes No
Suggested answer

Vinoth Selvam 1,592 Super User 2025 Season 1 on at

Like (0)

Report
Copy link

Link copied!

Hi WF-19092047-0 ,

Double-check that each document has a consistent format and labeling for the carrier and plan name fields. If fields are labeled differently or if similar terms are used interchangeably (e.g., "HMO" vs. "Health Maintenance Organization"), the bot may miss certain entries.

Generative Topics and High Precision may sometimes cause the bot to focus too narrowly, potentially leading it to summarize rather than list all items. Consider experimenting with lower precision or broader topic scope to see if the bot can retrieve more complete information.

Providing detailed prompts might help it avoid summarizing or misinterpreting the request. Configure your topics with proper details, inputs and descriptions for the LLM to understand the context more.

thanks.

Was this reply helpful? Yes No