Unanswered

Improving Chatbot Accuracy and Handling Uncertain Answers

(0) Share

Report

Posted on by MSPPQ

Hi All,

I have created a chatbot with a link to a knowledge base as the data source. The chatbot works well; however, in many cases, it makes up answers that are not correct, which makes it less reliable. I have set the moderation level to high (which is the default), and I have added a prompt at the "Create Generative Answers" topic's "Content Instructions" setup instructing the bot NOT to make up answers when it is not 100% sure, but this is not working.

I'm reaching out to the community to seek advice on improving the accuracy of the chatbot's answers. If you've succeeded in using the "content instructions" feature, could you share which prompts worked effectively for you?

If the bot is not 100% sure of an answer, I would like it to let the user know it cannot answer that question.

Thanks in advance for your help!

Categories:

General topics

I have the same question (0)

All responses (3)

Answers (0)

JT-28081628-0 6 on at

Like (0)

Report

Will let others answer with more specifics, but it's my understanding that 100% is not achievable using a generative AI model even with RAG and custom instructions.

Was this reply helpful? Yes No
citron-truc 394 on at

Like (1)

Report

Hello !

I hope you are doing well. If you want your chatbot to have deterministic answers, you should set the temperature & top_p to low values (works only for azure openAI models, not for copilots unfortunately).

From what I understood, content instructions are used to give context and ask for specific answer forms ("Pretend you are a pirate and start every answers with 'shiver me timbers'"). Giving a few example questions and answers might help your chatbot (see this : https://platform.openai.com/docs/guides/prompt-engineering/strategy-write-clear-instructions)

When you use generative AI on your data, there is an intermediary step where an indexer ranks documents according to how relevant they are to the question. There is then a hidden prompt where we ask the generative AI "given these five documents and this question, find a suitable answer". The AI doesn't know the exact probabilities of each document matching the question and therefore always "feel" 100% sure. Telling it to not answer if it isn't sure won't help on this one.

Hope it helps, have a great day.

Was this reply helpful? Yes No
CleanFox 20 on at

Like (1)

Report

Telling a LLM Bot something like "Dont", "Do Not", "Never", is generally considered bad practice.
It would be better to enforce the "Do's" instead of the "Dont's".
The "Do's" is like telling you to do a task.
Using "Do's" and "Dont's" is like telling you to do a task, but do NOT think about trains (now you will probably think about trains).

I can't guarantee better results, but I think this should at least improve your models accuracy.

Was this reply helpful? Yes No