Unanswered

Generative Answers with SharePoint provides false citations

(0) Share

Report

Posted on by TP_Phil

I created a topic for generative answers and added a SharePoint site. If I ask questions about topics from documents (regarding self-written dynamics manuals) inside the SharePoint site, it provides me a good answer and the correct citation. However, if I ask a question about other Microsoft related topics which are not covered by the manuals (e.g. How can I upload a PowerBI report?) it is providing me the correct answer, BUT it provides a made up citation of the provided SharePoint documents. I know that LLMs do tend to hallucinate, but this is kind of weird. Does somebody have made similar experiences? Any ideas how to fix that? Or is it just the hallucinations of the LLMs?

Categories:

General topics

I have the same question (0)

All responses (3)

Answers (0)

Sort by

HenryJammes Microsoft Employee on at

Like
a
(0)

Report
Copy link

Link copied!

This is indeed the nature of large language models. Copilot Studio adds a number of controls to validate that the generated answer is sourced in the input data/documents, but it can't guarantee that answers are always accurate. To have the highest level of moderation controls, you should leave that setting on 'High'.

Was this reply helpful? Yes No
joaomelou4 3 on at

Like
a
(0)

Report
Copy link

Link copied!

So that's it? no suggestions on how to mitigate or avoid this?

Was this reply helpful? Yes No
citron-truc 394 on at

Like
a
(0)

Report
Copy link

Link copied!

In order to create an answer, you give the content of the document to the AI and in the background, there is a prompt that looks like that in much more complicated : "Here are a lot of documents, could you answer this question with the documents ? Don't answer anything false".
Since the model already knows your answer (it was trained on lots of data pre-2022), it answers.
Generative AI is still a bit of a novelty (ChatGPT became available in december 2022, Azure OpenAI in june 2023 and Copilot in november 2023. Everybody is racing to integrate generative AI).
Right now, moderation is the best method to avoid out of context answers. Technology is improving. Hopefully, we'll soon be able to get more control on generative AI and how to personalize them.

Was this reply helpful? Yes No