Intelligent Copilot with Generative Answers, knowledge and LLM Orchestration

Hello everyone.

I am trying to build a Copilot that would be able to:

Understand the user's question or query in a way a LLM can (such as ChatGPT)
Decide on which topic to select, based on the user's query and natural language understanding
Answer with the Generative Answers feature, based on a certain restricted set of documents
AND it should be able to understand one or more follow-up questions, remaining in the current topic, until it understands that the user is not talking about the current topic anymore

For this, I have been using both Generative Answers on uploaded knowledge (SharePoint and uploaded files), and LLM Orcherstration, where a trigger is not defined by phrases, but by a description of when a certain topic should be triggered.

For example:

I have 2 documents: one that has information about a company's work-from-home policy, and another that has information about expenses and reimbursements for specific clients (X and Y).
Therefore I create a topic for work-from-home (containing the first document), and another topic for expenses and reimbursements (containing the second document). I write a good description for when each topic should be triggered. Each one uses generative answers to answer, but based on a different document.

Here's how it should work:

The user asks a question: "How do I get reimbursed for my 3 day trip to see client X?"
The Copilot understand the question is related to topic expenses and reimbursements, goes inside there, scans the knowledge, and answers the question.
The user then asks "What about client Y for the same duration?"
The Copilot, following the flow of the conversation, also understands that this question is related to topic expenses and reimbursements. Following the previous question, it scans the same knowledge and answers the question.
The user now asks "I need to work from home 5 days this month, is that ok?"
The Copilot now understand the question is related to topic work-from-home, therefore goes in that topic and answers accordingly

However, for the second question "What about client Y for the same duration?", the Copilot will not be able to classify this into either topic because it doesn't explicitly mention either topic, but acts as a follow up question to another. The best way would be to pass the whole chat into the copilot so it has context and understanding of the discussion, much like ChatGPT does, for example.

Please keep in mind that this is a simple example and the Copilot I'm trying to build is much more complex than that.

The classic way I've seen people use Generative Answers with knowledge, is by using the "On unknown Intent" trigger, and just gather all the knowledge together in that one node. In this case, the second question would also go into the "on unknown intent topic".
The main problem with this is that the Copilot will frequently hallucinate and answer differently, often pulling knowledge from the bad document on an answer that was quite straightforward. For example, in this example, there was a document called "work-from-home policy" in which it was specified that employees can work from home 50% of the time, but instead it pulled knowledge from another unrelated document about unpaid leave, and this was even with high content moderation !!

In short: is it possible to build this type of intelligent Copilot, that retains context, understands queries, and answers based on existing knowledge, today with Microsoft Copilot Studio? Or should I be looking at other options?

Categories:

General

Quick Links

Leaderboard