Answered

Best approach to use Teams feedback on Copilot Studio responses for knowledge refinement

(1) Share

Report

Posted on by HB-09041311-0

Hi everyone,

I'm working on a Microsoft Copilot Studio agent integrated with Teams and looking for ideas on implementing a feedback-driven knowledge refinement process.

Objective: Capture users' 👍/👎 reactions and comments on agent responses and use that feedback to automatically improve the knowledge base. For example:

Delete or deactivate an incorrect document summary stored in a Dataverse table
Flag knowledge for review
Enrich or regenerate summaries
Notify knowledge owners for manual validation

One approach I'm considering is:

1. User Feedback (Teams)
        ↓
2. Feedback stored in ConversationTranscript table (Dataverse)
        ↓
3. Power Automate trigger (On Dataverse row created)
        ↓
4. Parse & validate feedback
        ↓
5. Execute actions (Purge / Enrich / Notify)

A few questions for the community:

Is Dataverse + Power Automate the best pattern, or are there better approaches using:
- Agent Flows
- Copilot Studio APIs
- Conversation analytics APIs
- Event-driven architecture?
Any recommended patterns for implementing a human-in-the-loop review process before modifying knowledge sources?

I'd love to hear how others have implemented feedback loops and continuous knowledge refinement for Copilot Studio agents.

Thanks in advance!

Categories:

Publish & channel management

Building Copilot Studio chatbots in Microsoft Teams

Bot analytics

I have the same question (0)

All responses (2)

Answers (1)

Sort by

Suggested answer

11manish 3,333 on at

Like
a
(1)

Report
Copy link

Link copied!

For most enterprise Copilot Studio implementations, a combination of Dataverse + Power Automate + Agent Flows is the most maintainable and scalable

architecture. Use Dataverse as the central repository for structured feedback, Power Automate to orchestrate workflows and approvals, and Agent Flows for

AI-driven analysis and content enrichment. Most importantly, avoid automatically modifying or deleting knowledge based on individual user feedback. Instead,

implement a human-in-the-loop review process where feedback contributes to confidence scoring, triggers review tasks, and allows knowledge owners to

validate changes before republishing. This balances continuous improvement with governance, traceability, and content quality.

Was this reply helpful? Yes No
Verified answer

Haque 3,653 on at

Like
a
(1)

Report
Copy link

Link copied!
Hi @HB-09041311-0,

Your initial five steps seems kind of solid.

What else you can do -

1. Let's consider a dedicated AgentFeedback table with explicit columns for reaction, comments, knowledge source ID, and confidence score. This improves clarity and query efficiency.

2. Feedback Validation - you can implement logic to filter out noise (e.g., accidental thumbs down) by requiring multiple negative feedbacks or combining reaction with comments before triggering actions.

Reference:

Agent Evaluation in Microsoft Copilot Studio

How to evaluate AI agents

3. Notification routing is one more thing you can adapt - based on knowledge domain or content owner to ensure timely and relevant manual validation route notifications intelligently.

Quesoin-1/Answer: Is Dataverse + Power Automate the best pattern, or are there better approaches?
Yes, Dataverse + Power Automate is a widely used, robust, and low-code pattern for capturing feedback, triggering workflows, and automating knowledge refinement.

We have other options but to me not that better like above:

Option-1: Agent Flows (native to Copilot Studio) are optimized for real-time conversational logic

Option-2: Copilot Studio APIs (Development Effort) - Agent Flows (native to Copilot Studio) are optimized for real-time conversational logic.

And obviously Converasation analytics APIs and Event-driven architecutre are more advanced development oriented.

Quesoin-2/Answer: Any recommended patterns for implementing a human-in-the-loop review process before modifying knowledge sources?

It dependse - Having a human-in-the-loop review process before modifying knowledge sources is recommended. Hence, introudcint “Request for Information” (RFI) action in Copilot Studio agent flows or workflows to pause automation and collect structured input from human reviewers before proceeding. Also, alternativley we can use the Teams connector "Post adaptive card and wait for response" action to collect human input from Team chats or channels.

References:

Human in the loop: Request information from human review

Recieve feedback : Using adaptive card in team chat/channels.

Feedback Loop: Analyze conversational agents

Agent flow vs Workflows: Agent flows and workflows overview

Soltuion Architecture: Copilot Studio Agetn Lifecycle

The Signals Loop: Fine-tuning for world-class AI apps and agents

I am sure some clues I tried to give. If these clues help to resolve the issue brought you by here, please don't forget to check the box Does this answer your question? At the same time, I am pretty sure you have liked the response!

Was this reply helpful? Yes No