Unanswered

Manually triggering the End of Conversation Topic and capturing analytics

(1) Share

Report

Posted on by IanMcD

I have read the articles about capturing the analytics from the BOT. The resolution rate drivers seems to be driven by "the last non-system topic triggered". I can see why this makes sense when the BOT offers an answer to a question and then redirects to the "End of Conversation" topic, but I am attempting something different and having a few struggles!

The users of my BOT shall ask many questions in a day. I don't require/expect feedback for every answer provided, so want to avoid prompting them every time. It would drive them insane.

Instead I want to provide the answer and await the next question. If for any answer given the user wishes to leave feedback, I would like them to enter a phrase such as "leave feedback" and that would then trigger the end of conversation (EOC) topic for CSAT rating etc.

I have tried creating a "feedback" topic that redirects to the EOC topic, but the feedback topic is then "the last non-system topic triggered", so it appears in the analytics. I don't want that. I want the topic that outputted the last delivered answer.

I have tried modifying the EOC topic so that it is triggered by the "leave feedback" phrase. But by modifying the topic it no longer appeared in the system list, so it became "the last non-system topic triggered".

I know I'm fighting against the intended flow, but I'd like the behavior I describe above. Anyone have any ideas?

Thanks,

Ian

Categories:

Bot analytics

I have the same question (0)

All responses (12)

Answers (1)

Verified answer

HenryJammes on at

Like (2)

Report

Hi @IanMcD,

That's a great topic, and definitely something that is changing now that we interact with large language models as opposed to traditional chatbots.

I tweaked the way generative answers work by always sending the generated answer through the end of conversation topic, as a question. That way, I always reach a point in the conversation where I assume the question is answered and the session is resolved.

To display the generated answer, I also wrap it in an adaptive card, so that the user can at any time say if they are OK or not with the answer. Clicking 👍 and 👎 actually act as if the user had answered the traditional "Did this answer your question?" question of the end of conversation topic. Saying anything else will loop back into the Conversational Boosting topic (generative answers).

Here's an example where I ask a first question, a follow up question, and then click 👍.

For now, the only issue I have with this approach is that chat sessions that don't trigger any topic (i.e. an intent recognition event) and go to Conversational Boosting / Fallback (unrecognized intent), and these are not recorded as engaged sessions (so, neither resolved or abandoned). But this will be fixed soon.

Happy to share the bot as a sample, if you'd like.
Henry

Was this reply helpful? Yes No
IanMcD 26 on at

Like (0)

Report

@HenryJammes - thanks for the response. I like your solution from the users point of view. Essentially the question is still offered after each answer, but it is much more of a background feature, rather than breaking up the conversation flow. Pretty neat.

For my use case, it is unrelated to the conversational boosting, I have many topics each representing a business problem. The triggered topic calls our custom API to get a suggested answer. I hope to still pursue your above approach, but my "many topics" are the ones I want to appear in the resolution rate drivers.

I'll be returning to this soon. Currently have myself in a pickle where I have managed to delete the end of conversation topic, so currently trying to overcome that! (If you change the trigger for that system topic to be Phrases it seems to lost its system status)

Was this reply helpful? Yes No
HenryJammes on at

Like (2)

Report

Thanks @IanMcD -- yes, I believe we use the trigger type to determine what is system or not.
So, End of conversation can appear back in the list of system topics if you change its trigger to OnSystemRedirect, in the YAML code editor view.

Was this reply helpful? Yes No
IanMcD 26 on at

Like (0)

Report

@HenryJammes - that rescued me thanks. I was hoping your tip about the trigger type may have helped me for the feedback logic. I tried changing the End of Conversation topic to type "
OnSystemIntent" with my feedback phrase. It returned to the system list, but still seemed to take the spotlight in the resolution rate drivers list.

I'm trying to pursue your approach now. You offered to share your bot as a sample, could I take you up on that please? I haven't done much with adaptive cards as yet, so that is the gap I'm looking to fill. How to drive the feedback process from your thumbs up/down.

Was this reply helpful? Yes No
HenryJammes on at

Like (2)

Report

Sure, I packaged the bot in an unmanaged solution that you can import as is.
You can find it here: PowerVirtualAgentsSamples/SampleBotSolutions/MicrosoftLearnChatbot_1_0_0_1.zip at master · microsoft/PowerVirtualAgentsSamples (github.com)

Was this reply helpful? Yes No
IanMcD 26 on at

Like (0)

Report

Thanks for that, it definitely helps. I should perhaps have mentioned from the start that I'm developing for a Teams channel. Your approach is now working in test environment, but when used in Teams clicking the "thumbs up" just tells me "Your response was sent to the app". (I know there are issues showing markdown for tables in the adaptive card too but I can work around that).

I'll do some more reading on this and plug away at it myself. But your approach has definitely unblocked me so thank you for the help!

Was this reply helpful? Yes No
HenryJammes on at

Like (0)

Report
Ah yes - in Microsoft Teams you need to tweak the adaptive card a little.
Instead of a simple:
"data": "The generated answer was useful"

You actually need to send an record in an array with specific properties for msteams.
It's a bit buried but it's documented here: Add card actions in a bot - Teams | Microsoft Learn

I got it to work in this adaptive card format, let me know if that helps:

{ type: "AdaptiveCard", '$schema': "http://adaptivecards.io/schemas/adaptive-card.json", version: "1.5", body: [ { type: "Container", items: [ { type: "TextBlock", text: Global.Answer, wrap: true } ] }, { type: "Container", items: [ { type: "ColumnSet", columns: [ { type: "Column", width: "90", items: [ { type: "TextBlock", text: "AI-generated answer. Please rate it", wrap: true, size: "Small", color: "Accent" } ] }, { type: "Column", width: "5", items: [ { type: "TextBlock", text: "👎", wrap: true, size: "Small", color: "Light" } ], selectAction: { type: "Action.Submit", data: { msteams: { type: "messageBack", text: "This generated answer wasn't useful", value: "This generated answer wasn't useful" } } } }, { type: "Column", width: "5", items: [ { type: "TextBlock", text: "👍", wrap: true, size: "Small", color: "Light" } ], selectAction: { type: "Action.Submit", data: { msteams: { type: "messageBack", text: "This generated answer was useful", value: "This generated answer was useful" } } } } ] } ] } ] }

Was this reply helpful? Yes No
IanMcD 26 on at

Like (0)

Report

Thanks, that has it working for Teams now. Not sure why, but I had to change the type from messageBack to imBack.

BUT now I'm doubting if I can use it. We have the question in the End of Conversation topic. It expects a response. If they do not use the thumbs up/down for your example you bring them through to the conversational boosting. This makes sense, because that is where all the questions are targeting.

But if they type 'start over', that is also going to conversational boosting. For me it is more likely that they have entered the next question which is intended to trigger another topic.

The question is marked to allow switching to another topic but it doesn't seem to work.

Was this reply helpful? Yes No
HenryJammes on at

Like (0)

Report

True, this can be an issue.
Topic interruption actually doesn’t work when you capture the user’s entire response.
In the past, one way I had managed this was by also offering as a quick reply an escape message (e.g. OK I need something else), that would end the conversation (so, an additional path in your condition).

Was this reply helpful? Yes No
IanMcD 26 on at

Like (0)

Report

Your bit of guidance has again pointed me in the right direction. I changed the variable to a closed list of Useful/NotUseful. Now working as hoped, thank you sir!

Was this reply helpful? Yes No