Streaming Text from Copilot Studio to Teams

(1) Share

Report

Posted on by Stefan3000

Hi everyone,

I'm currently in the process of rolling out GenAI via Teams to my company.

I have released two apps so far:

First App: A simple GPT-4 connection for 1-to-1 chat via Teams and Azure OpenAI. For this, I utilized Azure OpenAI Services, CoPilot Studio, and Power Automate to manage aspects like chat history and session-based responses.
Second App: An Azure Prompt Flow Endpoint that utilizes the "Chat with your own data" service.

Both apps employ Power Automate to establish the connection with the LLM.

However, I'm facing two issues:

The responses sometimes take more than 2 minutes, leading to a timeout.
I would like to implement streaming text responses to enhance the user experience.

I have already found the KB article on setting up the Azure ML endpoint: https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-enable-streaming-mode?view=azureml-api-2

Setting this up is not a big issue, but how do I adapt the requirements for CoPilot Studio and Power Automate to handle the streamed texts?

Has anyone already done this, or is there another way to achieve it?

I look forward to your answers, and thank you very much!

Stefan

Categories:

Building Copilot Studio chatbots in Microsoft Teams

I have the same question (0)

All responses (5)

Answers (0)

ManikandanS 246 on at

Like (1)

Report

Hi @Stefan3000 ,

We have also undergone the same situation and don't think we have a way to show streaming response using power automate and co pilot studio. What we did from our side is we checked the other layers of system like Azure Open AI, Azure Search to speed up the overall turnaround time, thus improved our state and now we managed to send response to user question without timeout issues 99% of time.

Was this reply helpful? Yes No
Stefan3000 5 on at

Like (0)

Report

Hi @ManikandanS ,

thanks for your reply.
We have a RAG-powered Azure Endpoint in place, which I can't get under 10-15% timeout rate.

I have found a way to handle long HTTP calls:
https://sergeluca.wordpress.com/2018/02/01/pattern-for-making-long-async-http-calls-in-microsoft-flow/

Unfortunatly the way which describes how to get the URI of the specific action doesn't work.

Is their a new way to get the URL of the HTTP action, so that I can build a loop to check if the status is 200 or 202?

Thank you!

Does anybody know how to get the URL of a

Was this reply helpful? Yes No
HenryJammes on at

Like (1)

Report

Hi @Stefan3000

Copilot Studio doesn't support streamed messages -- at least not in its current architecture.
If this is an important requirement, may I ask that you submit a feature request detailing the business case, here: aka.ms/CopilotStudioFeatureRequest?

Henry

Was this reply helpful? Yes No
ManikandanS 246 on at

Like (0)

Report

@Stefan3000 - We are also having similar setup like you I assume, for each user question we have 1 Azure open AI call + Azure Search + 1 Azure open AI call again, please check these areas you should be able to get it.

Was this reply helpful? Yes No
jamesbio 29 on at

Like (0)

Report

Hi @HenryJammes, I went ahead and submitted using the link you shared here.

Does it need to be reviewed and then updated by staff before showing:
https://portal.productboard.com/fdeco3ykgkwvchtg4qbdowug/tabs/1-under-consideration

How do I know if the idea is actually being reviewed? Once it is added, I would like to share with other colleagues and developers at other large companies so they can upvote.

Was this reply helpful? Yes No