Copilot Bot responses differ from those in Azure Open AI Studio

(0) Share

Report

Posted on by SGbron

Hello,

I'm grappling with an intriguing behavior from a Copilot bot deployed through Azure Open AI Studio.

In my project within Azure Open AI Studio, I've trained the model using a mix of documents, with seven in German and one in English, sourced from an Azure Blob Storage container. During testing in the Azure playground, the bot provided excellent responses to questions in both German and English documents, showcasing its versatility.

Upon deploying the solution in a web app, the results remained consistent and satisfactory. However, when attempting to deploy the same solution in Copilot, the bot seems to struggle. It consistently responds with "I’m sorry, I’m not sure how to help with that. Can you try rephrasing?" even to straightforward prompts like "What information can you provide me?"

What perplexes me is this: when I deploy a bot trained solely on English documents in Copilot, it performs well. However, when the dataset includes German documents and one English document, the bot fails to respond coherently, even to prompts related to the English document.

Does anyone in the community have insights into this behavior? Any help or suggestions would be greatly appreciated.

Categories:

General topics

I have the same question (0)

All responses (7)

Answers (0)

citron-truc 394 on at

Like (1)

Report

Good morning,

The answer from the bot could hide a weirder / more problematic issue. In the way pva works, the bot sends the message "I’m sorry, I’m not sure how to help with that. Can you try rephrasing?" if the answer given is empty. This could also mean that there is a problem in you connection or that the moderation blocked the answer.

With foreign languages the moderation blocks a lot of answers, try to set the moderation of answers to low.
Also, for the life of me, I have never managed to make gpt-3.5-turbo work on pva. Try using gpt-3.5-turbo-16k instead.

Have a great day !

Was this reply helpful? Yes No
SGbron 27 on at

Like (0)

Report

Hello @citron-truc .

Thank you very much for your response. I have tried already this approach with the content moderation. I have tried all three posible choices but nothing changed.

Also I am using gpt-4 and gpt 4 32K ... So according the documentation the results had to be much more accurate. But this is not happening on my case.

Was this reply helpful? Yes No
citron-truc 394 on at

Like (1)

Report

Hello @SGbron,

I'm sorry to hear that it is not working.

Another suggestion (a bit far fetched maybe) : when working in the azure openai playground, keyword matching is used by default whereas semantic search is used by default in copilot studio.
To my mind, keyword extraction doesn't need to know the language to extract important documents but semantic search does. If you have a multi-lingual corpus of documents, maybe giving the language in the metadata might help ? https://learn.microsoft.com/en-us/azure/search/search-language-support

Was this reply helpful? Yes No
HenryJammes on at

Like (1)

Report

When configuring Azure OpenAI 'on your data' as a data source in generative answers, the generative answers capability actually adds its own summarization layer -- so it will summarize what your Azure OpenAI model returns.
We call this out in the aka.ms/CopilotStudioImplementationGuide

Even though it may sound counter intuitive, this is because generative answers supports multiple data sources, so for example if you have public website + Azure OpenAI on your data content, generative answers can provide a single answer with content from both sources.

For customers who don't want this behavior, they can do a custom integration with the HTTP request node (or through Power Automate). In the future, we're looking to support using the Azure OpenAI connector directly in the Copilot Studio canvas. We're also looking at exposing the last 10 turns as a variable for grounding any new question in past interactions, so that follow up questions are interpreted in context.
Would that help?

Henry

Was this reply helpful? Yes No
SGbron 27 on at

Like (0)

Report

Hello @citron-truc

To tell you the truth, I used both, Keyword and Semantic Search indexes and I didn't saw any difference in the responses. I thought that semantic search was the correct one according the recomendation from documentation Use your own data -Search Options for German Language but nothing changed insead of keaword search. From the other hand I saw a few articles that has no resources but mentioned that Copilot uses keaword search and not semantic. As you understand I am a little bit comfused. From the other hand, now I am trying to create a separate index for each language .

I will also check how to give the language in the metadata from the documentation that you mention.

Was this reply helpful? Yes No
SGbron 27 on at

Like (0)

Report

Hello @HenryJammes

I will read this document and I hope I will get some clarifications becasue I am really comfused.

About your last paragrahph and the future feature, yes its seems to be very intresting to have a connector dirrectly in the Copilot Studio ...

Was this reply helpful? Yes No
ashkanrad 10 on at

Like (0)

Report

I had the same issue and changed the "Query type" from "semantic" to "simple" and it started working.

Was this reply helpful? Yes No