Power Platform Community Forum Thread Details

I have a Copilot created using Copilot Studio, where I used custom instructions for generative answers to fine-tune the persona, tone, and format of the responses. It was working fine until Nov-07, 2024; I started to see that the characteristics specified in the instructions are not consistently followed by the Copilot. I tried changing the instructions multiple times, but the responses are still not reliable compared to the responses that I saw before update.

Also, the way the Actions used to work has changed. For example, when an action gets triggered and executed by the Copilot, the bot used to automatically generate a message with a summary of details the user provided as inputs to the action, but now the Copilot is not doing this, and I have to manually create a message.

I saw some updates in the Studio on Nov-06, 2024, but the major issues were noticed on Nov-07, 2024.

I have a Loan Copilot that was designed to help users get answers to loan queries, apply for and manage loan requests.

Samples:
1. In the below sample, when we asked for a question that has a positive tone.
  In older version, the Copilot answers with positive emotion :
  
  In newer version, there is no tone to the answers:
2. Below sample is to show the translation capability, we instructed the Copilot to use same language the user used for generating the response.
  In older version, it worked as expected:
  
  In newer version, it didn't follow the instruction and gave the response in English:
3. Below sample is to show the Action is not working the same. We have an action that inserts a row into a table,
  In older version, the bot generated a summary after the action gets executed:
  
  In newer version, there is no summary message like this.
4. And in many cases the there is a problem in Copilot triggering the actions/topics.
  For example, I had a simple cloud flow "Check Credit Score" that retrieves credit score from dataverse, this is configured as a AI Action in Copilot with proper name, description, inputs, outputs, credentials and instructions yet Copilot cannot trigger this action even with a most direct questions like "check credit score", "i want to check my credit score", etc.

These are few of many issues that we have faced. Kindly help me resolve this issue.

Thanks in advance.

Categories:

General topics

Hello,

as you probably already know - the cause is the update I guess. The only option to fix this is to alter the behaviour of the chatbot to new functionalities:

1. Try to manipulate the input prompt/instructions to enforce positive tone. Probably there are metaprompts which are added to the system message and are doing the mess. You can also try to instruct the model to perform a role, if it's not applied yet.

2. Probably it's due to metaprompts included in the system message. You've created a bot in English as a primary language? Tt must be included somewhere in the system message. Overall, it's not a good way to enforce multiple languages on the chatbot - it's not going to be stable and can lead to unexpected results as you already see.

3. I don't know how the summary message was generated - was it a separate generative answers action with the input from an action? You'd need to prompt engineer the input to be properly formatted, so that the model can work on it again.

In case of any other questions, let me know. If the answer helped you, mark it, so that others can benefit from it.

Best regards,

Artur Stepniak