Unanswered

Integrate a PVA chatbot with Azure OpenAI using your data - TBD

(0) Share

Report

Posted on by angerfire1213

First, I use the method from @HenryJammes ,It's good for my prod env to use PVA ChatBot to answer Employee questions,But sometime someone input key words it not good, and will get not good answer from my bot.

Two weeks ago,I read Azure OpenAI on your data (preview)This way support semantic search.

Maybe, It's a better way for use PVA chatBot answer your Employee questions.

Here is my Step to use Azure OpenAI on your data (preview) in my PVA:

1. I follow Quickstart: Chat with Azure OpenAI models using your own data complete the setting and upload my data.

2. I Create an Topic named "Cognitive search"

Screenshot 2023-07-07 at 9.02.41 AM.png

3. I create Power automate flow in step 2's action

Screenshot 2023-07-07 at 9.07.37 AM.png

4.I trigger the flow and got output from Azure Cognitive search

{
 "id": "12345678-1a2b-3c4e5f-a123-12345678abcd",
 "model": "",
 "created": 1684304924,
 "object": "chat.completion",
 "choices": [
 {
 "index": 0,
 "messages": [
 {
 "role": "tool",
 "content": "{\"citations\": [{\"content\": \"\\nCognitive Services are cloud-based artificial intelligence (AI) services...\", \"id\": null, \"title\": \"What is Cognitive Services\", \"filepath\": null, \"url\": null, \"metadata\": {\"chunking\": \"orignal document size=250. Scores=0.4314117431640625 and 1.72564697265625.Org Highlight count=4.\"}, \"chunk_id\": \"0\"}], \"intent\": \"[\\\"Learn about Azure Cognitive Services.\\\"]\"}",
 "end_turn": false
 },
 {
 "role": "assistant",
 "content": " \nAzure Cognitive Services are cloud-based artificial intelligence (AI) services that help developers build cognitive intelligence into applications without having direct AI or data science skills or knowledge. [doc1]. Azure Machine Learning is a cloud service for accelerating and managing the machine learning project lifecycle. [doc1].",
 "end_turn": true
 }
 ]
 }
 ]
}

not complete yet, and will update.

Categories:

General topics

I have the same question (0)

All responses (19)

Answers (0)

Sort by

HenryJammes Microsoft Employee on at

Like
a
(1)

Report
Copy link

Link copied!

Hi @angerfire1213 awesome to see you're working on this!
FYI @remidyon managed to get a custom integration working 👀

Was this reply helpful? Yes No
remidyon Microsoft Employee on at

Like
a
(2)

Report
Copy link

Link copied!
Hey @angerfire1213 , like @HenryJammes mentioned I was able to connect my PVA directly to Azure OpenAI Studio. Same as you for the first step, create the blob storage, indexes with azure cog. service, activate semantic search, etc. up to the point where it works in Azure OpenAI Studio Playground (here I uploaded a bunch of fake healthcare plans PDF in cog. service):

Then I created a Power Automate flow to call the Azure OpenAI Studio API directly and this is where it gets tricky: you need 3 URLs and 2 API Key to make this work!

The documentation for this is a little bit spread everywhere, I had to read all the following docs to make it work:
Using your data with Azure OpenAI Service - Azure OpenAI | Microsoft Learn
Azure OpenAI Service REST API reference - Azure OpenAI | Microsoft Learn
Create a Cognitive Services resource in the Azure portal - Azure Cognitive Services | Microsoft Learn
Azure OpenAI Service embeddings tutorial - Azure OpenAI | Microsoft Learn
How to work with the ChatGPT and GPT-4 models - Azure OpenAI Service | Microsoft Learn

I will save you some time and give you my HTTP request detail:
Notes:
- the API version in the URI is important, otherwise the API call will fail. Same for the chatgpt_url.
- the "extensions" part in the main URI is also required (but not in the chatgpt_url!!), this was also something that caused me troubles
- the 2 keys in the header are the same, but you need a different one in the body of the call (yes - this is tricky)

And here is the body:

{ "dataSources": [ { "type": "AzureCognitiveSearch", "parameters": { "endpoint": "YOUR_AZURE_COG_SEARCH_URL", "key": "YOUR_AZURE_COG_SEARCH_KEY", "indexName": "YOUR_AZURE_COG_SEARCH_INDEX" } } ], "messages": [ { "role": "user", "content": @{triggerBody()['text']} } ], "temperature": 0, "top_p": 1, "frequency_penalty": 0, "presence_penalty": 0, "max_tokens": 800, "stop": null }

After that I am able to parse the JSON in Power Automate, extract the result and pass that back to PVA (I am unable to make the link to the source work though - maybe that is something I can improve in the future)

Here is the result of "Azure OpenAI on your data" in PVA:

Was this reply helpful? Yes No
angerfire1213 90 on at

Like
a
(0)

Report
Copy link

Link copied!
Hey @remidyon,Thank you for great share! It's work fine in my env!
And hope link to the source!
I use select and expression try to do this but looks not good :
From:
take(split(outputs('Compose'),'metadata'),add(length(split(outputs('Compose'),'metadata')),-1))
Link:
take(concat('https',last(split(item(),'https'))),add(length(concat('https',last(split(item(),'https')))),-4))

Was this reply helpful? Yes No
angerfire1213 90 on at

Like
a
(0)

Report
Copy link

Link copied!

Hi @remidyon cloud you please share your "parse JSON","init Var","Apply to each" in the PA Follow?

Was this reply helpful? Yes No
remidyon Microsoft Employee on at

Like
a
(2)

Report
Copy link

Link copied!
Hi @angerfire1213

Here are the detailed steps:

ParseJSON is to extract the data from HTTP call.

Here is the Schema I used:
{ "type": "object", "properties": { "id": { "type": "string" }, "model": { "type": "string" }, "created": { "type": "integer" }, "object": { "type": "string" }, "choices": { "type": "array", "items": { "type": "object", "properties": { "index": { "type": "integer" }, "messages": { "type": "array", "items": { "type": "object", "properties": { "index": { "type": "integer" }, "role": { "type": "string" }, "content": { "type": "string" }, "end_turn": { "type": "boolean" } }, "required": [ "index", "role", "content", "end_turn" ] } } }, "required": [ "index", "messages" ] } } } }

And here is the detail for the loop:

But no need to bother with all this anymore: Microsoft just released the ability to deploy Azure OpenAI Studio "chat with your data" directly into PVA!
Create Generative AI solutions with Power Virtual Agents and Azure OpenAI Services | Microsoft Power Virtual Agents

Was this reply helpful? Yes No
angerfire1213 90 on at

Like
a
(1)

Report
Copy link

Link copied!

Wow! Big thank you @remidyon ! love it and can not wait to try it! https://powervirtualagents.microsoft.com/en-us/blog/create-generative-ai-solutions-with-power-virtual-agents-and-azure-openai-services/

Was this reply helpful? Yes No
angerfire1213 90 on at

Like
a
(0)

Report
Copy link

Link copied!

@remidyon Does it support PVA （classic）
https://powervirtualagents.microsoft.com/en-us/blog/create-generative-ai-solutions-with-power-virtual-agents-and-azure-openai-services/

Was this reply helpful? Yes No
HenryJammes Microsoft Employee on at

Like
a
(1)

Report
Copy link

Link copied!

As @remidyon said, Create Generative AI solutions with Power Virtual Agents and Azure OpenAI Services | Microsoft Power Virtual Agents is now the way to go 🙂

@angerfire1213 the native integration between Azure OpenAI on your data and PVA only supports the new unified authoring version. In general, no new features should be expected on the classic version of PVA.
For classic bots, you need to adopt a custom approach to integrate Azure OpenAI, through Power Automate cloud flows or HTTP requests in Bot Framework Composer dialogs.

Was this reply helpful? Yes No
AbigailCuadra 15 on at

Like
a
(0)

Report
Copy link

Link copied!

Hi @HenryJammes , I have connected Azure Open AI with my own data stored in Azure Blob Storage to Power Virtual agent and it works, however, the reference URL added to each responses is getting my blob storage path of each document which is not accessible properly by the users. How can we replicate the behaviour of Azure Open AI Chat playground where citations are getting shown properly to the users?

Was this reply helpful? Yes No
HenryJammes Microsoft Employee on at

Like
a
(0)

Report
Copy link

Link copied!

Hi @AbigailCuadra

There is work in progress to support citation management, but this will of course depend on the channel and where you deploy your PVA bot.

Henry

Was this reply helpful? Yes No