Integrate a PVA chatbot with Azure OpenAI ChatGPT using the Chat Completion API format

[HenryJammes]

OpenAI ChatGPT and GPT-4 models are both optimized for conversational interfaces. This means they accept input formatted as a conversation. The main benefit is retaining the conversation context (i.e. the history of past questions and answers), so that any new follow up question is made in the context of past interactions.

Example in ChatGPT, where the question "Who was it before?" only makes sense in the context of the previous questions and answers: 

To replicate this behavior in Power Virtual Agents, you integrate with the Azure OpenAI Service APIs using a Power Automate cloud flow.

For each of the requests you make to Azure OpenAI, you need to retain the context of previous questions and answers. 

The Chat Completion API is a new dedicated API for interacting with the ChatGPT and GPT-4 models in conversational manner. 

Chat Completion is optimized and will lead to better results than the previous Chat Markup Language (ChatML) format.

The format allows you to set the system prompt (with instructions, rules, data, etc.) as well as a format for the following question and answers, between the bot and the user. It's formatted in JSON.

Below is an example of the above example conversation:

[
    {
        "role": "system",
        "content": "You are an AI assistant that helps people find information."
    },
    {
        "role": "user",
        "content": "Who was Microsoft CEO in 2020?"
    },
    {
        "role": "assistant",
        "content": "In 2020, the CEO of Microsoft was Satya Nadella. He has been the CEO of Microsoft since 2014."
    },
    {
        "role": "user",
        "content": "Who was it before?"
    },
    {
        "role": "assistant",
        "content": "Before Satya Nadella, Steve Ballmer was the CEO of Microsoft. He served as the CEO from 2000 to 2014, following Bill Gates who co-founded Microsoft and served as CEO for many years."
    }
]

You can learn more about Chat Completion API here:

How to work with the ChatGPT and GPT-4 models (preview) - Azure OpenAI Service | Microsoft Learn

You cannot keep an infinitely large history of past interactions. Each model has a token limit. 

You can learn more on how to remain the thoken limit here.

In the below example, I show how to integrate Power Virtual Agents with Azure OpenAI with Power Automate, and use the Chat Completion API to retain conversation context.

• In the first post, I show how to do this using the Power Virtual Agents new unified authoring canvas.
• In the second post, I show how to do this using the classic version of Power Virtual Agents (using Bot Framework Composer).

Important note:

• There are existing native GPT-based capabilities in Power Virtual Agents.
• Be sure to check them out before buidling your own custom logic:
  
  • Conversation Boosters (public website and internal document search): Boost conversations (preview) - Power Virtual Agents | Microsoft Learn
  • Copilot: Create (and edit) with Copilot (preview) - Power Virtual Agents | Microsoft Learn

Prerequisites for the below articles:

• Power Virtual Agents web app.
  Documentation: Overview of Power Virtual Agents web app - Power Virtual Agents | Microsoft Learn
• Azure OpenAI Service with a ChatGPT (gpt-35-turbo) model deployed.
  Documentation: What is Azure OpenAI Service? - Azure Cognitive Services | Microsoft Learn

[angerfire1213]

Dear @HenryJammes , I got an error result as follow 

[HenryJammes]

---

@angerfire1213 wrote:

Dear @HenryJammes , I got an error result as follow 

---

My mistake @angerfire1213 , I think you need to use single quotes.

[Samuel-Zhou]

@HenryJammesThank you for sharing. Have you tested whether it is possible to directly send code to you for parsing, such as Python code? I have tried before, but parsing JSON will fail.

[HenryJammes]

I haven't tried with Python. What fails with JSON and where?

[Estee]

Hi @HenryJammes, 
Thank you for you post and detailed explanations.
I'm facing the issue of reaching the token limits and I twas thinking of different possibilities to fix it, but not sure what would be the easiest to implement in PVA (I'm used to python but new to PVA).

- Remove the oldest messages in the conversation transcript once the token count is reached. Every iteration, the oldest message is removed and so on.

- Reset the Global.FullDialog variable once there is no user activity for around 2 minutes. We can imagine a conversation between a user and the agent, and after 2 minutes of inactivity, the agent asks for user staisfaction, if the user is satisfied, the conversation start over on. Hopefully, there will be very few situations where the user is too chatty so that the token limit is reached.

- Inform the user the limit is reached and that he will have to start over but I think this option is a bit frustrating for the user.

What do you think? Thank you in advance for your help !

[Samuel-Zhou]

@HenryJammes Hi,I find that every time it seems to be an error in the Power Automate apply to each,the error code is as follows:

The execution of template action 'Return_value(s)_to_Power_Virtual_Agents' is failed: the client application timed out waiting for a response from service. This means that workflow took longer to respond than the alloted timeout value. The connection maintained between the client application and service will be closed and client application will get an HTTP status code 504 Gateway Timeout.

[Samuel-Zhou]

@HenryJammes Hi, I am now testing and found another error in return PVA, the screenshot is as follows

[HenryJammes]

---

@Samuel-Zhou wrote:

@HenryJammes Hi, I am now testing and found another error in return PVA, the screenshot is as follows

---

Hi @Samuel-Zhou  I believe this is because the flow took more than 2 minutes to execute and with a PVA integration it needs to return results under 2 minutes to avoid a timeout error.

[HenryJammes]

---

@Estee wrote:

Hi @HenryJammes, 
Thank you for you post and detailed explanations.
I'm facing the issue of reaching the token limits and I twas thinking of different possibilities to fix it, but not sure what would be the easiest to implement in PVA (I'm used to python but new to PVA).

- Remove the oldest messages in the conversation transcript once the token count is reached. Every iteration, the oldest message is removed and so on.

- Reset the Global.FullDialog variable once there is no user activity for around 2 minutes. We can imagine a conversation between a user and the agent, and after 2 minutes of inactivity, the agent asks for user staisfaction, if the user is satisfied, the conversation start over on. Hopefully, there will be very few situations where the user is too chatty so that the token limit is reached.

- Inform the user the limit is reached and that he will have to start over but I think this option is a bit frustrating for the user.

What do you think? Thank you in advance for your help !

---

Hi @Estee, these are valid options, aligned with that I read here: How to work with the ChatGPT and GPT-4 models - Azure OpenAI Service | Microsoft Learn.

You could also ask the GPT model to summarize the previous conversation to reduce its token size.

[Samuel-Zhou]

@HenryJammes Actually, he has already returned, but for some reason, he did not return to PVA, even though generatedanswer has a value.