Signalling error/failure condition to Power Automate from Async API

[Jcoolsen]

Dear Experts. 😊

I hope this is the right place/channel.

We have been using a custom connector in Power Automate for a while now. The endpoint is an Azure Function. This works fine.

Now, some of the operations can take a while depending on the input, so I have been looking into implementing the http async pattern.

I started extending the Azure function using durable functions and that also works ok, but they do not seem to match up with what the Power Automate caller/invocation expects.

Durable functions return a Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableOrchestrationStatus (json) object when complete and Power Automate does not appear to know how to unwrap this to yield only the output part of this.

Example response:

{

                             "statusCode": 200,

                             "headers": {

                                                          "Vary": "Accept-Encoding",

                                                          "Request-Context": "appId=cid-v1:8e4fcdf4-ed31-4196-8e4a-d3e3b9f94bb1",

                                                          "Date": "Tue, 08 Dec 2020 11:24:21 GMT",

                                                          "Content-Type": "application/json; charset=utf-8",

                                                          "Content-Length": "2253"

                             },

                             "body": {

                                                          "name": "WhoAmI_Orchestrator",

                                                          "instanceId": "4267e61e2a4649d1b06e981519a9733c",

                                                          "runtimeStatus": "Completed",

                                                          "input": {

                                                                                       "Authorization": "***",

                                                                                       "WebFullUrl": "https://sxyz.sharepoint.com/sites/jco"

                                                          },

                                                          "customStatus": null,

                                                          "output": {

                                                                                       "Title": "*Obfuscated Name*",

                                                                                       "LoginName": "i:0#.f|membership|jco@xyz.dev",

                                                                                       "Email": "jco@ xyz.dev"

                                                          },

                                                          "createdTime": "2020-12-08T11:24:09Z",

                                                          "lastUpdatedTime": "2020-12-08T11:24:12Z"

                             }

}

This may be something that we could work around, but the thing that I really struggle with is how to make Power Automate understand when a request has failed. Of all the examples and documentation I have found anywhere, there is no error handling in this scenario.

When Power Automate receives the following response from the function status endpoint, it happily accepts it as a success without any notion that the runtimeStatus is failed.

{

                             "statusCode": 200,

                             "headers": {

                                                          "Vary": "Accept-Encoding",

                                                          "Request-Context": "appId=cid-v1:8e4fcdf4-ed31-4196-8e4a-d3e3b9f94bb1",

                                                          "Date": "Tue, 08 Dec 2020 11:51:25 GMT",

                                                          "Content-Type": "application/json; charset=utf-8",

                                                          "Content-Length": "4449"

                             },

                             "body": {

                                                          "name": "WhoAmI_Orchestrator",

                                                          "instanceId": "dfd429917296412ebe107938dc93b641",

                                                          "runtimeStatus": "Failed",

                                                          "input": {

                                                                                       "Authorization": "***",

                                                                                       "WebFullUrl": "https://xyz.sharepoint.com/sites/cs-dms-demo1"

                                                          },

                                                          "customStatus": {

                                                                                       "Type": null,

                                                                                       "Title": "The remote server returned an error: (401) Unauthorized.",

                                                                                       "Status": 400,

                                                                                       "Detail": "Lms365.CS.Function.Controllers.ControllerException: The remote server returned an error: (401) Unauthorized.\r\n[*cut short*]",

                                                                                       "Instance": null,

                                                                                       "Extensions": {}

                                                          },

                                                          "output": "Orchestrator function 'WhoAmI_Orchestrator' failed: The remote server returned an error: (401) Unauthorized.",

                                                          "createdTime": "2020-12-08T11:51:14Z",

                                                          "lastUpdatedTime": "2020-12-08T11:51:16Z"

                             }

}

Now, Durable functions have a setting that allows them to set http status code “500 Internal Server Error” on failure (returnInternalServerErrorOnFailure). Trouble is that Power Automate (incorrectly) assumes that all status codes 5xx should be retried (it may be valid to retry “503 Service Unavailable”). Therefore, there will be a 10-minute delay before the user gets the actual error while Power Automate retries the operation.

To try and work around this, I made my own implementation for the status endpoint while still using the underlying durable function mechanics. This way I can control the output and status code of the successful operation, but when I return http status 400 Bad Request along with some error details (Microsoft.AspNetCore.Mvc.ProblemDetails), this is surprisingly captured as a 404 Not Found without any further details by Power Automate. The same goes if I send a 302 Found with a location to a different URL that contains the actual result (or error) depending on the outcome.

This article does not speak of redirection at the end:

https://docs.microsoft.com/en-us/azure/logic-apps/logic-apps-create-api-app#perform-long-running-tas...

This resource suggests using 302/303 redirect result codes:

https://docs.microsoft.com/en-us/azure/architecture/patterns/async-request-reply

To debug my way to a solution, I created a logic app, but found that the results were the same and it didn’t give me the necessary information.

I suspect that there are custom status headers that can be returned to Power Automate to signal that an operation has failed, but I simply cannot find any information on it.

[ShubhamGogna]

A solution that might work for you is to create an new endpoint or Azure function that can return a response in the format that Power Automate or Logic Apps accepts. 

Action in Power Automate
   |
   V
Azure Function 1 (start long-running action)
  (returns the 202+Location header to Azure Function 2)

Power Automate (following Location header as part of async contract)
   |
   V
Azure Function 2 (check long-running action)
  (returns 200, 400, 500 and the expected response for the long-running action)
  or
  (returns 202+Location header to Azure Function 2)

[Jcoolsen]

Thank you for responding. Much appreciated. 🙂

@ShubhamGogna wrote:
A solution that might work for you is to create an new endpoint or Azure function that can return a response in the format that Power Automate or Logic Apps accepts. 

Absolutely, and this is what I tried. I may not have been able to convey this clearly, but that is what I meant.

I made my own implementation for the status endpoint while still using the underlying durable function mechanics. This way I can control the output and status code of the successful operation, but when I return http status 400 Bad Request along with some error details (Microsoft.AspNetCore.Mvc.ProblemDetails), this is surprisingly captured as a 404 Not Found without any further details by Power Automate. The same goes if I send a 302 Found with a location to a different URL that contains the actual result (or error) depending on the outcome.

But when ever I try and throw an error, the platform does not recognize it properly. 500 makes Pwr Auto retry the operation and 400 shows up as 404 Not found. I cannot for the life of me figure out what it expects when I want to report an error and the guides/examples I have found do not cover this scenario.

It is completely impossible to debug because the underlying operation is hidden.
If there exists a test tool for http async pattern that I could run against my long-running function, I haven't been able to find it.

[ShubhamGogna]

I'm not that familiar with durable functions, but is it possible to "unwrap" the response?

If the response from Azure Functions is some payload like:

HTTP STATUS 200

{
 "statusCode": 400,
 "headers": [ ... ],
 "body": { "output": "Hello" }
}

Could your function "unwrap" it and return:

HTTP STATUS 400

"Hello"

It seems like LogicApps is accepting the "wrapped" response and treating it as a successful run of the action. The behavior with LogicApps and HTTP 500 is expected, but I'm not sure why there's a 400 -> 404 issue. 

[Jcoolsen]

Let's forget that I'm using durable functions. I just assumed that they would be my best bet for something compatible out of the box, but I can twist the responses in any way I want, so this is nearly irrelevant now.

I have done some more testing and have now concluded that the "404 Resource not found" is coming right after starting the operation, but only if I use the custom responses, so maybe there is some endpoint in play that is covered by the default durable implementation (?).

In order to understand what is really going on, I have made a new anonymous simple delay operation. You can call it too and verify that it returns what it should.

https://compliance-qa.365.systems/api/WaitTest?duration=10&success=true

The duration is a number of seconds to wait before completing and the success is whether the status endpoint should return "200 OK" or "400 Bad request" at the end. I have simplified the body of the "202 Accepted" response to just be the same URL as the Location header. E.g.

https://compliance-qa.365.systems/api/WaitTest_Status?instance=xyz123abc

When success is true, the end result is "200 OK" with body "null".

When success is false, the end result is "400 Bad request" with body "Orchestrator function 'WaitTest_Orchestrator' failed: This error was thrown on purpose".

When testing the endpoints via PostMan, it looks good to me, but it would seem that I'm missing something.

I have made a openapi.json that covers the operation and can be imported in a custom connector, but since I cannot attach it here, I'll paste it instead:

{
  "swagger": "2.0",
  "info": {
    "title": "WaitTest",
    "description": "",
    "version": "1.0"
  },
  "host": "compliance-qa.365.systems",
  "basePath": "/api",
  "schemes": [
    "https"
  ],
  "consumes": [],
  "produces": [],
  "paths": {
    "/WaitTest": {
      "get": {
        "responses": {
          "default": {
            "description": "default",
            "schema": {}
          }
        },
        "summary": "WaitTest",
        "operationId": "WaitTest_Start",
        "parameters": [
          {
            "name": "duration",
            "in": "query",
            "required": false,
            "type": "integer"
          },
          {
            "name": "success",
            "in": "query",
            "required": false,
            "type": "boolean"
          }
        ]
      }
    }
  },
  "definitions": {},
  "parameters": {},
  "responses": {},
  "securityDefinitions": {},
  "security": [],
  "tags": []
}

Calling the flow

Calling the flow

Result

404 result

I'm happy to change the output of the endpoints in any way that you can suggest.

Thank you for any assistance or pointers, you can provide.

[ShubhamGogna]

I found the issue, but I'm still trying to figure out why it's happening. The location header being returned by you is being modified somewhere in our system. This is leading to a 404 error because the modified URL doesn't exist.

I'll send an update when I have more information.

[ShubhamGogna · Accepted solution]

I got a confirmation that the behavior is by design and that custom connectors are expected to expose a separate operation in the OpenApi definition for status checks (your "WaitTest_Status" endpoint). This is done to make sure the status checks are authenticated.

Can you create a new operation, try it, and let me know if it works? FYI, the new operation can be marked as internal so the users of your custom connector will not see the operation in the Flow or Logic Apps UI: https://docs.microsoft.com/en-us/connectors/custom-connectors/openapi-extensions#x-ms-visibility 

[Jcoolsen]

I think I understand. It's not sufficient that the location header tells PA where to get status, it has to be part of the OpenAPI spec. But how do I mark it in the spec such that PA knows about it? Should it have a specific name or form?

[I wonder why this is not a problem when running with the default durable response... maybe there is a convention for the route/path to have a specific pattern?]

[Jcoolsen]

Thinking further about this, I now see that PA will match the location header to an endpoint in the spec. Thereby it can ensure that any, code query parameters for instance, are added to the getstatus call. It's late now, but I'll give it a go first thing in the morning and get back to you. 🙂

[ShubhamGogna]

There is no specific name or form that PA is expecting. The location URL just had to match one of the operations in the OpenAPI definition. 

  "paths": {
    "/WaitTest_Status": { // New path that matches the location you would send back
      "get": {
        "responses": {
          "default": {
            "description": "default",
            "schema": {}
          }
        },
        "summary": "WaitTest",
        "operationId": "WaitTest_Status", // Different operation ID
        "parameters": [
          {
            "name": "instance",
            "in": "query",
            "required": false,
            "type": "string"
          }
        ]
      }
    }
  },

Bonus: In order to avoid having to create a status operation for every async operation, you can try creating a generic status operation where one of the query parameters describes the original async operation. Ex: `AsyncOperation1, AsyncOperation2, AsyncOperation3` can all send a location header to `AsyncOperationCheckStatus`. This may not work in all cases, but if your async operations are very similar, you can reduce some work for yourself.

[Jcoolsen]

Thank you so much, @ShubhamGogna! 😁

The clue about adding the location operation to the openapi spec made all the difference and now I get the expected results both on success and on error.

---

@ShubhamGogna wrote:

Bonus: In order to avoid having to create a status operation for every async operation, you can try creating a generic status operation where one of the query parameters describes the original async operation.

---

I was actually already planning to do just that. It ties nicely into the durable orchestration api, but it's helpful to know that it is a proper way of doing things. 👍

You have solved my problem and I'll accept your answer shortly. Only one question still lingers in the open, however: The default endpoint for the durable getstatus operation looks like this:

https://server.host/runtime/webhooks/durabletask/instances/ag44f1f3eb0e4f15b557867de3bdfed7?taskHub=...
Are there special provisions within Power Automate/Logic App engine to accept location headers that look like this?

Best regards

[ShubhamGogna]

You could take advantage of policy templates like "Set Host URL": https://docs.microsoft.com/en-us/connectors/custom-connectors/policy-templates/dynamichosturl/dynami... 

However, you would still need to add something to the default response so that you could reference the header or query parameter in the policy template. 

[Jcoolsen]

Good to know. This was not what I meant, though. I was wondering why the durable function endpoint worked without explicit operation in the openapi definition. Sorry for not being clear about that.

[ShubhamGogna]

It seems the logic is the following:

if (response contains Location header)
  if (hostname(location) == hostname(OpenApi definition))
    location = reformat_location_to_operation_in_OpenApi_definition(location)
  else
    do not touch location

This is why your durable functions did not run into the issue with the location header being modified. The hostname of the durable functions does not match the host in the OpenAPI definition ("compliance-qa.365.systems").

[Jcoolsen]

Yes, it’s the same host.

”server.host” was just meant as a generalized example.

[ShubhamGogna]

Did your durable function return a Location header in the header section or in the body?

If it was in the body (as part of the JSON), it won't be interpreted as an async operation and will "finish" with just the one call.

[Jcoolsen]

It does both. I have exchanged the implementation with the durable in the WaitTest operation so you can see for yourself.
https://compliance-qa.365.systems/api/WaitTest?duration=30&success=true 

I have not made any changes to the openapi definition, but it still succeeds in PA.

Notice how both operations are taken as success by PA.

[ShubhamGogna]

I think I finally figured it out. I made the following call:

GET https://compliance-qa.365.systems/api/WaitTest?duration=3&success=true

which returned the following response:

HTTP/1.1 202 Accepted
Content-Length: 1365
Content-Type: application/json; charset=utf-8
Location: https://compliance-qa.365.systems/runtime/webhooks/durabletask/instances/[redacted]?taskHub=lms365complianceserviceqanortheurope&connection=Storage&code=[redacted]
Retry-After: 10

{
    "id": "[redacted]",
    "statusQueryGetUri": "https://compliance-qa.365.systems/runtime/webhooks/durabletask/instances/[redacted]?taskHub=lms365complianceserviceqanortheurope&connection=Storage&code=[redacted]",
    "sendEventPostUri": "https://compliance-qa.365.systems/runtime/webhooks/durabletask/instances/[redacted]/raiseEvent/{eventName}?taskHub=lms365complianceserviceqanortheurope&connection=Storage&code=[redacted]",
    "terminatePostUri": "https://compliance-qa.365.systems/runtime/webhooks/durabletask/instances/[redacted]/terminate?reason={text}&taskHub=lms365complianceserviceqanortheurope&connection=Storage&code=[redacted]",
    "purgeHistoryDeleteUri": "https://compliance-qa.365.systems/runtime/webhooks/durabletask/instances/[redacted]?taskHub=lms365complianceserviceqanortheurope&connection=Storage&code=[redacted]",
    "restartPostUri": "https://compliance-qa.365.systems/runtime/webhooks/durabletask/instances/[redacted]/restart?taskHub=lms365complianceserviceqanortheurope&connection=Storage&code=[redacted]"
}

As you said, the hostname is the same, but I was wrong about the exact logic. It's not the hostnames being compared, but the "service URL" which is (host + basePath). The updated logic is:

if (response contains Location header)
  if (starts_with_OpenApi_host_and_base_path(location))
    location = reformat_location_to_operation_in_OpenApi_definition(location)
  else
    do not touch location

This explains the behavior you saw:

var openApiHostAndBasePath = "https://compliance-qa.365.systems/api";
var nonDurableFunctionPath = "https://compliance-qa.365.systems/api/WaitTest?...";
var durableFunctionPath = "https://compliance-qa.365.systems/runtime/webhooks/...?...";

Assert.True( starts_with_OpenApi_host_and_base_path( nonDurableFunctionPath ) );
Assert.False( starts_with_OpenApi_host_and_base_path( durableFunctionPath ) );

[Jcoolsen]

That makes sense. That’s some special mechanics. Thank you for sticking with me.

❤️