PowerAutomate Cloud Flow takes too long with ForEach component

[karr]

I'm having a strange issue with PowerAutomate.

I have a basic Cloud Flow which consists of the following steps:

• Dataverse connector (a simple query: list EventEditions)
• For Each (EventEdition)
  • Dataverse connector (a simple query: list CustomerProfiles)
  • For Each (CustomerProfile)
    • Dataverse connector (a simple query: list WebEntries)
    • For Each (WebEntry)
      • Compose a string with CustomerProfile and WebEntries

The issue is that, while when I call the flow for one EventEdition (the most outer for each), it takes 30 mins at most (it is for the EventEdition with the largest data), a run for all EventEditions takes longer than a day. The number of EventEditions is 11, the flow for the largest one takes 30 mins at most, and most of them take only a few mins.

• Handling each EventEdition in separate flow instances: 30 + 6 + 5 + ... ~= 1 h
• Handling all EventEditions in the same flow instance: > 24 h

What would cause this?

I'm trying the simplest approach to overcome this which is to run the flow for each Event Edition sequentially. To do this, I set the Degree of Parallelism of the relevant ForEach components to 1 to make them run sequentially. But it seems this doesn't impact the running time.

If my explanation is not clear or detailed enough, I'll update it based on your comments.

I'd really appreciate your comments and suggestions on this. Thank you in advance.

Edit: Adding a screenshot of the flow.

The main flow, including one child flow.

[leo85]

Hi @karr ,

There are a few things you can check and try.

Complexity: You've got three nested for-each loops. So you're dealing with exponential complexity, which is not ideal. The inner most loop (list WebEntries) has got the biggest influence. If you have 11 events, 100 customers and 100 web entries (just as an example), "compose string" will be called 110.000 times.

You may run into throttling problems here:

• API request limits. The limit for API request is currently 40.000 actions, if you have a seeded license (Office 365) (https://docs.microsoft.com/en-us/power-platform/admin/api-request-limits-allocations). When you open the flow details, do you get a message that your flow is currently slow?
• Data limits. The limit for data consumption (the amount of data that is written to the flow history within 24 hours) is currently 1GB if you have got a seeded license (https://docs.microsoft.com/en-us/power-automate/limits-and-config). If you have a lot of webEntries and the list WebEntries action provides 1MB of data. If it is executed 1000 times in your flow run, you will run into the data consumption limit. You can decrease data consumption by using filters in your queries (e.g. list WebEntries). When you open the flow details, do you get a banner saying that your flow is using a lot of data?
• Connector limits. The Dataverse connector also has got some limitations, which you can have a look at here: https://docs.microsoft.com/en-us/connectors/commondataserviceforapps/.

You have already changed the parallelism. But running things sequentially will slow your flow down. Try to increase the parallelism on every level if possible by setting the degree of parallelism to a number higher than 1. You need to watch out, because running things in parallel can mess up your flow.

Did you list all of the actions in your flow or are there any other actions (e.g. conditions) which you did not list?

Regards,

Leo

----------------------------

If my answer solved your issue, please mark it as a solution.

If what I wrote helped you, please give it a thumbs up.

[karr]

Hi @leo85,

First of all, thank you for your help.

You are right, and I'm aware of the exponential complexity. But when I run the same flow for each data record (EventEdition) separately, I don't run into any performance issues, which must be equal to the same number of actions in total.

• API request limits. I run into this limit when I include all the necessary actions in the flow. Here, in my question, I put only the main actions such as Dataverse calls, loops, and string operations. This way I don't observe the slow flow warning but the flow is still too slow.
• Data limits. I haven't seen any banner regarding the data usage. But I'll keep an eye on it.
• Connector limit. I'll read it.

But... Returning back to the beginning, I think I should be getting over this issue by ensuring each flow run for only one EventEdition. Because when I run the flow manually for only one EventEdition, it takes much less as I mentioned before. That would make only about 1 h for all events compared to a day.

But I couldn't achieve it yet for some reason. I tried to read/write to/from a Blob Storage to store the processed event IDs and the processing status (if an EventEdition is being processed, it is true) to be able to process one event each time. But it doesn't work. Maybe my design is still wrong.

I'm really surprised that the PowerAutomate falls short in doing that kind of task. It shouldn't be that difficult. The equivalent data process takes 15-30 mins as an Azure Function in C#.

You have already changed the parallelism. But running things sequentially will slow your flow down. Try to increase the parallelism on every level if possible by setting the degree of parallelism to a number higher than 1. You need to watch out, because running things in parallel can mess up your flow.

---

I already tried it and it didn't change anything. Also, as I mentioned before, the flow must take about 1 h in total, because when I run the flow manually for each EventEdition, it takes that much (1 h). The core of the problem must be to do with running one flow for all EventEditions (which, I guess, means using parallelism). Setting the most outer loop (ForEachEventEdition) to run sequentially must do the trick, but it doesn't. Am I wrong?

Did you list all of the actions in your flow or are there any other actions (e.g. conditions) which you did not list?

---

There are many other actions in the original version of the flow and when used they cause the "slow flow" issue. To get rid of this, I split the entire flow into 3 flows. But this didn't help reduce the running time as well.

[leo85]

Can you post a screenshot of the flow? Maybe there are other actions that are causing the problem.

Can you go through the flow run, which lasted 24h and post screenshots of the actions that take the longest amount of time?

Can you also post a screenshot of the "actions" chart of the flow's "analytics" page?

[karr]

Single EventEdition (19 m - the longest one)All EventEditions (13 h - 11 event editions)

PS. When I put all the necessary actions in the flow, the running time for all EventEditions is about a day. Here, for the light/test version of the flow, it took 13 h.

[leo85]

Can you also post a screenshot of the "actions" chart of the flow's "analytics" page?

[karr]

Do you mean Run Details as in Logic Apps? As far as I know, it is available for only Logic Apps. My flow is a PowerAutomate Cloud Flow and I don't see any option to see more details about runs.

[leo85]

No, not the run details. When you are in the flow studio on the "Cloud flows" page, click the flow name to open its details page. In the top banner there is a button "Analytics". If you click on that, you will see the actions.

[karr]

But I don't have it. Maybe it's because of the subscription/plan or my permissions.

Edit: Yes, it is. The documentation says "Admins with the following roles and a license can view the reports in Power Automate analytics." https://docs.microsoft.com/en-us/power-platform/admin/analytics-flow

[karr]

Another strange issue is that, in the same run, a smaller record took much longer than the largest one as you can see below.

"26 m" of running time for "the largest" record"5 h" of running time for a "smaller" record

[leo85]

Hi @karr,

no, this is a standard feature that should be available to every creator of a flow irrespective of their license. What you quote is a feature of the Power Platform Admin Center. This is not was I was referring to.

Are you the owner/co-owner of the flow?

Is your flow part of a solution? In that case, you may need to select "See analytics" by clicking "more options" (see screenshot).

[karr]

Hi @leo85,

I've found the link. Here is the chart:

PS. I am able to get rid of the low-performance warning by moving some actions to child flows so it's not the actual issue.

[karr]

My questions:

1- How can I run the flow for only one EventEdition each time and do this for all EventEditions? (I tried Blob Storage to store variables but it didn't work. #M157868)

I mean, I'm looking for a way to avoid the flow from running for more than one EventEdition. It should finish the entire process for one EventEdition before starting it over for another EventEdition.

(I'm going to add screenshots from my try soon.)

2- What's the explanation of this situation that I mentioned before? #M157889

Would it be related to the main issue?

[leo85]

Hi @karr ,

looking at your screenshot, we can see that your flow is being throttled (slow) either because of data or API usage. The API request limit is a user limit, so all flows running under your account count towards this limit. If you have a seeded lisence the current flow already uses all of the API request available.

For 1:

• Firstly, if you create a solution, you can create one main flow, which calls one child flow for each EventEdition.
• Secondly, if you want to do it manually, you can also use a manual trigger to start the flow for one specific Event Edition at a time.
• Thirdly, you may be able to put all of it in one flow, if you reduce the number of API request. For example, you could try to get rid of the Compose actions (CustomerProfileId, CustomerProfile, ExhibitorBody) and use the results from the previous actions directly.

For 2:

• Power Automate works on a best-effort basis. Meaning how fast the flow runs, depends on the general server load. In times of high load (globally) you flows may be slower, in times of low load faster.
• Additionally, you've got an additional inner loop "Web Entries". So if the EventEdition with less customer profiles has got a profiles with a lot of web entries, this loop may take longer.

Regards,

Leo

[karr]

Hi @leo85,

Many thanks for your reply.

But to me, it still doesn't answer why it takes that long to "process all EventEditions at once" compared to "processing them one by one" (1 day vs 1 h).

I believe I can overcome this issue by processing each EventEdition at a time. It seems that ForEachLoop can't manage this scenario for some reason. What I want to achieve, as in a regular programming language such as C#, is this:

For Each EventEdition
    Process the current EventEdition (fetch data from Dataverse and write it into XML file) (but never jump into the next EventEdition until the current one has been finished!)

That is:
Start processing EventEdition1 (10 s)
EventEdition1 has been processed
Start processing EventEdition2 (30 m)
Start processing has been processed
Process EventEdition3 (5 m)
Start processing has been processed
...
Start processing EventEdition11 (2 m)

In total, it takes about 1 h.

But in practice, it works this way:
Start processing EventEdition1
Start processing EventEdition2 (Processing EventEdition1 has NOT been finished yet)
Start processing EventEdition3 (Processing EventEdition1 and EventEdition2 has NOT been finished yet)
...

It seems that ForEachLoop in PowerAutomate does not wait for the current item to be processed to pass to the next item even though I set the ForEachLoop to run sequentially (no parallelism (default mode) or parallelism with the value of 1).

Maybe, it's because PowerAutomate doesn't have a thread mechanism as in C# so that Dataverse calls are not being traced by it - instead, it does a call to Dataverse and passes to the next action immediately.

As a workaround, I used a Blob Storage to save the IDs of the processed EventEditions (ProcessedEventEditionIds) and a boolean (Processing) indicating an EventEdition is currently being processed. This way, for the first unprocessed EventEdition, I can immediately set Processing to true so that other EventEditions are not going to be processed.

But... While Processing is being updated to 1, a few next items in the loop start to be handled because Processing is still 0 at that moment. As I said before, this (processing multiple EventEditions) causes the flow to run too slowly.

Currently, this is the main issue I haven't been able to solve yet. (I'll check our licenses as well as you suggested.)

Thank you.

[leo85]

Hi @karr,

how exactly did you check whether the loops are being executed sequentially or in parallel? When you set "degree of parallelism" to 1 there should not be any parallel runs.

How often did you run the individual EventEditions test (1h)? Did you run them before or after the bulk EventEditions test (24h)?

Instead of using a loop (looping over all EventEditions), you can also create parallel branches (one for each Event Edition). This is a lot of manual effort for copy/paste and not recommended if you have changing EventEditions.