cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
karr
Regular Visitor

PowerAutomate Cloud Flow takes too long with ForEach component

I'm having a strange issue with PowerAutomate.

I have a basic Cloud Flow which consists of the following steps:

  • Dataverse connector (a simple query: list EventEditions)
  • For Each (EventEdition)
    • Dataverse connector (a simple query: list CustomerProfiles)
    • For Each (CustomerProfile)
      • Dataverse connector (a simple query: list WebEntries)
      • For Each (WebEntry)
        • Compose a string with CustomerProfile and WebEntries

The issue is that, while when I call the flow for one EventEdition (the most outer for each), it takes 30 mins at most (it is for the EventEdition with the largest data), a run for all EventEditions takes longer than a day. The number of EventEditions is 11, the flow for the largest one takes 30 mins at most, and most of them take only a few mins.

  • Handling each EventEdition in separate flow instances: 30 + 6 + 5 + ... ~= 1 h
  • Handling all EventEditions in the same flow instance: > 24 h

What would cause this?

 

I'm trying the simplest approach to overcome this which is to run the flow for each Event Edition sequentially. To do this, I set the Degree of Parallelism of the relevant ForEach components to 1 to make them run sequentially. But it seems this doesn't impact the running time.

 

If my explanation is not clear or detailed enough, I'll update it based on your comments.

 

I'd really appreciate your comments and suggestions on this. Thank you in advance.

 

Edit: Adding a screenshot of the flow.

 

The main flow, including one child flow.The main flow, including one child flow.

15 REPLIES 15
leo85
Continued Contributor
Continued Contributor

Hi @karr ,

There are a few things you can check and try.

 

Complexity: You've got three nested for-each loops. So you're dealing with exponential complexity, which is not ideal. The inner most loop (list WebEntries) has got the biggest influence. If you have 11 events, 100 customers and 100 web entries (just as an example), "compose string" will be called 110.000 times.

 

You may run into throttling problems here:

You have already changed the parallelism. But running things sequentially will slow your flow down. Try to increase the parallelism on every level if possible by setting the degree of parallelism to a number higher than 1. You need to watch out, because running things in parallel can mess up your flow.

 

Did you list all of the actions in your flow or are there any other actions (e.g. conditions) which you did not list?

 

Regards,

Leo

 

----------------------------

If my answer solved your issue, please mark it as a solution.

If what I wrote helped you, please give it a thumbs up.

Hi @leo85,

 

First of all, thank you for your help.

 

You are right, and I'm aware of the exponential complexity. But when I run the same flow for each data record (EventEdition) separately, I don't run into any performance issues, which must be equal to the same number of actions in total.

 

  • API request limits. I run into this limit when I include all the necessary actions in the flow. Here, in my question, I put only the main actions such as Dataverse calls, loops, and string operations. This way I don't observe the slow flow warning but the flow is still too slow.
  • Data limits. I haven't seen any banner regarding the data usage. But I'll keep an eye on it.
  • Connector limit. I'll read it.

 

But... Returning back to the beginning, I think I should be getting over this issue by ensuring each flow run for only one EventEdition. Because when I run the flow manually for only one EventEdition, it takes much less as I mentioned before. That would make only about 1 h for all events compared to a day.

 

But I couldn't achieve it yet for some reason. I tried to read/write to/from a Blob Storage to store the processed event IDs and the processing status (if an EventEdition is being processed, it is true) to be able to process one event each time. But it doesn't work. Maybe my design is still wrong.

 

I'm really surprised that the PowerAutomate falls short in doing that kind of task. It shouldn't be that difficult. The equivalent data process takes 15-30 mins as an Azure Function in C#.

 

You have already changed the parallelism. But running things sequentially will slow your flow down. Try to increase the parallelism on every level if possible by setting the degree of parallelism to a number higher than 1. You need to watch out, because running things in parallel can mess up your flow.


I already tried it and it didn't change anything. Also, as I mentioned before, the flow must take about 1 h in total, because when I run the flow manually for each EventEdition, it takes that much (1 h). The core of the problem must be to do with running one flow for all EventEditions (which, I guess, means using parallelism). Setting the most outer loop (ForEachEventEdition) to run sequentially must do the trick, but it doesn't. Am I wrong?

 

Did you list all of the actions in your flow or are there any other actions (e.g. conditions) which you did not list?


There are many other actions in the original version of the flow and when used they cause the "slow flow" issue. To get rid of this, I split the entire flow into 3 flows. But this didn't help reduce the running time as well.

leo85
Continued Contributor
Continued Contributor

Can you post a screenshot of the flow? Maybe there are other actions that are causing the problem.

Can you go through the flow run, which lasted 24h and post screenshots of the actions that take the longest amount of time?

Can you also post a screenshot of the "actions" chart of the flow's "analytics" page?

karr
Regular Visitor

 

flow-single-eventedition-1.pngSingle EventEdition (19 m - the longest one)Single EventEdition (19 m - the longest one)flow-all-eventeditions-1.pngAll EventEditions (13 h - 11 event editions)All EventEditions (13 h - 11 event editions)

 

PS. When I put all the necessary actions in the flow, the running time for all EventEditions is about a day. Here, for the light/test version of the flow, it took 13 h.

leo85
Continued Contributor
Continued Contributor

Can you also post a screenshot of the "actions" chart of the flow's "analytics" page?

karr
Regular Visitor

Do you mean Run Details as in Logic Apps? As far as I know, it is available for only Logic Apps. My flow is a PowerAutomate Cloud Flow and I don't see any option to see more details about runs.

leo85
Continued Contributor
Continued Contributor

No, not the run details. When you are in the flow studio on the "Cloud flows" page, click the flow name to open its details page. In the top banner there is a button "Analytics". If you click on that, you will see the actions.

 

leo85_1-1641062243910.png

 

 

But I don't have it. Maybe it's because of the subscription/plan or my permissions.

 

Edit: Yes, it is. The documentation says "Admins with the following roles and a license can view the reports in Power Automate analytics." https://docs.microsoft.com/en-us/power-platform/admin/analytics-flow

ribbon.png

karr
Regular Visitor

Another strange issue is that, in the same run, a smaller record took much longer than the largest one as you can see below.

"26 m" of running time for "the largest" record"26 m" of running time for "the largest" record"5 h" of running time for a "smaller" record"5 h" of running time for a "smaller" record

leo85
Continued Contributor
Continued Contributor

Hi @karr,

no, this is a standard feature that should be available to every creator of a flow irrespective of their license. What you quote is a feature of the Power Platform Admin Center. This is not was I was referring to.

Are you the owner/co-owner of the flow?

Is your flow part of a solution? In that case, you may need to select "See analytics" by clicking "more options" (see screenshot).

leo85_0-1641072062440.png

 

Hi @leo85,

 

I've found the link. Here is the chart:

 

flow-analytics-actions.png

 

PS. I am able to get rid of the low-performance warning by moving some actions to child flows so it's not the actual issue.

karr
Regular Visitor

My questions:

 

1- How can I run the flow for only one EventEdition each time and do this for all EventEditions? (I tried Blob Storage to store variables but it didn't work. #M157868)

I mean, I'm looking for a way to avoid the flow from running for more than one EventEdition. It should finish the entire process for one EventEdition before starting it over for another EventEdition.

(I'm going to add screenshots from my try soon.)

 

2- What's the explanation of this situation that I mentioned before? #M157889

Would it be related to the main issue?

leo85
Continued Contributor
Continued Contributor

Hi @karr ,

looking at your screenshot, we can see that your flow is being throttled (slow) either because of data or API usage. The API request limit is a user limit, so all flows running under your account count towards this limit. If you have a seeded lisence the current flow already uses all of the API request available.

 

For 1:

  • Firstly, if you create a solution, you can create one main flow, which calls one child flow for each EventEdition.
  • Secondly, if you want to do it manually, you can also use a manual trigger to start the flow for one specific Event Edition at a time.
  • Thirdly, you may be able to put all of it in one flow, if you reduce the number of API request. For example, you could try to get rid of the Compose actions (CustomerProfileId, CustomerProfile, ExhibitorBody) and use the results from the previous actions directly.

 

For 2:

  • Power Automate works on a best-effort basis. Meaning how fast the flow runs, depends on the general server load. In times of high load (globally) you flows may be slower, in times of low load faster.
  • Additionally, you've got an additional inner loop "Web Entries". So if the EventEdition with less customer profiles has got a profiles with a lot of web entries, this loop may take longer.

 

Regards,

Leo

Hi @leo85,

 

Many thanks for your reply.

 

But to me, it still doesn't answer why it takes that long to "process all EventEditions at once" compared to "processing them one by one" (1 day vs 1 h).

 

I believe I can overcome this issue by processing each EventEdition at a time. It seems that ForEachLoop can't manage this scenario for some reason. What I want to achieve, as in a regular programming language such as C#, is this:

 

For Each EventEdition
    Process the current EventEdition (fetch data from Dataverse and write it into XML file) (but never jump into the next EventEdition until the current one has been finished!)

That is:
Start processing EventEdition1 (10 s)
EventEdition1 has been processed
Start processing EventEdition2 (30 m)
Start processing has been processed
Process EventEdition3 (5 m)
Start processing has been processed
...
Start processing EventEdition11 (2 m)

 

In total, it takes about 1 h.

 

But in practice, it works this way:
Start processing EventEdition1
Start processing EventEdition2 (Processing EventEdition1 has NOT been finished yet)
Start processing EventEdition3 (Processing EventEdition1 and EventEdition2 has NOT been finished yet)
...

 

It seems that ForEachLoop in PowerAutomate does not wait for the current item to be processed to pass to the next item even though I set the ForEachLoop to run sequentially (no parallelism (default mode) or parallelism with the value of 1).

 

Maybe, it's because PowerAutomate doesn't have a thread mechanism as in C# so that Dataverse calls are not being traced by it - instead, it does a call to Dataverse and passes to the next action immediately.

 

As a workaround, I used a Blob Storage to save the IDs of the processed EventEditions (ProcessedEventEditionIds) and a boolean (Processing) indicating an EventEdition is currently being processed. This way, for the first unprocessed EventEdition, I can immediately set Processing to true so that other EventEditions are not going to be processed.

 

But... While Processing is being updated to 1, a few next items in the loop start to be handled because Processing is still 0 at that moment. As I said before, this (processing multiple EventEditions) causes the flow to run too slowly.

 

Currently, this is the main issue I haven't been able to solve yet. (I'll check our licenses as well as you suggested.)

 

Thank you.

leo85
Continued Contributor
Continued Contributor

Hi @karr,

 

how exactly did you check whether the loops are being executed sequentially or in parallel? When you set "degree of parallelism" to 1 there should not be any parallel runs.

 

How often did you run the individual EventEditions test (1h)? Did you run them before or after the bulk EventEditions test (24h)?

 

Instead of using a loop (looping over all EventEditions), you can also create parallel branches (one for each Event Edition). This is a lot of manual effort for copy/paste and not recommended if you have changing EventEditions.

Helpful resources

Announcements

Community will be READ ONLY July 16th, 5p PDT -July 22nd

Dear Community Members,   We'd like to let you know of an upcoming change to the community platform: starting July 16th, the platform will transition to a READ ONLY mode until July 22nd.   During this period, members will not be able to Kudo, Comment, or Reply to any posts.   On July 22nd, please be on the lookout for a message sent to the email address registered on your community profile. This email is crucial as it will contain your unique code and link to register for the new platform encompassing all of the communities.   What to Expect in the New Community: A more unified experience where all products, including Power Apps, Power Automate, Copilot Studio, and Power Pages, will be accessible from one community.Community Blogs that you can syndicate and link to for automatic updates. We appreciate your understanding and cooperation during this transition. Stay tuned for the exciting new features and a seamless community experience ahead!

Check Out | 2024 Release Wave 2 Plans for Microsoft Dynamics 365 and Microsoft Power Platform

On July 16, 2024, we published the 2024 release wave 2 plans for Microsoft Dynamics 365 and Microsoft Power Platform. These plans are a compilation of the new capabilities planned to be released between October 2024 to March 2025. This release introduces a wealth of new features designed to enhance customer understanding and improve overall user experience, showcasing our dedication to driving digital transformation for our customers and partners.    The upcoming wave is centered around utilizing advanced AI and Microsoft Copilot technologies to enhance user productivity and streamline operations across diverse business applications. These enhancements include intelligent automation, AI-powered insights, and immersive user experiences that are designed to break down barriers between data, insights, and individuals. Watch a summary of the release highlights.    Discover the latest features that empower organizations to operate more efficiently and adaptively. From AI-driven sales insights and customer service enhancements to predictive analytics in supply chain management and autonomous financial processes, the new capabilities enable businesses to proactively address challenges and capitalize on opportunities.    

Summer of Solutions | Week 3 Results | Win free tickets to the Power Platform Conference

We are excited to announce the Summer of Solutions Challenge!    This challenge is kicking off on Monday, June 17th and will run for (4) weeks.  The challenge is open to all Power Platform (Power Apps, Power Automate, Copilot Studio & Power Pages) community members. We invite you to participate in a quest to provide solutions to as many questions as you can. Answers can be provided in all the communities.    Entry Period: This Challenge will consist of four weekly Entry Periods as follows (each an “Entry Period”)   - 12:00 a.m. PT on June 17, 2024 – 11:59 p.m. PT on June 23, 2024 - 12:00 a.m. PT on June 24, 2024 – 11:59 p.m. PT on June 30, 2024 - 12:00 a.m. PT on July 1, 2024 – 11:59 p.m. PT on July 7, 2024 - 12:00 a.m. PT on July 8, 2024 – 11:59 p.m. PT on July 14, 2024   Entries will be eligible for the Entry Period in which they are received and will not carryover to subsequent weekly entry periods.  You must enter into each weekly Entry Period separately.   How to Enter: We invite you to participate in a quest to provide "Accepted Solutions" to as many questions as you can. Answers can be provided in all the communities. Users must provide a solution which can be an “Accepted Solution” in the Forums in all of the communities and there are no limits to the number of “Accepted Solutions” that a member can provide for entries in this challenge, but each entry must be substantially unique and different.    Winner Selection and Prizes: At the end of each week, we will list the top ten (10) Community users which will consist of: 5 Community Members & 5 Super Users and they will advance to the final drawing. We will post each week in the News & Announcements the top 10 Solution providers.  At the end of the challenge, we will add all of the top 10 weekly names and enter them into a random drawing.  Then we will randomly select ten (10) winners (5 Community Members & 5 Super Users) from among all eligible entrants received across all weekly Entry Periods to receive the prize listed below. If a winner declines, we will draw again at random for the next winner.  A user will only be able to win once overall. If they are drawn multiple times, another user will be drawn at random.  Individuals will be contacted before the announcement with the opportunity to claim or deny the prize.  Once all of the winners have been notified, we will post in the News & Announcements of each community with the list of winners.   Each winner will receive one (1) Pass to the Power Platform Conference in Las Vegas, Sep. 18-20, 2024 ($1800 value). NOTE: Prize is for conference attendance only and any other costs such as airfare, lodging, transportation, and food are the sole responsibility of the winner. Tickets are not transferable to any other party or to next year’s event.   ** PLEASE SEE THE ATTACHED RULES for this CHALLENGE**   Week 1 Results: Congratulations to the Week 1 qualifiers, you are being entered in the random drawing that will take place at the end of the challenge.   Community MembersNumber SolutionsSuper UsersNumber Solutions Deenuji 9 @NathanAlvares24  17 @Anil_g  7 @ManishSolanki  13 @eetuRobo  5 @David_MA  10 @VishnuReddy1997  5 @SpongYe  9JhonatanOB19932 (tie) @Nived_Nambiar  8 @maltie  2 (tie)   @PA-Noob  2 (tie)   @LukeMcG  2 (tie)   @tgut03  2 (tie)       Week 2 Results: Congratulations to the Week 2 qualifiers, you are being entered in the random drawing that will take place at the end of the challenge. Week 2: Community MembersSolutionsSuper UsersSolutionsPower Automate  @Deenuji  12@ManishSolanki 19 @Anil_g  10 @NathanAlvares24  17 @VishnuReddy1997  6 @Expiscornovus  10 @Tjan  5 @Nived_Nambiar  10 @eetuRobo  3 @SudeepGhatakNZ 8     Week 3 Results: Congratulations to the Week 3 qualifiers, you are being entered in the random drawing that will take place at the end of the challenge. Week 3:Community MembersSolutionsSuper UsersSolutionsPower Automate Deenuji32ManishSolanki55VishnuReddy199724NathanAlvares2444Anil_g22SudeepGhatakNZ40eetuRobo18Nived_Nambiar28Tjan8David_MA22  

Updates to Transitions in the Power Platform Communities

We're embarking on a journey to enhance your experience by transitioning to a new community platform. Our team has been diligently working to create a fresh community site, leveraging the very Dynamics 365 and Power Platform tools our community advocates for.  We started this journey with transitioning Copilot Studio forums and blogs in June. The move marks the beginning of a new chapter, and we're eager for you to be a part of it. The rest of the Power Platform product sites will be moving over this summer.   Stay tuned for more updates as we get closer to the launch. We can't wait to welcome you to our new community space, designed with you in mind. Let's connect, learn, and grow together.   Here's to new beginnings and endless possibilities!   If you have any questions, observations or concerns throughout this process please go to https://aka.ms/PPCommSupport.   To stay up to date on the latest details of this migration and other important Community updates subscribe to our News and Announcements forums: Copilot Studio, Power Apps, Power Automate, Power Pages

Users online (2,867)