cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
karr
Regular Visitor

PowerAutomate Cloud Flow takes too long with ForEach component

I'm having a strange issue with PowerAutomate.

I have a basic Cloud Flow which consists of the following steps:

  • Dataverse connector (a simple query: list EventEditions)
  • For Each (EventEdition)
    • Dataverse connector (a simple query: list CustomerProfiles)
    • For Each (CustomerProfile)
      • Dataverse connector (a simple query: list WebEntries)
      • For Each (WebEntry)
        • Compose a string with CustomerProfile and WebEntries

The issue is that, while when I call the flow for one EventEdition (the most outer for each), it takes 30 mins at most (it is for the EventEdition with the largest data), a run for all EventEditions takes longer than a day. The number of EventEditions is 11, the flow for the largest one takes 30 mins at most, and most of them take only a few mins.

  • Handling each EventEdition in separate flow instances: 30 + 6 + 5 + ... ~= 1 h
  • Handling all EventEditions in the same flow instance: > 24 h

What would cause this?

 

I'm trying the simplest approach to overcome this which is to run the flow for each Event Edition sequentially. To do this, I set the Degree of Parallelism of the relevant ForEach components to 1 to make them run sequentially. But it seems this doesn't impact the running time.

 

If my explanation is not clear or detailed enough, I'll update it based on your comments.

 

I'd really appreciate your comments and suggestions on this. Thank you in advance.

 

Edit: Adding a screenshot of the flow.

 

The main flow, including one child flow.The main flow, including one child flow.

15 REPLIES 15
leo85
Continued Contributor
Continued Contributor

Hi @karr ,

There are a few things you can check and try.

 

Complexity: You've got three nested for-each loops. So you're dealing with exponential complexity, which is not ideal. The inner most loop (list WebEntries) has got the biggest influence. If you have 11 events, 100 customers and 100 web entries (just as an example), "compose string" will be called 110.000 times.

 

You may run into throttling problems here:

You have already changed the parallelism. But running things sequentially will slow your flow down. Try to increase the parallelism on every level if possible by setting the degree of parallelism to a number higher than 1. You need to watch out, because running things in parallel can mess up your flow.

 

Did you list all of the actions in your flow or are there any other actions (e.g. conditions) which you did not list?

 

Regards,

Leo

 

----------------------------

If my answer solved your issue, please mark it as a solution.

If what I wrote helped you, please give it a thumbs up.

karr
Regular Visitor

Hi @leo85,

 

First of all, thank you for your help.

 

You are right, and I'm aware of the exponential complexity. But when I run the same flow for each data record (EventEdition) separately, I don't run into any performance issues, which must be equal to the same number of actions in total.

 

  • API request limits. I run into this limit when I include all the necessary actions in the flow. Here, in my question, I put only the main actions such as Dataverse calls, loops, and string operations. This way I don't observe the slow flow warning but the flow is still too slow.
  • Data limits. I haven't seen any banner regarding the data usage. But I'll keep an eye on it.
  • Connector limit. I'll read it.

 

But... Returning back to the beginning, I think I should be getting over this issue by ensuring each flow run for only one EventEdition. Because when I run the flow manually for only one EventEdition, it takes much less as I mentioned before. That would make only about 1 h for all events compared to a day.

 

But I couldn't achieve it yet for some reason. I tried to read/write to/from a Blob Storage to store the processed event IDs and the processing status (if an EventEdition is being processed, it is true) to be able to process one event each time. But it doesn't work. Maybe my design is still wrong.

 

I'm really surprised that the PowerAutomate falls short in doing that kind of task. It shouldn't be that difficult. The equivalent data process takes 15-30 mins as an Azure Function in C#.

 

You have already changed the parallelism. But running things sequentially will slow your flow down. Try to increase the parallelism on every level if possible by setting the degree of parallelism to a number higher than 1. You need to watch out, because running things in parallel can mess up your flow.


I already tried it and it didn't change anything. Also, as I mentioned before, the flow must take about 1 h in total, because when I run the flow manually for each EventEdition, it takes that much (1 h). The core of the problem must be to do with running one flow for all EventEditions (which, I guess, means using parallelism). Setting the most outer loop (ForEachEventEdition) to run sequentially must do the trick, but it doesn't. Am I wrong?

 

Did you list all of the actions in your flow or are there any other actions (e.g. conditions) which you did not list?


There are many other actions in the original version of the flow and when used they cause the "slow flow" issue. To get rid of this, I split the entire flow into 3 flows. But this didn't help reduce the running time as well.

leo85
Continued Contributor
Continued Contributor

Can you post a screenshot of the flow? Maybe there are other actions that are causing the problem.

Can you go through the flow run, which lasted 24h and post screenshots of the actions that take the longest amount of time?

Can you also post a screenshot of the "actions" chart of the flow's "analytics" page?

karr
Regular Visitor

 

flow-single-eventedition-1.pngSingle EventEdition (19 m - the longest one)Single EventEdition (19 m - the longest one)flow-all-eventeditions-1.pngAll EventEditions (13 h - 11 event editions)All EventEditions (13 h - 11 event editions)

 

PS. When I put all the necessary actions in the flow, the running time for all EventEditions is about a day. Here, for the light/test version of the flow, it took 13 h.

leo85
Continued Contributor
Continued Contributor

Can you also post a screenshot of the "actions" chart of the flow's "analytics" page?

karr
Regular Visitor

Do you mean Run Details as in Logic Apps? As far as I know, it is available for only Logic Apps. My flow is a PowerAutomate Cloud Flow and I don't see any option to see more details about runs.

leo85
Continued Contributor
Continued Contributor

No, not the run details. When you are in the flow studio on the "Cloud flows" page, click the flow name to open its details page. In the top banner there is a button "Analytics". If you click on that, you will see the actions.

 

leo85_1-1641062243910.png

 

 

karr
Regular Visitor

But I don't have it. Maybe it's because of the subscription/plan or my permissions.

 

Edit: Yes, it is. The documentation says "Admins with the following roles and a license can view the reports in Power Automate analytics." https://docs.microsoft.com/en-us/power-platform/admin/analytics-flow

ribbon.png

karr
Regular Visitor

Another strange issue is that, in the same run, a smaller record took much longer than the largest one as you can see below.

"26 m" of running time for "the largest" record"26 m" of running time for "the largest" record"5 h" of running time for a "smaller" record"5 h" of running time for a "smaller" record

leo85
Continued Contributor
Continued Contributor

Hi @karr,

no, this is a standard feature that should be available to every creator of a flow irrespective of their license. What you quote is a feature of the Power Platform Admin Center. This is not was I was referring to.

Are you the owner/co-owner of the flow?

Is your flow part of a solution? In that case, you may need to select "See analytics" by clicking "more options" (see screenshot).

leo85_0-1641072062440.png

 

karr
Regular Visitor

Hi @leo85,

 

I've found the link. Here is the chart:

 

flow-analytics-actions.png

 

PS. I am able to get rid of the low-performance warning by moving some actions to child flows so it's not the actual issue.

karr
Regular Visitor

My questions:

 

1- How can I run the flow for only one EventEdition each time and do this for all EventEditions? (I tried Blob Storage to store variables but it didn't work. #M157868)

I mean, I'm looking for a way to avoid the flow from running for more than one EventEdition. It should finish the entire process for one EventEdition before starting it over for another EventEdition.

(I'm going to add screenshots from my try soon.)

 

2- What's the explanation of this situation that I mentioned before? #M157889

Would it be related to the main issue?

leo85
Continued Contributor
Continued Contributor

Hi @karr ,

looking at your screenshot, we can see that your flow is being throttled (slow) either because of data or API usage. The API request limit is a user limit, so all flows running under your account count towards this limit. If you have a seeded lisence the current flow already uses all of the API request available.

 

For 1:

  • Firstly, if you create a solution, you can create one main flow, which calls one child flow for each EventEdition.
  • Secondly, if you want to do it manually, you can also use a manual trigger to start the flow for one specific Event Edition at a time.
  • Thirdly, you may be able to put all of it in one flow, if you reduce the number of API request. For example, you could try to get rid of the Compose actions (CustomerProfileId, CustomerProfile, ExhibitorBody) and use the results from the previous actions directly.

 

For 2:

  • Power Automate works on a best-effort basis. Meaning how fast the flow runs, depends on the general server load. In times of high load (globally) you flows may be slower, in times of low load faster.
  • Additionally, you've got an additional inner loop "Web Entries". So if the EventEdition with less customer profiles has got a profiles with a lot of web entries, this loop may take longer.

 

Regards,

Leo

karr
Regular Visitor

Hi @leo85,

 

Many thanks for your reply.

 

But to me, it still doesn't answer why it takes that long to "process all EventEditions at once" compared to "processing them one by one" (1 day vs 1 h).

 

I believe I can overcome this issue by processing each EventEdition at a time. It seems that ForEachLoop can't manage this scenario for some reason. What I want to achieve, as in a regular programming language such as C#, is this:

 

For Each EventEdition
    Process the current EventEdition (fetch data from Dataverse and write it into XML file) (but never jump into the next EventEdition until the current one has been finished!)

That is:
Start processing EventEdition1 (10 s)
EventEdition1 has been processed
Start processing EventEdition2 (30 m)
Start processing has been processed
Process EventEdition3 (5 m)
Start processing has been processed
...
Start processing EventEdition11 (2 m)

 

In total, it takes about 1 h.

 

But in practice, it works this way:
Start processing EventEdition1
Start processing EventEdition2 (Processing EventEdition1 has NOT been finished yet)
Start processing EventEdition3 (Processing EventEdition1 and EventEdition2 has NOT been finished yet)
...

 

It seems that ForEachLoop in PowerAutomate does not wait for the current item to be processed to pass to the next item even though I set the ForEachLoop to run sequentially (no parallelism (default mode) or parallelism with the value of 1).

 

Maybe, it's because PowerAutomate doesn't have a thread mechanism as in C# so that Dataverse calls are not being traced by it - instead, it does a call to Dataverse and passes to the next action immediately.

 

As a workaround, I used a Blob Storage to save the IDs of the processed EventEditions (ProcessedEventEditionIds) and a boolean (Processing) indicating an EventEdition is currently being processed. This way, for the first unprocessed EventEdition, I can immediately set Processing to true so that other EventEditions are not going to be processed.

 

But... While Processing is being updated to 1, a few next items in the loop start to be handled because Processing is still 0 at that moment. As I said before, this (processing multiple EventEditions) causes the flow to run too slowly.

 

Currently, this is the main issue I haven't been able to solve yet. (I'll check our licenses as well as you suggested.)

 

Thank you.

leo85
Continued Contributor
Continued Contributor

Hi @karr,

 

how exactly did you check whether the loops are being executed sequentially or in parallel? When you set "degree of parallelism" to 1 there should not be any parallel runs.

 

How often did you run the individual EventEditions test (1h)? Did you run them before or after the bulk EventEditions test (24h)?

 

Instead of using a loop (looping over all EventEditions), you can also create parallel branches (one for each Event Edition). This is a lot of manual effort for copy/paste and not recommended if you have changing EventEditions.

Helpful resources

Announcements

Win free tickets to the Power Platform Conference | Summer of Solutions

We are excited to announce the Summer of Solutions Challenge!    This challenge is kicking off on Monday, June 17th and will run for (4) weeks.  The challenge is open to all Power Platform (Power Apps, Power Automate, Copilot Studio & Power Pages) community members. We invite you to participate in a quest to provide solutions to as many questions as you can. Answers can be provided in all the communities.    Entry Period: This Challenge will consist of four weekly Entry Periods as follows (each an “Entry Period”)   - 12:00 a.m. PT on June 17, 2024 – 11:59 p.m. PT on June 23, 2024 - 12:00 a.m. PT on June 24, 2024 – 11:59 p.m. PT on June 30, 2024 - 12:00 a.m. PT on July 1, 2024 – 11:59 p.m. PT on July 7, 2024 - 12:00 a.m. PT on July 8, 2024 – 11:59 p.m. PT on July 14, 2024   Entries will be eligible for the Entry Period in which they are received and will not carryover to subsequent weekly entry periods.  You must enter into each weekly Entry Period separately.   How to Enter: We invite you to participate in a quest to provide "Accepted Solutions" to as many questions as you can. Answers can be provided in all the communities. Users must provide a solution which can be an “Accepted Solution” in the Forums in all of the communities and there are no limits to the number of “Accepted Solutions” that a member can provide for entries in this challenge, but each entry must be substantially unique and different.    Winner Selection and Prizes: At the end of each week, we will list the top ten (10) Community users which will consist of: 5 Community Members & 5 Super Users and they will advance to the final drawing. We will post each week in the News & Announcements the top 10 Solution providers.  At the end of the challenge, we will add all of the top 10 weekly names and enter them into a random drawing.  Then we will randomly select ten (10) winners (5 Community Members & 5 Super Users) from among all eligible entrants received across all weekly Entry Periods to receive the prize listed below. If a winner declines, we will draw again at random for the next winner.  A user will only be able to win once overall. If they are drawn multiple times, another user will be drawn at random.  Individuals will be contacted before the announcement with the opportunity to claim or deny the prize.  Once all of the winners have been notified, we will post in the News & Announcements of each community with the list of winners.   Each winner will receive one (1) Pass to the Power Platform Conference in Las Vegas, Sep. 18-20, 2024 ($1800 value). NOTE: Prize is for conference attendance only and any other costs such as airfare, lodging, transportation, and food are the sole responsibility of the winner. Tickets are not transferable to any other party or to next year’s event.   ** PLEASE SEE THE ATTACHED RULES for this CHALLENGE**

Celebrating the June Super User of the Month: Markus Franz

Markus Franz is a phenomenal contributor to the Power Apps Community. Super Users like Markus inspire others through their example, encouragement, and active participation.    The Why: "I do this to help others achieve what they are trying to do. As a total beginner back then without IT background I know how overwhelming things can be, so I decided to jump in and help others. I also do this to keep progressing and learning myself." Thank you, Markus Franz, for your outstanding work! Keep inspiring others and making a difference in the community! 🎉  Keep up the fantastic work! 👏👏 Markus Franz | LinkedIn  Power Apps: mmbr1606  

Copilot Cookbook Challenge | Week 1 Results | Win Tickets to the Power Platform Conference

We are excited to announce the "The Copilot Cookbook Community Challenge is a great way to showcase your creativity and connect with others. Plus, you could win tickets to the Power Platform Community Conference in Las Vegas in September 2024 as an amazing bonus.   Two ways to enter: 1. Copilot Studio Cookbook Gallery:  https://aka.ms/CS_Copilot_Cookbook_Challenge 2. Power Apps Copilot Cookbook Gallery: https://aka.ms/PA_Copilot_Cookbook_Challenge   There will be 5 chances to qualify for the final drawing: Early Bird Entries: March 1 - June 2Week 1: June 3 - June 9Week 2: June 10 - June 16Week 3: June 17 - June 23Week 4: June 24 - June 30     At the end of each week, we will draw 5 random names from every user who has posted a qualifying Copilot Studio template, sample or demo in the Copilot Studio Cookbook or a qualifying Power Apps Copilot sample or demo in the Power Apps Copilot Cookbook. Users who are not drawn in a given week will be added to the pool for the next week. Users can qualify more than once, but no more than once per week. Four winners will be drawn at random from the total qualifying entrants. If a winner declines, we will draw again at random for the next winner.  A user will only be able to win once. If they are drawn multiple times, another user will be drawn at random. Prizes:  One Pass to the Power Platform Conference in Las Vegas, Sep. 18-20, 2024 ($1800 value, does not include travel, lodging, or any other expenses) Winners are also eligible to do a 10-minute presentation of their demo or solution in a community solutions showcase at the event. To qualify for the drawing, templates, samples or demos must be related to Copilot Studio or a Copilot feature of Power Apps, Power Automate, or Power Pages, and must demonstrate or solve a complete unique and useful business or technical problem. Power Automate and Power Pagers posts should be added to the Power Apps Cookbook. Final determination of qualifying entries is at the sole discretion of Microsoft. Weekly updates and the Final random winners will be posted in the News & Announcements section in the communities on July 29th, 2024. Did you submit entries early?  Early Bird Entries March 1 - June 2:  If you posted something in the "early bird" time frame complete this form: https://aka.ms/Copilot_Challenge_EarlyBirds if you would like to be entered in the challenge.   Week 1 Results:  Congratulations to the Week 1 qualifiers, you are being entered in the random drawing that will take place at the end of the challenge. Copilot Cookbook Gallery:Power Apps Cookbook Gallery:1.  @Mathieu_Paris 1.   @SpongYe 2.  @Dhanush 2.   @Deenuji 3.  n/a3.   @Nived_Nambiar  4.  n/a4.   @ManishSolanki 5.  n/a5.    n/a

Your Moment to Shine: 2024 PPCC’s Got Power Awards Show

For the third year, we invite you, our talented community members, to participate in the grand 2024 Power Platform Community Conference's Got Power Awards. This event is your opportunity to showcase solutions that make a significant business impact, highlight extensive use of Power Platform products, demonstrate good governance, or tell an inspirational story. Share your success stories, inspire your peers, and show off some hidden talents.  This is your time to shine and bring your creations into the spotlight!  Make your mark, inspire others and leave a lasting impression. Sign up today for a chance to showcase your solution and win the coveted 2024 PPCC’s Got Power Award. This year we have three categories for you to participate in: Technical Solution Demo, Storytelling, and Hidden Talent.      The Technical solution demo category showcases your applications, automated workflows, copilot agentic experiences, web pages, AI capabilities, dashboards, and/or more. We want to see your most impactful Power Platform solutions!  The Storytelling category is where you can share your inspiring story, and the Hidden Talent category is where your talents (such as singing, dancing, jump roping, etc.) can shine! Submission Details:  Fill out the submission form https://aka.ms/PPCCGotPowerSignup by July 12th with details and a 2–5-minute video showcasing your Solution impact. (Please let us know you're coming to PPCC, too!)After review by a panel of Microsoft judges, the top storytellers will be invited to present a virtual demo presentation to the judges during early August. You’ll be notified soon after if you have been selected as a finalist to share your story live at PPCC’s Got Power!  The live show will feature the solution demos and storytelling talents of the top contestants, winner announcements, and the opportunity to network with your community.  It's not just a showcase for technical talent and storytelling showmanship, show it's a golden opportunity to make connections and celebrate our Community together! Let's make this a memorable event! See you there!   Mark your calendars! Date and Time: Thursday, Sept 19th Location: PPCC24 at the MGM Grand, Las Vegas, NV 

Tuesday Tip | Accepting Solutions

It's time for another TUESDAY TIPS, your weekly connection with the most insightful tips and tricks that empower both newcomers and veterans in the Power Platform Community! Every Tuesday, we bring you a curated selection of the finest advice, distilled from the resources and tools in the Community. Whether you’re a seasoned member or just getting started, Tuesday Tips are the perfect compass guiding you across the dynamic landscape of the Power Platform Community.   To enhance our collaborative environment, it's important to acknowledge when your question has been answered satisfactorily. Here's a quick guide on how to accept a solution to your questions: Find the Helpful Reply: Navigate to the reply that has effectively answered your question.Accept as Solution: Look for the "Accept as Solution" button or link, usually located at the bottom of the reply.Confirm Your Selection: Clicking this button may prompt you for confirmation. Go ahead and confirm that this is indeed the solution.Acknowledgment: Once accepted, the reply will be highlighted, and the original post will be marked as "Solved". This helps other community members find the same solution quickly. By marking a reply as an accepted solution, you not only thank the person who helped you but also make it easier for others with similar questions to find answers. Let's continue to support each other by recognizing helpful contributions. 

Reminder: To register for the Community Ambassador Call on June 13th

Calling all Super Users & User Group Leaders   Reminder: To register for the Community Ambassador Call on June 13th—for an exclusive event for User Group Leaders and Super Users! This month is packed with exciting updates and activities within our community.   What's Happening: Community Updates: We'll share the latest developments and what's new in our vibrant community.Special Guest Speaker: Get ready for an insightful talk and live demo of Microsoft Copilot Studio templates by our special guest.Regular Updates: Stay informed with our routine updates for User Groups and Super Users.Community Insights: We'll provide general information about ongoing and upcoming community initiatives. Don't Miss Out: Register Now: Choose the session that fits your schedule best.Check your private messages or Super User Forum for registration links. We're excited to connect with you and continue building a stronger community together.   See you at the call!  

Users online (3,197)