cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Marty3012
Helper II
Helper II

Apply to each - Parallel Processing - 50 limit

Hi,

 

I posted this before but for some reason i cant find it within any of the forums. 

 

I have the following flow;

Parallel Processing 50 plus.PNG

 

Description

1 - Connect to an excel sheet and get all rows

2 - Within the Apply to each, loop through each row. (Set to Parallelism)

3 - Set condition that checks if the 'Expiration Date' field for each row equals the current date

4 - If so, set off approvals

5 - If it is approved then the comments provided are sent back to the excel file into another field.

 

All is working well. However there is the limit of 50 parallel tasks. After the dates are compared I could have between 1-100 rows returned. One possible solution I thought of was to create 2 copies of the Apply to Each loop described above. I could then have a condition that checks if there is say 57 rows returned, one loop could process 50 and the other could process 7. Would this be possible?

 

I've tried testing using variables and count functions but I'm getting nowhere. Any help appreciated.

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Ok, got it.  In that case I think you'll have to put an intermediate Loop between your Excel List action and your main processing Loop.  Prior to that Loop create two array variables. Inside the loop take the values from the first 50 objects and append them as objects to the first array variable.  Once the count reaches 50 append the rest to the second array variable.  Then create two parallel Apply to Each Loops for your main processing.  Process the first array in one and the second array in the other.  Set both for maximum parallel of 50.  If you have more than 100 you'll have to add a third variable and parallel loop.



-------------------------------------------------------------------------
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.

Join me at 365 and PWR EduCon in Chicago
EduCon Chicago 2022

View solution in original post

5 REPLIES 5
Pstork1
Dual Super User
Dual Super User

Other than the performance implications.  Why do you need to process all the rows in Parallel?  If you have 57 rows and set Parallel to 50 it will still process all 57 rows.  It will start processing for 50 of the rows and then start the final 7 as soon as other rows finish. So all 57 will be processed without changing anything.  They will just be throttled so that no more than 50 are being processed at a time.



-------------------------------------------------------------------------
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.

Join me at 365 and PWR EduCon in Chicago
EduCon Chicago 2022

The reason for parallel is that for each excel row there is an 'email address' field which is used to send the approval. For example, if 57 rows are returned then 57 different approvals must be sent out in parallel to 57 different email addresses on that particular day (Expiration Date).

 

So for in the example I gave, 7 approvals would not go out on the current day (Expiration Date) as they would have to wait for the other 50 to finish.

 

thanks for your reply

Ok, got it.  In that case I think you'll have to put an intermediate Loop between your Excel List action and your main processing Loop.  Prior to that Loop create two array variables. Inside the loop take the values from the first 50 objects and append them as objects to the first array variable.  Once the count reaches 50 append the rest to the second array variable.  Then create two parallel Apply to Each Loops for your main processing.  Process the first array in one and the second array in the other.  Set both for maximum parallel of 50.  If you have more than 100 you'll have to add a third variable and parallel loop.



-------------------------------------------------------------------------
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.

Join me at 365 and PWR EduCon in Chicago
EduCon Chicago 2022

Due to holidays and trying to work this out, I have eventually created a work around.

 

First of all, the flow which i created initially was too slow as the condition within the loop was being evaluated on each Excel row. This was causing a problem with the limit of 256 lines when Flow reads Excel. Instead I created a series of filters so that I got all the rows I needed and then fed them to the loops (see below);

 

50Limit Split.png

I created 2 Apply to each loops. The 1st one reads the filtered data but only the first 50 rows. This is done with the following Expression:  

 

take.PNG

This is also with the Apply to Each concurrency control set to 50.

 

The 2nd loop does the same but for rows only after the first 50:  

 

Skip.PNG

Again, concurrency control is set to 50.

 

I could then add a 3rd apply to each with the expression skip100.png and so on.

 

Downside of course is that you have to keep creating more Apply to Each loops if you are working with a lot of data.

 

I've tested this and it works.

After further testing I realised that my expressions were incorrect. To use multiple Apply to Each functions to get round the 50 parrallel limit, you would use the following expressions instead;

 

delete_this_image.PNG

 

 

Helpful resources

Announcements
 WHAT’S NEXT AT MICROSOFT IGNITE 2022

WHAT’S NEXT AT MICROSOFT IGNITE 2022

Explore the latest innovations, learn from product experts and partners, level up your skillset, and create connections from around the world.

Register for a Free Workshop.png

Register for a Free Workshop

Learn to digitize and optimize business processes and connect all your applications to share data in real time.

September Events 2022

Check out all of these events

Attend in person or online, there are incredible conferences and events happening all throughout the month of September.

Users online (2,939)