cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
PURNA429
Resolver II
Resolver II

Set variable and Compose actions in apply to each loop with Concurrency Control and Degree of Parallelism

Hi,

I have a flow which queries a SharePoint list and then create a csv out of it.  List contains 100k items and GetItems action with filter condition would give me around 20k items. I need to process these 20k items inside an apply to each loop to convert date time formats  . For this I am setting a variable to store the field value and then using a compose action to convert the date time format like below.

 

ApplytoEach.PNG

 

Before having this date conversion functionality inside apply to each loop , to speedup the process, I have set the Concurrency Control with Degree of Parallelism as 30 and my flow was running fast and could get the csv quickly. Here is the setting

SettingsAppltoEach.PNG

 

But with the introduction of these additional actions (like set variable, compose etc.) inside apply to each I got warning message "Enable concurrency control for the apply to each loop and set its degree of parallelization to 1 when using a Set Variable action inside an apply to each loop."

 

With the concurrency ON and degree of parallelism set to 1,  my flow is taking 3-4 hours to process 20k records . So question here is, with the concurrency setting on, should we not to use Set variable and compose actions.? Can some one clearly explain me what will happen if we set. And in case if we should not use the parallelism what are the best options to speed up this kind of process.?

 

-Purna

1 ACCEPTED SOLUTION

Accepted Solutions
ManishJain
Solution Sage
Solution Sage

Hi @PURNA429 ,

Compose action inside loop becomes object and then you can access it outside using another compose action to get the output of compose as an array. so you do not need to use set variable inside loop. Also the set variable will not work with concurrency. 

 

Do refer to this blog:

 

https://sharepains.com/2019/07/09/compose-apply-to-each-power-automate/

 

Thanks

 

View solution in original post

6 REPLIES 6
ManishJain
Solution Sage
Solution Sage

Hi @PURNA429 ,

 

If your aim is to get array of {Email and Start Date} for large number(10k) then in the loop just use compose action with your variables set inside it . You have to use expressions in one step for setting formatted date. 

 

Also in order to get the array of data just use Compose action outside loop and just use the expression outputs('Compose'), which you are using inside loop, please change the outputs('Compose') to what you are using.

 

Thanks

Thanks @ManishJain  for the reply. Can you please elaborate more on "use Compose action outside loop"? if I keep compose action outside, how would i get the iterated value ?

 

Also with this change can I set back the parallelism to 30 . Will this speedup the process?

 

Thanks

Purna

ManishJain
Solution Sage
Solution Sage

Hi @PURNA429 ,

Compose action inside loop becomes object and then you can access it outside using another compose action to get the output of compose as an array. so you do not need to use set variable inside loop. Also the set variable will not work with concurrency. 

 

Do refer to this blog:

 

https://sharepains.com/2019/07/09/compose-apply-to-each-power-automate/

 

Thanks

 

View solution in original post

Thanks alot @ManishJain  for the guidance. 

Here is my flow, will this be okay with concurrency ON and with parallelism of 20.?

Compose in apply to each.png

 

Here I am writing this output to a csv. But I need to format the date before creating the csv.

Here my StartDate column is datetime type and I am not finding a way to format it (expressions needs string input) in inner compose action. How can we achieve this.. Can you suggest.

Thanks again for your help !

 

-Purna

ManishJain
Solution Sage
Solution Sage

Hi @PURNA429 ,

 

If your date coming from Sharepoint is actually of type datetime then you can use formatdatetime in expressions and choose the format of your choice like 'dd-MM-yyyy'. 

 

 

 

Thanks

Thank you @ManishJain .

 

Implemented logic like below . Putting it here for community purpose.

 

  • Inside apply to each, one compose action to format the date and second compose to create an array with other fields ( In original workflow I have more fields ). 
  • Outside of apply to each loop used the outputs('InnerComposeAction') as input to my create csv.

 

FinalOutput.png

 

This worked well without any issues. 

 

In actual workflow with few other business conditions and setting degree of parallelism as 40 , flow took ~16 minutes to create a csv with 25k records from a list with 200k+ items .

 

-Purna

Helpful resources

Announcements
UG GA Amplification 768x460.png

Launching new user group features

Learn how to create your own user groups today!

Community Connections 768x460.jpg

Community & How To Videos

Check out the new Power Platform Community Connections gallery!

M365 768x460.jpg

Microsoft 365 Collaboration Conference | December 7–9, 2021

Join us, in-person, December 7–9 in Las Vegas, for the largest gathering of the Microsoft community in the world.

Top Solution Authors
Top Kudoed Authors
Users online (1,148)