Set variable and Compose actions in apply to each loop with Concurrency Control and Degree of Parallelism

[PURNA429]

Hi,

I have a flow which queries a SharePoint list and then create a csv out of it.  List contains 100k items and GetItems action with filter condition would give me around 20k items. I need to process these 20k items inside an apply to each loop to convert date time formats  . For this I am setting a variable to store the field value and then using a compose action to convert the date time format like below.

Before having this date conversion functionality inside apply to each loop , to speedup the process, I have set the Concurrency Control with Degree of Parallelism as 30 and my flow was running fast and could get the csv quickly. Here is the setting

But with the introduction of these additional actions (like set variable, compose etc.) inside apply to each I got warning message "Enable concurrency control for the apply to each loop and set its degree of parallelization to 1 when using a Set Variable action inside an apply to each loop."

With the concurrency ON and degree of parallelism set to 1,  my flow is taking 3-4 hours to process 20k records . So question here is, with the concurrency setting on, should we not to use Set variable and compose actions.? Can some one clearly explain me what will happen if we set. And in case if we should not use the parallelism what are the best options to speed up this kind of process.?

-Purna

[ManishJain]

Hi @PURNA429 ,

If your aim is to get array of {Email and Start Date} for large number(10k) then in the loop just use compose action with your variables set inside it . You have to use expressions in one step for setting formatted date. 

Also in order to get the array of data just use Compose action outside loop and just use the expression outputs('Compose'), which you are using inside loop, please change the outputs('Compose') to what you are using.

Thanks

[PURNA429]

Thanks @ManishJain  for the reply. Can you please elaborate more on "use Compose action outside loop"? if I keep compose action outside, how would i get the iterated value ?

Also with this change can I set back the parallelism to 30 . Will this speedup the process?

Thanks

Purna

[ManishJain · Accepted solution]

Hi @PURNA429 ,

Compose action inside loop becomes object and then you can access it outside using another compose action to get the output of compose as an array. so you do not need to use set variable inside loop. Also the set variable will not work with concurrency. 

Do refer to this blog:

https://sharepains.com/2019/07/09/compose-apply-to-each-power-automate/

Thanks

[PURNA429]

Thanks alot @ManishJain  for the guidance. 

Here is my flow, will this be okay with concurrency ON and with parallelism of 20.?

Here I am writing this output to a csv. But I need to format the date before creating the csv.

Here my StartDate column is datetime type and I am not finding a way to format it (expressions needs string input) in inner compose action. How can we achieve this.. Can you suggest.

Thanks again for your help !

-Purna

[ManishJain]

Hi @PURNA429 ,

If your date coming from Sharepoint is actually of type datetime then you can use formatdatetime in expressions and choose the format of your choice like 'dd-MM-yyyy'. 

Thanks

[PURNA429]

Thank you @ManishJain .

Implemented logic like below . Putting it here for community purpose.

• Inside apply to each, one compose action to format the date and second compose to create an array with other fields ( In original workflow I have more fields ). 
• Outside of apply to each loop used the outputs('InnerComposeAction') as input to my create csv.

This worked well without any issues. 

In actual workflow with few other business conditions and setting degree of parallelism as 40 , flow took ~16 minutes to create a csv with 25k records from a list with 200k+ items .

-Purna