Flow Performance with parallel flows

[Jambo1]

Hi There

I have created a flow which reads through a CSV file and posts the results to a list - it is not very fast c9 mins per file - but I have a couple of things I can do to improve that. 

one thing I though I would be able to do is to kick off mulitple flows at the same time to process files in parallel. I did a test with 3 in parallel and they took 25 mins - nearly  3 times as long!

so my question is are we likely to see locking in this situation or are there other resource constraints that would limit true parallel running of flows

See in the image, the test run is on its own first in 9.32 and the 3 later runs are in parallel requiring 15, 20 & 25 mins

thanks for any hints JamboFlow run times

[manuelstgomes]

HI @Jambo1 

The issue could be in the destination where the information will be stored. I don't think SharePoint handles efficiently parallel bulk loading of data since it has the overhead of having to validate the relationships between fields, for example, and persisting that into a database.

If you want speed, you should use any database that is suited for these kinds of jobs. SharePoint is better suited for other types of operations.

If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.
​
Cheers
Manuel

[Jambo1]

Thanks @manuelstgomes ,

yes a DB would be better, but it is only a handful of rows that end up getting inserted and that step is done in 3 or 4 secs which is quite acceptable. 

the slow part is looping round an array of the CSV data which doesnt make sense that it would be impacted by other flows doing the same. hence my question about whether there are other locks that I should consider

Jambo

[manuelstgomes]

 Hi @Jambo1 

If the CSV is significant, then Flow probably throttles the loading to keep the platform stable. I don't have any information or could find anything online, but this may be the issue.

I'm not sure, so I won't make claims and let someone from Microsoft take this one to confirm if my suspicion is correct or not.

[Jcook]

Hello @Jambo1,

Do you have any Apply to each loops?

If you do please try turning up the concurrency.

This can be done by clicking the 3 dots on the apply to each shape, than selecting settings. Here turn on Concurrency and set it’s value to the highest. (50).

—Josh
If you like my post please hit the "Thumbs Up" -- If my post solved your issue please "Mark as a Solution" to help others

[Jambo1]

@Jcook weirdly adding parallelism to the Apply to each loops made it run slower. 9--> 15 mins. 

Also as it was updating an array during that looping, you lose the control over what goes in there and I ended up with duplicates. so bad all round...

I suspect @manuelstgomes  idea that there is some internal throttling from microsoft is the most likely scenario, but how to work around it I'm not sure

Jambo