Hi There
I have created a flow which reads through a CSV file and posts the results to a list - it is not very fast c9 mins per file - but I have a couple of things I can do to improve that.
one thing I though I would be able to do is to kick off mulitple flows at the same time to process files in parallel. I did a test with 3 in parallel and they took 25 mins - nearly 3 times as long!
so my question is are we likely to see locking in this situation or are there other resource constraints that would limit true parallel running of flows
See in the image, the test run is on its own first in 9.32 and the 3 later runs are in parallel requiring 15, 20 & 25 mins
thanks for any hints JamboFlow run times
HI @Jambo1
The issue could be in the destination where the information will be stored. I don't think SharePoint handles efficiently parallel bulk loading of data since it has the overhead of having to validate the relationships between fields, for example, and persisting that into a database.
If you want speed, you should use any database that is suited for these kinds of jobs. SharePoint is better suited for other types of operations.
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.
Cheers
Manuel
Thanks @manuelstgomes ,
yes a DB would be better, but it is only a handful of rows that end up getting inserted and that step is done in 3 or 4 secs which is quite acceptable.
the slow part is looping round an array of the CSV data which doesnt make sense that it would be impacted by other flows doing the same. hence my question about whether there are other locks that I should consider
Jambo
Hi @Jambo1
If the CSV is significant, then Flow probably throttles the loading to keep the platform stable. I don't have any information or could find anything online, but this may be the issue.
I'm not sure, so I won't make claims and let someone from Microsoft take this one to confirm if my suspicion is correct or not.
Proud to be a Flownaut!
@Jcook weirdly adding parallelism to the Apply to each loops made it run slower. 9--> 15 mins.
Also as it was updating an array during that looping, you lose the control over what goes in there and I ended up with duplicates. so bad all round...
I suspect @manuelstgomes idea that there is some internal throttling from microsoft is the most likely scenario, but how to work around it I'm not sure
Jambo
User | Count |
---|---|
101 | |
36 | |
29 | |
25 | |
16 |
User | Count |
---|---|
133 | |
54 | |
53 | |
37 | |
26 |