I'm trying to use Skip token feature for splitting the total data count to batches , so that we can enable parallelism.
We set the Apply to each action Concurrency control to 20, but still the performance didn't improve.
Scenario : We need to process 20k records in dataverse(trying to update) using Power Automate, but while using Apply to each it's taking 50 mins approx for processing ,
A) How to improve the performance for this scenario ,let say reduce the processing time
B) How to achieve this in dynamic Parallelism using batches
Appreciate your suggestion.
So, really the answer depends on the service you're trying to call (I am guessing Graph API?), but almost certainly Skip Token isn't going to open you up for parallelization. Skip Token tells the API what the last page was you called so it knows to start you on the next. So, if you call for (let's just call it) page one and get Skip Token XXXXXX, then call that same API 1, 20, or 2000 times with Skip Token XXXXXX, it will always give you page two. So, if you're pulling 100 records per page, then on each iteration each of your 20 threads is just running the same 100 records.
Instead, take a look at the API documentation for whatever service you are trying to call. See if it offers you a $skip or similar query parameter that could allow you to skip a set number of rows per call. If so, then you can set each of your threads to skip a different number of rows, forcing them to look at different pages instead of all at the same one.
Thanks @cchannon for your response, but the real issue here is the parallelism is not working as expected.
In our scenario we use skip token variable for holding the next page token value, with that value we are creating 4 array condition to execute concurrently.
First Array - 1 to 5k
Second Array - 5001 to 10k
Third Array - 10001 to 15 k
Fourth Array - 15001 to 20 K
Here even after the process got completed flow continues to run, if you see the flow history First Array is not stopping even after execution completed. It is running for a while and then it is stopping.
Those 4 arrays aren't executing in parallel if you are using skip token. You use the skip token from the first array to create the second array, third array needs seconds token before it can get created, and so on. The apply to each is working in parallel one array at a time. The only way apply to each is going to be noticeably faster is when all your data is going to separate places, it doesn't fix database waits. Try running it without doing anything inside the apply to each, if its fast than the issues is on the db insert side.
Join us for the next call on August 17, 2022 at 8am PDT.
The first Microsoft-sponsored Power Platform Conference is coming in September. 100+ speakers, 150+ sessions, and what's new and next for Power Platform.