cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
KishoreJ
Frequent Visitor

Skip Token variable handling large volume of data

Hi All,

 

I'm trying to use Skip token feature for splitting the total data count to batches , so that we can enable parallelism.

We set  the Apply to each action Concurrency control to 20, but still the performance didn't improve.

Scenario : We need to process 20k records in dataverse(trying to update) using Power Automate, but while using Apply to each it's taking 50 mins approx for processing ,

 

A) How to improve the performance for this scenario ,let say reduce the processing time

B) How to achieve this in dynamic Parallelism using batches 

 

Appreciate your suggestion.

3 REPLIES 3
cchannon
Super User
Super User

So, really the answer depends on the service you're trying to call (I am guessing Graph API?), but almost certainly Skip Token isn't going to open you up for parallelization. Skip Token tells the API what the last page was you called so it knows to start you on the next. So, if you call for (let's just call it) page one and get Skip Token XXXXXX, then call that same API 1, 20, or 2000 times with Skip Token XXXXXX, it will always give you page two. So, if you're pulling 100 records per page, then on each iteration each of your 20 threads is just running the same 100 records.

 

Instead, take a look at the API documentation for whatever service you are trying to call. See if it offers you a $skip or similar query parameter that could allow you to skip a set number of rows per call. If so, then you can set each of your threads to skip a different number of rows, forcing them to look at different pages instead of all at the same one.

Thanks @cchannon  for your response, but the real issue here is  the parallelism is not working as expected.

 

In our scenario we use skip token variable for holding the next page token value, with that value we are creating 4 array condition to execute concurrently.

First Array - 1 to 5k

Second Array - 5001 to 10k

Third Array - 10001 to 15 k

Fourth Array  - 15001 to 20 K

Here even after the process got completed flow continues to run, if you see the flow history First Array is not stopping even after execution completed. It is running for a while and then it is stopping.

Those 4 arrays aren't executing in parallel if you are using skip token. You use the skip token from the first array to create the second array, third array needs seconds token before it can get created, and so on. The apply to each is working in parallel one array at a time. The only way apply to each is going to be noticeably faster is when all your data is going to separate places, it doesn't fix database waits. Try running it without doing anything inside the apply to each, if its fast than the issues is on the db insert side.

Helpful resources

Announcements
Power Platform Call June 2022 768x460.png

Power Platform Community Call

Join us for the next call on August 17, 2022 at 8am PDT.

Power Platform Conf 2022 768x460.jpg

Join us for Microsoft Power Platform Conference

The first Microsoft-sponsored Power Platform Conference is coming in September. 100+ speakers, 150+ sessions, and what's new and next for Power Platform.

Users online (3,161)