How to use $skipToken to get more than 18000 data out when the limit is 1000-HTTP request

[meklulose]

I am using Http post request to get Resources out and into storage. However the http request has limitation of 1000 row per request. It does give you skip token which you can use to get more data. I am confused about how do I got about this. I can use the $skiptoken to get another 1000 rows and then will have to use $skiptoken from that run to get another 1000 and so on.  If i create a for each loop, is there way to get $skiptoken from previous run and use it for the next run and carry on doing so until there is no $skip token? What is the best way of approaching this?

[Nived_Nambiar]

Hi @meklulose 

is the skiptoken can only accept 1000 as max value?

also could you do any filter query which helps to limit search and be more effiecient ?

[meklulose]

@Nived_Nambiar after using the skiptoken from previous http request and running it again only brings back 1000. This is the documentation I am looking at Resources - Resources - REST API (Azure Azure Resource Graph) | Microsoft Learn Resources - Resources - REST API (Azure Azure Resource Graph) | Microsoft Learn. I need to get all of the data out so can't filter query. 

[Nived_Nambiar]

Hi @meklulose 

I think i have understood your question, give me sometime, i will come back with a flow  design 🙂

[Nived_Nambiar · Accepted solution]

Hi @meklulose 

This is the flow design i am taking about, to continue till skiptoken is not avaialble, i would recommend to try do until loop 

Flow design

Step 1:

we will initialize two variables, first one is array Final_Data, which would storing responses from http request

second one is a boolean variable skiptokenavailable, which checks skiptoken is available or not in response of API.

Step 2:

will use a do until loop like below

In the loop, we will do as below

we will invoke http request to call that azure resource manager (arm) API using the HTTP Request using below action

Now using condition, we will check whether the skiptoken is present or not

expression used on left side

body('Invoke_an_HTTP_request')?['$skipToken']

expression used in right side : null

it is checking whether the skiptoken is present there or not

if it skiptoken is present, in yes branch, do like this 

append the details from http request body to final_data variable and set skiptokenavailable variable as true

if the condition is false, do like below

append the body of http response to FinalData array variable and set skiptokenavailable variable as false 

if the skiptokenavailable becomes false, then loop will be skipped as it runs until skiptokenavailable becomes false.

Hope this helps !

Mark it as solution if it resolves your query !

[VictorIvanidze]

You request should return the following:

"@odata.nextLink"

Just use its value as the URL of the next request.

[meklulose]

thank you for the solution!

[Anonymous]

how do you loop through several 

"@odata.nextLink"

The first http request will get you one, the next request will get you a different  one and so on.  I would like to avoid manually adding another action to get the next page and so on.

[VictorIvanidze]

Create a value and you don't need to "manually add another action".