cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Highlighted
degvalentine
Level: Powered On

How to Avoid Do-Until Loops and Improve Performance

A common use case is to iterate paginated results from some API when we don't know how many pages there will be until we start querying. On our first request we get the first page of results along with a count of total items or remaining pages.

 

The natural solution is a Do Until loop with a page index variable, i.e. "keep getting next pages until the returned page is the last page".

 

2019-05-08 13_12_50-Edit your flow _ Microsoft Flow.png

 

There are 2 issues with this approach: [1] we must process pages sequentially (i.e. slowly) and [2] Do-Until loops are difficult to debug because they often obscure errors within an iteration.

 

You can save yourself from both by replacing the Do-Until with an Apply-to-Each over a pre-computed page range.

 

General approach:

  1. make a request to find the total pages we will have
  2. make a "range()" of pages
  3. Apply-to-Each over the range

 

Doing this will allow you to process multiple pages concurrently and to get better error reporting inside the loop. It comes at the tiny cost of 1 additional request.

 

2019-05-08 13_28_25-Edit your flow _ Microsoft Flow.png

 

Composing the page range could be a little tricky, since Flow doesn't have any "round up" function (hint, hint).

 

Here's your workaround:

 

// assumes you have variables for total items to fetch and a page size inside vars
range(
1, add( div( variables('itemCount'), variables('pageSize') ), if ( greater( mod( variables('itemCount'), variables('pageSize') ), 0 ), 1, 0 ) )
)

 

If you have so many pages that setting "Degree of Parallelism" to 50 just doesn't seem like enough, you could chunk your page list and toss that into an Apply-to-Each as well (boosting your concurrent page limit from 50 to 2,500).

 

2019-05-08 13_38_34-Edit your flow _ Microsoft Flow.png

 

// CHUNK PAGES INTO GROUPS OF 50

range(
  0,
  add(
    div(
      length(outputs('Compose_range(...)_of_pages')),
      50
    ),
    if (
      greater(
        mod(
          length(outputs('Compose_range(...)_of_pages')),
          50
        ),
        0
      ),
      1,
      0
    )
  )
)

// EXECUTE YOUR MAIN FLOW ON EACH CHUNK

take(
  skip(
    outputs('Compose_range(...)_of_pages'),
    mul(
      item(),
      50
    )
  ),
  50
)

Good luck avoiding those slow-running, error-hiding Do Untils!

 

1 REPLY 1
MichelleWongNL
Level: Powered On

Re: How to Avoid Do-Until Loops and Improve Performance

Thx for sharing!

Helpful resources

Announcements
firstImage

Microsoft Flow Online Conference

Join us for a FULL day of FREE Microsoft Flow Sessions from some of the best minds in the industry!

firstImage

Watch Sessions On Demand!

Continue your learning in our online communities.

Power Platform 2019 Release Wave 2 Plan

Power Platform 2019 Release Wave 2 Plan

Features releasing from October 2019 through March 2020

thirdimage

Flow Community User Group Member Badge

Fill out a quick form to claim your user group badge now!

thirdImage

New Flow Community Board!

Check out the new Microsoft Flow Community Blog Topic Suggestion board!

fifthimage

Microsoft Learn

Learn how to build the business apps that you need

sixthImage

Power Platform World Tour

Find out where you can attend!

seventhimage

Webinars & Video Gallery

Watch & learn from the Flow Community Video Gallery!

Top Kudoed Authors
Users Online
Currently online: 49 members 4,081 guests
Please welcome our newest community members: