cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Helper II
Helper II

Get files (properties only) - over 5000 files

Hi everyone

I'm using the Do until method to get more than 5,000 files outlined in this link as I need to filter a large library of around 50,000 files.  I can't use an OData filter query in the Get files steps as the columns I want to filter on are {Path} and {IsFolder} which don't seem to be accessible to the OData filter.  I was hoping to get all the documents in the library using the method in the link and then use a Filter array step to filter the data down (which would be a small fraction of the whole library and well under 5,000 files).

However, for some reason the Do until isn't working.  It does the first 5,000 files and then passes the ID of the last file successfully but then the second loop doesn't return any files:

Capture.JPG

In the Output for this the following shows:

Capture2.JPG

 

I've even tried setting up a test flow with just a Get files step with the Oata filter ID gt 18502 but this does the same and doesn't return any files.  The link I've used to set this up references a list rather than a library but according to the comments it should work in a library?  Is this not the case or is there something different I'd need to do to get more than 5,000 files?

Thanks in advance for any help.

3 REPLIES 3
Super User III
Super User III

Hello,

You will want to use pagination on the SharePoint action.

You can set this by clicking the 3 dots and settings on the action.

Enable pagination, and set the threshold value to a number higher than the files you want to return. (Example: 60000)

You may also want to configure retry settings in here as well. Set retry policy to Fixed interval,
And set the retry time to PT10S and number of retries to 50.

—Josh
If you like my post please hit the "Thumbs Up" -- If my post solved your issue please "Mark as a Solution" to help others




Did I answer your question? Mark my post as a solution!

If you like my post please hit the Thumbs Up


Proud to be a Flownaut!


Check out my blog for Power Automate tips, tricks, and guides
FlowAltDelete




@Jcook 

Thanks for the reply.  I've had a look and don't see the pagination option.  As I understand it pagination is only available for Get items (SP list) where I'm using Get files (SP Doc Library)?

Hi,

Hmm yes, so it seems to be a indexing limit and not a file limit.

Check this blog post out:

https://sharepointmaven.com/how-to-overcome-sharepoint-5000-item-limit-threshold/

—Josh
If you like my post please hit the "Thumbs Up" -- If my post solved your issue please "Mark as a Solution" to help others




Did I answer your question? Mark my post as a solution!

If you like my post please hit the Thumbs Up


Proud to be a Flownaut!


Check out my blog for Power Automate tips, tricks, and guides
FlowAltDelete




Helpful resources

Announcements
Microsoft Ignite

Microsoft Ignite

Join digitally, March 2–4, 2021 to explore new tech that's ready to implement. Experience the keynote in mixed reality through AltspaceVR!

New Super Users

Meet the Power Automate Super Users!

Many congratulations to the Season 1 2021 Flownaut Crew!

Microsoft Ignite

Microsoft Power Platform: 2021 Release Wave 1 Plan

Power Platform release plan for the 2021 release wave 1 describes all new features releasing from April through September 2021.

MPA Community Blog

Power Automate Community Blog

Check out the community blog page where you can find valuable learning material from community and product team members!

Users online (22,187)