cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
gj84uw
Advocate II
Advocate II

Get files (properties only) - over 5000 files

Hi everyone

I'm using the Do until method to get more than 5,000 files outlined in this link as I need to filter a large library of around 50,000 files.  I can't use an OData filter query in the Get files steps as the columns I want to filter on are {Path} and {IsFolder} which don't seem to be accessible to the OData filter.  I was hoping to get all the documents in the library using the method in the link and then use a Filter array step to filter the data down (which would be a small fraction of the whole library and well under 5,000 files).

However, for some reason the Do until isn't working.  It does the first 5,000 files and then passes the ID of the last file successfully but then the second loop doesn't return any files:

Capture.JPG

In the Output for this the following shows:

Capture2.JPG

 

I've even tried setting up a test flow with just a Get files step with the Oata filter ID gt 18502 but this does the same and doesn't return any files.  The link I've used to set this up references a list rather than a library but according to the comments it should work in a library?  Is this not the case or is there something different I'd need to do to get more than 5,000 files?

Thanks in advance for any help.

4 REPLIES 4
Jcook
Super User
Super User

Hello,

You will want to use pagination on the SharePoint action.

You can set this by clicking the 3 dots and settings on the action.

Enable pagination, and set the threshold value to a number higher than the files you want to return. (Example: 60000)

You may also want to configure retry settings in here as well. Set retry policy to Fixed interval,
And set the retry time to PT10S and number of retries to 50.

—Josh
If you like my post please hit the "Thumbs Up" -- If my post solved your issue please "Mark as a Solution" to help others

Did I answer your question? Mark my post as a solution!

If you like my post please hit the Thumbs Up


Proud to be a Flownaut!


Check out my blog for Power Automate tips,
tricks, and guides
FlowAltDelete





@Jcook 

Thanks for the reply.  I've had a look and don't see the pagination option.  As I understand it pagination is only available for Get items (SP list) where I'm using Get files (SP Doc Library)?

Hi,

Hmm yes, so it seems to be a indexing limit and not a file limit.

Check this blog post out:

https://sharepointmaven.com/how-to-overcome-sharepoint-5000-item-limit-threshold/

—Josh
If you like my post please hit the "Thumbs Up" -- If my post solved your issue please "Mark as a Solution" to help others

Did I answer your question? Mark my post as a solution!

If you like my post please hit the Thumbs Up


Proud to be a Flownaut!


Check out my blog for Power Automate tips,
tricks, and guides
FlowAltDelete





DeeTronSEAM
Advocate V
Advocate V

FWIW as of now you can use SharePoint's Get files (properties only) action with pagination.

 

Here is an excellent article on how to deal with the collection size limits (e.g. SharePoint page size, Top count, Threshold):

 

https://alextofan.com/2019/08/22/how-to-get-more-than-5000-item-from-sharepoint-online-in-flow/ 

Helpful resources

Announcements
UG GA Amplification 768x460.png

Launching new user group features

Learn how to create your own user groups today!

Community Connections 768x460.jpg

Community & How To Videos

Check out the new Power Platform Community Connections gallery!

Welcome Super Users.jpg

Super User Season 2

Congratulations, the new Super User Season 2 for 2021 has started!

Carousel 2021 Release Wave 2 Plan 768x460.jpg

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

Users online (2,842)