Working with SharePoint lists larger than 5000 items?

[Noah]

I have built several flows that successfully use the "When existing item is modified" trigger for SharePoint. However in a list that exceeds 5000 items, the flow fails and returns the error below (output body of trigger action). Has anyone used a SharePoint list greater than 5000 items with Flow? How can I avoid the error below?

I'm not trying to query for the entire list, just simply triggering when any single item is updated.

{ "status": 502, "message": "The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.", "source": "https://workcycle.sharepoint.com/sites/hub/assetmgmt/_api/SP.APIHubConnector.GetUpdatedListItems", "errors": [ "-2147024860", "Microsoft.SharePoint.SPQueryThrottledException" ] }

[v-monli-msft]

Hi @Noah,

I thought there actually is a limitation in Flow when using the trigger, but I am not sure the exact number. I will report this to the pg and will update here when there is any answer.

Regards,

Mona 

[v-monli-msft]

Hi @Noah,

Yes, 5000 items is the limit for modified trigger since we run a complex query that needs to go through all items to find recently modified ones. And those get throttled when hit a list bigger than 5000 items.

So this is a known limitation, if you want to extend, please consider to submit an idea in the Flow Ideas forum, so it might be considered for future releases.

https://powerusers.microsoft.com/t5/Flow-Ideas/idb-p/FlowIdeas

Regards,

Mona Li

[Noah]

I see. Shouldn't there be a big red warning in this trigger description that tells users it will stop working the day they add item #5000? This seems to me a fundamental blocker to using Flow with SharePoint, if you run the risk of your entire process breaking without recourse after an arbitrary limit is reached. This is the conversation with the business: "You know that fancy automated workflow we created and came to rely on? Well the other day it stopped working and will never work again -- sorry!"

Hopefully webhooks can be used instead for this trigger? This is GA and seems like it wouldn't have this issue:

https://dev.office.com/blogs/sharepoint-webhooks-is-now-generally-available-build-service-oriented-p...

[Adi2ro]

I can't even use a simple Filter if the list is above 5000 so nice job there limiting ALL operations withing Powerapps if the item count is greated than 5000.

[krainer]

You can use a filter if the returned number is lower than 5000 and you use indexes - just note that the order in your filter is important - e.g.:

If I have 10000 items in a list, and of these 7500 has "Status2" = "Active" + 500 has "Status1" not equal "Disabled" + both Status1 and Status2 are indexed columns

This will work:

(Status1 ne 'Disabled') and (Status2 eq 'Active')

This will fail (even though the combined filter is less than the 5000):

(Status2 eq 'Active') and (Status1 ne 'Disabled')

But really an issue with the trigger not working (not even with subfolders) -> Seems either a webhook or multiple lists is the solution here 😞

[ohthreesixfive]

*Posted this as a response elsewhere to a question, but putting it here in case someone stumbles onto this and finds it useful.

Hi @Noah,

I ran into this very issue recently after having gone live with a new document library to store a large number of files that were being hosted on a different (and costly) cloud platform. Of course, during testing the Flow was working great (picking up newly created and modified files) - but as soon as I brought everything over (73k+ files in total) and 'turned the lights on' the trigger would no longer trigger!

Anyways, I was able to resolve it by turning on indexing on the 'Modified' column. Since I hadn't seen this documented anywhere (and I did a lot of searching), I went ahead and created a post to walk you through the process in case you need more detailed guidance. Here's the link:

https://www.ohthreesixtyfive.org/2018/03/microsoft-flow-overcoming-the-list-view-threshold.html

Of course, if you have exceeded 20k items you'll need to trim it below that to be able to turn on indexing, but otherwise (Update: The 20k item limit to turn on indexing has, mercifully, been lifted! You can proceed to turn on indexing without needing to trim back your content now.) it should be pretty painless. Let me know how it goes!

Best regards,

Ramiro