cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
LYNCHJL
Helper II
Helper II

Filter Search Gallery with more than 3000 SP List Items

Hello, 

 

I have a SP list with more than 3000 items.  

 

My gallery has a search box with the gallery set as below:

 

Filter(SOURCE, User().Email in 'Users', Year = 'Year Search Select_1'.Selected.Value, TextSearchBox1_1.Text in Description || TextSearchBox1_1.Text in TAGS || TextSearchBox1_1.Text in Atmosphere || TextSearchBox1_1.Text in 'HR' || TextSearchBox1_1.Text in 'Finance' || TextSearchBox1_1.Text in 'MGT ' || TextSearchBox1_1.Text in Location)

 

How can I pull in all data into a collection and then sort?  I've tried adding the below to OnStart but still only able to pull first 2000 records.

 

Concurrent(ClearCollect(CollectionA,Filter('SOURCE',ID<2000)),ClearCollect(CollectionB,Filter('SOURCE',ID >=2000 And ID<4000)));ClearCollect(AllIncidentsSource,CollectionA,CollectionB)

 

thanks!

3 ACCEPTED SOLUTIONS

Accepted Solutions

Hi @LYNCHJL ,

That code will never work as ID is not delegable.

I posted a workaround here a little while back.

 

Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.

 

View solution in original post

@LYNCHJL ,

No matter what you do with the ID, you cannot use > (greater than) on more than the first 2000 records. The only delegable operator you can use with ID is = (equals). I will post the full solution below

I keep a "shadow" ID number numeric field in the list and do this on the OnSuccess of my new record Forms. 

UpdateIf(
    MyListName,
    ID = MyNewForm.LastSubmit.ID,
    {MyShadowID: MyNewForm.LastSubmit.ID}
)

You will have to populate all existing records with the present ID number.

You can then Collect as many records as you want with

ClearCollect(
   colTemp,
   MySPList
);
If(
   CountRows(colTemp) = 2000,
   Set(
      vID,
      Max(colTemp,IDRef)  //My Shadow field
   );
   Collect(
      colTemp,
      Filter(
         MySPList,
         IDRef>vID
      )
   )
);
If(
   CountRows(colTemp) = 4000,
   Set(
      vID,
      Max(colTemp,IDRef)
   );
   Collect(
      colTemp,
      Filter(
         MySPList,
         IDRef>vID
      )
   )
)

Then keep going in batches of 2000 - If you do this to both lists, you will have fully delegable In filters.

 

Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.

 

View solution in original post

Hi @LYNCHJL ,

Strangely that is exactly what I do, but usually when I mention Workflows, I get confused questions . . .

Yes, by far the easiest way to set the number is a SharePoint Designer 2013 Workflow - set on Item Creation with Set Field in Current Item - set ItemID to ID. It works very quickly in the background and required no other attention.
You do however need to create a new numeric field and set it to the ID (easiest way to get a sequential number) I call mine IDRef.

Also do not use Concurrent - you may have deleted records and the Max() in my code picks up the highest number in the 2000 collected previously.

 

Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.

 

 

View solution in original post

8 REPLIES 8
eka24
Super User III
Super User III

Have you checked the delegation limit in File....Settings...Advanced Setting if the delegation is increased to 2000 because you are pulling in bataches of two 2000

 

You can then try:

 

Concurrent(

            ClearCollect(CollectionA, Filter(SOURCE,ID >= 1 && ID <= 2000)),

            ClearCollect(CollectionB, Filter(SOURCE,ID >= 2001 && ID <= 4000)));

ClearCollect(AllIncidentSource, CollectionA, CollectionB)

 

Edited

------------

If you like this post, give a Thumbs up. Where it solved your request, Mark it as a Solution to enable other users find it.

 

Hi @LYNCHJL ,

That code will never work as ID is not delegable.

I posted a workaround here a little while back.

 

Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.

 

View solution in original post

Hi @eka24,

 

yes I did check and it is set at 2000. 

eka24
Super User III
Super User III

I Updated my post please check

------------

If you like this post, give a Thumbs up. Where it solved your request, Mark it as a Solution to enable other users find it.

@LYNCHJL ,

No matter what you do with the ID, you cannot use > (greater than) on more than the first 2000 records. The only delegable operator you can use with ID is = (equals). I will post the full solution below

I keep a "shadow" ID number numeric field in the list and do this on the OnSuccess of my new record Forms. 

UpdateIf(
    MyListName,
    ID = MyNewForm.LastSubmit.ID,
    {MyShadowID: MyNewForm.LastSubmit.ID}
)

You will have to populate all existing records with the present ID number.

You can then Collect as many records as you want with

ClearCollect(
   colTemp,
   MySPList
);
If(
   CountRows(colTemp) = 2000,
   Set(
      vID,
      Max(colTemp,IDRef)  //My Shadow field
   );
   Collect(
      colTemp,
      Filter(
         MySPList,
         IDRef>vID
      )
   )
);
If(
   CountRows(colTemp) = 4000,
   Set(
      vID,
      Max(colTemp,IDRef)
   );
   Collect(
      colTemp,
      Filter(
         MySPList,
         IDRef>vID
      )
   )
)

Then keep going in batches of 2000 - If you do this to both lists, you will have fully delegable In filters.

 

Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.

 

View solution in original post

Thanks @WarrenBelz .

 

Can I replace ID with a new number field (lets say ItemID) that gets assigned by a workflow on creation?  I could set the workflow to set ItemID as ID.  I would then use ItemID in my ClearConnect statement.

 

Would this work or would i run into the same issue?

 

thanks!

eka24
Super User III
Super User III

The number filed should be an incremental field, so you cant use the ItemID Field.

Concurrent(

            ClearCollect(CollectionA, Filter(SOURCE,ID >= 1 && ID <= 2000)),

            ClearCollect(CollectionB, Filter(SOURCE,ID >= 2001 && ID <= 4000)));

ClearCollect(AllIncidentSource, CollectionA, CollectionB)

------------

If you like this post, give a Thumbs up. Where it solved your request, Mark it as a Solution to enable other users find it.

 

Hi @LYNCHJL ,

Strangely that is exactly what I do, but usually when I mention Workflows, I get confused questions . . .

Yes, by far the easiest way to set the number is a SharePoint Designer 2013 Workflow - set on Item Creation with Set Field in Current Item - set ItemID to ID. It works very quickly in the background and required no other attention.
You do however need to create a new numeric field and set it to the ID (easiest way to get a sequential number) I call mine IDRef.

Also do not use Concurrent - you may have deleted records and the Max() in my code picks up the highest number in the 2000 collected previously.

 

Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.

 

 

View solution in original post

Helpful resources

Announcements
PA User Group

Welcome to the User Group Public Preview

Check out new user group experience and if you are a leader please create your group

secondImage

Demo Extravaganza Winner Announcement

Please join us on Wednesday, July 21st at 8a PDT. We will be announcing the Winners of the Demo Extravaganza!

V3_PVA CAmpaign Carousel.png

Community Challenge - Giveaways!

Participate in the Power Virtual Agents Community Challenge

Carousel 2021 Release Wave 2 Plan 768x460.jpg

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

Top Solution Authors
Top Kudoed Authors
Users online (1,718)