Filter Search Gallery with more than 3000 SP List Items

[LYNCHJL]

Hello, 

I have a SP list with more than 3000 items.  

My gallery has a search box with the gallery set as below:

Filter(SOURCE, User().Email in 'Users', Year = 'Year Search Select_1'.Selected.Value, TextSearchBox1_1.Text in Description || TextSearchBox1_1.Text in TAGS || TextSearchBox1_1.Text in Atmosphere || TextSearchBox1_1.Text in 'HR' || TextSearchBox1_1.Text in 'Finance' || TextSearchBox1_1.Text in 'MGT ' || TextSearchBox1_1.Text in Location)

How can I pull in all data into a collection and then sort?  I've tried adding the below to OnStart but still only able to pull first 2000 records.

Concurrent(ClearCollect(CollectionA,Filter('SOURCE',ID<2000)),ClearCollect(CollectionB,Filter('SOURCE',ID >=2000 And ID<4000)));ClearCollect(AllIncidentsSource,CollectionA,CollectionB)

thanks!

[eka24]

Have you checked the delegation limit in File....Settings...Advanced Setting if the delegation is increased to 2000 because you are pulling in bataches of two 2000

You can then try:

Concurrent(

            ClearCollect(CollectionA, Filter(SOURCE,ID >= 1 && ID <= 2000)),

            ClearCollect(CollectionB, Filter(SOURCE,ID >= 2001 && ID <= 4000)));

ClearCollect(AllIncidentSource, CollectionA, CollectionB)

Edited

------------

If you like this post, give a Thumbs up. Where it solved your request, Mark it as a Solution to enable other users find it.

[WarrenBelz · Accepted solution]

Hi @LYNCHJL ,

That code will never work as ID is not delegable.

I posted a workaround here a little while back.

Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.

[LYNCHJL]

Hi @eka24,

yes I did check and it is set at 2000. 

[eka24]

I Updated my post please check

------------

If you like this post, give a Thumbs up. Where it solved your request, Mark it as a Solution to enable other users find it.

[WarrenBelz · Accepted solution]

@LYNCHJL ,

No matter what you do with the ID, you cannot use > (greater than) on more than the first 2000 records. The only delegable operator you can use with ID is = (equals). I will post the full solution below

I keep a "shadow" ID number numeric field in the list and do this on the OnSuccess of my new record Forms. 

UpdateIf(
    MyListName,
    ID = MyNewForm.LastSubmit.ID,
    {MyShadowID: MyNewForm.LastSubmit.ID}
)

You will have to populate all existing records with the present ID number.

You can then Collect as many records as you want with

ClearCollect(
   colTemp,
   MySPList
);
If(
   CountRows(colTemp) = 2000,
   Set(
      vID,
      Max(colTemp,IDRef)  //My Shadow field
   );
   Collect(
      colTemp,
      Filter(
         MySPList,
         IDRef>vID
      )
   )
);
If(
   CountRows(colTemp) = 4000,
   Set(
      vID,
      Max(colTemp,IDRef)
   );
   Collect(
      colTemp,
      Filter(
         MySPList,
         IDRef>vID
      )
   )
)

Then keep going in batches of 2000 - If you do this to both lists, you will have fully delegable In filters.

Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.

[LYNCHJL]

Thanks @WarrenBelz .

Can I replace ID with a new number field (lets say ItemID) that gets assigned by a workflow on creation?  I could set the workflow to set ItemID as ID.  I would then use ItemID in my ClearConnect statement.

Would this work or would i run into the same issue?

thanks!

[eka24]

The number filed should be an incremental field, so you cant use the ItemID Field.

Concurrent(

            ClearCollect(CollectionA, Filter(SOURCE,ID >= 1 && ID <= 2000)),

            ClearCollect(CollectionB, Filter(SOURCE,ID >= 2001 && ID <= 4000)));

ClearCollect(AllIncidentSource, CollectionA, CollectionB)

------------

If you like this post, give a Thumbs up. Where it solved your request, Mark it as a Solution to enable other users find it.

[WarrenBelz · Accepted solution]

Hi @LYNCHJL ,

Strangely that is exactly what I do, but usually when I mention Workflows, I get confused questions . . .

Yes, by far the easiest way to set the number is a SharePoint Designer 2013 Workflow - set on Item Creation with Set Field in Current Item - set ItemID to ID. It works very quickly in the background and required no other attention.
You do however need to create a new numeric field and set it to the ID (easiest way to get a sequential number) I call mine IDRef.

Also do not use Concurrent - you may have deleted records and the Max() in my code picks up the highest number in the 2000 collected previously.

Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.