cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
CLS720
Advocate I
Advocate I

Collections maxes out at 2000 when populating with SharePoint list datasource

I have an app where on the OnView property for a screen, a collection is created from a SharePoint list datasource.

ClearCollect(CompletedRequestsCollection, CompletedRequests)

I then try to work around the delegation issue  within a gallery by sorting/filtering the collection that is created from OnView. However, even though there is no filtering on this datasource and there is no indication of non-delegation, the collection that is created OnView always maxes out with the first 2000 items in the list. Is this expected? Are there any workarounds?

1 ACCEPTED SOLUTION

Accepted Solutions
Drrickryp
Super User
Super User

HI @CLS720,

Collect() and ClearCollect() are not delegatable so they max out at 2000k items.  However, once a collection is created, all powerapps functions can be delegated to the collection including Search() and all numeric functions.   @martinav and I (well mostly him) came up with a way to rapidly collect a maximum of 4000 items from a Sharepoint list.  As long as the list doesn't exceed 4000 items it will work and it works by collecting the first 2k items then sorting the list descending and collecting the bottom 2k items.  Then merging the lists and then removing duplicates.  It works this way.: In the OnVisible property of the screen:

Concurrent(
ClearCollect(List1,Filter(SortByColumns(List,"ID",Ascending),ID>0)),
ClearCollect(LIst2,Filter(SortByColumns(List,"ID",Descending),ID>0))
);
 

So two collections (List1 and List2) are created concurrently.  This greatly increases the speed of the process.

Then the two two collections are merged, elimininating duplicates, into ListAll and the two collections, List1 and List2 are emptied.  Just add this formula to follow the semicolon in the upper formula box. 

ClearCollect(ListAll,List1,Filter(List2, Not(ID in List1.ID)));Clear(List1);Clear(List2)

 ListAll now contains up to 4000 unique rows and can be used with all Powerapps functions without regard to delegation,  and as an items property for galleries,  datatables, and Comboboxes (but not Dropdowns as these are hardcode limited to a maximum number of items, no one wants to scroll through thousands of items in a dropdown anyway.).    The execution speed of the formulas like Search() or Lookup() will astonish you. 

View solution in original post

15 REPLIES 15
Drrickryp
Super User
Super User

HI @CLS720,

Collect() and ClearCollect() are not delegatable so they max out at 2000k items.  However, once a collection is created, all powerapps functions can be delegated to the collection including Search() and all numeric functions.   @martinav and I (well mostly him) came up with a way to rapidly collect a maximum of 4000 items from a Sharepoint list.  As long as the list doesn't exceed 4000 items it will work and it works by collecting the first 2k items then sorting the list descending and collecting the bottom 2k items.  Then merging the lists and then removing duplicates.  It works this way.: In the OnVisible property of the screen:

Concurrent(
ClearCollect(List1,Filter(SortByColumns(List,"ID",Ascending),ID>0)),
ClearCollect(LIst2,Filter(SortByColumns(List,"ID",Descending),ID>0))
);
 

So two collections (List1 and List2) are created concurrently.  This greatly increases the speed of the process.

Then the two two collections are merged, elimininating duplicates, into ListAll and the two collections, List1 and List2 are emptied.  Just add this formula to follow the semicolon in the upper formula box. 

ClearCollect(ListAll,List1,Filter(List2, Not(ID in List1.ID)));Clear(List1);Clear(List2)

 ListAll now contains up to 4000 unique rows and can be used with all Powerapps functions without regard to delegation,  and as an items property for galleries,  datatables, and Comboboxes (but not Dropdowns as these are hardcode limited to a maximum number of items, no one wants to scroll through thousands of items in a dropdown anyway.).    The execution speed of the formulas like Search() or Lookup() will astonish you. 

Hey  @Drrickryp,  THanks for the tag!  It was a combined effort, but I thought you came up with the first/last scenario... anyhwo, it is working very well.  I have a list with 2,400 items, and it does indeed work well.  Also, by using the ShowColumns() command, you can pull in only the columns you wish to have in the collection.  If you dont need it, no sense in wasting time and memory space with data you dont need.  

 

Here are my actual commands, which I put in the "OnStart" control for the app.  

Concurrent(
ClearCollect(MDL_List1,Filter(ShowColumns(SortByColumns(Master_Drawing_List,"ID",Ascending),"ID","MD_PartNumber","MD_Project","MD_LatestRevNumber","MD_DrawingRevDate","MD_EngrgRelDate","Title","MD_PDM_Link","MD_LinkToDrawing","MD_IsActive","MD_Data_Flagged"),ID>0)),
ClearCollect(MDL_List2,Filter(ShowColumns(SortByColumns(Master_Drawing_List,"ID",Descending),"ID","MD_PartNumber","MD_Project","MD_LatestRevNumber","MD_DrawingRevDate","MD_EngrgRelDate","Title","MD_PDM_Link","MD_LinkToDrawing","MD_IsActive","MD_Data_Flagged"),ID>0)) ); ClearCollect(MDL_All,MDL_List1,Filter(MDL_List2, Not(ID in MDL_List1.ID)));Clear(MDL_List1);Clear(MDL_List2)

So, everything is pulled one time.  I do have a data refresh button that executes the exact same commands if the user thinks his data is stale.  I also have a timer that runs, and reloads the data every 30min.  If you dont do that, since data in your collection does not update automatically, you might want to recollect the data from sharepoint regularly.  It will depend on your specific use.  Also, note that I cleared the two collections used to pull in the two separate lists.  To minimize memory usage.

 

This really did work well.  Much better than I could have hoped!  I did have another idea..  To perhaps get more items.  I was wondering if can collect 2,000 items at a time using the ID of the sharepoint list.  (if(ID>2000 AND ID<4000) etc etc. It might take using a Flow, though, because I think that using gt or lt functions are not delegable.  However, flow allows you to to get items, and use ID.  I think you can do get items using the get items filter.  Then sending back to powerapps using the Response action.  I have no idea how long it would take.   

 

Thank you both! I applied this solution and it worked beautifully.

martinav
Community Champion
Community Champion

Sweet!  Another happy customer!

Anonymous
Not applicable

Hey Guys, can you help me please?

 

I have a excel spreadsheet wich cointains 37k rows and i want to bring it to powerapps.

 

the data does not change, so i uploaded the excel via import via excel, and when i collect it brings me 15k rows.

 

but how can i separate in 15k15k and another one with 8k to bring all the datas?

Hold that thought...  while you are waiting... what commands/objects are you using?

Anonymous
Not applicable

actually @timl figured out a way to do this with.

 

spread the excel table in 3 parts then collect all of them in powerapps.

 

worked 😉

 

thanks!  

Seems to work @Drrickryp and @martinav , but more fun!

 

I have a SharePoint list with thousands of items, but I only want to pull items where the Start Date is greater than today. New items will come in over time. Is there a way to do this? I tried adding the Filter in, but with no change.

 

Concurrent(
ClearCollect(List1,Filter(SortByColumns('Events','Start Time'>Now(),"ID",Ascending),ID>0)),
ClearCollect(List2,Filter(SortByColumns('Events','Start Time'>Now(),"ID",Descending),ID>0))
);
ClearCollect(ListAll,List1,Filter(List2, Not(ID in List1.ID)));Clear(List1);Clear(List2)

 

I have a feeling I'm going to have to somehow create a Flow to keep a 2nd copy of the SharePoint List that is cleaned up. Any advice would be appreciated - thanks!

@ubikwitous ,

 

Filter first, then sort.  The parenthesis work from the inside out.  

 

Concurrent(
ClearCollect(List1,SortByColumns(Filter('Events','Start Time'>Now()),"ID",Ascending),ID>0)),
ClearCollect(List2,SortByColumns(Filter('Events','Start Time'>Now()),"ID",Descending),ID>0))
);
ClearCollect(ListAll,List1,Filter(List2, Not(ID in List1.ID)));Clear(List1);Clear(List2)

I didnt actually try this, I think the syntax is all ok.  But, this may not work because the ">" will be a delegation issue itself. This may not work.

 

Getting 2,000 rows first, and then filtering would look like this... (I think)

 

Concurrent(
ClearCollect(List1,Filter(SortByColumns('Events',"ID",Ascending),'Start Time'>Now()),ID>0)),
ClearCollect(List2,Filter(SortByColumns('Events',"ID",Descending),'Start Time'>Now()),ID>0))
);
ClearCollect(ListAll,List1,Filter(List2, Not(ID in List1.ID)));Clear(List1);Clear(List2)

I'm not sure about your syntax for 'Start Time'.  You also will need to be sure you use time zone correction to get your time adjusted to UTC().

Helpful resources

Announcements

Super User of the Month | Drew Poggemann

As part of a new monthly feature in the Community, we are excited to share that Drew Poggemann is our featured Super User for the month of February 2024. If you've been in the Community for a while, we're sure Drew's name is familiar to you, as he is one of our most active contributors--he's been a Super User for five consecutive seasons!   Since authoring his first reply 5 years ago to his 514th solution authored, Drew has helped countless Community members with his insights and expertise. In addition to being a Super User, Drew is also a User Group leader and a Microsoft MVP. His contributions to our Super User sessions and to the new SUIT program are always welcome--as well as his sense of humor and fun-loving way of sharing what he knows with others.   When Drew is not solving problems and authoring solutions, he's busy overseeing the Solution Architecture team at HBS, specializing in application architecture and business solution strategy--something he's been doing for over 30 years. We are grateful for Drew and the amazing way he has used his talent and skills to help so many others in the Community. If you are part of the SUIT program, you got to hear some great tips from Drew at the first SUIT session--and we know he still has much more to share!You can find him in the Community and on LinkedIn. Thank you for all you do, Drew!

Announcing Power Apps Copilot Cookbook Gallery

We are excited to share that the all-new Copilot Cookbook Gallery for Power Apps is now available in the Power Apps Community, full of tips and tricks on how to best use Microsoft Copilot as you develop and create in Power Apps. The new Copilot Cookbook is your go-to resource when you need inspiration--or when you're stuck--and aren't sure how to best partner with Copilot while creating apps.   Whether you're looking for the best prompts or just want to know about responsible AI use, visit Copilot Cookbook for regular updates you can rely on--while also serving up some of your greatest tips and tricks for the Community. Our team will be reviewing posts using the new "Copilot Studio" label to ensure we highlight and amplify the most relevant and recent content, so you're assured of high-quality content every time you visit. If you share a post that gets featured in the curated gallery, you'll get a PM in the Community to let you know!The curated gallery is ready for you to experience now, so visit the new Copilot Cookbook for Power Apps today: Copilot Cookbook - Power Platform Community. We can't wait to see what you "cook" up!      

Celebrating a New Season of Super Users with Charles Lamanna, CVP Microsoft Business Applications

February 8 was the kickoff to the 2024 Season One Super User program for Power Platform Communities, and we are thrilled to welcome back so many returning Super Users--as well as so many brand new Super Users who started their journey last fall. Our Community Super Users are the true heroes, answering questions, providing solutions, filtering spam, and so much more. The impact they make on the Communities each day is significant, and we wanted to do something special to welcome them at our first kickoff meeting of the year.   Charles Lamanna, Microsoft CVP of Business Applications, has stressed frequently how valuable our Community is to the growth and potential of Power Platform, and we are honored to share this message from him to our 2024 Season One Super Users--as well as anyone who might be interested in joining this elite group of Community members.     If you want to know more about Super Users, check out these posts for more information today:    Power Apps: What is A Super User? - Power Platform CommunityPower Automate: What is A Super User? - Power Platform Community Copilot Studio: What is A Super User? - Power Platform Community Power Pages: What is A Super User? - Power Platform Community

February 2024 User Group Update: Welcoming New Groups and Highlighting Upcoming Events

It's a new month and a brand-new year, which means another opportunity to celebrate our amazing User Groups!Each month, we highlight the new User Groups that have joined the community. It's been a busy season for new groups, because we are thrilled to welcome 15 New User Groups! Take a look at the list below, shared by the different community categories. If your group is listed here, give this post a kudo so we can celebrate with you!   We love our User Groups and the difference they make in the lives of our Community! Thank you to all the new User Groups, new User Group leaders--we look forward to hearing about your successes and the impact you will leave!   In addition to our monthly New User Group spotlight, it's a great time to share some of the latest events happening in our User Group community! Take a look at the list below to find one that fits your schedule and need! There's a great combination of in-person and virtual events to choose from. It's a great time of year to connect and engage with User Groups both locally and online. Please Welcome Our NEW User Groups   Power Platform: Heathcare Power Platform User Group Power Platform Connect Hub Power Platform Usergroup Denmark Mexico Norte- Power Platform User Group Pune Power User Group Sudbury Power Platform User GroupMicrosoft User Group GhanaMPPBLR - Microsoft Power Platform Bengaluru User Group Power Apps:   Myrtle Beach Power Platform User GroupAnanseTechWB PowerApps Copilot Studio: Pathfinders Power Platform Community Dynamics365: Cairo, Egypt MSD 365 Business Central/NAV/F&O User GruopMS Dynamics 365 Business Central LatamCincinnati OH D365 F&O User Group February User Group Events February 2024 Cleveland Power Platform User GroupPortallunsj - Februar 2024Indiana D365/AX February User Group MeetingQ1 2024 KC Power Platform and Dynamics 365 CRM Users Group 

Super Users 2024 Season One is Here!

   We are excited to announce the first season of our 2024 Super Users is here! Our kickoff to the new year welcomes many returning Super Users and several new faces, and it's always exciting to see the impact these incredible individuals will have on the Community in 2024! We are so grateful for the daily difference they make in the Community already and know they will keep staying engaged and excited for all that will happen this year.   How to Spot a Super User in the Community:Have you ever written a post or asked for help in the Community and had it answered by a user with the Super User icon next to their name? It means you have found the actual, real-life superheroes of the Power Platform Community! Super Users are our heroes because of the way they consistently make a difference in the Community. Our amazing Super Users help keep the Community a safe place by flagging spam and letting the Community Managers know about issues. They also make the Community a great place to find answers, because they are often the first to offer solutions and get clarity on questions. Finally, Super Users share valuable insights on ways to keep the Community growing, engaging, and looking ahead!We are honored to reveal the new badges for this season of Super Users! Congratulations to all the new and returning Super Users!     To better answer the question "What is a Super User?" please check out this article: Power Apps: What is A Super User? - Power Platform CommunityPower Automate: What is A Super User? - Power Platform Community Copilot Studio: What is A Super User? - Power Platform Community Power Pages: What is A Super User? - Power Platform Community

Did You Attend the Microsoft Power Platform Conference in 2022 or 2023? Claim Your Badge Today!

If you were one of the thousands of people who joined us at the first #MPPC Microsoft Power Platform Conference in 2022 in Orlando--or attended the second-annual conference in Las Vegas in 2023--we are excited to honor you with a special community badge! Show your support for #MPPC Microsoft Power Platform Conference this year by claiming your badge!         Just follow this link to claim your badge for attending #MPPC in 2022 and/or 2023: MPPCBadgeRequest    Want to earn your badge for 2024? Just keep watching our News & Announcements for the latest updates on #MPPC24.

Top Solution Authors
Top Kudoed Authors
Users online (3,030)