PowerApp when having a sharepoint list with over 10000 records

[Ekelund]

I have a powerapp for a sharepoint online app (iGlobe CRM) that works perfectly when I have under approximately 4000 records (still very slow). However many of our customer have over 5000 customers, some over 10.000. This gives a problem in Power Apps. I have set up different filters on the company screen and on the contact screen, fx by selecting only records where the current user is account manager. Again it is working but is slow.

Example of the contact screen

     

My problem however is when I have to use lookup. Fx on the contact detial screen I am making a lookup to get the company address, zip, city and country. This is where I am getting an error because of list with to much data. 

LookUp('CRM Company'; IGlobeCRM_Company_Name=ThisItem.IGlobeCRM_Contact_Company.Value).IGlobeCRM_Company_City

I also tried this 

LookUp(Filter('CRM Company', User().Email=IGlobeCRM_AccountManager.Email), IGlobeCRM_Company_Name=ThisItem.IGlobeCRM_Contact_Company.Value).IGlobeCRM_Company_City

Both gives an error if I am working with a lot of data. 

Question

1. Should I limit the amount of data I am working on when I start the app. Can I do that if so how? and how do I make lookup and use this in the app

2. Can I make a filter of the data and make a look on on that?

3. Or any other way of handling app that are connected to application with much of data? 

It is a bit complex I will happely take a call with some one that can assist me. 

[RusselThomas]

Hi Ekelund,

It does seem a little complex, and doing separate lookups for each row of data in a gallery does tend to be slow as PowerApps has to run a separate query for every single row of data in there, in the context of that row.  If it's running a main query to get the data from source, then multiple queries per row of data returned, from another source - well, you get the picture, otherwise you wouldn't have posted this 🙂

The other thing is, what kind of errors are you getting?  Are they delegation warnings or list view threshold errors?

Without knowing this, there are perhaps a few things I can suggest you try, although I can't say if any of them will solve your current issue;

First - if you haven't already, perhaps make sure all your lists have an index - preferably on a column you typically use to Filter/LookUp.  You can do this by going to the List settings in SharePoint and selecting "Indexed Columns" and then "Create a New Index" and select the column you want to index.  

Second - assuming you might be getting delegation warnings, try to avoid complex objects for functions in your filter/lookup conditions - by this I mean pretty much anything in your conditions that you are referencing as a property of an object or are applying a function to - like User().Email or ThisItem.Something.Value or even functions applied to values. 

This sometimes presents delegation issues as SharePoint can't do anything with the object or function it can't understand (that isn't "delegable"), so powerapps will fetch the first 500 rows regardless of condition, then resolve the condition object/function, then apply the filter/lookup condition on those 500 rows.

Lastly, try use collections where possible to prepare results you might be expecting - Your app looks fantastic, I assume you must be familiar with collections, but if not it might be worth looking into - for example, your filter of accounts based on the user could be done up front and stored in a collection (assuming of course the results are less than 500 rows :)) - it might be slow while creating the initial collection, but subsequent galleries using the collection would be much faster, and you may find you can collect stuff on one screen that you will use on a later screen, making the user experience a little better.

All in all it depends a lot on what the user story is and how you get them to the data they need as fast as possible - while we as creators of the app can prove we have the data by presenting a gallery with thousands of rows, does the user need them or use them, or can we present them with the data they're looking for faster and easier (for us and them)?

That's about all I can think of for now, let me know if anything here doesn't make sense and apologies up front if it doesn't solve your problem, but perhaps you can post the errors and we can try see exactly what's happening?

Hope this helps anyway 🙂

RT

[v-xida-msft]

Hi @Ekelund,

Could you please share more details about the error message (associated with too much data) that you mentioned within your app?

For your first question, your should check if the formula that you provided could be delegabled within your app.

If your formula could be delegabled within your app, you don't need to limit the amount of data your are working, because the processing of data would be delegabled to the SharePoint list data source. In addition, the amount of data would take effect on the execution speed of your formula (executes slowly).

But if your formula could not be delegabled within your app, you must limit the amount of data your are wokring, because all the data must be brought to the device first and then process locally. By default, you could only process 500 records locally within your app. Currently, you could consider take a try to change the Non-delegable limits and change it into the maximum value -- 2000.

More details about the Delegation in PowerApps, please check the following article:

Delegation in PowerApps

More details about changing the Non-delegable limits in PowerApps, please check the followng article:

https://docs.microsoft.com/en-us/powerapps/maker/canvas-apps/delegation-overview#changing-the-limit

For your second question, could you please share more details about the error meesage with the formula that you provided? 

For your third question, please check and see if the following blog would help in your scenario:

https://blog.coeo.com/davidmorrison/2017/06/21/powerapps-500-record-limit-delegation-and-how-to-work...

Best regards,

Kris

[Ekelund]

Thank you both.

This is the error I am getting. The screenshot is in Danish 

 

This is a translation

"The requested operation is invalid. Server response: The attempted operation was not allowed because it exceeded the list view limit set by the administrator. ClientRequestId: e1645f4d-027a-4526-bd69-66aed21ef4b6 ServiceRequestId: 5470779e-10ee-6000-1a4e-35d3e9aa6a6b"

This error I am getting at a customer that have about 10.000 customer record. 

Btw I did change  Non-delegable limits and change it into the maximum value -- 2000.

If you want to take a look I have a demo tenant where the app is running. Just notice that this versiion don't have that much data. So it is working but slow.... I am sure you would have many suggestuions for improvments.  

The app is running on a demo tenant.

jab@iglobedemo.com
pass@word1

The app is connected to this app - https://iglobedemo-7f47668eda3b9d.sharepoint.com/sites/iGlobeCRMApp/IGlobeCRM 

Don't worry this is a demo teant. No live data. 

Btw the idea of filter accounts based on the user (fx by account mananger) could be done up front and stored in a collection (assuming of course the results are less than 2000 rows ) - is great. I need to connect to 4 lists and the can all be filtered by account manager. Question is how 🙂

Thanks

[Ekelund]

Anyone? can someone please provide with input. 

Thanks

[danderson2692]

Just in case anyone else ends up looking here, I believe the solution is to set the columns that you're filtering by as index columns as described in this thread. This seemed to work for me: https://powerusers.microsoft.com/t5/Building-Power-Apps/Help-with-Large-Sharepoint-List/td-p/66007