Separating big collections

[Fernanda]

Hello!

I am working with an offline app that has a photo collection that later on gets uploaded to a SP document folder through Power Automate. This is working really good, until I reach a certain limit of photos (app runs slower, and it crashes when I want to upload it).

I need to collect at least 300 photos, but my collection gets too heavy to deal with, so my idea was to split it or create new collections when a certain limit (let's say 50 photos) is reached.

I have been trying to get this going, but I am failing hugely.

Could you please help me??

Thanks a lot!!

[iAm_ManCat]

Hi,

Unless your photos are absolutely tiny, you will not be able to use 300 photos with the offline functionality.

This is generally limited by the amount of memory the phone provides for the running App, but average limits are around 30-70Mb, so 300 photos would likely be more data than the app can use at any one time for offline use (regardless of if you split it into multiple, they still need to be stored in the app session)

Details of limitations are listed here:

https://learn.microsoft.com/en-us/power-apps/maker/canvas-apps/offline-apps

[Fernanda]

Today I was able to work 150 photos 278kb average each with no problem. So what I need is a collection reaches 50 photos, a new collection is created and so on.

This is the formula I am working on

Set(varName;
varPlanta & "-" & VarQuimico & "-"& RandBetween(1;500)
)
;;
Collect(colFotos;
{
    DisplayName: varName & ".jpg";
    Id: varName & ".jpg";
    Value: Bt_TomarFoto.Media;
    Timestamp: Now()
} );;
Set(varApress;JSON(Im_FotoTomada.Image;JSONFormat.IncludeBinaryData));;
Set(varBase64only;Mid(varApress;Find(",";varApress)+1;Len(varApress)-Find(",";varApress)-1));;
If(CountRows(colCompleta)<50;Collect(colCompleta;
{  
    Nombre: varName & ".jpg";
    Base: varBase64only
});If(CountRows(colCompleta)>50 && CountRows(colCompleta2)<50;Collect(colCompleta2;{  
    Nombre: varName & ".jpg";
    Base: varBase64only
});If(CountRows(colCompleta2)>50 && CountRows(colCompleta3)<50;Collect(colCompleta3;{  
    Nombre: varName & ".jpg";
    Base: varBase64only
});If(CountRows(colCompleta3)>50 && CountRows(colCompleta4)<50;Collect(colCompleta4;{  
    Nombre: varName & ".jpg";
    Base: varBase64only
});If(CountRows(colCompleta4)>50 && CountRows(colCompleta5)<50;Collect(colCompleta5;{  
    Nombre: varName & ".jpg";
    Base: varBase64only
});If(CountRows(colCompleta5)>50 && CountRows(colCompleta6)<50;Collect(colCompleta6;{  
    Nombre: varName & ".jpg";
    Base: varBase64only});
    If(CountRows(colCompleta6)>50 && CountRows(colCompleta7)<50;Collect(colCompleta7;{  
    Nombre: varName & ".jpg";
    Base: varBase64only}))))))))


;;
SaveData(colFotos;"ColeccionFotos");;
SaveData(colCompleta;"ColeccionCompleta");;SaveData(colCompleta2;"ColeccionCompleta2");;SaveData(colCompleta3;"ColeccionCompleta3");;SaveData(colCompleta4;"ColeccionCompleta4");;SaveData(colCompleta5;"ColeccionCompleta5");;SaveData(colCompleta6;"ColeccionCompleta6");;SaveData(colCompleta7;"ColeccionCompleta7")

But not quite there yet, I am sure there's gotta be a better way to do it, but I can't find it

[iAm_ManCat]

Hi,

Just to be clear, I'm not saying that the limit is on the collection, the limit is for the whole App's offline storage.

Yes it is possible to split the collection into even hundreds of collections, but the total space they use, and the total space used by the App will be the limit that will have the App crash.

You said you have 150 photos at 278Kb average, that is 41Mb. As is shown on the link I provided, the amount of memory granted to the App is dependent on the phone make and many other factors but averages between 30-70Mb, whereas in your case 300 photos at 278Kb average would be 82Mb.

This is the limit I am referring to.

Even if you split the collections, once you reach the size limit given to the App by the mobile device, the app will run into issues.

[Fernanda]

Hi

Oh, ok, yes, of course there will be a limit, but so far, what we have tested, we are ok to handle 300 photos. (This app is being designed to work with Samsung Tablet S8). If at some point we encounter data limit, we plan to set a count limit to upload, clean and then continue.

So, back to my initial question... is it possible to separate big collections?? so whenever the collection reaches 50 items, a new collection is created for another 50 items and so on??

[iAm_ManCat]

In your case, I would start by using the Monitor function to see which part is struggling before the app crashes

could you provide some more information around the slowness - is it slow to save to phone memory? slow to load from phone memory?  is it slow when you try to upload? how are you uploading? If you are passing it to a flow (as you've said) then your app will be sending to flow an 40-80Mb file based on what you've said, would this be an accurate assessment?

I would look at dealing with the collection in batches.

Use addColumns to create an index ( Matt Devaney has a good article on that here ), then you can create collections that are subsets of the original by using:

ClearCollect(filteredCol1, Filter(yourCollection, RowNumber > 0 & RowNumber <=50));
ClearCollect(filteredCol2, Filter(yourCollection, RowNumber > 50 & RowNumber <=100));
ClearCollect(filteredCol3, Filter(yourCollection, RowNumber > 100 & RowNumber <=150));
ClearCollect(filteredCol4, Filter(yourCollection, RowNumber > 150 & RowNumber <=200));
ClearCollect(filteredCol5, Filter(yourCollection, RowNumber > 200 & RowNumber <=250));
ClearCollect(filteredCol6, Filter(yourCollection, RowNumber > 250 & RowNumber <=300));
ClearCollect(filteredCol7, Filter(yourCollection, RowNumber > 300 & RowNumber <=350));

[Fernanda]

Hey there!

Thank you for taking the time with me!!

Let me elaborate a little bit more about when it starts getting slow and finally crashing.

First of all, my app is an offline app that will work as a chemicals inventory app. Since our warehouses and plants are not always within cellphone range, it must work offline. So, what I did was:

For every plant and chemical, evidence photos must be uploaded (this is why we have an average of 250-300 photos)

When we add the photo, the following formula is ran. Inicially, in this very same button, I ran the JSON formula and created data to send to power automate, but it made it really slow to work with after 30 photos. So I trimmed it and up to here, I am only saving the images and info I need

Set(varName;
varPlanta & "-" & VarQuimico & "-"& varFotoID+Last(colFotos).ID)
;;
Collect(colFotos;
{
    DisplayName: varName & ".jpg";
    ID: varFotoID+Last(colFotos).ID;
    Value: AddMediaButton1.Media;
    Timestamp: Now();
    Apress:UploadedImage1.Image;
    Observación: Toggle1.Value;
    'Con Observación': TextInput2.Text
} );;SaveData(colFotos;"ColeccionFotos");;Notify("Foto agregada";NotificationType.Information);;Reset(AddMediaButton1);;Reset(Toggle1);;UpdateContext({varComentarioFoto:false})

Once inventory is over and data is ready to be uploaded to our sharepoint lists, what I do is creating the data to send to power automate by clicking a button with the following formula.

ForAll(colFotos;Collect(colCompleta;{ Nombre: ThisRecord.DisplayName;
            Base: Mid(JSON(ThisRecord.Apress;JSONFormat.IncludeBinaryData);Find(",";JSON(ThisRecord.Apress;JSONFormat.IncludeBinaryData))+1;Len(JSON(ThisRecord.Apress;JSONFormat.IncludeBinaryData))-Find(",";JSON(ThisRecord.Apress;JSONFormat.IncludeBinaryData))-1)
        }))

And here is where it gets slow. Binary data created in order to process the photos crashes the app after 150 photos ("Base" column info in the collection created above) Size is an average of 250-300 kb each. 80Mb is quite right 

So, what I tried was separating the collection into smaller ones, so it won't crash when sending it to Power Automate. I did that manually and it work. Takes a bit, but it doesn't crash. But I was having serious problems trying to make it work automatically. Meaning, reducing the amount of buttons, but I was not succeding.

This is my flow

Thank you for the formula... now that I see it, I feel kinda stupid for not visualizing it before haha.. will give it a try adding my additional info and i will let you know how it goes!