Limitation on Variable and Microsoft Dataverse Connector in Handling More Than 100000 Records

[otep-kunnn]

Hi!

I am trying to process more than 100000 records in CRM via Power Automate through utilizing the skip token in the 'List rows' Action to bypass the pagination limit of 100000 records and appending to an array variable. Unfortunately, I am reaching the limit on the allowed data within an array variable. I would like to ask your opinion on how can we create a solution to handle this scenario while not affecting the overall runtime performance of the flow, if possible.

Thank you in advance! 🙂

[MarconettiMarco]

Hello @otep-kunnn ,

what's the source? An excel file?

Have you ever considered to use the following XRMToolbox tools?

https://www.xrmtoolbox.com/plugins/JosephMerhej.DataImport/

https://www.xrmtoolbox.com/plugins/Cinteros.XrmToolBox.BulkDataUpdater/

BR,

Marco

[otep-kunnn]

Hello @MarconettiMarco,

Thank you for reaching out. The source of the data is Dynamics 365 CRM/MS Dataverse. Unfortunately, we cannot use the XRM Toolbox tools as we need to process the large number of records everyday on a certain schedule, in an automated way.

Regards,
Otep

[MarconettiMarco]

Hello @otep-kunnn ,

for this kind of volume, I suggest to use Azure services like LogicApps, Data Factory.

BR,

Marco

[otep-kunnn]

Hi @MarconettiMarco,

Thank you for your suggestion. I will definitely check the said services.

I will let you know the outcome.

Regards,

Otep