cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
CharlesS
Kudo Kingpin
Kudo Kingpin

Scale-up compute power for complex data-heavy dataflows

Is there any way to scale up the compute power used to process dataflows? Dataflows can get quite complex and can process pretty large amounts of data. As such, many of my flows can take two or three hours to load into Dataverse.

 

It would be great if we could pay to "scale-up" the compute power, a little bit like you would do with a Virtual Machine in Azure. Yes, yes, I know, Dataverse isn't infrastructure as a service (IaaS). Nevertheless, bottom line, we would be willing to pay to process our dataflows faster.

 

Is there any way to do this?

2 ACCEPTED SOLUTIONS

Accepted Solutions
ChrisPiasecki
Super User
Super User

Hi @CharlesS,

Dataflows are currently best suited for smaller data integration needs and currently have limitations/throttling limits on how much data it can handle.

You may want to look at Azure Data Factory for larger data volumes that you can scale up/down as needed. 

Have a review of the documentation below to see if it fits your need. 

https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/why-cds-import-export

---
Please click Accept as Solution if my post answered your question. This will help others find solutions to similar questions. If you like my post and/or find it helpful, please consider giving it a Thumbs Up.

 

View solution in original post

Hi @CharlesS ,

 

To add to what @ChrisPiasecki mentioned, you also need to consider the Dataverse API limits.  Azure Data Factory may be more scalable in terms of processing data but at the end of the day you are still putting the data into Dataverse and it does not matter what method you use, you will be throttled by the Dataverse API limits.

 

There isn't an option to scale up the processing speed but if you have a genuine interest in this and would be willing to pay for the scale then it might be worth talking to Microsoft directly about this.

View solution in original post

4 REPLIES 4
ChrisPiasecki
Super User
Super User

Hi @CharlesS,

Dataflows are currently best suited for smaller data integration needs and currently have limitations/throttling limits on how much data it can handle.

You may want to look at Azure Data Factory for larger data volumes that you can scale up/down as needed. 

Have a review of the documentation below to see if it fits your need. 

https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/why-cds-import-export

---
Please click Accept as Solution if my post answered your question. This will help others find solutions to similar questions. If you like my post and/or find it helpful, please consider giving it a Thumbs Up.

 

Hi @CharlesS ,

 

To add to what @ChrisPiasecki mentioned, you also need to consider the Dataverse API limits.  Azure Data Factory may be more scalable in terms of processing data but at the end of the day you are still putting the data into Dataverse and it does not matter what method you use, you will be throttled by the Dataverse API limits.

 

There isn't an option to scale up the processing speed but if you have a genuine interest in this and would be willing to pay for the scale then it might be worth talking to Microsoft directly about this.

@HSheild @ChrisPiasecki 

 

Thanks for your replies! I had indeed considered using Azure DataFactory, but there are problems with this:

1. Many users have access to PowerPlatform, but not Azure.

2. Data Factory requires a different skill set.

 

My work often puts me on the blurry line between Azure and the PowerPlatform. Ultimately, my greatest wish would be for PowerPlatform to extend its functionalities further, to allow for the treatment of more data and more complex data (with premium associated fees to be paid to MSFT and greater API limits), and for Azure to "democratize" further to a "citizen developer" level. You know what? I think it will eventually happen because the business opportunity here for Microsoft is huge.

That will inevitably happen and it has been happening for a number of years now. As the Azure platform expands its capabilities those capabilities eventually get democratized via the Power Platform. That said, there will always be a blurred line at the edge and we'll have to navigate it, by helping citizen developers understand where that limitation sits (at the current time) and the options for getting past it by utilizing more pro-dev based solutions.

 

And then certain things may not get democratized because the usage telemetry tells Microsoft there is not enough demand to make it a successful offering. This is why I've come to very much appreciate the forward guidance Microsoft gives us twice a year on Power Platform.

Helpful resources

Announcements
Ignite 2022

WHAT’S NEXT AT MICROSOFT IGNITE 2022

Explore the latest innovations, learn from product experts and partners, level up your skillset, and create connections from around the world.

Power Apps Africa Challenge 2022

Power Apps Africa Challenge

Your chance to join an engaging competition of Power Platform enthusiasts.

Super User 2 - 2022 Congratulations

Welcome Super Users

The Super User program for 2022 - Season 2 has kicked off!

September Events 2022

Check out all of these events

Attend in person or online, there are incredible conferences and events happening all throughout the month of September.

Users online (2,969)