Unanswered

Scale-up compute power for complex data-heavy dataflows

(0) Share

Report

Posted on by Charles Seguin

401

Is there any way to scale up the compute power used to process dataflows? Dataflows can get quite complex and can process pretty large amounts of data. As such, many of my flows can take two or three hours to load into Dataverse.

It would be great if we could pay to "scale-up" the compute power, a little bit like you would do with a Virtual Machine in Azure. Yes, yes, I know, Dataverse isn't infrastructure as a service (IaaS). Nevertheless, bottom line, we would be willing to pay to process our dataflows faster.

Is there any way to do this?

Categories:

Microsoft Dataverse with Power Apps

I have the same question (0)

All responses (4)

Answers (2)

Verified answer

ChrisPiasecki 6,422 Most Valuable Professional on at

Like (1)

Report

Hi @CharlesS,
Dataflows are currently best suited for smaller data integration needs and currently have limitations/throttling limits on how much data it can handle.
You may want to look at Azure Data Factory for larger data volumes that you can scale up/down as needed.
Have a review of the documentation below to see if it fits your need.
https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/why-cds-import-export
---
Please click Accept as Solution if my post answered your question. This will help others find solutions to similar questions. If you like my post and/or find it helpful, please consider giving it a Thumbs Up.

Was this reply helpful? Yes No
Verified answer

Hamish Sheild 3,005 Most Valuable Professional on at

Like (2)

Report

Hi @CharlesS ,

To add to what @ChrisPiasecki mentioned, you also need to consider the Dataverse API limits. Azure Data Factory may be more scalable in terms of processing data but at the end of the day you are still putting the data into Dataverse and it does not matter what method you use, you will be throttled by the Dataverse API limits.

There isn't an option to scale up the processing speed but if you have a genuine interest in this and would be willing to pay for the scale then it might be worth talking to Microsoft directly about this.

Was this reply helpful? Yes No
Charles Seguin 401 on at

Like (3)

Report

@HSheild @ChrisPiasecki

Thanks for your replies! I had indeed considered using Azure DataFactory, but there are problems with this:
1. Many users have access to PowerPlatform, but not Azure.
2. Data Factory requires a different skill set.

My work often puts me on the blurry line between Azure and the PowerPlatform. Ultimately, my greatest wish would be for PowerPlatform to extend its functionalities further, to allow for the treatment of more data and more complex data (with premium associated fees to be paid to MSFT and greater API limits), and for Azure to "democratize" further to a "citizen developer" level. You know what? I think it will eventually happen because the business opportunity here for Microsoft is huge.

Was this reply helpful? Yes No
nicknow 18 on at

Like (1)

Report

That will inevitably happen and it has been happening for a number of years now. As the Azure platform expands its capabilities those capabilities eventually get democratized via the Power Platform. That said, there will always be a blurred line at the edge and we'll have to navigate it, by helping citizen developers understand where that limitation sits (at the current time) and the options for getting past it by utilizing more pro-dev based solutions.

And then certain things may not get democratized because the usage telemetry tells Microsoft there is not enough demand to make it a successful offering. This is why I've come to very much appreciate the forward guidance Microsoft gives us twice a year on Power Platform.

Was this reply helpful? Yes No