Standard (Dataverse) Dataflow vs Power Automate

(0) Share

Report

Posted on by KoJaPD

601

Dear all,

I have an external data source that is being updated regularly (every few minutes during a workday), from which I get its data through a REST API.

Currently I am fetching all the data of the current day with a Standard Dataflow, every 5 minutes. This means that I read the same (hundreds of) records each time, plus any additional records that are created during the day. My dataflows will CRUD these records into multiple (standard and custom) entities in my Dataverse.

It seems that this method is causing a lot of overhead on the system, and that these dataflow executions are getting beyond the available capacity and are being slown down after a while...

Should I try to convert these Dataflows into Power Automate flows that are only triggered when data is changed in the source data (if I somehow can detect that)?

I start wondering why one should choose either a Standard Dataflow or a Power Automate Scheduled Cloud Flow? In this case, they both achieve the same, namely reading an external source and copying the data into Dataverse...

Could anyone shine a light on the "best-practices" on choosing between Dataflows and Cloud Flows?

Thanks,

Koen

Categories:

Power Apps Pro Dev & ISV

I have the same question (0)

All responses (10)

Answers (1)

Sort by

cchannon 4,702 Moderator on at

Like (0)

Report
Copy link

Link copied!

Try using Incremental Refresh with Dataflows. It can let you selectively pull content based on when it was created or modified (assuming your source data tracks these)

Using incremental refresh with dataflows | Microsoft Docs

Was this reply helpful? Yes No
KoJaPD 601 on at

Like (0)

Report
Copy link

Link copied!

Hi @cchannon , that seems a legit solution; I will try that, but I will be struggling, since not all entities that we fetch from this source have audit tracks...
I am still wondering how can get more details on the usage of these flows. The flows I am talking about suddenly stopped working this night, while the flow itself and datasource have not been changed...
The error in the logs only says "Failed: There was a problem refreshing the dataflow. please try again later. (request id: e36b2f5e-703d-4877-8b89-5ef382c6bd1e)"... Is there any place where we can get more details on these (failing) requests?
This is driving us crazy and makes the system unusable for our use case...

Was this reply helpful? Yes No
cchannon 4,702 Moderator on at

Like (0)

Report
Copy link

Link copied!

hmmm... That is frustrating.

Did you try executing them manually to see if you might get a clearer error message?

Was this reply helpful? Yes No
KoJaPD 601 on at

Like (1)

Report
Copy link

Link copied!

Hi @cchannon ,
manual refresh gave the same problem...
After quite some messaging with Microsoft support the last two weeks, they admitted there were some intermittent problems with Power Platform Dataflows.
It still strikes me that we, as a Microsoft Partner, do not have access to status reports on these kind of issues in the Power Platform. This means we have no clue when such problems arise and we have nothing to tell our customers when this happens... Also the lack of good documentation and analytics about the available and used capacity for dataflows is problematic...

Was this reply helpful? Yes No
cchannon 4,702 Moderator on at

Like (0)

Report
Copy link

Link copied!

This is a good point, @KoenJanssensPD . This would have been helpful recently with Dataflows in particular; I have seen several posts from apps in EU regions complaining about DataFlows running crazy long, though this is the first I have seen of them failing unexpectedly.

@chass - a "Known/Active Issues" section might prove helpful for things like this, since people often come here to the community first. Something to consider.

Was this reply helpful? Yes No
Verified answer

dave-jorgensen 502 on at

Like (0)

Report
Copy link

Link copied!

Did you try using Azure Data Factory instead of Dataflow / flow for syncing your data?

Was this reply helpful? Yes No
KoJaPD 601 on at

Like (0)

Report
Copy link

Link copied!

Hi @dave-jorgensen , never heard of it , but will take a look for sure, thanks!!!

Was this reply helpful? Yes No
dave-jorgensen 502 on at

Like (1)

Report
Copy link

Link copied!

No problem. Azure data factory (ADF) is intended for these exact scenarios, migrating data. The only tricky bit is setting up alternate keys. Basically, the unique id field in your datasource, you add that field to your dataverse table (I'm sure you already have this). In dataverse table, you designate this as an alternate key. Then when you setup the data source / data sink, you map fields in source to destination, and there is a dropdown to set the alternate key. That's what allows ADF to determine if it's an insert or update - that alternate key.

Was this reply helpful? Yes No
cchannon 4,702 Moderator on at

Like (0)

Report
Copy link

Link copied!

@KoenJanssensPD Did these posts resolve your issue? If so, please mark the solution so future visitors will see this issue as resolved.

Was this reply helpful? Yes No
KoJaPD 601 on at

Like (0)

Report
Copy link

Link copied!

Hi @cchannon , thanks for the follow up; I have marked the ADF as a solution, however I have not implemented it yet.
Anoter "meta" question for you: how do you keep track of (new, unanswered,...) questions in this community? I would like to help out other people by answering their question, but I don't see a "stackoverflow-like" way to get an overview of questions that might be interesting for me to answer...?

Was this reply helpful? Yes No