You’re offline. This is a read only version of the page.
595
Dear all,
I have an external data source that is being updated regularly (every few minutes during a workday), from which I get its data through a REST API.
Currently I am fetching all the data of the current day with a Standard Dataflow, every 5 minutes. This means that I read the same (hundreds of) records each time, plus any additional records that are created during the day. My dataflows will CRUD these records into multiple (standard and custom) entities in my Dataverse.
It seems that this method is causing a lot of overhead on the system, and that these dataflow executions are getting beyond the available capacity and are being slown down after a while...
Should I try to convert these Dataflows into Power Automate flows that are only triggered when data is changed in the source data (if I somehow can detect that)?
I start wondering why one should choose either a Standard Dataflow or a Power Automate Scheduled Cloud Flow? In this case, they both achieve the same, namely reading an external source and copying the data into Dataverse...
Could anyone shine a light on the "best-practices" on choosing between Dataflows and Cloud Flows?
Thanks,
Koen
Hi @cchannon , thanks for the follow up; I have marked the ADF as a solution, however I have not implemented it yet.
Anoter "meta" question for you: how do you keep track of (new, unanswered,...) questions in this community? I would like to help out other people by answering their question, but I don't see a "stackoverflow-like" way to get an overview of questions that might be interesting for me to answer...?
@KoenJanssensPD Did these posts resolve your issue? If so, please mark the solution so future visitors will see this issue as resolved.
No problem. Azure data factory (ADF) is intended for these exact scenarios, migrating data. The only tricky bit is setting up alternate keys. Basically, the unique id field in your datasource, you add that field to your dataverse table (I'm sure you already have this). In dataverse table, you designate this as an alternate key. Then when you setup the data source / data sink, you map fields in source to destination, and there is a dropdown to set the alternate key. That's what allows ADF to determine if it's an insert or update - that alternate key.
Hi @dave-jorgensen , never heard of it , but will take a look for sure, thanks!!!
Did you try using Azure Data Factory instead of Dataflow / flow for syncing your data?
This is a good point, @KoenJanssensPD . This would have been helpful recently with Dataflows in particular; I have seen several posts from apps in EU regions complaining about DataFlows running crazy long, though this is the first I have seen of them failing unexpectedly.
@chass - a "Known/Active Issues" section might prove helpful for things like this, since people often come here to the community first. Something to consider.
Hi @cchannon ,
manual refresh gave the same problem...
After quite some messaging with Microsoft support the last two weeks, they admitted there were some intermittent problems with Power Platform Dataflows.
It still strikes me that we, as a Microsoft Partner, do not have access to status reports on these kind of issues in the Power Platform. This means we have no clue when such problems arise and we have nothing to tell our customers when this happens... Also the lack of good documentation and analytics about the available and used capacity for dataflows is problematic...
Hi @cchannon , that seems a legit solution; I will try that, but I will be struggling, since not all entities that we fetch from this source have audit tracks...
I am still wondering how can get more details on the usage of these flows. The flows I am talking about suddenly stopped working this night, while the flow itself and datasource have not been changed...
The error in the logs only says "Failed: There was a problem refreshing the dataflow. please try again later. (request id: e36b2f5e-703d-4877-8b89-5ef382c6bd1e)"... Is there any place where we can get more details on these (failing) requests?
This is driving us crazy and makes the system unusable for our use case...
Try using Incremental Refresh with Dataflows. It can let you selectively pull content based on when it was created or modified (assuming your source data tracks these)
Using incremental refresh with dataflows | Microsoft Docs
#1
WarrenBelz
109
Most Valuable Professional
#2
Michael E. Gernaey
76
Super User 2025 Season 1
#3
mmbr1606
71
Super User 2025 Season 1
Share
Report
All responses (
Answers (
