CDS/Dataverse Bulk load- error handling

(0) Share

Report

Posted on by amarendher_v

Hi all,

Need help for the following

1) Bulk load with sql DB(sql query) loading into CDS with (nearly 1 million)

2) Error handling for CDS with dataflows dataloads.

3) how/where can I find the failed records (Not the log from refresh history).

Thanks.

Categories:

Microsoft Dataverse with Power Apps

I have the same question (0)

All responses (5)

Answers (0)

EricRegnier 8,720 Most Valuable Professional on at

Like (1)

Report
Hi @amarendher_v,

Here are quick answers to your questions:

Although Dataflows is a great low code/no code ETL tool, for 1M records it might not be the best. Suggest to look into Azure Data Factory or SSIS (with optionally Kingswaysoft).

Look into the Dataflow Power Automate connector which can send notifications: https://powerapps.microsoft.com/en-us/blog/announcing-dataflows-power-automate-connector-public-preview/

From what I'm aware, you can't unfortunately. You only have the error message from clicking the warning icon next to the Refresh Date. To see the failed records, you'll need determine possibly by the successful records, and/or modified/created on...

Hope this helps...

Was this reply helpful? Yes No
Marcel Lathouwers 237 on at

Like (0)

Report

You could try simego. It is more a sync tool to compare to datasets. But is a record exists in source, but not in destination, it will create the record.
SharePoint and Dynamics 365 Data Integration Solutions (simego.com)

Inserting 1 mln records will definitaly increase the demand on storage in dataverse, not very cheap!
Maybe if your customer hears the cost, the number of 'must have' record might vanish 😁

Was this reply helpful? Yes No
amarendher_v 5 on at

Like (0)

Report

@EricRegnier thanks for the info. With Data Factory/SSIS options how could be the data intake speed into CDS/Dataverse. Currently with dataflows the data loads are very slow-(100k/2 hrs).

Was this reply helpful? Yes No
amarendher_v 5 on at

Like (0)

Report

@Mglat Thanks for the info.

Was this reply helpful? Yes No
EricRegnier 8,720 Most Valuable Professional on at

Like (0)

Report

Much faster than Dataflows. Its' hard to give an exact throughput, as it depends on the latency, system traffic, level of customizations (e.g plugins), and how optimized your ETL is...

Was this reply helpful? Yes No