Unanswered

Importing data to DataVerse using ADF

(1) Share

Report

Posted on by Agila

Hi All!

We need to create an ETL solution to load Data verse tables. Source is Excel files (max 400Mb) and Data from DataVerse which will be transformed into 1 to 4 GB of data which needs to be loaded to Data Verse. Now we are facing challenges in below areas:

1. Data input from Data verse tables using filters to fetch required data. Currently we are using Xrm tool to create xml query to fetch data with filters on attributes. This works if table size is low but not for huge data

2. Copy activity to load Data Verse takes very long time., 1GB data takes 13 hrs to load data to Data Verse table. Need to increase the performance.

Any idea for above 2 challenges will help us.

Categories:

Microsoft Dataverse with Power Apps

I have the same question (0)

All responses (2)

Answers (0)

Sunil Kumar Pashikanti 1,316 Moderator on at

Like (0)

Report
Copy link

Link copied!

Hi @Agila,

Data Input from Dataverse Tables with Filters:
When dealing with large datasets, consider using an ETL (Extract, Transform, Load) pipeline that can efficiently filter and extract data from Dataverse.

Instead of using the Xrm tool, explore other options:
Custom FetchXML Queries: You can create custom FetchXML queries directly in your ADF pipeline to fetch data from Dataverse. This allows you to apply filters and retrieve only the necessary records.
Incremental Loading: Implement incremental loading by tracking changes (e.g., modified date) in Dataverse records. Fetch only the updated or new records since the last extraction.
Parallelization: Split large tables into smaller chunks and process them in parallel to improve performance.

Optimizing Data Loading Performance:
Loading 1GB of data in 13 hours is quite slow. To improve performance:
Batching: Break down the data into smaller batches for loading. Use parallel copy activities to load multiple batches concurrently.
Bulk Loading: If possible, use bulk insert methods provided by Dataverse (e.g., ExecuteMultipleRequest in C#) instead of individual record inserts.
Indexing: Ensure that Dataverse tables have proper indexes to speed up data insertion.
Network Latency: Check if network latency between ADF and Dataverse is causing delays. Optimize network connectivity.
Azure Data Factory Integration Runtime: Use a dedicated integration runtime in ADF for better performance.
Data Compression: Compress data before loading to reduce transfer time.
Remember to monitor your ADF pipeline execution and fine-tune based on performance metrics.

Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.

Sunil Pashikanti - Tech Blog: PowerApps

Was this reply helpful? Yes No
AllanDeCastro 412 Most Valuable Professional on at

Like (1)

Report
Copy link

Link copied!

Hello
I was able to perform data migrations with large volumes using ADF, I recommend rather using Dataflows and especially to cut the data in batch using several connections with different App User.

Was this reply helpful? Yes No