Incremental refreshing in the Dataverse

(0) Share

Report

Posted on by nleuck_101

421

Hello All,

I need some help in understanding the Incremental refresh in the Dataverse. I tried using that to import a SharePoint list to my Dataverse. I set the refresh schedule for every hour. I thought it would only add new records or update existing records but instead it duplicated the records every hour. What am I not understanding or missing when setting this up? Any help would be greatly appreciated.

Thanks,

Categories:

Dataflows

Microsoft Dataverse

I have the same question (0)

All responses (7)

Answers (0)

ronaldwalcott 3,827 Super User 2025 Season 2 on at

Like (0)

Report

Incremental refreshing in the Dataverse

There should be a datetime field in the record which you use in configuring the incremental refresh which identifies the age of the record.

Was this reply helpful? Yes No
nleuck_101 421 on at

Like (0)

Report

Incremental refreshing in the Dataverse

I was using the import option within Dataverse. I don't see there being an option on setting the refresh to look at a certain column. Is there a different approach I need to take.

Was this reply helpful? Yes No
ronaldwalcott 3,827 Super User 2025 Season 2 on at

Like (0)

Report

Incremental refreshing in the Dataverse

Import doesn't understand if a record is new or existing so it will just create a new primary key for anything that you import.

I believe that you have to set up Duplicate Detection Rules. What makes the records unique other than the key generated within Dataverse.

Set up duplicate detection rules to keep your data clean - Power Platform | Microsoft Learn

If you want to merge records then you'll have to check the documentation at Import Data (Microsoft Dataverse) - Power Apps | Microsoft Learn as I am unsure how to configure that except to know it needs some way to identify that the record has changed.

Was this reply helpful? Yes No
nleuck_101 421 on at

Like (0)

Report

Incremental refreshing in the Dataverse

Setting up a Duplicate detection rule didn't seem to work. Maybe I set it up incorrectly. I need to import the data first and then add new or update old records when changes are made in SharePoint. I really wanted to use the virtual tables but you can't connect to them in Power BI (at least I couldn't find a way).

Was this reply helpful? Yes No
ronaldwalcott 3,827 Super User 2025 Season 2 on at

Like (0)

Report

Incremental refreshing in the Dataverse

Since you need to update data you could take two approaches.

Use Dataflows

Create and use dataflows in Power Apps - Power Apps | Microsoft Learn

or import the data into a separate table and use Power Automate or Power Apps to merge the records.

Was this reply helpful? Yes No
Suggested answer

Garima_PowerPlatform 170 on at

Like (2)

Report

Incremental refreshing in the Dataverse

Hi,

If you want more control over how the data is merged, especially when records in SharePoint are updated or added, you could import the data into a separate table in Dataverse and then use Power Automate or Power Apps to merge the records. This approach might be more suitable if you want to run complex logic or need to handle merging records manually.

Was this reply helpful? Yes No
Suggested answer

nleuck_101 421 on at

Like (0)

Report

Incremental refreshing in the Dataverse

I ended it up creating a flow that when an Item is created or modified in SharePoint to add or update the record in Dataverse.

Was this reply helpful? Yes No