Unanswered

Duplicate data with every refresh

(0) Share

Report

Posted on by AveryLesser

Hi guys,

I setup a dataflow for Dataverse and it automatically adds "Modified by" and "Modified On" columns that result in duplicate rows. I've turned off "Track Changes". How do I stop these duplicate rows from happening? For a Power BI/Analytical Dataflow, it clears out the table when refreshing. Dataverse/Standard Dataflow refreshes look like they just append the data?

I would do incremental refresh, but not all of my dataflow tables have a date column?

Categories:

Microsoft Dataverse with Power Apps

Duplicates.png

I have the same question (0)

All responses (4)

Answers (0)

Community Power Pla... on at

Like (1)

Report

You can add a key to your table. If you create one to your table on the column which is the unique identifier from your source, Thinks should work properly.

Was this reply helpful? Yes No
MrOgilvie123 4 on at

Like (0)

Report

I think what you are looking for is this article. Revolving around making keys within the table.

https://powerusers.microsoft.com/t5/Microsoft-Dataverse/How-do-you-prevent-a-dataflow-from-creating-duplicates/td-p/1337669

Was this reply helpful? Yes No
Ram Prakash Duraisamy 5,593 Super User 2025 Season 2 on at

Like (0)

Report

Hi @AveryLesser

Hope you are doing good.

As per Dataflow you need to have KEYS to avoid duplicate.

Steps to follow.

1. Identify a unique coloum from your Source System Table.

2. Navigate to make.powerapps.com then tables then Click Keys

3. Click Add new Keys and Select the Column

.

Unique column Source = Unique Column Destination

It can be either string, number etc

Please mark as Answer if it is helpful and provide Kudos

Subscribe : https://www.youtube.com/channel/UCnGNN3hdlKBOr6PXotskNLA

Blog : https://microsoftcrmtechie.blogspot.com

Was this reply helpful? Yes No
Community Power Pla... on at

Like (1)

Report
Hello @AveryLesser , I had same issue for a couple of times, and here is my solution, the idea is to designate individual index as a primary key for each record so that Dataverse can identify and stop duplicating:
Dataverse - Dataflow-Right click ellipse (three dots) and Select Edit your Dataflow.

Choose the table you are having trouble with.
Edit
I'll use a dummie dataset as an example

Add Column - Index Column

Then you click "Next" and go to "Publish".

If this post helps, then please consider Accept it as the solution to help the other members find it and grow more quickly.

With Warm Regards,
Liangying

Was this reply helpful? Yes No