web
You’re offline. This is a read only version of the page.
close
Skip to main content
Community site session details

Community site session details

Session Id : rYgwv6LghyXvDQqj58ST+/
Power Apps - Microsoft Dataverse
Answered

Delete Duplicates in Dataverse Table Import

Like (0) ShareShare
ReportReport
Posted on 31 Mar 2021 15:06:19 by

I am working with custom tables for a project and it involves updating the data daily, but whenever I run the job to refresh the data it duplicates all the existing records.  Is there a way to automatically delete duplicates when the import happens?

  • rkhalid81 Profile Picture
    2 on 01 Aug 2023 at 17:59:36
    Re: Delete Duplicates in Dataverse Table Import

    Hi, It won't let me set duplicate detection on custom tables. It just does not show custom tables in the list where you are setting up a new duplicate detection job. Any other ideas?

  • takolota1 Profile Picture
    4,900 Moderator on 09 Jun 2023 at 16:57:58
    Re: Delete Duplicates in Dataverse Table Import

    @Anonymous @NyomanLukas @ChrisPiasecki 

     

    You could also try this custom template for finding & removing duplicates in any Power Automate datasource: https://powerusers.microsoft.com/t5/Power-Automate-Cookbook/Find-and-Remove-Duplicates/m-p/2191403#M1611

  • NyomanLukas Profile Picture
    18 on 02 Aug 2022 at 14:31:47
    Re: Delete Duplicates in Dataverse Table Import

    Hi @ChrisPiasecki, would you mind to elaborate on how to setup the bulk deletion job? I understand how to create a job that deletes ALL items based on a filter (e.g. field X contains Y). But how to remove the duplicates only and keep ONE item?

     

    Thanks!

  • Verified answer
    ChrisPiasecki Profile Picture
    6,397 Most Valuable Professional on 31 Mar 2021 at 16:50:07
    Re: Delete Duplicates in Dataverse Table Import

    Hi @Anonymous,

     

    You can setup a Duplicate Detection Job to detect all the records that have been identified as duplicates. You can then setup a Bulk Deletion Job to delete the records based on the criteria you specify.

     

    There is also an MS provided code sample on how to do a bulk detection and delete in one go in a programmatic way if desired. 

     

    ---
    Please click Accept as Solution if my post answered your question. This will help others find solutions to similar questions. If you like my post and/or find it helpful, please consider giving it a Thumbs Up.

     

     

  • Community Power Platform Member Profile Picture
    on 31 Mar 2021 at 16:40:03
    Re: Delete Duplicates in Dataverse Table Import

    I set up the duplicate detection but it only notifies the user about the duplicates and then I have to manually decide what to keep and what not to keep.  Is there a way to set all duplicates found to be deleted?

  • ChrisPiasecki Profile Picture
    6,397 Most Valuable Professional on 31 Mar 2021 at 15:36:16
    Re: Delete Duplicates in Dataverse Table Import

    Hi @Anonymous ,

     

    You should setup duplicate detection rules to better detect them. If you are using the Data Import Wizard, you can set it to prevent duplicates from being imported. 

     

    Additionally, you can set up Alternate Keys on your table to ensure uniqueness amongst a record using 1 or more columns. If you try to create a duplicate in the system based on the key, then the system will prevent it at the database level.

     

     

    ---
    Please click Accept as Solution if my post answered your question. This will help others find solutions to similar questions. If you like my post and/or find it helpful, please consider giving it a Thumbs Up.

     

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Announcing our 2025 Season 2 Super Users!

A new season of Super Users has arrived, and we are so grateful for…

Paul Stork – Community Spotlight

We are honored to recognize Paul Stork as our July 2025 Community…

Congratulations to the June Top 10 Community Leaders!

These are the community rock stars!

Leaderboard > Power Apps

#1
WarrenBelz Profile Picture

WarrenBelz 791 Most Valuable Professional

#2
MS.Ragavendar Profile Picture

MS.Ragavendar 410 Super User 2025 Season 2

#3
mmbr1606 Profile Picture

mmbr1606 275 Super User 2025 Season 2

Featured topics

Loading complete