I am working with custom tables for a project and it involves updating the data daily, but whenever I run the job to refresh the data it duplicates all the existing records. Is there a way to automatically delete duplicates when the import happens?
Hi, It won't let me set duplicate detection on custom tables. It just does not show custom tables in the list where you are setting up a new duplicate detection job. Any other ideas?
@Anonymous @NyomanLukas @ChrisPiasecki
You could also try this custom template for finding & removing duplicates in any Power Automate datasource: https://powerusers.microsoft.com/t5/Power-Automate-Cookbook/Find-and-Remove-Duplicates/m-p/2191403#M1611
Hi @ChrisPiasecki, would you mind to elaborate on how to setup the bulk deletion job? I understand how to create a job that deletes ALL items based on a filter (e.g. field X contains Y). But how to remove the duplicates only and keep ONE item?
Thanks!
Hi @Anonymous,
You can setup a Duplicate Detection Job to detect all the records that have been identified as duplicates. You can then setup a Bulk Deletion Job to delete the records based on the criteria you specify.
There is also an MS provided code sample on how to do a bulk detection and delete in one go in a programmatic way if desired.
---
Please click Accept as Solution if my post answered your question. This will help others find solutions to similar questions. If you like my post and/or find it helpful, please consider giving it a Thumbs Up.
I set up the duplicate detection but it only notifies the user about the duplicates and then I have to manually decide what to keep and what not to keep. Is there a way to set all duplicates found to be deleted?
Hi @Anonymous ,
You should setup duplicate detection rules to better detect them. If you are using the Data Import Wizard, you can set it to prevent duplicates from being imported.
Additionally, you can set up Alternate Keys on your table to ensure uniqueness amongst a record using 1 or more columns. If you try to create a duplicate in the system based on the key, then the system will prevent it at the database level.
---
Please click Accept as Solution if my post answered your question. This will help others find solutions to similar questions. If you like my post and/or find it helpful, please consider giving it a Thumbs Up.
WarrenBelz
791
Most Valuable Professional
MS.Ragavendar
410
Super User 2025 Season 2
mmbr1606
275
Super User 2025 Season 2