Skip to main content

Notifications

Power Automate - General Discussion
Answered

Remove Duplicates from a Sharepoint List

(0) ShareShare
ReportReport
Posted on by 37

I have an issue where duplicates are getting made in a Sharepoint List and I would like Flow to on a schedule get all the items in a list and then for each duplicate   The Key to finding the duplicates would be Employee ID so Find items with the same EEID.

then for every duplicate, it finds send me an email

I am open to using REST if someone can help me with how to accomplish this.

Thanks

 

 

  • takolota1 Profile Picture
    takolota1 4,781 on at
    Re: Remove Duplicates from a Sharepoint List

    @jeckard @SunshineK @testasdfasfsafa @v-alzhan-msft @jkeckard007 

     

    There is now a more efficient way to find and remove duplicates from any dataset: https://powerusers.microsoft.com/t5/Power-Automate-Cookbook/Find-and-Remove-Duplicates/m-p/2191403#M1611

  • SunshineK Profile Picture
    SunshineK 102 on at
    Re: Remove Duplicates from a Sharepoint List

    Can this be used if I want to update an item? 

     

    For e.g. I have a flow from Survey123 to Sharepoint list. I want to be able to create items (this works all good) and then update the item if an answer has changed (update is based on the unique ID I have created that passes through from Survey123 to Sharepoint List). 

    Currently it just keeps creating a new item. 

    Cheers, 

    Summer 

  • testasdfasfsafa Profile Picture
    testasdfasfsafa 307 on at
    Re: Remove Duplicates from a Sharepoint List

    I can't believe Microsoft doesn't have a simple array action called "removed duplicates" like Nintex had since 2007!!!

  • Re: Remove Duplicates from a Sharepoint List

    Hi @jkeckard007 ,

     

    I will help you mark the post as answered.

     

    Best regards,

    Alice       

     

    Community Support Team _ Alice Zhang
    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

  • jkeckard007 Profile Picture
    jkeckard007 108 on at
    Re: Remove Duplicates from a Sharepoint List

    There is no accept as a solution just reply and mark as new.

  • Re: Remove Duplicates from a Sharepoint List

    Hi @jeckard ,

     

    If your problem been solved, please go ahead and mark the post as solved by clicking “Accept as Solution” so that this thread will be marked for other users to easily identify!

     

    Best regards,

    Alice       

    Community Support Team _ Alice Zhang
    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

  • Verified answer
    jeckard Profile Picture
    jeckard 37 on at
    Re: Remove Duplicates from a Sharepoint List

    Here is what I ended up doing and it works.  It would be nice to get all the ID in an array and then delete them all but the first one...That will be the next thing.

    Duplicate1.PNGDuplicate2.PNGDuplicate3.PNG

  • Pstork1 Profile Picture
    Pstork1 64,839 on at
    Re: Remove Duplicates from a Sharepoint List

    If you compare your screenshot to mine you will see that I have two get items statements and two loops.  You have one get items statement and three loops.  From what I can see I assume your first get items isn't shown in the screenshot.  If so, then I don't see a need for Apply to each 2.  Remove that loop.  The outside loop should be the first Get items value and the inside loop should be the second.

  • Jay-Encodian Profile Picture
    Jay-Encodian 2,920 on at
    Re: Remove Duplicates from a Sharepoint List

    @jkeckard007 

    I think you'll need @Pstork1 to provide some more guidance here... but I think you'll need to do the following:

    1. Delete the 'Apply to each 2' and 'Apply to each 3' (and contained actions)

    2. Re-add a 'Condition' action directly underneath the second 'Get Items' action

    3. Select the 'Employee ID' value from the 'Get Items 1 and compare to the the 'Employee ID' value from the 'Get Items' 2... that should set the loop up correctly.

    This will find duplicates but its a pretty expensive query, and it won't work for large lists (above 5000) even after enabling pagination... Working with @Pstork1 excellent suggestion I updated to the following but it still suffers the same limitation with the original query... the second query is slightly quicker as it will return a smaller result set and only do something if the array is greater than 1... i..e a duplicate exists.

    1.png

      This is still not really correct and needs more logic to be added a single duplicate would result in 2 separate emails (One for each duplicate item)... if your list contains less than 5000 items this may work, but I'd really suggest disabling duplicates on the list as per my original suggestion.

    If you're list is above 5000 items I think will need a direct REST query using a CAML query to perform a GroupBy type query (I know distinct isn't support by ODATA)...

    HTH

    Jay

     

  • Re: Remove Duplicates from a Sharepoint List

    Hi @jeckard ,

     

    Please refer to screenshot below to create the flow:

    1.png

     

    Best regards,

    Alice       

    Community Support Team _ Alice Zhang
    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Microsoft Kickstarter Events…

Register for Microsoft Kickstarter Events…

Announcing Our 2025 Season 1 Super Users!

A new season of Super Users has arrived, and we are so grateful for the daily…

Announcing Forum Attachment Improvements!

We're excited to announce that attachments for replies in forums and improved…

Leaderboard

#1
WarrenBelz Profile Picture

WarrenBelz 145,508

#2
RandyHayes Profile Picture

RandyHayes 76,287

#3
Pstork1 Profile Picture

Pstork1 64,839

Leaderboard