web
You’re offline. This is a read only version of the page.
close
Skip to main content

Announcements

News and Announcements icon
Community site session details

Community site session details

Session Id :
Power Platform Community / Forums / Power Automate / How to remove duplicat...
Power Automate
Answered

How to remove duplicates from a "tricky" array

(0) ShareShare
ReportReport
Posted on by 294

Hi,

I have the following array and would like to remove the duplicates. The "Email" and "MainOrgID" are common in all, but the "SHID" (stakeholder ID) is unique. I tried using the 'union' function but clearly didn't work. Can someone please help?

Thanks!

[
  {
    "EmailinSelect": "Maria.Conner@fakecorp.com",
    "MainOrgIDinSelect": "Org-ID-1075",
    "SHIDinSelect": "SH-ID-0657"
  },
  {
    "EmailinSelect": "Ruth.Guercio@fakecorp.com",
    "MainOrgIDinSelect": "Org-ID-1075",
    "SHIDinSelect": "SH-ID-0659"
  },
  {
    "EmailinSelect": "Maria.Conner@fakecorp.com",
    "MainOrgIDinSelect": "Org-ID-1075",
    "SHIDinSelect": "SH-ID-0660"
  },
  {
    "EmailinSelect": "Calvin.Rigdon@fakecorp.com",
    "MainOrgIDinSelect": "Org-ID-1075",
    "SHIDinSelect": "SH-ID-0658"
  },
  {
    "EmailinSelect": "Calvin.Rigdon@fakecorp.com",
    "MainOrgIDinSelect": "Org-ID-1075",
    "SHIDinSelect": "SH-ID-0661"
  }
]
Categories:
I have the same question (0)
  • SudeepGhatakNZ Profile Picture
    14,396 Most Valuable Professional on at

    @karimabdelrazek ,

     

    Add a Select action to pick only "EmailinSelect" and "MainOrgIDinSelect" from the given array.

    Apply the union function to the output of the Select

    SudeepGhatakNZ_0-1675033682081.png

     

  • karimabdelrazek Profile Picture
    294 on at

    @SudeepGhatakNZ . Thank you for the reply. How fo I retain SHID values of the remaining records?

  • grantjenkins Profile Picture
    11,063 Moderator on at

    What are you trying to achieve with your flow overall? If you wanted to remove duplicates, but keep all the SHIDinSelect values, would you be expecting the following JSON?

     

    [
     {
     "EmailinSelect": "Maria.Conner@fakecorp.com",
     "MainOrgIDinSelect": "Org-ID-1075",
     "SHIDinSelect": [
     "SH-ID-0657",
     "SH-ID-0660"
     ]
     },
     {
     "EmailinSelect": "Ruth.Guercio@fakecorp.com",
     "MainOrgIDinSelect": "Org-ID-1075",
     "SHIDinSelect": [
     "SH-ID-0659"
     ]
     },
     {
     "EmailinSelect": "Calvin.Rigdon@fakecorp.com",
     "MainOrgIDinSelect": "Org-ID-1075",
     "SHIDinSelect": [
     "SH-ID-0658",
     "SH-ID-0661"
     ]
     }
    ]

     

    To get exactly what you're looking for, we'd need to understand what the main use of the data is for. If you could provide screenshots of what you currently have, that also help.

  • karimabdelrazek Profile Picture
    294 on at

    @grantjenkins . Thank you for the reply. The scenario and context of this use case are as follows:

    • I have a master list of employees that contains the above-mentioned fields
    • This master list is populated by uploading an excel from different users using a flow.
    • There is a possibility that duplicates will end up being created in the master list
    • I want to run a flow (periodically) to find and remove those duplicates in the master list

    Please let me know if this is sufficient or if you need more info. Thanks for your help!

  • grantjenkins Profile Picture
    11,063 Moderator on at

    What would you expect to happen with the following two objects?

     

     {
     "EmailinSelect": "Calvin.Rigdon@fakecorp.com",
     "MainOrgIDinSelect": "Org-ID-1075",
     "SHIDinSelect": "SH-ID-0658"
     },
     {
     "EmailinSelect": "Calvin.Rigdon@fakecorp.com",
     "MainOrgIDinSelect": "Org-ID-1075",
     "SHIDinSelect": "SH-ID-0661"
     }

     

    They have the same EmailinSelect and MainOrgIDinSelect, but different SHIDinSelect. I'm still a bit confused what you want to do with the different SHIDinSelect values.

  • karimabdelrazek Profile Picture
    294 on at

    In the above output, I want to keep one and remove the other. I was going to use the SHID to find and remove one of those two from the master list. I'm open to better ways to do this. Disclaimer: i'm a newbie at power automate 🙂

    My ultimate objective is to keep a clear master list with no duplicates, taking into consideration that users will periodically, and accidently create duplicates.

  • grantjenkins Profile Picture
    11,063 Moderator on at

    Can definitely do it - just need to understand which of the duplicate SHID values would be removed, and which one remains. Do you have the logic for this? Maybe keep the one with the largest ID, or something like that?

  • karimabdelrazek Profile Picture
    294 on at

    @grantjenkins Yes, I was thinking the one with the smallest value would remain, as that would be the "original" one that was added. In the end, it won't make a difference to me which one stays/ goes because I tag everything in my app to the user's email ID. 

    Do you have something I can implement to achieve this? I've been going around in circles for days now and it's driving me nuts.

  • karimabdelrazek Profile Picture
    294 on at

    @grantjenkins . Could you please tell me how you generated this JSON?

  • Verified answer
    grantjenkins Profile Picture
    11,063 Moderator on at

    @karimabdelrazek Sorry for the delay in getting back to you. I've been in workshops the last couple of weeks. Hopefully this is what you're looking for. It will give you the JSON as per the previous post.

     

    See full flow below. I'll go into each of the actions.

    grantjenkins_0-1677389092616.png

     

    Data (Compose) contains your original JSON data.

    grantjenkins_1-1677389123431.png

     

    Select Base (Select) maps the properties, excluding the SHIDinSelect, using the following expressions.

    //EmaininSelect
    item()?['EmailinSelect']
    
    //MainOrgIDinSelect
    item()?['MainOrgIDinSelect']

    grantjenkins_2-1677389209071.png

     

    XML (Compose) uses the output from Data and transforms it into XML so we can use XPath in the next action. The expression used is:

    xml(json(concat('{"root": { value:', outputs('Data'), '}}')))

    grantjenkins_3-1677389281579.png

     

    Select (Select) uses the following expressions to get the unique objects from Select Base. It then uses Map to getthe current items and adds a new property that contains an array of all the related SHIDinSelect values.

    //From
    union(body('Select_Base'), body('Select_Base'))
    
    //Map
    addProperty(item(), 'SHIDinSelect', xpath(outputs('XML'), concat('//root/value[EmailinSelect="', item()?['EmailinSelect'], '" and MainOrgIDinSelect="', item()?['MainOrgIDinSelect'], '"]/SHIDinSelect/text()')))

    grantjenkins_4-1677389491002.png

     

    After running the flow, Select would have the following output.

    [
     {
     "EmailinSelect": "Maria.Conner@fakecorp.com",
     "MainOrgIDinSelect": "Org-ID-1075",
     "SHIDinSelect": [
     "SH-ID-0657",
     "SH-ID-0660"
     ]
     },
     {
     "EmailinSelect": "Ruth.Guercio@fakecorp.com",
     "MainOrgIDinSelect": "Org-ID-1075",
     "SHIDinSelect": [
     "SH-ID-0659"
     ]
     },
     {
     "EmailinSelect": "Calvin.Rigdon@fakecorp.com",
     "MainOrgIDinSelect": "Org-ID-1075",
     "SHIDinSelect": [
     "SH-ID-0658",
     "SH-ID-0661"
     ]
     }
    ]


    ----------------------------------------------------------------------
    If I've answered your question, please mark the post as Solved.
    If you like my response, please consider giving it a Thumbs Up.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Introducing the 2026 Season 1 community Super Users

Congratulations to our 2026 Super Users!

Kudos to our 2025 Community Spotlight Honorees

Congratulations to our 2025 community superstars!

Congratulations to the March Top 10 Community Leaders!

These are the community rock stars!

Leaderboard > Power Automate

#1
Haque Profile Picture

Haque 592

#2
Valantis Profile Picture

Valantis 340

#3
11manish Profile Picture

11manish 284

Last 30 days Overall leaderboard