cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
jeckard Helper I
Helper I

Remove Duplicates from a Sharepoint List

I have an issue where duplicates are getting made in a Sharepoint List and I would like Flow to on a schedule get all the items in a list and then for each duplicate   The Key to finding the duplicates would be Employee ID so Find items with the same EEID.

then for every duplicate, it finds send me an email

I am open to using REST if someone can help me with how to accomplish this.

Thanks

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
jeckard Helper I
Helper I

Re: Remove Duplicates from a Sharepoint List

Here is what I ended up doing and it works.  It would be nice to get all the ID in an array and then delete them all but the first one...That will be the next thing.

Duplicate1.PNGDuplicate2.PNGDuplicate3.PNG

View solution in original post

14 REPLIES 14
Super User
Super User

Re: Remove Duplicates from a Sharepoint List

Hey @jeckard 

I'd suggest setting the column to enforce unique values to prevent the addition of duplicates:

1.png

HTH

Jay

Highlighted
jeckard Helper I
Helper I

Re: Remove Duplicates from a Sharepoint List

Hey, Jay, I think you meant to upload an image but nothing came through??

Super User
Super User

Re: Remove Duplicates from a Sharepoint List

Hey @jeckard, sorry try now?

Dual Super User
Dual Super User

Re: Remove Duplicates from a Sharepoint List

If you just want to scan for duplicates and send an email for each duplicate then you can just do two Apply to Each loops, with one inside the other.  Before the first loop do a get items and inside the first loop do a get items again.  Then in the second loop compare the value of the record from the first loop with the record from the second.  If its a match then send an email.  I used a list that has a Project choice field, just substitute the Employeeid field.

screenshot.png



-------------------------------------------------------------------------
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.
jeckard Helper I
Helper I

Re: Remove Duplicates from a Sharepoint List

I will give you solution a try tonight and let you know!

jkeckard007 Helper III
Helper III

Re: Remove Duplicates from a Sharepoint List

 

 I tried the solution you suggested I believe I might be setting up the loops incorrectly

Can you clarify Value one and Value two as I am getting the system creating a  apply to each loop three?

 

Loop2.png

Community Support
Community Support

Re: Remove Duplicates from a Sharepoint List

Hi @jeckard ,

 

Please refer to screenshot below to create the flow:

1.png

 

Best regards,

Alice       

Community Support Team _ Alice Zhang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Super User
Super User

Re: Remove Duplicates from a Sharepoint List

@jkeckard007 

I think you'll need @Pstork1 to provide some more guidance here... but I think you'll need to do the following:

1. Delete the 'Apply to each 2' and 'Apply to each 3' (and contained actions)

2. Re-add a 'Condition' action directly underneath the second 'Get Items' action

3. Select the 'Employee ID' value from the 'Get Items 1 and compare to the the 'Employee ID' value from the 'Get Items' 2... that should set the loop up correctly.

This will find duplicates but its a pretty expensive query, and it won't work for large lists (above 5000) even after enabling pagination... Working with @Pstork1 excellent suggestion I updated to the following but it still suffers the same limitation with the original query... the second query is slightly quicker as it will return a smaller result set and only do something if the array is greater than 1... i..e a duplicate exists.

1.png

  This is still not really correct and needs more logic to be added a single duplicate would result in 2 separate emails (One for each duplicate item)... if your list contains less than 5000 items this may work, but I'd really suggest disabling duplicates on the list as per my original suggestion.

If you're list is above 5000 items I think will need a direct REST query using a CAML query to perform a GroupBy type query (I know distinct isn't support by ODATA)...

HTH

Jay

 

Dual Super User
Dual Super User

Re: Remove Duplicates from a Sharepoint List

If you compare your screenshot to mine you will see that I have two get items statements and two loops.  You have one get items statement and three loops.  From what I can see I assume your first get items isn't shown in the screenshot.  If so, then I don't see a need for Apply to each 2.  Remove that loop.  The outside loop should be the first Get items value and the inside loop should be the second.



-------------------------------------------------------------------------
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.
jeckard Helper I
Helper I

Re: Remove Duplicates from a Sharepoint List

Here is what I ended up doing and it works.  It would be nice to get all the ID in an array and then delete them all but the first one...That will be the next thing.

Duplicate1.PNGDuplicate2.PNGDuplicate3.PNG

View solution in original post

Community Support
Community Support

Re: Remove Duplicates from a Sharepoint List

Hi @jeckard ,

 

If your problem been solved, please go ahead and mark the post as solved by clicking “Accept as Solution” so that this thread will be marked for other users to easily identify!

 

Best regards,

Alice       

Community Support Team _ Alice Zhang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

jkeckard007 Helper III
Helper III

Re: Remove Duplicates from a Sharepoint List

There is no accept as a solution just reply and mark as new.

Community Support
Community Support

Re: Remove Duplicates from a Sharepoint List

Hi @jkeckard007 ,

 

I will help you mark the post as answered.

 

Best regards,

Alice       

 

Community Support Team _ Alice Zhang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

testasdfasfsafa Advocate II
Advocate II

Re: Remove Duplicates from a Sharepoint List

I can't believe Microsoft doesn't have a simple array action called "removed duplicates" like Nintex had since 2007!!!

Helpful resources

Announcements
firstImage

New Ranks and Rank Icons released on April 21!

The time has come: We are finally able to share more details on the brand-new ranks coming to the Power Automate Community!

firstImage

Now Live: Power Virtual Agents Community!

We are excited to announce the launch of Power Virtual Agents Community. Check it out now!

firstImage

New & Improved Power Automate Community Cookbook

We've updated and improved the layout and uploading format of the Power Automate Cookbook!

thirdimage

Power Automate Community User Group Member Badge

Fill out a quick form to claim your user group badge now!

sixthImage

Community Summit North America

The top training and networking event across the globe for Microsoft Business Applications

Top Solution Authors
Users online (7,550)