cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
der_cosmicyes
Helper I
Helper I

Delete duplicates in a SharePoint List

Hello all,

I have got a SharePoint List with about 1400 items. It only should be arround 800 items and I found out that many items have duplicates. I don´t know why and that´s also not the point here.
My idea is to create a Flow which will get each row of the list and goes through all other rows and checks if *every* column is equal. If yes then delete this row.

I found this here in the forum:
Solved: Delete Duplicate in SharePoint List - Power Platform Community (microsoft.com)
But this is not exactly what I need I think.
I created this flow (unfortunately it is german) - only 3 conditions as an example. In reallity there are 8 columns.

 
 

Screenshot 2021-01-18 115100.jpg


I did not run it yet as I am pretty sure that this Flow will delete everything - or am I wrong?
How can I achieve that the flow deletes only duplicates but leaves one entry of each?

Thank you in advance!

1 ACCEPTED SOLUTION

Accepted Solutions
fchopo
Super User
Super User

Hello @der_cosmicyes 

In your solution you should another apply to each. I mean, when do the comparison, you are comparing the values of the same item, so it will be always true, and therefore, it will delete all items.

You need an algorithm like that:

1) Get ALL items.

2) For each item.

3) Get ALL items AGAIN

4) For each item in step 3, compare it against the item in step 2.

5) If the values are the same, remove it.

If you find the use of OData filters too complicated, you could do something like this:

getitems.png

 

Hope it helps!

Ferran

 

Did I answer your question? Please consider to mark my post as a solution to help others.
Proud to be a Flownaut!

View solution in original post

30 REPLIES 30
fchopo
Super User
Super User

Hello @der_cosmicyes 

You could do the following:

1) Get all items.

2) For each item.

3) Look for an identical one, using get items and an Odata Filter query expression.

4) If more than one element is found (check the lenght of the ouput of the previous step) -> there is a duplicate.

5) Delete the item.

Have a look at this example:

filterQuery2.png

 

Therefore, in your "Get Items 2" action, you should write an OData Filter query with the 8 fields to compare.

 

Have a look at this sources on how to work with this kind of expressions: 

Every Power Automate (MS Flow) Filter Query You Ever Wanted To Know As A Functional Consultant – DIY...

OData Filter query in SharePoint Get items in Flow (sharepains.com)

Hope it helps!

Ferran

Did I answer your question? Please consider to mark my post as a solution to help others.
Proud to be a Flownaut!

Thank you Ferran,
Thank you! This sounds a little complicated to a PowerAutomate Newbie. I haven´t heard of OData Filters but I am willing to dive deeper into it.
But one question:
Is there no way to combine "my" solution with the idea to check if more than 1 item is found?
I think my solution would delete everything but if I can check if more than 1 one was found this could be the solution. What do you think?

fchopo
Super User
Super User

Hello @der_cosmicyes 

In your solution you should another apply to each. I mean, when do the comparison, you are comparing the values of the same item, so it will be always true, and therefore, it will delete all items.

You need an algorithm like that:

1) Get ALL items.

2) For each item.

3) Get ALL items AGAIN

4) For each item in step 3, compare it against the item in step 2.

5) If the values are the same, remove it.

If you find the use of OData filters too complicated, you could do something like this:

getitems.png

 

Hope it helps!

Ferran

 

Did I answer your question? Please consider to mark my post as a solution to help others.
Proud to be a Flownaut!

Thank you Ferran,
I have not tried yet but this seems very good to me!
Espacially the condition ID(1) is NOT equal to ID(2) is a very good idea.
I will try as soon as possible and then Accept as Solution (I hope 😉 )

Best regards

Hi Ferran,
ok, I know why you wanted me to use OData Query: Performance! 🙂
Unfortunately I could not solve it using OData.
I am using now your second approach (which is really very slow...) but it works.
The only thing I changed ist to use ID of 'Get Items' and not of 'Get Items 2':

I set up a very small list with Items
* Paul McCartney
* John Lennon
* George Harrison
* Ringo Starr
* Paul McCartney

and used your second approach.
When deleting ID of 'Get Items 2' both Pauls are gone and the Beatles are a Trio 😉
Using ID of 'Get Items' worked.
You helped me very much - thank you!

@fchopo how would you remove all but one? I want to remove duplicates but keep one.

Anonymous
Not applicable

Same here! I need to remove all items but one. How can this be achieved?

 

Thank you!

Anonymous
Not applicable

I solved this recently. I have an automation that finds duplicates from a Sharepoint Online List and removes duplicates, leaving the original one.

 

The start of the flow is the same as the example before, so you use Odata to filter the items and check that the length of the results is more than 1. There are some changes I made after that and one change at the beginning of the flow. I created a variable. The variable contains the Sharepoint List data and the variable is updated after each filtering. At the end of the flow, I extract the List item ID's from the Array and use those ID's in the "Delete item" part. My flow is in finnish, but i'll try to explain with a picture:

automaatio.png

 

Hope this helps!

Thank you on your sharing.

Would it be okay for your to share more detail how the ODATA FILTERED ITEMS configured? 

 

I've tried, but it does not work. I'm new in Power automate, I wish to learn more how it works to improve the performance. 

Anonymous
Not applicable

Sure!

 

Select the "Get items" step, and open the advanced settings. Then type the code in the Odata filtering field. My code is below, but you can use whatever suits your needs. The field names are case sensitive, so keep that in mind.

 

odatafilter.PNG

 

Does this answer your question?

Thank you @Anonymous . 

 

I would like to learn the logic. Would it be okay to get your input from few items below: 

  1. From the benksyong_1-1656913167952.png

    if Title has to match with Sharepoint List column's header, then, 'benksyong_0-1656912328677.png' is refer in Dynamic content with same column header? 

  2. During Odata filtering Dynamic content selection, should I refer to 'Get items' or 'Get items 2'? why? what logic behind?
  3. If I've 8 columns in my SharePoint List that I need to match with logic if all 8 column contents are identical, then consider the line item is duplicated. In this case, if that means i need to repeat all 8 column entry in OData filtering with example,
    • Title eq 'Title' AND column2 SP header eq 'Dynamic content: column2' AND column3 SP header eq 'Dynamic content: column3' AND....... column 8 SP header eq 'Dyanamic contect: column 8' 
  4. if the 'Create the variable' is necessary step to define which specific item should be deleted?

 

 

Anonymous
Not applicable

 

I'll go trough this step by step.

 

The first step in the flow is to get the items without any filtering. Let's say that's called "Get items 1". You put the result of that into a variable (because you need to define the variable at the start of the flow) and let's say the variable is called YourVariable:

 

ilied84_0-1656914522007.png

 

Then you add the "apply to all" step, and under that you create the Odata filtered get items -step. Let's call that "Get items 2". The logic in the Odata filtering is that you are comparing two lists of items. So basically "If the title of get items 2 is equal to a title in get items 1 and Endtime in get items 2 is equal to Endtime in get items 1". This of course returns all the values, but it groups the items based on that Odata filtering. So you can have an item with one result, or an item with 4 results, or however many duplicates that filtering can find.

 

After the filtering is done, you update the variable (YourVariable) you created at the start of the flow with the results of the filtering. Then you create an if-statement step in which you check the "length" of the variable items. Length in this case means, that how many results does one item contain. When you add the if-statement, add the code:
Length(variables('YourVariable')) and select "is greater than 1" from the dropdown menu. This skips all the results that are not duplicates.

 

ilied84_1-1656915429173.png

 

The next step is "Compose". What we are doing here is a step, that skips the first result of YourVariable after the length has been checked. At this point all the non-duplicate entries are not present and we are left with ONLY the duplicates, we need to skip the first result, so that there will be one result left in the sharepoint online list. The code for the Compose -step is: skip(variables('YourVariable'),1)

 

Now it's time to update YourVariable again with the results of the Compose -step.

 

The final step is to get all the ID-numbers from YourVariable and delete the corresponding items from the Sharepoint Online list. You add the "delete items" step and in the item ID field you add the following code:

variables('YourVariable')[0]['ID']

 

This finds all the ID's and removes the list items based on those values. I'm not completely sure about the [0], but I think it removes null values, if there are any. I found this in an example and it was not explained. Anyway, the Flow works very well in my case.

 

ilied84_2-1656916536421.png

 

hi,  I ran into problem as below step: Column 'status does not exist'

benksyong_0-1657034276035.png

 

but the Status column indeed remained in my SharePoint List, and my Odata filter entry: 

benksyong_1-1657034489489.png

benksyong_2-1657034607802.png

 

With example data in SharePoint list, the duplicated item is 

Title = a AND Status = aaa. 

 

Expected result post-automate is to remove one of the item named Title= a AND Status = aaa. ...still can't get it right. seek advise. 

 

Regards,

Yong

Thanks for sharing this.

I thought it was working beautifully until I realized it's creating duplicates for each array and deleting them.

My list, which was originally at 2k (no duplicates because the other flow only run once), now only has 1 row...

Any suggestions? I followed your step exactly.

Thanks for this solution! Have you figured out a quicker version of this? My version takes more than 2 minutes to delete a duplicate entry.

Here's the flow I built:
(Overview)

kymramos_0-1657456411463.png

 

(First pair of "get items from SharePoint" + "apply action if condition was met")

kymramos_1-1657456500793.png

 

(Second pair)

kymramos_2-1657456961763.png

 

(Third and last pair)

*A replicate of the first pair, but instead, referencing the version of the SharePoint list (product of the series of Japanese-English translations).

 

New data gets loaded into the query (Power BI) whenever it gets refreshed, and so whenever the button is clicked, I think Power Automate always starts from the very first row of data and hence the duplicate entries generated. Is there a way to tell Power Automate to only create items that are not currently in the SharePoint list so that it won't have to generate duplicate entries?

 

I hope my explanation was clear!

Please let me know if you'd need me to expand.

 

Thanks a bunch!

Hi,

I used this method, but I have a problem when a title has an apostrophe in it. It breaks the ODATA query.

 

Do you have an idea of an expression that would not break the "Get the items"?

Thanks by advance

it doesn't work

@fchopo 

Hi,

 

you need to check what name/code "Get the items" creates for you columns.

The link below shows you, where you can find the correct spelt name of your columns (Post from: ScottShearer 08-16-2019 09:56 AM)


https://powerusers.microsoft.com/t5/General-Power-Automate/OData-filter-column-with-space/td-p/34241...

 

Does it help?

 

Regards

Helpful resources

Announcements

Tuesday Tips: Getting Started in the Community

TUESDAY TIPS is back!   This weekly series of posts is our way of sharing helpful things we've learned or shared that have helped members of the Community. Whether you're just getting started or you're a seasoned pro, Tuesday Tips will help you know where to go, what to look for, and navigate your way through the ever-growing--and ever-changing--world of the Power Platform Community! The original run of Tuesday Tips was a highlight of last year, and these all-new Tips will hopefully prove to be just as informative as helpful. We will cover some basics about the Community, a few "insider tips" to make your experience even better, and sharing best practices gleaned from our most active community members and Super Users. Make sure to watch the News & Announcements each week for the latest and greatest Tuesday Tips!   THIS WEEK: I'm Brand New! What Do I Do? The number of new community members we have each week is pretty amazing, and we are so glad to welcome all of you to the Community! You may be wondering. "What do I do? Where do I get started? Will anyone be willing to help me? What I have a question? Help!"   Let's start with this: Welcome to the low-code revolution, and more importantly, welcome to the Power Platform Community! This is a great place to start. Whether you're busy with Power Apps, getting familiar with Power Automate, engaging Copilot Studio, or building in Power Pages, there are a few key places you should check out as you begin your journey: FORUMS: The forums are THE place to ask questions, look at questions asked by other Community members—and see answers and solutions from our Super Users and other helpful people in the Community. Power Apps ForumsPower Automate ForumsCopilot Studio ForumsPower Pages Forums   NEWS & ANNOUNCEMENTS: Our News & Announcements section highlights the newest and greatest updates in the Community, news from the product team, and so much more. It’s updated a few times each week and will also help you find ways to connect with what’s going on in the ever-growing world of Power Platform. Power Apps News & AnnouncementsPower Automate News & AnnouncementsCopilot Studio News & AnnouncementsPower Pages News & Announcements   GALLERIES: The Galleries section of the Community features tons of tips and tricks, features and benefits, and more—through videos created by our Super Users, product teams, and other helpful members of the Community. Power Apps GalleriesPower Automate Galleries Copilot Studio GalleriesPower Pages Galleries BLOGS: The community blogs section is full of handy step-by-step tips from members of the Community—and some of them include detailed answers to some of the questions most frequently asked questions, as well as how they solved a problem they faced. Power Apps Community BlogPower Automate Community BlogCopilot Studio Community BlogPower Pages Community Blog POWER UP PROGRAM: If you’d like to really take a huge step forward in your journey, we recommend checking out the Power Up Program, a Microsoft-sponsored initiative that trains new Power Platform users and has been a huge success since it launched a little over a year ago. There’s a waiting list, so definitely apply soon if you’re interested! Find out more here: Microsoft Power Up Program for career switchers.   There's so much more you'll discover in your Power Platform experience, and this Community is here for YOU! We are glad you've discovered us and can't wait to see where you grow! If you're new to the Community and just getting started, make sure to give this post a kudo and introduce yourself so we can welcome you!

Super User of the Month | Drew Poggemann

As part of a new monthly feature in the Community, we are excited to share that Drew Poggemann is our featured Super User for the month of February 2024. If you've been in the Community for a while, we're sure Drew's name is familiar to you, as he is one of our most active contributors--he's been a Super User for five consecutive seasons!   Since authoring his first reply 5 years ago to his 514th solution authored, Drew has helped countless Community members with his insights and expertise. In addition to being a Super User, Drew is also a User Group leader and a Microsoft MVP. His contributions to our Super User sessions and to the new SUIT program are always welcome--as well as his sense of humor and fun-loving way of sharing what he knows with others.   When Drew is not solving problems and authoring solutions, he's busy overseeing the Solution Architecture team at HBS, specializing in application architecture and business solution strategy--something he's been doing for over 30 years. We are grateful for Drew and the amazing way he has used his talent and skills to help so many others in the Community. If you are part of the SUIT program, you got to hear some great tips from Drew at the first SUIT session--and we know he still has much more to share!You can find him in the Community and on LinkedIn. Thank you for all you do, Drew!

Announcing Power Apps Copilot Cookbook Gallery

We are excited to share that the all-new Copilot Cookbook Gallery for Power Apps is now available in the Power Apps Community, full of tips and tricks on how to best use Microsoft Copilot as you develop and create in Power Apps. The new Copilot Cookbook is your go-to resource when you need inspiration--or when you're stuck--and aren't sure how to best partner with Copilot while creating apps.   Whether you're looking for the best prompts or just want to know about responsible AI use, visit Copilot Cookbook for regular updates you can rely on--while also serving up some of your greatest tips and tricks for the Community. Our team will be reviewing posts using the new "Copilot Studio" label to ensure we highlight and amplify the most relevant and recent content, so you're assured of high-quality content every time you visit. If you share a post that gets featured in the curated gallery, you'll get a PM in the Community to let you know!The curated gallery is ready for you to experience now, so visit the new Copilot Cookbook for Power Apps today: Copilot Cookbook - Power Platform Community. We can't wait to see what you "cook" up!    

Celebrating a New Season of Super Users with Charles Lamanna, CVP Microsoft Business Applications

February 8 was the kickoff to the 2024 Season One Super User program for Power Platform Communities, and we are thrilled to welcome back so many returning Super Users--as well as so many brand new Super Users who started their journey last fall. Our Community Super Users are the true heroes, answering questions, providing solutions, filtering spam, and so much more. The impact they make on the Communities each day is significant, and we wanted to do something special to welcome them at our first kickoff meeting of the year.   Charles Lamanna, Microsoft CVP of Business Applications, has stressed frequently how valuable our Community is to the growth and potential of Power Platform, and we are honored to share this message from him to our 2024 Season One Super Users--as well as anyone who might be interested in joining this elite group of Community members.     If you want to know more about Super Users, check out these posts for more information today:    Power Apps: What is A Super User? - Power Platform CommunityPower Automate: What is A Super User? - Power Platform Community Copilot Studio: What is A Super User? - Power Platform Community Power Pages: What is A Super User? - Power Platform Community

Super Users 2024 Season One is Here!

   We are excited to announce the first season of our 2024 Super Users is here! Our kickoff to the new year welcomes many returning Super Users and several new faces, and it's always exciting to see the impact these incredible individuals will have on the Community in 2024! We are so grateful for the daily difference they make in the Community already and know they will keep staying engaged and excited for all that will happen this year.   How to Spot a Super User in the Community:Have you ever written a post or asked for help in the Community and had it answered by a user with the Super User icon next to their name? It means you have found the actual, real-life superheroes of the Power Platform Community! Super Users are our heroes because of the way they consistently make a difference in the Community. Our amazing Super Users help keep the Community a safe place by flagging spam and letting the Community Managers know about issues. They also make the Community a great place to find answers, because they are often the first to offer solutions and get clarity on questions. Finally, Super Users share valuable insights on ways to keep the Community growing, engaging, and looking ahead!We are honored to reveal the new badges for this season of Super Users! Congratulations to all the new and returning Super Users!     To better answer the question "What is a Super User?" please check out this article: Power Apps: What is A Super User? - Power Platform CommunityPower Automate: What is A Super User? - Power Platform Community Copilot Studio: What is A Super User? - Power Platform Community Power Pages: What is A Super User? - Power Platform Community

Microsoft Power Platform | 2024 Release Wave 1 Plan

Check out the latest Microsoft Power Platform release plans for 2024!   We have a whole host of exciting new features to help you be more productive, enhance delegation, run automated testing, build responsive pages, and so much more.    Click the links below to see not only our forthcoming releases, but to also try out some of the new features that have recently been released to market across:     Power Apps  Power Automate  Copilot Studio   We can’t wait to share with you all the upcoming releases that will help take your Power Platform experience to the next level!    Check out the entire Release Wave: Power Platform Complete Release Planner 

Users online (3,929)