cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
jeckard
Helper I
Helper I

Remove Duplicates from a Sharepoint List

I have an issue where duplicates are getting made in a Sharepoint List and I would like Flow to on a schedule get all the items in a list and then for each duplicate   The Key to finding the duplicates would be Employee ID so Find items with the same EEID.

then for every duplicate, it finds send me an email

I am open to using REST if someone can help me with how to accomplish this.

Thanks

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
jeckard
Helper I
Helper I

Here is what I ended up doing and it works.  It would be nice to get all the ID in an array and then delete them all but the first one...That will be the next thing.

Duplicate1.PNGDuplicate2.PNGDuplicate3.PNG

View solution in original post

16 REPLIES 16
Jay-Encodian
Community Champion
Community Champion

Hey @jeckard 

I'd suggest setting the column to enforce unique values to prevent the addition of duplicates:

1.png

HTH

Jay

Hey, Jay, I think you meant to upload an image but nothing came through??

Hey @jeckard, sorry try now?

Pstork1
Most Valuable Professional
Most Valuable Professional

If you just want to scan for duplicates and send an email for each duplicate then you can just do two Apply to Each loops, with one inside the other.  Before the first loop do a get items and inside the first loop do a get items again.  Then in the second loop compare the value of the record from the first loop with the record from the second.  If its a match then send an email.  I used a list that has a Project choice field, just substitute the Employeeid field.

screenshot.png



-------------------------------------------------------------------------
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.

I will give you solution a try tonight and let you know!

 

 I tried the solution you suggested I believe I might be setting up the loops incorrectly

Can you clarify Value one and Value two as I am getting the system creating a  apply to each loop three?

 

Loop2.png

v-alzhan-msft
Community Support
Community Support

Hi @jeckard ,

 

Please refer to screenshot below to create the flow:

1.png

 

Best regards,

Alice       

Community Support Team _ Alice Zhang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

@jkeckard007 

I think you'll need @Pstork1 to provide some more guidance here... but I think you'll need to do the following:

1. Delete the 'Apply to each 2' and 'Apply to each 3' (and contained actions)

2. Re-add a 'Condition' action directly underneath the second 'Get Items' action

3. Select the 'Employee ID' value from the 'Get Items 1 and compare to the the 'Employee ID' value from the 'Get Items' 2... that should set the loop up correctly.

This will find duplicates but its a pretty expensive query, and it won't work for large lists (above 5000) even after enabling pagination... Working with @Pstork1 excellent suggestion I updated to the following but it still suffers the same limitation with the original query... the second query is slightly quicker as it will return a smaller result set and only do something if the array is greater than 1... i..e a duplicate exists.

1.png

  This is still not really correct and needs more logic to be added a single duplicate would result in 2 separate emails (One for each duplicate item)... if your list contains less than 5000 items this may work, but I'd really suggest disabling duplicates on the list as per my original suggestion.

If you're list is above 5000 items I think will need a direct REST query using a CAML query to perform a GroupBy type query (I know distinct isn't support by ODATA)...

HTH

Jay

 

Pstork1
Most Valuable Professional
Most Valuable Professional

If you compare your screenshot to mine you will see that I have two get items statements and two loops.  You have one get items statement and three loops.  From what I can see I assume your first get items isn't shown in the screenshot.  If so, then I don't see a need for Apply to each 2.  Remove that loop.  The outside loop should be the first Get items value and the inside loop should be the second.



-------------------------------------------------------------------------
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.
jeckard
Helper I
Helper I

Here is what I ended up doing and it works.  It would be nice to get all the ID in an array and then delete them all but the first one...That will be the next thing.

Duplicate1.PNGDuplicate2.PNGDuplicate3.PNG

Hi @jeckard ,

 

If your problem been solved, please go ahead and mark the post as solved by clicking “Accept as Solution” so that this thread will be marked for other users to easily identify!

 

Best regards,

Alice       

Community Support Team _ Alice Zhang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

There is no accept as a solution just reply and mark as new.

Hi @jkeckard007 ,

 

I will help you mark the post as answered.

 

Best regards,

Alice       

 

Community Support Team _ Alice Zhang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

I can't believe Microsoft doesn't have a simple array action called "removed duplicates" like Nintex had since 2007!!!

Can this be used if I want to update an item? 

 

For e.g. I have a flow from Survey123 to Sharepoint list. I want to be able to create items (this works all good) and then update the item if an answer has changed (update is based on the unique ID I have created that passes through from Survey123 to Sharepoint List). 

Currently it just keeps creating a new item. 

Cheers, 

Summer 

takolota
Multi Super User
Multi Super User

Helpful resources

Announcements

Check out the Copilot Studio Cookbook today!

We are excited to announce our new Copilot Cookbook Gallery in the Copilot Studio Community. We can't wait for you to share your expertise and your experience!    Join us for an amazing opportunity where you'll be one of the first to contribute to the Copilot Cookbook—your ultimate guide to mastering Microsoft Copilot. Whether you're seeking inspiration or grappling with a challenge while crafting apps, you probably already know that Copilot Cookbook is your reliable assistant, offering a wealth of tips and tricks at your fingertips--and we want you to add your expertise. What can you "cook" up?   Click this link to get started: https://aka.ms/CS_Copilot_Cookbook_Gallery   Don't miss out on this exclusive opportunity to be one of the first in the Community to share your app creation journey with Copilot. We'll be announcing a Cookbook Challenge very soon and want to make sure you one of the first "cooks" in the kitchen.   Don't miss your moment--start submitting in the Copilot Cookbook Gallery today!     Thank you,  Engagement Team

Tuesday Tip | How to Report Spam in Our Community

It's time for another TUESDAY TIPS, your weekly connection with the most insightful tips and tricks that empower both newcomers and veterans in the Power Platform Community! Every Tuesday, we bring you a curated selection of the finest advice, distilled from the resources and tools in the Community. Whether you’re a seasoned member or just getting started, Tuesday Tips are the perfect compass guiding you across the dynamic landscape of the Power Platform Community.   As our community family expands each week, we revisit our essential tools, tips, and tricks to ensure you’re well-versed in the community’s pulse. Keep an eye on the News & Announcements for your weekly Tuesday Tips—you never know what you may learn!   Today's Tip: How to Report Spam in Our Community We strive to maintain a professional and helpful community, and part of that effort involves keeping our platform free of spam. If you encounter a post that you believe is spam, please follow these steps to report it: Locate the Post: Find the post in question within the community.Kebab Menu: Click on the "Kebab" menu | 3 Dots, on the top right of the post.Report Inappropriate Content: Select "Report Inappropriate Content" from the menu.Submit Report: Fill out any necessary details on the form and submit your report.   Our community team will review the report and take appropriate action to ensure our community remains a valuable resource for everyone.   Thank you for helping us keep the community clean and useful!

Hear what's next for the Power Up Program

Hear from Principal Program Manager, Dimpi Gandhi, to discover the latest enhancements to the Microsoft #PowerUpProgram, including a new accelerated video-based curriculum crafted with the expertise of Microsoft MVPs, Rory Neary and Charlie Phipps-Bennett. If you’d like to hear what’s coming next, click the link below to sign up today! https://aka.ms/PowerUp  

Back to Basics Tuesday Tip #5: How to Become a Community Blog Author

This is the fifth post in our series dedicated to helping the amazing members of our community--both new members and seasoned veterans--learn and grow in how to best engage in the community! Each Tuesday, we feature new content that will help you best understand the community--from ranking and badges to profile avatars, from Super Users to blogging in the community. Our hope is that this information will help each of our community members grow in their experience with Power Platform, with the community, and with each other!   This Week's Topic: How to Become a Community Blog Author We want YOU to be part of the community blog! Sharing your knowledge of Power Platform is an essential part of our community! Here's why:   It helps create a vibrant and dynamic community of makers who can learn from each other’s experiences and insights.It provides a platform for collaboration and innovation, where members can share their ideas and work together to develop new solutions.It helps promote Power Platform by showcasing its capabilities and real-world applications.It helps build trust and credibility in the community because you are providing valuable information and resources your fellow community members can use to improve their skills and knowledge. By sharing your knowledge of Power Platform in the community blog, you help us create a more engaged and informed community, better equipped to tackle complex challenges. To get started with blogging across the Power Platform communities, please visit the following links:   Power Apps: https://powerusers.microsoft.com/t5/Power-Apps-Community-Blog/bg-p/PowerAppsBlog Power Automate: https://powerusers.microsoft.com/t5/Power-Automate-Community-Blog/bg-p/MPABlog Power Virtual Agents: https://powerusers.microsoft.com/t5/Power-Virtual-Agents-Community/bg-p/PVACommunityBlogPower Pages: https://powerusers.microsoft.com/t5/Power-Pages-Community-Blog/bg-p/mpp_blog   When you follow the link, look for a button like the one below on the right rail of your community blog, and let us know you're interested. We can't wait to connect with you and help you get started. Thanks for being part of our incredible community--and thanks for becoming part of the community blog!  

Back to Basics Tuesday Tip #4: What is a Super User?

This is the fourth post in our series dedicated to helping the amazing members of our community--both new members and seasoned veterans--learn and grow in how to best engage in the community! Each Tuesday, we feature new content that will help you best understand the community--from ranking and badges to profile avatars, from Super Users to blogging in the community. Our hope is that this information will help each of our community members grow in their experience with Power Platform, with the community, and with each other! This week's topic: Super Users   Have you ever been exploring the community and come across a user with this unique icon next to their name? It means you have found the actual, real-life superheroes of the Power Platform Community! Super Users are our heroes because of the way they are consistently helpful with everything from solutions to flagging spam, offering insight on the community, and so much more!If you've ever wondered exactly what a Super User does and how they earn that special badge, please check out this article: Power Apps: What is A Super User? - Power Platform CommunityPower Virtual Agents: What is A Super User? - Power Platform Community Power Automate: What is A Super User? - Power Platform CommunityPower Pages: What is A Super User? - Power Platform Community   Still interested in finding out more about how to become a Super User? Then it's time to think about "suiting up" with SUIT   Our new Super User in Training program launched at the recent Microsoft Power Platform Conference and the results have been incredible! With space for only 500 "new recruits," we are nearing capacity. It's not too late to find out more and get your Super User suit on, learning from the incredible current crop of community superheroes, our community team, and more!Find out about SUIT and sign up today!Power Apps: Are you ready to SUIT UP and become a Super User?Power Virtual Agents: Are you ready to SUIT UP and become a Super User? Power Automate: Are you ready to SUIT UP and become a Super User? Power Pages: Are you ready to SUIT UP and become a Super User? We are so thankful for our awesome Super Users and can't wait to see even more of you SUIT UP! Check back here next Tuesday for the next iteration of our "Back to Basics" series. #TuesdayTips  

Empowering Communities: Máté’s Journey as a Power Platform Leader

In the heart of Hungary, a community of tech enthusiasts is thriving under the guidance of Máté, the passionate leader of the Hungary Power Platform User Group. Máté embarked on his Power Platform journey as a citizen developer in 2018. By 2021, he had transitioned into a full-time Power Platform Developer, and in 2022, he ventured into freelancing as a Power Platform consultant and Microsoft Certified Trainer. His journey has been nothing short of inspirational, and his commitment to empowering others is reshaping the tech landscape in Hungary.   "I wanted to connect with like-minded individuals in my country. Breaking down language barriers for non-English speakers and creating networking opportunities has been and continues to be a significant goal for me," shared Máté, reflecting on the motivation behind leading the user group.   Máté’s initiative to lead the Hungary Power Platform User Group in 2022 proved transformative. The group, previously inactive, became part of the Global Power Platform Bootcamp initiative, marking a significant milestone. Inspired by Mark Smith's 90 Days Mentoring Challenge, Máté envisioned a platform where like-minded individuals could connect.   "My favorite part is bringing people together, regardless of their role or level of knowledge," said Máté, emphasizing his passion for community building. His advice for aspiring user group leaders echoed his encouraging nature, "Just give it a shot; it might feel a bit intimidating to put yourself out there publicly initially, but you'll quickly get the hang of it. Starting your user group is simple – a few clicks, and you're ready to roll."   For Máté, being part of a user group is more than a role; it's a commitment to fostering connections and facilitating knowledge sharing. His dedication is palpable as he discusses the impact of the group on his life. "Being part of a user group has made a significant difference in my life by expanding my network, enhancing my skills, and boosting my confidence," he stated. The past year was marked by notable achievements for the Hungary Power Platform User Group. Hungary's inclusion in the Global Power Platform Bootcamp map and a unique "Power Platform Picnic" meetup in Budapest were highlights, showcasing the group's growth and creativity.   Through his role as a user group leader, Máté has learned the transformative power of community. "From being a user group leader, I've learned the value of community and the incredible impact it can have on growth," he shared. His journey stands as a testament to the incredible impact a dedicated leader can have in empowering and nurturing a community. As the Hungary Power Platform User Group continues to grow under Máté's leadership, the future of tech in Hungary looks promising, with a community that thrives on collaboration, learning, and shared enthusiasm for the Power Platform.  

Users online (4,319)