cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
jeckard
Helper I
Helper I

Remove Duplicates from a Sharepoint List

I have an issue where duplicates are getting made in a Sharepoint List and I would like Flow to on a schedule get all the items in a list and then for each duplicate   The Key to finding the duplicates would be Employee ID so Find items with the same EEID.

then for every duplicate, it finds send me an email

I am open to using REST if someone can help me with how to accomplish this.

Thanks

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
jeckard
Helper I
Helper I

Here is what I ended up doing and it works.  It would be nice to get all the ID in an array and then delete them all but the first one...That will be the next thing.

Duplicate1.PNGDuplicate2.PNGDuplicate3.PNG

View solution in original post

16 REPLIES 16
Jay-Encodian
Community Champion
Community Champion

Hey @jeckard 

I'd suggest setting the column to enforce unique values to prevent the addition of duplicates:

1.png

HTH

Jay

Hey, Jay, I think you meant to upload an image but nothing came through??

Hey @jeckard, sorry try now?

Pstork1
Most Valuable Professional
Most Valuable Professional

If you just want to scan for duplicates and send an email for each duplicate then you can just do two Apply to Each loops, with one inside the other.  Before the first loop do a get items and inside the first loop do a get items again.  Then in the second loop compare the value of the record from the first loop with the record from the second.  If its a match then send an email.  I used a list that has a Project choice field, just substitute the Employeeid field.

screenshot.png



-------------------------------------------------------------------------
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.

I will give you solution a try tonight and let you know!

 

 I tried the solution you suggested I believe I might be setting up the loops incorrectly

Can you clarify Value one and Value two as I am getting the system creating a  apply to each loop three?

 

Loop2.png

v-alzhan-msft
Community Support
Community Support

Hi @jeckard ,

 

Please refer to screenshot below to create the flow:

1.png

 

Best regards,

Alice       

Community Support Team _ Alice Zhang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

@jkeckard007 

I think you'll need @Pstork1 to provide some more guidance here... but I think you'll need to do the following:

1. Delete the 'Apply to each 2' and 'Apply to each 3' (and contained actions)

2. Re-add a 'Condition' action directly underneath the second 'Get Items' action

3. Select the 'Employee ID' value from the 'Get Items 1 and compare to the the 'Employee ID' value from the 'Get Items' 2... that should set the loop up correctly.

This will find duplicates but its a pretty expensive query, and it won't work for large lists (above 5000) even after enabling pagination... Working with @Pstork1 excellent suggestion I updated to the following but it still suffers the same limitation with the original query... the second query is slightly quicker as it will return a smaller result set and only do something if the array is greater than 1... i..e a duplicate exists.

1.png

  This is still not really correct and needs more logic to be added a single duplicate would result in 2 separate emails (One for each duplicate item)... if your list contains less than 5000 items this may work, but I'd really suggest disabling duplicates on the list as per my original suggestion.

If you're list is above 5000 items I think will need a direct REST query using a CAML query to perform a GroupBy type query (I know distinct isn't support by ODATA)...

HTH

Jay

 

Pstork1
Most Valuable Professional
Most Valuable Professional

If you compare your screenshot to mine you will see that I have two get items statements and two loops.  You have one get items statement and three loops.  From what I can see I assume your first get items isn't shown in the screenshot.  If so, then I don't see a need for Apply to each 2.  Remove that loop.  The outside loop should be the first Get items value and the inside loop should be the second.



-------------------------------------------------------------------------
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.
jeckard
Helper I
Helper I

Here is what I ended up doing and it works.  It would be nice to get all the ID in an array and then delete them all but the first one...That will be the next thing.

Duplicate1.PNGDuplicate2.PNGDuplicate3.PNG

Hi @jeckard ,

 

If your problem been solved, please go ahead and mark the post as solved by clicking “Accept as Solution” so that this thread will be marked for other users to easily identify!

 

Best regards,

Alice       

Community Support Team _ Alice Zhang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

There is no accept as a solution just reply and mark as new.

Hi @jkeckard007 ,

 

I will help you mark the post as answered.

 

Best regards,

Alice       

 

Community Support Team _ Alice Zhang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

I can't believe Microsoft doesn't have a simple array action called "removed duplicates" like Nintex had since 2007!!!

Can this be used if I want to update an item? 

 

For e.g. I have a flow from Survey123 to Sharepoint list. I want to be able to create items (this works all good) and then update the item if an answer has changed (update is based on the unique ID I have created that passes through from Survey123 to Sharepoint List). 

Currently it just keeps creating a new item. 

Cheers, 

Summer 

takolota
Multi Super User
Multi Super User

Helpful resources

Announcements

Check out the Copilot Studio Cookbook today!

We are excited to announce our new Copilot Cookbook Gallery in the Copilot Studio Community. We can't wait for you to share your expertise and your experience!    Join us for an amazing opportunity where you'll be one of the first to contribute to the Copilot Cookbook—your ultimate guide to mastering Microsoft Copilot. Whether you're seeking inspiration or grappling with a challenge while crafting apps, you probably already know that Copilot Cookbook is your reliable assistant, offering a wealth of tips and tricks at your fingertips--and we want you to add your expertise. What can you "cook" up?   Click this link to get started: https://aka.ms/CS_Copilot_Cookbook_Gallery   Don't miss out on this exclusive opportunity to be one of the first in the Community to share your app creation journey with Copilot. We'll be announcing a Cookbook Challenge very soon and want to make sure you one of the first "cooks" in the kitchen.   Don't miss your moment--start submitting in the Copilot Cookbook Gallery today!     Thank you,  Engagement Team

Tuesday Tip | How to Report Spam in Our Community

It's time for another TUESDAY TIPS, your weekly connection with the most insightful tips and tricks that empower both newcomers and veterans in the Power Platform Community! Every Tuesday, we bring you a curated selection of the finest advice, distilled from the resources and tools in the Community. Whether you’re a seasoned member or just getting started, Tuesday Tips are the perfect compass guiding you across the dynamic landscape of the Power Platform Community.   As our community family expands each week, we revisit our essential tools, tips, and tricks to ensure you’re well-versed in the community’s pulse. Keep an eye on the News & Announcements for your weekly Tuesday Tips—you never know what you may learn!   Today's Tip: How to Report Spam in Our Community We strive to maintain a professional and helpful community, and part of that effort involves keeping our platform free of spam. If you encounter a post that you believe is spam, please follow these steps to report it: Locate the Post: Find the post in question within the community.Kebab Menu: Click on the "Kebab" menu | 3 Dots, on the top right of the post.Report Inappropriate Content: Select "Report Inappropriate Content" from the menu.Submit Report: Fill out any necessary details on the form and submit your report.   Our community team will review the report and take appropriate action to ensure our community remains a valuable resource for everyone.   Thank you for helping us keep the community clean and useful!

Tuesday Tip | How to Get Community Support

It's time for another Tuesday Tip, your weekly connection with the most insightful tips and tricks that empower both newcomers and veterans in the Power Platform Community! Every Tuesday, we bring you a curated selection of the finest advice, distilled from the resources and tools in the Community. Whether you’re a seasoned member or just getting started, Tuesday Tips are the perfect compass guiding you across the dynamic landscape of the Power Platform Community.       This Week: All About Community Support Whether you're a seasoned community veteran or just getting started, you may need a bit of help from time to time! If you need to share feedback with the Community Engagement team about the community or are looking for ways we can assist you with user groups, events, or something else, Community Support is the place to start.   Community Support is part of every one of our communities, accessible to all our community members.   Within each community's Community Support page, you'll find three distinct areas, each with a different focus to help you when you need support from us most. Power Apps: https://powerusers.microsoft.com/t5/Community-Support/ct-p/pa_community_support Power Automate: https://powerusers.microsoft.com/t5/Community-Support/ct-p/mpa_community_support Power Pages: https://powerusers.microsoft.com/t5/Community-Support/ct-p/mpp_community_support Copilot Studio: https://powerusers.microsoft.com/t5/Community-Support/ct-p/pva_community-support   Community Support Form If you need more assistance, you can reach out to the Community Team via the Community support form. Choose the type of support you require and fill in the form accordingly. We will respond to you promptly.    Thank you for being an active part of our community. Your contributions make a difference!   Best Regards, The Community Management Team

Community Roundup: A Look Back at Our Last 10 Tuesday Tips

As we continue to grow and learn together, it's important to reflect on the valuable insights we've shared. For today's #TuesdayTip, we're excited to take a moment to look back at the last 10 tips we've shared in case you missed any or want to revisit them. Thanks for your incredible support for this series--we're so glad it was able to help so many of you navigate your community experience!   Getting Started in the Community An overview of everything you need to know about navigating the community on one page!  Community Links: ○ Power Apps ○ Power Automate  ○ Power Pages  ○ Copilot Studio    Community Ranks and YOU Have you ever wondered how your fellow community members ascend the ranks within our community? We explain everything about ranks and how to achieve points so you can climb up in the rankings! Community Links: ○ Power Apps ○ Power Automate  ○ Power Pages  ○ Copilot Studio    Powering Up Your Community Profile Your Community User Profile is how the Community knows you--so it's essential that it works the way you need it to! From changing your username to updating contact information, this Knowledge Base Article is your best resource for powering up your profile. Community Links: ○ Power Apps ○ Power Automate  ○ Power Pages  ○ Copilot Studio    Community Blogs--A Great Place to Start There's so much you'll discover in the Community Blogs, and we hope you'll check them out today!  Community Links: ○ Power Apps ○ Power Automate  ○ Power Pages  ○ Copilot Studio    Unlocking Community Achievements and Earning Badges Across the Communities, you'll see badges on users profile that recognize and reward their engagement and contributions. Check out some details on Community badges--and find out more in the detailed link at the end of the article! Community Links: ○ Power Apps  ○ Power Automate  ○ Power Pages  ○ Copilot Studio    Blogging in the Community Interested in blogging? Everything you need to know on writing blogs in our four communities! Get started blogging across the Power Platform communities today! Community Links: ○ Power Apps  ○ Power Automate  ○ Power Pages  ○ Copilot Studio   Subscriptions & Notifications We don't want you to miss a thing in the community! Read all about how to subscribe to sections of our forums and how to setup your notifications! Community Links: ○ Power Apps  ○ Power Automate  ○ Power Pages  ○ Copilot Studio   Getting Started with Private Messages & Macros Do you want to enhance your communication in the Community and streamline your interactions? One of the best ways to do this is to ensure you are using Private Messaging--and the ever-handy macros that are available to you as a Community member! Community Links: ○ Power Apps  ○ Power Automate  ○ Power Pages  ○ Copilot Studio   Community User Groups Learn everything about being part of, starting, or leading a User Group in the Power Platform Community. Community Links: ○ Power Apps  ○ Power Automate  ○ Power Pages  ○ Copilot Studio   Update Your Community Profile Today! Keep your community profile up to date which is essential for staying connected and engaged with the community. Community Links: ○ Power Apps  ○ Power Automate  ○ Power Pages  ○ Copilot Studio   Thank you for being an integral part of our journey.   Here's to many more Tuesday Tips as we pave the way for a brighter, more connected future! As always, watch the News & Announcements for the next set of tips, coming soon!    

Hear what's next for the Power Up Program

Hear from Principal Program Manager, Dimpi Gandhi, to discover the latest enhancements to the Microsoft #PowerUpProgram, including a new accelerated video-based curriculum crafted with the expertise of Microsoft MVPs, Rory Neary and Charlie Phipps-Bennett. If you’d like to hear what’s coming next, click the link below to sign up today! https://aka.ms/PowerUp  

Monthly Community User Group Update | April 2024

The monthly Community User Group Update is your resource for discovering User Group meetings and events happening around the world (and virtually), welcoming new User Groups to our Community, and more! Our amazing Community User Groups are an important part of the Power Platform Community, with more than 700 Community User Groups worldwide, we know they're a great way to engage personally, while giving our members a place to learn and grow together.   This month, we welcome 3 new User Groups in India, Wales, and Germany, and feature 8 User Group Events across Power Platform and Dynamics 365. Find out more below. New Power Platform User Groups   Power Platform Innovators (India) About: Our aim is to foster a collaborative environment where we can share upcoming Power Platform events, best practices, and valuable content related to Power Platform. Whether you’re a seasoned expert or a newcomer looking to learn, this group is for you. Let’s empower each other to achieve more with Power Platform. Join us in shaping the future of digital transformation!   Power Platform User Group (Wales) About: A Power Platform User Group in Wales (predominantly based in Cardiff but will look to hold sessions around Wales) to establish a community to share learnings and experience in all parts of the platform.   Power Platform User Group (Hannover) About: This group is for anyone who works with the services of Microsoft Power Platform or wants to learn more about it and no-code/low-code. And, of course, Microsoft Copilot application in the Power Platform.   New Dynamics365 User Groups   Ellucian CRM Recruit UK (United Kingdom) About: A group for United Kingdom universities using Ellucian CRM Recruit to manage their admissions process, to share good practice and resolve issues.    Business Central Mexico (Mexico City) About:  A place to find documentation, learning resources, and events focused on user needs in Mexico. We meet to discuss and answer questions about the current features in the standard localization that Microsoft provides, and what you only find in third-party locations. In addition, we focus on what's planned for new standard versions, recent legislation requirements, and more. Let's work together to drive request votes for Microsoft for features that aren't currently found—but are indispensable.   Dynamics 365 F&O User Group (Dublin) About: The Dynamics 365 F&O User Group - Ireland Chapter meets up in person at least twice yearly in One Microsoft Place Dublin for users to have the opportunity to have conversations on mutual topics, find out what’s new and on the Dynamics 365 FinOps Product Roadmap, get insights from customer and partner experiences, and access to Microsoft subject matter expertise.  Upcoming Power Platform Events    PAK Time (Power Apps Kwentuhan) 2024 #6 (Phillipines, Online) This is a continuation session of Custom API. Sir Jun Miano will be sharing firsthand experience on setting up custom API and best practices. (April 6, 2024)       Power Apps: Creating business applications rapidly (Sydney) At this event, learn how to choose the right app on Power Platform, creating a business application in an hour, and tips for using Copilot AI. While we recommend attending all 6 events in the series, each session is independent of one another, and you can join the topics of your interest. Think of it as a “Hop On, Hop Off” bus! Participation is free, but you need a personal computer (laptop) and we provide the rest. We look forward to seeing you there! (April 11, 2024)     April 2024 Cleveland Power Platform User Group (Independence, Ohio) Kickoff the meeting with networking, and then our speaker will share how to create responsive and intuitive Canvas Apps using features like Variables, Search and Filtering. And how PowerFx rich functions and expressions makes configuring those functionalities easier. Bring ideas to discuss and engage with other community members! (April 16, 2024)     Dynamics 365 and Power Platform 2024 Wave 1 Release (NYC, Online) This session features Aric Levin, Microsoft Business Applications MVP and Technical Architect at Avanade and Mihir Shah, Global CoC Leader of Microsoft Managed Services at IBM. We will cover some of the new features and enhancements related to the Power Platform, Dataverse, Maker Portal, Unified Interface and the Microsoft First Party Apps (Microsoft Dynamics 365) that were announced in the Microsoft Dynamics 365 and Power Platform 2024 Release Wave 1 Plan. (April 17, 2024)     Let’s Explore Copilot Studio Series: Bot Skills to Extend Your Copilots (Makati National Capital Reg... Join us for the second installment of our Let's Explore Copilot Studio Series, focusing on Bot Skills. Learn how to enhance your copilot's abilities to automate tasks within specific topics, from booking appointments to sending emails and managing tasks. Discover the power of Skills in expanding conversational capabilities. (April 30, 2024)   Upcoming Dynamics365 Events    Leveraging Customer Managed Keys (CMK) in Dynamics 365 (Noida, Uttar Pradesh, Online) This month's featured topic: Leveraging Customer Managed Keys (CMK) in Dynamics 365, with special guest Nitin Jain from Microsoft. We are excited and thankful to him for doing this session. Join us for this online session, which should be helpful to all Dynamics 365 developers, Technical Architects and Enterprise architects who are implementing Dynamics 365 and want to have more control on the security of their data over Microsoft Managed Keys. (April 11, 2024)       Stockholm D365 User Group April Meeting (Stockholm) This is a Swedish user group for D365 Finance and Operations, AX2012, CRM, CE, Project Operations, and Power BI.  (April 17, 2024)         Transportation Management in D365 F&SCM Q&A Session (Toronto, Online) Calling all Toronto UG members and beyond! Join us for an engaging and informative one-hour Q&A session, exclusively focused on Transportation Management System (TMS) within Dynamics 365 F&SCM. Whether you’re a seasoned professional or just curious about TMS, this event is for you. Bring your questions! (April 26, 2024)   Leaders, Create Your Events!    Leaders of existing User Groups, don’t forget to create your events within the Community platform. By doing so, you’ll enable us to share them in future posts and newsletters. Let’s spread the word and make these gatherings even more impactful! Stay tuned for more updates, inspiring stories, and collaborative opportunities from and for our Community User Groups.   P.S. Have an event or success story to share? Reach out to us – we’d love to feature you. Just leave a comment or send a PM here in the Community!

Users online (4,380)