cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
StretchFredrik
Multi Super User
Multi Super User

FIlter array by both path and modified date

Hello!

 

I am wondering how to format my filter array step to get the following:

 

I got 2 arrays that are the files from 2 different libraries that are supposed to be "synced" (The same files in both libraries) where one of the libraries are considered the "source" and the other library are just copies.

 

Now i dont want to loop every item just to check if its already "synced" or not, which is why im trying to do it in a filter array step.

 

Basically i want to filter out any files that have the same URL and where the modified date is less in the source than in the "mirrored library" (This means that it's already the correct version in the mirrored library).

 

Any help is very much appreciated. 😃

 

Maybe you can guide me in the right direction? @Expiscornovus 

1 ACCEPTED SOLUTION

Accepted Solutions

@StretchFredrik Hopefully this works as expected. It should copy over update files and newly added files. I combined what @Chriddle did with the solution as a lot nicer and easier to handle the new files added.

 

See full flow below. I'll go into each of the actions.

grantjenkins_0-1677760541190.png

 

Get files Library A and Get files Library B are the same as the original solution.

grantjenkins_1-1677760645339.png

 

Select B extracts just the Modified date and the Full Path (excluding the Library Name) from Get files Library B. The expression used is:

join(skip(split(item()?['{FullPath}'], '/'), 1), '/')

grantjenkins_2-1677760792459.png

 

Select A extracts out the Identifier, Modified date and Path from Get files Library A, plus the matching Modified date from Library B. The expression used to get the Modified date from Library B is:

xpath(
    xml(json(concat('{"root": { value:', body('Select_B'), '}}'))),
    concat('string(//root/value[FullPath="', join(skip(split(item()?['{FullPath}'], '/'), 1), '/'), '"]/Modified/text())')
)

grantjenkins_3-1677760888349.png

 

Filter array uses the output from Select A with the following filter.

//ModifiedB is empty (new file added) or ModifiedA is greater than ModifiedB (file updated in Library A)

@or(
    equals(item()?['ModifiedB'], ''),
    greater(item()?['ModifiedA'], item()?['ModifiedB'])
)

grantjenkins_4-1677761008930.png

 

Apply to each iterates over each of the items in our Filter array.

grantjenkins_5-1677761044855.png

 

Copy file uses the following expressions to copy the new/updated files from Library A to Library B.

//File to Copy
item()?['Identifier']

//Destination Folder - NOTE that you would need to put your Library names here
slice(replace(item()?['Path'], 'LibraryA', 'LibraryB'), 0, lastIndexOf(replace(item()?['Path'], 'LibraryA', 'LibraryB'), '/'))

grantjenkins_6-1677761158596.png


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.

View solution in original post

21 REPLIES 21
grantjenkins
Super User
Super User

I've got an idea on how to do this within a single Filter array, but just off to sleep now so won't be able to get you something until later (12:30AM for me at the moment). If someone else can provide a solution prior to that then even better 🙂

 

Are they both Document Libraries that would have identical folder structures/files?

 

Assuming this would be a scheduled flow that ran daily/weekly?


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.
Expiscornovus
Most Valuable Professional
Most Valuable Professional

Hi @StretchFredrik,

 

Normally I would say, have a look at the Except method described in this blog:

https://pnp.github.io/blog/post/comparing-two-arrays-without-an-apply-to-each/

 

However, you want to check two things (Url and Modified date time).

 

Only workaround I can think of at the moment is getting like the max modified date as the latest sync time. Not a great workaround because it doesn't compare it with the modified of the target item itself, but just with the max modified date time of the whole collection of items.

 

But because it is a sync process, that might be ok? 😁

 

 

@and(contains(body('Select_-_Target_Paths'), item()['File']), greater(ticks(item()['Modified']), max(body('Select_-_Target_Modified'))))

 

 

maxmodifieddate.png



Happy to help out! 🙂

Interested in more #PowerAutomate #SharePointOnline or #MicrosoftCopilotStudio content?
Visit my blog, Subscribe to my YouTube channel or Follow me on Twitter


StretchFredrik
Multi Super User
Multi Super User

Thank you for your reply @Expiscornovus , i would need to compare each items modified date since the sync might take different documents each time since they will be approved or worked on at different times. So a file might not be touched for a year meanwhile another document is worked on daily. The library in question has around 50 000 documents. 

StretchFredrik
Multi Super User
Multi Super User

Thank you for your reply @grantjenkins 

 

Are they both Document Libraries that would have identical folder structures/files?

 

Yes they have the same folder structure and files.

 

Assuming this would be a scheduled flow that ran daily/weekly?

 

It will run every 15 or 30 minutes, which is why i want it to only spend time on the documents that are out of sync. 

Expiscornovus
Most Valuable Professional
Most Valuable Professional

Ok, thanks for clarifying. In that case what I provided isn't sufficient.

 

Let's wait for the response of @grantjenkins. I am sure he can come up with a great solution 😀



Happy to help out! 🙂

Interested in more #PowerAutomate #SharePointOnline or #MicrosoftCopilotStudio content?
Visit my blog, Subscribe to my YouTube channel or Follow me on Twitter


Chriddle
Super User
Super User

Have you considered using Sharepoint (and maybe OneDrive) sync capabilities instead of a Power Automate flow?

StretchFredrik
Multi Super User
Multi Super User

Yes, that does not work since the main goal of this is to have one library with ONLY major versions of files and only read permissions @Chriddle

StretchFredrik
Multi Super User
Multi Super User

Yes, that does not work since the main goal of this is to have one library with ONLY major versions of files and only read permissions @Chriddle 

Chriddle
Super User
Super User

Maybe that helps:

I created an array "source" with 500 objects (I hope this is a resonable number of changed files between two flow runs)
and an array "destination" with 50000 objects.
Maybe the amount of destination objects can be reduced by a clever odata filter.

For each object in "source", the Select action "combined" does a lookup in the destinations for an entry with same name and get its "created" with the help of xpath.

You can filter this output with date comparison,

 

With this amount of values this flow runs in round about 2 minutes.

 

Of course this is only a POC and you would have to add times, check what happens if the objects are bigger (because of longer names) and probably more 😉

 

Chriddle_0-1677672392384.png

source (Select):

 

"inputs": {
        "from": "@range(0,500)",
        "select": {
            "name": "@concat('file-',string(item()))",
            "created": "@concat('2023-02-', rand(1, 28))"
        }
    }

 

 

destination (Select):

 

"inputs": {
        "from": "@range(0, 50000)",
        "select": {
            "name": "@concat('file-', string(item()))",
            "created": "@concat('2023-02-', rand(1, 28))"
        }
    }

 

 

destinationXML (Compose):

 

"inputs": "@xml(json(concat('{\"root\":{\"item\":', body('destination'),'}}')))"

 

 

combined (Select):

 

"inputs": {
        "from": "@body('source')",
        "select": {
            "name": "@item()['name']",
            "created": "@item()['created']",
            "created_destination": "@first(xpath(outputs('destinationXML'), concat('//item[name=\"',item()['name'],'\"]/created/text()')))"
        }
    }

 

 

Filter array:

 

"inputs": {
        "from": "@body('combined')",
        "where": "@greater(item()['created'], item()['created_destination'])"
    }

 

 

I think I might have something that will work.

 

One concern is the number of files in each library (50,000). We can't just apply a Filter Query within our Get files, so would need to return all files from both libraries (100,000+ files) then apply filtering. The filtering will be quick - just the initial retrieval of files.

 

The other concern is if there are a lot of files out of sync it will take some time to copy them across which could take quite a while depending on number of files and size of those files. One thing I haven't done here is copied the actual metadata (properties) across. Can easily do that, but not sure if your requirement is just the file sync, or properties too.

 

If you go with this approach, you may need to run the flow manually a few times to see how long it takes to complete, then schedule the flow accordingly.

 

See full flow below. I'll go into each of the actions.

grantjenkins_6-1677673275523.png

 

Get files Library A and Get files Library B are both using Get files (properties only) actions. They both have the filter FSObjType eq 0 which means only get files (not folders). I've also set the Top Count to 5000 for each of them.

grantjenkins_2-1677672438361.png

 

I've also gone into the Settings for Get files Library A and Get files Library B, turned on Pagination, and set the Threshold to 60000 (needs to be a number larger than the number of files you will have over the next couple of years at least). This will take a while to retrieve all your files.

grantjenkins_3-1677672501618.png

 

grantjenkins_4-1677672540888.png

 

Select extracts out a couple of properties from Get files Library B that we will convert to XML so we can apply XPath within the filter later. The expressions used are:

//FullPath - removes the library name from the full path
join(skip(split(item()?['{FullPath}'], '/'), 1), '/')

//Modified - replaces characters so we are left with a number (required for XPath comparison)
replace(replace(replace(replace(item()?['Modified'], '-', ''), 'T', ''), ':', ''), 'Z', '')

grantjenkins_5-1677673047373.png

 

Filter array uses the output from our Select and the following expression to filter our items that have been updated in Library A since being copied to Library B (in need of updating). It uses an XPath expression to compare both the FullPath (including Filename) and the Modified Date. And if the length of items returned is greater than 0 then we need to update the item.

@greater(
    length(
        xpath(
            xml(json(concat('{"root": { value:', body('Select'), '}}'))), 
            concat('//root/value[FullPath = "', join(skip(split(item()?['{FullPath}'], '/'), 1), '/'), '" and Modified < "', replace(replace(replace(replace(item()?['Modified'], '-', ''), 'T', ''), ':', ''), 'Z', ''), '"]')
        )
    ), 
    0
)

grantjenkins_7-1677673609019.png

 

Apply to each iterates over each of the items in our Filter array (files that need to be updated).

grantjenkins_8-1677673662844.png

 

Copy file uses the following expressions for File to Copy and Destination folder.

//File to Copy
item()?['{Identifier}']

//Destination Folder - NOTE that you would need to put your Library names here
slice(replace(item()?['{Path}'], 'LibraryA', 'LibraryB'), 0, lastIndexOf(replace(item()?['{Path}'], 'LibraryA', 'LibraryB'), '/'))

grantjenkins_9-1677673768050.png


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.
StretchFredrik
Multi Super User
Multi Super User

Thank you! I will give it a try and get back to you! @grantjenkins 

StretchFredrik
Multi Super User
Multi Super User

I think i got it close to working, will test a bit more tomorrow. But i would need the the filter array to also include any items that don't already exist in destination. @grantjenkins 

 

Thank you for your help this far 😃 

@StretchFredrik Yea I was thinking about that and already started a bit of a redesign, but off to sleep now, so won't be able to get anything to you until a bit later. I know how to achieve it, but just need to rebuild a part of it.

 

Also, what if you delete an item from the source - would you also need to delete that from the destination?


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.
StretchFredrik
Multi Super User
Multi Super User

Sounds awesome take your time and sleep well. the delete part ive already done by comparing urls with the filter array step. It then loops the ones present in destination and not in source and tries to find the file by document-id to see if its moved in source but not published yet. If it doesnt find it by document-id, it gets deleted.

 

Thank you for your time @grantjenkins 

@StretchFredrik Hopefully this works as expected. It should copy over update files and newly added files. I combined what @Chriddle did with the solution as a lot nicer and easier to handle the new files added.

 

See full flow below. I'll go into each of the actions.

grantjenkins_0-1677760541190.png

 

Get files Library A and Get files Library B are the same as the original solution.

grantjenkins_1-1677760645339.png

 

Select B extracts just the Modified date and the Full Path (excluding the Library Name) from Get files Library B. The expression used is:

join(skip(split(item()?['{FullPath}'], '/'), 1), '/')

grantjenkins_2-1677760792459.png

 

Select A extracts out the Identifier, Modified date and Path from Get files Library A, plus the matching Modified date from Library B. The expression used to get the Modified date from Library B is:

xpath(
    xml(json(concat('{"root": { value:', body('Select_B'), '}}'))),
    concat('string(//root/value[FullPath="', join(skip(split(item()?['{FullPath}'], '/'), 1), '/'), '"]/Modified/text())')
)

grantjenkins_3-1677760888349.png

 

Filter array uses the output from Select A with the following filter.

//ModifiedB is empty (new file added) or ModifiedA is greater than ModifiedB (file updated in Library A)

@or(
    equals(item()?['ModifiedB'], ''),
    greater(item()?['ModifiedA'], item()?['ModifiedB'])
)

grantjenkins_4-1677761008930.png

 

Apply to each iterates over each of the items in our Filter array.

grantjenkins_5-1677761044855.png

 

Copy file uses the following expressions to copy the new/updated files from Library A to Library B.

//File to Copy
item()?['Identifier']

//Destination Folder - NOTE that you would need to put your Library names here
slice(replace(item()?['Path'], 'LibraryA', 'LibraryB'), 0, lastIndexOf(replace(item()?['Path'], 'LibraryA', 'LibraryB'), '/'))

grantjenkins_6-1677761158596.png


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.
StretchFredrik
Multi Super User
Multi Super User

Thank you @grantjenkins , the filter array step fails on "Greater expects all of its paramteres to be either integer or decimal numbers, Found invalid parameter type "Null". So guessing the filter array query fails when the modifiedB is null(empty).

Did you put the filters in the same order that I had? If ModifiedB is empty, then it wouldn't try to evaluate the second filter.


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.
StretchFredrik
Multi Super User
Multi Super User

This is what i have, should be the same: 

 

StretchFredrik_0-1677762414617.png

 

StretchFredrik
Multi Super User
Multi Super User

Nevermind, im stupid, i forgot to change the input of filter array @grantjenkins . Will try again! 😃 

Helpful resources

Announcements

Exclusive LIVE Community Event: Power Apps Copilot Coffee Chat with Copilot Studio Product Team

It's time for the SECOND Power Apps Copilot Coffee Chat featuring the Copilot Studio product team, which will be held LIVE on April 3, 2024 at 9:30 AM Pacific Daylight Time (PDT).     This is an incredible opportunity to connect with members of the Copilot Studio product team and ask them anything about Copilot Studio. We'll share our special guests with you shortly--but we want to encourage to mark your calendars now because you will not want to miss the conversation.   This live event will give you the unique opportunity to learn more about Copilot Studio plans, where we’ll focus, and get insight into upcoming features. We’re looking forward to hearing from the community, so bring your questions!   TO GET ACCESS TO THIS EXCLUSIVE AMA: Kudo this post to reserve your spot! Reserve your spot now by kudoing this post.  Reservations will be prioritized on when your kudo for the post comes through, so don't wait! Click that "kudo button" today.   Invitations will be sent on April 2nd.Users posting Kudos after April 2nd at 9AM PDT may not receive an invitation but will be able to view the session online after conclusion of the event. Give your "kudo" today and mark your calendars for April 3, 2024 at 9:30 AM PDT and join us for an engaging and informative session!

Tuesday Tip: Unlocking Community Achievements and Earning Badges

TUESDAY TIPS are our way of communicating helpful things we've learned or shared that have helped members of the Community. Whether you're just getting started or you're a seasoned pro, Tuesday Tips will help you know where to go, what to look for, and navigate your way through the ever-growing--and ever-changing--world of the Power Platform Community! We cover basics about the Community, provide a few "insider tips" to make your experience even better, and share best practices gleaned from our most active community members and Super Users.   With so many new Community members joining us each week, we'll also review a few of our "best practices" so you know just "how" the Community works, so make sure to watch the News & Announcements each week for the latest and greatest Tuesday Tips!     THIS WEEK'S TIP: Unlocking Achievements and Earning BadgesAcross the Communities, you'll see badges on users profile that recognize and reward their engagement and contributions. These badges each signify a different achievement--and all of those achievements are available to any Community member! If you're a seasoned pro or just getting started, you too can earn badges for the great work you do. Check out some details on Community badges below--and find out more in the detailed link at the end of the article!       A Diverse Range of Badges to Collect The badges you can earn in the Community cover a wide array of activities, including: Kudos Received: Acknowledges the number of times a user’s post has been appreciated with a “Kudo.”Kudos Given: Highlights the user’s generosity in recognizing others’ contributions.Topics Created: Tracks the number of discussions initiated by a user.Solutions Provided: Celebrates the instances where a user’s response is marked as the correct solution.Reply: Counts the number of times a user has engaged with community discussions.Blog Contributor: Honors those who contribute valuable content and are invited to write for the community blog.       A Community Evolving Together Badges are not only a great way to recognize outstanding contributions of our amazing Community members--they are also a way to continue fostering a collaborative and supportive environment. As you continue to share your knowledge and assist each other these badges serve as a visual representation of your valuable contributions.   Find out more about badges in these Community Support pages in each Community: All About Community Badges - Power Apps CommunityAll About Community Badges - Power Automate CommunityAll About Community Badges - Copilot Studio CommunityAll About Community Badges - Power Pages Community

Tuesday Tips: Powering Up Your Community Profile

TUESDAY TIPS are our way of communicating helpful things we've learned or shared that have helped members of the Community. Whether you're just getting started or you're a seasoned pro, Tuesday Tips will help you know where to go, what to look for, and navigate your way through the ever-growing--and ever-changing--world of the Power Platform Community! We cover basics about the Community, provide a few "insider tips" to make your experience even better, and share best practices gleaned from our most active community members and Super Users.   With so many new Community members joining us each week, we'll also review a few of our "best practices" so you know just "how" the Community works, so make sure to watch the News & Announcements each week for the latest and greatest Tuesday Tips!   This Week's Tip: Power Up Your Profile!  🚀 It's where every Community member gets their start, and it's essential that you keep it updated! Your Community User Profile is how you're able to get messages, post solutions, ask questions--and as you rank up, it's where your badges will appear and how you'll be known when you start blogging in the Community Blog. Your Community User Profile is how the Community knows you--so it's essential that it works the way you need it to! From changing your username to updating contact information, this Knowledge Base Article is your best resource for powering up your profile.     Password Puzzles? No Problem! Find out how to sync your Azure AD password with your community account, ensuring a seamless sign-in. No separate passwords to remember! Job Jumps & Email Swaps Changed jobs? Got a new email? Fear not! You'll find out how to link your shiny new email to your existing community account, keeping your contributions and connections intact. Username Uncertainties Unraveled Picking the perfect username is crucial--and sometimes the original choice you signed up with doesn't fit as well as you may have thought. There's a quick way to request an update here--but remember, your username is your community identity, so choose wisely. "Need Admin Approval" Warning Window? If you see this error message while using the community, don't worry. A simple process will help you get where you need to go. If you still need assistance, find out how to contact your Community Support team. Whatever you're looking for, when it comes to your profile, the Community Account Support Knowledge Base article is your treasure trove of tips as you navigate the nuances of your Community Profile. It’s the ultimate resource for keeping your digital identity in tip-top shape while engaging with the Power Platform Community. So, dive in and power up your profile today!  💪🚀   Community Account Support | Power Apps Community Account Support | Power AutomateCommunity Account Support | Copilot Studio  Community Account Support | Power Pages

Super User of the Month | Chris Piasecki

In our 2nd installment of this new ongoing feature in the Community, we're thrilled to announce that Chris Piasecki is our Super User of the Month for March 2024. If you've been in the Community for a while, we're sure you've seen a comment or marked one of Chris' helpful tips as a solution--he's been a Super User for SEVEN consecutive seasons!   Since authoring his first reply in April 2020 to his most recent achievement organizing the Canadian Power Platform Summit this month, Chris has helped countless Community members with his insights and expertise. In addition to being a Super User, Chris is also a User Group leader, Microsoft MVP, and a featured speaker at the Microsoft Power Platform Conference. His contributions to the new SUIT program, along with his joyous personality and willingness to jump in and help so many members has made Chris a fixture in the Power Platform Community.   When Chris isn't authoring solutions or organizing events, he's actively leading Piasecki Consulting, specializing in solution architecture, integration, DevOps, and more--helping clients discover how to strategize and implement Microsoft's technology platforms. We are grateful for Chris' insightful help in the Community and look forward to even more amazing milestones as he continues to assist so many with his great tips, solutions--always with a smile and a great sense of humor.You can find Chris in the Community and on LinkedIn. Thanks for being such a SUPER user, Chris! 💪 🌠  

Find Out What Makes Super Users So Super

We know many of you visit the Power Platform Communities to ask questions and receive answers. But do you know that many of our best answers and solutions come from Community members who are super active, helping anyone who needs a little help getting unstuck with Business Applications products? We call these dedicated Community members Super Users because they are the real heroes in the Community, willing to jump in whenever they can to help! Maybe you've encountered them yourself and they've solved some of your biggest questions. Have you ever wondered, "Why?"We interviewed several of our Super Users to understand what drives them to help in the Community--and discover the difference it has made in their lives as well! Take a look in our gallery today: What Motivates a Super User? - Power Platform Community (microsoft.com)

March User Group Update: New Groups and Upcoming Events!

  Welcome to this month’s celebration of our Community User Groups and exciting User Group events. We’re thrilled to introduce some brand-new user groups that have recently joined our vibrant community. Plus, we’ve got a lineup of engaging events you won’t want to miss. Let’s jump right in: New User Groups   Sacramento Power Platform GroupANZ Power Platform COE User GroupPower Platform MongoliaPower Platform User Group OmanPower Platform User Group Delta StateMid Michigan Power Platform Upcoming Events  DUG4MFG - Quarterly Meetup - Microsoft Demand PlanningDate: 19 Mar 2024 | 10:30 AM to 12:30 PM Central America Standard TimeDescription: Dive into the world of manufacturing with a focus on Demand Planning. Learn from industry experts and share your insights. Dynamics User Group HoustonDate: 07 Mar 2024 | 11:00 AM to 01:00 PM Central America Standard TimeDescription: Houston, get ready for an immersive session on Dynamics 365 and the Power Platform. Connect with fellow professionals and expand your knowledge. Reading Dynamics 365 & Power Platform User Group (Q1)Date: 05 Mar 2024 | 06:00 PM to 09:00 PM GMT Standard TimeDescription: Join our virtual meetup for insightful discussions, demos, and community updates. Let’s kick off Q1 with a bang! Leaders, Create Your Events!  Leaders of existing User Groups, don’t forget to create your events within the Community platform. By doing so, you’ll enable us to share them in future posts and newsletters. Let’s spread the word and make these gatherings even more impactful! Stay tuned for more updates, inspiring stories, and collaborative opportunities from and for our Community User Groups.   P.S. Have an event or success story to share? Reach out to us – we’d love to feature you!

Users online (5,971)