cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
StretchFredrik
Multi Super User
Multi Super User

FIlter array by both path and modified date

Hello!

 

I am wondering how to format my filter array step to get the following:

 

I got 2 arrays that are the files from 2 different libraries that are supposed to be "synced" (The same files in both libraries) where one of the libraries are considered the "source" and the other library are just copies.

 

Now i dont want to loop every item just to check if its already "synced" or not, which is why im trying to do it in a filter array step.

 

Basically i want to filter out any files that have the same URL and where the modified date is less in the source than in the "mirrored library" (This means that it's already the correct version in the mirrored library).

 

Any help is very much appreciated. 😃

 

Maybe you can guide me in the right direction? @Expiscornovus 

1 ACCEPTED SOLUTION

Accepted Solutions

@StretchFredrik Hopefully this works as expected. It should copy over update files and newly added files. I combined what @Chriddle did with the solution as a lot nicer and easier to handle the new files added.

 

See full flow below. I'll go into each of the actions.

grantjenkins_0-1677760541190.png

 

Get files Library A and Get files Library B are the same as the original solution.

grantjenkins_1-1677760645339.png

 

Select B extracts just the Modified date and the Full Path (excluding the Library Name) from Get files Library B. The expression used is:

join(skip(split(item()?['{FullPath}'], '/'), 1), '/')

grantjenkins_2-1677760792459.png

 

Select A extracts out the Identifier, Modified date and Path from Get files Library A, plus the matching Modified date from Library B. The expression used to get the Modified date from Library B is:

xpath(
    xml(json(concat('{"root": { value:', body('Select_B'), '}}'))),
    concat('string(//root/value[FullPath="', join(skip(split(item()?['{FullPath}'], '/'), 1), '/'), '"]/Modified/text())')
)

grantjenkins_3-1677760888349.png

 

Filter array uses the output from Select A with the following filter.

//ModifiedB is empty (new file added) or ModifiedA is greater than ModifiedB (file updated in Library A)

@or(
    equals(item()?['ModifiedB'], ''),
    greater(item()?['ModifiedA'], item()?['ModifiedB'])
)

grantjenkins_4-1677761008930.png

 

Apply to each iterates over each of the items in our Filter array.

grantjenkins_5-1677761044855.png

 

Copy file uses the following expressions to copy the new/updated files from Library A to Library B.

//File to Copy
item()?['Identifier']

//Destination Folder - NOTE that you would need to put your Library names here
slice(replace(item()?['Path'], 'LibraryA', 'LibraryB'), 0, lastIndexOf(replace(item()?['Path'], 'LibraryA', 'LibraryB'), '/'))

grantjenkins_6-1677761158596.png


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.

View solution in original post

21 REPLIES 21
grantjenkins
Super User
Super User

I've got an idea on how to do this within a single Filter array, but just off to sleep now so won't be able to get you something until later (12:30AM for me at the moment). If someone else can provide a solution prior to that then even better 🙂

 

Are they both Document Libraries that would have identical folder structures/files?

 

Assuming this would be a scheduled flow that ran daily/weekly?


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.
Expiscornovus
Most Valuable Professional
Most Valuable Professional

Hi @StretchFredrik,

 

Normally I would say, have a look at the Except method described in this blog:

https://pnp.github.io/blog/post/comparing-two-arrays-without-an-apply-to-each/

 

However, you want to check two things (Url and Modified date time).

 

Only workaround I can think of at the moment is getting like the max modified date as the latest sync time. Not a great workaround because it doesn't compare it with the modified of the target item itself, but just with the max modified date time of the whole collection of items.

 

But because it is a sync process, that might be ok? 😁

 

 

@and(contains(body('Select_-_Target_Paths'), item()['File']), greater(ticks(item()['Modified']), max(body('Select_-_Target_Modified'))))

 

 

maxmodifieddate.png



Happy to help out! 🙂

Interested in more #PowerAutomate #SharePointOnline or #MicrosoftCopilotStudio content?
Visit my blog, Subscribe to my YouTube channel or Follow me on Twitter


StretchFredrik
Multi Super User
Multi Super User

Thank you for your reply @Expiscornovus , i would need to compare each items modified date since the sync might take different documents each time since they will be approved or worked on at different times. So a file might not be touched for a year meanwhile another document is worked on daily. The library in question has around 50 000 documents. 

StretchFredrik
Multi Super User
Multi Super User

Thank you for your reply @grantjenkins 

 

Are they both Document Libraries that would have identical folder structures/files?

 

Yes they have the same folder structure and files.

 

Assuming this would be a scheduled flow that ran daily/weekly?

 

It will run every 15 or 30 minutes, which is why i want it to only spend time on the documents that are out of sync. 

Expiscornovus
Most Valuable Professional
Most Valuable Professional

Ok, thanks for clarifying. In that case what I provided isn't sufficient.

 

Let's wait for the response of @grantjenkins. I am sure he can come up with a great solution 😀



Happy to help out! 🙂

Interested in more #PowerAutomate #SharePointOnline or #MicrosoftCopilotStudio content?
Visit my blog, Subscribe to my YouTube channel or Follow me on Twitter


Chriddle
Super User
Super User

Have you considered using Sharepoint (and maybe OneDrive) sync capabilities instead of a Power Automate flow?

StretchFredrik
Multi Super User
Multi Super User

Yes, that does not work since the main goal of this is to have one library with ONLY major versions of files and only read permissions @Chriddle

StretchFredrik
Multi Super User
Multi Super User

Yes, that does not work since the main goal of this is to have one library with ONLY major versions of files and only read permissions @Chriddle 

Chriddle
Super User
Super User

Maybe that helps:

I created an array "source" with 500 objects (I hope this is a resonable number of changed files between two flow runs)
and an array "destination" with 50000 objects.
Maybe the amount of destination objects can be reduced by a clever odata filter.

For each object in "source", the Select action "combined" does a lookup in the destinations for an entry with same name and get its "created" with the help of xpath.

You can filter this output with date comparison,

 

With this amount of values this flow runs in round about 2 minutes.

 

Of course this is only a POC and you would have to add times, check what happens if the objects are bigger (because of longer names) and probably more 😉

 

Chriddle_0-1677672392384.png

source (Select):

 

"inputs": {
        "from": "@range(0,500)",
        "select": {
            "name": "@concat('file-',string(item()))",
            "created": "@concat('2023-02-', rand(1, 28))"
        }
    }

 

 

destination (Select):

 

"inputs": {
        "from": "@range(0, 50000)",
        "select": {
            "name": "@concat('file-', string(item()))",
            "created": "@concat('2023-02-', rand(1, 28))"
        }
    }

 

 

destinationXML (Compose):

 

"inputs": "@xml(json(concat('{\"root\":{\"item\":', body('destination'),'}}')))"

 

 

combined (Select):

 

"inputs": {
        "from": "@body('source')",
        "select": {
            "name": "@item()['name']",
            "created": "@item()['created']",
            "created_destination": "@first(xpath(outputs('destinationXML'), concat('//item[name=\"',item()['name'],'\"]/created/text()')))"
        }
    }

 

 

Filter array:

 

"inputs": {
        "from": "@body('combined')",
        "where": "@greater(item()['created'], item()['created_destination'])"
    }

 

 

I think I might have something that will work.

 

One concern is the number of files in each library (50,000). We can't just apply a Filter Query within our Get files, so would need to return all files from both libraries (100,000+ files) then apply filtering. The filtering will be quick - just the initial retrieval of files.

 

The other concern is if there are a lot of files out of sync it will take some time to copy them across which could take quite a while depending on number of files and size of those files. One thing I haven't done here is copied the actual metadata (properties) across. Can easily do that, but not sure if your requirement is just the file sync, or properties too.

 

If you go with this approach, you may need to run the flow manually a few times to see how long it takes to complete, then schedule the flow accordingly.

 

See full flow below. I'll go into each of the actions.

grantjenkins_6-1677673275523.png

 

Get files Library A and Get files Library B are both using Get files (properties only) actions. They both have the filter FSObjType eq 0 which means only get files (not folders). I've also set the Top Count to 5000 for each of them.

grantjenkins_2-1677672438361.png

 

I've also gone into the Settings for Get files Library A and Get files Library B, turned on Pagination, and set the Threshold to 60000 (needs to be a number larger than the number of files you will have over the next couple of years at least). This will take a while to retrieve all your files.

grantjenkins_3-1677672501618.png

 

grantjenkins_4-1677672540888.png

 

Select extracts out a couple of properties from Get files Library B that we will convert to XML so we can apply XPath within the filter later. The expressions used are:

//FullPath - removes the library name from the full path
join(skip(split(item()?['{FullPath}'], '/'), 1), '/')

//Modified - replaces characters so we are left with a number (required for XPath comparison)
replace(replace(replace(replace(item()?['Modified'], '-', ''), 'T', ''), ':', ''), 'Z', '')

grantjenkins_5-1677673047373.png

 

Filter array uses the output from our Select and the following expression to filter our items that have been updated in Library A since being copied to Library B (in need of updating). It uses an XPath expression to compare both the FullPath (including Filename) and the Modified Date. And if the length of items returned is greater than 0 then we need to update the item.

@greater(
    length(
        xpath(
            xml(json(concat('{"root": { value:', body('Select'), '}}'))), 
            concat('//root/value[FullPath = "', join(skip(split(item()?['{FullPath}'], '/'), 1), '/'), '" and Modified < "', replace(replace(replace(replace(item()?['Modified'], '-', ''), 'T', ''), ':', ''), 'Z', ''), '"]')
        )
    ), 
    0
)

grantjenkins_7-1677673609019.png

 

Apply to each iterates over each of the items in our Filter array (files that need to be updated).

grantjenkins_8-1677673662844.png

 

Copy file uses the following expressions for File to Copy and Destination folder.

//File to Copy
item()?['{Identifier}']

//Destination Folder - NOTE that you would need to put your Library names here
slice(replace(item()?['{Path}'], 'LibraryA', 'LibraryB'), 0, lastIndexOf(replace(item()?['{Path}'], 'LibraryA', 'LibraryB'), '/'))

grantjenkins_9-1677673768050.png


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.
StretchFredrik
Multi Super User
Multi Super User

Thank you! I will give it a try and get back to you! @grantjenkins 

StretchFredrik
Multi Super User
Multi Super User

I think i got it close to working, will test a bit more tomorrow. But i would need the the filter array to also include any items that don't already exist in destination. @grantjenkins 

 

Thank you for your help this far 😃 

@StretchFredrik Yea I was thinking about that and already started a bit of a redesign, but off to sleep now, so won't be able to get anything to you until a bit later. I know how to achieve it, but just need to rebuild a part of it.

 

Also, what if you delete an item from the source - would you also need to delete that from the destination?


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.
StretchFredrik
Multi Super User
Multi Super User

Sounds awesome take your time and sleep well. the delete part ive already done by comparing urls with the filter array step. It then loops the ones present in destination and not in source and tries to find the file by document-id to see if its moved in source but not published yet. If it doesnt find it by document-id, it gets deleted.

 

Thank you for your time @grantjenkins 

@StretchFredrik Hopefully this works as expected. It should copy over update files and newly added files. I combined what @Chriddle did with the solution as a lot nicer and easier to handle the new files added.

 

See full flow below. I'll go into each of the actions.

grantjenkins_0-1677760541190.png

 

Get files Library A and Get files Library B are the same as the original solution.

grantjenkins_1-1677760645339.png

 

Select B extracts just the Modified date and the Full Path (excluding the Library Name) from Get files Library B. The expression used is:

join(skip(split(item()?['{FullPath}'], '/'), 1), '/')

grantjenkins_2-1677760792459.png

 

Select A extracts out the Identifier, Modified date and Path from Get files Library A, plus the matching Modified date from Library B. The expression used to get the Modified date from Library B is:

xpath(
    xml(json(concat('{"root": { value:', body('Select_B'), '}}'))),
    concat('string(//root/value[FullPath="', join(skip(split(item()?['{FullPath}'], '/'), 1), '/'), '"]/Modified/text())')
)

grantjenkins_3-1677760888349.png

 

Filter array uses the output from Select A with the following filter.

//ModifiedB is empty (new file added) or ModifiedA is greater than ModifiedB (file updated in Library A)

@or(
    equals(item()?['ModifiedB'], ''),
    greater(item()?['ModifiedA'], item()?['ModifiedB'])
)

grantjenkins_4-1677761008930.png

 

Apply to each iterates over each of the items in our Filter array.

grantjenkins_5-1677761044855.png

 

Copy file uses the following expressions to copy the new/updated files from Library A to Library B.

//File to Copy
item()?['Identifier']

//Destination Folder - NOTE that you would need to put your Library names here
slice(replace(item()?['Path'], 'LibraryA', 'LibraryB'), 0, lastIndexOf(replace(item()?['Path'], 'LibraryA', 'LibraryB'), '/'))

grantjenkins_6-1677761158596.png


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.
StretchFredrik
Multi Super User
Multi Super User

Thank you @grantjenkins , the filter array step fails on "Greater expects all of its paramteres to be either integer or decimal numbers, Found invalid parameter type "Null". So guessing the filter array query fails when the modifiedB is null(empty).

Did you put the filters in the same order that I had? If ModifiedB is empty, then it wouldn't try to evaluate the second filter.


----------------------------------------------------------------------
If I've answered your question, please mark the post as Solved.
If you like my response, please consider giving it a Thumbs Up.
StretchFredrik
Multi Super User
Multi Super User

This is what i have, should be the same: 

 

StretchFredrik_0-1677762414617.png

 

StretchFredrik
Multi Super User
Multi Super User

Nevermind, im stupid, i forgot to change the input of filter array @grantjenkins . Will try again! 😃 

Helpful resources

Announcements

Check out the Copilot Studio Cookbook today!

We are excited to announce our new Copilot Cookbook Gallery in the Copilot Studio Community. We can't wait for you to share your expertise and your experience!    Join us for an amazing opportunity where you'll be one of the first to contribute to the Copilot Cookbook—your ultimate guide to mastering Microsoft Copilot. Whether you're seeking inspiration or grappling with a challenge while crafting apps, you probably already know that Copilot Cookbook is your reliable assistant, offering a wealth of tips and tricks at your fingertips--and we want you to add your expertise. What can you "cook" up?   Click this link to get started: https://aka.ms/CS_Copilot_Cookbook_Gallery   Don't miss out on this exclusive opportunity to be one of the first in the Community to share your app creation journey with Copilot. We'll be announcing a Cookbook Challenge very soon and want to make sure you one of the first "cooks" in the kitchen.   Don't miss your moment--start submitting in the Copilot Cookbook Gallery today!     Thank you,  Engagement Team

Announcing Power Apps Copilot Cookbook Gallery

We are excited to share that the all-new Copilot Cookbook Gallery for Power Apps is now available in the Power Apps Community, full of tips and tricks on how to best use Microsoft Copilot as you develop and create in Power Apps. The new Copilot Cookbook is your go-to resource when you need inspiration--or when you're stuck--and aren't sure how to best partner with Copilot while creating apps.   Whether you're looking for the best prompts or just want to know about responsible AI use, visit Copilot Cookbook for regular updates you can rely on--while also serving up some of your greatest tips and tricks for the Community. Check Out the new Copilot Cookbook for Power Apps today: Copilot Cookbook - Power Platform Community.  We can't wait to see what you "cook" up!    

Welcome to the Power Automate Community

You are now a part of a fast-growing vibrant group of peers and industry experts who are here to network, share knowledge, and even have a little fun.   Now that you are a member, you can enjoy the following resources:   Welcome to the Community   News & Announcements: The is your place to get all the latest news around community events and announcements. This is where we share with the community what is going on and how to participate.  Be sure to subscribe to this board and not miss an announcement.   Get Help with Power Automate Forums: If you're looking for support with any part of Power Automate, our forums are the place to go. From General Power Automate forums to Using Connectors, Building Flows and Using Flows.  You will find thousands of technical professionals, and Super Users with years of experience who are ready and eager to answer your questions. You now have the ability to post, reply and give "kudos" on the Power Automate community forums. Make sure you conduct a quick search before creating a new post because your question may have already been asked and answered. Galleries: The galleries are full of content and can assist you with information on creating a flow in our Webinars and Video Gallery, and the ability to share the flows you have created in the Power Automate Cookbook.  Stay connected with the Community Connections & How-To Videos from the Microsoft Community Team. Check out the awesome content being shared there today.   Power Automate Community Blog: Over the years, more than 700 Power Automate Community Blog articles have been written and published by our thriving community. Our community members have learned some excellent tips and have keen insights on the future of process automation. In the Power Automate Community Blog, you can read the latest Power Automate-related posts from our community blog authors around the world. Let us know if you'd like to become an author and contribute your own writing — everything Power Automate-related is welcome.   Community Support: Check out and learn more about Using the Community for tips & tricks. Let us know in the Community Feedback  board if you have any questions or comments about your community experience. Again, we are so excited to welcome you to the Microsoft Power Automate community family. Whether you are brand new to the world of process automation or you are a seasoned Power Automate veteran - our goal is to shape the community to be your 'go to' for support, networking, education, inspiration and encouragement as we enjoy this adventure together.     Power Automate Community Team

Hear what's next for the Power Up Program

Hear from Principal Program Manager, Dimpi Gandhi, to discover the latest enhancements to the Microsoft #PowerUpProgram, including a new accelerated video-based curriculum crafted with the expertise of Microsoft MVPs, Rory Neary and Charlie Phipps-Bennett. If you’d like to hear what’s coming next, click the link below to sign up today! https://aka.ms/PowerUp  

Tuesday Tip | How to Report Spam in Our Community

It's time for another TUESDAY TIPS, your weekly connection with the most insightful tips and tricks that empower both newcomers and veterans in the Power Platform Community! Every Tuesday, we bring you a curated selection of the finest advice, distilled from the resources and tools in the Community. Whether you’re a seasoned member or just getting started, Tuesday Tips are the perfect compass guiding you across the dynamic landscape of the Power Platform Community.   As our community family expands each week, we revisit our essential tools, tips, and tricks to ensure you’re well-versed in the community’s pulse. Keep an eye on the News & Announcements for your weekly Tuesday Tips—you never know what you may learn!   Today's Tip: How to Report Spam in Our Community We strive to maintain a professional and helpful community, and part of that effort involves keeping our platform free of spam. If you encounter a post that you believe is spam, please follow these steps to report it: Locate the Post: Find the post in question within the community.Kebab Menu: Click on the "Kebab" menu | 3 Dots, on the top right of the post.Report Inappropriate Content: Select "Report Inappropriate Content" from the menu.Submit Report: Fill out any necessary details on the form and submit your report.   Our community team will review the report and take appropriate action to ensure our community remains a valuable resource for everyone.   Thank you for helping us keep the community clean and useful!

Users online (3,632)