cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Highlighted
Regular Visitor

Processing past emails is VERY slow

I am tasked to process past emails by copying all attachments to a SharePoint Online Library along with a few pieces of data (sender, subject, received date/time stamp). The process works except that it is excruciatingly slow. The source mailboxes have over 10k emails to process so I can't rely on Flow if it is going to move this slowly.

 

Here are some rough metrics from my executions:
3 emails with 12 attachments total took roughly 15 minutes
20 days of emails with roughly 135 attachments processed in over 2 hours when I eventually had to cancel

 

The Flow is constructed as follows:

I manually start the flow with 2 date pickers for a starting date and ending date. I use an integer counter with addDays function to set a CurrentDate variable which is then used as a search query by Get Emails (V3) to restrict emails to a received date. Ex: CurrentDate = addDays(variables('StartDate'),variables('intCount'),'yyyy-MM-dd')

 

So basically, inside a Do Until, the Get Emails will process each calendar date from Start Date to End Date. An (Apply to Each) will handle emails with attachments. A second (Apply to Each) will copy each attachment to SharePoint and update the metadata fields with the source email's sender, subject, and received date. After the first cancelled run, I went back and enabled the Concurrency option on both of the (Apply to Each) controls. That didn't seem to help.

 

Here are screenshots of the Flow.

Flow_1.png

 

Flow_2.png

 

Flow_3.png

 

Flow_4.png

 

The run that succeeded shows 14 minutes occurring at the second (Apply to Each), but the actual Create File and Update File operations then only show a few seconds each. So the time doesn't equate to those operations.

Flow_5_Create_Time.png

 

Any thoughts are appreciated. This is basically for a one-time "get all attachments" operation. If I can't speed this up I may just cut my losses and go to C#.

 

Thanks,

Anthony

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted
Microsoft
Microsoft

Re: Processing past emails is VERY slow

H Anthony, 

 

Which plan are you using? Maybe the performance issue may be in the type of license you have.

 

Your plan determines how often your flows run. For example, your flows may run every 15 minutes if you're on the free plan. If a flow is triggered less than 15 minutes after its last run, it's queued until 15 minutes have elapsed.

https://docs.microsoft.com/en-us/power-automate/billing-questions

 

Regards, 

Javier

View solution in original post

2 REPLIES 2
Highlighted
Microsoft
Microsoft

Re: Processing past emails is VERY slow

H Anthony, 

 

Which plan are you using? Maybe the performance issue may be in the type of license you have.

 

Your plan determines how often your flows run. For example, your flows may run every 15 minutes if you're on the free plan. If a flow is triggered less than 15 minutes after its last run, it's queued until 15 minutes have elapsed.

https://docs.microsoft.com/en-us/power-automate/billing-questions

 

Regards, 

Javier

View solution in original post

Highlighted
Regular Visitor

Re: Processing past emails is VERY slow

Javier,

 

I'm certain you are correct. Thanks for the link with info on confirming the subscription plan for the executing account.

 

For any others looking for an alternative, I created a simple C# console app using Exchange Web Service (EWS) api and SharePoint Client api. For uploading to SPO, I followed guidance from this link:

https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/upload-large-files-sample-app-for-... 

 

I also borrowed ideas from this link on downloading email attachments:

https://blog.sqltreeo.com/read-emails-from-exchange-online-mailbox-office-365-into-sql-server/ 

 

With this approach (for each email - download local then upload then update metadata then delete local), I was able to process emails as hoped (thousands of emails in a few hours).

 

Thanks,

Anthony

 

Helpful resources

Announcements
secondImage

August 2020 CYST Winners!

Check out the winners of the recent 'Can You Solve These?' community challenge!

thirdImage

Experience what's new for Power Automate

Join us for an in-depth look at the new Power Automate features and capabilities at the free Microsoft Business Applications Launch Event.

firstImage

Power Platform 2020 release wave 2 plan

Features releasing from October 2020 through March 2021

thirdimage

Power Automate Community User Group Member Badge

Fill out a quick form to claim your user group badge now!

Users online (5,463)