cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
New Member

Best practices: downloading thousands of files

Hi. I want to start a business that downloads a file from thousands of different websites. And then organize all those files into 1 master file. I'm not super technical, but know enough to battle my way through specifications. My question centers on best practices. I plan on hiring high school kids to create flows that fetches the files from around 3,000 different websites. What is the best way to batch all of these flows into 1 job so that it runs on a monthly schedule? Do I run this batch job from my notebook or is there a more robust and strategic solution out there? I welcome any feedback. Thank you.

1 REPLY 1
Super User II
Super User II

Hi @leowis1 

 

First of all, good luck with your new adventure :).

 

Regarding Power Automate, it runs on the cloud, so no need for notebook concerns.

 

I would recommend the following:

  1. Cluster sites that download the same information in the same Power Automate so that you don't need to repeat steps
  2. Create Power Automates that do small tasks and then call these Power Automates inside other Power Automates. If you don't know how to do it, here's an article that I wrote on the topic.
  3. Don't use one job. It's too complex to maintain. Also, you'll have all your jobs triggering at the same time. Spread the jobs around the month so that you have only a few a day and not all at the same time.
  4. If you use multiple Power Automates, it's easier to have multiple people creating in parallel, making your job easier.
  5. Be careful with the limits of triggers. You may run into limits of some sort so take a look at the documentation.
  6. If possible trigger the Power Automates when there are changes and not on a schedule. Then you'll have smaller differences that are easier to parse and debug if something goes wrong but be aware of the limits mentioned before

 

I hope this helps. I know it's generic, but I would need more information to be able to advise you better relating to each Power Automate and what you intend to do on each website.

 

If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.

Cheers
Manuel

Helpful resources

Announcements
Microsoft Ignite

Microsoft Ignite

Join digitally, March 2–4, 2021 to explore new tech that's ready to implement. Experience the keynote in mixed reality through AltspaceVR!

New Super Users

Meet the Power Automate Super Users!

Many congratulations to the Season 1 2021 Flownaut Crew!

Microsoft Ignite

Microsoft Power Platform: 2021 Release Wave 1 Plan

Power Platform release plan for the 2021 release wave 1 describes all new features releasing from April through September 2021.

MPA Community Blog

Power Automate Community Blog

Check out the community blog page where you can find valuable learning material from community and product team members!

Users online (43,904)