Showing results for 
Search instead for 
Did you mean: 
New Member

Best practices: downloading thousands of files

Hi. I want to start a business that downloads a file from thousands of different websites. And then organize all those files into 1 master file. I'm not super technical, but know enough to battle my way through specifications. My question centers on best practices. I plan on hiring high school kids to create flows that fetches the files from around 3,000 different websites. What is the best way to batch all of these flows into 1 job so that it runs on a monthly schedule? Do I run this batch job from my notebook or is there a more robust and strategic solution out there? I welcome any feedback. Thank you.

Community Champion
Community Champion

Hi @leowis1 


First of all, good luck with your new adventure :).


Regarding Power Automate, it runs on the cloud, so no need for notebook concerns.


I would recommend the following:

  1. Cluster sites that download the same information in the same Power Automate so that you don't need to repeat steps
  2. Create Power Automates that do small tasks and then call these Power Automates inside other Power Automates. If you don't know how to do it, here's an article that I wrote on the topic.
  3. Don't use one job. It's too complex to maintain. Also, you'll have all your jobs triggering at the same time. Spread the jobs around the month so that you have only a few a day and not all at the same time.
  4. If you use multiple Power Automates, it's easier to have multiple people creating in parallel, making your job easier.
  5. Be careful with the limits of triggers. You may run into limits of some sort so take a look at the documentation.
  6. If possible trigger the Power Automates when there are changes and not on a schedule. Then you'll have smaller differences that are easier to parse and debug if something goes wrong but be aware of the limits mentioned before


I hope this helps. I know it's generic, but I would need more information to be able to advise you better relating to each Power Automate and what you intend to do on each website.


If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.


Helpful resources

Power Automate News & Announcements

Power Automate News & Announcements

Keep up to date with current events and community announcements in the Power Automate community.

Community Calls Conversations

Community Calls Conversations

A great place where you can stay up to date with community calls and interact with the speakers.

Power Automate Community Blog

Power Automate Community Blog

Check out the latest Community Blog from the community!

Users online (4,832)