Power Platform Community Forum Thread Details

Hi. I want to start a business that downloads a file from thousands of different websites. And then organize all those files into 1 master file. I'm not super technical, but know enough to battle my way through specifications. My question centers on best practices. I plan on hiring high school kids to create flows that fetches the files from around 3,000 different websites. What is the best way to batch all of these flows into 1 job so that it runs on a monthly schedule? Do I run this batch job from my notebook or is there a more robust and strategic solution out there? I welcome any feedback. Thank you.

Categories:

General topics

Hi @leowis1

First of all, good luck with your new adventure :).

Regarding Power Automate, it runs on the cloud, so no need for notebook concerns.

I would recommend the following:

Cluster sites that download the same information in the same Power Automate so that you don't need to repeat steps
Create Power Automates that do small tasks and then call these Power Automates inside other Power Automates. If you don't know how to do it, here's an article that I wrote on the topic.
Don't use one job. It's too complex to maintain. Also, you'll have all your jobs triggering at the same time. Spread the jobs around the month so that you have only a few a day and not all at the same time.
If you use multiple Power Automates, it's easier to have multiple people creating in parallel, making your job easier.
Be careful with the limits of triggers. You may run into limits of some sort so take a look at the documentation.
If possible trigger the Power Automates when there are changes and not on a schedule. Then you'll have smaller differences that are easier to parse and debug if something goes wrong but be aware of the limits mentioned before

I hope this helps. I know it's generic, but I would need more information to be able to advise you better relating to each Power Automate and what you intend to do on each website.

If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.

Cheers
Manuel