Skip to main content

Notifications

Power Automate - General Discussion
Unanswered

Best practices: downloading thousands of files

(0) ShareShare
ReportReport
Posted on by

Hi. I want to start a business that downloads a file from thousands of different websites. And then organize all those files into 1 master file. I'm not super technical, but know enough to battle my way through specifications. My question centers on best practices. I plan on hiring high school kids to create flows that fetches the files from around 3,000 different websites. What is the best way to batch all of these flows into 1 job so that it runs on a monthly schedule? Do I run this batch job from my notebook or is there a more robust and strategic solution out there? I welcome any feedback. Thank you.

  • manuelstgomes Profile Picture
    manuelstgomes 6,625 on at
    Re: Best practices: downloading thousands of files

    Hi @leowis1 

     

    First of all, good luck with your new adventure :).

     

    Regarding Power Automate, it runs on the cloud, so no need for notebook concerns.

     

    I would recommend the following:

    1. Cluster sites that download the same information in the same Power Automate so that you don't need to repeat steps
    2. Create Power Automates that do small tasks and then call these Power Automates inside other Power Automates. If you don't know how to do it, here's an article that I wrote on the topic.
    3. Don't use one job. It's too complex to maintain. Also, you'll have all your jobs triggering at the same time. Spread the jobs around the month so that you have only a few a day and not all at the same time.
    4. If you use multiple Power Automates, it's easier to have multiple people creating in parallel, making your job easier.
    5. Be careful with the limits of triggers. You may run into limits of some sort so take a look at the documentation.
    6. If possible trigger the Power Automates when there are changes and not on a schedule. Then you'll have smaller differences that are easier to parse and debug if something goes wrong but be aware of the limits mentioned before

     

    I hope this helps. I know it's generic, but I would need more information to be able to advise you better relating to each Power Automate and what you intend to do on each website.

     

    If I have answered your question, please mark your post as Solved.
    If you like my response, please give it a Thumbs Up.

    Cheers
    Manuel

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Microsoft Kickstarter Events…

Register for Microsoft Kickstarter Events…

Announcing Our 2025 Season 1 Super Users!

A new season of Super Users has arrived, and we are so grateful for the daily…

Announcing Forum Attachment Improvements!

We're excited to announce that attachments for replies in forums and improved…

Leaderboard

#1
WarrenBelz Profile Picture

WarrenBelz 145,343

#2
RandyHayes Profile Picture

RandyHayes 76,287

#3
Pstork1 Profile Picture

Pstork1 64,703

Leaderboard