Running power automate for huge number of records.

(0) Share

Report

Posted on by Shashank Bhide

942 Moderator

I've to update a field in contact by calling a 3rd party API in a power automate.
The 3rd party API scales extremely well, we've an existing power automate running on a single contact record's create/update,
We are planning to call the same power automate as a child flow in a concurrent for each loop with flow retrials turned off.
Expected number of contacts and power automate runs are going to be 200,000 minimum.
Failures are planned to be recorded in a cosmos DB table and retried max 5 times using another power automate.

Question :

do you see any challenges in this design?
can I catch errors with detailed stack trace happening anywhere inside the entire power automate, including some URL to jump to see the failed instance?

Categories:

Microsoft Dataverse with Power Apps

I have the same question (0)

All responses (2)

Answers (1)

Verified answer

ivan_apps 2,187 Moderator on at

Like (0)

Report

seems like you gave it some good thought. My main concern would be hitting service limits for power automate. You would at minimum need a dedicated service account with Power Automate premium solely for this workload to qualify in the ‘Medium’ performance profile. If I’m reading the request limits correctly, your 24 hour limit is then 200k requests which is exactly in line with your minimum. I would venture to say you might need a Power Automate process plan.
For retry - you can do scoped retries inside the flow and gather error info there. Take a look at the articles below. A new feature that made GA is also power automate work queues. This helps in the retry behavior and ensures prioritization and visibility in failing flows much better than inside the flow ‘runs’.
I would recommend taking a look at work queues unless you are set with cosmos db.

Request Limits:
https://learn.microsoft.com/en-us/power-automate/limits-and-config#power-platform-request-limits

try catch in power automate
https://d365demystified.com/2022/01/20/try-catch-error-handling-using-scope-in-power-automate/

Automate Work Queues:
https://powerautomate.microsoft.com/en-us/blog/microsoft-power-automate-work-queues-are-generally-available/

Was this reply helpful? Yes No
Shashank Bhide 942 Moderator on at

Like (1)

Report

Thanks @ivan_apps for giving the idea of power automate work queues, I had completely forgotten about it. I missed to add that we aren't reading from Dataverse, but from a SQL Online which will have a copy of the data all the time, so I think hitting service limits for reading is covered, need to discuss with the customer about premium license though because AFAIK we need a premium license even for the developer?

Was this reply helpful? Yes No