PowerAutomate Cloud Flow takes too long with ForEach component

(0) Share

Report

Posted on by karr

I'm having a strange issue with PowerAutomate.

I have a basic Cloud Flow which consists of the following steps:

Dataverse connector (a simple query: list EventEditions)
For Each (EventEdition)
- Dataverse connector (a simple query: list CustomerProfiles)
- For Each (CustomerProfile)
  - Dataverse connector (a simple query: list WebEntries)
  - For Each (WebEntry)
    - Compose a string with CustomerProfile and WebEntries

The issue is that, while when I call the flow for one EventEdition (the most outer for each), it takes 30 mins at most (it is for the EventEdition with the largest data), a run for all EventEditions takes longer than a day. The number of EventEditions is 11, the flow for the largest one takes 30 mins at most, and most of them take only a few mins.

Handling each EventEdition in separate flow instances: 30 + 6 + 5 + ... ~= 1 h
Handling all EventEditions in the same flow instance: > 24 h

What would cause this?

I'm trying the simplest approach to overcome this which is to run the flow for each Event Edition sequentially. To do this, I set the Degree of Parallelism of the relevant ForEach components to 1 to make them run sequentially. But it seems this doesn't impact the running time.

If my explanation is not clear or detailed enough, I'll update it based on your comments.

I'd really appreciate your comments and suggestions on this. Thank you in advance.

Edit: Adding a screenshot of the flow.

The main flow, including one child flow.

Categories:

Building flows

I have the same question (0)

All responses (15)

Answers (0)

leo85 609 on at

Like (1)

Report
Hi @karr ,
There are a few things you can check and try.

Complexity: You've got three nested for-each loops. So you're dealing with exponential complexity, which is not ideal. The inner most loop (list WebEntries) has got the biggest influence. If you have 11 events, 100 customers and 100 web entries (just as an example), "compose string" will be called 110.000 times.

You may run into throttling problems here:
API request limits. The limit for API request is currently 40.000 actions, if you have a seeded license (Office 365) (https://docs.microsoft.com/en-us/power-platform/admin/api-request-limits-allocations). When you open the flow details, do you get a message that your flow is currently slow?
Data limits. The limit for data consumption (the amount of data that is written to the flow history within 24 hours) is currently 1GB if you have got a seeded license (https://docs.microsoft.com/en-us/power-automate/limits-and-config). If you have a lot of webEntries and the list WebEntries action provides 1MB of data. If it is executed 1000 times in your flow run, you will run into the data consumption limit. You can decrease data consumption by using filters in your queries (e.g. list WebEntries). When you open the flow details, do you get a banner saying that your flow is using a lot of data?
Connector limits. The Dataverse connector also has got some limitations, which you can have a look at here: https://docs.microsoft.com/en-us/connectors/commondataserviceforapps/.
You have already changed the parallelism. But running things sequentially will slow your flow down. Try to increase the parallelism on every level if possible by setting the degree of parallelism to a number higher than 1. You need to watch out, because running things in parallel can mess up your flow.

Did you list all of the actions in your flow or are there any other actions (e.g. conditions) which you did not list?

Regards,
Leo

----------------------------
If my answer solved your issue, please mark it as a solution.
If what I wrote helped you, please give it a thumbs up.

Was this reply helpful? Yes No
karr 16 on at

Like (0)

Report
Hi @leo85,

First of all, thank you for your help.

You are right, and I'm aware of the exponential complexity. But when I run the same flow for each data record (EventEdition) separately, I don't run into any performance issues, which must be equal to the same number of actions in total.

API request limits. I run into this limit when I include all the necessary actions in the flow. Here, in my question, I put only the main actions such as Dataverse calls, loops, and string operations. This way I don't observe the slow flow warning but the flow is still too slow.
Data limits. I haven't seen any banner regarding the data usage. But I'll keep an eye on it.
Connector limit. I'll read it.

But... Returning back to the beginning, I think I should be getting over this issue by ensuring each flow run for only one EventEdition. Because when I run the flow manually for only one EventEdition, it takes much less as I mentioned before. That would make only about 1 h for all events compared to a day.

But I couldn't achieve it yet for some reason. I tried to read/write to/from a Blob Storage to store the processed event IDs and the processing status (if an EventEdition is being processed, it is true) to be able to process one event each time. But it doesn't work. Maybe my design is still wrong.

I'm really surprised that the PowerAutomate falls short in doing that kind of task. It shouldn't be that difficult. The equivalent data process takes 15-30 mins as an Azure Function in C#.

You have already changed the parallelism. But running things sequentially will slow your flow down. Try to increase the parallelism on every level if possible by setting the degree of parallelism to a number higher than 1. You need to watch out, because running things in parallel can mess up your flow.
I already tried it and it didn't change anything. Also, as I mentioned before, the flow must take about 1 h in total, because when I run the flow manually for each EventEdition, it takes that much (1 h). The core of the problem must be to do with running one flow for all EventEditions (which, I guess, means using parallelism). Setting the most outer loop (ForEachEventEdition) to run sequentially must do the trick, but it doesn't. Am I wrong?

Did you list all of the actions in your flow or are there any other actions (e.g. conditions) which you did not list?
There are many other actions in the original version of the flow and when used they cause the "slow flow" issue. To get rid of this, I split the entire flow into 3 flows. But this didn't help reduce the running time as well.

Was this reply helpful? Yes No
leo85 609 on at

Like (0)

Report

Can you post a screenshot of the flow? Maybe there are other actions that are causing the problem.
Can you go through the flow run, which lasted 24h and post screenshots of the actions that take the longest amount of time?
Can you also post a screenshot of the "actions" chart of the flow's "analytics" page?

Was this reply helpful? Yes No
karr 16 on at

Like (0)

Report

Single EventEdition (19 m - the longest one)All EventEditions (13 h - 11 event editions)

PS. When I put all the necessary actions in the flow, the running time for all EventEditions is about a day. Here, for the light/test version of the flow, it took 13 h.

Was this reply helpful? Yes No
leo85 609 on at

Like (0)

Report

Can you also post a screenshot of the "actions" chart of the flow's "analytics" page?

Was this reply helpful? Yes No
karr 16 on at

Like (0)

Report

Do you mean Run Details as in Logic Apps? As far as I know, it is available for only Logic Apps. My flow is a PowerAutomate Cloud Flow and I don't see any option to see more details about runs.

Was this reply helpful? Yes No
leo85 609 on at

Like (0)

Report

No, not the run details. When you are in the flow studio on the "Cloud flows" page, click the flow name to open its details page. In the top banner there is a button "Analytics". If you click on that, you will see the actions.

Was this reply helpful? Yes No
karr 16 on at

Like (0)

Report

But I don't have it. Maybe it's because of the subscription/plan or my permissions.

Edit: Yes, it is. The documentation says "Admins with the following roles and a license can view the reports in Power Automate analytics." https://docs.microsoft.com/en-us/power-platform/admin/analytics-flow

Was this reply helpful? Yes No
karr 16 on at

Like (0)

Report

Another strange issue is that, in the same run, a smaller record took much longer than the largest one as you can see below.
"26 m" of running time for "the largest" record"5 h" of running time for a "smaller" record

Was this reply helpful? Yes No
leo85 609 on at

Like (0)

Report

Hi @karr,
no, this is a standard feature that should be available to every creator of a flow irrespective of their license. What you quote is a feature of the Power Platform Admin Center. This is not was I was referring to.
Are you the owner/co-owner of the flow?
Is your flow part of a solution? In that case, you may need to select "See analytics" by clicking "more options" (see screenshot).

Was this reply helpful? Yes No