Set variable and Compose actions in apply to each loop with Concurrency Control and Degree of Parallelism

(0) Share

Report

Posted on by PURNA429

195

Hi,

I have a flow which queries a SharePoint list and then create a csv out of it. List contains 100k items and GetItems action with filter condition would give me around 20k items. I need to process these 20k items inside an apply to each loop to convert date time formats . For this I am setting a variable to store the field value and then using a compose action to convert the date time format like below.

Before having this date conversion functionality inside apply to each loop , to speedup the process, I have set the Concurrency Control with Degree of Parallelism as 30 and my flow was running fast and could get the csv quickly. Here is the setting

But with the introduction of these additional actions (like set variable, compose etc.) inside apply to each I got warning message "Enable concurrency control for the apply to each loop and set its degree of parallelization to 1 when using a Set Variable action inside an apply to each loop."

With the concurrency ON and degree of parallelism set to 1, my flow is taking 3-4 hours to process 20k records . So question here is, with the concurrency setting on, should we not to use Set variable and compose actions.? Can some one clearly explain me what will happen if we set. And in case if we should not use the parallelism what are the best options to speed up this kind of process.?

-Purna

Categories:

Building flows

I have the same question (0)

All responses (9)

Answers (1)

Sort by

MJain 2,450 on at

Like (0)

Report
Copy link

Link copied!

Hi @PURNA429 ,

If your aim is to get array of {Email and Start Date} for large number(10k) then in the loop just use compose action with your variables set inside it . You have to use expressions in one step for setting formatted date.

Also in order to get the array of data just use Compose action outside loop and just use the expression outputs('Compose'), which you are using inside loop, please change the outputs('Compose') to what you are using.

Thanks

Was this reply helpful? Yes No
PURNA429 195 on at

Like (0)

Report
Copy link

Link copied!

Thanks @ManishJain for the reply. Can you please elaborate more on "use Compose action outside loop"? if I keep compose action outside, how would i get the iterated value ?

Also with this change can I set back the parallelism to 30 . Will this speedup the process?

Thanks
Purna

Was this reply helpful? Yes No
Verified answer

MJain 2,450 on at

Like (2)

Report
Copy link

Link copied!

Hi @PURNA429 ,
Compose action inside loop becomes object and then you can access it outside using another compose action to get the output of compose as an array. so you do not need to use set variable inside loop. Also the set variable will not work with concurrency.

Do refer to this blog:

https://sharepains.com/2019/07/09/compose-apply-to-each-power-automate/

Thanks

Was this reply helpful? Yes No
PURNA429 195 on at

Like (0)

Report
Copy link

Link copied!

Thanks alot @ManishJain for the guidance.
Here is my flow, will this be okay with concurrency ON and with parallelism of 20.?

Here I am writing this output to a csv. But I need to format the date before creating the csv.
Here my StartDate column is datetime type and I am not finding a way to format it (expressions needs string input) in inner compose action. How can we achieve this.. Can you suggest.
Thanks again for your help !

-Purna

Was this reply helpful? Yes No
MJain 2,450 on at

Like (0)

Report
Copy link

Link copied!

Hi @PURNA429 ,

If your date coming from Sharepoint is actually of type datetime then you can use formatdatetime in expressions and choose the format of your choice like 'dd-MM-yyyy'.

Thanks

Was this reply helpful? Yes No
PURNA429 195 on at

Like (2)

Report
Copy link

Link copied!
Thank you @ManishJain .

Implemented logic like below . Putting it here for community purpose.

Inside apply to each, one compose action to format the date and second compose to create an array with other fields ( In original workflow I have more fields ).
Outside of apply to each loop used the outputs('InnerComposeAction') as input to my create csv.

This worked well without any issues.

In actual workflow with few other business conditions and setting degree of parallelism as 40 , flow took ~16 minutes to create a csv with 25k records from a list with 200k+ items .

-Purna

Was this reply helpful? Yes No
ggracia 6 on at

Like (0)

Report
Copy link

Link copied!

The best way to do parallelism if you use variables is to use nested flows as part of a solution.
Greetings.

Was this reply helpful? Yes No
Yasir_rd 91 on at

Like (0)

Report
Copy link

Link copied!

Can you give an example of nested flow please?

Was this reply helpful? Yes No
rw_ga 137 on at

Like (0)

Report
Copy link

Link copied!

I know this is old and the author asked about compose and variables, but wouldn't a "Select" data action be the fastest here? You would just put the 'Get Items' Value as the from, add two map entries any custom label and then use item()?['FIELDLABELHERE'] as the value. Use the formatDateTime expression to manipulate the date format however you want. I would think this would be much faster than the loop with max concurrency, but I haven't tested.

Was this reply helpful? Yes No