Apply to each - Parallel Processing - 50 limit

(0) Share

Report

Posted on by Marty3012

Hi,

I posted this before but for some reason i cant find it within any of the forums.

I have the following flow;

Parallel Processing 50 plus.PNG

Description

1 - Connect to an excel sheet and get all rows

2 - Within the Apply to each, loop through each row. (Set to Parallelism)

3 - Set condition that checks if the 'Expiration Date' field for each row equals the current date

4 - If so, set off approvals

5 - If it is approved then the comments provided are sent back to the excel file into another field.

All is working well. However there is the limit of 50 parallel tasks. After the dates are compared I could have between 1-100 rows returned. One possible solution I thought of was to create 2 copies of the Apply to Each loop described above. I could then have a condition that checks if there is say 57 rows returned, one loop could process 50 and the other could process 7. Would this be possible?

I've tried testing using variables and count functions but I'm getting nowhere. Any help appreciated.

Thanks

I have the same question (0)

All responses (7)

Answers (1)

Pstork1 68,326 Most Valuable Professional on at

Like (0)

Report

Re: Apply to each - Parallel Processing - 50 limit

Other than the performance implications. Why do you need to process all the rows in Parallel? If you have 57 rows and set Parallel to 50 it will still process all 57 rows. It will start processing for 50 of the rows and then start the final 7 as soon as other rows finish. So all 57 will be processed without changing anything. They will just be throttled so that no more than 50 are being processed at a time.

Was this reply helpful? Yes No
Marty3012 on at

Like (0)

Report

Re: Apply to each - Parallel Processing - 50 limit

The reason for parallel is that for each excel row there is an 'email address' field which is used to send the approval. For example, if 57 rows are returned then 57 different approvals must be sent out in parallel to 57 different email addresses on that particular day (Expiration Date).

So for in the example I gave, 7 approvals would not go out on the current day (Expiration Date) as they would have to wait for the other 50 to finish.

thanks for your reply

Was this reply helpful? Yes No
Verified answer

Pstork1 68,326 Most Valuable Professional on at

Like (0)

Report

Re: Apply to each - Parallel Processing - 50 limit

Ok, got it. In that case I think you'll have to put an intermediate Loop between your Excel List action and your main processing Loop. Prior to that Loop create two array variables. Inside the loop take the values from the first 50 objects and append them as objects to the first array variable. Once the count reaches 50 append the rest to the second array variable. Then create two parallel Apply to Each Loops for your main processing. Process the first array in one and the second array in the other. Set both for maximum parallel of 50. If you have more than 100 you'll have to add a third variable and parallel loop.

Was this reply helpful? Yes No
Marty3012 on at

Like (0)

Report

Re: Apply to each - Parallel Processing - 50 limit

Due to holidays and trying to work this out, I have eventually created a work around.

First of all, the flow which i created initially was too slow as the condition within the loop was being evaluated on each Excel row. This was causing a problem with the limit of 256 lines when Flow reads Excel. Instead I created a series of filters so that I got all the rows I needed and then fed them to the loops (see below);

I created 2 Apply to each loops. The 1st one reads the filtered data but only the first 50 rows. This is done with the following Expression:

This is also with the Apply to Each concurrency control set to 50.

The 2nd loop does the same but for rows only after the first 50:

Again, concurrency control is set to 50.

I could then add a 3rd apply to each with the expression and so on.

Downside of course is that you have to keep creating more Apply to Each loops if you are working with a lot of data.

I've tested this and it works.

Was this reply helpful? Yes No
Marty3012 on at

Like (2)

Report

Re: Apply to each - Parallel Processing - 50 limit

After further testing I realised that my expressions were incorrect. To use multiple Apply to Each functions to get round the 50 parrallel limit, you would use the following expressions instead;

Was this reply helpful? Yes No
NV-27100430-0 2 on at

Like (0)

Report

Apply to each - Parallel Processing - 50 limit

Hi @Marty3012, in which action are the expressions you listed above were pasted into? I am still new and having the same issue. Thanks for the help!

Was this reply helpful? Yes No
takolota1 4,946 Moderator on at

Like (0)

Report

Apply to each - Parallel Processing - 50 limit

@NV-27100430-0

There have been updates since this thread was created. You could now use chunk( ) on the values of your dataset for the from of one apply to each, then create a nested apply to each inside that with the from set to the outer loop’s current item. That way you can set up to 50 x 50 rows of data to run at once.

Was this reply helpful? Yes No