Power Platform Community Forum Thread Details

Hi,

I'm quite new to the power automate platform, trying to crack on with a flow. I have a scenario where my dynamic JSON will have N number of objects (0<N<x*1,00,000), following are the steps inside my flow/logic.

steps:

1) Initialize a variable of type array.

2) Foreach object inside the huge json (foreach loop configured to run concurrently with maximum degree of parallelism - 100)

3) Do some transformations like converting into CSV table(bcz this is a dynamic object, json key names are unpredictable/unknown), select operation to get the data in a pattern so that it will be serialized into fixed json objects.

4) Add the serialized object into declared array- This step is failing in my current flow apparently initialized array couldn't able to append data concurrently.

Any suggestions how to make array to accept concurrent data? (I could have solved it using concurrent bag in C#) but doesn't look like power automate provides similar variable.

Note: You might be wondering why I need this data in single array? I would like to do bulk ingestion into my sql DB. Opening up a connection every time inside a foreach loop is a costly operation in terms of price and time.

Regards

Kiran.

Categories:

General topics

Hi @Anonymous ,

Instead of using a variable of type array, try using a compose action in it's place. As far as I understand, when Power Automate updates a variable, it locks the variable so it can be updated, then it releases the variable. While the variable is locked any other concurrent action won't be able to update the variable, so that concurrent thread will wait until the variable is unlocked and is free again. This variable locking will affect performance of the loop. The compose action doesn't suffer from this issue.

Example:

Populate the Compose action as you did with the array variable, example:

Then outside of the Apply to Each (i.e. after the Apply to each loop), you can reference the Compose variable (which will be an array) using the following expression:

If you share the flow, there may be other optimizations that could be suggested and tried.

See: The advanced Pieter's method in Power Automate (sharepains.com)

Hope this helps.

Ellis
____________________________________
If I have answered your question, please mark the post as ☑️ Solved.
If you like my response, please give it a Thumbs Up.
My Blog Site