Answered

Flow to import parsed lines to SharePoint list errors with 5000 limit

(0) Share

Report

Posted on by DB7176

Hello, first, want to say I'm new to the community and this topic has been discussed but don't see anything that resolves my issue. So please bear with me. I have a log that is generated (pipe delimited) every hour and only has three columns. A notification email with the log attached is sent every hour. I set up a SharePoint list with proper column names and generated a flow to import to the list when a new email arrives. Standard stuff. The attachment is base64 decoded, parsed by line, parsed by pipe, and converted to JSON. A connection to the SharePoint list is made and an Apply to each operation is triggered to import the values. However, the reports average about 10K lines every hour and I'm getting the "The number of foreach items limit exceeded for action 'Apply_to_each_2': maximum '5000' and actual '31075'" error (this is a bigger number because I need to import a backfill). Questions:

1. Is this a SharePoint configuration issue? I know the default is 50000.

2. If so, can this be changed in the Admin portal? I am admin and owner of the site.

3. Or is this Power Automate issue? If so, can this be configured to a larger number? I have a premium subscription.

4. Or do I just need to import the max and change my flow? If this is the case, can someone give an example or point me in the right direction?

Really appreciate any help. Thank you, DB7176

Categories:

General topics

I have the same question (0)

All responses (4)

Answers (1)

SamLed 2,338 Moderator on at

Like (0)

Report
Copy link

Link copied!
Hi,

That's a Power Automate limit (not issue): Limits of automated, scheduled, and instant flows - Power Automate | Microsoft Learn

I would suggest to process your data by batches:
Compute batch count: items count / Batch size
use Do until loop
Use take() function to get n first items
On next loops, use take() from skip() function output to get next batch. skip count is: (Batch size x (current loop index +1))
______________________________________________________________
If I have answered your question, please Accept the post as solution.
If you like my response, please Thumbs Up.

Was this reply helpful? Yes No
DB7176 12 on at

Like (0)

Report
Copy link

Link copied!

Hello, thank you for responding. That makes sense and understand it's a feature not an issue. Great suggestion. Going to give it a try this weekend and see how it goes. New to Power Automate processes so will have to do some research and dig in. Will update the post with results. Thanks!

Was this reply helpful? Yes No
Verified answer

SamLed 2,338 Moderator on at

Like (0)

Report
Copy link

Link copied!
Hi,

Here's a flow to illustrate what I meant:

I've created an array containing 20 items in a Compose action as source:

I initialize 4 variables:
And

Expressions:
length(outputs('Compose'))
add(div(variables('ItemCount'), variables('BatchSize')), if(equals(mod(variables('ItemCount'), variables('BatchSize')), 0), 0, 1))
NB: Last expression compute required batch count, we need to increment by 1 when the rest of the division "ItemCount / BatchSize" is not zero .

Now let's make batches:
Expression: take(skip(outputs('Compose'), mul(variables('BatchSize'), iterationIndexes('Do_until'))), variables('BatchSize'))

NB: I use IterationIndexes('Do_until') rather than variables('BatchNum'), because the index must be base 0 so that skip() does not exclude an element during the first loop. It turns out that here "BatchNum" is base 0, but it should be able to change base as needed.
______________________________________________________________
If I have answered your question, please Accept the post as solution.
If you like my response, please Thumbs Up.

Was this reply helpful? Yes No
DB7176 12 on at

Like (0)

Report
Copy link

Link copied!

Thank you SamLed for your detailed instruction. Really appreciate it!

Was this reply helpful? Yes No