Unanswered

"Get items" return different data on same filter run

(0) Share

Report

Posted on by grahamworth

I am running "Get Items" with pagination set to 100000 items on a large list but I only really pull data back in the 50k most recent items so order by "Created desc", I found if I did not do this I only get a 10th of the items. I believe this is correct functionality as it only scans the first 100k items. Ordering by "Created desc" resolves this initial issue.

Manually exporting the list I can see there are 1336 items that fit my filter. When I run the flow I see the following if i count the rows:

I run the exact same flow again with no change to the back end data and get this:

If I run it again I might get the other result again or this again or a different number. The items is sometimes misses meet the filter criteria so should be shown. It seems random to how many items it is able to return which gives me very low confidence. This is not an isolated scenario, after many tests I found the same issue occurring. Even when reducing the filters to just one I has the same problem of different quantities of data returned.

As an alternative I am using http get request on a continuous loop then filtering the items, but this is much slower (taking +5 minutes) than get items (less than 1 minute).

Any thoughts on why this is happening or ways to resolve it would be greatly appreciated.

Categories:

Using flows

I have the same question (0)

All responses (2)

Answers (0)

David_MA 12,966 Super User 2025 Season 2 on at

Like (0)

Report

I cannot confirm this as I don't have any SharePoint lists with that many items. However, I believe that the OData filters function somewhat like a SharePoint view would. To meet threshold limits with views, you must use indexed columns and the first filter must return less than 5,000 items.

Are the fields in your filter query indexed? If not, that may be your problem. If they are indexed, would supplier reference return less than 5,000 items?

I do have some lists with around 50,000 items, and using index columns and ensuring the first filter returns less than 5,000 items has worked for me.

Was this reply helpful? Yes No
grahamworth 19 on at

Like (1)

Report

All key columns are indexed and I am using simple data types (string, number date) to prevent other issues

To handle anything returning +5k items I am reverting to the http call method. I expect Get Items to work with filters when their are less than 5k items returning. My current list sits at around 200k so not huge, I have others with 3million+ items but this is the only list returning +1000 items on a get items call

I'm becoming pretty certain this is a bug in the "get items" connector. I'm not sure if in the background it is using a http call or the web service. My assumption is the web service as I've not come across this when writing my own rest calls.

Was this reply helpful? Yes No