The performance of collecting multiple batches of 2000 rows from sharepoint is seriously impacted

Like (0) Share

Report

Posted on 5 Feb 2025 16:58:20 by JD_26

101

I am using this code to create a collection for my works CMS tool , activity has 60,000 rows , and when I went to switch on the New analysis engine , I had to switch off Collection access in component scope.

The impact is that the performance is seriously impacted , with users complaining about how slow the tool had become.

I have switched off New analysis engine and turned the Collection access in component scope back on , and the performance is back to the normal speeds.

I am concerned that this feature is in Retired section and the impact it will have on my works CMS tool if removed.

Concurrent(
        ClearCollect( activity_00, Filter( Activity_List, ID_Code < 1 )),
        ClearCollect( activity_0, Filter( Activity_List, ID_Code >= 1 && ID_Code < 2000 )),
        ClearCollect( activity_1, Filter( Activity_List, ID_Code >= 2000 && ID_Code < 4000 )),
        ClearCollect( activity_2, Filter( Activity_List, ID_Code >= 4000 && ID_Code < 6000 )),
        ClearCollect( activity_3, Filter( Activity_List, ID_Code >= 6000 && ID_Code < 8000 )),
        ClearCollect( activity_4, Filter( Activity_List, ID_Code >= 8000 && ID_Code < 10000 )),
        ClearCollect( activity_5, Filter( Activity_List, ID_Code >= 10000 && ID_Code < 12000 )),
        ClearCollect( activity_6, Filter( Activity_List, ID_Code >= 12000 && ID_Code < 14000 )),
        ClearCollect( activity_7, Filter( Activity_List, ID_Code >= 14000 && ID_Code < 16000 )),
        ClearCollect( activity_8, Filter( Activity_List, ID_Code >= 16000 && ID_Code < 18000 )),
        ClearCollect( activity_9, Filter( Activity_List, ID_Code >= 18000 && ID_Code < 20000 )),
        ClearCollect( activity_10, Filter( Activity_List, ID_Code >= 20000 && ID_Code < 22000 )),
        ClearCollect( activity_11, Filter( Activity_List, ID_Code >= 22000 && ID_Code < 24000 )),
        ClearCollect( activity_12, Filter( Activity_List, ID_Code >= 24000 && ID_Code < 26000 )),
        ClearCollect( activity_13, Filter( Activity_List, ID_Code >= 26000 && ID_Code < 28000 )),
        ClearCollect( activity_14, Filter( Activity_List, ID_Code >= 28000 && ID_Code < 30000 )),
        ClearCollect( activity_15, Filter( Activity_List, ID_Code >= 30000 && ID_Code < 32000 )),
        ClearCollect( activity_16, Filter( Activity_List, ID_Code >= 32000 && ID_Code < 34000 )),
        ClearCollect( activity_17, Filter( Activity_List, ID_Code >= 34000 && ID_Code < 36000 )),
        ClearCollect( activity_18, Filter( Activity_List, ID_Code >= 36000 && ID_Code < 38000 )),
        ClearCollect( activity_19, Filter( Activity_List, ID_Code >= 38000 && ID_Code < 40000 )),
        ClearCollect( activity_20, Filter( Activity_List, ID_Code >= 40000 && ID_Code < 42000 )),
        ClearCollect( activity_21, Filter( Activity_List, ID_Code >= 42000 && ID_Code < 44000 )),
        ClearCollect( activity_22, Filter( Activity_List, ID_Code >= 44000 && ID_Code < 46000 )),
        ClearCollect( activity_23, Filter( Activity_List, ID_Code >= 46000 && ID_Code < 48000 )),
        ClearCollect( activity_24, Filter( Activity_List, ID_Code >= 48000 && ID_Code < 50000 )),
        ClearCollect( activity_25, Filter( Activity_List, ID_Code >= 50000 && ID_Code < 52000 )),
        ClearCollect( activity_26, Filter( Activity_List, ID_Code >= 52000 && ID_Code < 54000 )),
        ClearCollect( activity_27, Filter( Activity_List, ID_Code >= 54000 && ID_Code < 56000 )),
        ClearCollect( activity_28, Filter( Activity_List, ID_Code >= 56000 && ID_Code < 58000 )),
        ClearCollect( activity_29, Filter( Activity_List, ID_Code >= 58000 && ID_Code < 60000 )),
        ClearCollect( activity_30, Filter( Activity_List, ID_Code >= 60000 && ID_Code < 62000 ))
    );
    // Clear and collect all filtered data into a single 'activity' collection
    ClearCollect(activity,activity_00, activity_0, activity_1, activity_2, activity_3, activity_4, activity_5, activity_6, activity_7, activity_8, activity_9, activity_10, activity_11, activity_12, activity_13, activity_14, activity_15, activity_16, activity_17, activity_18, activity_19, activity_20, activity_21, activity_22, activity_23, activity_24, activity_25, activity_26, activity_27, activity_28, activity_29, activity_30);

All responses (3)

Answers (0)

iAm_ManCat 18,206 Most Valuable Professional on 06 Feb 2025 at 14:32:15

Like (0)

Report

The performance of collecting multiple batches of 2000 rows from sharepoint is seriously impacted

As I said, I think it would be better to re-architect. Resolve the delegation issues and that should mean you can refer to the datasources directly instead of trying to cache large volumes of data.

Can you share which parts are showing as non-delegable and what the field types are in each case?
JD_26 101 on 06 Feb 2025 at 08:17:59

Like (1)

Report

The performance of collecting multiple batches of 2000 rows from sharepoint is seriously impacted

@iAm_ManCat

Thanks for your response.

I have to create collections , due to the delegation issues.

My issue is that I have CMS built from 4 different sharepoint list

Master , where a new organization is added , the ID from that is then used

to connect the other 3 list

I have a supplier list , which can have up to 15 different suppliers per organization

The large list is the activity list , which shows activity for each of the suppliers

The CMS consist of galleries and forms , with filters.

When a user selects an organization it then populates all the matched suppliers in a gallery

Then when one of those is selected then the activity is shown in another gallery.

This is why there are 60,000 rows of activity 1200 orgs , 6000 suppliers , and 60,000 activity records.

the galleries are populated with this code

suppliers

Filter(

Master_Profiles,

account_id = master_search.Selected.ID,

(combo_sm_rep.Selected.Result = Blank() || combo_sm_rep.Selected.Result = "All" || 'SM Rep' = combo_sm_rep.Selected.Result),

(combo_dataset.Selected.Value = "All" || Dataset = combo_dataset.Selected.Value),

(combo_status.Selected.Value = "All" || Status = combo_status.Selected.Value)

)

activity

SortByColumns(

Filter(

activity,

'Dataset Subtype' = supplier_search.Selected.'Dataset Subtype' &&

DupeID = supplier_search.Selected.'Dupe ID' &&

AccountID = supplier_search.Selected.account_id

),

"ID",

SortOrder.Descending

)
iAm_ManCat 18,206 Most Valuable Professional on 06 Feb 2025 at 07:51:25

Like (0)

Report

The performance of collecting multiple batches of 2000 rows from sharepoint is seriously impacted

Hiya,

Regardless of whether or not the new analysis engine is slowing down your data retrieval, I would strongly suggest re-architecting this as soon as you can - you should not be loading more than a small number of records in locally, as that's performant design practice.

Could you explain the reason that 60k records are needing to be pulled in instead of referring to the datasource directly - is it because of offline usage?

Cheers,

Sancho

Community site session details

The performance of collecting multiple batches of 2000 rows from sharepoint is seriously impacted

Helpful resources

News and Announcements

Quick Links

Announcing the Engage with the Community forum!

🌸 Community Spring Festival 2025 Challenge Winners! 🌸

Warren Belz – Community Spotlight

Leaderboard > Power Apps - Power Apps Experimental Features

Featured topics

Product updates

Community site session details

The performance of collecting multiple batches of 2000 rows from sharepoint is seriously impacted

Helpful resources

News and Announcements

Quick Links

Announcing the Engage with the Community forum!

🌸 Community Spring Festival 2025 Challenge Winners! 🌸

Warren Belz – Community Spotlight

Subscribe to this forum!

Leaderboard > Power Apps - Power Apps Experimental Features

Featured topics

Product updates