Hi @CU21050757-0,
Your current approach—replacing the existing Excel file with updated data until it reaches the 512 MB limit, then uploading a new file to the SharePoint folder to stay within file count limits—is a practical strategy for sure.
If our target is to make sure the agent always refers to the most updated data, let's follow these steps:
File Replacement and Naming: When replacing the existing file, please keep the file name consistent so the knowledge source reference remains unchanged. For new files after reaching size limits, use a clear naming convention with timestamps or version numbers to track files. For example, each time you update the data, you replace this file with the new Excel file but keep the exact same file name (BusinessData_Current.xlsx). Also, when the file size approaches the 512 MB limit, let's upload a new file with a new name, for example: BusinessData_2024Q2.xlsx. Let's keep the old file (BusinessData_Current.xlsx) until the new file is fully indexed and ready.
Knowledge Source Configuration: If the agent uses a SharePoint Folder knowledge source, ensure it is configured to include all relevant files in the folder. When adding a new file, the folder-based knowledge source will index all files, so the agent can access data from multiple files seamlessly.
Indexing and Synchronization: Please remember that SharePoint Folder knowledge sources sync data to Dataverse on a scheduled job that can take several hours (6-8 hours). Need plan updates accordingly and allow time for indexing to complete before expecting the agent to reflect new data.
Splitting Large Files: Alreayd mentioned above, but again, if a single Excel file approaches the 512 MB limit, consider splitting it logically into smaller files by date ranges, departments, or categories. This improves indexing performance and reduces risk of sync failures.
Automate File Management: Let's use Power Automate flows to manage file uploads, replacements, and folder organization to maintain consistency and reduce manual errors.
Agent Refresh: After updating files, republish or refresh the agent if needed to ensure it picks up the latest indexed data.
I am sure some clues I tried to give. If these clues help to resolve the issue brought you by here, please don't forget to check the box Does this answer your question? At the same time, I am pretty sure you have liked the response!
