Copilot using Sharepoint library as source only looks responds certain topics based on one document

(2) Share

Report

Posted on by daniel27ad_work

I see a lot of posts without replies (even similar ones to mine) but I'll have a go anyways:

My company produces proprietary software that project engineers need to know how to correctly use and configure following best practice documents. Certain settings in the software have initial values that best practice documents recommend they be set to when first creating a config, but this values can over time once a project has been implemented and data has bee collected, in what we call the "tuning process".

We have two different Best Practice documents where the same settings can be mentioned: One of them when we create the configuration for the first time, and another for the tuning phase of the project.

I have noticed that when I ask for the best practice of a certain setting when creating the configuration file, copilot will always respond to me based on the best practice document that was created for tuning, and not for the initial configuration creation.

Word matching wise, it makes no sense:
One document is called:

And the other is called:

And my question is:
What is the Best Practice for the "XXXXXXXXX" option when creating the configuration file in (PRODUCT NAME)? (This is just one of the many questions I've attempted, all of them unsuccessful).

The word configuration should clearly lead it to the document I want! There is another specific word I'm using (can't share it for company privacy) that's present both in my question and in the titles of the pertinent sections in the document I want. So that's even further evidence for me that it is doing something wrong.

I've even replied back to it trying to make it reconsider, and it will insist on the same source, or at worst, even give me other sources in other secondary folders I've provided it with.

I've not done more than watching Youtube videos on the Subject, sadly I can't seem to find any clues as to why it is not responding my question correctly.

I would not like the type of solution where I'd need to give it specific instructions to match specific keywords to that document and other keywords to the other, when it should it do it on its own to begin with. But beyond that, my problem is that my library is extensive and this is just the first test I'm making with one of my library folders. If I was to configure specifc stuff for something like this, I can't begin to imagine the amount of work I'd have to do to get it to work for the whole library.

Thanks in advance!

Categories:

General topics

I have the same question (0)

All responses (4)

Answers (0)

Suggested answer

Vinoth Selvam 1,590 Super User 2025 Season 2 on at

Like (1)

Report

Hi daniel27ad_work,

It looks like Copilot is prioritizing certain document sources over others since it perceives some content as more contextually relevant or detailed, even when keywords indicate otherwise.

Microsoft always suggest to have the knowledge sources as much as unique as we can. Since we don't have control over how Generative answer picks the content from our files.

If possible in Copilot’s configuration, categorize/tag each document based on its purpose (“Initial Configuration” and “Tuning”) to help it distinguish between sources. Provide clear descriptions for the copilot to understand the knowledge source.

You could place the initial configuration and tuning documents in different folders and specify to Copilot which folder to search in based on the query. Mentioning the specific folder (like “Configuration Best Practices folder”) in your question may help Copilot focus on the right source.

Thanks.

Was this reply helpful? Yes No
SS-15100851-0 15 on at

Like (0)

Report

We have been trying to integrate sharepoint with copilot studio as knowledge base without any success.

All the authentication secret key part has been configured by the admin as per Microsoft sharepoint integration guidelines of Copilot studio.

But still when we ask a question the bot says as per the below image; Seems like the bot is not able to fetch the document uploaded in sharepoint.

Was this reply helpful? Yes No
daniel27ad_work 19 on at

Like (0)

Report

Thank you for the help and apologies for replying this late. I did sort this out.

I made a dummy library with only a couple of documents, and then even with just the one document that mattered: It still didn't give me the answer I wanted!

That's when I found that the issue was in the document. Even though MS claims that it can work with up to 80 k words per document (hopefully I'm remembering right), this document was just 11k words, and when I split it in half and left just 6k, it worked, not only in the test environment, but also in the full one mixed with all the other documents.

So there you go.

Was this reply helpful? Yes No
Pitfrog911 Microsoft Employee on at

Like (0)

Report

Do you mean that the agent is not generating a response when asking it? Problem with the SharePoint integration is that it is using Microsoft Graph, which in its basic form is using a lexical/ keyword search.

This ofc is not very powerful because the end user needs to define a precise query so that MS Graph can find somnething. If you want a more powerful search, i.e. a semantic search we also need a semantic index which comes when you have a M365 Copilot license in the tenant.

Was this reply helpful? Yes No