Unanswered

Search results for Sharepoint knowledge source does not work well

Like (0) Share

Report

Posted on 27 Dec 2024 20:24:54 by YM-27122000-0

I have built a Copilot Studio agent with a Sharepoint site knowledge source and the search performs poorly. I ran several tests with multiple users.
A few examples of the issues I am finding:
- Asking the bot 3 time the same question: the first 2 times it asked me to re-phrase, the third time it provide me a good enough answer. A business user will immediately think this is not working, this is not good. The bot is configured with high content moderation.
- My site has product manuals for various machine models, in PDF format. When asking to provide all machine models with a particular feature, it only returned results from one product manual, for one model. I expected to see 3 models sharing the same feature. This feature was shown in a PDF file showing a comparison between the 3 models, also in each model's product manual. The regular Sharepoint search returns all matching documents, however the bot does not. I expected a response similar to "this feature is available for model a, b and c."
- I had 3 users asking the same question: one received a very short and incomplete answer (8 lines), the other 2 received a very good long, detailed answer (60 lines). I expected to see a good enough amount of detail, even if it's phrased differently.
- I asked the bot several times the same kind of question, phrased slightly different, with more or less context. The first day I tested this I got "don't understand, please rephrase". The next day, asking the same question, I got qreat answers.
- I used the bot quite a bit. The analytics are empty, the logs are empty. I have no ability to analyze the logs or produce them for product support.

The bot is inconsistent, unreliable and is not production ready for business users.

What can I do to improve the performance?

Categories:

General

All responses (2)

Answers (0)

Suggested answer

VASANTH KUMAR BALMADI 266 on 31 Dec 2024 at 05:20:59

Like (1)

Report
Search results for Sharepoint knowledge source does not work well

Hi,

To address the poor search results when using SharePoint as a knowledge source in Copilot Studio, here are the steps you can take based on your suggestions:

1. Add a System Prompt

Purpose: System prompts provide additional context to guide the bot in how it should interpret and respond to queries. This can significantly improve the accuracy and relevance of responses.

How to Implement:

You can configure system prompts within the Copilot Studio settings to provide guidance on expected behavior. For example:

"Please provide a detailed answer based on all available documents for the queried feature."

"Make sure to consider all models when looking for features in the documentation."

These prompts can help the bot better understand how to prioritize information and improve its response consistency.

2. Change Content Moderation Settings

Purpose: High content moderation levels may be filtering out answers that could be valid, resulting in missed or incomplete responses. Lowering the moderation sensitivity can help in allowing more relevant information to be shared.

How to Implement:

Navigate to the Copilot Studio settings and review the content moderation settings.

Lower the sensitivity if it's set too high, and experiment with different moderation levels to find an optimal balance between filtering and providing relevant answers.

Test with a moderation-free setup to see if the bot provides better results for your queries.

3. Document Size Limitation

Purpose: Large documents, especially PDFs over 7MB, may not be indexed properly or may not be included in search results, leading to incomplete answers.

How to Address:

Optimize Document Size: Try to reduce the size of your PDFs if possible. Consider breaking up large PDFs into smaller files, each focusing on specific products or models.

Split Information Across Multiple Documents: Instead of relying on large single documents, consider distributing your content across multiple smaller, more focused documents to ensure they are indexed and retrieved accurately.

Check for Indexing Issues: Ensure that SharePoint's indexing system is correctly configured to handle large documents and that documents are fully searchable.

Additional Steps to Improve Search Performance:

Use Metadata and Structured Data: As PDFs may not always yield great results, leverage structured data from SharePoint lists or libraries, which will allow the bot to query specific fields (e.g., product names, features) more effectively.

Reindex SharePoint Content: If documents aren't appearing in search results, try reindexing your SharePoint site. This will force SharePoint to rebuild the search index, improving the accuracy of search results.

By taking these steps—using system prompts, adjusting content moderation, and managing document sizes—you should see an improvement in the Copilot Studio bot's search performance with SharePoint knowledge sources.
Mahesh Chintha 158 on 28 Dec 2024 at 02:56:54

Like (0)

Report

Search results for Sharepoint knowledge source does not work well

Try below options to improve the copilot performance:

1. Add a system prompt with the details of what are you expecting

2. change the content moderation

3. You may not get results from the document if the size is larger than 7MB