Unanswered

Generative answers (conversation boost) performance relative to Bing and Bing Copilot

(0) Share

Report

Posted on by chrislrobert

Hi community,

I've been testing generative answers (conversation boost) on a public website, and I've found the performance to be quite poor relative to how Bing and Bing Copilot perform (using the "site:" specifier as recommended here). It's a site with thousands of reports, and if I ask for reports on a specific topic it'll typically just give me just one. If I ask for a list, it'll say that it doesn't understand or that it can't provide a list and I should go search on the website (see examples below). Bing and Bing Copilot do great with the same queries, surfacing lots of relevant reports.

I'm testing this as a copilot tool for the website, and in that setting it would obviously be awkward to suggest that the user go to the website and search there (since they'll already be using the copilot on the site to do that very thing). An example of follow-up questions gone badly:

You said: Can you list other studies?

Bot said: I’m sorry, I’m not sure how to help with that. Can you try rephrasing?

You said: You responded with one study, I'd like a list of others, please.

Bot said: I’m sorry, I’m not sure how to help with that. Can you try rephrasing?

You said: List all studies available on adolescent health in india

Bot said: I’m sorry, but I don’t have access to the specific list of studies available on adolescent health in India. However, you can visit the xyz website at [citation] to explore a wide range of research studies and resources related to adolescent reproductive and sexual health in India.

I can obviously fall back to scraping the website myself, indexing the content in a vector database, and building a custom chatbot within the AI Studio. But this Copilot Studio seems like it's meant to meet this need, and I'd prefer something my clients' IT teams can maintain. Any suggestions would be welcome.

Thanks!

Chris

Categories:

General topics

I have the same question (0)

All responses (3)

Answers (0)

chrislrobert 9 on at

Like (0)

Report

Another very specific example:

"Can you list the 2023 submissions for gender equality and equity global goal? site:https://knowledgecommons.popcouncil.org/" in Bing Copilot results in a very helpful answer that lists the first 15 2023 submissions from https://knowledgecommons.popcouncil.org/focus_gender-equality/.

In my custom Copilot with generative answers:

You said: Can you list the 2023 submissions for gender equality and equity global goal?

Bot said: I’m sorry, I’m not sure how to help with that. Can you try rephrasing?

You said: What are the global goals?

Bot said: The Global Goals, also known as the Sustainable Development Goals (SDGs), are a set of 17 goals established by the United Nations to address global challenges and promote sustainable development. These goals cover a wide range of areas including poverty eradication, gender equality, climate action, and access to quality education and healthcare. They aim to create a more equitable and sustainable world for all. To learn more about the Global Goals and the work being done in this area, you can visit the Population Council website’s Global Goals section 1. The website provides valuable information and resources related to gender equality and equity, which is one of the key areas addressed by the Global Goals. Please note that as the PopCouncil Evidence Assistant, I can provide information based on the documents retrieved from the Population Council’s Knowledge Commons. If you have any specific questions or need further assistance, feel free to ask.

Note: the citation in the above was to https://knowledgecommons.popcouncil.org/focus_gender-equality/, which included precisely the answer I was originally seeking.

You said: Can you list the 2023 submissions for gender equality and equity global goal?

Bot said: I’m sorry, but I don’t have access to the specific list of 2023 submissions for gender equality and equity global goal 1.

Again it cited the very page with the answer I sought, but it would not answer my question by listing the submissions. I believe that even ChatGPT 3.5 would be able to conduct that kind of basic summarization task. Somehow, the generative answers engine is finding the relevant page, but not generating a helpful answer.

Curious if others have seen this kind of behavior.

Chris

Was this reply helpful? Yes No
remidyon on at

Like (0)

Report

Hi @chrislrobert
I share your frustration - Copilot Studio is supposed to do exact what you are looking for. I have experienced that issue firsthand a couple of times with very easy data sources.
I would recommend trying to lower the moderation level to see if that helps - sometimes the internal moderation is a little "too strong".
I tried your website and some of the questions you asked after lowering the moderation and it got decent answers.
One thing that you need to be aware is that not all PDFs the website contains are indexed in Bing (it depends how the website stores them). Could be valuable if you could download them and index them (either upload them to copilot or use an azure storage)

Was this reply helpful? Yes No
chrislrobert 9 on at

Like (1)

Report

Thanks, @remidyon, for the tips! Unfortunately, my trial period ended and we have resorted to custom-building a RAG system with LangChain and Azure AI Search for the moment. I love the potential simplicity and IT-level maintainability of the Copilot Studio, but it feels a little too black-box given its baseline level of performance. It would need to really perform well by default if it's to have so few levers of control.

Thanks again,

Chris

Was this reply helpful? Yes No