Copilot Studio seems like a super cool tool with huge potential, so I wanted to test it out. So far it isn't working well, which I suspect is an error I've made setting it up since it works so well for many others, and I'm new at this. I work at a university, so I used our main website, our athletics website, and our library web site. These are listed in Generative AI -> Public websites & SharePoint sources. I've now also uploaded a one-page PDF document with facts about our school in the Upload a Document section. I have medium content moderation set and enabled "Dynamic chaining with generative actions".
I was getting some strange errors, like when I'd ask what hours the library is open, sometimes it would ask me if I meant the Kirkland, Seattle, or Redmond office. We are not close to Washington state. I think that answer was generated by a Topic. Specifically, the built-in topic "Lesson 2 - A simple topic with a condition and variable" asks which of those 3 cities is right. I have now deactivated Lesson 1, 2, and 3 from the Topics.
My test question is "Who is the president?" If it realizes it can only answer from the provided websites, it should answer with the president of the university instead of the president of the USA or some other country. Here are 2 tests. In Test 2, it got the answer correct on the second try, after answering my first query with "Hello, how can I help you today?". OK - not bad. But below is the transcript for my first attempt, in which it did not correctly answer until the 10th time I asked, at which point it answered correctly. In between I asked a few questions about my university, like who teaches math or computer science, or what is the mascot (which I misspelled with a typo), some of which it got right on the first and second attempts, including providing the correct references from our main website. This time it did not identify the computer science faculty, but if often does. I've tried many other questions with very mixed results - sometimes it works, sometimes it does not.
What am I doing wrong? I saw a demo of Copilot Studio where they used the Microsoft website as a source, and while our site is a decent size with over 100 academic departments, it is nowhere near the size nor depth of microsoft.com. Do I need to write a new Topic to answer any question about my university, or disable all pre-built Topics, or something like that?
Here is a transcript of my first attempt to ask who our president is (I've removed references to my school and people to keep this all generic):
Hello, I’m X Test Copilot 1, a virtual assistant. Just so you are aware, I sometimes use AI to answer your questions. If you provided a website during creation, try asking me about it! Next try giving me some more knowledge by setting up generative AI.
Sent at May 17 at 11:45 AM
3 minutes ago
You said: Who is the president?
You said:
Who is the president?
Sent at May 17 at 11:45 AM
2 minutes ago
Bot true said: Hello, how can I help you today?
Bot true said:
Hello, how can I help you today?
Sent at May 17 at 11:45 AM
2 minutes ago
You said: Who is the president?
You said:
Who is the president?
Sent at May 17 at 11:45 AM
2 minutes ago
Bot true said: I’m sorry, I’m not sure how to help with that. Can you try rephrasing?
Bot true said:
I’m sorry, I’m not sure how to help with that. Can you try rephrasing?
Sent at May 17 at 11:46 AM
2 minutes ago
You said: Who teaches math?
You said:
Who teaches math?
Sent at May 17 at 11:46 AM
2 minutes ago
Bot true said: The Mathematics and Statistics Department at X has a dedicated faculty who teach math courses 12. The faculty members include ...
Bot true said:
The Mathematics and Statistics Department at X has a dedicated faculty who teach math courses 12. The faculty members include ...
2 references
1
Mathematics and Statistics - X
2
Faculty & Staff | Mathematics and Statistics | X
Sent at May 17 at 11:46 AM
2 minutes ago
You said: Who is the president?
You said:
Who is the president?
Sent at May 17 at 11:46 AM
2 minutes ago
Bot true said: Hello, how can I help you today?
Bot true said:
Hello, how can I help you today?
Sent at May 17 at 11:46 AM
2 minutes ago
You said: Who is the president?
You said:
Who is the president?
Sent at May 17 at 11:46 AM
A minute ago
Bot true said: You’re welcome.
Bot true said:
You’re welcome.
Sent at May 17 at 11:46 AM
A minute ago
You said: Who is the president?
You said:
Who is the president?
Sent at May 17 at 11:46 AM
A minute ago
Bot true said: Hello, how can I help you today?
Bot true said:
Hello, how can I help you today?
Sent at May 17 at 11:47 AM
A minute ago
You said: who teaches computer science?
You said:
who teaches computer science?
Sent at May 17 at 11:47 AM
A minute ago
Bot true said: Escalating to a live agent is not currently configured for this bot, however this is where the bot could provide information about how to get in touch with someone another way. Is there anything else I can help you with?
Bot true said:
Escalating to a live agent is not currently configured for this bot, however this is where the bot could provide information about how to get in touch with someone another way.
Is there anything else I can help you with?
Sent at May 17 at 11:47 AM
A minute ago
You said: who teaches computer science?
You said:
who teaches computer science?
Sent at May 17 at 11:47 AM
A minute ago
Bot true said: You’re welcome.
Bot true said:
You’re welcome.
Sent at May 17 at 11:47 AM
Bot true said: Would you like to end our conversation?
Bot true said:
Would you like to end our conversation?
Sent at May 17 at 11:47 AM
A minute ago
You said: No
You said:
No
Sent at May 17 at 11:47 AM
A minute ago
Bot true said: Go ahead. I’m listening.
Bot true said:
Go ahead. I’m listening.
Sent at May 17 at 11:47 AM
A minute ago
You said: Who is the president?
You said:
Who is the president?
Sent at May 17 at 11:47 AM
A minute ago
Bot true said: Hello, how can I help you today?
Bot true said:
Hello, how can I help you today?
Sent at May 17 at 11:47 AM
A minute ago
You said: Who is the president?.
You said:
Who is the president?.
Sent at May 17 at 11:47 AM
A minute ago
Bot true said: I’m sorry, I’m not sure how to help with that. Can you try rephrasing?
Bot true said:
I’m sorry, I’m not sure how to help with that. Can you try rephrasing?
Sent at May 17 at 11:47 AM
A minute ago
You said: Who is the president?
You said:
Who is the president?
Sent at May 17 at 11:47 AM
Just now
Bot true said: Hello, how can I help you today?
Bot true said:
Hello, how can I help you today?
Sent at May 17 at 11:47 AM
Just now
You said: is X a good school?
You said:
is X a good school?
Sent at May 17 at 11:47 AM
Just now
Bot true said: Escalating to a live agent is not currently configured for this bot, however this is where the bot could provide information about how to get in touch with someone another way. Is there anything else I can help you with?
Bot true said:
Escalating to a live agent is not currently configured for this bot, however this is where the bot could provide information about how to get in touch with someone another way.
Is there anything else I can help you with?
Sent at May 17 at 11:47 AM
Just now
You said: what is the macot?
You said:
what is the macot?
Sent at May 17 at 11:48 AM
Just now
Bot true said: The mascot of X is ...
Bot true said:
The mascot of X is ...
1 reference
1
History and Traditions - X
Sent at May 17 at 11:48 AM
Just now
You said: Who is the president?
You said:
Who is the president?
Sent at May 17 at 11:48 AM
Just now
Bot true said: Hello, how can I help you today?
Bot true said:
Hello, how can I help you today?
Sent at May 17 at 11:48 AM
Just now
You said: Who is the president?
You said:
Who is the president?
Sent at May 17 at 11:48 AM
Just now
Bot true said: The president of X is Y
Bot true said:
The president of X is Y 1.
1 reference
1
Introducing X's President | X
Sent at May 17 at 11:48 AM
Just now
Connectivity Status: Connected
Type your message
