Power Platform Community Forum Thread Details

I’m seeing inconsistent behavior with the “Capability use” test method in evaluation of test case methods in Copilot Studio.

Even when my tool is successfully invoked (visible in the Trace and returning correct output),
the test case shows Capability use = Fail.

If I delete the test case and recreate it (with the same question and expected result),
Capability use suddenly shows Pass. No changes to the agent or the tool.

This looks like the test case is caching older capability–tool mappings and only refreshes
when the test case is recreated.

Steps:
1. Agent with a tool/ connector eg: (get current weather/ get current time)
2. Create a test case using Capability use and adding tool get current weather.
4. Run test → tool is invoked → but Capability use = Fail.
5. Delete and recreate test → Capability use = Pass.

In the screenshot added, once i readded the test case the capability started working again and working for the first time.  

Expected:
Capability use should pass whenever the mapped tool is invoked.

Is this a known issue or is there a workaround to force capability metadata to refresh
without recreating the test every time ?

Categories:

Calling actions from Copilot Studio

Why This Happens
Caching/Metadata Stale: The test case may cache the capability–tool mapping at creation. If the agent or tool configuration changes, the test case may not update its internal mapping, leading to false negatives.
UI/Backend Sync Delay: Sometimes, the UI or backend does not immediately reflect changes, especially after edits to tools or capabilities.
Workarounds
Delete and Recreate Test Case: As you found, this reliably refreshes the mapping.
Edit and Save Test Case: Sometimes, simply editing (e.g., reselecting the capability/tool) and saving the test case can force a refresh.
Agent Publish/Republish: Republishing the agent after tool/capability changes may help but is less reliable than recreating the test case.

------------------------------------------------------------------------------------
No Official “Refresh” Button
Currently, there is no built-in way to force a metadata refresh for test cases without recreating them.