Know when the agent stops knowing.

A green connection light does not prove that an agent can still find the truth. Build a small set of known-answer questions, replay them after launch, and catch silent retrieval or tool drift before users become the monitoring system.

About 8 minutes · no signup · exportable evidence pack

01Define truth

Pick questions whose correct answer, source, or safe refusal is known.

02Save a baseline

Record what the healthy agent returns before a platform or knowledge change.

03Replay the pack

Paste current outputs and test the assertions without exposing business data.

04Act on drift

Export the failed prompts, timestamps, evidence, owner, and containment plan.

Set the operating target.

Describe the live agent and choose the reliability threshold that should trigger investigation.

Self-reported operating context

Agent or workflow Business owner Environment or channel Primary knowledge or tool Test cadence Incident owner

Reliability target 98% Consecutive failures to page Evidence window

This is an evidence harness, not an uptime monitor.A canary proves whether a few important known answers still work. It does not prove that every possible question is correct.

Use business-owned questions such as a current refund rule, an approved escalation path, a known customer status, or a safe refusal.

Define the known answers.

Each row should test one business-critical fact, source, action, or safe refusal. Required phrases use a vertical bar between alternatives.

6 active canaries

#	Canary question	Assertion	Required phrase(s)	Source or tool marker

Exact match is best for IDs and fixed values. Required phrase works for natural language. Safe refusal is for questions the agent should not answer.

Replay baseline and current outputs.

Paste what the agent returned while healthy and what it returns now. The lab checks required language and source or tool markers.

Outputs are user-supplied evidence

#	Question	Healthy baseline	Current output

Remove sensitive data before pasting.The lab stores progress only in this browser. Exported files contain whatever you enter.

For repeated-run testing, run this same pack three to five times and keep each exported evidence record. Reliability includes consistency, not only one successful response.

Research basis, scoring notes, and source limits

Continuous evaluation. Microsoft’s agent lifecycle guidance says continuous evaluation should regularly retest defined cases to catch regressions caused by knowledge, model, or configuration changes. Microsoft Learn, updated July 14, 2026.

Regression design. Microsoft recommends realistic prompts, verifiable assertions, repeat runs, and near-100% consistency for core regression tests. This lab applies those principles to business-owned known answers. Microsoft Learn.

Practitioner signal. Multiple Microsoft community users reported Copilot Studio agents returning no SharePoint matches while sources still appeared Ready. Reports continued through August 21, 2026. No official root-cause analysis or platform-wide incident notice is posted, so Aule treats this as a multi-user community signal, not proof of a broad outage. Microsoft Tech Community thread.

Scoring limit. A pass means the pasted output satisfied the chosen phrase and source-marker assertions. It does not prove factual correctness outside the canary pack. Outputs and thresholds are user-supplied; calculations are deterministic.

Know when the agent stops knowing.

Set the operating target.

Define the known answers.

Replay baseline and current outputs.

Reliability evidence pack.

No current evidence

Canary drift map

Immediate operating response

30-day knowledge reliability protocol

Leadership memo