A nightly job samples your production traces and scores them against six ethics criteria — toxicity, demographic bias, sentiment consistency, fairness, misinformation, and privacy leakage. It runs asynchronously and never touches the request path, so live traffic is never slowed or blocked by it.
A job runs at 03:00 UTC and scores a deterministic, hash-based sample of roughly 17% of the previous day’s trace IDs across the project — not hand-picked, and repeatable.
Each sampled trace is scored by LLM judges on toxicity, demographic bias, sentiment consistency, fairness, misinformation, and privacy leakage — every criterion on a 0–100 scale.
A trace scoring below 60 on any single evaluator is flagged for review. Each flagged row stores an excerpt of the prompt and response (about 500 characters) with a link to the full trace — the finding is never a number without the conversation behind it.
An overall score with Good (80) and Risk (60) thresholds, a 30-day trend line, a score-vs-prior-run delta on a 7-day rolling average, and an average per agent — so a single agent regressing does not hide inside the project total.
Evaluation is fully asynchronous. Nothing in this report adds latency to a model call or blocks a response — it observes what already happened.
Scores, trends and flagged traces are kept in the portal as a record that fairness and privacy are actually measured on live traffic. The judge is an LLM, so results can vary between runs. Completed runs count as partial evidence for NIST MEASURE 2.1 and ISO/IEC 42001 clause 9 in Framework Mapping.
Toxicity, demographic bias, sentiment consistency, fairness, misinformation, privacy leakage.
A deterministic, hash-based sample of the previous day’s trace IDs, taken each run.
Score below 60 on any evaluator and the trace is flagged for human review.
Runs asynchronously after the fact — never on the request path.
What the six judges measure, how sampling works, and why this sits beside guardrails rather than inside them.
Talk to an SE →This report observes; inspection and Policy Sets act. To probe an agent before it ever serves traffic, run Project Moonshot evaluations; for coverage against recognised frameworks and the audit trail, see framework mapping and the Audit Ledger. Everything together is on the Governance AI page.
