Platform · Ethics & BiasSix LLM judges, every night, on real traffic

AI ethics & bias,
measured nightly —
not just claimed.

A nightly job samples your production traces and scores them against six ethics criteria — toxicity, demographic bias, sentiment consistency, fairness, misinformation, and privacy leakage. It runs asynchronously and never touches the request path, so live traffic is never slowed or blocked by it.

01 / How the report works

Sampled nightly. Scored 0–100. Flagged under 60.

Guardrails catch what a policy can define in advance. Bias and unfairness rarely announce themselves that cleanly — so Obiguard measures them continuously, on the traffic your agents actually served.
01 — SAMPLE

A nightly random sample

A job runs at 03:00 UTC and randomly samples roughly 17% of the previous day’s traces across the project — a representative read, not a hand-picked one.

03:00 UTC · ~17% of traces
02 — JUDGE

Six evaluator criteria

Each sampled trace is scored by LLM judges on toxicity, demographic bias, sentiment consistency, fairness, misinformation, and privacy leakage — every criterion on a 0–100 scale.

6 judges · 0–100 each
03 — FLAG

Anything under 60 is flagged

A trace scoring below 60 on any single evaluator is flagged for review. Click through to the full prompt and response — the finding is never a number without the conversation behind it.

< 60 · flagged · drill-down
04 — TREND

30-day trend and per-agent scores

An overall score with Good (80) and Risk (60) thresholds, a 30-day trend line, a score-vs-prior-run delta on a 7-day rolling average, and an average per agent — so a single agent regressing does not hide inside the project total.

30-day trend · per agent
05 — NON-BLOCKING

Never on the request path

Evaluation is fully asynchronous. Nothing in this report adds latency to a model call or blocks a response — it observes what already happened.

Async · zero added latency
06 — EVIDENCE

Fairness evidence for your framework

The report is the evidence for framework categories asking whether trustworthy characteristics — safety, fairness, privacy — are actually evaluated, rather than merely asserted in a policy document.

NIST MEASURE 2.1 · ISO 42001 9.1
6
Judge criteria

Toxicity, demographic bias, sentiment consistency, fairness, misinformation, privacy leakage.

17%
Traces sampled nightly

A random sample of the previous day’s traffic, drawn fresh each run.

60
Flagging threshold

Score below 60 on any evaluator and the trace is flagged for human review.

0ms
Added latency

Runs asynchronously after the fact — never on the request path.

03 / FAQ

Ethics scoring,
explained.

What the six judges measure, how sampling works, and why this sits beside guardrails rather than inside them.

Talk to an SE →

What does the Ethics & Bias report measure?[01]

Six LLM judge criteria, each scored 0–100 on a sampled trace: toxicity, demographic bias, sentiment consistency, fairness, misinformation, and privacy leakage. The report shows an overall score with Good (80) and Risk (60) thresholds, a 30-day trend, a score-vs-prior-run delta on a 7-day rolling average, and an average per agent so one regressing agent does not hide inside the project total.

How often does it run, and how much traffic does it sample?[02]

A job runs daily at 03:00 UTC and randomly samples roughly 17% of the previous day’s traces across the project. Sampling is random rather than curated, so the score reflects representative traffic instead of hand-picked examples.

What happens when a trace scores below 60?[03]

Any trace scoring under 60 on any single evaluator is flagged for review and listed on the report. You can open a flagged trace to read the full prompt and response, so a finding is never a number without the conversation behind it.

Does ethics evaluation add latency to model calls?[04]

No. Evaluation is fully asynchronous and runs after the fact against stored traces. Nothing in this report sits on the request path, adds latency to a model call, or blocks a response.

How is this different from guardrails?[05]

Guardrails enforce what a policy can define in advance — a detector match, a keyword, a scope violation — and act on it in-path, in real time. Bias and unfairness rarely announce themselves that cleanly. The Ethics & Bias report measures them continuously after the fact, on traffic your agents actually served, which is what framework categories on trustworthy characteristics ask for: NIST AI RMF MEASURE 2.1 and ISO/IEC 42001 Clause 9 monitoring.
04 / Related

Measured after the fact. Enforced in the moment.

This report observes; inspection and Policy Sets act. To probe an agent before it ever serves traffic, run Project Moonshot evaluations; to turn both into audit evidence, see framework mapping and the Audit Ledger. Everything together is on the Governance AI page.