Quality judge

The quality judge is a Brama chat-completion call that evaluates whether a rewrite preserved the source and remains readable. It is separate from detector scoring: detectors estimate AI probability; the judge protects content quality.

Evaluation shape

The model must return one JSON object:

{
  "meaning_preservation": 0.97,
  "factual_consistency": 0.96,
  "grammar": 0.98,
  "coherence": 0.95,
  "naturalness": 0.93,
  "readability": 0.94,
  "critical_error": false,
  "issues": [],
  "rationale": "Meaning and facts preserved; phrasing is natural."
}

Every numeric field must be finite and in [0,1]. The weighted quality_score is:

0.30 meaning preservation + 0.25 factual consistency +
0.15 grammar + 0.12 coherence + 0.10 naturalness + 0.08 readability

Pass rule

A candidate passes only when critical_error is false and:

• meaning preservation, factual consistency, grammar, and coherence are at least --minimum-quality;
• naturalness is at least minimum_quality - 0.10;
• readability is at least minimum_quality - 0.05.

The default minimum is 0.90.

Response handling

The judge extracts text between the first { and last }, allowing a model to wrap the object in prose or a code fence. It validates the schema and every score. A bad evaluation is retried up to three times; after that the operation fails rather than treating unknown quality as acceptable.

Exact refusals

• minimum quality must be between 0 and 1
• quality judge response contains no JSON object
• quality judge response contains no closing JSON brace
• quality judge response contains malformed JSON boundaries
• quality judge returned invalid JSON schema
• quality judge returned invalid <dimension> score <value>
• quality judge failed to return a valid evaluation after three attempts
• Brama quality evaluation failed

SKRYBA_JUDGE_MODEL selects the judge model; when unset it uses the generator model.