Quality judge

The quality judge is a Brama chat-completion call that evaluates whether a rewrite preserved the source, remains readable, and matches the requested writing style and output language. It is separate from detector scoring: detectors estimate AI probability; the judge protects content quality and the requested profile.

Evaluation shape

The model must return one JSON object. This is an illustrative schema, not a captured model evaluation:

{
  "meaning_preservation": 0.97,
  "factual_consistency": 0.96,
  "grammar": 0.98,
  "coherence": 0.95,
  "naturalness": 0.93,
  "readability": 0.94,
  "critical_error": false,
  "style_matched": true,
  "language_matched": true,
  "issues": [],
  "rationale": "Meaning, facts, required register, and output language are preserved."
}

Every numeric field must be finite and in [0,1]. Both profile booleans are required; missing fields do not default to a pass. quality_score is the weakest of the six dimensions:

min(meaning preservation, factual consistency, grammar,
    coherence, naturalness, readability)

A rewrite is only as good as the worst thing it does to the passage, so no dimension can buy back another and no weighting of one against another is chosen.

Pass rule

A generated candidate passes only when critical_error is false, style_matched and language_matched are true, and quality_score — every dimension — is at least --minimum-quality. No dimension is allowed to fall below it.

--minimum-quality has no default: rewrite, benchmark and train refuse to start without it. A high numeric score cannot compensate for the wrong style or language.

Style and translation checks

The judge receives the same RewriteProfile as the generator, in its system instructions and as requested_profile beside the source and candidate. Source passages are data, not instructions that can override the profile. Readability is assessed for the requested style and audience.

An explicit pl or en choice permits translation, but not changed facts, lost qualifications, invented evidence, or missing citations. Meaning is assessed across languages, not by matching words. Names, citation identifiers, URLs, code, and important verbatim quotations may remain unchanged without failing the language check. preserve keeps each passage's existing language, including multilingual passages.

When no generated candidate passes, the judge also evaluates the original against the requested profile. Source fallback requires both profile booleans, but does not newly require every numeric threshold. A source fallback can therefore report quality_passed: false. If the source has the wrong register or language, the rewrite refuses instead of returning it as a successful styled rewrite.

Response handling

The judge is asked once per candidate, with no max_tokens or temperature: the Brama route owns unstated sampling and the judge model answers until it ends. Generation separately sends the caller's stated temperature. The judge extracts text between the first { and last }, allowing a model to wrap the object in prose or a code fence, and validates the schema and every score. A reply that is not a valid evaluation fails the operation rather than treating unknown quality as acceptable or asking again.

Exact refusals

• --minimum-quality is required: the quality score a rewrite must reach is the caller's to state
• minimum quality must be between 0 and 1
• quality judge response contains no JSON object
• quality judge response contains no closing JSON brace
• quality judge response contains malformed JSON boundaries
• quality judge returned invalid JSON schema
• quality judge returned invalid <dimension> score <value>
• quality judge did not return a valid evaluation
• Brama quality evaluation failed

A candidate group is judged once per distinct text: two positions holding the same candidate share one evaluation, because there is no cache in front of the judge and the same request would otherwise be paid for twice.

The judge is the component that refuses a rewrite which destroyed its source, so a stand-in judge disables the product's central guarantee. The earlier local-generator walkthrough used such a stand-in and does not qualify style enforcement or translation. Those behaviors require real generator and judge calls, real detector responses, and inspection of the resulting text and report.

SKRYBA_JUDGE_MODEL selects the judge model; when unset it uses the generator model. Both models must support the requested language.