Skip to main content

Evaluators

Last updated 10/30/2026

Evaluators use LLM Judge to assess the quality of answers or execution evidence, turning scoring criteria into reusable rules. Each scoring attempt always returns Score and Reason.

For scoring rule maintainers and evaluation users. Verification baseline: the project's current implementation as of 2026-10-30. Entry points in deployed environments may vary by version and permissions.

Search keywords: LLM Judge, Prompt, rubric, variables, Score, Reason, evaluator versions.

1. Create and configure an evaluator​

  1. Go to Evaluators, click Create, and enter a name. You can also select Create from template to preview and choose an enabled template.
  2. After creation, open Basic configuration on the details page, select a category and model configuration, and enter the Prompt and description. The initial draft cannot yet be used for evaluation.
  3. Reference variables with double braces, such as {{input}} and {{output}}. Variables are extracted automatically from the Prompt. Names must start with a lowercase letter, contain only lowercase letters, numbers, and underscores, and be no longer than 64 characters.
  4. Check the scoring criteria, model, and variables, then click Save evaluator. A valid save automatically generates a published version. Confirm it in Version history.

2. Define clear scoring criteria​

The Prompt should explain the assessment goal, the meaning of inputs and reference answers, scoring levels, deduction conditions, and output requirements. Do not simply write "Assess whether it is good or bad." The following example can be adapted to your business.

You are an answer consistency evaluator.
Question: {{input}}
Reference answer: {{reference_output}}
Answer to evaluate: {{output}}
Using the reference answer as the basis: assign 1 for complete consistency; 0.5 if correct but missing key points; 0 for clear contradictions.
Return only one JSON object containing a numeric Score (0 to 1) and a nonempty string Reason.

Score ranges from 0—1. Higher scores indicate closer alignment with this evaluator's criteria. Reason explains the basis for the judgment. If the output cannot be parsed, the score is out of range, or Reason is empty, scoring fails for that item and its score remains empty rather than 0. Only LLM Judge is currently supported; custom output fields and Code Evaluator are not supported.

3. How variables receive values​

Variable sourceConfiguration location and rules
Regular variables, such as input and reference_outputBind to a dataset field from the current Case in the evaluation task; the variable and field names do not need to match.
outputUses actual output for online Agent / large language model tasks; for preset-result tasks, you must manually select an existing answer field.
execution_status、tool_summary、artifact_summaryAvailable only for online Agents; the system provides execution evidence.
execution_trace、tool_callsAvailable only for online Agent process evaluation; requires a complete execution trace snapshot suitable for scoring.

Evaluators that depend on tools or execution traces cannot be used for direct model calls or preset results. You can use separate evaluators for the final answer and the sequence of tool calls to explain result quality and process quality independently.

4. Optimize the Prompt and calibrate​

  1. To optimize a Prompt, click Prompt optimization, select an existing model, and generate suggestions. Read the results before clicking Confirm and fill.
  2. Filling updates only the form. You must still save the evaluator to create a new reusable version.
  3. Use Reliability and calibration to maintain samples with known expected scores and check parsing and scoring consistency. See the corresponding subpage for details.

5. Reuse, versions, and deletion​

The list supports searching, copying, and deletion. Copying creates an independent evaluator. Check the name, model, and Prompt before saving. Editing the working copy or publishing a new version does not change tasks bound to older versions or their historical results. Explicitly select the published version each task should use. Later template changes do not automatically update existing evaluators either.

6. Next steps and related pages​

Return to the manual home page:

AI evaluation platform

Was this page helpful?