Skip to main content

Evaluation templates

Last updated 10/30/2026

Evaluation templates provide reusable Judge Prompts. Creating from a template produces an independent evaluator; the template itself does not score tasks.

For business users who use templates and administrators who maintain project templates. Verification baseline: the project's current implementation as of 2026-10-30. Entry points in deployed environments may vary by version and permissions.

Search keywords: built-in templates, project templates, enable, disable, template versions.

1. Create an evaluator from a template​

  1. Go to Evaluators and click Create from template.
  2. Enter an evaluator name, select an enabled template, and preview its purpose and Prompt.
  3. After creation, open the evaluator details, configure the model, and adjust the Prompt for your scenario. Save and confirm the published version.

A template is only a starting point. Even when names match, check which variables it requires, whether it assesses answer quality or execution processes, and what execution evidence it needs.

2. Manage project templates​

  1. Click Manage templates in the evaluator list. As an administrator, click New Template and enter a name, category, Judge Prompt, and optional description.
  2. Variables are extracted automatically from the Prompt using double-brace syntax. Check the name, category, Prompt, and variables before creating the template. The current creation form does not require you to enter a template code manually.
  3. After creation, check the code, version, source, and status in the list. The system retains template versions. Creating a new template does not rewrite existing evaluators.
  4. Disable a project template to stop future use, and enable it to resume use. Built-in templates remain read-only.

For the same code, the latest enabled version in the current project takes priority. If no project version is available, the system can fall back to a built-in template. Disabling a project template does not necessarily make the built-in template with the same code unavailable. Enabling or disabling a template affects only future creation; it does not change existing evaluators or historical scores.

3. Language and applicability limits​

Built-in template names, descriptions, and Prompts follow the UI language. Creating from a built-in template saves its content in the language used at that time. Later UI language changes do not retranslate saved evaluators. Custom project templates retain the user's original text.

The tool-call sequence reasonableness template depends on an online Agent's execution trace and cannot be used for preset results or direct model calls. It cannot establish remote internal processes that are not visible. Incomplete traces may cause that score to fail.

4. Next steps and related pages​

Return to the manual home page:

AI evaluation platform

Was this page helpful?