Skip to main content

AI evaluation platform

Last updated 10/30/2026

The AI evaluation platform helps teams validate the quality of Agents, large language models, or existing answers in batches: prepare samples in evaluation datasets, define criteria with LLM Judge evaluators, create evaluation tasks to run them, and review scores, reasons, and execution evidence.

For business testers, Agent / model maintainers, scoring rule maintainers, and project administrators. This manual is organized by reader tasks, with a fixed entry point for each topic and links to related pages across modules.

Initial verification date: 2026-10-30. This content is based on the project's current product materials, page implementations, and permission rules. Deployed versions may differ; check which entry points are available in your version.

1. Get started​

Follow Quick start to complete a minimal evaluation, then switch to an online Agent or large language model as needed.

2. Choose by goal​

3. Common tasks​

4. Understand key concepts​

ObjectWhat it addresses
Evaluation dataset / CaseWhich data to test with; a Case is one sample.
Model / Agent configurationWhich service to call, and which identity and parameters to use.
Evaluator / LLM JudgeWhich rules to score by; always returns Score (0—1) and Reason.
Task version / RunHow the configuration is saved, and the status, snapshot, and results of an actual execution.

5. Find instructions​

See Evaluation datasets for data import and versions; Evaluators for scoring rules, templates, and calibration; Evaluation tasks for input templates, field bindings, and run results; Implementation for service connections and credentials; and Permissions and members for sharing and member management. UI paths are marked in bold. Example field names and values do not represent built-in business rules.

6. Scope and result limitations in this version​

  • Supports three types of evaluation targets: Agents, large language models, and preset results, as well as evaluations on fixed schedules.
  • A scoring failure or missing score is not a score of 0. Average scores use only valid scores.
  • Published asset versions and Run snapshots preserve historical facts. Later edits do not automatically change historical results.
  • Execution traces explain only the evidence actually collected in this evaluation. They do not provide complete model reasoning or a general online observability platform.
  • The current default pages do not expose AI Monitor import or data reporting configuration. Creating Azure models or OpenCode Agents is also outside the operational scope of this version.
  • Automated release gates, general manual review, Code Evaluator, and automatic cross-version comparisons are not available.

Parent manual: Agentic Engine.

Was this page helpful?