AI evaluation platform
The AI evaluation platform helps teams validate the quality of Agents, large language models, or existing answers in batches: prepare samples in evaluation datasets, define criteria with LLM Judge evaluators, create evaluation tasks to run them, and review scores, reasons, and execution evidence.
For business testers, Agent / model maintainers, scoring rule maintainers, and project administrators. This manual is organized by reader tasks, with a fixed entry point for each topic and links to related pages across modules.
Initial verification date: 2026-10-30. This content is based on the project's current product materials, page implementations, and permission rules. Deployed versions may differ; check which entry points are available in your version.
1. Get started
Follow Quick start to complete a minimal evaluation, then switch to an online Agent or large language model as needed.
- Complete your first evaluation: Quick start.
2. Choose by goal
- Prepare, import, and maintain test samples: Evaluation datasets.
- Define scoring criteria and validate rules: Evaluators.
- Start evaluations, troubleshoot issues, and schedule runs: Evaluation tasks.
- Configure models, Agents, and credentials: Implementation.
- Manage sharing scope, members, and access permissions: Permissions and members.
3. Common tasks
- Select an Agent, large language model, or preset results and bind inputs: Create evaluation tasks and map fields.
- Understand valid scores, low scores, failures, and Traces: View results and execution traces.
- Run evaluations hourly, daily, weekly, or monthly: Scheduled tasks.
- Build rules from built-in or project templates: Evaluation templates.
- Check whether a Judge meets expectations: Reliability and calibration.
4. Understand key concepts
| Object | What it addresses |
|---|---|
| Evaluation dataset / Case | Which data to test with; a Case is one sample. |
| Model / Agent configuration | Which service to call, and which identity and parameters to use. |
| Evaluator / LLM Judge | Which rules to score by; always returns Score (0—1) and Reason. |
| Task version / Run | How the configuration is saved, and the status, snapshot, and results of an actual execution. |
5. Find instructions
See Evaluation datasets for data import and versions; Evaluators for scoring rules, templates, and calibration; Evaluation tasks for input templates, field bindings, and run results; Implementation for service connections and credentials; and Permissions and members for sharing and member management. UI paths are marked in bold. Example field names and values do not represent built-in business rules.
6. Scope and result limitations in this version
- Supports three types of evaluation targets: Agents, large language models, and preset results, as well as evaluations on fixed schedules.
- A scoring failure or missing score is not a score of 0. Average scores use only valid scores.
- Published asset versions and Run snapshots preserve historical facts. Later edits do not automatically change historical results.
- Execution traces explain only the evidence actually collected in this evaluation. They do not provide complete model reasoning or a general online observability platform.
- The current default pages do not expose AI Monitor import or data reporting configuration. Creating Azure models or OpenCode Agents is also outside the operational scope of this version.
- Automated release gates, general manual review, Code Evaluator, and automatic cross-version comparisons are not available.
Parent manual: Agentic Engine.

