Skip to main content

Create evaluation tasks and map fields

Last updated 10/30/2026

Combine data, an evaluation target, and scoring rules into an executable task. The page is organized into Task Configuration, Evaluators, Field Mapping, and Advanced Settings. Once configured, you can save a draft or start the task.

For users who have prepared evaluation datasets and scoring rules. Verification baseline: the project's current implementation as of 2026-10-30. Entry points in deployed environments may vary by version and permissions.

Search keywords: new task, Agent, large language model, preset results, field mapping, input template, random sampling.

1. Prerequisites​

  • The evaluation dataset has finished importing, contains samples, and has a published version.
  • The scoring model is configured, and the evaluator has been fully saved with a published version.
  • Online Agent mode requires an accessible Agent configuration. Large language model mode requires a model under test. Preset-result mode requires only existing answers in the data.

2. Select the evaluation target and versions​

  1. Go to Evaluation tasks → New evaluation task and enter a task name and optional description.
  2. Select the evaluation target, dataset, and published dataset version. For online modes, also select an Agent or model configuration.
  3. Select 1—8 evaluators and choose a published version for each. Use View rules to read the full Prompt.
Evaluation targetExecution modeUse cases
AgentCalls the Agent for each Case, then sends the result to the Judge for scoring.Validate answer quality or, when evidence supports it, tool execution processes.
Large language modelCalls the model for each Case, then sends the result to the Judge for scoring.Validate model answers to specified inputs; no Agent tool evidence is produced.
Preset resultsReads existing Case content and sends it directly to the Judge.Evaluate offline answers, historical outputs, or results prepared by other systems.

3. Complete field mappings​

Variables required for calls or scoring appear on the left. On the right, select their sources from the current dataset version. Display names may differ from field keys. Template references must use the actual field keys provided in the UI.

Target variableOnline Agent examplePreset-result example
Agent inputBind the question field.No Agent input is configured.
Judge inputBind the question field.Bind the question field.
Judge outputThe system injects the Agent's actual output.Manually bind the answer field.
Judge reference_outputBind the reference_answer field.Bind the reference_answer field.

The field names above are examples only. Use the actual fields in your data. Map all required Agent inputs and bind every regular variable for each evaluator. Similar variable names do not imply automatic associations. Each evaluator variable binds to one field.

Large language model mode first attempts to detect the main input field automatically. If detection is ambiguous, select an input field or use an input template. The model's actual answer is automatically supplied to output. In preset-result mode, output has no system-output meaning; select its source just like any other regular variable.

execution_status, tool_summary, artifact_summary, execution_trace, and tool_calls depend on Agent execution. Large language model and preset-result modes cannot use evaluators that depend on this evidence.

4. Combine multiple input fields​

  1. To combine context and a question, open the template editor for the corresponding input.
  2. Copy placeholders from the available fields and write an input template in the following format.
  3. Check the notices for unknown fields, placeholder syntax, and length, then save the template. The template replaces the single-field input for that item and is frozen in the run snapshot.
Business context: {{context}}
User question: {{question}}

An online Agent's TEXT input template can reference only TEXT fields. A large language model input template can reference fields of any type. Templates must reference valid fields and cannot exceed 16000 UTF-8 bytes. Validation and startup are unavailable until the current version's fields are loaded. Nonexistent fields produce explicit errors. If a Case lacks a value for a declared field in a model template, that placeholder renders as empty text. Check whether the data meets your business expectations.

5. Sample, save a draft, and start​

  1. Under Advanced Settings, you can enter the number of Cases to sample randomly. Leave it blank to run all Cases. If specified, it must be an integer greater than 0 and no greater than the total sample count.
  2. Check the sample count and estimated scoring count. The estimate equals the number of samples in this run multiplied by the number of evaluators; it does not mean all scores will succeed.
  3. Click Save draft if the configuration is incomplete. When ready, click Confirm and start evaluation and wait for the results page.
  4. If startup fails, read the notice and correct the issue. The draft is retained. Check the actual task status first to avoid duplicate submissions.

Each start samples randomly and independently. To use a fixed sample set, use a dedicated dataset version. The page may display a Report to ThinkingAI switch, but the data reporting configuration entry point is not currently available. Leave it off by default for your first evaluation. For deployed reporting, have an administrator verify the configuration before use.

A task can carry up to 5 frozen result filter fields (TEXT, NUMBER, BOOLEAN). The creation page in this version has no separate selection control. Do not assume every dynamic column is filterable by default; the results page uses the actual frozen configuration.

6. Next steps and related pages​

Return to the manual home page:

AI evaluation platform

Was this page helpful?