Evaluation datasets
Evaluation datasets store reusable test samples. Define fields and Cases first, then specify how each field is used in an evaluation task.
For users who need to create, import, and maintain test data. Verification baseline: the project's current implementation as of 2026-10-30. Entry points in deployed environments may vary by version and permissions.
Search keywords: Dataset, Case, CSV, XLSX, JSONL, fields, data versions.
1. Create an empty evaluation dataset
- Go to Evaluation datasets, click New evaluation dataset, enter a name and optional description, and click Create evaluation dataset.
- On the Fields tab of the details page, click Add field, enter a name and select a type, then click Save new field.
- Go to the Sample tab, click Add sample, enter data for each field, and click Save new sample. Save the fields before adding their samples.
| Field type | How to enter values |
|---|---|
| TEXT | Text; suitable for questions, reference answers, and scenario labels. |
| NUMBER | Valid numbers; do not enter plain text directly into numeric columns. |
| BOOLEAN | true or false; select an option in the editor. |
| JSON | A valid JSON object, array, or value. |
| DATE | A valid date as required by the UI; text validation uses YYYYMMDD. |
| TIME | A valid time in HH:mm:ss format. |
2. Import from a file
- Click Import evaluation dataset, enter a name and description, and select a CSV, XLSX, or JSONL file.
- Confirm the file and click Start importing. A successful submission only means asynchronous processing has started. Wait for the import to finish on the details page.
- Check the detected fields, types, and sample count. If the import fails, read the issue description. If the page provides a row-by-row error report, download it, correct the file, and import again.
The maximum file size is 50 MiB, with up to 100000 samples and 200 fields. CSV files must use UTF-8 encoding, with nonempty, unique field names in the first row and the same number of columns in every row. Each nonempty line in JSONL must be a separate JSON object with no duplicate keys. To preserve numeric, Boolean, or structured types, prefer JSONL and verify the import results.
General dynamic fields does not require fixed business columns. Agent task case V1 validation requires case_id and prompt. Select a case specification only if your version's UI offers that option. Do not assume prompt is required for a general file.
3. Edit fields and samples
- Rename, reorder, or delete fields on the Fields tab. New fields remain local drafts until you confirm and save them.
- Add, edit, or delete Cases on the Sample tab. Fields and samples are saved separately. Saving one section does not automatically submit drafts in the other.
- Batch additions are submitted one at a time. If only some succeed, saved content is retained. Process only the remaining drafts to avoid duplicate additions.
Check the content before deleting fields or samples. Historical tasks use frozen snapshots, so changes to the working copy do not affect past evaluations. If simultaneous editing causes a conflict, preserve your input before refreshing and checking the latest content.
4. Understand working copies and versions
Use the working copy to continue editing. A valid save generates a read-only version. New tasks bind to published versions, not unsaved local drafts. Go to Version history to view version numbers, publication times, and frozen content. No new version is generated when there are no substantive changes.
If you see Changes saved, but version publication failed, use Retry version publication. The data is already saved, so do not add the fields or samples again. Later saves do not overwrite old versions or update historical Runs.
5. Export and view references
- Click Export on the details page, select CSV or JSONL, and wait for the background process to generate the file for download. Prefer JSONL when you need to preserve the meaning of structured fields.
- Open Reference records to view tasks that use the evaluation dataset, then open the corresponding task to check its results and frozen version.
The list supports searching by name and filtering by status. Deleting an evaluation dataset places it in a recycled state while retaining historical results. This does not mean the UI offers a restore option. AI Monitor online import is not currently exposed as a default operation in this version.
6. Next steps and related pages
- Use the data for evaluation: Create evaluation tasks and map fields.
- Resolve editing or visibility issues: Permissions and members.
Return to the manual home page:

