Reliability and calibration
Use historical call signals and samples with known expected scores to assess whether an evaluator is stable and meets your business scoring expectations. The results highlight risks.
For rule maintainers with permission to edit evaluators. Verification baseline: the project's current implementation as of 2026-10-30. Entry points in deployed environments may vary by version and permissions.
Search keywords: Golden Case, sample library, expected score, tolerance, score deviation, reliability.
1. Check run stability
- Open the evaluator details and go to Reliability and calibration.
- Review call and parsing performance over the last 30 days, the number of scoring attempts in a terminal state, call success rate, retries, and latency.
- Distinguish No observations yet from failure. No scoring records does not mean a success rate of 0, nor does it prove that the evaluator is reliable.
2. Add calibration samples
- In Sample library, click Add sample. When you add the first sample, the system automatically prepares the internal sample library. You do not need to create a calibration suite manually.
- Enter Test variables (JSON), including all variables required by the current Prompt and the questions, answers, and reference content you prepared.
- Enter an expected score from 0—1 and an allowed error. After saving, confirm that the sample appears in the saved list.
- Prepare high-score, low-score, and boundary samples. Unsaved content in the form does not participate in calibration.
Example variables:
{"input":"Can all orders be refunded?","output":"All orders can be refunded unconditionally.","reference_output":"Refund eligibility depends on the order conditions and refund rules. Not all orders can be refunded."}
For example, with an expected score of 0 and an allowed error of 0.1, the absolute difference between the actual and expected scores must be no greater than 0.1 to meet this expectation. These values are for demonstration only and should be determined by your business rules.
3. Run calibration and interpret results
- Click Run calibration and wait for it to finish. Do not submit again while it is running.
- Review each sample's expected score, actual score, deviation, tolerance, and Reason. Distinguish Deviation exceeds tolerance from Execution error.
- Use Show only samples requiring attention to locate issues first, then view all results. Adjust scoring rules based on the parsing success rate, alignment pass rate, mean absolute error, and maximum absolute error.
- Recalibrate after changing the model or Prompt. If a past calibration is marked as expired, do not continue to treat its results as validation of the current configuration.
4. Maintenance and limitations
Confirm your selection before deleting samples. Deleted samples no longer participate in future calibration, but existing calibration records are unaffected. Passing a few samples demonstrates performance only on those samples and does not prove coverage of all scenarios. Calibration does not block evaluation tasks or automatically decide whether to release an Agent. It is not a general manual annotation platform.
5. Next steps and related pages
- Adjust the Prompt or model: Evaluators.
- Investigate scoring failures in actual evaluations: View results and execution traces.
Return to the manual home page:

