Experiment list
Create an experiment
Basics
Go to Experiment → Experiments and click Create Experiment in the upper-right corner.
Enter the experiment name, group, and experiment hypothesis, and select the experiment type:
- Feature Experiment: Used for experiments on product features, UI styles, backend algorithms, and more. Different experiment groups receive different Feature values.
- External Experiment: Uses the experiment statistical analysis capabilities to evaluate external experiment data (coming soon)
For the experiment hypothesis, we recommend the structure "If a certain change is made, a certain metric will increase or decrease, because the change has a certain effect." For example:
If Manual Sample Switch V2 is enabled, Manual Sample Conversions V2 will increase, because the new feature shortens the user's operation path.
Click Confirm to open the experiment editing page.
Configure traffic, experiment variables, and experiment groups
Before you start, confirm the following:
- You have entered the project where you want to run the experiment, and you have management permissions for Feature, metrics, traffic layers, and experiments.
- You have determined the bucket key of the experiment. The system currently supports user ID (
#user_id), account ID (#account_id), distinct ID (#distinct_id), and device ID (#device_id). - You have completed the tracking design for the experiment result events. When you create an event metric, you need to select an existing business event and its aggregation method.
- You have defined the experiment hypothesis, the control group plan, the experiment group plan, the key metrics, and the expected running duration.
The traffic percentages of all experiment groups should add up to 100%. Click Split Evenly to quickly distribute traffic evenly among the groups.
Set stop conditions and statistical metrics
Set the experiment stop condition on the right. This example selects runtime and sets the experiment to end automatically after 30 days.
Add at least one key metric in Statistical Metrics. Click Add next to key metrics, move Manual Sample Conversions V2 to the selected area, and click Save. Secondary metrics are optional. In addition to key metrics and secondary metrics, Statistical Metrics contains a third type: Guardrail Metrics.
The completed experiment configuration includes:
- The traffic layer and the traffic percentage of the layer.
- Audience conditions.
- Feature parameters, and the control group and experiment group configurations.
- The experiment hypothesis and stop conditions.
- Key metrics and secondary metrics.
The upper-right corner of the page provides two save options:
- Save draft: The experiment enters Draft. You can continue editing it, and it does not take part in rule evaluation.
- Save and Submit: The experiment enters Pending, which completes the configuration submission before it runs.
In this example, click Save and Submit. The experiment ID is 0010, and the status changes to Pending.
Run an experiment
In the actions column of a pending experiment, click the run icon. The system notes that the experiment will take part in rule evaluation once it runs. After confirming, click Run.
After the experiment runs successfully, its status changes to Running, the Feature rules take effect, and the system starts collecting experiment data.
View experiment statistics and configuration
Click the experiment name to open the experiment details.
Experiment Statistics includes:
- The experiment hypothesis and experiment conclusion.
- Key metrics, the performance of experiment groups relative to the control group, P values, and confidence intervals.
- The experiment timeline.
- Statistical test results.
- Group entry data and metric trends.
When an experiment has just started or no users have entered the groups yet, users in groups, P values, and confidence intervals may be shown as 0 or "-". Wait until there are enough experiment samples before you judge the experiment results.
Experiment Configuration shows the basic information, traffic layer, bucket key, experiment traffic, Feature parameters, group ratios, Feature values of each group, and statistical metrics.
View the report and operation log after an experiment ends
After an experiment ends, you can still open Experiment Statistics to view the frozen report. The timeline keeps milestones such as run, pause, resume, and actual end.
Click Log at the top of the details page to view when the experiment was created, submitted, launched, paused, resumed, and ended, along with the corresponding operators and pause reasons.
Experiment list features
Pause and resume an experiment
For a running experiment, you can click the pause icon. You must enter a pause reason before pausing. After the experiment is paused, its rules stop taking effect, data collection is interrupted, and Feature falls back to the default value.
After the experiment is paused successfully, its status changes to Paused.
To continue the experiment, click the run icon again and click Confirm in the Resume window. After it resumes, the experiment rules take effect again.
End an experiment
In the actions column of a running or paused experiment, click the more menu to view the report or end the experiment.
After you click End Experiment, the system notes that:
- The experiment will be terminated permanently and cannot be resumed.
- Feature assignment falls back to the next-priority rule.
- Experiment data is frozen, but the report can still be viewed.
After confirming, click End.
The experiment status changes to Finished.
After the experiment ends, the traffic layer list shows 0% currently used, which means the traffic has been released and is available for subsequent experiments.
The experiment list keeps the Occupied 100% record from when the experiment was configured. To determine whether a traffic layer is currently available, refer to the Used percentage in the traffic layer list.
The details page of an ended experiment provides an Archive entry. After archiving, you can view the experiment through Archived in the upper-left corner of the experiment list.