Push experiments

Last updated 10/02/2026

1. What is A/B testing

A/B testing is an experimental method that compares two or more different strategies, designs, or changes to determine which one has a greater impact on users or the business. A/B testing helps you determine which experiment group gets better user feedback, a higher conversion rate, or better business performance.

In an A/B test, the subjects are randomly divided into two (or more) groups: one is the control group and the others are experiment groups. The control group uses the current strategy or design, while the experiment groups use the changed strategy or design. Both groups of users receive different versions of the experience during the same period.

By comparing user feedback, behavior, or business metrics between the groups, you can determine which version performs better. This testing method can validate all kinds of changes, such as different push copy, different game gift pack prices, different ways of reaching users, or feature changes.

The goal of A/B testing is to make decisions based on real data and user feedback. Through continuous testing and optimization, you can gradually improve the user experience, increase conversion rates, and grow business revenue.

2. Feature overview

The AE Engage module supports A/B testing on Message Configuration. By configuring multiple sets of content in the same operation task and comparing their performance, it helps operators make data-driven decisions on the best strategy and improve target conversion.

3. How to use

3.1 Turn on A/B testing

In the Operation Tasks module, click + Create task to open the operation task creation page

Then, in Configuration, click Turn On A/B Test. The operation task then enables the split-flow test settings.

3.2 Choose an experiment type

Triggered tasks can use either type of A/B test: Split-flow Test or Horse-racing Test. Tasks of other push types (Manual, Scheduled - One-time, Scheduled - Recurring) support only split-flow tests.

3.2.1 Split-flow test

• Users who meet the target audience conditions are randomly assigned to multiple experiment groups according to the sampling and grouping ratios, and multiple plans are pushed at the same time for comparison to validate their effect.
• Split-flow tests suit scenarios where you compare the effect of push strategies across groups. They don't support a subsequent push to all users.

3.2.2 Horse-racing test

Currently, only triggered operation tasks support horse-racing tests.

• Before pushing the official plan, test multiple plans on a small scale in experiment groups. After the experiment identifies the best plan, push it to all target users.
• Horse-racing tests suit scenarios where you need to push to all users after the experiment.
• Horse-racing tests let you set an automatic push to all audience strategy before the experiment starts or midway through it, so that once the experiment produces a winning group, the winning group's content strategy is pushed to all audience.

3.3 Configure a split-flow test

• Sampling Ratio

After you turn on A/B testing, you can manually set the sampling ratio, that is, randomly select a certain proportion of the target users to take part in the A/B test. Other users won't be reached.

The sampling percentage can be an integer in (0, 100]. You can set it by dragging the slider or entering a value.

• Runtime

The runtime of a split-flow test is the same as the task duration, and the experiment runs at the same time as the push task.

• Winning Metric

You can also set the winning metric of the A/B test. Currently, you can choose Same as Primary goal of the operation task or Click-through rate. To use Click-through rate, you must first configure the Clicked event in the funnel setting of the corresponding channel in Channel Management.

• Activation Event

By default, the winning metric is calculated based on users who were pushed successfully. In practice, there may be some traffic loss between a successful push and experiment exposure. In such cases, you can add an activation event. After you add it, the system treats users who were "pushed successfully and completed the activation event" as the users entering the experiment, and then analyzes whether their experiment metrics differ significantly.

• Experiment group settings

After completing the settings above, you also need to configure the content for the control group and the experiment groups in Message Configuration.

The control group uses the current strategy or design, while the experiment groups use the changed strategy or design. Both groups of users receive different versions of the experience during the same period.

• Control Group

Each A/B test has 1 control group by default, and you can choose whether the control group reaches users.

If the control group reaches users, it works the same way as the experiment groups, so you can observe how users respond to different content during the same period.

If the control group doesn't reach users, you can use it to evaluate whether applying the operation strategy to users is effective, by observing whether the experiment groups that reach users show a clear improvement in target conversion over the control group.

• Experiment Groups

You can set up to 4 experiment groups. Click the small gear icon on the right to configure them: you can add and delete experiment groups and customize their names. You can also drag the slider to allocate traffic among the experiment groups. Based on your target audience, the sampling ratio, and the traffic allocation ratios, the system estimates how many users each experiment group will reach.

3.4 Configure a horse-racing test

• Sampling Ratio

After you turn on A/B testing, you can manually set the sampling ratio, also known as the gradual rollout sampling ratio. It means randomly selecting a certain proportion of the target users for the A/B test. Other users won't be reached.

The sampling percentage can be an integer in (0, 100]. You can set it by dragging the slider or entering a value.

• Runtime

A horse-racing test usually runs in the first half of the operation task's duration.

A horse-racing test starts together with the operation task. You need to set the experiment's end time yourself, and it can't be later than the end time of the operation task. After the horse-racing test ends and a winning experiment group is identified, you can push the winning strategy to all users for the rest of the task duration.

• Winning Metric

You can also set the winning metric of the A/B test. Currently, you can choose Same as Primary goal of the operation task or Click-through rate. To use Click-through rate, you must first configure the Clicked event in the funnel setting of the corresponding channel in Channel Management.

• Experiment group settings

After completing the settings above, you also need to configure the content for the control group and the experiment groups in Message Configuration.

The control group uses the current strategy or design, while the experiment groups use the changed strategy or design. Both groups of users receive different versions of the experience during the same period.

• Control Group

Each A/B test has 1 control group by default, and you can choose whether the control group reaches users.

If the control group reaches users, it works the same way as the experiment groups, so you can observe how users respond to different content during the same period.

If the control group doesn't reach users, you can use it to evaluate whether applying the operation strategy to users is effective, by observing whether the experiment groups that reach users show a clear improvement in target conversion over the control group.

• Experiment Groups

You can set up to 4 experiment groups. Click the small gear icon on the right to configure them: you can add and delete experiment groups and customize their names. You can also drag the slider to allocate traffic among the experiment groups. Based on your target audience, the sampling ratio, and the traffic allocation ratios, the system estimates how many users each experiment group will reach.

• automatically select the strategy and Push to all audience with it

You can turn on the automatically select the strategy and Push to all audience with it switch and select the experiment group to push to all audience. Once this is set, when the horse-racing test ends, if a best-performing (winning) experiment group emerges, the winning group's strategy is automatically pushed to all audience. If no winning group emerges, the fallback strategy group you selected is pushed to all audience.

If you don't turn on the automatically select the strategy and Push to all audience with it switch, you can go to the A/B Test Report page after the experiment ends and choose to push to all audience manually.

4. Experiment status

Note: Operation task status is the parent of Test Status. The status of the operation task affects the experiment status. For example, if the operation task is paused, the experiment is paused; if the operation task ends, the experiment ends.

• Operation task statuses are Draft, Approving, In progress, Paused, Finished, and Rejected

• Experiment status describes the current state of the A/B test. The statuses are:

  • Not Started: an experiment that hasn't been pushed yet while the operation task isn't finished
  • Experiment Canceled: an experiment that was closed without ever being pushed when the operation task became Finished
  • In Experiment: the operation task is Running and the experiment is still within the target conversion window
  • Paused: when the operation task is Paused, the experiment also enters the Paused status
  • Data Collecting: the operation task is Finished, but the target conversion window hasn't ended yet, so the experiment's data is still being counted dynamically
  • Report Completed: the operation task is Finished and the experiment has reached the end of the target conversion window. The experiment report is complete, and the experiment officially ends.

• In addition, in a Horse-racing Test, if you turned on the strategy of automatically pushing to all audience when the experiment ends, the push to all audience starts when the experiment ends. If you chose the manual push to all audience strategy, the experiment enters the Push Waiting status when it ends.

5. Experiment report

In the A/B test report module, you can visually compare the result data of the control group and each experiment group. You can not only determine the winning experiment group from precise statistics, but also observe how the data changes throughout the experiment.

5.1 Open the report

For an operation task with A/B testing turned on, you can view the A/B test results on the A/B Test Report tab of the operation task details page.

5.2 Report overview

5.2.1 Basic information

Split-flow tests and horse-racing tests share the same basic information layout.

The basic information in the experiment report includes experiment type, duration, experiment status, winning metric, number of users pushed successfully, and number of experiment groups.

• Experiment type: the type of the A/B test, that is, a split-flow test or a horse-racing test

• Duration: the time from the start of the experiment to the completion of the report

  • Default start time: the experiment start time
  • Default end time: today; the latest selectable time is the report completion time
  • The time filter controls the whole experiment report and also affects the experiment group data, the winner determination, and the change rates in the line chart.
  • The duration can't exceed 90 days

• Experiment status: describes the current state of the A/B test. For details, see Experiment status description

• Winning metric: the key metric for determining the winning group in the experiment

• Users pushed successfully: the total number of users pushed successfully across all experiment groups in the A/B test

• Number of experiment groups: the total number of control and experiment groups in the A/B test

5.2.2 Statistical conclusions

• Push Succeeded: the number of users pushed successfully in each experiment group
• Activated Users (when an activation event is added): the number of users who were pushed successfully and completed the activation event within the winning metric window (after an activation event is added, the number of achieved users and the metric achieve rate are calculated based on activated users).
• Number of Achieved Users: the number of users, among those pushed successfully (or activated), who completed the conversion of the Winning Metric
• Metric Achieve Rate: number of achieved users / number of users pushed successfully (or activated)
• Lifted by: the rate at which the experiment group's metric rises or falls compared with the control group's metric
• Rules for determining the winner

The experiment determines the winning experiment group by its "lift rate". The lift rate is the rate at which the experiment group's metric rises or falls compared with the control group's metric. The experiment group with the highest lift rate is the winning group. If no experiment group clearly improves the metric achieve rate compared with the control group, the experiment has no winning group.

• Confidence Interval of Improvement Rate: the range within which the true value of the metric lift rate is likely to fall, based on a two-sided test at a 5% significance level

Calculation logic:

• Experiment group metric: M(experiment); experiment group sample size: U(experiment)
• Control group metric: M(control); control group sample size: U(control)

First, get the standard deviations of the experiment group and control group samples. Because these are ratio metrics, the standard deviations are

$${σ(实验)} = \sqrt{M(实验)*(1-M(实验))\over U(实验)}$$ $${σ(对照)} = \sqrt{M(对照)*(1-M(对照))\over U(对照)}$$

Calculate the lift value and lift:

$$提升值 = {M(实验) \over M(对照)} - 1$$ $${σ(lift)} = {M(实验) \over M(对照)}* \sqrt{(\frac{{σ(实验)}}{M(实验)})^2 +(\frac{{σ(对照)}}{M(对照)})^2}$$

From the results above:

$$置信区间上限 = 提升值 + 1.96*{σ(lift)}$$ $$置信区间下限 = 提升值 - 1.96*{σ(lift)}$$

Here, 1.96 is a variable converted from the confidence level. The default is a 95% confidence level

• Conclusion

  • If the lower bound of the confidence interval for the experiment group's percentage lift is negative and the upper bound is positive: no significant change
  • If the lower bound of the confidence interval for the experiment group's percentage lift is positive and the upper bound is positive: significant increase
  • If the lower bound of the confidence interval for the experiment group's percentage lift is negative and the upper bound is negative: significant decrease

• Action

1. View the specific push content of each experiment group
2. When the experiment report produces a winning group, you can click the copy button next to the experiment group to create a new operation task that pushes to all users.

5.2.3 Metric trends

In Data Details, you can see the push details and goal completion of each experiment group from a batch perspective

You can also view the trend chart of the metric achieve rate at Daily granularity

5.2.4 Horse-racing test rollout settings

In the task settings of a horse-racing test, whether you turn on the automatically select the strategy and Push to all audience with it switch determines how the push to all audience works after the experiment ends.

1. Adjust the push strategy

Until the experiment enters the Experiment ended status, that is, Data collecting or Completed-Report ready, you can use Adjust Strategy to change the experiment group that is finally pushed to all audience.

Tip: You can't adjust the push strategy of an experiment that has already ended.

Before the horse-racing test ends

Before the horse-racing test ends

1. Automatic push to all audience

When you turn on the automatically select the strategy and Push to all audience with it switch, after the horse-racing test ends, all users are pushed automatically according to the strategy you set.

After the horse-racing test ends

1. Manual push to all audience

If you don't turn on the automatically select the strategy and Push to all audience with it switch, the task enters the Push Waiting status after the horse-racing test ends. You need to select the experiment group to push to all audience and click Push To All Audience.

After the horse-racing test ends