Admost A/B Testing

Admost A/B Testing is a tool for dividing the users in your application into groups and testing them simultaneously with different parameters. This allows you to try out new features such as zone/placement adjustments, network settings, and remote configuration management on a small portion of your users and observe their impacts before applying these changes to your general audience.

New A/B Test

A/B Testing tool can be found on the left menu under the Test tab on the Admost dashboard. Please click “New A/B Test” under the “Manage A/B Tests” tab to specify the details of your test.

First, you will need to choose the application you would like to use for your test and select it as “active”.

It is recommended that you apply the A/B test to about 5-10% of your total users so that your revenues would not be negatively affected. The A/B test will be applied only to the new users that install your app from the date you start the test onwards to prevent any possible deviations caused by other parameters.

You can identify the name of your test, which percentage of your users the test will be applied to, on which dates the test will take place, and (optionally) which countries’ users the test will be open to.

Afterward, please click on the “+Groups” option to create as many groups as you need (e.g. A, B, C) and define the parameters you want to test in these groups.

The number of new users that you set (in percentage) will be distributed randomly to these groups that will evaluate the different parameters of your choosing. The number of users in each group will be the same.

If you would like, you can set parameters only in Group A and do not specify any parameters in Group B to make a comparison between Group A and the current state of your application.

Alternatively, you can set parameters only in Group A and compare its results with the general audience without adding any other groups. You can also select the zone that your test will be applied to.

During your A/B test, it would be more meaningful to choose a single parameter and observe how the changing values in this parameter would affect your groups. If you choose more than one parameter, it would be difficult to identify how each parameter affected your groups.

Parameters

• Zone: This option will allow you to make tests on the parameters below. The parameters will be applied to the zones that you select (and to all zones if you do not select any).

Daily Freq. Capping: For setting a daily impression limit
Hourly Freq. Capping: For setting an hourly impression limit
Impression Interval: Minimum amount of time between two impressions
Impression Delay: Minimum amount of time before an ad is shown to a first-time user (The screen-time while the app is open is taken into account)
Zone Overall Timeout: Maximum amount of time to wait for an ad to load
Adaptive Enabled: For enabling Admob’s adaptive ad feature
Adaptive Cutout: The total distance of an adaptive ad from the sides of the screen (If not specified, the screen width will be taken into account) 

• Networks Exclude: This option will allow you to test how your application and revenue will be affected when one or some of your already existing networks are not in the waterfall. With the help of this parameter, you can make a test on your new users without having to change your waterfall. You can set the zones that your test will be applied to with the “Zones” tab.

• Placement Exclude: This option will allow you to test how your application and revenue will be affected when one or some of your already existing placements are not in the waterfall. With the help of this parameter, you can make a test on your new users without having to change your waterfall. You can set the zones that your test will be applied to with the “Zones” tab.

• Remote Config: You can use remote configuration to make changes and adjustments in your application without having to update your application or make any changes in your database. A/B test will allow you to test the effect of these changes you plan on your users.

Important: The config you define here will override the already existing config you defined on your application earlier, therefore it is required to define all the parameters you would like to use in your application (even if with the same values for all groups) while designing your A/B test.

Cohort Analysis

You can review the results of your A/B test with “Cohort Analysis” on the left menu under the “Analytics” tab on the Admost dashboard. Please select the application the test was applied to, the test you would like to review under the “A/B Tests” option, the dates the test took place, and filter “Test” as the “Cohort Type”.

With this report, you can see how many users were included in the different A/B groups you defined with different parameters, how much revenue you generated from these users, and the average revenue per user (ARPU) for these groups. You can also compare the retention rates, engagement rates, and in-app purchases of the new users who became part of your A/B test.