Experiments
Overview
Experiments are a type of experience that enable you to test different variations of your content to determine what resonates best with your visitors. You can set up one or multiple variants per Experiment, show them to all users or just specific Audiences, and analyze the performance either with your own tools or the Experience Insights.
Create an experiment
To create an Experiment:
Log in to the Contentful web app.
Navigate to the "Content" tab.
Click Add entry, and create a
Ninetailed Experience entry type.Choose Create experiment in the config field.
Enter a title.
Optional: Enter a description.
Configure the options for your experiment: Primary metric, Distribution, Traffic allocation, Audience and Components.
Primary metric
The primary metric determines how the success of an experiment is measured, or what defines success. Selecting a primary metric:
Helps your team understand what the goal of your experiment is.
Automatically selects that metric in the Insights section.
Enables the multi-armed bandit distribution type.
Distribution
The distribution of an Experiment decides what percentage of users that are part of this Experiment are allocated to which variant. You can also add or remove variants, enabling you to run A/B/n tests. Every Experiment consists of a control and at least one variant. The control represents your baseline content.
NOTE: Variants are maintained per user, as long as the variant distribution is not changed for the given Experiment. This means that each user will always see the same variant of an Experiment. If the variant distribution is changed, some users might be re-allocated to see another variant.
You can choose from the following distribution models:
manual distribution
even split distribution
multi-armed bandit distribution.
Manual distribution
Manually select what percentage of traffic should be distributed to each individual variant. The sum of all values must equal 100%. If you do not want to allocate all of your traffic towards the experiment in general, you can use traffic allocation.
Even split distribution
The traffic will be split evenly among all variants.
Multi-armed bandit distribution
Contentful Personalization can automatically and gradually allocate more traffic towards the winning variant. This smart capability is referred to as the multi-armed bandit approach, a theoretically optimal way of distributing traffic among variants. The approach follows the reasoning that more traffic should be allocated to a variant as soon as there is evidence the variant is superior. Using the multi-armed bandit approach, the variant distribution is updated based on Experience Insights derived from your selected primary metric.
NOTE: To choose multi-armed bandit distribution, you must first select a primary metric.
In practice, the distribution follows the Probability To Be Best (PTBB) for the selected primary metric, for example, a variant with 80% PTBB will receive 80% of traffic.
Important: The variant distribution is evaluated and updated once per day.
Weighting
Weighting is based on Probability To Be Best (PTBB). As soon as the PTBB for variants is calculated, re-weighting starts. The calculation process starts immediately.
The minimum threshold for the PTBB is 5% per variant.
To test your setup, we recommend you create a separate development environment and run your experiments. Note that the dashboards are not real-time. They get updated once per hour and show conversions, views, conversion rate, uplift per variant, and PTBB.
Traffic allocation
Traffic allocation allows you to allocate only a portion of all visitors that would be eligible for this Experiment.
NOTE: By default, 100% of your eligible traffic is allocated to your Experiment.
Example
Imagine you have an Experiment running that is only shown to a "Returning Visitors" Audience, with a 50/50 split between two variants and a traffic allocation of 30%.
We now have 1000 users visiting your website, and 200 of them are returning visitors. The traffic allocation will allocate 30%*200 users = 60 users. Each variant will be displayed to 50% of these users (30 users each).
NOTE: Similar to variant distribution, a user is permanently either allocated to an Experiment or not. Only if the traffic allocation changes, some users are reallocated. If the traffic allocation is increased, all users that have already seen the Experiment will be kept in that Experiment.
Difference between traffic allocation and the control group
Visitors not participating in an experiment and those participating in the control group of an experiment may see the same content. This case arises when some of your visitors are not assigned to an experience. You should recognize those users as statistically distinct from one another. If you were to consider these users as belonging to a single group and compared their conversion rate against those seeing experiment variant content, you may introduce base rate fallacies in your analysis.
The control group helps you understand the performance uplift of your variants. Only visitors that are specifically chosen to see this control group are counted towards your control group in any analytical tools such as Experience Insights.
If you are running multiple Experiences on the same component, being in the control group will exclude the visitor from any other Experiences that have a lower priority at that time. In contrast, being excluded from an Experiment (either due to traffic allocation or Audience eligibility) will still evaluate any other Experiences that have a lower priority and potentially show a variant for these.
Audience
Choosing an Audience will set an eligibility requirement for this Experiment. Therefore, only users that are eligible for this Audience at that moment will see that experiment.
Components
Mapping components enable you to choose which entry should be exchanged with another one. Depending on the number of variants you have set up in your distribution configuration, you can map one or several variants.