A/B test: personalization and customer experience

Personalization A/B test: personalization and customer experience

Published on 15.06.2021 by Stephan Lamprecht, journalist

Personalization of offers is one of the keys to greater success in digital retail. But what’s the best way for shop operators to find out what goes down well with their customers? A/B testing offers a solution in this area.

The more strongly the offer is tailored to the individual wishes of customers, the better. If users feel that they are being spoken to directly in an app, retention is improved; in a shop, a tailored offer increases the incentive to add something to the shopping basket. A/B testing helps to find out what people actually want.

How an A/B test works

A/B tests take their name from the way they work. Test subjects are presented with two variants of some form of content. This can be a CTA element (call-to-action), a product recommendation or a content module, for example different headlines for the same article in a newsletter.

The actions taken by the subjects are measured. If a direct comparison shows that one of the two variants is better received, that variant is used in the future.

While it is generally possible to compare more than two variants, doing so increases the complexity of the design of the test and its evaluation.

Where A/B testing performs well

A/B testing is suitable as a tool for use in various areas in digital retail:

  • Design and processes: To begin with, there is an idea or a specific question. Is there a measurable difference from the customer’s point of view between a button that says “Buy now” or “Go to order”? Are there any differences at all to be discerned from the position of the button? Amazon, for example, did a great deal of experimentation before positioning the “Buy Box” where it is. Experiments of this kind can also be applied to processes, for example with regard to the sequence of data queries during the ordering process.
  • Prices and recommendations: If a shop operator wants to display product recommendations on the details page, one possible question is whether it makes sense to provide recommendations from the same product category. In an A/B test, variant A of the recommendations displayed contains items from the current category, while variant B includes products from other categories. At its core, almost every recommendation engine works on this principle, with machine learning now being used for variant production and evaluation. A/B testing also constitutes the basis for determining the optimum price.

Tips for implementing an A/B test

For an A/B test to succeed, there are a number of basic things to bear in mind.

  1. Analyse data and formulate hypotheses
    Everything starts with data analysis. For example, are there points in the customer journey where a particularly large number of users cancel? Does the choice of payment method in a shop fall well short of expectations? On the basis of specific figures of this kind, it is then possible to develop hypotheses regarding the changes that can be made to sway behaviour in the desired direction. Another example, not so easy to answer (and much more complex), is the question of whether bolder language in the headlines in a newsletter leads to more interaction.
  2. Define a test scenario
    Based on the hypothesis, a scenario is defined in which two variants of the headlines are always developed. For presentation of the payment methods, the order is changed or other labels for buttons are developed. The important thing here is not to change too many things all at once. This reduces the informative value of the test and can lead to false conclusions.
  3. Pay attention to the samples
    It is important to ensure that the subjects are distributed evenly, and preferably automatically, between the two variants. The results are statistically meaningful only if as large a number of tests as possible are carried out. Drawing your conclusions after only 100 displays of the two variants runs the risk of being fooled by chance.
  4. Define the duration of the test
    It is very important to define the duration of a test in advance. For example, the period within which the variants are to be presented should be specified, or the number of users should be defined. The results are quantified only on reaching the end of the test.
  5. Examine the result critically
    Ideally, the result is that one variant changes the measured variable positively and significantly. The decision in favour of implementation is then (almost) made. It is also possible for the change to be negative and significant. It is then better not to go down that road. If there were no significant differences even after an adequate period of time, the change was not constructive. In the case of cancelled purchases, new hypotheses are therefore required.

Generally speaking, the results of an A/B test must always be critically examined, and must also be seen in the context of the system as a whole. For example, if the hypothesis is that a different colour order button increases the number of orders and the test confirms the hypothesis, the result must be seen in relation to other figures from the company’s point of view. After all, if the number of sales has increased but the value of the shopping baskets has at the same time fallen, implementation of the measures may ultimately lead to less revenue.

Solutions for testing

Thanks to a large number of inexpensive tools, A/B testing is also available to smaller retailers. Google for example offers “Optimize”, a tool for smaller tests that can be used free of charge. And in mailing tools for newsletters, testing functions of this kind are generally included anyway.

Stephan Lamprecht, journalist

Stephan Lamprecht has been following e-commerce developments in Germany, Austria and Switzerland for two decades as a journalist and consultant.

((commentsAmount)) Comments

There was an error during request.
  • (( comment.firstname )) (( comment.lastname )) (( comment.published )) (( comment.content ))

Contact us

Do you have questions for our experts, or do you need advice? We will be only too happy to help!

Contact us