eBay
A/B testing for improved online performance reporting
The Challenge...
eBay Inc. is an internet auction and shopping website that was established in 1995. With more than 100 million active users globally, eBay is the world’s largest online marketplace, and one in which people and businesses can buy and sell a very broad range of goods and services.
eBay’s challenge is to understand how changes made to the website can help to increase revenue by altering visitors’ behaviour. Even a tiny change to behaviour can make or lose hundreds of thousands of pounds, so it is very important to be able to quickly and reliably detect very small changes. As a result, eBay are interested in any techniques that might improve their A/B testing, for instance by giving them additional insight into customer behaviour, or allowing more tests to be run more rapidly.
THE SOLUTION...
CACI’s expertise was sought by eBay in order to create a framework for Bayesian significance testing for their webpage A/B testing. They chose us because we've been working in data analytics since 1996 and have delivered insight into a multitude of challenging business areas across all sectors.
CACI worked closely with the eBay team throughout the research and development stages to create and benchmark a new Bayesian framework for reporting. This framework provides additional metrics at a daily level, delivering increased transparency and understanding of campaign performance. This new methodology also allows tests to be shortened or extended as required, enabling more efficient use of testing bandwidth.
This approach uses a simpler and more nuanced language when dealing with significance testing. For instance, replacing ‘I reject the null hypothesis’ with ‘I’m 91% sure this campaign has had a positive effect’.
This naturally lends itself to a more intuitive graphical display similar to that shown adjacent. This example output shows the probability that a campaign has a negative, positive, or no effect, as well as error bars on all quantities of interest.
Throughout the process, a primary focus was the socialisation of the approach within eBay, through a mix of workshops, documentation, training sessions, and presentations.
The Techniques...
The use of conjugate distributions throughout the tests ensures that the new approach is straightforward to calculate, and is easily wrapped up into Excel macros or R functions for use by eBay. Implementation on other platforms such as SAS is also possible.
The Results...
This Bayesian approach to significance testing at eBay has shown that it can provide greatly improved insight into the various aspects of a visitor’s interaction with the website. In turn this creates increased confidence in the data supporting strategic decisions. In addition, the increased flexibility in being able to stop tests early can allow more tests to be run, leading directly to improved customer experience.
The new testing framework has provided eBay with the means to fully understand and evaluate the benefits of Bayesian significance testing, and is already proving a powerful tool in making the case for its adoption.
"CACI provided deep technical skills and prior experience to accelerate eBay's adoption of a Bayesian framework for A/B testing. This has helped lay the foundations for a much improved framework for interpreting the thousands of experiments we run globally to optimise our buyer and search experience."
Matt Gardner
Senior Manager EU Buyer and Search Analytics, eBay
Lets Talk
Why not get in touch with us to discuss how we can help you implement and optimise A/B split testing.
