Summary
Stale Homepage
The DriveTime UX team had taken a small detour during the pandemic to help their sister company get an app off the ground; consequently their homepage remained unchanged in those few years. In this time, DriveTime had been celebrating the success of new TV creative. These new commercials emphasized the flexibility of our financing, and we wanted to test refreshed branding and language that had a similar outlook.
We started with the most popular page on our site — the mobile homepage. Drivetime.com averages just under 2 million unique visitors a month and 56% of them land on our homepage. 86% of our total traffic comes from mobile devices, so we were confident this high-traffic page would reach statistical significance faster.
Since there were so many ideas coming out of our design sessions, we decided to let our users choose which brand-direction spoke most to them. We came up with a multivariate test that ran every combination of our potential imagery, header verbiage, and the main CTA on the page.
We were expecting lively results since, in the past, even slight changes to our homepage revealed compelling outcomes for lead conversion and even lead-to-sale.
Obstacles
Fresh Team Fresh Look
While the multi-variate design of the experiment could facilitate a large amount of ideas, it could not produce limitless variations. Still, the business strategy team wanted a lot of options and flexibility in what they could present to our customers for feedback in this experiment. After gathering an enormous cache of ideas from across the creative teams the challenge was distilling them down to a handful of intentional visual and verbiage options.
It was also important that the ideas displayed had enough contrast between them so as to be sure that our independent variable was causing the observed differences in outcome.
Solutioning
Multi-Armed Bandit
All these elements fit into three UI categories:
- Page imagery
- Headline verbiage
- CTA verbiage on the button.
Hundreds of concepts across these categories were filtered out. First within the design group, then we had product pick promising options, and finally, the business strategy group anonymously voted on their favorites. We ended with a matrix of elements that included 5 visual concepts, 6 headline directions, and 3 previously proven CTA’s. These elements would be mixed-and-matched to create 90 variations for testing.
Results
New Direction
In order to shorten the length of the experiment, we shut off sub-optimal variations after 2 weeks and continued to route more traffic to the ones that showed promise. In the final week of the experiment, there were only 7 variations running. All of these variations out-performed our control. The final winner had significant lifts in our main KPIs (see graphic) and was a huge step forward for our new brand direction.
