Disrupting the digital experience
Watch every single session from a morning of insights from leading DX experts at ASOS, Vitality and Virgin Media O2.
3 tips for your experimentation strategy (plus ideas you’ll want to copy)
According to Doychin Sakutov, Director of Experimentation and Optimization at Virgin Media O2, it’s because most marketers are not going far enough when they run experiments. Too many within the industry talk the talk of experimentation while staying within clear arbitrary boundaries that make their brand look too similar to the competition.
The result? No real differences. No real innovation or forward momentum. The fear of standing out–and of potential failure–results in no real experimentation at all.
This was Doychin’s argument at a recent Contentsquare Champagne Breakfast event.
The title of his talk was “Dare to push the boundaries”, in which he set out some key learnings–with practical examples–from recent successful experimentation at Virgin Media O2. The thought process and results were eye-opening, and so we want to share them here.
Hopefully, these tips and examples will help you run more effective experiments as you level up your own strategy.
Watch now
Tip 1: Don’t (just) look at your competitors
An effective experimentation strategy starts with trying things that you’re not sure will work, Doychin believes. This is inherently frightening because most of us have been trained in life to fear looking incompetent or to be caught out looking like we don’t know the answer to a question. But really, this is the only way forward.
“Don’t look at your competitors, because they’re looking at you too, and you stay on the same level,” Doychin said in his talk. “There is no growth. There is no innovation. There are no advantages that you can explore.”
He went on to give the examples of two different companies, both Virgin Media O2 resellers, who copied each other and came up with almost identical web cards for their services:
No innovation–no difference. “Your customers will get bored and you will gain nothing,” Doychin said.
So what’s the answer?
“You need to lose sight of the shore and completely forget about your competitors when you’re deciding what experiments to do,” he added, before going on to give a great example of where Virgin Media O2 had taken inspiration from the travel industry.
TUI Group is a well-known German travel and tourism company that offers a wide range of holidays, flights and resorts. He pointed to an example web page where TUI created urgency by offering social proof–pointing out how popular a resort is by showing how many people had booked it.
Not only does this show that the resort is good because plenty of other customers have booked. It also gives a sense of urgency by implying that the resort might sell out, which encourages users to make a purchase.
This is all well and good, but how did Virgin Media O2 adapt this idea? How could a telco create a sense of urgency? As Doychin said: “It’s not like we’re going to run out of broadband if you don’t buy it now!”
It turned out that just testing the idea on a Virgin Media O2 page was enough to trigger a sense of urgency in users. Even though logically it makes no sense.
The point is that they tried it because it worked in a different industry and the company was curious. Would it work for them? They didn’t know. Which, for Doychin, was exactly what made the experiment worthwhile.
Adding the message to say how many people had added the product to their basket in the previous 24 hours resulted in a statistically significant increase in order conversion rate. All from testing an idea that looked unlikely to work. But the experiments didn’t stop there, as Doychin went on to talk about his second rule.
Tip 2: Browse the internet
This means you need to look around at where your customers are and the sites they tend to visit.
“See what’s out there. See the familiar patterns. Think about all the times that you didn’t have to think,” Doychin said, expanding on his theory that our online behavior that feels so intuitive and easy is actually learned.
The biggest tech companies and the biggest websites have trained us to use the internet in ways that benefit them. This is why it’s so important to look at popular websites outside your niche for inspiration.
Take the example of Instagram. “We all love spending time on Instagram, even during meetings,” Doychin said, adding that Instagram users are now well-used to Instagram stories.
“Everybody started using stories after Instagram invented them because it soon became a familiar pattern,” Doychin said. “It’s a really easy and quick way of absorbing lots of information.”
So Virgin Media O2 wondered if they could use this same idea to communicate customers’ final bills to them in a way that avoided them having to read through endless PDF documents of fine print and complex jargon. So the company tried it. They put together their own form of Instagram-style stories for this:
The result?
Increased retention and increased digital containment, meaning customers didn’t feel the need to call in and have their bill explained, which also resulted in significant customer support savings. Or, as Doychin summarized it: “It worked magic!”
Tip 3: The more obscure, the better
While browsing the internet and looking for ideas, the more obscure those ideas are for your industry, the better. Obscure ideas will force you further out of your comfort zone to try things that are genuinely new and different.
Of course, the obvious next question is how you can know if something that’s working in another industry is going to work in yours. But this is precisely Doychin’s point: “That’s the hidden gem. You don’t know, and you have to embrace that.”
The point is to try out new patterns constantly. As an example, Doychin cited digital bank Monzo’s Split the Bill feature, which allows friends to split their bills digitally:
At first glance, it’s not obvious how to apply this to telco. But Virgin Media O2 found a way, primarily by offering a “Split the Cost” function on broadband for students living in shared accommodation:
As a result of this experiment, Virgin Media O2 increased conversion rates and next step progression. “It was a success because what we’d adopted was something that was very familiar for our customers,” Doychin said.
High velocity is key to experimentation strategy success
Experimenting in this way is not neat. “We are lucky enough to be given the platform to push the boundaries to discover these different examples. But it’s never easy and requires A LOT of testing,” Doychin warned.
This is where key velocity becomes the key to success.
The more experiments you try over time, the better at it you’ll get. In fact, he described it in terms of a classic learning curve, where you start by experimenting with every idea you can think of, many of which are unlikely to succeed. Then, as you go on, you come up with more interesting experiments:
With this model, you could get through the initial experimentation phase in just a few months, forcing you to go further, find more patterns and embrace more obscure ideas. This in turn could result in more creative experiments by month five. But remember this timeline is different for everyone and it could take longer than 6 months.
A Doychin explains: “You push yourself. You go out there. You search for familiar patterns. You’re trying to be creative. You’re trying to be innovative for your own industry. Innovation doesn’t need to be something that you just come up and create and invent. You need to be innovative within your industry.”
Disrupting the digital experience Watch every single session from a morning of insights from leading DX experts at ASOS, Vitality and Virgin Media O2.
Watch now
Why “I don’t know” are the three most important words for your experimentation strategy
This is how brands can use experimentation strategy to get great results—by pushing themselves to go further, to dare to try things where they can’t be sure of the outcome.
Virgin Media O2 has tested over 600 variants year-to-date, Doychin added in his talk, making them 165% better than the industry standard. And that’s by following the three rules to move beyond being a telco, A/B testing the most obscure ideas, and always pushing forward.
Or, as Doychin concluded:
“Digital disruption starts with the power of pattern, broadening your horizon, and embracing the I don’t know. You have to have experimentation on steroids for that which is backed up by a really high velocity.”
