Split-Testing 101: How to Know Which Words Work

The Transcript

Split-Testing 101: How to Know Which Words Work

Brian Clark: Hey there Rainmaker’s. Welcome to the show, as always. I am Brian Clark, founder and CEO of Copyblogger Media. Robert is off today, somewhere in a metaphorical coal mine getting his work done but that’s okay.

Today we’ve got a special guest, that will more than compensate for the lovely, deep tones of Mr Bruce. Her name is Joanna Wiebe of Copy Hackers, which is a site you should be paying religious attention to, if you are not already.

Joanna completely crushed it at last year’s Authority live event, talking about copywriting and testing the difference between this word and that word, this button and that button. It’s fascinating stuff. And of course, we have been talking a bit about split-testing in the previous episode, so I thought, “Let’s get someone in here who really kind of lives and breathes both aspects of this.”

She is a creative copywriter, and yet she understands the importance of figuring out scientifically what works, and what doesn’t.

Joanna, thank you so much for being here.

Joanna Wiebe: Brian, thank you so much for having me here because I have been admiring you for so long. It always gets me nervous to hear you say this stuff. I feel all nervous now. Like, “Oh-oh, what if I disappoint him?” Anyway, no, it’s great for you to have me here. Thank you.

Brian Clark: Oh, that’s just silly. Come on now.

All right. Now that we have got that out of the way, why don’t you share a little bit about your story. Kind of how you got to being this really go-to expert and running it through Copy Hackers like you do. You’ve got a story that got you there in the first place, and I know everyone does but I want to hear yours.

Joanna Wiebe: Right. Mine is like a lot of people, where you kind of fall into things.

I was a creative writing student and undergrad, which I really liked. I went to Japan for a year to teach, while writing a book, and I didn’t write a word. I came back and almost went to law school. Then some stuff happened in my life and I had this kind of switch where I knew I didn’t want to do some of the things I had done.

At that point I got offered a job at an agency. What they were calling a creative writer and I was like, “Yeah, that sounds good.” So I did that for a couple of years and then moved over to Intuit. The tech company for turbo-tax, quickbooks and all that stuff.

I worked as their senior copywriter for about 5 years and it was really in that time when I was calling myself a creative copywriter, I figured out that copywriting isn’t about the creative all at. But when you are in an agency, that’s a better title to have, than a copywriter, which just sounds dull and boring or something.

Brian Clark: Yeah.

Joanna Wiebe: Which is like a crazy idea, right? So that’s where I was and I went into it. There was this huge testing culture, and I was like, “Tell me more. I like it.”

When I was going through the whole law school thing, my favorite part of the LSAT was the puzzles. Like, “Figure something out and see how it works.” And this has always been very interesting to me. So when I moved over to this testing culture as a copywriter, and had everything so informed by data, it was just like a revelation to me. I loved it.

So I was there for about 5 years and I worked with Conversion Rate Experts for a while too, which is one of the first conversion rate optimization consultancies. They are incredible. They are out of the UK.

Then I went out on my own and started Copy Hackers. Basically, it’s to help the smaller businesses that don’t get access to the same sort of resources that the Intuit’s of the world get. For those people that can’t work with conversion rate experts because as much as it’s worth, they just truly don’t have the budget right now to afford for somebody to come in and do their optimization for them. So that’s what Copy Hackers is for.

I have been at that for about 3 years, and last year I got to talk at Authority, which so far, is one of the highlights of my career. It’s been very cool.

Brian Clark: Awesome. That’s a great story.

A couple of things come to mind. Sonia Simone will always take a creative writing degree because you can teach people the testing and the principles of copy and all that, but you can’t teach initial drive and talent to actually want to write. So that’s why Robert was a poet and now he’s a VP of Marketing. That was a tough case but don’t tell him.

Joanna Wiebe: He was a poet?

Brian Clark: He still is actually.

Joanna Wiebe: Well yeah, I guess that never stops, right?

Brian Clark: Yeah. But yeah, that’s how we first met. And the other thing is, I did really well on the LSAT and unfortunately did go to law school. Actually, I don’t mind that I went to law school, it was the years of practice before I quit that was annoying. So I think you took the right path.

Joanna Wiebe: Thank you.

What Every Real Copywriter Should Focus On

Brian Clark: Okay. Let’s get into this. We want to really talk about split-testing fundamentals. You know, the technology keeps getting easier for normal people like, you know, creative writing majors and ex-attorneys to use, but we have to have the proper perimeters here.

So we’ve been talking a lot about getting people on an email list, building an audience, as a precursor to maybe starting a new business or a content marketing initiative for an existing business, whatever the case may be.

So when you are building an audience, and you are starting a new site, you have your ideas about what’s a good headline, what’s good body copy and what’s the right button text. You know, all these things, but really where do we start when we are trying to build an audience in a new context?

Joanna Wiebe: I think that for us, it’s going to be a new audience but naturally, as we all know, the audience exists, so where are they, what are they doing and what are they looking for?

I would start just by going out and doing that initial research. I’m sure everybody does this, but may be they don’t consciously do it with tactics that are specific and documented along the way.

So like with everything we do in any sort of optimization, which is business optimization, or conversion rate optimization, or whatever, optimize your copy.

It all starts with going out and listening to your prospects. You may not have a single customer or subscriber yet but they are out there. They are talking and they are doing that all online in this very documented way. By that I mean, either in forums or they are leaving reviews on the Amazon products that they buy, that are related to what you want to sell. All those sorts of things. Going out, learning and just soaking them up, which every copywriter I think knows, but I don’t know that the world necessarily knows.

Brian Clark: Yeah. Well it’s interesting and I am so glad you started there because that is where the battle is either lost or partially won, before you ever think about testing. And you are right, pre-Internet, every copywriter, direct response to creative agency type, that’s the first job and they do this insane dive down into “Who are these people that we are trying to reach?” and I hope everyone still does that. And I say it every time I can but sometimes I think when it comes to content, as opposed to pure sales copy, people think maybe it’s not so necessary. It’s just whatever I want to talk about. And then it doesn’t connect with anyone and they are like, “Why?” And I’m like, “Well.”

Joanna Wiebe: Yeah. Exactly. “Why is my blog dead?” Right. There’s no, “Why is nobody reading it? Why is nobody signing up for it? Why is nobody coming to it at all and I am telling their friends about it?”

Unless you are Seth Godin, but then you already have a brand established, so then if you don’t have this established brand, I think it’s very hard to build anything, if you are not building it. I feel like I am saying the most obvious thing right now but if you are not building it for that audience, that means you have to absolutely know that audience intimately.

You know, if you are trying to attract students or write copy for students, you go and sit among them and listen to them, and they don’t know that you are listening to them.

Brian Clark: Yeah and now we have social media, which is the biggest eavesdropping thing in the world. It’s all free.

Joanna Wiebe: You don’t even have to leave your desk. You can do it all right here in this little box around you. It’s brilliant and I think that copywriters do that. But as usual, maybe I’m going to rant for a bit here, but we do things and we just accept them as part of how it has to work but I feel like there’s sort of this sense that you should get a bit of a pat on the back in marketing today, if you go do research.

I’m seeing that in a lot of blogs people are like, “Oh, good on you for picking up the phone and calling customers.” And I am like, “How else are you going to do it?”

Brian Clark: What?

Joanna Wiebe: Sure, good on you. Just like good on you for putting your pants on this morning. Good on you for doing the most basic thing.

Brian Clark: That’s right. Continuing to breathe.

Joanna Wiebe: That’s confusing to me, especially as a copywriter.

Brian Clark: Yeah, now I know, and it’s just a matter of perspective, because when things become democratized outside of the way things are done rightly in an existing industry, honestly that’s why Copyblogger came into existence 9 years ago.

The tactics I was teaching weren’t any different. It was the context in which they were applied.

It’s funny that the newer people eventually seem to get it, and yet some of the old school people still don’t get it. I find that odd too. Maybe when you are hitting the mailbox for your own career, you just can’t shift your mind to social and online. But that’s another episode.

Joanna Wiebe: I know, right? I’ve got lots of thoughts.

The Starting Point for Building Any “New” Audience

Brian Clark: Okay. Let’s assume that we all did the crucial up-front homework to know this prospective audience, better than they know themselves. That’s the goal at least.

Joanna Wiebe: Yes.

Brian Clark: We have made some educated guesses about them and the benefit that we are trying to give in exchange for their attention. Headlines, copy and all that kind of thing. It’s still a new site though. We’ve made some educated guesses but if we are smart, we wrote many headlines and then went with the one we kind of felt was best but we don’t know for sure. And this might be where some testing may come in handy but if it’s a new site and it’s not a huge traffic generator, how do you solve that problem?

Joanna Wiebe: Yeah, that’s a big one. When we are talking about testing, a lot of people will say, “Well I’ll just push some more traffic at it. I’ll go buy traffic.” But we don’t necessarily recommend that. I think the bigger thing that we would say to do, and the statement that we have, between Lance and myself, “The lower your traffic, the bigger your change is.”

So if you decide that you want to test your headline, like you said, but you don’t have a lot of traffic, for example you get 300 visitors to your homepage a day and you want to test that, your traffic is just going to be really low to make that happen.

So when you are testing one small thing, the impact is unlikely to be big enough to cause a big win in your testing tool like Optimizely or VWO or whatever it is that you are actually using to run the test, or even if you are using an Unbounce landing page.

Something like that that you are testing in, you need to have this huge difference that’s measured between your control and that new variation. So a small change on your new variation is unlikely. In most cases, I’d say 99% of the time, a small change like a headline, and as important as headlines are, just that alone is unlikely to bring you the huge numbers. The differences that you need, right?

Where if you have five people convert or 25 people clicking through on your homepage a day, let’s say per variation, you need the control to keep having that 25 people clicking through but you need the variation to get like 75, 100 people clicking through. You need huge differences in order for that test to ever get to a point of completion.

If it’s 25 versus 30, the numbers are way too low to say anything about it, right? So for us, the big thing that we try to recommend is if your traffic is low, make sure that new creative you test against the control, is dramatically different.

Brian Clark: Yeah. I kind of figured that out myself. Like I said, we have got split-testing tools built into the Rainmaker Platform, which are really simple to use and it’s tempting to just jump right into. Then I started looking around online going, “What’s going to make this statistically significant?” and I ran straight into that problem.

Now when we test on StudioPress or Copyblogger, traffic is not a problem. It’s huge, right? It’s just an issue that I really wanted to address with someone like you.

The Number of Conversions You Need to Make a Good Call

Brian Clark: Okay. So before we maybe look at some alternatives, what is the lowest amount of conversions per option that you would consider, something even paying attention to? I found some stuff online but I don’t want to say anything. I want to hear it from you.

Joanna Wiebe: There’s a lot online. At Unbounce’s CTA conference last September, Peep Laja from Conversion Excel quoted somebody who said “350 paid conversions per variation” and I’m like, “In crazy land. No way, for a small business.” Forgive me if I am getting this wrong, but it really does sound like the takeaway is, “Oh, crap. If I am not Amazon, I can’t test.”

I don’t think that you have to have that many, like 250 or 350 paid conversions per variation, in order for it to be a statistically confident test. For me, I look at things like the data is good and you have to listen to the data. Follow what it says and pay attention. Don’t jump to your own conclusions or say, “Oh, it looks like it’s trending up, so we are just going to call it a winner and go.”

Brian Clark: Yeah.

Joanna Wiebe: But at the same time, I think waiting around for these huge numbers is actually something that’s not going to happen in a lot of cases.

We like to see about 100 conversions per variation of whatever that is. So it’s unlikely on a homepage, unless it’s a one pager site. I wouldn’t expect a homepage to be responsible for a paid conversion. But for a micro-conversion like a click-through, then definitely.

If you get 100 people clicking through on each variation at minimum, then from that point you can use the calculators that are available within the tool. You can use Evan Miller’s calculator to confirm that the data is good, rather than just trusting VWO or Optimizely. But get to about 100 if you can. That usually means running your tests for well over a month. But that’s what we like to see. Other’s say more but I think that more is often.

Brian Clark: Okay. Let me reassure you because 100 is the number that I use.

Joanna Wiebe: Oh, okay. Good.

Brian Clark: And in this context, obviously we are talking about email opt-ins, which as you know, is a discreet action which does not rise to the level of paid, and yet, it is definitely trackable.

Everything you have said so far, I kind of figured out in my own experience. I tested a “how to” versus basically the same substantive headline, thinking “how to” would kill it, and it wasn’t significant a change because it’s too similar.

Then I tested another idea that was almost a different positioning statement that I had had in my research. It was almost like another way to go. And my first choice of headline destroyed the radically different one, which score one for intuition. But it’s not intuition. It’s like hard core paying attention and understanding what these people respond to, even though you are not currently serving them.

We are going to go back and say, “Start with research” for every question of this show.

Joanna Wiebe: I know, right. That’s always the answer. And, end podcast. That’s it.

How to Know What Site Areas to Test in the First Place

Brian Clark: Okay, so let’s assume that we have got enough traffic and we can do the 100 conversion threshold in a reasonable amount of time. Let’s say, it’s a week or something.

My rule of thumb has always been, you test the headline first because if people don’t make it past the headline, what does it matter if your button text is optimized?

Now do you agree with that and regardless, what are your tips as far as testing headlines? You already gave us one.

Joanna Wiebe: No Brian, I disagree with you.

Brian Clark: Okay. Cool.

Joanna Wiebe: Just kidding. This is why I hesitate there. For me, I know headlines are very, very important but of course, as I talked about at Authority, the button is that site of conversion. Of course, it’s where the activity happens. It’s hugely the headline but I would say the button is equally important. And I would further say that they work together.

We’ve actually run a few tests and we reported on one of them this fall. We ran this test on a UK fashion site called Dressipi.com. I guess our research question was, “Does riskier copy lead to bigger conversion wins?” largely because you are actually saying something that’s sticky and noticeable. So we were testing on a bunch of different sites around this, so we had a whole bunch of headlines that we were testing on different sites and Giuseppe was one of them. And so we tested these two headlines against each other.

The control was “Clothes you’ll love perfect for your shape and style.” Okay. A perfectly fine homepage headline and then the one we tested against it, we pulled from some language that one of their prospects or someone who fits into their market were saying on the forum. Obviously they are targeting women. These women were talking to each other on this fashion type forum, and they were saying things like, “I have got a big bum. And that’s not good on my bum.” Or “My waist is too thick for that.”

So against the headline, “Clothes you’ll love perfect for your shape and style” we tested the headline, “Big bum, thick waist, not so perky boobs? Find an outfit you’ll look fab in, just as you are.”

So it was much bigger and of course, a longer headline but it had the stickier words in it. Things that people perhaps weren’t going to see a lot online but it was language that we saw they were using. So we thought it was a good test.

When we tested we saw about a 15% lift. That held throughout the test for a couple of weeks, but it didn’t actually reach confidence. We never actually got to a place of confidence. We were like, “What the hell? This headline, in my humble opinion, is superior to the non-sticky one, that it was being tested against. It should be doing something. Even if that means polarizing people. The polarized group that is ‘pro’, unless they’re a very small group, shouldn’t they be acting on it?”

So from that point, I was like “Okay, well let’s take a look at the button on there,” which again is the point of conversion. The site of conversion was “Sign up now.” So we had that on both variations. And I was like, “Ugh, ‘Sign up now’ is a terrible button” right? I would never recommend that as a button. It’s the action you are about to take but there is no value associated with it.

Brian Clark: Yep.

Joanna Wiebe: We were like, “Okay” so again, we ran another test. We had those same two headlines tested against each other but on the second one we tested “Show me outfits I’ll love” as the button copy on there. And with that we saw a 124% lift in clicks, with 100% confidence and it held for weeks. I think we even left it running for 6 weeks.

It was so dramatically different. We had two great headlines against each other. And once we optimized the button as well, I think the headline was able to perform better because it wasn’t restricted by this really sucky button that was pulling it back before. Does that make sense?

Brian Clark: It does and I have to admit, I kind of set myself up there because I knew you were going to say that.

Joanna Wiebe: I knew, you knew. That’s why I wasn’t afraid to say that I disagreed with you.

Brian Clark: But it’s so prevalent that we think the headline is just God, and it is so important in content and copy.

I think I love that story and I love what you elaborated on that at the event. That’s why I think you blew people away because you were scientifically and very politely, in the most Canadian sense, destroying a bunch of myths that we all carry around with each other. So, yes, thank you for disagreeing with me, although I was being a little sly there.

Joanna Wiebe: Thank you for setting me up.

Joanna’s Favorite Email Opt-in Button Copy

Brian Clark: Okay. Then I’m going to put you on the spot here. In the context of opt-ins, getting the email address, what’s your favorite button copy, to the extent you can choose something that might work better than most, as opposed to “Subscribe” which is awful?

Joanna Wiebe: Yeah. Let’s say you’ve got an opt-in form, a pop-up, or an exit-intent pop-up. Typically, a great, consistent winner is to take your headline, which usually begins with an action word like “Get”. So the headline has some sort of action statement in it. Just repeat that in the button. We have seen that work all over the place. Not just for opt-in but especially for opt-ins.

So if you say, “Get the conversion rate kit” as the headline, then the button maybe just “Get the kit.”

Brian Clark: Yeah.

Joanna Wiebe: So that’s not a very sexy test but it seems to work, again, and again, and again. If the headline attracted them, and if they liked that and there’s an action to it, then why not just repeat it in the button?

Brian Clark: Yeah, it makes a lot of sense. It’s action oriented and it’s mirroring or continuing the scent that Bryan Eisenberg likes to talk about.

Joanna Wiebe: Yes. Totally.

Brian Clark: Excellent. That’s already worth the price that everyone has paid so far. We joke around that I just invite my friends on for free consulting and then share it with everyone.

Joanna Wiebe: That’s awesome.

The Recurring Theme of Conversion Testing That Works

Brian Clark: Okay. I did want to primarily focus on email and I do of course want to respect your time. Let me just ask you a couple of things about when we are actually selling something, services or products. Lance had a great article just a few days ago that I found very interesting and it had a nice little click-bait title but he totally backed it up with some really smart ideas.

But before we get to that. When you’ve got a sales page, you know, you’ve got a benefit oriented headline, you’ve got features that are written in terms of benefits. If you are doing baseline practices as it is, from there, how do you develop one or more hypothesis that tell you what to test at all?

Joanna Wiebe: Yeah. I think this is a really good question and it’s hard for a lot of people to do. So hard I think that people don’t do it, which is problematic.

I like to break it up into two simple things. As you said, Lance writes great posts on testing. I write posts on copy. So Lance is definitely into the science of it all.

Brian Clark: Yeah.

Joanna Wiebe: I run my stuff by him and have him approve it. For me to get my head around it, I like to have a problem statement followed by a couple of research questions. I choose one of those and build creative against it.

Obviously, first consider your conversion goal. Go through all those things that you normally do to identify an opportunity. The problem statement is a question around what’s keeping people from converting at this point. Again, we are talking about sales funnels.

Brian Clark: Right.

Joanna Wiebe: You’ve identified on your product catalogue that there’s a high exit rate on there, so people are not converting. They are leaving, instead of even moving forward. So if you’re goal is to get them to choose one of those products or categories to move forward with, that’s your conversion goal. So choose one they have to click. It will be an engagement test, which is perfect.

Then it’s a matter of identifying what’s getting in the way of them moving forward or what’s making them want to exit on this page. And in most cases, at that point, that’s where we look to the creative.

Obviously there’s lots of ways to go about this but I would say, “Okay. Well if they are exiting here, it’s probably got something to do with what they are failing to find on the page.” So it might not be the right information that they need etc, or the way the information is presented is wrong. So as usual, it comes down to the “What?” versus the “How?” And in that, you have to identify which one of those it is.

At this point, if I said, “Okay, the product catalogue is brutal. We are losing people. It’s bleeding all of our visitors. Or paying for all those people to come here and now they are all just exiting before they even get to a product detail page.” Then I would say, “Okay, well let’s go and run a usertesting.com session. There are about five of those and see what people are saying when they are on this page.

At the point, we go and listen to a lot of what people are doing and saying. Like watch what they are doing on the page and listen to what they are saying about the page.

If it means doing a Qualaroo pop-up survey as well, to get a sense of what they came to this page to do, and what there expectations were, then that can help us see if it’s a “What?” or a “How?”

The “What” is, “Are the right products on this page organized in a way that it’s easy?” Yes, organization is kind of a “How”. However, if it’s buying boots and they are on a catalog page for boots, but they had an expectation that they were able to sort them by tall boots and short boots, but they can’t do that, then that’s a, “What’s wrong with this page?” And that could help us understand what we need to do to optimize this page. Then that leads to a question, which is a hypothesis or phrased another way, it’s your research question if you pose it like a question.

And that could be, “Okay, if we add better filtering on this page or better ways to filter through the products, will we see a lift in click-throughs?” That’s how we would get to the point of identifying what’s wrong with it.

First, what page is causing the problem that’s keeping you from getting that conversion? Next you need to ask, listen and then watch what people are doing on it and develop an hypothesis from there. And of course, you want to ideally go with the one that’s the lowest hanging fruit.

So if only one person says, “Oh, I thought I could sort by short boots versus tall boots,” but other people are saying, “Is anybody else even using this site?” Or it’s indicating that there’s no social proof or a lack of authority about the site, then you choose the best one between those two possible problems and test from there.

When you set up that good hypothesis, or it’s informed by something and you can say something as specific as, “Will adding filtering or sorting to this page increase my conversion rate?” Then you add that filtering or sorting and then you see if it did or it didn’t. You then actually have an answer and you can move forward with that. You can then take away that learning, which I would say, in most cases, is more important than just a lift in conversion.

Brian Clark: Yes. Absolutely.

Joanna Wiebe: I know that’s a long answer.

Brian Clark: It’s a long answer but there is the recurring theme of listen, watch and observe. And at the risk of oversimplifying Lance’s post, he said, “You know, if you are selling stuff to people, why would you jump straight to testing, why don’t you talk to them first?” And it’s so simple. And he wasn’t denigrating testing, of course, Lance wouldn’t denigrate testing and yet people don’t start at the right place.

You mentioned some really cool tools where you can listen, but why not just talk to the people who bought from you and find out from stuff right there?

Joanna Wiebe: It’s so true. I think when you are trying to identify what’s wrong with a page, it can be a little tough. That’s where I think usertesting.com and Qualaroo are fantastic. When you’ve got a page identified and you could sit around at a table brainstorming what’s wrong with it. I have learned this from Copyblogger too. You are not your audience.

Brian Clark: Right.

Joanna Wiebe: So listen to your audience first. I know with usertesting.com the objection is, “Well, are they really my prospect?”

Usertesting has a new peek service, where you can get your visitors while they are right there on your site, to opt-in to share their feedback with you. So there are options and ways to get around it.

I’ve found if you want to find out what’s wrong with a page, then a really great way is to get people on that page to try and use it. Don’t ask them to tell you what’s wrong with it because oh Lord, it will get awful.

Brian Clark: Yeah.

Joanna Wiebe: Just watch them and listen to them, which can then lead to a great hypothesis.

Brian Clark: Awesome. Joanna thank you so much for your time. I am still getting over the fact that you are not rejoining us this year but you did have a good excuse.

Joanna Wiebe: It’s U2’s world tour opening night, Brian.

Brian Clark: Wait a minute, I thought you were helping your sister with her food truck? I think I have been bamboozled here.

Joanna Wiebe: No, I told you. Remember in Twitter, I was like “Oh yeah, I think I am doing my sister’s food truck” but that’s because I have two sisters and they are constantly asking for things. No, I’m just kidding. But the other one wanted to go to U2 and I was like “I know I am doing something with my sister that weekend.”

Brian Clark: Oh yeah, that’s right. That’s right. Okay, so Bono wins over me, fine.

Joanna Wiebe: Sorry. This one time.

Brian Clark: Yeah. Okay.

All right. Well thank you again and I can’t wait to do something else with you guys. Obviously, we’ve got to have Lance on the show. And I’ve already got you down for 2016, so don’t even try.

Joanna Wiebe: I’m on it. It’s on my calendar. I can’t wait.

Brian Clark: All right. Take care. Bye everyone.

Joanna Wiebe: Bye.