November 15, 2017
Approaching an anniversary commemorating the time I’ve spent in the glorious arena of conversion rate optimization under the umbrella of a stellar PPC agency (#8 on AdAge’s Best Places to Work list), I wanted to take this small corner of the Interwebs to share a few key lessons with you all. Why? Because I am a doer. I learn by doing. Thus, I tend to make mistakes. It solidifies my lesson retention, but sometimes mistakes just don’t need to be made. For those of you lucky enough to effectively learn vicariously, learn from my mistakes and prevent your own! Because trust me, when you make these mistakes you generally embrace the look of shame; downward-turned eyes and all.
#1 Limit Risk
I pride myself on being one of those folks that has to do ALL of the research before I carry out any task; in life, in CRO, you name it. However, when first stepping into CRO I also believed that it made sense to test on 100% my traffic straight out of the gate.
No matter how much data you put behind a testing recommendation and how much time you pour into QA there are always variables you may not be aware of and therefore unable to control. One of these may come out of left field and slam your test and your conversion rate right into floor-level metrics. No one wants to experience that.
When you launch a test of any capacity, limit your traffic for the first few days while you closely monitor everything in order to ensure that the test is running smoothly. We want to disrupt traffic by nature. We strive to optimize that traffic’s performance. But if it goes south quickly, there is no logical argument you can hold to keep it running in hopes that it will turn up and to the right. It won’t happen reliably and consistently and pulling the plug early is better than forcing that traffic, and the site, to suffer while you hold your breath and hope for the best.
#2 Data is Your Friend
Experience is a well-rounded teacher, but in an ever-changing industry that rests on human behavior your gut and your experience may not be the most reliable foundation to stand on. Data (preferably quantitative) is rooted and solid and will enable you to ground yourself and your testing endeavors.
I promise not to lecture you on bias, but I will assert that you are biased and you should come to terms with that sooner than later. You may absolutely hate a specific element of a site or how it functions. That element may make no sense to you, but it may be completely intuitive to your users. Assuming that your experience is similar or identical to everyone else’s experience on that site is also known as the false-consensus effect.
Let the data do the talking and try not to place your own personal lens onto that data. If it helps, I try to think of every data analysis as a learning experience. Prove me wrong. Let’s bring it back to square one of the scientific method: form a hypothesis, then try to disprove your own hypothesis.
#3 Learn Regex
I cannot tell you how many times I have failed to capture all traffic to a thank you page because I had a very specific destination goal set up to match a URL. There are all sorts of ways that anyone and everyone can add a parameter to the URL you’re tracking and if none of those people loop you in, you’re goal is invalid. If you copied the URL wrong, your goal will be invalid.
In order to eliminate most threats seeking to invalidate your goal tracking, isolate the critical pieces of that URL and blend it together using regular expressions.
RegexBuddy lovingly describes regular expressions as “wildcards on steroids.” The language is rather robust and you can get very granular using regex, but I use 3 symbols in total: .,*, +.
- “.” – wildcard that represents one character (used like “*” in excel with the limitation on reach)
- “*” – qualifies the wildcard to include 0 or more characters
- “+” – qualifies the wildcard to include 1 or more characters
Imagine the thank you page we want to track hits to is as follows: www.ppchero.com/thank-you
You know that someone is going to track traffic source via parameters so we already know that folks may be coming in on a URL with all sorts of shenanigans following “thank-you”.
I would use the following expression to track this URL:
Do NOT forget the drop-down menu.
Note that I’ve replaced every symbol in the URL with “.” because “.” is a wildcard and, as I mentioned earlier, I like knowing what every character is doing. I am not a regex expert but I do know that it is technically a language and has the capacity to interpret many characters and perform functions based on those characters. I take no risks in regex. Wildcards are safe.
If you’re still confused or wish to dive deeper into regex here is a very informative site: RegexOne
While you practice your skills, double-check your phrases for complete functionality with this site: Regex Tester
#4 There is Always a Catalyst to Your Problem. Or 2. Or more…
Be very wary when assuming that any singular variable is responsible for a problem.
This seems counterintuitive to testing because we attempt to control every variable beyond the alteration to ensure similarity and can attribute performance to that alteration. However, “attempt” is critical there. There are too many variables involved in anything we do and you may not always be able to find every variable that affected your performance.
For instance, while hunting for a catalyst responsible for an irregular drop in conversion rate on mobile devices we discovered several key factors on a similar timeline and none of them could be ignored.
- Spend in AdWords for specific campaigns was abnormally low indicating that we weren’t as prevalent as we thought we were on the SERPs.
- The mobile site is a responsive desktop site that had most likely reached the end of its efficacy in the age of highly functional and attractive mobile sites.
- Competitors in that industry had also become more and more prevalent and were becoming more competitive in the PPC space as well as the user experience space.
All of these are critical and most definitely impacted that conversion rate. All at once? Maybe not. Nonetheless, compounded problems are still problems.
Never rule a problem as completely solved (unless you’re positive you caused the problem). Always keep your eyes and ears open and ask yourself if anomalies could have worked together to cause your problem. Keep digging.
Also keep in mind that while most our work lives on the Internet, people’s lives do not. Anything and everything our users see, hear, or even smell can affect your conversion rate if it reaches a significant portion of your traffic.
#5 QA Your Landing Page Tests
This is a basic step in the test set-up process and as such it tends to go by the wayside when your personal confidence concerning testing starts to increase (Or is this just me? Bueller?).
Whether you are testing for your own site or for your client’s site, you should always be aware of what your test looks like. In an ideal world, we would test on every single browser (including various browser sizes) and operating system. However, most of us don’t have that kind of time on our hands. To compromise, test on the browsers and operating systems that the majority of your traffic utilizes.
Never forget this step. The worst message you can receive, the most “tail between your legs”-inducing, is the message with the screenshot of your test rendering in a very unintended format. Even if you change a button color, QA that nonsense. You can never be too safe, right?
You’d also be amazed how many times I catch random browsers having severe difficulty rendering the variation correctly. Don’t deprive your niche browser users of aesthetically pleasing or functional experiences purely because you didn’t want to spend time in QA.
Final Thoughts
- Limit performance risks by limiting traffic flowing through your tests in the first few days (when you can).
- Build a reliable foundation of data from which you can make decisions that are not biased by your experience.
- Learn regex. I promise you it will be useful if you target anything via URL.
- Each and every problem has a robust array of catalysts. Hunt them down and keep an eye out for the stragglers.
- Make sure you know what the users are seeing. Feel their feelings. Experience the site as they would. Empathize! And don’t forget those that utilize “strange” or little known browsers.
When you get excited and go a little crazy, take a step back, breathe, and remember that you’ve totally got this and that you don’t have to repeat Kate Wilcox’s newb mistakes because you’ve read this article and learned vicariously.
If you feel like you’re on the breach of a good mistake or you would simply enjoy sharing an “I’ve been there” story, reach out via Twitter (@katewilcoxkcco) and make my day.