What to Do When you Don’t Have a Baseline
We’ve tried to describe some useful baselines. But if you’ve read through the past seven chapters, you’ll know that these numbers are rudimentary at best: you want churn below 2.5%; you want users to spend 17 minutes on your site if you’re in media or UGC; fewer than 2.5% of people will interact with content; 65% of your users will stop using your mobile app within 90 days. For many metrics, there’s simply no “normal.” The reality is you’ll quickly adjust the line in the sand to your particular market or product. That’s fine. Just remember that you shouldn’t move the line to your ability; rather, you need to move your ability to the line. Nearly any optimization effort has diminishing returns. Making a website load in 1 second instead of 10 is fairly easy; making it load in 100 milliseconds instead of 1 second is much harder. Ten milliseconds is nearly impossible. Eventually, it’s not worth the effort, and that’s true of many attempts to improve something. That shouldn’t be discouraging. It’s actually useful, because it means that as you approach a local maximum, you can plot your results over time and see an asymptote. In other words, the rate at which your efforts are producing diminishing results can suggest a baseline, and tell you it’s time to move to a different metric that matters. Consider the 30-day optimization effort for a site that’s trying to convince visitors to enroll, shown in Figure 28-1. At first, out of over 1,200 visitors, only 4 sign up—an abysmal 0.3% conversion rate. But each day, the
company tweaks and tests enrollment even as site traffic grows modestly. By the end of the month, the site is converting 8.2% of its 1,462 visitors.
The question is: should this company keep working on enrollment, or has it hit diminishing returns? By applying a trend line to the conversion rate, we can quickly see the diminishing returns (Figure 28-2).
Ultimately, the best the company will be able to do with all else being equal is achieve a conversion rate of around 9%. So on the one hand, that’s a good baseline, and gives a sense of the universe it’s in. On the other hand, all else is seldom equal. A new strategy for user acquisition could change things significantly.
This recalls our earlier discussion of local maxima. Iterating and improving the current situation will deliver diminishing returns, but that may be good enough to satisfy part of your business model and move forward. In this example, if the company’s business model assumes that 7% of visitors will subscribe, then it’s time to move on to something else, such as increasing the number of visitors. If you don’t have a good sense of what’s normal for the world, use this kind of approach. At least you’ll know what’s normal—and achievable— for your current business. At this point, you’ve got an idea of your business model, the stage you’re at, and some of the baselines against which you should be comparing yourself. Now let’s move beyond startups into other areas where Lean Analytics still plays an important role: selling to the enterprise and intrapreneurs
re 