This article is an excerpt from the Shortform summary of "The Lean Startup" by Eric Ries. Shortform has the world's best summaries of books you should be reading.
Like this article? Sign up for a free trial here .
What are vanity metrics? Why are they so bad for understanding your startup? Learn the key vanity metrics you must avoid, and how to avoid them with the right approach.
Introduction to Startup Metrics
As Eric Ries says in The Lean Startup, The job of a startup is to:
- Build an MVP to measure where it is now
- Experiment to improve the metrics
- Decide whether to continue in the same direction or pivot in a new direction.
Defining the right metrics that actually matter to your business is critical. Before giving examples of good metrics, we’ll define common bad metrics startups choose.
Why Vanity Metrics are Bad
The most insidious kind of metrics, vanity metrics, give you false optimism – it SEEMS like you’re making great progress, but in reality you’re actually stuck.
Often, vanity metrics are metrics that have no choice but to keep increasing over time. One common example is total user count. Let’s say your app adds 1,000 users every week. At the end of 10 weeks of hard work, you have 10,000 users. This is a big number!
Except you aren’t growing any faster – you’re still adding 1,000 users every week. Your hard work actually didn’t change your growth rate.
These vanity metrics are misleading because they keep increasing over time, making you feel good. But think about a silly metric like number of hours worked. Every day, you and your team each add 10 hours to the total count. At the end of the month, you proudly announce to the team, “this month, we added 1,250 hours to our Total Hours Worked metric. This is great progress. Good job, team!”
This may sound silly, but think about how many founders you’ve heard brag about how many hours they work. What actually matters are the results they achieve. They might not focus on these non-vanity metrics as much!
Another type of vanity metric tracks results that aren’t actually critical to business function. For instance, early startups worry about racking up press mentions or making new hires. While these may help the business move, they’re not the actual core of the business – your company doesn’t exist to hire employees, it exists to create value for customers and create profits for shareholders.
These common vanity metrics are all problematic:
- Total number of anything – users, sales, actions in product
- Money raised from investors
- Press articles written about your company
- Number of employees hired
- Number of features added to product
- Meetings scheduled
- Emails written
Avoiding Vanity Metrics: Cohort Analysis
Instead of looking at cumulative vanity metrics over time, the more accurate analysis is to separate users into groups based on the time they joined, then measure your metric for each group independently. Each group of users is called a cohort. Learn more about cohort analysis here.
Choosing Good Non-Vanity Metrics
You need to define metrics that really matter to your business, and measure improvements to that metric.
The metrics that really matter vary from business to business. Often, they reflect your Value Hypothesis and your Growth Hypothesis.
For example, an app might aim for these metrics:
Value Hypothesis: Each new user will upload an average of 10 photos in the first week of using the app.
Growth Hypothesis: Every user who joins the app will refer 10 new visitors, 1 of whom will join as a new user (leading to virality).
The important part is that you need to believe these numbers are vital to the success of your business. Eric Ries suggests choosing the riskiest assumptions first – the metrics that you have the least confidence on, yet have the most impact to your business. For example, if your company is going to be supported by advertising, advertising rates are probably not the riskiest assumption – getting user engagement will be.
Here are examples of good metrics that reveal the health of your business (only some will apply to your startup):
- Engagement
- Time in product per user per week
- % of users who return in the first day / first week / first month
- Growth
- Viral factor
- Net Promoter Score
- Conversion rate of each major step – visiting to signup to payment
- New users gained per week
- Finances
- Customer acquisition cost
- Lifetime value per user
Unlike vanity metrics, these metrics don’t improve unconditionally over time – if you don’t put in work, it’s unlikely that the revenue per user will increase.
———End of Preview———
Like what you just read? Read the rest of the world's best summary of "The Lean Startup" at Shortform . Learn the book's critical concepts in 20 minutes or less .
Here's what you'll find in our full The Lean Startup summary :
- How to create a winning Minimum Viable Product
- How to understand how your startup will grow
- The critical metrics you need to track to make sure your startup is thriving
