The measure of Success (tend to be customer-centric) are a common way to describe the success of a product.
“Our Customer Retention score is 90%.”
“Our Net Promoter Score is 80%.”
“Our Churn Rate has never crossed 10%.”
“Our conversation rate from trial to paying is 50%.”
“Our Monthly Active Users are growing at a rate of 210% month-on-month.”
“Our Monthly Recurring Revenue is growing at a rate of 230% month-on-month.”
“Our Customer Lifetime Value is $300 and growing.”
Such metrics are very important and should be well crafted. They allow a business to focus on the right behaviour expected from a product.
The Flip Side of Measure of “Success”
Consider this scenario. You have chosen a conversion rate of 50% from trial to paying customers as your Measure of Success.
Say early in the journey, you see 49% conversion rate. What does that mean? Has it failed? Not really. Just a bit behind but still a success. You continue on the same path along with continuous improvement.
Say instead, you had seen 45% conversion rate. What would that mean? Had it failed? Just a bit behind but still a success? Most probably, you would have continued on the same path along with continuous improvement.
Maybe same at 40% conversion rate.
At what point will the alarm bell ring? At what point will you decide you have some basics wrong and you need a pivot?
Say instead, you had seen 30% conversion rate. What would that mean? Had it failed? Still a success? Most probably, at this threshold, you would decide to go back to basics and maybe pivot.
Say then it falls down to 20% conversion rate. More user studies, more analysis, more advice seeking, more market research. Not calling it a success but not giving up. Pivots and updates would occupy your mind.
Maybe same at 10% conversion rate.
At what point will the alarm bell ring? At what point will you decide you have it all wrong and need to consider closing the product as sunk cost?
Maybe when the conversion rate is less than 5%? For straight 3 months? Even when you have made drastic changes based on real data and feedback? Most probably, at this threshold, you would decide to wind up.
Startups get this. Enterprises won’t. And that is a whole another topic.
A Case for Measure of “Failure”
In my Product Management Canvas, there is a clear place for calling out Failure Metrics. In the example below, there are three clear thresholds:
|Conversion Rate > 30%||–||Success! Keep learning and keep improving|
|Conversion Rate <= 30%||–||O Oh! Go back to basics. Something is wrong. We are not successful but we are not giving up either. We will get there.|
|Conversion Rate <= 5%||Over 3 month||Am I throwing good money on a bad idea? Am I stuck in sunk-cost fallacy? Time to call it quits? Looks like that.|
While you struggle to find the right product-market fit, you have not failed.
Not achieving Success Metrics does not imply failure.
It is obvious that not achieving Success Metrics does not imply failure as in between the two is a large gap, the Zone of Struggle.
Then how does one know when to stop and move on? For that, we need a very clear Measure of Failure.
Measure of Failure
In the above example, it is obvious that Measure of Success and Measure of Failure are distinct measurements and face values. Defining Metrics of Failure is an important activity that we may find hard to dwell on. It is human nature to avoid dwelling too much on the worst case. This is where a disciplined Product Manager can make a difference.
Some characteristics of Metrics of Failure:
- Like Metrics of Success, Metrics of Failure is a band of values rather than a single value
- Unlike Metrics of Success, Metrics of Failure has a definite time component. In the above example, it is “Conversion Rate <= 5%, over 3 month“.
- It can be the same as Metrics of Failure but a band on the other side of the spectrum
- There is a sufficient gap between Metrics of Success and Metrics of Failure for the Zone of Struggle
- Metrics of Failure can possibly be a totally different metric than the products Metrics of Success
The Measure of Success, the Zone of Struggle and The Measure of Failure
Defining Measure of Success and Measure of Failure is an important part of your Product Ideation process.
|Measure of Success||Conversion Rate > 30%||–||Keep going. Keep learning. Keep improving.|
|Zone of Struggle||Conversion Rate <= 30%||–||Pivot|
|Measure of Failure||Conversion Rate <= 5%||Over 3 month||Record Learning and wind up.|
What’s your experience with this? Please share, especially if you are in an enterprise.