When choosing metrics to describe a product or a specific e-commerce feature, it is essential to approach this process systematically. Various indicators can describe each product feature or change. For example, e-commerce product page changes can be described by such metrics as conversion, average check, or a number of items added to shopping cart.
I want to start this article with a story of how we fixed a situation when a client’s product team made a mistake by choosing the wrong metric. This choice could lead the company to a financial and time loses.
Imagine that an e-commerce company decided to implement a new recommendation system. The new recommendation system algorithm intended to show related products to users. The team conducted an AB testing to validate the new recommendation system and compare the results with a previous algorithm. Analyzing the AB testing results, it was imperative to select the indicator that would validate this change accurately. Product manager decided that the new recommendation system should increase the conversion rate. However, the conversion rate has not changed.
The thing is that in this example the new recommendation system could improve conversion, but also other indicators such as average check and total revenue. Focusing only on one indicator was a mistake and could reject a valuable feature, which can significantly increase such important business indicators.
It is necessary to have a prioritized metrics map rather than one or a couple of metrics. Metrics map helps to evaluate the quality of the idea or a feature accurately.
Metrics prioritization works as a guide. It helps to determine product features that would drive a positive growth as well as to avoid making wrong decisions in many cases.
The metrics influenced by a large number of internal and external factors. We cannot control external factors, but internal factors can be controlled quite easy (not too easy, of course, but let's imagine the world of pink ponies). If you told that the conversion rate is 3%, this means nothing. Conversion rate 3% by what factor and combination of factors? By factors and its combination I mean, for example, users from a specific device + traffic channel + a particular history of visits or users of a certain gender, age and with a certain number of purchases. There are a lot of such factors in any e-commerce product. Before analyzing any metric, it is critical to identify what factors are the most influencing for the metric.
In the picture above, we can see which factors have the greatest scatter in the metric’s mean value. To determine what influence our metric the most we can use specific statistical methods such as ANOVA, Logistic regression, Random forest, for example. We will review these methods in the future articles, but now it is worth to remember one thing - it is not recommended to look at the metrics globally, we need to take into account different factors in different segments. Ignoring this can lead to missed product growth points in non-obvious segments.
Each metric should be compared with something. If I’ll tell you that my conversion rate is 1,434% is it good or bad? Or if I’ll tell you that my conversion rate is 1,434%, and my competitor conversion rate is 2,3434% - is it good or bad? Alas, there is no obvious answer to this question. The bottom line is that conversion to the first purchase/order/subscription could be even 10% - but my expenses could be more than that. To determine precisely what is good and what is bad, it is necessary to add new metrics.
Let's look at an example. I spend $30 on getting a new customer, this metric called customer acquisition cost or CAC. On average, one paying customer spends $20 - average revenue per paying user, or ARPPU. In this case, my customer acquisition cost is more than the customer pays me. If such customer doesn’t make a second and third purchase, then the product economics is unprofitable. We reviewed a product economics basics in the earlier article. This example implies that a good conversion is not the one that is the biggest, but the one which positively effects product economics and business.
When you choose metrics for your product, it is crucial to look at a comprehensive picture and additionally to macro indicators also take into account micro metrics and product characteristics. The best way to identify vital product metrics or specific feature metrics is to create a metrics map. You can read more about metrics map in the article How to chose relevant product metrics.