In the process of product development, different teams have various improvement ideas to test. However, the launch of the experiment requires resources and time. Obvious that it is not possible to test absolutely all ideas at once, and it is crucial to understand which of the hypotheses are most important and in which sequence they will be tested.
In this article, we will share what logic we use to prioritize the hypotheses and how this helps us to create a hypotheses roadmap.
This is a simple one. The more complicated the hypothesis development and implementation process, the more time and money required.
In order to achieve representative results, a certain amount of data required. It is important to look at the amount of traffic on a particular funnel step where we are going to test the hypothesis, not at the total product traffic amount. It is critical to estimate in advance how long it will take to test the hypothesis, because one experiment can take several months.
Different metrics can validate experiments. Not every experiment validated by conversion rate. Such metrics as time per scenario / time per page / number of pages viewed and similar do not require accumulation, because these metrics do not have a business cycle. If the experiment metric is conversion / revenue / average check, then these metrics have a business cycle, but typically not too long. However, if our experiment aimed to test metrics like Life Time Value, then it’s more challenging, and you have to wait long enough before getting to analysis.
This is perhaps the most ambiguous parameter that appears in the priority assessment. It can even be called cumulative, and usually answers the following questions:
Based on these 4 parameters, we are able to prioritize an extensive list of hypotheses. Let's try this with an example.
Let's look at the table in details. We describe the hypothesis so that all team members are aware of what is being verified and where. Based on the knowledge gained, we prioritize one or another hypothesis. When all priorities set, we find the average between all quantitative estimates. The hypothesis with the highest SCORE is the most important, and then descending. According to this logic, our hypotheses will have the next priority.
Now the hypotheses are prioritized and can be executed.
Obviously, hypothesis prioritization is an important process. This document is an excellent way to align all team members and make decisions together based on the data you have. The format we presented is easily adapted to different teams and products, and you can easily add different requirements you think are necessary.