Before launching a feature, it is crucial to analyze existing data to make informed decisions about what to build. This process often leads to questions about the potential impact of adding, changing, or improving a feature.
Understanding Feature Change
To evaluate a feature change, it is essential to understand why someone proposed the feature and what goals it aims to achieve (similar to measuring success in a product. To determine if utilizing the feature change will benefit your product long-term goals, we must establish a connection between the goals of the proposed feature and the overall objectives.
Decide the Metric for the Feature update
see measuring success in a product for more details
Design Experiment
see ab-testing
Interpret Results
In general:
- Launch If:
- Success Metrics are all practically and statistically significant
- No violation of assumptions
- No negative change in guardrail metrics
- Do Not Launch if:
- Violation of assumptions
- Negative change in guardrail metrics
Understand Trade-off between metrics
How to make a decision when one metric goes up and one metric goes down? Mental Model: Eg. user acquisition & revenue: Expensive campaigns increase user acquisition but degrade revenue.
Solution: Align and discuss with stakeholders! Discuss about:
- Unify both positive and negative impact to one metric (Overall Evaluation Criterion).
- Segmentation of positive and negative on user groups and features.
- Discuss whether the overall result is desirable.