Investigating Metrics In A Product

5-minute read
product-analyst

To investigate Why is the metric for a feature X dropping by Y%

Introduction

It’s crucial to investigate and understand why certain metrics in a product are dropping. This guide will provide you with a step-by-step framework to approach such investigations and help you uncover the reasons behind the metric decline.

Step 1: Clarify

To begin the investigation, you need to clarify certain aspects related to the dropping metric:

  1. Definition and Calculation: Understand how the metric is defined and calculated. This will give you a clear understanding of what the metric represents.
  2. Magnitude of Change: Determine the extent of the drop in the metric. By quantifying the percentage decrease (Y%), you can assess the severity of the decline.

Step 2: Gather the Context

To gain a comprehensive understanding of the situation, gather contextual information regarding the dropping metric.

Timeframe

Evaluate the timeframe over which the metric change occurred. This will help you determine whether the drop was sudden or gradual.

  • One-time Sudden Change: Investigate if the drop was a result of external or internal factors.

    • External Factor: Look for any external events that could have impacted the metric, such as natural disasters or new competitors entering the market.
    • Internal Factor: Check for any bugs or recent feature releases that might have affected the metric.

  • Gradual (Progressive) Change: Determine if the drop is associated with a gradual effect.

    • Analyze if growing A/B test variant features or a gradual increase in the user base within a specific group (activity/demographic) is contributing to the decline.

Generic or Specific

Explore whether the dropping metric is correlated with other features or products.

  • Related Products/Features: Investigate if other related products or features have experienced a similar change in metrics.
  • Consider the concept of Cannibalization, where other related features may have experienced an increase in usage or metrics.

Segmentation

Understand if the dropping metric is specific to certain segments within various attributes. Apply the MECE Framework to analyze the segmentation.

  • User Demographic: Segment users based on age, geographic location, or platform.
  • User Activity: Analyze metrics based on daily active users (DAU), monthly active users (MAU), time spent, funnel to the feature, or user types (new, active, power users, etc.).
  • Market/Features: Assess if the drop is caused by a new feature cannibalizing existing ones or by the presence of bugs.

Step 3: Decompose the Metrics and Funnel

To pinpoint the cause of the dropping metric, decompose the metric itself and analyze the associated funnel and user journey.

  • Decompose the Metrics: Break down the metric into its underlying components. This will help you identify which part of the funnel is experiencing the highest change.
  • Analyze the Funnel: Understand the user journey leading to the metric. Examine each stage of the funnel to identify potential areas of improvement or issues.

Example Scenario: Food Ordering Company

Question: “Why has the average order value (AOV) dropped by 15% in the past month?”

Step-by-Step Answer:

  1. Clarify: Start by understanding how AOV is defined and calculated, and determine the exact percentage decrease.

    • AOV Definition: The average order value represents the average monetary value of orders placed by customers.
    • Calculation: AOV is calculated by dividing the total revenue generated from orders by the total number of orders within a given timeframe.
    • Magnitude of Change: The AOV dropped by 15% compared to the previous month.

  2. Gather the Context:

    • Timeframe: Analyze if the drop occurred suddenly or gradually.

      • Sudden Change: The drop occurred suddenly in the past month.
      • External Factors: Investigate if any external events or changes may have influenced the drop. For example, were there new competitors entering the market, or were there any disruptions in supply chains affecting food prices?

    • Generic or Specific: Investigate if related products or features have experienced similar metric changes or if cannibalization is occurring.

      • Related Products/Features: Check if there have been any changes or updates to the platform, such as the introduction of a new feature or a change in pricing structure, that could impact AOV.

    • Segmentation: Break down the AOV drop across user demographics or user activity metrics.

      • User Demographic: Analyze if the drop is consistent across different age groups, geographic locations, or platforms.
      • User Activity: Assess if the decline is more prominent among new users, active users, or power users, and identify any specific patterns in the user journey leading to lower AOV.

  3. Decompose the Metrics and Funnel: Analyze the underlying components of AOV and examine each stage of the ordering process to identify areas of significant change or potential issues.

    • Decompose AOV: Break down AOV by analyzing factors such as the number of items per order, average order size, or any changes in pricing and discounts.

    • Funnel Analysis: Evaluate each stage of the ordering process, including browsing the menu, adding items to the cart, and completing the checkout process. Identify any potential bottlenecks or issues that may have led to a decrease in AOV, such as a complicated checkout flow, lack of relevant product recommendations, or pricing inconsistencies.

By following these steps, you will be able to systematically investigate the drop in AOV and uncover the factors contributing to the decline. Remember to adapt and refine the framework based on the specific needs of your analysis and the nature of the product you are working with.