Skip to main content

​
Use Cases

Ratio metrics are used to add a more nuanced understanding - revenue went up, as did purchase volume, but did revenue/purchase go up in a way that was meaningful? Ratio metrics are also used to normalize metrics. For example, if you’re a company that rents out devices like scooters, you might run a scooter-level experiment but want to measure average revenue per distinct rider. You could make a metric of SUM(revenue)/COUNT_DISTINCT(rider_id) to calculate this normalized metric and have the metric be less influenced by scooters in popular areas which get lots of riders.

​
Calculation

At the unit level, ratio metrics will calculate both component metric’s unit level aggregation. At the group level, the mean is calculated as the total group calculation of the first metric, divided by the total group value of the second metric.
The denominator is not the number of units in the experiment; the normalization is by the denominator metric.
This would look like the SQL below: 
-- Denominator (Checkouts)
SELECT
  source_data.unit_id,
  exposure_data.group_id,
  COUNT(1) as denominator
FROM source_data
JOIN exposure_data
ON
  -- Only include users who saw the experiment
  source_data.unit_id = exposure_data.unit_id
  -- Only include data from after the user saw the experiment
  -- In this case exposure_data is already deduped to the "first exposure"
  AND source_data.timestamp >= exposure_data.timestamp
GROUP BY
  source_data.unit_id,
  exposure_data.group_id;

-- Numerator (Revenue)
SELECT
  unit_id,
  group_id,
  SUM(revenue) as numerator
FROM source_data
GROUP BY unit_id, group_id;

-- Group Level
SELECT
  group_id,
  SUM(numerator)/SUM(denominator) as mean
FROM denominator
-- full outer join depending on settings
LEFT JOIN numerator
USING (group_id)
GROUP BY group_id;

​
Methodology Notes

Ratio metrics require adjustment due to potential unit-level covariance between the numerator and the denominator. Statsig uses the delta method to estimate this adjustment. By default, Statsig treats ratio metrics as a conversion rate (unordered). That is, we only count numerator events for units that also performed the denominator event. This is configurable in Advanced settings, where you can switch to a simple ratio.

​
Options

  • Treat as conversion rate (unordered).
    • Control whether to include numerator events only if the unit also performed the denominator event, regardless of order. Uncheck for a simple ratio, which counts numerator events for all units, even if they never performed the denominator event.
  • Cohort Windows (Numerator and Denominator)
    • You can specify a window for data collection after a unit’s exposure. For example, a 0-1 day cohort window would only count actions from days 0 and 1 after a unit was exposed to an experiment
      • Only include units with a completed window can be selected to remove units out of pulse analysis for this metric until the cohort window has completed
  • Winsorization
    • Specify a lower and/or upper percentile bound to winsorize at. Winsorization and its thresholds can be specified for both the numerator and denominator of the ratio metric independently. All values below the lower threshold, or above the upper threshold, will be clamped to that threshold to reduce the outsized impact of outliers on your analysis
  • Baked Metrics
    • Baked Metrics allow you to specify how long a metric needs to mature. This is common in situations like chargebacks or cancellations. Statsig will delay loading the data until the window has elapsed, and only calculate pulse results for that metric if a unit’s metric has matured.