Separate who joins from what membership changes.
Matched difference-in-differences, monthly diagnostics and a constructed untreated world. The numbers below are generated from one current simulation, with uncertainty and assumptions kept visible.
The estimated change in customer payments
The two percentages use different denominators: the naive gap uses nonmembers’ post-launch spending; matched DiD uses members’ pre-launch spending. Neither is an estimate of profit.
Members were already different before launch
The spending gap compares all eventual adopters with all non-adopters before membership existed. The SMDs describe balance in pre-launch spending before and after matching; balanced levels do not establish parallel trends.
What the comparison identifies
Each eventual adopter is matched to one unique non-adopter using observed pre-launch spending, transactions, tenure, region and channel preference. The outcome includes all customers and zero-purchase periods. It estimates the average contrast over the full post-launch window for the retained adopters, including delayed joining and cancellation.
This is observational selection into membership, not randomized assignment or an intent-to-treat experiment. Causal interpretation requires conditional parallel trends, overlap, no anticipation and no interference. Passing a diagnostic cannot establish these assumptions.
The monthly path
September 2024 is the reference, fixed at zero. Intervals are pointwise 95% intervals, not simultaneous bands. Standard errors cluster by matched pair. The path mixes enrollment cohorts, cancellations and calendar conditions; it is not an effect by months since enrollment.
Diagnostics that can challenge the conclusion
Pre-launch level balance is a matching diagnostic, not a test of parallel trends. The table reports the placebo, joint monthly and linear pretrend tests, with Holm correction across seven distinct diagnostics. The pre-discount pretrend duplicates the overall net outcome and is excluded from this family.
A rejected diagnostic remains a warning. No seed was selected to make these tests pass. These checks reuse the pre-period used for matching; uncertainty from choosing the propensity model and matches is not fully captured.
Channels and simulation truth
All entries are dollars per customer per day. Coupled ATT compares factual payments with one untreated world using shared basket draws and independent Poisson thinning during active membership. This is a simulation benchmark under that coupling. One interval covering truth does not prove unbiasedness or correct repeated-sampling coverage.
How much would a trend violation change the estimate?
Adjusted effect = reported DiD − assumed differential untreated trend. The interval shifts by the same amount. This is an assumption scenario, not a fitted confounding model or a confidence bound on the violation.
Robustness to matching choices
Calipers and processing orders can change who is retained, so these rows need not target exactly the same population. Matching-order stability is not a correction for matching uncertainty.
Revenue is not profit
Fee collections and benefit amounts are descriptive cash totals. Causal net payments already include discounts and redeemed coins: subtracting them again double-counts benefits. The matched effect cannot automatically be extrapolated to unmatched adopters. Product costs, operating costs, refunds and revenue recognition are not modeled, so profitability is not established.