Roadmap slides tend to draw the same arrow: customers, then the initiative, then a blank, then profit.
On this site that blank already has a name: the Phase 2 Problem, after the South Park gnomes whose plan runs Phase 1 collect underpants, Phase 2 a question mark, Phase 3 profit. Teams fill the “?” with whatever is handy to report: engagement, NPS, sign-ups. Those figures are rarely the outcome leadership actually wants.
The field session stopped at a five-step whiteboard exercise. This report continues from there with one concrete launch instead of a generic chain. A bank ships a “set a savings goal” feature, and each stage gets a named metric plus the proxy that usually pretends to stand in for it.
“People tried it” is only the middle of that map. The last question is whether the goal is still active and still funded months later, and launch reviews almost never get that far.
Each link assumes something specific about what people will notice, find, try, finish, and keep doing. Break any one of those bets and the initiative can fail while the dashboard still looks fine.
Writing the chain down early is how those assumptions get checked while a fix still costs little.
The story in four parts
A release date, a goal record, a screen view all happened. None of them show that saving behaviour changed.
Knowing the feature exists does not mean finding it later. One try does not mean finishing setup. Finishing setup does not mean the goal is still funded next month.
Sign-ups, opens, and starts land in the log the same day. Whether a goal survives real life takes months to learn, so that check gets deferred and then dropped.
Name a metric for each join before release, and the weak step shows up while there is still time to redesign it, instead of after a quarter spent arguing about whether the feature “worked.”
The evidence at a glance
On Microsoft's experimentation platform, about a third of features under test actually moved the metric they were meant to improve.
List's applied work finds that programmes with a real pilot lift usually keep only a fraction of that lift once the full rollout begins.
In real bank data, daily-login engagement tracked negatively with financial wellbeing: the opposite reading from what the metric was treated as measuring.
In a randomised field trial, plain reminder messages increased how much people saved well after account opening, past the sign-up window.
Program evaluators have a precise vocabulary for the gnomes' blank middle. An output is whatever the activity itself produces: a shipped feature, a training session held, a goal record created. An outcome is the later change in behaviour or condition that output is supposed to cause, counted on its own timeline. Outputs are easy to tally the day they appear. Outcomes need time, and a separate check.
Logic models have drawn that split for decades. The W.K. Kellogg Foundation's guide is the version most cited. It keeps outputs apart from short-term, medium-term, and long-term outcomes so “we ran the activity” cannot pass for “the activity worked.” Shipping the savings-goal feature is the output. Whether customers save differently because of it is the outcome, several links downstream.
From here the report walks that gap on one familiar product shape. A bank launches “set a savings goal”: name, target amount, date, progress on the home screen. Each stage gets a metric and a trap.
The stages below are sequential filters, not five independent audits. Each step keeps a smaller share of whoever arrived. Applied to the savings-goal example, they start from an illustrative 1,000 customers who noticed the feature. The counts are illustrative, drawn to show how a funnel of this shape shrinks, not results from a live product. Expand any stage for the smaller steps inside it. Even “send a notification” hides three things that each have to succeed.
Sees the feature: banner, notification, or homepage tile.
Opens goal-setting days later, with no campaign push.
Names the goal: first substantive input.
Finishes setup: amount, date, and funding source.
Goal still active and funded at day 90.
Of the original 1,000, about 95 remain three months later, roughly one in ten. Each join shed about half of whoever reached it, in the same ballpark List saw between pilot and full rollout. That persistence count is the one worth taking into a steering meeting. The 1,000 at the top of the funnel, and the raw tally of goals ever created, both look healthier and both say less.
The map already split this stage into sent, delivered, and opened. Trial cannot start until someone knows the feature is there. The usual trap is logging impressions as if they were people. Show the same customer a banner five times with no notice taken, and the log records five impressions and zero real awareness. What matters is the share of eligible customers reached at least once through a channel they would plausibly see, not how many times the banner painted to a screen.
How can a banner rack up a million impressions and still miss awareness?
Impressions count how often something rendered. A push sent to 500,000 customers and opened by 4,000 still shows 500,000 impressions in many dashboards. The people who actually became aware sit closer to 4,000, more than a hundred times smaller.
What it teaches: report unique customers reached once, through a channel that records an open or view. Technical delivery volume is a different number.
On the map above, 62% of the aware group reached this stage: open the app, find the tile. Awareness decays. Someone who saw the March announcement may still want the feature in July, and by then it has to be findable without the original banner. The trap is mixing campaign-driven traffic with organic finds into one visit count. Only the organic share answers whether the feature is discoverable in ordinary app use after the launch push ends.
Why is a launch week's traffic figure silent about month three?
Almost all first-week visits to the goal-setup screen ride on the campaign. They say little about whether people can still find the flow later. Measure organic entry: visits that arrive through normal in-app navigation after the campaign has stopped.
What it teaches: a feature that only appears while it is being actively pushed has not solved discovery. It borrowed the campaign's attention, and that attention expires.
Fifty-five percent of the discovery group made it here on the map: screen opens, then names the goal. This is where “clicked a button” most often gets filed as “used the feature.” Counting an entry-point tap as trial is the trap. Plenty of people open the flow and back out immediately. Trial means reaching at least the first meaningful step inside the flow, past the front door.
Why does a tapped button fail to prove anyone tried the feature?
“Set a Goal” taps include everyone who poked out of curiosity and left on the next screen. Real trial is the share who reached the first substantive input: naming the goal. Opening the entry screen alone does not clear that bar.
What it teaches: the first screen of a flow is still marketing. Trial begins on the first screen that asks the customer to do something.
Another 53% dropped before this stage on the map: amount, date, funding source. Starting a flow and finishing it are different outcomes, and most of the feature's real friction lives in that gap. Reporting “flow starts” as adoption is the trap. Full completion means name, target amount, target date, and funding source all set, whether in one sitting or across return visits.
Where does a savings-goal flow actually lose people?
A four-step flow (name, amount, date, funding source) that only reports an overall “started” count hides which step is bleeding users. Per-step completion rates turn a launch review from one disappointing aggregate into a concrete fix list.
What it teaches: this is the field session's Rube Goldberg check applied literally: find the weakest individual join, not only the chain's overall pass rate.
After one more 53% drop, the map ends on three checkpoints at 30, 60, and 90 days. Launch reviews rarely get here: the wait is longest, and skipping is easiest. Completing setup does not keep a goal alive. Someone can finish every field and then stop funding, or delete the goal within weeks, while a “goals created” dashboard never shows the decay.
Why does “goals created” say nothing about whether the feature is working?
“Goals created” is a one-time output that never shrinks, even after every goal has been abandoned. Use a cohort persistence rate instead: of goals created in a given month, what share are still active and have received at least one contribution at 30, 60, and 90 days.
What it teaches: persistence does not arrive free with completed onboarding. A randomised field trial found that simple reminder messages raised how much people actually saved months after opening an account. Sustained saving responds to ongoing prompts, beyond a one-time setup flow.
The five stages above are the same mistake in five costumes: taking a metric that is easy to collect and treating it as the outcome that matters. This site's experiment-validity report names that failure construct validity: a real number standing in for a concept it does not capture. “Engagement” for “loyalty,” and “goals created” for “customers who are actually saving more,” are the same substitution, one in a paper and one on a product dashboard.
This is the field session's review list written down for a deadline. Four checks, not forty, because a step people will run tired on a Friday has to stay short.
Each stage above reads as one tidy step. Underneath, each is a bet that a specific behavioural assumption holds.
A notification earns real attention. Intention survives without a cue. A future goal beats today's spend. One more form field does not cost the customer. A finished setup stays funded without more help.
Mapping the chain names those assumptions early enough to check them before a quarter's spend is gone. A named assumption points at a mechanism (salience, a missing signpost, present bias) that behavioural economics can target for a fix, not only for measurement.