Where this gets tested in front of a real room

My own kids sometimes ask what I actually do at a bank, and I've learned the answer that sticks is always a demonstration. I got the same chance with a room of Year 8 and 9 students, and opened with the one that always lands hardest: a pair of sneakers, marked down live in front of them.

From the actual session: the envelopes, the handout, and the room.

To a teenager's brain, money isn't a calculator, it's closer to a calculator crossed with Instagram.

None of them will remember the word “anchoring” by lunch, and that's fine. What stuck was thirty hands going up for “$80 in savings” and getting told, on the spot, that they'd just fallen for it. That moment worked because it was theirs, said out loud, in front of the room. The envelope demo only half-had that: I narrated it from the front instead of putting real coins in real hands, and the parts of the talk they argued about afterwards were, without exception, the parts they'd actually touched. Next run, every pair of hands gets coins for that one.

I opened with South Park. The gnomes in the underpants-collecting episode have a business plan on a whiteboard: Phase 1, collect underpants. Phase 2, a question mark. Phase 3, profit. Pressed on Phase 2, they don't have an answer. I put that slide up first so the room could laugh before I told them it's also, more or less, our own roadmap slides.

The example that landed hardest: “just increase engagement”

I asked the room to map the outcome logic for a familiar one: a bank wants customers using the app more. The chain nearly writes itself: Customers → Engagement → Loyalty → Retention → Products per Customer → Profit. Simple. Or is it?

Everyone in the room could draw the arrow from Initiative to Profit. Almost nobody could defend it.

Neither the Kohavi statistic nor the Sowell quote did the real work here. What landed was watching people try to map their own Phase 2, live, for something they'd actually shipped, and hit the gnomes' wall themselves. Nobody enjoys realising their own success metric was a proxy for something else. The sequencing gave some of them an out, though: people who mapped their own initiative after seeing the digital-engagement twist already knew what was coming, so their version was less honest than the ones who went first. Worth running the exercise before the reveal next time, not after.

The assumption behind most banking-app design for the last decade has been: more visibility into your money is always good. Show people their real-time balance, show them what's “safe to spend” after upcoming bills, and they'll spend more responsibly. I opened this talk with a number that quietly breaks that assumption.

The less paying hurts, the more you buy, a “safe to spend” number is just credit's trick wearing a helpful face.

Most of the room had shipped a feature that removed friction somewhere in a payment flow, in the name of a smoother experience, without checking whether the friction they'd removed had been doing real protective work, an uncomfortable thing to sit with. Nobody sets out to build the credit-card effect into a savings feature. I ran the audit live against a stranger's balance screen, not theirs, and it showed anyway; a room auditing its own screens instead of watching me do it to someone else's would find more, and mean it more.

Economist John List has spent years asking a specific, unglamorous question: why do so many ideas that work beautifully in a small pilot fall apart the moment you try to run them for everyone? His book The Voltage Effect calls the pattern a “voltage drop,” and I'd already used it lightly in an earlier session on outcome logic. This talk was the deeper version, built around one of List's own field experiments, and the story I think most directly explains the “very best” trap product teams fall into without noticing.

The pilot that used your best people isn't evidence the idea works. It's evidence your best people work.

Ask any room who staffed their last pilot with their A-team, and most hands go up, on purpose, to give the idea its best shot. That instinct is exactly what CHECC's design pushes against, and it's the story that actually landed harder than Uber's, because it runs against instinct instead of just being a good result. List's point isn't subtle once you sit with it: stacking a pilot with your strongest people answers a different question than the one you meant to ask, and most teams never notice they asked the wrong one.

I went in for a routine check-up and a filling, mentioned somewhere between the X-ray and the drill what I actually spend my time thinking about, and spent the rest of the appointment quietly cataloguing behavioural economics. None of it looked designed. It didn't need to. It was happening anyway, and I'd bet it's already sitting in that practice's revenue numbers whether anyone there could name a single one of these mechanisms.

None of this was designed as behavioural economics. It didn't need to be. It was working anyway, and it's probably already sitting in that practice's revenue line, whether anyone there could name a single mechanism.

What stuck with me wasn't any one moment. It was how many of these were stacked into one ordinary appointment, unplanned, unnamed, and each one doing real work. A dental practice isn't a fintech or a subscription business, and nobody there is running an experimentation programme. It doesn't matter. The mechanisms don't ask permission. If I ran a version of this for a room of clinicians or small-business owners, I'd open with the same question I'd ask any product team: which of these are you already doing by accident, and which one, done on purpose, would actually move the number you care about?

Every version of this talk opens with the same problem: the room usually already has one of two wrong ideas about what behavioural economics is. Either it's “a fancy word for common sense,” or it's “psychology with extra steps.” Both miss the actual claim, and if I don't fix that in the first five minutes, every example after it lands as a trick instead of a discipline. So the first thing I put on the screen is this chart.

Economics is incentives and human decision-making, which is just another way of saying human behaviour. Classical economics sits at one end of that line: high on simplicity and scalability, because it assumes you think like a spreadsheet: rational, consistent, always maximising. Psychology sits at the other end: high on explainability, because it studies the real mechanism, but a full account of one person's decision doesn't scale into a rule you can ship. Behavioural economics sits deliberately between the two, informed by psychology, but still asking “so what should we predict, and when?” It assumes you think in part like a spreadsheet and part like Instagram: mostly rational, with specific, predictable biases layered on top.

The example I use to make that concrete: classical economics predicts a free product and a one-cent product should behave about the same: price is price, and a cent is close enough to nothing. It doesn't. Zero Price Effect research finds that dropping a price to genuinely free doesn't just lower the cost, it changes the category of the decision: demand for a free item jumps far more than a one-cent gap would predict. That's the line I put under the chart: classical economics gets that wrong, behavioural economics gets it right, and the reason is a specific, citable, replicated finding, not a vibe.

Why any of this matters for product and design work

Definition sorted, the next question is always the same: why does this matter here specifically? Because product and experience work already has a one-line job description most teams never say out loud: it exists to influence human behaviour, on purpose, towards an outcome. Jeff Patton's outcome framework puts that chain plainly: find it → try it → use it → keep using it → say good things, sitting underneath the acquisition, activation, retention, referral and revenue funnel a growth team already reports on every week. If product work is behaviour change, a real model of how people actually decide isn't a nice-to-have layered on top of the roadmap. It's the thing the roadmap is supposedly for.

“So it's just marketing with a science degree?”

Fair pushback, and one I get most times I give this talk. The honest answer is falsifiability: marketing wants something to work, and will keep the story if it does. Behavioural economics wants to know exactly when a mechanism stops working, and for whom, that's why every principle on this site carries a named study, a sample, and a methodology critique, not just a claim. I used to leave that distinction for the Q&A. Now I put it straight after this chart, because the room trusts everything that follows a lot more once they've heard it.

There's one objection to running a proper experiment that comes up in almost every room, usually from the most senior person in it: “why can't we just make the change and monitor a metric?” It's a reasonable question, a controlled test costs time, traffic, and engineering effort a simple launch doesn't. So instead of arguing the point, I show a real chart and let the room talk themselves into the wrong answer first. It's a live demo, not a controlled study of my own, but it's the fastest way I've found to make the actual risk felt instead of just stated.

The real source, and what it actually found

The example isn't hypothetical or mine. It's a real post from Airbnb's own engineering blog, and it's the one I build this whole talk around.

“So basically, I can't trust anything I haven't tested?”

Someone asks a version of this most times I run the reveal, usually sounding a little defeated. That's not quite the lesson, and I've learned to say so faster than I used to: it's not that metrics are untrustworthy, it's that a metric alone can't tell you why it moved, and a good story about why is exactly what makes people stop checking. The Airbnb example is memorable precisely because the story was so plausible: faster to scan, better mobile experience, matches how people search, that nobody would have thought to doubt it without the actual A/B result sitting underneath. I used to close this section with the confounder list. Now I close with the reveal line again, because that's the sentence people actually carry out of the room.

Agile Australia gave me a room of 200+ product and agile leaders and about 45 minutes. I've given a version of this talk before, the Uber apology finding shows up here too, and so does the Airbnb filter reveal, but this was the first time I'd built the whole thing around one frame: a “10x” way of thinking about product work, and why most organisations never get there. I opened the way I open most rooms: not with a definition, but with a number.

From the actual talk: AgileAus23, August 2023.

It's powerful, like a chainsaw, which means it can cut trees with ease, but also limbs.

That line closed the talk on purpose. The two heavyweight case studies I lean on most, the Uber apology finding and the Airbnb filter reveal, already have their own full write-ups on this site, and I didn't want to re-tell either one here just to pad this session out; both are linked below, exactly as I'd point a room to them afterwards. What this talk added on top was the frame: a “10x” way of judging whether applied behavioural economics actually worked: confidence of impact, speed to learning, scalable learning, probability of working, and the size of the measurable outcome, and most of the room's own honest answer, when I asked which of the five they were weakest on, was measurement. Not the theory. Not the idea. The part where you find out if you were right.