On Iterating and Growth

[Product & Experimentation]

         An overview of motivations and metrics emerging in growth
Isaac Gelman, 17 Nov, 2021

The Customer is Always Right

Digital products are effective at connecting consumers, but our increasingly online transactions have placed product creators far from their user.

When you sell your shoes in a physical store all day, you get to experience your customers emotions through their words, body language, facial expressions. The inflection in their voice. You learn with each customer where to improve: why did they turn away? And you learn what delights them — the quality material, the comfortable shape, the stylish design. With care, iteration, and skill, you’re suddenly the hottest shoe shop on the block.


So, for products where that interaction is online, how do you get that deep understanding of your customer? How do you introspect about your product?1 What changes can you make to your product to make it truly resonate with customers?

In the modern age, short attention spans make fast resonance required for viral growth.  

Breaking it Down

Notice that the customer’s journey through the shoe store probably has a consistent temporal structure.

First they hear about the store, maybe from a colleague. Then, they window shop. They then step into the store and wander. A certain item catches their eye. They ask a salesperson a few clarifying questions. Maybe they haggle on the price. Finally, they leave the store. The next day, they put on the shoes, and a colleague asks where they got their shoes.

Nearly every customer will have a “journey” similar to this one, albeit with many permutations. The funnel, shown below, is a framework for tracking the stages of this customer journey. The funnel is “above” an income statement; the bottom of the funnel is exactly the top line of the income statement.



The framework is equally applicable to startups and large corporations alike to refine product market fit by delivering the must have experience to the right customers.

Experimentation and Learning

The iterative formula:
    1. Create a hypothesis about a marketing or product change to improve one of these metrics
    2. Code the change
    3. A/B test (or something more sophisticated from AirBnB) the change to track whether the hypothesis was correct
    4. If the change improved the metric, then adopt it. Else, scrap it
    5. Record what you learned
    6. Repeat

In laboring to make the formula concise, it has become brutish and impersonal.

At the core, the goal of experimentation isn’t really to optimize a metric by a few points. It is to learn what the customer wants and needs. What they respond to. What their motivations are. Synthesizing and recording the experiment’s takeaways are key to understanding users, and thus improving all future hypotheses to come. Remember the shoe shop.

Some of these successful hypotheses can even become new features as a new product niche is discovered. You might be surprised by the results of experiments, or even that products we use everyday.  

One interesting such surprise at Spotify:
    When [Matt Ogle] joined Spotify, the music-streaming company, he helped build a product called Discover Weekly, a personalized list of 30 songs delivered every Monday to tens of million of users.
    The original version of Discover Weekly was supposed to include only songs that users had never listened to before. But in its first internal test at Spotify, a bug in the algorithm let through songs that users had already heard. “Everyone reported it as a bug, and we fixed it so that every single song was totally new,” Ogle told me.
    But after Ogle’s team fixed the bug, engagement with the playlist actually fell. “It turns out
having a bit of familiarity bred trust, especially for first-time users,” he said. “If we make a new playlist for you and there’s not a single thing for you to hook onto or recognize—to go, ‘Oh yeah, that’s a good call!’—it’s completely intimidating and people don’t engage.” It turned out that the original bug was an essential feature: Discover Weekly was a more appealing product when it had even one familiar band or song. The Atlantic.

Without step #5, a growth team doesn’t synthesize their learning to improve their next product hypotheses.

Spotify’s Discover Weekly team could easily have missed the insight that new users need a familiar stepping stone to be comfortable with a new personalized product. Each experiment is an opportunity to learn about the market.

The Metrics

The move to online trade has, however, given us an empirical way to optimize the quantitative metrics we do have. The list of metrics for each stage of the funnel is below.

To outline:
1. How users discover the product (acquisition)
2. How users convert to a paying customer (activation)
3. How long they stick around as a paying customer (retention)
4. How do customers spread the word (referral)

The proper functioning of each of these stages in the journey is critical.

Acquisition and Activation

What gets your users “in the door”?
Acquisition is the mirror side of the “creator economy” where individuals and companies create quality, free content. Advertisers can then purchase space in these digital assets. The sponsors drive value to users by funding the maintenance of the digital asset and recommending (hopefully) relevant products.

What drives your users to make their first purchase? What diverts them?
An excellent case study in activation is F2P mobile games. They “hook” users and focus on retention and activation. They use simple “tricks” like investment in a digital asset and social competition.2

While many users will continue to play freely, a significant portion of users activate into revenue by making in-game purchases to get a status boost or “leg up” against their friends. Furthermore, these purchases are randomized to create more addictive variable rewards for the user.


Cost per Click (CPC)
    This is getting expensive.
Conversion Rate
Customer Acquisition Cost (CAC) = CPC / Conversion Rate
Bounce Rate
Blended CAC

Retention and Revenue

Why do users return to your product?
How do you get paid?

Churn rate
Payback period


Why do users share your product with their friends?

Referral Engines:
Explicit referral:
    Give $5 / get $5 (AirBnB, Robinhood)
    Discounts, money, free stuff (Dropbox), early access
Implicit referral: social proof (passive visibility of the product) — users share with their audience on social media, in their physical communities, and in their digital communities

Referrals and WOM are key in network effect situations, when a platform becomes

Viral Cycle Time
    The time it takes for an user to invite another user
Viral Coefficient
    Average invites per user * conversion rate
Net Promoter Score
    “How likely would you be to recommend X?” NPS = Promoters% - Detractors%

Depart from Averages

It’s hard to learn about people from averages.3

    Conversion rates are averages
    Bounce rates are averages
    LTVs are averages
Average metrics are very helpful in measuring improvement, but hypotheses are more likely to come from being in the shoe shop.

I look forward to seeing the emerging tools and levers for founders and PMs to get more personal interaction with customers. A close friend of mine is building such a widget called Mantis Chat - you can check it out here if interested.


[1] One on one sales calls? Discord and Slack communities? What is your community-market-fit?
[2] As outlined in a way-too-verbose book called Hooked by Nir Eyal. OK, there are some good frameworks in the book. But it’s still way too long. And the author is obsessed with email.
[3] A classic example: If you know that there are 11 people in a room with a total wealth of $101M, it’s more likely that we have one individual worth $100M and 10 individuals worth $100k, rather than ten individual deca-millionaires.


Created with the Wolfram Language