post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Customer

How behavioral economics can improve product strategy

Product Roadmap

Why most product roadmaps fail: Phil Hornby (Product Leadership Coach) – ProductTank Porto

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Join us at #mtpcon London in 2025 – mark your calendars

Connecting fans to creators at Spotify (Dariusz Dziuk, Product Lead, Spotify)

Why teams often struggle with OKRs: A conversation with Jeff Gothelf

Deep dive: Crafting your product vision and mission

5 ways to invent and simplify as an AI/ML product manager

Eira is an editor for Mind the Product. She's been a business journalist, editor, and copywriter for longer than she cares to think about.

Eira Hayward

<p>Experiments allow us to measure the impact of our products and determine what works and what doesn’t. They should be a quick and inexpensive way for product managers to validate and prioritise their ideas. In the first of this two-part series we’ll dig into the types of experimentation product managers can use and reveal some of the pitfalls and challenges.</p>
<h3>In brief:</h3>
<ul>
<li><span style="font-weight: bold;">Generative experiments are qualitative and start with learning about your customers’ context and problems.</span></li>
<li><span style="font-weight: bold;">Evaluative or quantitative experiments start with a hypothesis which defines the type of test you use.</span></li>
<li><span style="font-weight: bold;">Make sure you understand the value of qualitative research and know how to analyse metrics before you embark on quantitative research. </span></li>
<li><span style="font-weight: bold;">Every experienced product manager should know how to run a high-quality customer research interview and how to conduct an A/B test.</span></li>
<li><span style="font-weight: bold;">Document your biases and get others to review your experiments before you put anything into production.</span></li>
<li><span style="font-weight: bold;">Make sure you always refer back to the bigger picture.</span></li>
</ul>
<h2>What kinds of experimentation can we/should we do?</h2>
<p>Experiments are either generative, in that they don’t necessarily start with a hypothesis and generate many ideas, or evaluative, where they test a hypothesis for a clear yes or no result.</p>
<p>Product consultant, and former director of training at Mind the Product, <a href="https://www.linkedin.com/in/rosemaryelizabethking/" target="_blank" rel="noopener">Rosemary King</a> comments that the place to start in any business is learning about and understanding your customers&#8217; context and problems. That’s done through generative qualitative research.</p>
<p>The first experiment then for any product manager is to validate customers through customer interviews. Says Rosemary: “It’s very generative. You’re not asking anything specific, you’re asking what problems they have and how they solve them currently. You generate a lot of information around the challenges and opportunities that exist for clients and that moves you on to brainstorming and ideation about what you can build.”</p>
<p>From there you can move to evaluation and experiments. “In the evaluative stage you want to gather as much realistic data as possible without having to build anything really expensive,” says Rosemary.</p>
<p>In a startup especially it’s important to be able to build up a community and have a group of early adopters who pay attention to what you say. As first steps Rosemary suggests doing things like creating a landing page for people to sign up, or a Facebook group to see how many people you attract. “For a startup, building a community is your first experiment. Once you&#8217;ve built up a semblance of a community, you can start testing the content, to see what content resonates, what makes people engage, what drives behaviour.”</p>
<h2>How do we know what test to use when?</h2>
<p>The hypothesis defines the type of evaluative test you use. So make sure you start with a hypothesis and don’t fall into the trap of getting excited by a methodology and failing to take a holistic view of what you need to. <a href="https://www.linkedin.com/in/terry-lee-296a089/?originalSubdomain=uk" target="_blank" rel="noopener">Terry Lee</a>, Senior Product Manager at <a href="https://flo.health/" target="_blank" rel="noopener">Flo Health</a>, comments: “Don’t start with ‘I want to run a fake door test’. Start with the hypothesis that you think ‘you can improve retention by 5%’, or whatever it may be.”</p>
<p>Sometimes it may not be obvious what test to use, even with a strong hypothesis. So draw on the expertise of the people around you. If the business has analysts or UX designers who work closely with the product team, then discuss your potential experiments with them.</p>
<p>Terry says that you must also understand and document your biases along the way so that anyone else looking at the test understands they have been taken into account. “At Flo, our analysts independently review all our experiments,” he says. “As a product manager I can look at the live data coming in. But I will always discuss with our analysts before saying this is the winner and putting something into production.”</p>
<p>Terry doesn’t try to match specific experiments to a given situation, but rather to keep in mind a general idea of what’s appropriate where: “For example, if you&#8217;re working on a low traffic product or a low traffic area of the product, then you should be running things like A/B tests  &#8211; just this or that. You should be limiting your variation because you’ll struggle to get to statistical significance in an acceptable time with multi-variant testing.”</p>
<h2>What tests should all product managers know how to run?</h2>
<p>There are a multitude of proven tests and experiments for product managers, from straightforward A/B testing to complex and highly nuanced card sorting exercises. If they don’t have access to colleagues like analysts and UX experts, then, as Rosemary says, every product manager should at least have read a few books (<a href="https://www.amazon.co.uk/Lean-Startup-Innovation-Successful-Businesses/dp/0670921602" target="_blank" rel="noopener">Eric Ries’ Lean Startup</a> is the obvious place to start) or blogs on experimentation. She says that the resources on the <a href="https://kromatic.com/innovation-resources" target="_blank" rel="noopener">Kromatic website</a> are very useful.</p>
<p>She says that a high-quality customer research interview is the first experiment everybody should know. She adds: “Usually I don&#8217;t recommend PMs jump into A/B testing or multivariate testing until they’ve become at running more qualitative experiments and are structuring hypotheses around what they need to learn with ease. They need to have absorbed experimentation templates and structures, and understand the boundaries of experiments.”</p>
<p>Rosemary recommends only moving on from qualitative to quantitative experiments once you understand:</p>
<ul>
<li>How long an experiment should run</li>
<li>How your audience should be segmented for experimentation</li>
<li>What false negatives and false positives might look like</li>
</ul>
<p>As an intermediate step between qualitative and quantitative experiments, Rosemary also recommends you develop the skills to analyse metrics. “You should be starting with qualitative experiments,” she says. “Then when you really understand how to think about metrics and how to measure if something is working, then you can move to quantitative experiments.”</p>
<p>But if there’s one test that every experienced product manager should know how to conduct, it’s an A/B test and its multivariate version. Says Terry: “Simple A/B tests, simple split tests, fake door tests, everything else apart from that is quite nuanced to your particular situation. And, to be honest, you could probably achieve almost anything with an A/B test.”</p>
<h2>What pitfalls of experimentation should product managers be aware of?</h2>
<ul>
<li>Resist the urge to leapfrog over qualitative data in favour of quantitative data because it feels more scientific</li>
<li>Try to keep your tests as simple as possible. Don’t jump into complex tests without a full understanding of the questions you want to ask.</li>
<li>If your organisation’s data tracking is not robust enough then any testing you do won’t tell you anything. As Rosemary says, “it’s just going to make you more confused”.</li>
<li>Make sure you address the bigger plan. Product teams are often working at such speed that they fail to fully address biases or take an holistic view of what they’re trying to achieve, says Terry.  He’s seen teams just focus on short-term metrics and end up adversely affecting other teams and delivering an inconsistent user experience.</li>
</ul>
<p>Look out for the next part of this series (publishing on Monday 28th March)  when we’ll dig into some case studies and examine some experimentation successes and failures.</p>
<h2>Further reading</h2>
<ul>
<li><a href="https://www.mindtheproduct.com/when-a-b-tests-arent-enough-by-scott-castle/" target="_blank" rel="noopener">When A/B Tests Aren’t Enough, by Scott Castle</a></li>
<li><a href="https://www.mindtheproduct.com/data-driven-blunders-and-how-to-avoid-them/" target="_blank" rel="noopener">Data-Driven Blunders and how to Avoid Them by Jorge Rodriguez-Ramos</a></li>
<li><a href="https://www.mindtheproduct.com/evaluating-experiments-when-the-numbers-lie/" target="_blank" rel="noopener">Evaluating Experiments: When the Numbers lie by Jacob de Lichtenberg</a></li>
<li><a href="https://www.mindtheproduct.com/how-do-you-implement-effective-product-experimentation/" target="_blank" rel="noopener">How do you Implement Effective Product Experimentation? By John Noronha</a></li>
</ul>

Experimentation for product teams: Part 1

Eira Hayward

Read more

Comments