post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Artificial Intelligence (AI) & Machine Learning (ML)

Microsoft lays off 3% of workforce while renogiating partnership with OpenAI

Podcast

How to build the right product culture during transformation – Joca Torres (Product Consultant)

B2B product management

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Context, intent, change, purpose, constraints – a model for communicating requirements

5 difficult conversations and how to have them

DoorDash eats Deliveroo while OpenAI makes its biggest buy: This week’s news roundup

Why great product managers think like therapists

Tools and techniques to scale product teams – Charlotte King (Lead Product Manager, eBay)

Eira is an editor for Mind the Product. She's been a business journalist, editor, and copywriter for longer than she cares to think about.

Eira Hayward

<p>Experiments allow us to measure the impact of our products and determine what works and what doesn’t. They should be a quick and inexpensive way for product managers to validate and prioritise their ideas. In the first of this two-part series we’ll dig into the types of experimentation product managers can use and reveal some of the pitfalls and challenges.</p>
<h3>In brief:</h3>
<ul>
<li><span style="font-weight: bold;">Generative experiments are qualitative and start with learning about your customers’ context and problems.</span></li>
<li><span style="font-weight: bold;">Evaluative or quantitative experiments start with a hypothesis which defines the type of test you use.</span></li>
<li><span style="font-weight: bold;">Make sure you understand the value of qualitative research and know how to analyse metrics before you embark on quantitative research. </span></li>
<li><span style="font-weight: bold;">Every experienced product manager should know how to run a high-quality customer research interview and how to conduct an A/B test.</span></li>
<li><span style="font-weight: bold;">Document your biases and get others to review your experiments before you put anything into production.</span></li>
<li><span style="font-weight: bold;">Make sure you always refer back to the bigger picture.</span></li>
</ul>
<h2>What kinds of experimentation can we/should we do?</h2>
<p>Experiments are either generative, in that they don’t necessarily start with a hypothesis and generate many ideas, or evaluative, where they test a hypothesis for a clear yes or no result.</p>
<p>Product consultant, and former director of training at Mind the Product, <a href="https://www.linkedin.com/in/rosemaryelizabethking/" target="_blank" rel="noopener">Rosemary King</a> comments that the place to start in any business is learning about and understanding your customers&#8217; context and problems. That’s done through generative qualitative research.</p>
<p>The first experiment then for any product manager is to validate customers through customer interviews. Says Rosemary: “It’s very generative. You’re not asking anything specific, you’re asking what problems they have and how they solve them currently. You generate a lot of information around the challenges and opportunities that exist for clients and that moves you on to brainstorming and ideation about what you can build.”</p>
<p>From there you can move to evaluation and experiments. “In the evaluative stage you want to gather as much realistic data as possible without having to build anything really expensive,” says Rosemary.</p>
<p>In a startup especially it’s important to be able to build up a community and have a group of early adopters who pay attention to what you say. As first steps Rosemary suggests doing things like creating a landing page for people to sign up, or a Facebook group to see how many people you attract. “For a startup, building a community is your first experiment. Once you&#8217;ve built up a semblance of a community, you can start testing the content, to see what content resonates, what makes people engage, what drives behaviour.”</p>
<h2>How do we know what test to use when?</h2>
<p>The hypothesis defines the type of evaluative test you use. So make sure you start with a hypothesis and don’t fall into the trap of getting excited by a methodology and failing to take a holistic view of what you need to. <a href="https://www.linkedin.com/in/terry-lee-296a089/?originalSubdomain=uk" target="_blank" rel="noopener">Terry Lee</a>, Senior Product Manager at <a href="https://flo.health/" target="_blank" rel="noopener">Flo Health</a>, comments: “Don’t start with ‘I want to run a fake door test’. Start with the hypothesis that you think ‘you can improve retention by 5%’, or whatever it may be.”</p>
<p>Sometimes it may not be obvious what test to use, even with a strong hypothesis. So draw on the expertise of the people around you. If the business has analysts or UX designers who work closely with the product team, then discuss your potential experiments with them.</p>
<p>Terry says that you must also understand and document your biases along the way so that anyone else looking at the test understands they have been taken into account. “At Flo, our analysts independently review all our experiments,” he says. “As a product manager I can look at the live data coming in. But I will always discuss with our analysts before saying this is the winner and putting something into production.”</p>
<p>Terry doesn’t try to match specific experiments to a given situation, but rather to keep in mind a general idea of what’s appropriate where: “For example, if you&#8217;re working on a low traffic product or a low traffic area of the product, then you should be running things like A/B tests  &#8211; just this or that. You should be limiting your variation because you’ll struggle to get to statistical significance in an acceptable time with multi-variant testing.”</p>
<h2>What tests should all product managers know how to run?</h2>
<p>There are a multitude of proven tests and experiments for product managers, from straightforward A/B testing to complex and highly nuanced card sorting exercises. If they don’t have access to colleagues like analysts and UX experts, then, as Rosemary says, every product manager should at least have read a few books (<a href="https://www.amazon.co.uk/Lean-Startup-Innovation-Successful-Businesses/dp/0670921602" target="_blank" rel="noopener">Eric Ries’ Lean Startup</a> is the obvious place to start) or blogs on experimentation. She says that the resources on the <a href="https://kromatic.com/innovation-resources" target="_blank" rel="noopener">Kromatic website</a> are very useful.</p>
<p>She says that a high-quality customer research interview is the first experiment everybody should know. She adds: “Usually I don&#8217;t recommend PMs jump into A/B testing or multivariate testing until they’ve become at running more qualitative experiments and are structuring hypotheses around what they need to learn with ease. They need to have absorbed experimentation templates and structures, and understand the boundaries of experiments.”</p>
<p>Rosemary recommends only moving on from qualitative to quantitative experiments once you understand:</p>
<ul>
<li>How long an experiment should run</li>
<li>How your audience should be segmented for experimentation</li>
<li>What false negatives and false positives might look like</li>
</ul>
<p>As an intermediate step between qualitative and quantitative experiments, Rosemary also recommends you develop the skills to analyse metrics. “You should be starting with qualitative experiments,” she says. “Then when you really understand how to think about metrics and how to measure if something is working, then you can move to quantitative experiments.”</p>
<p>But if there’s one test that every experienced product manager should know how to conduct, it’s an A/B test and its multivariate version. Says Terry: “Simple A/B tests, simple split tests, fake door tests, everything else apart from that is quite nuanced to your particular situation. And, to be honest, you could probably achieve almost anything with an A/B test.”</p>
<h2>What pitfalls of experimentation should product managers be aware of?</h2>
<ul>
<li>Resist the urge to leapfrog over qualitative data in favour of quantitative data because it feels more scientific</li>
<li>Try to keep your tests as simple as possible. Don’t jump into complex tests without a full understanding of the questions you want to ask.</li>
<li>If your organisation’s data tracking is not robust enough then any testing you do won’t tell you anything. As Rosemary says, “it’s just going to make you more confused”.</li>
<li>Make sure you address the bigger plan. Product teams are often working at such speed that they fail to fully address biases or take an holistic view of what they’re trying to achieve, says Terry.  He’s seen teams just focus on short-term metrics and end up adversely affecting other teams and delivering an inconsistent user experience.</li>
</ul>
<p>Look out for the next part of this series (publishing on Monday 28th March)  when we’ll dig into some case studies and examine some experimentation successes and failures.</p>
<h2>Further reading</h2>
<ul>
<li><a href="https://www.mindtheproduct.com/when-a-b-tests-arent-enough-by-scott-castle/" target="_blank" rel="noopener">When A/B Tests Aren’t Enough, by Scott Castle</a></li>
<li><a href="https://www.mindtheproduct.com/data-driven-blunders-and-how-to-avoid-them/" target="_blank" rel="noopener">Data-Driven Blunders and how to Avoid Them by Jorge Rodriguez-Ramos</a></li>
<li><a href="https://www.mindtheproduct.com/evaluating-experiments-when-the-numbers-lie/" target="_blank" rel="noopener">Evaluating Experiments: When the Numbers lie by Jacob de Lichtenberg</a></li>
<li><a href="https://www.mindtheproduct.com/how-do-you-implement-effective-product-experimentation/" target="_blank" rel="noopener">How do you Implement Effective Product Experimentation? By John Noronha</a></li>
</ul>


core/freeform

Lean Product Management

Hooked! Connect, engage and inspire with storytelling by Yamini Naidu

Lean Product Management by Itamar Gilad

Evidence-Based Product Backlogs, by John Pagonis

How Design Thinking, Lean, and Agile Work Together by Jonny Schneider

How Medicine and Research Translates to Patients by Shree Patel

How to Influence the Business by David J Bland

Lean, Agile, & Design Thinking by Jeff Gothelf

Lucia Adams – How to Innovate in Organisations That Don’t Like Failure

Tom-Tom Erik Isaksen – Disrupt Yourself or Die

Debbie Wren – Scaling Autonomous Teams

Experimentation for product teams: Part 1

ProductTank Pune

ProductTank Hyderabad

ProductTank Valencia

ProductTank Auckland

ProductTank Helsinki

ProductTank Barcelona

ProductTank Lomé

ProductTank Wellington

ProductTank San Diego

ProductTank Canberra

ProductTank Perth

ProductTank Hong Kong

ProductTank Bangkok

ProductTank Doha

ProductTank Tel Aviv-Yafo

ProductTank Baku

ProductTank Reston

ProductTank Raleigh

ProductTank Charlotte

ProductTank Washington

ProductTank Richmond

ProductTank Cluj-Napoca

ProductTank Zagreb

ProductTank Amsterdam

ProductTank Edinburgh

ProductTank Paris

ProductTank Hamburg

ProductTank München

ProductTank Manchester

ProductTank Bristol

ProductTank Oxford

ProductTank Madrid

ProductTank Des Moines

ProductTank Dallas

ProductTank Portland

ProductTank Curitiba

ProductTank Chicago

ProductTank Austin

ProductTank Philadelphia

ProductTank Montevideo

ProductTank São Paulo

ProductTank Los Angeles

ProductTank Gdansk

ProductTank Stuttgart

ProductTank Pittsburgh

ProductTank Bangalore

ProductTank Chennai

ProductTank Mumbai

ProductTank Fribourg

ProductTank Norwich

ProductTank Zürich

ProductTank Newcastle Upon Tyne

ProductTank Jacksonville

ProductTank Florianópolis

ProductTank Tallinn

ProductTank Denver

ProductTank San Francisco

ProductTank Sheffield

ProductTank Tokyo

ProductTank Valletta

ProductTank Vancouver

ProductTank Milano

ProductTank Columbus

ProductTank Belfast

ProductTank Montréal

ProductTank Seattle

ProductTank Lausanne

Product Management Foundations BST

Product Management Foundations ET

ProductTank Reading

ProductTank Bielefeld

Communication & Alignment BST

Communication & Alignment ET

ProductTank Frankfurt

ProductTank Bonn

Experimentation for product teams: Part 1

Eira Hayward

Read more