post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Customer

How behavioral economics can improve product strategy

Product Roadmap

Why most product roadmaps fail: Phil Hornby (Product Leadership Coach) – ProductTank Porto

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Join us at #mtpcon London in 2025 – mark your calendars

Connecting fans to creators at Spotify (Dariusz Dziuk, Product Lead, Spotify)

Why teams often struggle with OKRs: A conversation with Jeff Gothelf

Deep dive: Crafting your product vision and mission

5 ways to invent and simplify as an AI/ML product manager

Eira is an editor for Mind the Product. She's been a business journalist, editor, and copywriter for longer than she cares to think about.

Eira Hayward

In this #mtpcon <a href="https://www.mindtheproduct.com/mtpcon/san-francisco/" target="_blank" rel="noopener">SF+Americas 2022</a> Spotlight session, sponsored by <a href="https://launchdarkly.com/" target="_blank" rel="noopener">LaunchDarkly</a>, the company’s Principal Product Manager Cameron Savage looks at what stops teams from running experiments.
Watch this video or read on for key highlights from the talk.
<ul>
<li>Experiments can be seen as additional work</li>
<li>Commonly challenges come with engineering, data ingestion, decision making and product culture</li>
<li>Teams can find it hard to capture data and to work with it</li>
<li>Psychological safety and support are needed for successful experiments</li>
</ul>
Cameron heads up the decision science team at feature management platform LaunchDarkly and talks to hundreds of customers of all shapes and sizes in different industries and with different goals.  He says these conversations show a fairly consistent set of themes that prevent product teams from measuring their work in a way that they feel good about. In this talk he looks at the friction areas and challenge points in experimentation and at what you can do to get your team to start running experiments or to run more of them.
<h2>What is an experiment?</h2>
An experiment, says Cameron, is a way to compare an old version against a new version or many versions against themselves so that you quantify the impact of decisions you make and increase certainty in your decision making. He says: “One of the superpowers of experimentation is that you isolate exclamation points and question marks such that you can understand the relative impact of a change on your system regardless of all the other changes in your system.” A product manager’s goal is to identify the right decisions to make, he adds, and experiments remove subjective noise to give you a higher level of certainty.
<h2>Why aren’t more product teams running experiments?</h2>
Cameron has found there are four friction areas that prevent teams from running experiments or running as many as they would like. They are:-
Engineering: issues around workflow, speed, risk. Teams see experiments as additional work to implement when they need to focus on getting the things they’ve already scoped out the door. It adds to complexity and delivery time and can add to security and performance risks.
Cameron recommends that by placing equal priority on tooling that fits the dev teams’ workflows you can run more experiments in the long run.
Data ingestion: issues around effort, reliability, trust. Teams may feel they have a hard time connecting the data they need and trusting that is reliable enough to use to make decisions. Setting up data intake is real work, says Cameron, and it needs to be properly planned. There may be challenges in ensuring data is captured in a scalable way and in validating that data is accurate. Cameron recommends that you partner with data science/ engineering upfront to map out data flows and it should mean that going forward you get less pushback about adding new data points or locations.
Decision making: issues around comprehension and trust. Teams aren’t confident about running their own experiments. This lack of confidence may start with concern about how to interpret statistics, says Cameron, a lack of trust that teams know how to set up experiments, or that there is enough data to get results. To overcome these challenges Cameron recommends bringing delivery teams into evaluation and planning of tooling in order to assess how the results will be used in practice. You can then better democratise data-driven decisions and free up your experts to focus on more strategic efforts.
Product culture: issues around momentum, fear, buy-in. Cameron says teams can be so focused on speed of delivery that they are afraid that experiments will slow them down. Often there’s a lack of priority, comfort or habit around using data to make decisions, teams quantify their success by looking at how often they launch things and failed experiments equate with failure. “If you’re unwilling to take the time to test and validate something that you’re launching then essentially you’re in a feature factory, “ Cameron says. It’s a challenge for leadership, he adds, and recommends that teams are given safety and support so that they can take time to validate releases.
<h2>There&#8217;s more where that came from!</h2>
Explore more <a href="https://www.mindtheproduct.com/mtpcon-sf-video-library-2022/" target="_blank" rel="noopener">#mtpcon SF+Americas 2022 video content</a> or use our <a href="https://www.mindtheproduct.com/content-a-z/" target="_blank" rel="noopener">Content A-Z</a> to find even more product management insights.

core/freeform

Hypothesis and Testing

The journey to product experimentation by Split

Progressive product development: release products that deliver by Split

Surveillance art, dying phones, and fake likes – Dries Depoorter at MTP Engage Hamburg

Understanding and executing effective research

What’s stopping teams from running experiments by Cameron Savage

How to be a better data-driven product manager

Product’s role in experimentation culture by Terry Lee

Stay Humble or Be Humbled — Case Studies from the Experimentation Trenches by Anthony Rindone