post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Innovation

Sunday Rewind: Show your true innovative colours by Julia Shalet

Customer

How behavioral economics can improve product strategy

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Join us at #mtpcon London in 2025 – mark your calendars

Connecting fans to creators at Spotify (Dariusz Dziuk, Product Lead, Spotify)

Why teams often struggle with OKRs: A conversation with Jeff Gothelf

Deep dive: Crafting your product vision and mission

5 ways to invent and simplify as an AI/ML product manager

Megan is a Senior Product Manager with a background in advertising. She is currently working on audience and data products for global advertising agencies and their clients. As a Speaker Curator for ProductTank SF, Megan finds, interviews and hosts speakers at San Francisco ProductTank events. Megan believes in continued learning and emphasizes the importance of connecting with the product community.

Senior Product Manager

Megan Hart (Conlin)

<p>In this ProductTank San Francisco talk <a href="https://www.linkedin.com/in/jonnoronha/" target="_blank" rel="noopener">Jon Noronha,</a> then Senior Director of Product Management at <a href="https://www.optimizely.com/" target="_blank" rel="noopener">Optimizely</a>, what he’s learned about the benefits of product experimentation and how to avoid the common pitfalls that may lead you astray.</p>
<p>Watch the video to see the talk in full, or read on for an overview of the key points:</p>
<ul>
<li>Product experimentation can support profitability</li>
<li>Many things that can go wrong when applying an experimentation mindset to product</li>
<li>There&#8217;s much to consider to ensure that your experimentation is implemented correctly – iterate on your metrics, know your numbers, and test</li>
</ul>
<h2>Product experimentation delivers value</h2>
<p>In 2011  Jon Noronha joined the product team for <a href="https://www.bing.com/" target="_blank" rel="noopener">Microsoft’s Bing search engine</a> at a time when the product, as he puts it, was “in a big hole&#8221;. As the years went by, however, that turned around, and in late 2015, Microsoft reported that Bing had become profitable. Jon and his colleagues at Microsoft attribute that growth to a change towards experimentation. As the then Microsoft GM of Analytics and Experimentation, <a href="https://www.linkedin.com/in/ronnyk/" target="_blank" rel="noopener">Ronny Kohavi</a> commented: “The growth of experimentation is the major reason Bing is profitable and its share of US desktop searches nearly tripled.”</p>
<h2>Product experimentation Pitfalls</h2>
<p>Seeing the value of product experimentation first-hand, Jon moved from Microsoft to the experimentation platform <a href="https://www.optimizely.com/" target="_blank" rel="noopener">Optimizely</a>, where he is now Director of Product. Jon learned that though powerful, there are many things that can go wrong when applying this mindset to product. He shares three common pitfalls:</p>
<h3>1. Testing with the wrong metrics</h3>
<p>At first, Jon and team focused their product experiments at Bing on driving more searches per user. The more time a user spends on your page the better, right? Not necessarily when you’re a search engine. After some tests around this metric, they figured out this was actually counter to what users really want and what the market leader (i.e. Google) was focusing on. Instead, they decided to try <em>decreasing</em> searches per session but <em>increasing</em> sessions per user. These metrics in tandem got the user to what they were looking for faster and meant they turned to Bing to get them what they needed.</p>
<p>To avoid this pitfall, Jon recommends:</p>
<ul>
<li>Put yourself in the users’ shoes. What if they knew what your goal was? Would they be aligned with that experience?</li>
<li>Think about your goal as part of the big picture. If this metric went up and everything else remained flat, would this be good for your business?</li>
<li>Constantly reevaluate.</li>
</ul>
<h3>2. Being mislead by statistics</h3>
<p>Peeking at results too soon or getting fooled by false positives can mislead your direction. If you’re testing 20 variations with a 5% chance of a false positive, you need to consider that at least one result may be incorrect.</p>
<p>To avoid this pitfall, Jon recommends two things:</p>
<ol>
<li>Involve a professional who knows the numbers — have a data scientist or statistician join the team</li>
<li>Use a testing platform that’s built to consider nuances in the data</li>
</ol>
<h3>3. Testing too few variations</h3>
<p>We tend to focus on <a href="https://www.mindtheproduct.com/tag/ab-testing/" target="_blank" rel="noopener">A/B testing</a>, but at Optimizely, Jon and team have found that testing many variations is more likely to lead you to a significant winner. On average, two variations lead to a significant winner 14% of the time while five variations lead to a winner 27% of the time.</p>
<p>To avoid this pitfall, Jon recommends that, if you have the resources, test more than just A and B. This helps the team think out of the box around other possible solutions.</p>
<p>As Jon shared, experimentation is a valuable tool in the product managers’ tool kit, but there’s a lot to consider to make sure it’s implemented correctly. Iterate on your metrics, know your numbers and test! Check out Jon’s talk to hear more stories about product experimentation. You can also read Jon&#8217;s article, <a href="https://www.mindtheproduct.com/how-do-you-implement-effective-product-experimentation/" target="_blank" rel="noopener">How do you Implement Effective Product Experimentation?</a></p>
<h2>Enjoy more from ProductTank</h2>
<p>ProductTanks are informal meetups, created by Mind the Product, to bring local product people together and to enable speakers to share amazing product insights. Today we have ProductTanks in more than 200 cities across the globe and there’s probably one near you.</p>
<p><a href="https://www.mindtheproduct.com/producttank-about/content/" target="_blank" rel="noopener">Explore more ProductTank talks</a></p>

core/freeform

How to lead data-centric product teams: Jon Mora (Chief AI Officer, Zefr)

How to be a better data-driven product manager

User insights for product decisions by Cris Valerio

Data scaling for startups by Crystal Widjaja

Make Better Product Decisions by Using Quantitative and Qualitative Data, by Brian Tran

Product experimentation pitfalls and how to avoid them by Jon Noronha

Megan Hart (Conlin)