post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

mission statement

Deep dive: Crafting your product vision and mission

Podcast

Connecting fans to creators at Spotify (Dariusz Dziuk, Product Lead, Spotify)

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Join us at #mtpcon London in 2025 – mark your calendars

How Google makes AI work at Enterprise scale – Miku Jha (Director, AI/ML and Generative AI, Google Cloud)

5 ways to invent and simplify as an AI/ML product manager

Why teams often struggle with OKRs: A conversation with Jeff Gothelf

Building your own user experience and software stack: This week’s product news

Being a problem solver by heart and product manager by trade, Thomas has experience discovering, validating and developing digital products. Employing a hypothesis-driven approach, he takes feature or product ideas from conceptualisation to realisation by applying lean & user-centered methods. Currently he is a Senior Growth Product Manager at Personio, a holistic HR software for companies from 10 - 2000 employees. Previously he led Sign up for DAZN, the worlds leading OTT sports streaming provider. Prior he worked with German tech consultancy inovex in their internal incubator, after gaining experience as Product Manager at insurtech GetSafe and ticketing provider CTS Eventim. During the inaugural MTP Engage Hamburg he enjoyed the sense of community so much that he decided to co-found ProductTank Karlsruhe shortly afterwards – and after moving to Amsterdam has joined ProductTank Amsterdam as organiser.

Senior Growth Product Manager

Thomas Leitermann

<p>Ready to get started with your experiment? Only need to find the right tool now? But which one to choose? It feels like being in a jungle with tools upon tools &#8211; how to navigate this situation?</p>
<p>In this talk from MTP Engage in Hamburg, I share learnings and examples from some of the numerous experiments I ran over the past two years at Inovex. I look at how we can learn from our mistakes, see how easy it is to get started with today’s tools and hear why I believe experimentation is more than A/B testing.</p>
<h2>Pack the Treasure Map and Compass</h2>
<p>Running experiments for experiments’ sake is a waste of time and money. We should always know WHY we want to run an experiment &#8211; what is our goal? How do we know we have achieved our goal? To structure the numerous assumptions and ideas and to think about what success looks like and how to measure it, at Inovex came to rely on the <a href="https://blog.strategyzer.com/posts/2015/3/5/validate-your-ideas-with-the-test-card" target="_blank" rel="noopener noreferrer">Test Card</a> from Strategyzer.</p>
<ul>
<li>It serves as the treasure map for our jungle expedition. On it, we gather all the important information to navigate the tool jungle. The metric defined on the Test Card serves us our compass, informing our decisions.</li>
</ul>
<h2>Pick the Right Machete</h2>
<p>When choosing the type of experiment, we have discovered two constraints. Is our hypothesis generative or evaluative? And, how strong does our evidence need to be, compared to the time we have?</p>
<p>When our hypothesis is generative it is either about the customer segment or the solution idea. If our hypothesis is evaluative it is either about revenue and willingness to pay, or acceptance of the solution. This positioning determines the experiment types at our disposal. We can then choose from a wide variety of experiments, for example. from the more than 30 experiment types described in the <a href="http://www.realstartupbook.com/" target="_blank" rel="noopener noreferrer">Real Startup Book</a>.</p>
<p>Once we find an experiment type that matches both our hypothesis and our desired strength of evidence, we need to find time and budget to be able to start.</p>
<ul>
<li>For a jungle expedition, this means we now can pack our machete, the experiment type that helps us to cut through the tool jungle. Time and budget can be thought of as food and water for our jungle expedition.</li>
</ul>
<h2>Ready to Go? &#8211; Almost</h2>
<p>Now we have all the necessities and are ready to go, right? Before going head over heels into the unknowns of the tool jungle I’d offer some knowledge acquired from previous expeditions.</p>
<p>In my experience experiments help to enforce the &#8211; hopefully old &#8211; mantra “you’re not the user” very well. Rather than build final product features based on assumptions, experiments help us expose those. With experiments we can learn from the people living in the jungle &#8211; our users.</p>
<p>In order to expose and learn from our users we first need to connect with them. Automating the recruitment of users for research and testing helps tremendously. There are a lot of useful tools available to find, acquire, schedule, and set up users for research. With those, we can put our user research recruitment on auto-pilot.</p>
<ul>
<li>In the jungle, we can navigate using maps and compass &#8211; but a GPS device gets us to the goal much quicker.</li>
</ul>
<h2>The Simpler, the Better &#8211; and Faster</h2>
<p>When looking for tools it&#8217;s easy to get down the rabbit hole to find the best, easiest, or latest tool. At Inovex we once spent a number of days rebuilding an experiment on a new tool after the first tool failed us in a small but vital part. So it’s best always to choose the simplest tool. Instead of a high-fidelity prototyping tool &#8211; why not use Keynote or PowerPoint? Simple tools also have the added benefit that everyone should know them well. So last-minute changes can be done quickly and without someone becoming a bottleneck.</p>
<ul>
<li>If you’re in the jungle looking to chop firewood then you could use a power saw. But an axe also gets the job done and never runs out of fuel.</li>
</ul>
<h2>User-Test your User Testing Tools</h2>
<p>There are fantastic tools available to help us understand user behaviour in our experiments, tools like Hotjar, Mixpanel, or Amplitude. They also help with measuring our metric, the compass of our experiments. To be able to rely on metrics we need to make sure to test the tools end-to-end. Only then are we sure they are capable of delivering the metric we need.<br />
At Inovex we learned this the hard way. We ran an experiment for two days before discovering the metric we needed could not be produced by the tool. Only after switching tools and re-running the experiment were we able to answer our experiment hypothesis.</p>
<ul>
<li>Before going on a multi-day jungle expedition, it&#8217;s probably better to survive one night. Make sure your user testing and measuring tools survive one user.</li>
</ul>
<h2>Iterate Within the Experiment</h2>
<p>Once the experiment is running there is nothing to do but wait, right? I’d argue differently. The probability of getting everything right the first time is negligible. I believe the well-known agile motto of inspect and adapt should also be used within an experiment. In one of our experiments at Inovex, we set up ads to drive traffic to the experiment throughout the night, then inspected the results and identified issues in the morning, applied changes in the afternoon and started the cycle again in the evening.</p>
<ul>
<li>Making fire with a flint rarely works the first time &#8211; but if we inspect and adapt we can get the fire for our overnight stay in the jungle going quickly.</li>
</ul>
<h2>Find a Path Through the Tool Jungle</h2>
<p>Equipped with learnings from previous expeditions as well as a backpack with treasure map, compass, machete and supplies I am confident there is a path through the tool jungle.</p>
<p><img decoding="async" src="https://www.mindtheproduct.com/wp-content/uploads/2019/08/e0521808-321c-424a-b49e-2936c447f46e_MTP-Engage_Thomas-Leitermann_EN.png" alt="" /></p>
<p>The slidedeck from this talk can be found <a href="https://speakerdeck.com/thomasleiterman/navigating-the-tool-jungle-how-to-pick-the-right-tools-for-your-hypothesis-experiments" target="_blank" rel="noopener noreferrer">here (English)</a> and <a href="https://speakerdeck.com/thomasleiterman/experimente-and-tools-wie-man-sich-in-diesem-dschungel-zurechtfindet" target="_blank" rel="noopener noreferrer">here (German)</a>.</p>

core/freeform

Hypothesis and Testing

The journey to product experimentation by Split

Progressive product development: release products that deliver by Split

Surveillance art, dying phones, and fake likes – Dries Depoorter at MTP Engage Hamburg

Understanding and executing effective research

What’s stopping teams from running experiments by Cameron Savage

How to be a better data-driven product manager

Product’s role in experimentation culture by Terry Lee

Stay Humble or Be Humbled — Case Studies from the Experimentation Trenches by Anthony Rindone

How to Pick the Right Tools for your Experiments by Thomas Leitermann

Thomas Leitermann