post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Artificial intelligence (AI)

How to implement effective AI evaluations

Guest post

Strength through tension – in defence of the product triad

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Why self-care is your secret weapon as a product manager

What if pricing was your best feature?

Navigating the workplace as a young product manager

Klarna goes on recruitment drive after ditching AI-first policy

How Stack Overflow is competing with AI – Jody Bailey (CPTO, Stack Overflow)

George Fletcher is a Senior Product Manager that has been working in product for over a decade.  A passionate product manager promoting lean product management, rapid prototyping and the adoption of design sprints throughout the businesses he's worked for.  Having built successful products in a variety of different industries George has found a home in the acceleration and incubation hubs in larger corporate environments helping transform mindsets and processes towards lean, digital thinking.

George Fletcher

<p>In this ProductTank London talk, Ben Fields, formerly Lead Data Scientist at FutureLearn, looks at getting the most from the relationship between product managers and data people, working out people&#8217;s background understanding, the benefits of leading with a story, and how best to spread knowledge.</p>
<p>There are three key aspects of how data science can effectively sit within a product and software organisation:</p>
<ol>
<li>Gear and orient to your audience</li>
<li>Optimise for diversity</li>
<li>Embed with product</li>
</ol>
<h2>1. Gear and Orient to Your Audience</h2>
<p>How can you show your audience content that they would not otherwise have found? Whether a site is selling ecommerce products or promoting editorial content, delivering an accurate relationship between what the user is presently viewing and then where you direct them next is more likely to lead to conversion.</p>
<h4>A Story About Popularity Bias</h4>
<p>Ben says that a recommender system and widget built at FutureLearn attempted to orientate the user to the course they were viewing and then gear and direct them to other possible, related options &#8211; options that they may not have encountered without this widget.</p>
<p>A recommender system should make effective use of data that an application has available. However, this can push a model to decisions that might not be the best for the viewing user. Popularity bias can creep into data models.</p>
<p>The FutureLearn recommender system would promote courses based upon similar enrolment patterns and behaviour from the wider user base.  o, at its simplest if Person A enrolled in Course 1 and Course 2, then when Person B views Course 1 they may also be recommended Course 2.</p>
<p>However, FutureLearn found that its recommender system kept promoting a similar set of courses that didn&#8217;t necessarily have a fair relation to the course being viewed. This was because the system took popularity into account and one provider, the British Council, provided a number of courses that were popular among FutureLearn users. As a result, a lot of British Council courses appeared in the widget. This meant that the widget was not meeting the aim of showing courses that users would not have found otherwise.</p>
<blockquote><p>If we&#8217;re just repeating what we already saw then what work are we doing?</p></blockquote>
<p>Ben has another story about popularity bias. A popular book retailer had with a “people who bought this also bought…” widget. When browsing the product page for a Java RMI (remote method invocation) book the widget recommended two other software engineering related books and then finally &#8230; Harry Potter.</p>
<p>It this wasn&#8217;t incorrect, it just isn’t what was wanted and likely had little value. The system picked out Harry Potter because “everyone” has bought Harry Potter.</p>
<p>It is very difficult to move from popular courses, books, content, artists, products, deep into the long-tail, less popular and less visible items. However, that longer tail can hold a lot of dormant value which can be unlocked if it is seen by users at relevant opportunities.</p>
<h2>2. Optimise for Diversity</h2>
<p>One way to combat popularity bias is to optimise for diversity. When designing a complicated system such as a behavioural recommender, it is key to spend time working with the people who will find the edges of the requirements.</p>
<h4>Telling and Supporting Your Stories With Data</h4>
<p>FutureLearn wanted to build a prediction engine that would predict course enrolment and revenue. The system would take the known information about the course due to run and model it against behaviour seen on the site to get enrolment and revenue figures. Start date, course length, how many times has it been run, how much learning there is, and course density, are some of those elements that can be coupled with other factors such as seasonality, peaks, and troughs.</p>
<p>Using a machine learning tool called TPOT, the team at FutureLearn was able to cut short system setup and prove it worked.  It took a week to show that the system was able to predict revenue when given a set of data attributes.</p>
<p>This lean approach gave FutureLearn an immediate output and evidence base, supported by data. While not pristine, the system gave better predictions about which courses might be better than others than the previous rule-based, business process methods.</p>
<h2>3. Embed With Product</h2>
<p>Finally, the FutureLearn team found success by keeping their data scientists close to product. The product team outnumbered the data scientists in the business but this was offset by using an embedded rotation system. Data scientists moved around and embedded into the various product teams based upon priority and focus.</p>
<h4>Priority</h4>
<p>Here the data team would consider need and ask when/where are the data scientists going to be most effective?  Which product team needs them, when and for what scale of project?</p>
<h4>Focus</h4>
<p>The battle here is to ensure that the decision aligns with the strategic aims of the company. By serving and orienting to the business focus the team could better put the resources where they could be most effective.</p>
<p>Ben’s final parting thought about scarce resources was that effective internal communication and an understanding of the value that data science will allow companies to be far more effective.</p>


core/freeform

AI ethics advice from former White House technologist – Kasia Chmielinski (Co-founder, The Data Nutrition Project)

How Google makes AI work at Enterprise scale – Miku Jha (Director, AI/ML and Generative AI, Google Cloud)

LLM workflows for product managers: 3 key takeaways (Niloufar Salehi, Assistant Professor at UC Berkeley) – ProductTank SF

The future of product management: Insights from ProductTank San Francisco

Product lessons learned making early moves with AI in media: Lindsey Jayne (CPO, Financial Times)

A year with ChatGPT and product innovation: Navigating the AI landscape

How to keep your head about generative AI (when everyone is losing theirs) by Claire Woodcock

They’re Just Fancy Averages by Ben Fields

George Fletcher