Honing our ‘AI intuitions’ as product managers

As Aman Kahn finished his Thriving as an AI PM talk at #mtpcon and walked off the main stage at the Barbican Center in downtown London, I rushed over to my mtp colleague Steffi Crivellaro and asked, “can we do a follow-on article about this?” Thankfully, she said, ‘Yes’. I’m delighted because I think Aman’s message is timely and relevant to many of us and worth a more detailed conversation. In this quick 30-min interview, I hold that deeper conversation.

I believe Aman is giving voice to a latent unease that many product managers are feeling at this moment of accelerating change. Some of us were just starting to feel reasonably competent and then were thrown the full stack of AI/ML, GenAI and who knows what else to exploit, and now we must quickly make sense of all these capabilities, and the kicker - deliver groundbreaking new user experiences, plus some real ROI and business impact while we’re at it. 

During his keynote, a phrase Aman kept coming back to was ‘developing our AI intuitions’, and that resonated with me. It was the natural point of departure for our follow-on Zoom conversation. I hope you enjoy it as much as I did. 

—----------------

I went to Berkeley and studied mechanical engineering. After college, I ended up landing a job testing autonomous vehicles at Cruize. I was the intermediary between seeing all of the perception data and trying to understand what are the assertions or tests we should write for the car and then getting the data labelled from labellers. It was very manual and very scrappy. Later, I joined Spotify and worked on their ML platform. We were 50-plus teams of data scientists doing everything from Discover Weekly to personalizing the in-app experience. And, that same testing problem, figuring out whether the algorithm’s moves matched the user’s expectation - the evals issue - came up again. After Spotify, I joined up with Arize when they landed their Series A and I really connected with their whole mission of making AI work, and work for people.

If you're shipping some type of AI model in your app - could be any type of model, be it a classification model for detection all the way to computer vision, or an LLM - when you actually ship, are you getting back the right data to understand where it's performing well to improve on the model? That's what we do. We give you the tools to go from development through to production so that you know you're actually going beyond just ‘vibe coding’. 

I'm actually trying this new phrase out. We’re going from ‘vibe coding to thrive coding’. The bottom line is that we capture these logs, you own that data, and you can either host it yourself, or we can host it. We help you slice that data and conduct evaluations with any human feedback you have. And if you lack human feedback, we’ll help augment your eval data with AI.

It's impossible to keep up with everything. I think one of the other speakers at the conference hit on this, too, which is, in such an environment, ironically, using AI to gather info and try to stay on top of everything doesn't actually help. It’s just going to create more noise. 

So our job becomes filtering through the noise. There is such a range of content. There are scholarly papers going all the way down to how something works, and then there's what I would call the TikTok-ification of AI news, which is shallow, short-form content. Our job is to sit somewhere in the middle of that range and find the right people and curators who interest us.

A great starting point is Andre Karpathy. Somehow, he's found the ability to speak to people at all levels, from 4-year olds up to those who have PhDs in AI.  I do trust Lenny Rachitsky, too. His podcasts have been great as he’s made the transition into covering AI with meaningful depth, talking to the builders and CEOs. I think someone who also speaks from a product manager perspective is Peter Yang. He's doing a great job of presenting how to use AI in your day-to-day. And then the last, I'd say, who's kind of ‘out there’ but interesting is Greg Eisenberg. He really is getting pretty hands-on with a lot of these tools that can give product manager incredible leverage. I know these people to be of high integrity from working with them. To use a food analogy, they are all like my hearty oatmeal, not sugary cereal.

There are a bunch of tools out there that are all somewhat comparable. What matters most is just to start with one. So I'd say try tools like Replit, v0 from Vercel, Lovable or Bolt. The last two are probably the most end-to-end. But you won't get as far with them as you will with Vercel and Replit. Those are good starting points, and if you want to graduate, eventually, you might get to Cursor.

Yeah, I do. So we have a customer, a large enterprise SaaS customer that wanted to track how many of their staff were using our Arize product and on which days. Basically, they wanted a usage dashboard. And, we use Pendo internally actually, so we were trying to give them access to that, but we couldn't cordon off a part of it, etc. So while the team was debating some different approaches, considering putting an engineer on it for a week, I literally just went to Replit, gave it our Pendo API, and prompted it with “build me a usage dashboard”, and it built me a Streamlit-based dashboard for that specific company and hosted it at a URL with authentication. I was like, ‘Oh, my God’, while the team was talking about this problem, I literally just built the prototype. That was 6 months ago, and the tools have only gotten better since then.

I actually think, to some degree, there is a disadvantage of trying to solve problems with an engineering lens. We really honestly don't know how LLMs work. We know that they proxy our brain in some way, or proxy how we represent information with text. That should be a light bulb or an inspiration to people who don't have a coding background. You can manipulate and master these things with your superpower, language, or plain text. You’ll be the one on the team finding the boundaries of what the tech is good at and not so good at. You can just figure out what sort of prompts work and don't work. So I think people with a gravitation towards thinking about design from a human lens will actually have an unfair advantage and do quite well.