Aman Khan is the Director of Product, LLM at Arize AI, who has spent the last 10 years building products around AI. At #mtpcon London 2025, he led a practical session on how product managers can thrive with AI. Watch the video in full, or read on for our recap of his session.
Aman opens by sharing the five essential skills that differentiate a good AI product manager from a great one:
He notes that product managers must cover the fundamentals of AI intuition. You don’t need to be so deep in the weeds by reading “Attention is All You Need”, a landmark research paper in machine learning authored by eight scientists working at Google.
“It’s okay if, as a product manager, you don’t have a technical background, but it’s important to find AI content that you enjoy and pushes you in the right direction of getting that foundation of knowledge,” he says.
Reflecting on his experience building an AI feature for Arize’s customers, Aman stresses the importance of having those basic product management fundamentals to find out what the customer wants instead of just jumping into the technology.
Product people play an important role in making the best use of AI, as product teams are the ones who both understand the technology and talk to customers, he explains.
Another key skill is testing the feasibility and effectiveness of new AI tools in your product. “As you test new ideas with your customers, rapid prototyping becomes crucial.” Aman breaks down how we can get started with rapid prototyping:
He recommends building prototypes with Bolt or Lovable to try new things out. “Find one problem in your daily life, go to one of these tools and just experiment. You will find new boundaries of what is possible,” he explains. Doing this will also develop your AI intuition.
Aman also adds that we should learn from great AI experiences to understand what they mean in practice. For example, we should learn from Cursor's successes that go beyond terms like ‘RAG’ or ‘Agents’.
“Now that we’ve got the foundation and have talked to our customers, it’s important to understand that LLMs hallucinate,” Aman explains. “However, that doesn’t mean that you don’t use them; it’s the job of product managers to understand when and why they do so.”
By "hallucinate," Aman refers to when LLMs generate content that may sound plausible but is, in fact, inaccurate or made up. This is a common challenge when working with AI models, and it’s crucial for product managers to identify these instances and evaluate what impact this has on users.
Aman adds that it is becoming increasingly important to master Evals and Observability. These are tools for testing and evaluating AI models to ensure they perform well and provide real value.
“If you’re thinking of building with AI, the ability to write prompts is a skill that product managers can master to drive the end-user experience and also evaluate it with prompts,” Aman notes. Understanding how to craft effective prompts can guide the model’s responses and ensure they meet the needs of users while minimizing errors.
One of the highest-leverage things that product managers can do today is to know where the technology is headed and how to apply it intuitively to your product, Aman notes.
He encourages us to try out three new ways of working with AI:
Aman closes by saying you must get your hands dirty with AI. He adds that we’re still not fully sure of what great AI experiences look like; it’s up to product managers to define that. He urges us to take those tools and apply them to a product experience that doesn’t exist yet.
