How to Break into AI PM from Google AI Product Leader | Marily Nika (Google)

the first thing is to get out of your head and not worry that oh God I don't have a CS degree I don't have a PhD anyone can do it now you need to put your learning hat on and you need to be comfortable with just going back and getting uncomfortable and learning about what does each one of these models look like how do you train it what are the trade-offs so you need to have ai awareness and having that AI awareness should not overwhelm you like you don't need to get deep in the math but you need to understand how things work and how that affects your PM craft you also need to just demonstrate to the software engineers and scientists that you're willing to sit there next to them and really understand how things work really understand the architecture really understand the technical decisions they need to make so you lean in them and they lean you you're partnering and they just need to respect you and realize that oh okay marily is a trusted partner she knows your PM craft and it's going to help answer these questions because ultimately it's all about the users right experience all right welcome everyone so my guest today is mar Nika AI product Leia Google Mar got her PhD in machine learning and then she spent 12 years building AI products at Google and meta she's also taught over 5,000 students through her top rated AI courses so welcome thank you so much for having me Peter yeah so you know I feel like we got to know each other and why don't we uh start by

talking about just like a quick explainer of some core AI Concepts since you do have the PHD and all this EXP experence sounds good yeah why don't we start with like what is your definition of AI and how is it different from machine learning perfect um well I love you're getting straight to the important stuff so actually I put together this like use cases and applications map um is it okay if I share my screen as I talk about this awesome perfect so I put

together this AI algorithms and applications map just because I'm getting this question so much and I think this is a really nice way to visualize kind of everything that's happening in AI so as you see there are different types of learning right there supervised learning self-supervised learning unsupervised and then reinforcement learning I want to start with classification because AI is all about adding intelligence would say to a machine so that it can act and learn in the same manner as a human so I have a four-year-old and then we were walking around the other day and I was like oh here's a squirrel here's a dog here's a cat and then I did that enough times that eventually there was a little animal there and I'm like what do you think this is and my daughter was like oh yeah it's a doggy and I said how do you know it's a dog in and then she came up with some reason but this is what AI is all about right it's learning from data that's labeled specifically in the classification example and then when we have a new data point that has no label it predicts and it provides the low probability as to whether it's like a cat do square but it's interesting because in this map you can see the kind of applications you can have for example spam and not spam which is like the coolest feature that exists on email is exactly built with that manner right so you do classification you train it smart matching another fantastic capability that's provided by AI so you know dating apps is a good example on the use case image classification if you go on Google photos and then type in you know old wedding dresses or Sky that's blue it classifies based on that so that's supervised learning you also have regression so regression is actually good for forecasting so you will see that for you know player capabilities and how they perform during a game and predicting and guessing that or forecasting sales that's also machine learning um we also have let's go on the far left unsupervised learning right here so this is the kind of learning where AI learns Without You providing label data it's more like hey here's a bunch of info you figure it out and this is where AI does pattern matching and I think a good example I have here is you know clustering being able to segment images or customer segmentation right this is heavily used by targeted advertisements it's amazing for anomaly detection and fraud and so on we also have two more things and I'm almost done I promise so reinforcement learning that you see at the bottom this is where the cool stuff are well cool like everything is is pretty cool but um here's where you will see things like learning through trial and error so imagine having a dog and teaching the dog how to sit you give a reward and then it eventually does the right thing and then over time learns their behavior so this is used you know in the real world when an agent interacts with an environment and learns to achieve a goal like deep mind and of course learning you know chess or a robot learning how to navigate in maze essentially I have some cool applications here like recommender systems yeah you definitely use it there and then Robotics and Automation and it's just you know I love this map because it's a way to see apps and features you use dayto day and how what powers them and I just loved putting this together and it took forever and then last at the very top we're talking about llms and of course llms as folks know large language models these enable things like you know chat Bots and AI agents and content generation and some ization and search and all these things I know I spoke for too long but hopefully that was a helpful answer so basically a supervised learning is you give it this labeling data where like you have like given this image given set of images this is like I'm gonna tell you if this is a cat or dog and then based on this data that I have you tell me based on other set of images if this is a cat dog right yeah but it's important to know that there's always the probability so you will say here's I'm certain by 7 8% as whether this is a cat or a dog and that probability is so important for product managers because you build probabilistic products as a product manager no longer deterministic and that is a whole other mindset you need to have when you want to bring such products life makes sense so do you think LMS are supervised unsupervised or somewhere in between like yeah I'm going to skip this question Peter you're remove it because there's a lot to say there and I don't want some scientist to come to me and say well actually blah blah so let's skip this fine okay all right so that that's a really good overview so

now let's talk about AI product management right so there are so many different types of aipm roles maybe you can just give us a general framework for like what tra the best apms have in common oh that's a very good question yeah well first of all let's define aipm right so whenever you work that directly or indirectly with this type of models and you have a scientist team and part of your requirements need to be kind of refined and discussed with the science team because there is some model training required then you know you're an AI product manager and I was an AI product manager before it was school so I started in 20 2013 and it was kind of like I was working with you know scientists and stuff and I didn't realize that not all PMS do this type of work and have this same type of challenges so that's when I started to realize okay what are the unique traits and characteristics that you need to have in order to be able to do this well number one you need to be comfortable with experimentation and when I say experimentation I mean you will heavily pivot from your original hypothesis and requirements to the final product like it's going to be completely different and you need to be okay with that I know PMS that really need this kind of you know step by-step approach and road map and build against that but in AI it's almost never the chance that what you have in mind is what you end up building number two being comfortable with that probabilistic nature because as an aipm you have kind of this responsibility to advocate for AI and educate if you will other cross functional partners that are not familiar with it on how it works I've been in prodct reviews where you know leadership was trying this product we wanted to launch and they're like hey I tried this two three times and every time I tried it had a different but I don't think it's working as intended and I was like it does work as intended and here's why so understanding kind of this more technical nature and being able to advocate for it is important and then last but not least every single day is about trade-offs so when you have this strategic thinking and you're not afraid to take the risk and say all right given all these trade-offs here's what I think we're going to do and let's discuss it and having this strategic diplomatic kind of approach and discussion I think is very important and I feel strategic thinking is a skill that I traditionally used to see in mid to senior levels whereas for AI you got to have it walking in so strategic thinging is so important I like your point about uh s right expectations of leadership because uh like you said this thing is very non-deterministic and if you try hard enough you can always get it to spit out potentially the wrong answer yeah so you have to like what some ways to set the expectations up front that you seeing they're effective good question and I think this is an indirect question about product reviews all right because you know in the past there's kind of a forum for product reviews and you kind of sign up and you go there and you say what you want to launch or what the question is that you hav't you get a discussion you can answer you launch or you la you go ahead but with AI products

and that's my personal opinion if you go in to a product review and it's the first time this forum or steering committee or leadership if it's the first time they see what you want to do then you've lost like this is not what you should do so when you work with AI products you need to get constant Buy in you need to always align your thinking you need to make sure everyone around you leadership or not are comfortable with your vision and strategy because there's just so much uncertainty and this can cause a KN and of course we need all these you know exra resources for to build AI products so you really need everyone to be aligned as fast as possible at all times so I think that's a good way communicate often make sure people have seen what um you're doing and don't work alone like these are cool products you should involve as many people as possible it's a collective effort so I don't like seeing you know in the past I may have seen you know PMS working in silos like AI is cool let's all get there and work together towards this awesome cool new thing yeah like you have to partner with everyone else and maybe even customers and just iterate right just have to like um yeah you know get something out there prototype it and see if it actually works see if the aties even support the use case that you're thinking about yeah and I like that you're mentioning the use case right because so often in AI you kind of lose sight of what you're doing and why you're doing it so it's good to always say hey there's a problem and a reason behind this crazy you know AI system I'm building so we always need to be able to say okay there a use case we're solving for and we're this is not just a shiny object because it has AI so I think it's important to call this out as well and can you give us a look

into what a day in the life is like as a aipm I think the term sounds very glamorous but like a lot of is maybe clean up data or like what what is a day in life like yeah very nice question well the thing is I mean every pm has cross functional Partners but I find that you know with AI there's just so much more an throughout your cross functional partners and there are even more cross functional Partners so you just meet a lot and you need to run a lot of experiments and be very mindful about you know what success means so you just come up with kind of a role M and an idea and you want to get aligned with all these C functional partners and sometimes when you have kind of different job Lads you work with you realize that you know they have a different understanding of what success means in AI so there's higher sense of responsibility let's say of um and challenge to get alignment so ton of align thingss ton of prop reviews ton of thinking outside of the Parx and then tons of pivoting and by the way I don't like the word pivoting I want to use more evolving I have a friend called Hans and he's like oh we evolve we don't people like yes that's exactly it so you expect no day to be the same and alignment and discussions and strategy and Evol okay is is a like when you say alignment you talk about is one of the challenges is aligning on shared goals or like what are some the challenges yeah so of course you want to align on the high level goals and why are you doing this in the first place but when you're deep in the weeds and you are in active model training process there's a ton of conversations around you know what kind of data are you going to use what is that going to look like how do you communicate that to the users ton of privacy and Link go discussions right this is very sensitive we definitely want to get it right so there's just so many different parties you need to work with in order to make sure folks are comfortable with the work in what you're building that I feel it's just more heightened than classic pming if that makes sense got it you can have to be a little bit parner right because there's like so many different uh outputs and so many risks that you have to try to account for yeah and at the end of the day like what is success and there is a thing called mvq which is the minimum viable quality so you know let's say you have a scientist team and they build together they build this model then you have your software engineering team to take that and package it into an experience so you have your ex helping in there and then you test this out and you put it in production so the quality that you had that you thought that pure model had is just so different when it's in the in production and in the experience and it's kind of like yeah all right what's the minimum VI quality this should have in order for it to launch out in the world and there's no right or wrong that's all on you and of course getting alignment and it's right there's just so much responsibility in the aipm hands that again there's no boring day and I just love being in this situation you love the ambiguity yes okay so like

let's say you know I'm a PM I'm a growth PM or maybe I'm not even a PM I'm like a product marketer and the hot thing I really want to be become aipm how do I go about breaking into this field just start playing around with AI tools I mentioned there is AI product management which is you know working with scientists and bringing AI powered experiences to life but there's also AI for product management which is you know you being the consumer of these AI products that were created by some PM right so just start using them use perplexity use u. com use com with AI um create your own AI agents custom gbts and just get the confidence needed and just start becoming the expert in your own company as to you know how you can best serve your you know your end users I keep saying this like if you want to become a product manager before interviewing you need to reach the point where every single thing you do you always need to think about hey how would I improve this so I'm telling people hey if you have an upcoming interview and you go get a drink at Starbucks and you use app I want you to think subconsciously how would I improve this app always how do I improve and AI products are the same so start using these AI tools figure out ways to improve them figure out how to answer the question if you were to interview for that app for that company hey what would you do next strategically what would you improve and why right very important and you know I get a lot of people in my certification saying okay well I did the work I know that I want to be an aipm I have this certification to show for what's next I don't have anything to show for in my resume to get the first interview and my advice for people is attend hackathons participate hackathons there's a wonderful website it's called devpost and it shows you a list of H phones all over the world and the best part is that there is a tab called you know like teams and team members and there's a little filter called looking for a team so you'll go in there you'll submit your profile you'll say that you're the PM for an AI hackathon and you'll meet people that are developers and you ex folks and you'll all come up with an idea maybe create a prototype maybe win right there is a lot of perks and money to be won but it's not about winning it's about having to show something and your ASA you can say hey participated in five 10 projects and here's the GitHub or here's the website where you can see the Prototype and I concepted it and it launched it's in the app store if you do end up launching or it won so try to get kind of this experience in any way you can and other strategy I love is within your own company Shadow aipm if there are no aipm pitch you to become the aipm and don't expect for someone to come and say yep you're now named the aipm for the company to start doing the job work with some engineer um LED some smartphone functionality put a few things together create a little demo and a little prototype pitch it to leadership and just become the aipm don't ask for permission we cannot ask for permission in order to achieve anything right so become the aipm that you wish to be let's see and you kind of want to become the goto person in your company for AI and AI features and experiences and this is kind of something that's claimed it's not something that you know you're assigned to so you want to be that person and last but not least something I tell everyone is you'd be shocked by how many people that I teach in my boot um have already worked with AI and they're already AI PMS but they just don't know it so look into all your past products is there anything that has some smart functionality because if so you can tap on that you can leverage that if you interview you can reape kind of your role a little bit in your regiment just make to say hey launched end to endend feature or concepted feature or created a 5-year Road M for something that has AI in its core I think the fact of the matter is like no matter what role you're in like AI can probably help you do your job better so it's like learning the tools yeah help you build credibility yeah like this morning I spent like so you know with these interviews I've been trying to get the transcripts to clean up right so this morning I spent an hour like trying to work with this AI prompt to clean the transcripts automatically before I basically had to hire somebody to do it for me I finally got the prompt in a place where like it actually works like I don't actually have to hire a human to review it like it seems to work and that was like a really good feeling after spending it an hour doing CL cloud is amazing now you can create apps on cloud and it's been fascinating you just type in you're like do this and then if you don't like something you're like okay let's do this other thing but yeah there are two things the way I see it right there is AI product management which is what we talked about and then there's AI for product management which is the tools that you're going to use on your day-to-day in order to become better more productive and just evolve as a PM yeah I mean obviously if you're a product manager hopefully you're using the tools yourself including your own prod product yeah okay so let's talk about uh

building a products a little bit more so obvious is's like a lot of hype for generative AI products there's a lot of startups out there but I think the unset reality is like a lot of these uh products have like a huge retention problem like I just based on my personal experience you know I I've tried hundreds of AI products and I pretty much only use cloud and maybe per perplexity on a regular basis so I'm curious if you have any like high level principles or any tips for building a successful AI product yeah I think this goes back to the comment about hey are you solving a real problem like is there something that you will improve from users day-to-day life that they will have so much value from that they will just need to keep coming back I mean there are apps out there that got created just to write the H and they charg like $2 per person using it and they made a lot of money and that's what they were hoping for so sometimes you know the goal is not to have retention it's more like okay let's get everything we can but you know in the beginning and I keep saying AI is not a product right AI is a tool so in the beginning it was more like okay here's what AI can do for us here's a playground here let's try it out and people try it out got excited so it's natural to lose the hype there but I feel that by the fact that a lot of people started using it in its row form came up with ideas on how to take it and Infuse it in experiences examples I was using Google Sheets the other day and there was kind of this um PL again that lets you know opens up a prompt and you're like oh I want to create a deck about my new bakery I want to have like an introduction and this thing and it just created a deck for me with generated images with the right wording with the right bio props and everything so I think AI is about infusing it in current day-to-day workflows versus having a just a screen where you can type in them prompt so and that's the beauty of being an AI product manager right to figure out how to enhance the day-to-day life of people by leveraging yeah I think that's a really important point because a lot of people when they think about General AI to think about you know like chat GPT right like you talk to some sort of agent or something but like you said I think a lot of the most successful AI features are actually just like removing some friction in some sort of core workflow like using a spreadsheet or like trying to draft the e or something you know it's like these like tiny features that are not necessarily like super sexy but like people just use them all the time yeah I will say though there's something else that's super awesome right if you have like the promt and you can talk to it okay cool but if you change the form

factor is super cool so let me show you what I tried the other day and I'm just fascinated by it okay hey tgbt I'm here with uh my friend Peter Young and we're having a podcast about Ai and its capabilities and how retention drops after people try out AI that doesn't really have an enhancement people's day-to-day life what do you think it's a great topic for discussion retention often drops when AI doesn't add real value to users daily lives if the AI feels more like a gimmick than a useful tool people might try it once out of curiosity but won't stick with it to keep users engaged AI needs to solve a real problem or enhance their experience in a meaningful way for example an AI that helps people dis new music they love and connects them with but right so what did we do we used you know this like a what used to be a UI right but we used it in different form factor and because we were able to speak to it and spoke back in this human like voice now that's something people will use on day to day because it solves a different problem which is the convenience of talking to it and interactivity and conversation ality and yeah I just love using this thing it's just great yeah I haven't tried that that's the latest voice release from chbt right I haven't tried yeah I haven't tried it has this conversation is so you can interupt it and it can kind of adjust and say oh yeah so feels like a human like my husband I was asking hey who are you talking to about that stuff like oh it's ch beting that's funny because I joke with some people that like I talk to some of these tools more than I talk to my wife during the day like I dm' them more than you know DM my wife so um yeah I mean like I've I basically come to a point where maybe this is not right but like I basically just whenever I have a problem I basically just ask AI like what's your opinion and most of the time maybe it doesn't have the right answer but it gives me something useful to build off of you know I multimodal is really amazing like I had made a post about this there was like a little bird that was injured in my driveway and I was looking at it I'm like oh God who do I call what do I do should I pick it up and I took a photo and immediately CH B was like oh this is a bitchin it has fractured the wing do not move it here's the number to call based on where you're at and give it some water and crackers and it was just I outstanding it saved so much time and it probably saved the pigeon's life because the first thing was going to be I was going to pick it up and take it in but they're like no no your professional needs to do that outstanding yeah it's so helpful in everyday situations I I do you kind of worry that maybe it'll make us like more lazy and dumb basically yeah but maybe not yeah I can see lazy for sure I can see it yeah I kind of want to just Hammer this point home right like both of us working at big companies right or like have worked at many different big companies and uh a lot of times like you know you build like a one two three year strategy document and then you get a bunch of buying and then you yeah and then you build and then you maybe like at some point you ship it to customers they start getting feedback but I think I feel like that process is not going to work with AI products right or I don't know how you feel but like just kind of reemphasize some of the points that you said earlier yeah things really move fast and things really change you need to have some strategy in place but you also need to reconsider and reassess at all times and you need to make sure the team has this experimentation mindset and realize that oh okay you know this is what we're headed towards but you know we need to be prepared for some people but you need strategy in place you can just do things without strategy so that's the first thing that I got ask the other day and you need to always build for where technology is headed not where it's at right now so you make all these assumptions about what we'll be able to do in the future and you know because technology may not be there and resources and all these things you may need to reassess but try to think as far ahead as you can in order to minimize the risk of like evolving and pivoting and you the other thing the other challenge I had found is the team when they're working in this experimental experimentation environment they're like wait so how is you know progress assessed and how is my career getting assessed with products if we don't launch you know the traditional product management is like lunch launch but if you're an AI it's more like what question did I answer what input did I get what Insight how can I pivot so it's more a milestone basis rather than feature launch in0 to1 products so gotta be mindful about that you know assessment changes in AI product management as well okay so like uh you want to like the foundational models are getting better all the time right so you want to build your product in a way that leverages the fact that this stuff's getting better like every month or even every week that's one point right yeah and I guess the second point is some of the stuff like might not make it past like the Prototype or not notebook stage and like that is also a success if you figured out early that this use case isn't working right now yeah when I was doing my PhD my first experiment like just didn't work and it didn't have any like I didn't prove my hpoes it was just oh my God I lost like three four months of my work and my amazing Supervisor was like this is absolutely progress you can publish it in the conference and say hey here's what I tried here's what didn't work and here's what I would do next and I'm telling you this to support you and that paper got accepted and it ended up being like a big chapter of my PhD so it's an important point that in AI learning is success and also for product managers like it's very important yeah so sometimes in product management you try to structure experiment so that like you know like it's guaranteed to work and I'm going to get my next promotion but I think in this case you really have to keep an open mind yeah it depends yeah you know now that you work at Google I'm sure you can use like Gemini in house all you want like how do you use AI to do PM work like which parts of PM work can you automate through AI yes so when I'm talking about a zero to one product you know there's a lot of hypothesis making and you know I may test my hypothesis and say hey I think you know when it comes to maximizing the this is just an example like streaming time or you know someone watch time on you know music app or something like this I believe that this and this are factors that affect this decision and it kind of double checks my intuition and it gives me extra data or it can quantify things like hey here's the biger assignment than this but also for the implementation side of things as well I may have a product review and I may say hey here's my goal here's the pro and cons can you put together one sheater for me to just so that I can go in for the viiew and it saves so many hours because you take that you paste it and then you can kind of you know tweak it and adjust it the way you like it and it's kind of wonderful how much time it saves from the you know the main tasks now of course you shouldn't share anything confidential or anything like that to these apps but keeping at high level has changed my life to be honest yeah I think it's much easier to work off a first draft than like a blank page right so yeah get too I've also found it very helpful for like summarizing stuff that's like one of the best use cases that it's good at like um you know like a 50 message slack thread like I don't really want to read like just yeah like summarize for me yeah I was interviewing somewhere at some point and I think I had taken the transcript of the entire investor relations release that was like 4 hours long on YouTube and they uploaded it and I'm like hey I'm interviewing for this role what of all this matters summarize it for me and it was like oh okay cool and then I went to the interview and I was like hey oh by the way I read this and there was that like y perfect so it's kind of you know Changing Life in so many ways in you know productivity let say yeah it saves a lot of time on that let's just kind of do you also like use it to write prds and stuff or like do you use yeah I have a custom gbt that I can show with you it's called AI product gbt and it actually creates PDS for me and it just really as I said like saves time so it's this one you can say I am a PM for Netflix and want to improve retention for binge Watchers yeah and then I trained it and it kind of comes up with ideas that are AI enhanced and it tells you hey well these are three ideas which one do you want to go with and you choose there like AI powered content recommendation with a solution the problem solved dyam make episode interuption that's cool smart notifications and then it's like which one do you want you'll say let's go with free and you know it will take as an input a template I have provided for an AI PRD and then it will just start you doing the framing and the market and the pain points and you want the rice framework but the best part of what it does is it generates a mock which actually you can give to the designers and be done so it's just super fantastic like this type of work oh well you're actually able to get it to generate a mock yes check this out wow boom it's just been fantastic and I can see the data of how many people have using it's like I don't know like 10,000 or something like that there you go wow well that is fantastic yeah I never thought about the mark part helps I mean the designer the logo and yeah also it has become so much better from the hallucination side because now text renders okay like still has some issues but this is so much better than it used to be sze yeah that's really smart that generates a mock so yeah long story short yes I use it for PRD generation I mean PRD is like also keeping the PRD updated is like a Hu huge pain so like you know if you can just use AI like people have discussion about the PRD and they just use AI like hey can you update PRD based on this discussion I guess I'm admitting how lazy I am but uh you know you still have to you have to audit what it does but it saves a lot of time well not just that I wish you could like talk see the comments that people leave and address the comments too I think that's where the human intervation comes in but yeah maybe in the future and uh what is

your um favorite use case for AI in your life outside work I love this voice command you know what I like using uh that's recent I like generating music and there's this app that does it and you know I absolutely love Coldplay is kind of my thing and I was like Hey play generate cplay and it actually did generate Coldplay liking songs now you know it has some licensed data and then I realized that app had some unlicensed data so I stopped using it but I think a music is something that we are going to start seeing in the future because there's just so much potential in it is it suo or what does you use for mus yeah that's the one yep right yeah yeah it's amazing but uh you know like between me and you and I guess whoever listening to this like I'm pretty sure all the stuff is trained on licensed data I don't see how it cannot be so I hope so yeah I don't know I need to do more research yeah cool all right so let's uh wrap up want don't you talk a little bit about your AI courses and what you cover in your AI courses and where people can find out more yeah sounds good so yeah I really wanted people to break into AI product management and I created an AI product building AI products 101 course on me and that was like the top rated best course only s only course on Maven that taught people the AI product life cycle and what your role is as an aipm and all these algorithms were kind of discussed from the PM perspective as well as a ton of exercises and a live coding workshop with no code tools um and people really like that and then after they were graduating they were like hey we need more so I created an advanced AI product management course where I bring people from the industry like M from Microsoft open AI and all these places and they present their exclusive kind of case studies of what it is they worked on and it's more for like directors VPS SE level folks and you know they learn how to create aipm Works they learn how it's applied in other companies and they do all these exercises as well um and then you know that was another very popular course so I decided to create a 12 week course that was more like a bot cam AI product bot camp that includes the 101 the advanced in the cstone project where people can apply their skills to their real world problems and also pitch to a panel of VC's um bringing in and people have raised funding people have met fantastic colleagues we have an A Fine community of 5,000 people that people refer each other people hire each other people form startups with each other we do meetups and yeah it's been like the best rated course people have been loving it and I found my identity as an AI educator and it's been something I absolutely love doing so and I'm also launching a new course soon so if anyone is interested in learning more I'm on LinkedIn I'm on sapag and yeah I'm just very excited to show more about this cool awesome all right well mer thanks for a conversation thanks for sharing some of your insights and um hopefully we get more uh aipm that's my mission yeah absolutely thank you for having me Peter that has been super great all cool all right thank you