Why Prototypes Beat PRDs: How to Ship 10x Faster with AI | Jeremy Epling (CPO Vanta)

we used to write all these like 20 page prds that nobody wants to read. Like why do you think prototypes are better and like how do we normalize this stuff? — One of the first things we actually did was build a template. Well, I sent this out to the PM team with like a little loom explaining on them how to like fork from this and this allowed teams to kind of go out and like really start to experiment and build out more pages. — Jeremy, chief product officer at Vanta and former VP of product at GitHub and Microsoft. I don't think the PRD completely dies, but like seeing is believing. I want to like close my eyes and like open them and like try to like become the customer, you know, and I'm like, "Okay, if I was seeing this for the first time, is it working? " Do you have any advice for like the big tech PM like worry about their job or like, you know, paying their mortgage? — Yeah, I think it's um All right, welcome everyone. And my guest today is Jeremy, chief product officer at Vanta and former VP of product at GitHub and Microsoft. Really excited to talk to Jeremy today about normalizing prototypes over PRDS. You know, his favorite AI workflows to uh for PM work and the top skills that PMs need to learn uh in this new era. Uh we'll also cover how the team ran AI evaluations for Vanta's new AI agent. So, welcome Jeremy. — Yeah, thanks for having me. Excited for the conversation. All right. So, why don't we start with like a hot topic, right? So, you know, we both work at Microsoft. I think you for much longer than me. And, you know, we used to write all these like 20page PRDs that nobody wants to read. Like, why do you think prototypes are better? And like how do we normalize this stuff? Like it's hard to get people to actually do prototypes, you know? Like, how do you normalize this? — Yeah, definitely. I think when I first

started at Microsoft, I worked on Windows. I think the template was like 35 pages, so it was pretty crazy. Um, yeah. I mean, I think we definitely need to evolve. It's like I don't think the PRD completely dies. Like I don't know. There's two kind of catchphrases I use a lot internally is like writing is thinking. And I think it like really forces people to kind of think through what is the customer problem, what is the business impact, but like seeing is believing. And so to me, I think the faster we can get into like a prototype and what is the customer experience? How does it feel? Does it feel right? How will we talk to the customer about like the value? to think about like the marketing message, the key demo, those are the big things. And you know, as we looked at that transformation at Vanta, I think that you know, one of the big things we ended up doing was uh picking up Vzero for like everyone on the PM and design team. And so we leverage Vzero prototypes like very heavily um throughout the process and it's an easy way for and you know could be other tools you could be using you know Figma make lovable there's like a ton of like great tools out there but I think being able to get an idea of how the customer feels in the experience like I kind of think at this moment to me as like a product person of like I want to like close my eyes and like open them and like try to like become the customer you know and I'm like okay if I was seeing this for the first time, is it working? And I think it's so easy with really long PRDs and all the details. And I'm not saying they're not important, but you kind of lose the forest for the trees and don't understand, are we really building the right thing? Is this something a customer will get excited about? Is this something they'll pay money for? Is this actually solving the problem? And instead, you end up in these reviews where you're just talking about all these details that really don't matter at the beginning. What matters is like, should we even do this thing? And like, will customers get excited about it? And so I think prototypes really bring that out to me as a more like visual designoriented person and makes it much more real. — Yeah. Like my ideal review is like a press release like a blog post using customer centric language and maybe like a prototype from that. That's it. It's like — Yeah. Definitely. And we do a lot of the kind of Amazon style PR FAQs as well, you know, uh where it's kind of like write the blog post. What is that demo script going to look like that's going to have that wow moment for someone? and now let's look at it um you know and spend kind of the review time talking about that experience and why it's great or how it could be better. — So um can you screen share and like share some specific examples of prototypes that your team has built and you know what their impact was? — Yeah, definitely. I'd be happy to. So we have a couple different things that I can show here. Uh, one of the first things we actually did is we saw a bunch of these AI tools come up for prototyping and kind of like building full-on applications too was build a template. So, one of the things I did is like not all the PMs on Vanta are like super technical and have like written code and have more of a developer background. So, we leveraged just Vzero's import feature and to be able to kind of import some of our basic assets out of uh, Figma and create a simple prototype here, which was really helpful. I sent this out to the PM team with like a little loom explaining on them how to like fork from this. We got licenses for everybody in like our Vanta v0 account — and this allowed teams to kind of go out and like really start to experiment and build out more pages. This kind of like a shell. It's not our exact UI but you know it's good enough to talk about general flows and meet with customers and you know we use it for almost everything at this point. So over here is one that we built when we thought about the Vanta AI agent and we're

trying to understand what experiences do we want that to have. So when you think about Vanta, one of the big things we do is help people get compliance which is really about helping them set up a security program and then prove that to auditors or regulators. So a big part of that is building out your policies. This could be like a code of conduct, could be your vulnerability policy. And it's really easy when you're writing all these documents for them to kind of get uh out of date uh for like the things that you're monitoring or not. So maybe you have an SLA for vulnerabilities that's like 4 days. Your document says seven. You can have lots of contradictions across these. So a feature we shipped recently and this is like the actual original prototype for it was on the policies page. You can get these AI suggestions. You can click in here and see that hey we notice some of your AI policies are inconsistent. We kind of use this to kind of get an idea around what would it feel like to have an agent in the sidebar. What would be some of the experiences? Would we want little cards like this so people could expand and be like actually the thing I want to change is the SLA in my policy versus I want to change the setting. So like which one of these is accurate or not? I can go back and forth and ask some questions and refine. And so we use this to kind of explore multiple things. Do we want to kind of create many forms inside this uh inside the chat experience? Like chat's great. Sometimes I want to use text. Sometimes that's a it has a little bit of a blank page problem, you know, because you're like not sure what to ask or you know what I should go do. So can we give some forms and controls and little mini cards in there. Um so this was something that like really helped us like establishing some of these templates. — Yeah. for the team so that PMS even nontechnical ones could get going and then helping us start to feel out you know what works and what doesn't work in experience. — Yeah, that's really smart like starting with like the shell of uh what Vantest product looks like and then customizing from there is like a really smart approach. Yeah. — Yeah. And then if you go over we kind of over time PMs have kind of done a lot of things like our customer trust management product. We think about authoring tests you know these different MVP experiences. As you can see, every day we have kind of PMs adding, you know, new prototypes and building things and then even building some of using some of their own building blocks and sharing them out. Some of our more technical PMs have gone so far as to build like massive recreations of our product that they maintain to be able to experiment with ideas and we've seen a lot of designers start to use it as well. — That's awesome, man. Like that's like I I've talked to a lot of like uh CPOS at this point and I think um it's actually very hard to have this kind of culture, man. like especially if you're used to writing docs and like you know doing this stuff like to actually have this uh prototype repo and like and it's like if you have that repo for like uh prds I'll probably read maybe like one of them max but because it's those prototypes like yeah it's just fun to play with them right like I want to check them out you know — yeah and I think it like with like I mean I'm not to discount like words it's really important I think to like write we do write docs and you know I still love docs but I don't know it's so easy to misinterpret when someone says thing, you know, like what does that mean? Or you just get a bunch of superlatives in a sentence and we're like this will be the best customer experience. It will solve these three problems, but you're like, "Yeah, but what exactly does that mean? " And I think like bringing it to life in a prototype really helps and helps build that design engineering product collaboration early. Like one of the things I've heard from engineers is like, "Oh my god, I now understand what they were talking about. " You know what I mean? And it's also a good handoff to designer. Not saying that the design is done or their work is less valuable. There's still a ton of work to go do still from this point. Um, but I think it creates a an easier shared language between everybody working on making the product. — So, how does a typical product review go than if you have a prototype and maybe like a doc like you know how do you guys structure it? Like when you get to the prototype stuff?

— Yeah. Um, I ended up naming ours like product deep dives at my last job. I called them product walkthroughs. I didn't love the connotation of reviews even though I'm sure some people be like it still feels like a review to some degree. Um, but I want to make it more of a conversation. Yeah. And I think one of the things I really want to do is have everyone bring ideas at any stage. So, uh, this could be it's really about like setting the context for what type of feedback do you want? Like are we about to go ship something and we want polish level feedback or we're exploring a super early concept. It's a VZR prototype. It's not going to be exactly right, but we're really trying to figure out is the flow and the value and the marketing message and how we'll talk about it there. Um, so we end up doing these meetings. We have them every single week. Um, basically their schedule people can kind of sign up and get on them. We add them if we want to. We try to do things async. We have like a very remoteled culture. — So we have offices a couple people, you know, we have people that go to them, but a lot of people are remote, including me. Um, and so we try to do a lot async. I would say the big thing that I try to do is just like get into the pictures as quickly as possible. I think a lot of times PMs want to come in, especially PMs, but it could happen to anybody, and like set lots of context. And I think the advice I give people is like I probably have a lot more context than you think that I have. I've probably been like stalking your designs in Figma or looking at your PRD or just like I end up talking about the product so much and like where we're going that I have a lot of context. So I'm like hey let's just get into the experience quickly like hey here's what a customer is going to feel like I they've got this problem. Let's walk through something that's going to make it a better experience for them. So really jumping into the pixels, starting to click through and then setting that expectation of like the level of feedback we get. I also think it helps build a culture around lets me like quality and like sweating the details. — Yeah. — Um not to say that's what we're always doing in these is like detailed pixel analysis, but sometimes we are. And — letting people know that like hey the word choices really matter. Like for us as a security compliance company, we have tons of startups that have no idea what SOCK 2 is. It's super scary. Oh my gosh, I've got this audit. I've I don't know security. And so, how do we think about like the copy and the language choices, making it like friendly and supportive while still being like accurate? Um, is all the types of things I look for in these. — And do you get to a point uh during a review where like you're like, you know, you're walking through the customer journey and then you're like, "But what about this? " And then you just like prompt Vzero live in the review to to do it. Like, do you have that kind of interaction? Yeah. Uh, I think that has definitely happened at least twice that I could think of where I've been in one of these and like we just like pull it up. Um, there's definitely times when we're going through like a lot of Figma mocks as well. Um, and but yeah, I think when we have the live Vzero demos, which we do a lot, I've definitely had PMs that are more comfortable just be like or designers be like, "Hey, let's actually just try that now and like see what happens. " And a lot of times you get a pretty good result. And I think too as these tools have started to have a design mode as well has been really helpful. like there's a lot of the times I'm just like hey I just want to grab this button and change it or move it from here to here right and having that direct manipulation I think is really powerful um as opposed to trying to express through text exactly like oh I want to move this thing over here or do this thing — yeah I think that's one of my key lessons building AI products like I feel like AI can get you 80% of the way there but you still want to do like manual tweaks after that should be easy to do like manually like have to prompt the LM each time is kind of annoying like to change a button you know it's Oh yeah, definitely. Like I mean I still have a bunch of personal projects where I'm like writing code like every single week and you know the LLM's help me a lot but like there's definitely times where it's like no I just need to go in here and like get it exactly the way I want it or fix this one issue. — Exactly. So let's talk a little bit more about other ways to use AI to save time at work. For you personally like what

has been most helpful like in using AI to save time like is it like you know performance management? Is it like writing documents, brainstorming or what has been most helpful? Yeah. — Yeah. I think it's um you know LLM's obviously were like really taking off when I started Advant uh two years ago and I think that ramping up on this a new space like I'm someone who loves to learn new things and like I was not an expert in compliance having come from like GitHub and doing different things before that on like one drive and other products. So — to me just being able and there's so much jargon in the world of security and compliance. There's kind of a basic thing there of just like asking, okay, what is, you know, IL4 versus IL5 standard for like getting Fed frame certification, like things I kind of like need to know cuz we're building products around these, but like there's a lot of depth that like I don't have in those areas because I'm not a SME in all the areas. It helps me a lot with that. — I think processing feedback for people's reviews, you mentioned performance management is really helpful. We have a pretty big feedback culture which I love, but it can be a lot for me to go through for like each of my reports like the 12 people that all sent them feedback. So, I've created a GPT um and actually our last review our HR team actually created and sent out a GPT where they'll like upload all the feedback here and it does a great job of like synthesizing it, giving you direct quotes and I can even go and ask it be like oh what did Boris say about this person or what did Deb say about this person? It'll find that quote and tell me if it isn't already pulled out. — That's been really helpful. I think market research has been a big one for me. Um, I still create prototypes. I'm still in like Vzero using it, but I don't know the one that really is probably like top of my list for impact is actually some tooling that we built internally on top of Gong. M so we have like gong stream listeners when different subjects come up with customers like different competitors uh different features that we've shipped other topics and then we have a Slack channel that we've kind of built that goes ahead and like pulls that in by product and we can kind of understand hey what is the conversations happening live is this a pre-sales conversation a post- sales conversation what are customers saying about the product what are they saying about competitors because like I love are like we have an amazing sales team and amazing sales leaders but you know the detail you get in like a win- loss analysis like they're not really incentivized to make that great and they're not really thinking from a product mindset. So a lot of times it's directionally correct but there's a lot of details and questions I have and being able to go directly into the gone call at that moment and hear what did they say about a competitor, this feature, what are they saying about this area of the product like risk management or the type of work that they do in vendor reviews and jump in there. Um, we've had a technical PM that kind of like hacked together a bunch of like internal pieces on top of the Gong API using chat GPT and Slack and everything. That's made a nice internal tool for us there. — So, you can basically ask questions about the raw transcripts from the calls and stuff, right? Like that's kind of cool what — Yeah. And then we get a feed of the things that we say we care about. Um, which I think Gong I think has built in now with streams where you can get just a feed of some of the words that are mentioned and it's just so much better now. Um, but yeah, I think that's helpful for me. I mean, I'm just so like such a product customer person. I want to like get in and be like, "Okay, what did the customer say? How did they say it? What exactly was it? " Um, so that helps me a lot. — Yeah, that helps. Yeah, I do something similar like every product I build, I have a project where I upload like a bunch of like deep research stuff about the product space and then also like customer interactions like you know just on Discord or something like some chat threads — and then I have my like PRD template or whatever and then I just work with that. Like that's probably I mean I hate to admit this but that's that project is probably my most important coworker in com in making a product. — Yeah. And I mean I think a lot of this like gong customer analysis for us especially when we're doing features that are you know around like churn and they're talking to account managers and CSMS or they're talking to like sales. — Yeah. — Is so helpful for me just thinking about strategy wise as we think about growing the business and revenue and just like making customers happy. I can use that as a treasure trove to kind of like just ask it questions and aggregate across all these customer conversations. Um because not everybody can remember, you know, in the best way what exactly happened or summarizing it. This could be a personal one. One other one I do is I use some uh different uh dictation apps. I've been using uh trying to like leverage chat GBT directly more for this, but I've used a couple different ones as well where I don't know I go off during the day and just need to like get out of like the home office and like walk my dog and a lot of times it's easy for me just to like stream of consciousness kind of like ramble ideas and then now leveraging LLM to like grab all that and actually turn it into like a more structured document. Um, which has been really helpful for me just to think through — strategy or other things where it's just easy just to kind of talk with someone and like leverage like the AI to kind of be that person to like prompt me and to talk to. — Yeah, dude. The dictation is a game changer for me. Like I have a I have like a life coach project like where like I have all my like goals and like you know my fitness stats and everything and like every quarter I check in with it on how I'm doing and like it has so much context on me. Hopefully it doesn't leak, but Like I think it's better than any the It's probably therapist, you know, at this point. Yeah. — Oh, definitely. And I feel like I don't know, maybe some of product kind of is like when you're thinking through the strategy and stuff like that therapy session of just like you just need to like talk it out, you know, and be like, "Okay, what if we did this? that? " And I don't know. I feel like a lot of the things that were in my head, I then would have to remember them and then sit down and like write it all and do it and just short circuits the whole process a lot. And the last thing I'll mention is like lately I've been having meetings with like you know my team and stakeholders and sometimes you know we're trying to figure out product copy or like a problem and I I sometimes I just like screen share like you know like chat or something like hey let's ask chat this question and it still feels kind of weird but like you know it's like having our teammate uh trying to brainstorm with you. — Oh definitely. Yes. Yeah. I do that a lot and I don't know even as like a product like feature for us like a lot of things we've been looking at for the Vant agent has been that for like GRC professionals and when we kind of look through the data and talk with customers we see a lot of them doing that too you know of just like hey I just need somebody to go back and forth with on this thing um yeah — yeah okay so let let's talk a little about um like how how are functions

changing with all this AI stuff and hopefully changing for the better right like um maybe we can talk a little about careers first, you know, historically you have to become like, you know, like a manager, like a director, VP and CPO to kind of like, you know, make more money and get promoted and and now and now we're seeing like, you know, these AI researchers getting paid like $100 million or something like so do you think like the ICPM is coming back to you or like how do you balance like what's the best career? Is there like a viable IC path now for senior PMs or — I definitely think there is and like I don't know I definitely create one at every company that I've been at. Um — I think that I see work is so critical like I mean even for me as like a CPO. I look at my job as like there's also the IC stuff that I own. you know, Christina, our like CEO and founder, like she will still write PRDs for specific features that she's like excited about and she creates PZO prototypes and sends them to me and I send ones to her as well of just like, hey, what is this idea? And we're like, okay, great. Let's just like knock out like a quick prototype and see how to go do it. So, — I think that IC work is incredibly important if you're going to be a manager or not. Um, I think that people can easily I've worked with amazing IC's at the VP level, you know, or VP equivalent level. um there and see that work happening a lot because to me it's all about like impact and I think when you can get that like flow time and go like really deep on a problem and really drive that and not also be thinking about managing the team growing the team hiring the other things you can have a massive impact like we hired a principal engineer recently Tavanta who's had a massive impact we've hired a couple staff PMs some of which I expect to get to principal as well um who have had a really large impact like across the board. So I think there's definitely a lot of room to kind of grow as a senior IC. — And what do you think the like when you talk about like impact like growing your career as IC? Is it more like — Yeah. — Like you have to align more people or like your product is more complicated or like what what is the progression there for IC? — Yeah. Um I think for PM there is a little bit more breadth than you probably require for engineering like um I always liked for I lead the engineering team as well. So like I when I think about engineering I kind of really like the meta's archetypes that they have and they have an archetype around like the coding machine that just like cranks out like a ton of code which I think is great as well. I think for PM um the big things I always look for is like autonomy and independence I think is like the career progression. So I always like just to be able to drop off an ambiguous problem with our most senior PMs and just be like hey this is worrying me about like our strategy or our experience. Can you just go figure it out? and like almost to some degree like the less context. I'm not saying people shouldn't ask questions. They should ask a ton of questions and like they do. But I think the less I have to kind of be involved and then more I know they will just take the agency around saying, "Hey, I'm going to go find figure out who are the four people I need to go talk to. I'm going to set up these customer calls. I'm going to go coordinate with these other people across the company. Oh, this is a pricing and packaging thing. Let's pull in finance. Let's talk to a couple like AEES that are selling our product. Let's talk to the marketing team. Let's figure it all out. " So I think the more people can really take kind of independence and ownership and accountability to me like that's one of the keys about being a really successful IC. And I think one of the exciting things is you're kind of untethered from a lot of other responsibilities. So if you're someone that gets a lot of excitement out of like delivering which is big for me as well. Um it lets you just do that non-stop and maybe some of the other work that's that you know is exciting for some people but like not for others. — I love that. Yeah. It's about like how ambiguous the problem is and like how much agency this person is showing to figure stuff out like just do it right. Yeah. — Yeah. Definitely. And I think that the more you can kind of build relationships with people throughout the company in different disciplines is really helpful. I think that it doesn't mean you're going to fill up your schedule with a ton of ones, but like who are your key people that you can go to pre-sales and postales and support and like understand what's happening on, you know, the marketing side and everybody within EPD. So, I think the people that do this really well have a good kind of like network and knowledge of what's happening and they also know how to communicate at the right level. Like there's definitely like when someone on my team is like talking to Christina, they should know the context that she has as like the CEO and founder, the context I'll have versus they're talking with like the engineers they'll be working with to go build the product. And I think understanding how to work at those different altitudes and like what message does that person need? What decision do you need from them? And what's the minimal context? Because it's easy for me at least as like the CPO to kind of get a little bit like overwhelmed. You know what I mean? of like people want to bring me into every last detail and like I actually love details but like sometimes I'm like oh man this is almost too much for me to take it all in like can you digest it down for me um but I still want you to make sure you know all those different pieces so that you can like make it a great experience and — and obviously like AI is very good for that right like you know try to like uh — definitely — yeah but I think that's one of the key lessons I learned like as someone who likes to like get done uh one of the key lessons I learned is that like um building the right relationships like having those coffee chats is actually really important for getting done like it's not just about you like oh yeah sitting there. So like you know yeah for people who are like kind of drivers out there like you know hy agency you do have to like set up the right relationships up front before you like try to get them to do something or like work with them. — Yeah. I mean when I'm onboarding anybody new to my team one of the first things I have in their like 90-day plan is like these are the people you need to build deep relationships with and I think the more senior you get the more broad you want those to be. — Makes sense. So, um, so I guess like just to double click on this, like there's like a lot of ants right now, right, for like big tech PMs like you know, Microsoft's doing layoffs like pretty frequently. — Yeah. — And I guess I do believe that like the average successful company will get a lot smaller like just because of all this like AI stuff. Like do you have any advice for like the big tech PM like

worry about their job or like you know paying their mortgage like you know? — I do think there's going to be like fewer PM roles at these companies over time. Um, to some degree. I mean, I think it's a good thing and I think it also can be exciting depending on how you look at it. Like, as somebody who spent a lot of time at Microsoft, I felt like one of the things that kind of disappointed me and one of the reasons why I kind of switch teams frequently and like always tried to be on earlier products was I really like to do like shaping and felt like we own the direction of where the thing was. And when you get on some of these big products that are so huge, you feel a little bit more like a cog. And like I don't think there's anything wrong with being like a feature PM. I think there's great career tracks and things you can go do there, but I think people that can expand out more. I think the PM role even at larger tech companies is going to look more like a later stage startup PM role where you're still thinking more about the business, how it connects into marketing, working with a bunch of customers, like very directly on like a broader scope versus like a very narrow feature where you're just working with like a couple designers and a couple engineers. So, I think it's good. It's definitely like stressful as people think about it. Um I think that um too technical expertise is going to be more and more helpful like when you think about product engineers and how valuable product engineers are. — I know you know Advanta we've talked a lot about continuing to hire and like growing out design engineering and I know some companies like Verscell are like very heavy on design engineering and I think the overlap of the disciplines is going to start to happen a lot more and like already is happening with AI tools right where it's easier for PMs to produce prototypes to produce code. I've talked to our designers about like how many of them are going to be like submitting pull requests by the end of the year even if they're not technical, right? So like how are we pushing them forward? How can engineers now get more context around customers? Like now that we have like a lot of this gong infrastructure we've built internally and the tools on top of it like a smart-minded product engineer can go in and like learn about customers and ask a bunch of questions and figure out where we should be going. So I think the roles will be blending a lot more. Yeah, — I still believe that there's still like technical expertise people need. So I'm not telling anybody to like stop learning how to write code. Maybe I'll be proven wrong in like the very long term, but I still think there's, you know, a lot of value in, you know, having that technical expertise as well, even if it becomes more of an editing role over time. — Yeah. I mean like my engineering friends can like, you know, I I've been vibe coding stuff, but like my engineering friends can vibe code much better stuff than me because they actually understand like how everything works. They can get better prompts. — And I think we've seen the same thing. like we have uh at least one of the PMs on the team who started as an engineer which way back in the day that was like how I started. Um and just like the level of vzero prototypes and how he can like maintain that and how fast he can move with customers directly um is just a really big advantage, right? Yeah. — Yeah. So, so I think my key takeaway is like you know you got to have the a agency and you got to be able to wear multiple hats. Like you know people at startups all wear multiple hats, right? They're not like oh I only do design or I only do this. It's like, you know, — and I think too, like you said, because you were asking about big tech companies, I think not trying to understand how the business works and not let yourself get so pigeonholed. Like when I think about people that come to Vanta and it's like I spent a bunch of time at a big tech company, too. But like I definitely have some like warning signs I kind of like look out for when people are coming to Vanta where I'm like, "Oh, how much of a kind of cocoon were you put in within the company? " And I think making sure that you're not kind of so insulated because like we'll see people who come to Vanta and they could be amazing. It just seems unlikely they'll be successful here because they're so used to having all the support infrastructure. Like they have so focused on a narrow set of things and doing them great that they don't really understand, you know, they don't do a ton of like direct customer interactions that's all brokered through like another research team or like you know what I mean? They're not really thinking about pricing and packaging and never have and aren't really working with the sales team as much and like they don't have the full view of like a business and how to grow and make revenue because they're like part of a massively successful business which is super rewarding and financially lucrative but like you know what I mean? It's like um they don't get to kind of see how the big picture works. And I think when you're at startups even ones like Fanta that are you know like series C and things like that like you still need to know how all the pieces fit together. And PM is such an important glue there. So I would encourage people to be like get more technical, spend time with AI tools, like reach into design engineering doing that, but then also like understand how the rest of business like really works. Um — yeah, there's like too much subdivision, man. Like there's like a product manager that's customerf facing program manager that manages like the sprints and like way too many why you have so many roles, man. It's like have one person do everything like you know. — Yeah. And I mean that's definitely the philosophy that like I have at Vanta and I think our general philosophy is you know we don't have a lot of like uh like TPMs and it's like nothing against like TPMs but I'm just kind of like hey engineering managers like you're accountable for like execution and the rhythm and doing it and we do have an amazing EPD operations team but I think of them as like setting up the big processes that make whole disciplines successful right like they build a bunch of our and help us choose the tools we need for like customer feedback analysis and some of AI tools there and like how we're doing road mapping tools and other but they're like setting up the big systems you know like a capital S versus like oh I'm going to run the standup you know and it's like no like engineer can do that by themselves like — yeah like don't talk to me about like uh you know like the agile stuff like I didn't want to talk about it yeah cool so let's talk about uh for

last topic let's talk about Vantas AI a agent and what goes into building AI products right it's actually like from my experience it's actually a lot more difficult and annoying than building a noni product. So maybe you can talk a little about uh what does this Asian do first like what kind of problem does it solve or maybe share what it looks like. Yeah. Yeah definitely. Um yeah let me pull up what we've got. I mean I think one of the things that's really exciting for me and one of the things that I think not enough companies focus on uh with their AI products is really making sure the quality is extremely high. like there is so much kind of like hype around these different products that doesn't fully manifest in you know the product experience that you feel and so I think for me one of the big focus that we have is on the quality of the AI that we have responses and answers so I can walk you through a couple of these experiences I guess you saw the zero prototype of what became a bunch of this UI now but again as I mentioned before people have security policies. There are lots of documents. It's hard to keep them all in sync. consistent with your monitoring. So, we've kind of built an agent here where and we've even gone further. Instead of you having to ask the agent, hey, do I have any inconsistencies? Can you please go fix these for me, which I feel like is kind of work on the customer. You have a bit of this blank page problem. We're just going ahead and surfacing these insights directly. So, I can click right here and be like, hey, you need to fix 14 SLA inconsistencies. we'll pull this up, tell you what the inconsistencies are, giving you a list of sources. Again, building trust and confidence on like where the answer came from that you know it's good and I'm like, "Hey, actually, I do want to update this. We're going ahead and offering that next suggestion to you. " So, again, not putting it on the customer to think about what it is and then giving them like a simple card that kind of explains, "Hey, we're going to make these edits. Are you okay? " And you can go through and, you know, accept those edits and make them. Um and I think yeah and I think one of the big things for us you mentioned like AI products how to think about them is like what are the prompts that you're expecting customers to write and like what are they trying to get out of it. So, you know, in this case, I think not leaving them with a blank page problem where you're just coming in around like, "Oh my gosh, what do I need to go do? " Like, we have a couple prompts here where we're like, "Hey, give me a summary of my ma password management requirements. " Like letting people know which types of prompts give them an idea of what the agent can do and like what it can't do. Also, like refusing. We spend a lot of time on quality and on eval to make sure the answers are really high quality. — Yeah. and people are getting things that they accept and they like. Um so you know whether it be policies like we talked about here you know a big part of preparing for a compliance audit is also you have to go collect all this evidence. Vance automates collecting a lot of that evidence and we can now evaluate a lot of it. So here's a PDF that I had to go write. I created this with our policy builder which is like AI enabled and helps you write that policy which is a great experience. But then, you know, a lot of people producing the evidence maybe aren't on the security team or you're a startup and you don't even have a security team yet and you think you did it right, but you're not sure. — We can go through and we have subject matter experts at Fanta who are constantly evaluating the prompts, these different types of evidence that you need to submit. And we can tell you like, hey, we found these gaps and here are the suggested improvements that you should go make to this document that will make it, you know, likely that an auditor will then accept this and be like happy with the document that you're going through. So I think these are like a couple of different things whether it be questionnaire automation as well. Even when you think about like going to someone's trust center um you know you're understanding what is their security compliance requirements. I can go in here and kind of like ask a question like what is the you know encryption that's used it doesn't even um and we'll go ahead and answer that question which I think you know preAI when I think about this you kind of design the page of like again the customer has to go do all the work they have to like dig through it they have to read go figure out but what do they actually want to do they just want to ask a question so can we just like jump to the end you know what I mean um — yeah they don't have to like yeah I I feel like uh the best AI products actually like they basically help you save time. Like if I click around all this stuff, I can find an answer, but it'll take me much longer to just ask me a question, you know? It's like — Yeah, exactly. And so, I don't know, the way I think about AI is like it's a new tool in the toolbox, — and I don't think it's like rub a little AI on it. Like, it's the right solution for like everything. But I was like, — if we have a customer problem and AI is a great solution, we now have this whole new set of tech that we've never had before. And for processing tons of unstructured data which is pretty common in compliance like documents, keeping documents in sync, producing documents, analyzing them, it has been kind of a game changer for us. We also have to do a lot of analytics things as well. So being able to dynamically generate charts and graphs on the fly from our agent has been like super helpful as well. — Awesome. Let's talk about um I I really like your emphasis about quality. I think the difference between like a new AIPM and a mature AIPM like a new one is like oh check out this cool demo that I have you know it's like and a mature one has been around a blocking like made mistakes that they're like oh I got to look at the eval scores like just because this thing work one time for something it's always going to work right so yeah so maybe you can talk about like at a high level like how the eval works like you know do you start with like a manual eval you have some criteria or yeah how do you approach the eval process? Yeah, definitely. I mean, I think early on we

have a kind of AI platform that we've built internally. We kind of started with questionnaire automation. So, security questionnaires people get in. Same if you're sending them as a buyer vendor ones where you're the one sending the questionnaire. Someone drops a bunch of docs to you and you want to kind of automatically ask the questions. Um, we kind of for both of those let like a thousand flowers blooms if you will. We kind of went and we're just like, okay, what prompts are going to work? What models And like started to figure it out. only after we figured out what works in a given space like we really focused I think on iterating quickly and testing lots of models testing early prompts seeing what feels right only later did we then try to like platformize and productize some of these pieces so I felt like a year ago or two years ago it was a lot of kind of experimenting trying things letting teams be really flexible we have found certain things that work so we have a core kind of like rag pipeline and architecture that we leverage internally we have some shared eval tools that uh we like we work with like Raria AI and some of these other ones around evaluation and optimization. Um and I think all of these have become really helpful for us and then as the products mature I think then the team really focuses on quality. I never worked on search results, but I imagine if you're like a PM at Google working on search results or something like that, it's somewhat similar where I would say a lot of the emphasis I have with the PMs and the engineers on those teams is, hey, what is, you know, our coverage rate for like a questionnaire for a set of questions? What is the accuracy of our responses? How are customers scoring those up and down? and really kind of tuning the prompt, tuning the vector database, the chunking, all the different pieces that we kind of need to make sure people are getting like really high quality answers. And then now a lot of it is actually turning the APIs of our system into tools so that we can make the agents take more action on your behalf. I see. — So how are we exposing tools out to customers so we can just say, "Oh, great. You found a test that needs remediation. We know the name of the person. Like just go ahead and assign the test to them. " And it will like automatically do that. So I think that's been the latest evolution for us. And then there's some things that I would say are more out there. I know OpenAI just announced I think a big update to computer use. I haven't like checked it out yet, but um you know that's something we've been playing with a lot that felt like pretty early. But I feel like all these areas I'm always trying to be like let's have somebody constantly thinking about some of this tech even if it's not quite there. Like when computer use first came out for some of the use cases we'd want to use it for. Let's just keep a background thread kind of going where there's like an engineer or a product person like constantly playing with it over time because like when it does become a big deal, I want to be able to like jump on it really quickly or when we think it's hit that quality like bar that we can, you know, be in the position where we know how the technology works and can like jump on it quickly. — Yeah, you got to be constantly testing stuff so that when it does cross the threshold for acceptable like percent acceptable, then you can implement in the product, right? Yeah, it's uh — I think subject matter experts are really important for us too as like working in an industry where you know a lot of the PMs have not been security experts or former auditors or things like that. So we partner really closely with our CISO JD. She and I work closely together. She has a whole team of SMEES and a large part of their job and some of them it is solely their job is to help us with these eval loops and to kind of go through and be the definitive experts of like hey I have run audits at like large audit firms for a long time and they can go through our responses and really help us tweak and tune. So the product team can stay focused on the data and the customer experience features we need but not have to become like deep technical experts and you know is this the exact language that should go into a policy or the exact way to evaluate this evidence in a way that like a big audit firm like Shelman or a line or PWC would accept. — Got it. Yeah. That's a really good point. I feel like eval there's a lot of like fancy techniques right there like LM as a judge and all this stuff like you can that you can do but like I think it comes down to like human taste at the end of the day. You gota you got to have like gar garbage in garbage out, right? You got to have some like really good human evals like human criteria that powers all this other stuff. — Yeah. And we have like I think it's like over 50,000 evals. We have like you know multiple like synthetic golden data sets that we use. So we do all those things too. But — yeah, — I do find that especially when you're early and shipping a brand new feature, just telling the team it's like every day just look at like what everyone is asking and how it's working and like sweating those details just like you would shipping any feature. Um, yeah, and I think the how far we can push chat as an interface has also been something that's been like debated a lot that's interesting to me. I think there's things it's great at, as you probably saw in some of those prototypes and some of what we ship, we're starting to play with cards, more interactive things. you know, cursor Windsorf have done a lot there as well of like showing little mini command prompts asking you to approve when it's running commands in the terminal and things like that. And so I think it'll be interesting to see how that evolves and you know where you want it to be chat, where you don't chat and like what becomes this hybrid. — Yeah. And like chat is great because like the users telling you their intent but yeah they need a lot of guidance. There's probably a lot of innovation on top of that. Yeah. and like once you have chat like you know for my AI product I haven't shipped chat yet because once you have chat like then they can ask whatever question they want and you got to account for all these edge cases and all this stuff you know so — yeah definitely and yeah we had a couple experience where we started with pretty aggressive refusals as well for because of that same concern too where we're like hey sorry that like this is kind of like a labs feature we we can't help you with that problem yet like we can do these other ones um — but yep — cool Jeremy Well, this is awesome conversation, man. I really appreciate everything from like prototypes to your workflows to like real talk about PMing and uh so where can people find out more about Vanta's AI agents or just like, you know, where can people find you? — Yeah, definitely. I would say uh check out vanta. com, see what's going on there. Um we have a bunch of exciting resources for people. If you're a small company and looking to go get compliance sock 2 ISO for the first time, if you're a big established company, we help extremely large customers like Omni Hotels, GitHub, like others handle um compliance and move forward there with vendor risk management, customer trust management. Um you know, you can find me even though I don't post as much on like X or like on LinkedIn as well. But um yeah um I mean I think the other big thing for me is I would say is like we are growing a lot Advanta and we are hiring a lot. Um I think last quarter I hired over 50 people across engineering product and design. Um so if you are interested in a job uh please reach out check out like our listings or reach out to me because we're growing the team a lot have a lot of exciting work going on across AI and even other areas that aren't AI related. If you send Jeremy your uh Vzero prototype, you'll probably get a better response than like ask me to pick his brand. — Yes, that I guess that is the hack. Yeah. — Yeah, that's the hack. All right, man. Thanks so much for your time, man. This is an awesome conversation. Yeah. — Yeah. Thanks a lot, too. See you.