Stanford CS547 HCI Seminar | Spring 2026 | Observing the User Experience in 2026

All right. So, our talk is entitled Ground Truth. Yes, that's a picture of a mortar and a pestle. Keep it in mind. We'll get back to it later. We've already heard about us, so thank you, Christina. Um, important thing to note at the bottom. Yes, we're married. — This is important to note because a lot of people don't know and then they get confused and mildly taken aback when we like whack each other and talk over each other. — We even have children and everything. So, um, — so this will be a very conversational talk. So, be prepared for some back and forth and some chitchat. Generally speaking, Mike loves AI more than I do. — And so be prepared to hear some um — let me put one thing. This talk is much more of a why talk than a how talk. Uh we'll talk about this later, but the position your mind uh to think about like what we are doing is we are trying to uh ask big questions uh rather than give you specific techniques. All right, but first — to start off, why did we write the same four times? This is the book, Welcome to Observing the User Experience, third edition. — Um, the third edition is supposed to come out next year. — Yeah, it's — Elsir delayed it multiple times, — but please look for it. um pre-order it, tell all your friends to pre-order it, tell your professors to put it in their classes, do all the things that we would like you to do. — Oh, right. You're — to say that I think this is literally the single best book on user research that you can buy and I'm not picking that up. — I wouldn't believe that was true. — You know, we actually I don't want to say we think so too, but definitely. So one of the things that I want to point out is that one of the reasons why this is a why talk is that the writing of the book each edition kind of articulates and canonizes the relationship that we as the authors have to what we do as researchers and what we where we think the profession is going as a whole. Um, and so the first one came out in 2003 that was written by Mike Alone. And that was Mike kind of making sense of this new discipline, this new career that he found himself in. There weren't a whole lot of user researchers who called themselves that out there. And in a lot of ways, it was Mike writing the textbook that he wished he'd had and that no one could give him. Then in 2012, yes, I know, nine years later, the next edition comes out. At this point, we're married. — I've been teaching user this very book at Berkeley in a graduate class on user research for a couple years, — which I will say was selected independent of them selecting Liz as the person to teach the class, which was very awkward. — So awkward. Yes. — Teaching a book that is written by your then boyfriend to a class of students is just horribly embarrassing. — Um but in it so we got to the first second edition and I was like you know I don't think this first edition is really working. Like there are specific things that I need to say about what it is to be a professional user researcher in 20 at that point 2010 that need to be articulated because this is a textbook and I've been teaching it and I feel a responsibility to the next generation. — That's the second edition. — That's [clears throat] the second edition. Looks a little different. Same general cover theme. I was all set to write the third edition and I was in fact midway through writing the third edition. — This is uh approximately 2019. So our publisher came to us about 5 years 7 years after the first one, six years and said hey we need a third edition. We said great and uh then Liz started writing it uh because she had been the she's the lead author now. I wrote the original — and at that point I'd been managing a team of designers and researchers. So I had even more settled opinions about how I thought things should happen and what was working and what wasn't. And then the pandemic happened and there was chaos and we had two small children and we basically told our publisher to go away until the suffering stopped. And by the time our publisher came back and the suffering had stopped and we were all

ready to pick it up again, AI happened. And we realized this was about 6 months I think after the first big release of the first big models. — So like basically you know we uh wrapped up writing it in like 2022. So this is four years ago and then GBT4 comes out. We're like, "Oh, we should probably do something uh different and or GPG GPD3 at this point. " Then um anyway, to make a long story short, uh like chat GPT comes out everything and uh we realized that we needed to rewrite the book that we had just rewritten — and it's a thousand pages. — Yeah, it's a thousand page one. like — and every single chapter was touched — by both the switch to remote work during the pandemic but also by AI and so we overhauled the whole thing. — This is 100% not our cover. Our publisher uh gave us this. This they gave us a choice of covers. This is the ugliest one. — but it's bright yellow which is good. And so it's coming soon. It took us another few years to grapple with it and rewrite a thousand pages and get clear about where we thought — so — the field's going. — Yes. So this talk is based on what we learned rewriting the book over the last couple of years about what user experience research is and how it works in the modern AI era as we did this. — So um should I do this one? — Yes. — All right. So uh what we're going to talk about is the confluence of these trends today. So essentially UX practice has as a discipline has really changed. There's all these new tools, developments, there's all these new behaviors, there's all this new stuff that's out there. And so the actual practice of user research has deeply changed. One, two, we have seen major shift in the power dynamics. We talk a lot about power dynamics which is literally about like uh the power of the giant corporation versus the person who's working for who they are trying to make a product for. We talk a lot about the power dynamics because they have shifted a lot and we think it's really important to make that explicit and to discuss it because that's a thing that historically never got talked about. No one ever told you when you're a UX uh or when you're an HCI undergrad that when you're working for one of the big companies, you are a cog at a machine and this is how you have to be that cog uh if you want to work for those companies. And so we want to talk about that. And then the third thing is obviously the technology has blown up. And so uh and one of uh yeah so that's what this talk is about is how these three things kind of influence each other at the same time in this moment in history. All right on that note this is me right? — Mhm. — All right so uh how has the technology changed? So, as we know, uh, the, you know, AI is like this, it's like this tidal wave that goes through and like if you've ever seen pictures of tidal waves, like it's not like a single wave like you see in the pictures where it's like a big wave and then it's over. A tidal wave when you see it hit, it's not that big, but then it keeps going and going. And what that does is that keeps pulling everything along and mashing it up. And that's what we're in the middle of right now. So um the first thing is obviously that we have uh that we have all these technologies have all these tools. So what we did when we revised the book is we just dropped a bunch of methods from uh from it because they're just simply better tools. In the first edition and the second edition, we talked about transcription and how do you do it and when what do you do? It's free now. Free every transcribe everything like record and transcribe everything. We talked about how to do uh how to code those transcripts when you're reading them. Mark it mark up the quotes you want and kind of do concordance. Forget about it. It's all okay. We disagree. Uh so preliminary analysis of those recordings and transcripts. You can have something else do it. to do your video editing. You can have translation. You can uh use Notebook LM uh with all your transcripts. That's the Google tool for

those of you who don't uh to get kind of ground truth uh like when and what transcript did somebody say talk about this. So really like uh you know the model is actually pretty good at uh writing first drafts of discussion guides. They're pretty good at it. One of the funny things is that I actually had a model just do this couple uh days ago and it regurgitated something that looked almost exactly like things from our book. So that was kind of exciting. We were like, "Oh, right. It's us. — We're in the anthropic settlement. " It's like like uh yeah we are in fact in the anthropic settlement. So um and so like uh it writes uh you can get pretty good survey questions you can get uh you know first draft of all kinds of things. All right. So, uh, remote research tools are awesome. You know, you get prototyping tools. These are these auto. All right. So, all of this stuff is essentially taking existing practices and automating them and making them faster, making them easier. And that is on the whole a good thing — except — however this is a thing we never had to deal with when we wrote the first two editions. The thing is that you know user experience is about getting ground truth. It's about actually getting the ground truth of like human beings and their experience and their actual genuine experience. And what we have now thanks to all of these AI uh tools we have hyperscaled our fraud. We have uh every single thing that you can think about that can be faked is being faked. Survey answers. people. We've had conversations with people where they talk about the fact I mean, you've probably heard this about job interviews where they've interviewed somebody who says that they're a stock broker and uh that you've recruited them and they've answered all the stock broker questions and you get on the video call with them and you're asking them questions and there's this like pause in between the answers and uh then the person uh who's the interviewer realizes, "Oh, you're just reading off of a chat GPT screen. you don't actually know anything about this. And so that like kind of how to deal with that has become like a major uh problem. And so our advice about that, you know, we don't have like this advice is this like we wrote this deck like a couple like a month ago. This is probably already out of date, but uh our advice is that uh is that you do continuous verification of people during screening and then you keep probing them for anomalies. You ask them questions that are they're like, "Uh, oh, so you say uh you're that you're from Belgium. Uh, and you ask like a politics question about recent politics and if they can't answer it, then they're probably faking, you know, uh, or, you know, there's a lot of ways. Um, you ask harder questions. You have you ask specific questions. You know, that's — delay payment. — Oh, yeah. Do not autopay people. — Do not Exactly. do not pay subjects until you are sure that they're not lying to you. — Basically, what we're suggesting is like zero trust security, — right? Like continuous verification — for UX. — Do zero trust — vetting for participants. — All right. So, this is the uh this is the thing. — This is a money slide, everybody. — I don't know. Uh so this is the thing that uh when I was trying to put this together uh I was like okay like let's now talk about what uh is actually happening in terms of AI and UX research. So you have these two axes essentially fake versus real users and fake versus real researchers. And so you can have something in each one of those quadrants. And there are in fact startups so uh and what I will say is that actually simulated uh and synthetic users actually have and personas have a place in research for certain things. They can identify basic requirements. behaviors. They can uh also give you access to uh some perspectives that are really difficult to recruit for. You know, there are some groups of people that you're just not going to be able to find uh very many of to talk to. The problem is that whatever you learn from these models is guaranteed to be wrong in some details. You know, the way I'm kind of describing this is like these models give you anti-alias reality. they give you like reality with the corners uh with the corners

rounded off. So um uh let me talk about uh researchers. So um synthetic researchers they can ask questions for you. of people for you. But then you are back in that place where you have to trust them to ask the questions that are going to actually generate re uh answers and that the people they're talking to are real people who are generating real answers otherwise you have two AI models and you might as well just like do simulated research. Uh so um uh one of the things about that also about both is that the models are trained on specific corpuses and there are certain groups who are very well documented in Reddit whose opinions and views and behaviors and preferences are very well documented online. Great. — Cool. — Yeah. But there are lots of people whose docu whose behaviors are not — and those are the behaviors people that you actually want to know about. — So let me give you an example. — I work with um largely pol healthcare policy specialists who have excruciating amounts of detailed knowledge about very specific parts of US healthcare law which is notoriously painfully complicated. You would think that these people would be because they're online, because they're in a wealthy industrialized country, that there would be a whole lot that you could ask an LLM about how they do their work. Not so. They do all of their work behind government firewalls. Any conventional LLM will know nothing about what they do, how they do it, or how they feel about it. And that really gets I think to the heart of one of the big I don't know you don't know what you don't know that is very risky about the all AI method is that we have a lot of assumptions often about what who populations are what they're like and you know in the past I've sort of used an example here of like I don't know Mongolian tour guides we actually know a guy who's a Mongolian tour guide his life experience isn't necessarily all that well documented because most of it's in frankly Chinese and Mongolian and if and Russian and so if you have a English language LLM it's not really going to get to it but that kind of exotifies the problem right like that makes the LLM problem sort of out there whereas my problem like the normal LLM problem is in here in the United States in people who you probably know if you public health class It's not like somehow you're going to be safe to use AI if you're working with US-based populations or highly internet connected groups. It may be that you're actually asking about a part of their lives that is simply walled off from the corpus of LLM generated personas. — And this is why you need ground truth. And uh we're calling quote unquote traditional user research where you actually talk to people. Maybe you even go to where they live and you talk to them. You pet their dog. You uh sit down with them. You learn about uh you learn about their kids. You learn about things you don't care about but that are important to them. You know, this is why you uh you still have that because that helps you test your conclusions. Like if like it's an external validation of research you may have already done through other means, which we absolutely encourage you to do. Like don't do the most expensive kind. It's expensive. kind of research first, but do it anyway because you're going to need to uh identify whether all the other research you did is valid and you're going to need to identify what the edge cases are and you're going to need to collect stories. And so stories because of organizational power. And we're going to talk about that uh I think in a few minutes. — Yes, we are. — Yeah. So um what we recommend is that from a pragmatic perspective that like say you're in a startup, say you're doing uh a completely new thing, you're incredibly busy like everyone's working a 12-hour days, you can in fact use these automated tools and we recommend that you do, but we also recommend that on a regular cadence you go and talk to people. You got you

prepare for it. You prepare for it by making uh discussion guides. looking at the results that you have. You prepare uh for it by looking at how you or by reading about how you look at things. There's plenty of stuff out there like this is why this is a why talk not a how talk. There's so much out there about how to do all this stuff. It'll of course also be in our book and of course when our book comes out it will train the LLMs and so you'll be able to get it out of cloud without you having to read our book. However, it's um this is why it's important. This is why you do that. And so the power dynamics comes in because uh you need to have because the ground truth has meaning in an organizational context not just in a research context. All right. Is this you? — Yes. So we're going to talk a little bit about how UX research role how the role has changed within the broader tech industry. And to do that, we have to talk about organizational power. And I want to be clear that we're not going to get nostalgic for, I don't know, the Halion days of 2015 or whatever. But we do need to acknowledge that the world has changed. And that's a big part of why I want to do this, why we wanted to rewrite the book is that it's not new. You know, there's always a temptation to, you know, the opposite of nostalgia is kind of presentism where we sort of believe that everything that happens today is new and unprecedented and revolutionary. And that's just not true. Maybe the technology is very new, but the experiences that people have as they watch their jobs change, the emotions they feel, the decisions they have to make have happened before, and you can learn from them. auto workers in the 1980s when robots came around. Right. — Exactly. And so this quote comes from the automation of knowledge from Lewis Mumford. 1970. The ideas date back to the 1930s and they were probably updated in the 1950s based on the then new definition of cybernetics. And if someone here is a symbolic systems major and that is the genesis of the symbolic systems major at Stanford as we have now. So you can trace a bright line from this to this — and so it's happened again, right? It's happened before, it'll happen again. And so what we try to say is that AI is a symptom. It's not a cause. And what I mean by that, what we mean by that is that it's a symptom of a longer evolution of who has power, whose job is seen to be automatable. So when we say power we mean the ability to shape the world either through what happens or through what doesn't happen. So you can see power in what relevance is treated like how is relevance defined what skills are seen as salient or even comprehensible. Back in the day, I had a job working with a bunch of computational economists in a research lab. And my background in anthropology and social science was vaguely comprehensible to them, but it was not seen as salient to them in any way. So if I raised a question in a seminar or I asked about their product testing plans, I was met with like blank stairs, glass wall, and I remember thinking, "Oh, oh, I'm I have no power here. " Like I am a representative of a discipline that does not count. Okay, cool. Um, — you still have to have that job. figure out how to make a change. — Yeah. like I still have to figure out something to do but it's not going to be through this route of trying to appeal to people who do not see my work as meaningful. Then there's also reward and rejection. So as I said like what gets you rewarded depending on your definition of reward. What how do other people get rewarded or rejected? Who gets laid off? Who gets a bigger salary? Who gets to make decisions in sprint planning? who gets to speak here. Those are all forms of reward. And reward for the economists in the audience, I know there's one is revealed choice, right? It's revealed preference. You can see it happening. This is not like up in the sky imaginary power dynamics. It's very real. — And what I wanted to point out here in uh in this chart that I actually had GPT make for me. — Thanks, Mike. the data was out there uh in layoffs. fyi

FYI, but um what I want to point out is that like okay, yeah, there's a lot of layoffs right now. We hear a lot about layoffs, but actually the layoffs started before CH GPT actually became a big deal. Like the companies were already laying people off and this was like there was like a previous hump a couple years earlier uh during the pandemic. But like what we're seeing now is essentially like a justification of AI as you probably heard as a reason to just continue business as usual for a lot of these organizations. — So when you see layoffs and you hear someone say AI did it. — Yeah. — A obviously never trust what a CEO is telling you about the their company in public. Like just don't. Right. like they're talking they're doing this for the stock prices and they're doing it for the business analysts but also the data contradicts it and what that means is that the layoffs are indicative they're a symptom of something else that's going on and it can be multicausal like we know that situations in the world have many factors that play into them but the story that we're being given of AI as a sole cause or business efficiencies is just not And [snorts] however, they're really scary. They're really unsettling. There's a lot of panicked discussion on LinkedIn and Medium about the death of design and the death of UX. And that is completely understandable. You know, it happens every time jobs get automated. And historically, let's talk about historically it was robotics. It was manual labor. And so jobs that were the 3D, this is what I learned in grad school about what gets automated. — Wait, how many people know this framing? The 3D robotics framing. Okay, so this is a classic uh robotics and industrial operations engineering kind of consideration. Like where do you stick a robot? That's where you stick a robot, right? you stick a robot when you have a task that's at that's where it's — where do you replace a human right get rid of humans and — can I explain the upper right hand corner which is actually I did that and it's really comp I realize it makes no sense to anybody — okay sure — so what I what I want to say there is that what unless you were going to get this is that what we're seeing right now is that we're seeing a lot of Um like the perspective that we are taking on this is that there are roles, there are activities, there are titles and those are not the same thing. And so when you hear about UX, uh research that is not necessarily associated with a specific person or a specific title and um what we're seeing is we are seeing uh automation taking away essentially the activities for some of these things And uh but not necessarily taking away either the roles or the tasks and moreover not taking away the people because let me see uh because the responsibility of a person in a role is much greater than a specific activity that they do and their role in an organization can be much different than this that activity. Right? So for example uh UX research came up with this I find incredibly inf infelicitous acronym PWDR person who does research. I know. And they had to come up with this acronym because there are so many people in startups in medium like all over the place who do who are very clearly doing research as researchers define it. The systematic investigation of the world and the derivation of knowledge from it according to general theories. People are doing that. They don't call themselves researchers but it's happening. And the same thing is now happening to engineering. There are just all sorts of people writing code who would not describe themselves as engineers, who are not called engineers, who do not want to be called engineers, but they're still doing it. — And the same thing for interaction design. — Yeah. PE right now you'll see people are trying to scoot around the title thing and they're going like, "Oh, no, we're builders. That's what it we're builders. — We're technologist. — We're not engineers anymore because the AI does that. We're not UX researchers

because AI does that. What we are is we're builders. Like — fine. — Um I think a point that I wanted to make here is that uh the thing that AI does repeatedly or any automation technology does is it takes a job that is repetitive and that is considered to be um low value. And what that is it shifts and it moves that to uh to a machine. It's always done that. It's done that since the you know what whatever uh Manchester cotton mills in the 1840s whatever like if what you're doing is uh repetitive if you have to do it multiple times and okay thanks an organization can wrap a name around that they will automate — and so what we're leaning towards now is this 3E framework and you know take this with a grain of salt We're evolving this, but I think it complicates. It's a necessary complication to the dirty, dangerous, dull framework because what we're seeing now is essentially the automation via AI of knowledge work that is seen as expensive. So, you have to hire a separate researcher with benefits and everything to do this work — and it takes what? Not 45 seconds. — It takes weeks. You might have to wait for results. unthinkable and external. That is to say, do I really need to hire this external person? Is it really important to product development to have a separate person? Can't I just is it really like do I really need to have a separate person to be a good product manager? Maybe I can just do this thing myself cuz it's not deeply part of my job. It's, you know, kind of secondary and so extraneous, right? you can separate it and you can kind of pull it out and plop it down into automation because it's seen as repetitive or lower value. The question then becomes though, and this is why we keep talking about power dynamics, who defines extraneous, expensive, and internal? I feel pretty darn cheap. I feel pretty central. I think my work is not repetitive at all, but I'm not the one who gets to define in terms of a large organization what is or is not central. So instead, what we see is more user research. Paradoxically, lots and lots of research is happening, but fewer people might have that title. We have fewer kind of disciplinary specialties. So, if you've ever looked at um somebody who worked for Google, if you've ever looked at Google, they have these amazing like career ladders that are like qualitative researcher six or three. And that's what I mean by like a disciplinary specialty versus a broader shared organizational competency where you might expect that multiple people share the quantitative research responsibility. It becomes a culture rather than a title. And then of course as I said PWDRs all these different places where things that to me are recognizably research are happening and I would hope that they are happening better. So what we see again thinking about our book was that in the second edition of our book what we saw is the rise of discount methods right like — yeah first edition do everything by hand second edition the expectations are that you're going to be able are you going to do it faster and you're going to do more of it. So you started to see discount research, — gorilla research it was called sometimes. And then you see around like we're sort of this is all very handwavy. — Yeah, this is — but like think about it. You have the amount of time up top like 2 months 60 days and it took a while right in our first around the early 2000s. the time goes down via the discount methods and the expectation that you're going to send ah maybe a month 20 days. Now what we're getting down to is this expectation that AI can do it maybe through one of these synthetic users or synthetic researchers and it will take days if not hours to get the same results. All right. — So, this is sort of we're going to try and wrap this up quickly so we can have a conversation and this is the end of the like are there still careers in research? Y'all are mostly students. Maybe you're expecting to have a career in research. You're probably expecting to have some sort of career, I hope. And this is this our book is for you

even if you're not going to be a researcher. So, we want to have a conversation about what it means to do this work going forwards as a professional, not a hobbyist, not someone who does it for funsies on the weekend, but someone for whom this might be a component, if not the principal component of your working life. And let me take a step back though cuz I feel like an old. So, I graduated from college in 1998. And back then, the traditional career pro procession, well, there wasn't really a career. Let's be, let's be honest, we were making it up. But for about 10 years, the career progression was kind of guildlike, you know, very artal. You'd begin and you'd re you become an apprentice under a mid-level manager. And the mid-level manager was like a journeyman, right? Who supervised and taught groups of apprentices under the head of research, someone like Christina here, who would be kind of the overseer who would guide and lead the entire culture of research and set standards and coordinate work. Or like Mike was, you could be an independent consultant. You'd hustle. you just hustle, hustle. And a lot of the advances in our profession were made by people who hustled. So I don't disrespect it at all. Today, okay, like this is very again, this is like a little handwavy. I haven't done a like a statistical analysis of glass door or anything like that. Um but through a lot of interviews and a lot of talking to people who are kind of at the head of level what we see is that entrylevel designers/ress researchers are people who wear multiple hats. They can do a lot of different things. They're a generalist and there are fewer people obviously who are specifically doing research. product managers have taken over that like midlevel journeyman tell the apprentices what to do role and — yeah and so I don't know if we talk about this later I don't remember so product managers the title is an interestingly ambiguous title UX researcher gives uh an organization very specific bounds around what that uh role is — quantitative researcher Number five, — your job is you are talking about user experience which is a part of design possibly or marketing possibly and you are talking about research. So you are doing research like you're going to be product management has none of those. There are no prerequisites for calling yourself a PM and thus PMs have taken over the power structure of Silicon Valley. the title of PM has essentially subsumed all of these other titles not just in terms of the kind of uh work that uh work that they do but in terms of the power that they have within organizations. So one of the uh like if you want like a small piece of advice call yourself a PM right cuz — no one can prove you wrong. — Yeah. No one can tell you you're not. — This is what I say to designers. — Do the user research because that can also be part of the PM umbrella. the PM bubble will burst at some point because there whatever it always happens. Uh uh and somebody else like the builders will take over whatever uh and so but uh just think about that as like what is it that you are doing when you are in these roles and PMs can do a lot and should be doing a lot of research even if they're not. — All right. So this is sort of a set of maxims that we can leave you with no matter what you call yourself. Like we genuinely do not care what you call yourself at this point. One, we've talked about this that ground truth is like gold, not like flowers. And that's why we had our mortar and pestle, right? like you have to mine it, you have to refine shine it up, you have to make it pretty. Like it's not just like a flower. You don't like go out and just like pick a research insight up. And that work that you put into it is um really meaningful, right? Like I'm going to skip actually over and say like one of the great fallacies we both believe of a

lot of these AI enabled research tools and also a lot of AI design enabled design tools in general is they sort of think of the deliverable as um the outcome right like oh you're you made a wireframe you're done that's what your job is or you made a research report that's what your job is that is not actually True. If you think about the actual power dynamics, I did my dissertation on this in technology and sociology. So, I'm willing to fight for it. If you are in fact a working professional, you know that just like sending out your report into the void does not do anything. You have to think of a report or wireframe as a kind of contract between all the people who helped to make it. It's like I'm going to look at you and I'm going be like, "All right. " Like, we looked each other in the eye and we agree that this journey is an important journey and that there are some bright spots and there are some pain points and we agree that the worst pain point is that one and we're going to work on that one. And we both know that because you made this, you understand what's in it. The deliverable is essentially a contract between two people that says the person who's delivering it actually did the work and knows the information. You can get an AI to generate a perfectly reasonable deliverable, but then it is not a valid social contract between you or whoever makes that deliverable and whoever receives that deliverable because there is not this uh relationship that says this is actually based in ground truth. — So — ground truth is what gives the deliverable — its organizational power. — I saw a question. Yeah, I'm curious about this because I feel like before this point it was a lot harder to create the deliverable to create the outcome — without that implicit understanding like even if maybe you were the head of a research team somebody in your research team had the understanding and in theory you would know who like so but now you know so that contract came was it didn't wasn't necessary to question that before. — Exactly. Because it was hard to make — that meant that when it because it was expensive that essentially meant that a lot of work and thought had gone into it and that was the validation of that uh of that product. So what you would do like in the oldie days is uh of UX research is that you'd put together like a big binder full of all of your interviews and all of your stuff and you would deliver that to your client and there would be an executive summary at the front. They would never look at a single sheet of that binder, but that would validate the executive summary that you made because that w because they knew that if you had gone through all the work to collect all this information, they would believe you — that was uh that the conclusions were done. Similarly, if you make a design, if you're a UX design, if you are a visual designer, if you produce a design that uh clearly has to be done by hand, then that means you stand behind the decisions that were made when that uh deliverable was uh was done. If you have something else uh generate that, there's no guarantee on the other side for them. And this is not necessarily going to be a conscious thing on the people that you talk to. It's a power dynamic thing. They're not going to uh give it as much value if they know that it was generated by an air — or I mean quite literally. So I do a lot of facilitative group exercises and yes it's research because I work with a lot of people whose work practices are very poorly documented but also it is there to establish a relation of gratitude and trust. I talk to you, you tell me about your work. We have again have that like eye to eye. Then you get to see that work even if you didn't make the deliverable. you know that I was there. I talked to you. You see your words. You are now a part like quite literally a part of this deliverable and you have to take some ownership of it. Even if you don't feel like you're the owner, you've been incorporated into it. And that's a really valuable power hack, right? You are showing people that they have been heard. And if you have, it's not even if you have an AI generated deliverable, just the suspicion of it

starts to creep in. Was that a real person you talked to or not? Are those real quotes? Did you make them up? Like so much of what we did as researchers, for better or for worse, depended on this aura of authenticity and reality. And when you disrupt it, you start dislodging some of the credibility that your conclusions have. It's like imagine if I showed someone a log file, like just one of those, I don't know, Google Analytics generated files, and they we talked about it for hours and at the end of it, I said, "Hey, you know what? I had chat GPT make that for me. " Nothing in it is real. That is a massive betrayal of a kind of work social contract and it's something that anyone could do at any time. — So I want to talk about — yes — this a little bit. So one of the things that we recommend to everyone is that user research is the practice of understanding the perspective of someone else. The perspective of uh another person what uh success looks like to them what failure likes to them. uh it looks like to them what they want to uh what they want what they don't want. Do that with your with the internal stakeholders of the companies that you work for. Understand use the re same user research tools uh conceptually that you would use to understand someone outside of your organization to understand your own organization in order to understand how to be successful in within that organization. if like what does that uh senior vice president need in order for them to be successful? Can I using uh in my using my position using the work that I'm doing, can I help them be more successful because they value this? And this is how you get uh past the 3E uh uh problem. You are now you are not extraneous. You are not external. You are central. — And that furthermore, I know you did this. I'll tell a story about you. — Mike did this hack when he was uh working for a massive Scandinavian company that shall not be named who communicated internally exclusively in giant powerpoints. — Yeah. No one ever delivered the powerpoints. They're literally just mailed to each other. like everyone just mail emailed each other PowerPoints all the time. — So, so Mike realized that his only way to talk to a bunch of these very powerful people was to make PowerPoint slides that could be easily copied and pasted into these giant documents and that were kind of infinitely reusable and remixable. And so he just sat down and designed slides that could be copied without attribution to him. — Yep. Uh if they were stolen, awesome. That means my ideas are out there. Uh that and it'll for the most part come back to me. So great, go steal my PowerPoint, [snorts] — right? — People did. — Yeah. I've done something similar with um emails in organizations that only do emails. You have to make emails that are easily forwarded and, you know, digested into the words that people need to hear. It's exactly the same technique that uh we recommend when you're interviewing people. Use their words when you're talking to them. Like if someone uh if uh someone is calling something a different word than what you would call it, just use their word when you're talking to them because that uh that keeps the uh everyone focused. That keeps the context on uh the conversation. that helps them understand what you're talk what you're talking about. You do the same thing when you're uh communicating inside an organization. Don't try to make them understand what user experience research — terminology is or methodology is. — They do not care. — Yeah, — they really don't. It's sort of a terrible irony of being a researcher. I am obviously someone who loves research. I got a PhD. Nobody wants to hear the names of methods. Nobody cares about my methods. Like they want to know that they happened and that they're legitimate and I'm not making this stuff up. But I researchers are the only ones who want to hear about other researchers and how they work. Instead, you have to, as Mike said, like speak very directly to the needs and goals of the people that you're working for. And I know this sounds basic, like I feel like I'm giving you this advice and you're like, "Yeah, yeah, yeah. " But it's really hard in practice and it's especially really hard if you're using large language models because the large language models do not understand your organizational context. They are deliberately generic. So unless you have trained your own large language model on an incredibly deep corpus which you can like if you

worked for like Amazon you could totally give enough documents enough corporate speak documents to that LLM to train it to talk like an Amazon executive. Most of us I don't know Stanford but like most of us like if you work at a startup or like a niche thing like I do with healthcare policy there are not enough documents to train a normal model to talk with the experts I work with and when I have tried to get frontier models to do that they are extremely poor quality and I don't blame them because that's the corpus. So your job using these skills is to at least make sure even if you draft something with the help of whatever model or tool that it is still speaking the niche cultural vocabulary of your specific organization and its specific context because getting back to the technological capabilities you cannot expect sorry I'm going to yell but like really you cannot expect a large language model to talk in the correct terms for your specific moment in time and place. Particularly if you are designing something that is so new it has never existed before. There can be no expectation that the LLM will be able to predict that because it doesn't exist yet. So if you're inventing the future, please do not rely on a corpus built from the past. The end. — Good. — Bing. — Oh, and uh thank you. — Uh thank you very much. — Questions. We end with more Mumford always. — Do we have time? — Yeah, you got seven minutes questions. — Awesome. — Let's go. — We rushed through so we could get through the questions part. — All right. Oh, wait, wait. I have one more slide to show. — You do? — I do. I do. I used the wrong presenter. Okay, fine. — All right. So uh I with CL cloud vcoded a uh synthetic open source uh user interview tool that uh essentially is written entirely in Python. So it works on everything. Um so what this does just so you know it's very very basic. There is a set of personas in one document. There is a you a research essentially guide a set of questions for the researcher in another document. And then there is a list of um essentially uh AI models. And if you have API keys to those AI models and really it just uses um uh it what you can do is you can uh have one AI model act as the — interviewer and one AI model acts as the interviewee. It randomizes who does what uh every session, compiles the answers and then summarizes it at the end. So you can essentially run a com, you know, it's a very basic fake uh set of uh external interviews. And what I recommend you do is do not pretend that this is a real substitute, but it's a toy you can play with that it gives you an idea of how what these AI models do and how they do it. So there you go. — God, I'm horrified. — It was so fun. Uh so Gartner like a little while ago came up with this very provocative like claim that in like 2 years 80% of data that is going to be used to train like a lot of the foundation models that many companies are trying to build is going to be synthetic. I was wondering how that if we believe at least some part of that trend hype that they're kind of pointing to. How does that fit with some of the things that you were saying? — That is a amazing confirmation. basically of what we are I like to think it's an amazing confirmation of what we just said which is that if the models eat themselves you're getting further and further away from empirical validation of core bets. So I'm assuming that a lot of people here want to make products. I work with people who make policy. It's actually not that different. You're [clears throat] making a bet, a very expensive bet sometimes. And people like to think that code is cheap, which is sometimes true, but reputation is expensive, — and runway is expensive. And if you make a really bad bet based on totally synthetic data, talking to totally synthetic data, you've essentially just um Oh gosh, I don't know how to say this without swearing. You've eaten your own

BS. Yes, — enjoyed the talk. uh given that issue and also given the diffusion of the UX researcher task — among the many roles that you've just explained to us. How do you convince somebody of the quality of a piece of user experience when you research? What how do you do that? How do you tell them it's not based on this and these people don't really know what they're doing because they only do it part-time? you know, how do you validate that value? — I mean, I think it's an excellent question and and it's like it's yeah, you know, it's like how do you justify your work to an organization that may think that it is automatable? I think that that's the question. Uh I think that's a that's a big question. I think um honestly uh that's where going to going out and petting people's dogs like collecting some ground truth is actually uh has a lot of rhetorical power rhetorical vibe like look I took a selfie with this person in their kitchen right like thus when I tell you that Suzie actually does this thing that you thought no one would ever do. I'm telling you that there's at least one person I can tell you why she did it. And [clears throat] then that gives you rhetorical uh — it actually gets back to some an amazing piece of advice given to me when I was just starting out in research by uh Genevieve Bell, who some of you may know about as a very well-known research and research advocate. And Genevieve told me that I should always take a picture of myself doing the research if possible wearing a brightly colored shirt so that I stand out from the background and make sure that it is in the presentation because that is the you know like in crypto it's like the proof of work. — Um Jenny also did like she was she has some amazing so she was at Intel for years. She has some amazing hacks uh for getting herself uh her uh not herself but really like her work uh seen. Um and she made a lot of uh she had a lot of ended up having a lot of influence at Intel. She was only one time that when she was talking to uh essentially sea level people at Intel, she walked over and she was making a point about uh I'm not sure what she but there were stickers involved. She like picks up she like brings some stickers that they essentially talk about and she picks up a sticker and she's walking along and she walks over to the CEO of Intel and she's talking to him and she's peeling the sticker. She closes her laptop and she puts this his laptop. She closes his laptop so he has to look at her and then she puts her sticker on it and then she continues talking and walks away. — like uh like that is essentially someone who like that's — for power. — Exactly. I don't recommend it in all [clears throat] situations, but in that situation — like it got the CEO's attention. — Mhm. [clears throat] — Uh okay. — There's a question over there. I was just going to ask to what extent do you think finding ground truth from the communities you need to hear it most from will become more and more difficult because they are guarding said ground truth so it doesn't end up on itself I say that from a perspective I'm working with a range of tribal nations that are pulling back from just projects that are incorporating AI you know there are misperceptions there's a misunderstanding for like benefit but also real concerns about data — and I think that that's so it's funny you should say that because the sort of the gold metaphor of ground truth is a very ambivalent one — right like mining is an extractive industry it often destroys the landscapes that it pulls from and that is in some ways the history of user research is that you come in you ask these questions companies benefit from the lived experience of the people who have spoken to you and then they don't actually get a penny or even credit in the final product. And I think tribal nations are especially interesting because there are real sovereignty issues and particularly around certain kinds [snorts] of genetic or cultural information. And I think in so far as that makes research harder, I think that's a good thing. Um like friction is a good thing.

There is no law that says somehow or no moral right that says that you should be able to ask anyone a question and they have to answer it and that somehow they are unethical or wrong if they say no right like this is sort of the foundation of trauma informed research where you cannot go you know one of the reasons why I would recommend using synthetic personas is if you're planning to do research let's say with like people undergoing cancer treatment perhaps it's not a great time to take your untested uh interview protocol and just try asking them about their emotions. — So, one of the So, a couple of thoughts. One of the important roles that UX or whatever whoever is doing like people who do research do is that they learn to understand the audiences and how uh and what helps the audience trust the organization to be able to deliver certain kinds of services that it wants to deliver to those audiences. that is literally uh the responsibility and for difficult to reach audiences it's hard but if the goal is to create something like basically if the entire business model or organizational model is based on delivering something to that uh that population that is critical knowledge in order to be able to be successful at the thing that the thing is trying that you're — right so — I think that's going to have to be the final question Thank you and thank our guest for giving an amazing talk.