How AI Is Changing Google Search and SEO

— Hello and welcome to a new episode of Search Off the Record, the podcast where we take you a little bit behind the scenes of Google Search and hopefully have some fun along the way. Well, you probably have seen AI features in Search and uh whenever I have to talk about AI features in Search, I'm really, really happy that I got to see a presentation at Search Central Live in Zurich last year and uh I think it's time to open this up to more people. So, I invited a guest today. My guest today is Nikola Todorovich and um would you like to introduce yourself, Nikola? Yes, uh thank you, Martin. So, I have uh joined Google about 15 years ago over here in the Zurich office and for all of that time, I've been uh a part of the Search organization, what used to be called Search Quality, nowadays it's Search Intelligence. And I've been a part of the team that's called Safe Search and for the last several years, I've been leading that team and also in the last couple of years, I was uh more involved in the ecosystem work, uh working together with you, with the folks from Search Console, Google Trends, et cetera. And so, have some more experience on that front as well. And we pushed you into the cold water of our stage in Zurich as well and you had a really, really cool topic. You talked a bit more about AI in Search. Would you like to tell us what led to that talk and what was the thinking behind it and what you want people to take away from that? Yeah, well, clearly, AI is the topic that everybody's talking about right now. A lot of people are wondering how is Search evolving and uh what will be the future of Search, uh the future of AI, et cetera. And from that perspective, I think it was valuable to bring that particular presentation. Now, the presentation that you referred to is showed a lot more things before the new wave of AI came in. I think that was the context that I felt it was helpful to present to the audience over here. Yeah, because I think everyone is talking about AI in Search as if it's a new thing, but it has been there behind the scenes, so to speak, before that, right? So, what makes these AI features that people are using now and that are progressively enhancing the search experience for them so different from the features we had before? Would you consider these new features revolutionary and completely different from what we've been doing so far or is it more like an evolution of what we have been doing in the past? I think the way they are being used and I think it is a revolution that we're speaking right now, but clearly, in the whole process, there was like small steps, but if you compare Search now and Search 10 years ago, it's a very different product. So, I would say yes, this is like a big step change and it is absolutely changing the way the users are searching. So, if you think about it, any feature is changing in some way. For example, if you bring like more images, videos, et cetera, then it is bringing this kind of experience, so people are going more to image search. For example, when we added what we call the image universal uh blocks on the main page. Now, the this new wave is also changing the way the users are searching because they are uncovering that Search can actually answer to more complex questions. And for that reason, we do see that user queries or if you call them prompts now, so they're getting longer. They become uh more detailed and the average query length is growing. So, we do see the new traffic and this new wave of traffic is a consequence of users being able to see, "Aha, there is something new I can do over here. " So, that's from that perspective, it is revolution, but it is obviously a bunch of steps in between that happened and have been improving Search all the time. Can you shed some light on the steps in between that you think are outstanding and probably have kind of paved the way for this? Kind of before I jump into that, maybe it would be interesting to tell you a little bit about the process how the changes happen in Search. Oh, yeah. — And then I can add kind of what are the particular changes or that reflected this AI revolution. So, in principles, Google Search is a huge product. It has a lot of different components and, you know, all like you, Gary Illyes, uh John Mueller, and others like have been talking about this like it starts all start with web, with the crawling

indexing, the ranking components, and so on, the new features on top, et cetera. So, we have thousands of changes in Google Search per year. I'm not sure how many, but it certainly in thousands. We know that because we're tracking all of them and we're evaluating all of them. We're measuring because the key point is, yes, we have new technology, we have things that are, for example, we know problems that happen. Like very often, you know, the changes that come to Search are either consequence of the new technology that's coming up as we see, "Oh, let's use this new technology because it certainly will bring us something some improvements. " Or alternatively, we see how there is a problem. I'm typing this query, but I'm getting this result is not optimal to see this. And when we do this, we, as engineers on Search, we are making a kind of an experimental version of Google Search that has something new, that has something different compared to the production version of Google Search. And we need some way to tell us, okay, what is better? Because we we're not just launching these 5,000 changes because some engineer or some product manager has an intuition, "Ah, this probably will be better, so let me like add this thing there, this thing there. " No. So, we have to start and see, "Aha, I'm I have to like build a prototype of the new version. " Thankfully, all the infrastructure at Google is really amazing, so it help us like run this very quickly once we have a good idea. So, we can like build a new version, run a comparison with the baseline, which is the production system, and we run those things called side by sides. So, you're getting random user queries that will see a difference between the production and your experiment. And we have published the guidelines that help human raters review those changes, those differences between the baseline and experiment. And out of these reviews, out of these human reviews, we're getting statistics. And this statistic is telling us the experiment is better than the baseline. And if it is, then, well, you would think, yeah, let's uh submit it and commit the changes and like go launch. No. We will have a uh something called launch review and that is a uh process where we are where the engineers are talking to the leads who have the decision-making power in the end and make a call, yes, this is better. And sometimes it can be that your overall statistics look like improving, but you're you have a some really bad pattern of losses in your experiment and well, if there is a kind of reasonable way how to fix those patterns, we're going to bring the engineer back and let them fix those patterns and make an improvement. And so, right now, I'm just you know, talking about the kind of uh the standard good old process of the launch reviews and the new experiments and everything at Google in Search. And this process has been going on uh and is still there. So, let me know if this what I was just explaining is clear. Do you have any sub questions on that before I I'm moving to the kind of more AI territory? I'm just wondering if at some point we should break this out as a separate episode because I think we've mentioned both the Search Quality Rater Guidelines and experiments beforehand, but I don't think we've ever gotten such a nice explanation of how the process works and how the different bits and pieces fit together. So, that was really, really cool. But let's take it back to AI now. So, I'm guessing uh the AI features underwent more or less the same process, right? Yeah, absolutely, they do and I have to say, yes, given that uh the world is obviously changing, the competitive landscape has changed as well. We also need to adapt to this new world. However, the like a lot of AI inside of Google has been, you know, developed for years before the generative AI came to play. As I mentioned in the beginning, I am responsible for the Safe Search engineering team. And we're one of the first places where Google was able to comfortably apply artificial intelligence / machine learning uh models directly in Search. The reason why it was not so easy to just apply it everywhere is because these models function like a kind of like box. You don't always understand what's happening underneath. It's a complex set of, for example, neural networks or even the older kind of simpler, even, you know, the linear models are kind of the easiest ones to understand and to debug, right? Because it's not just you kind of can put your AI or ML system into Search and you know, you reap the most benefit from your side-by-side experiments that I just mentioned previously.

And now you will, you know, get to something and launch it. But then you will have problems with that as well because obviously the systems evolve like the searches evolve and so on. And then you will need to debug this and replace it. And like this kind of replacement and changes is complicated. So the more you can understand how these things work, what signals are you using, what signals are important for the relevance, for the quality, for the safety of the results. So you do need to understand the system and kind of the more complex the AI or the ML systems are, then the more challenging it is. But safe search has been one of the places where, you know, you could isolate outside of the main search ranking flow, you can isolate the systems that just do like process the images, process the videos, process the text, and they just give you kind of a signal on its own. How explicit for example, a result can be. And then the kind of the understanding of let's say the 10 years ago or 15, no, it's more like 12 years ago when really the convolutional neural networks came in to help us understand the images better and in many places they were actually already doing things better than humans in understanding images. Then, you know, we could apply this as a kind of a standalone AI system that runs on the topic. And if we have problems, yes, the engineers in the safe search team had the intuition and could run an iteration and improve the neural network itself. But it's a kind of a very isolated space, so you can more easily uh navigate. And then the rest of the search stack has still been on its own and running things. Along the way, there have been various new technologies. So starting with transformers, I think that that's the biggest one like that in the end introduced all the GenAI world. But we were reaping the benefits of transformers on search long before all this stuff came in. And we were open about it. So we have announced publicly uh the systems like BERT, like MUM. And they have been able to transform the search and ranking into a much better place. And again, these systems were built in kind of an isolation as well. Just like this the safe search systems, I think these systems were also built in isolation as a new signals. And these new signals were supporting the whole ranking infrastructure and was kind of one more thing on top of everything else. Hopefully that makes sense. — That makes sense. And I mean, if you look at it, the new AI features are kind of also they are integrated, but they are also somewhat isolated as in like there's an AI overview that lives in its own space. And AI mode is a completely different way of searching. So they are kind of also independent of the rest of the search stack even though they use the rest of the search infrastructure and search stack and ranking systems, right? Would you say that's the case as well or is that completely different from previous systems? Yeah, let's maybe start with AI overviews because that's where I think this holds the most still because if you think of AI overviews like this is your normal search with perhaps a few fan outs. Huh, I just introduced a new term. I probably shouldn't. Please explain that. I think it's for, you know, the experts out there. I don't think it's like probably many of them have heard about it. But anyway, a fan out is when you have a your own search query but then we might identify some additional search query that will yield the results uh that can be relevant for your original search query as well. And then we have like we can fork and in parallel do the retrieval for multiple search queries that can all come back into one original more complex query that you gave in. And so, as I initially said previously that we do see longer queries. This is also we can help in understand the kind of more directions of what you were initially typing. So we launch multiple queries. Now we get all of this retrieved back. And then AI overviews is combining an interesting selection of these uh of these results and making a summary from what it can see in those results. So in a sense, the whole retrieval system, the whole ranking system is the old style, uh the old school. And that one is the AI overviews is a feature that uh stamps on top of this and operates on its own in this I this is the kind of the isolated space for the AI overview where it combines and it's really fascinating what the

language models have been able to do. But yes, it can combine like text that it sees on these sources on the snippets, the titles, etc. And additional uh context it can get out of those pages and then make a really nice summary in the end. And I really like that. And I think that also goes back to what you said earlier that the behavior changes and queries get longer and more complicated. Because I remember in back in the days when, I don't know, the world was still monochrome or something, I when I searched even on Google, I searched kind of keyword like restaurant vegetarian Zurich. And then over the years that became more conversational as in like vegetarian restaurants in Zurich, which is already a change. And now nowadays I ask questions or I type in queries that are so much more vague and I still get usable uh results like based on dietary restrictions which restaurants would you recommend now for a lunch in Zurich? And then you get like a bunch of stuff and it works because of these fan out queries. It asks like a bunch of queries that I don't have to ask myself anymore to get to the right result. And what I find myself doing is I'm asking questions where I don't even know what a good question is. Right? Beforehand you would sit in front of Google and think like, "Ah, how do I even look for this? There's an effect in, I don't know, let's say like there's a physical effect and I ah what was the name of that? Uh Mm. So you would try to like find the name of the effect first and then Google for the specific effect once you had the name. And now you're like, "What is the physical effect that makes water glow when there's radiation there? " And then it kind of figures it out for you. And I think that's one of the possibilities of features like AI overview, right? So from AI overviews, what was the motivation and the idea behind then going further towards AI mode? Yeah, no, I I agree completely. These are the these are exactly kind of the nice examples of the way how search has evolved with the AI overviews and eventually also AI mode. But all the capability of understanding your intention with some vagueness or I mean even if you're more detailed, yeah, I want a vegetarian restaurant that serves falafels and that has like something like you should be able to get this or that's open now near me. Like all the kind of context that you're getting it. True, I didn't think of that. But yeah, even if you have like more details, you now get better results. Yeah, so either if you have vague like a query or if you have actually more details, so you both of this uh seems to work better. And, you know, clearly this doesn't stop there yet because what we're seeing with the large language models, they're able to gather a lot of information on their own, right? And so they're able to like things like uh what is the capital of France? You don't really need to kind of do the search for it, right? Like so this is kind of one part of like solving parametric uh memory of the model. And so AI mode is able to communicate with you in um obviously it's like even longer queries or longer discussions because it also enables you to do the multi-turn thing. And I mean it's you have like different tools that do all that, right? Uh so like with Gemini being like the Google's version, but uh obviously others like ChatGPT, etc. have been there. And we do see that users like that. So the users like the conversational aspect, the user like to communicate longer and so on. So AI mode is kind of search's answer to that. And we have also seen obviously not every user in the world is going to some of these chatbots. And obviously AI mode is kind of a part of search. So like the users of search might actually want to use that and see how it's like. And you do have also the option to transition from the AI overviews to AI mode if you want to kind of explore more and have like a longer conversation and more details. So I think it's a overall really nice addition. And I like get myself like many times entering query and search or maybe directly into AI mode or like going to the AI overviews and say, "Mm, maybe actually I want like a longer conversation. " And then I'll go to AI mode. AI mode is also still using the search, right? So it does have its own fan outs. Uh it does have the uh linked results and citations as well. So it is kind of in essence still based on this uh kind of standard concept of how we do things on search. But it on its own it has a kind of a bigger well, like the infrastructure is new and like all the it has kind of bigger

ownership or like it's no longer an isolation of it. It's like the AI mode is kind of it runs on search, but it's also has like a bigger platform for its own uh I'm I'm still processing the fact that yeah, of course it works in both directions. It also works with like if you have more details. And I just like the ability to have multimodal search and I think AI mode just adds to that really and that's pretty cool. But one thing that we keep hearing from the ecosystem pretty much at every event we do and it's everywhere is how do we make sure that with AI features being part of search now that the ecosystem continues to thrive. And I think that's an interesting challenge, but also there are like lots of opportunities thanks to AI features these days and I know that we as Google try our best to go on this journey together with the ecosystem. But how do you see it from your perspective? What is it that we do to make sure the ecosystem thrives with these new features. Yeah, the ecosystem impact and like uh I think as you said I've been on two three Search Central Live sites twice in Zurich, once in Madrid. This is uh clearly one of the key question and you see them a lot in the on the social media as well. And uh I don't think there is like a magic wand that can clearly give the guidance okay, what do I do now? Like what would the SEO experts do now in the new system? My kind of guiding principle or my like the way I see here is that uh the site owners I think they do need to continue making sure that their products, that their websites, that their platforms are providing value to the user because ultimately if you provide a particular value, then the users will continue coming to you and they through Google as well. So, if you know for example, you're selling something, you have like a product or a like platform that you have like some subscriptions etc. So, you clearly will if you are providing value to your clients, like they will continue coming to you. We were talking about restaurants, right? Obviously, if you're like putting a menu etc. So, yeah, well, the users will eventually come as well to either your restaurant so they will like go over and see. So, in the AI centric or AI oriented system, I think those kind of bringing the value still continues. But just like in kind of the previous evolutionary or revolutionary steps like on how the media has been disseminated thinking about the newspapers, the radio, the TV, like the internet, all the stuff like all these things also kind of remain to be in this world, but people needed to continue providing value because if you don't provide value, nobody's going to buy your like uh newspaper or book or like nobody's going to listen to the radio or to the podcast. — But so is I think everybody like including all of us like there is a lot of question, right? Like uh is AI going to take our jobs and so on. I think we all need to continue thinking like how do we provide value on top of all of this and in many cases this is about mastering the AI tools and being able to use them in the best possible way. So, kind of this is one of my recommendation to all the SEO professionals and site owners and like the whole ecosystem that they continue providing value, but then do not neglect the new technology and make sure you use it in the best possible way for you. Now, obviously, I don't think we would over here recommend like a the best possible way is to just multiply all the content and just generate because you know, it's cheap and easy and now we're going to generate like it's not going to provide a ton of value. So, so you but if you know, you're using it to like improve your grammar, to improve the style a little bit, make it kind of more interesting and so on. I don't think that's a wrong use of the technology. But then there's plenty of ways okay, maybe the AI can help you better understand your data, maybe understand the competition potentially better as well and so on. So, clearly this is something so we can advise. I find that really interesting because I'm seeing a lot of excitement at the same time a lot of worrying in the community in the ecosystem and I think it is like that because on one hand it democratizes a lot of stuff that has been traditionally difficult to do or just cumbersome to do. At the same time some people have misunderstood whatever

it was that they are trying to accomplish or to provide to be these cumbersome bits and only these cumbersome bits, right? So, to give you an example um when it comes to let's say um writing articles about I don't know, lifestyle or technical topics because I'm more like a geek, so I'm reading more technical things, right? I really enjoyed when people were giving me like interesting details of technology from the days past much older than I am, so I wouldn't have any touching points with technology from the '60s or the '70s and then if someone was like, "Hey, did you know that the displays in old hi-fi uh devices worked like this? " That was a really interesting article. But obviously, they also went and explained what their experiences were with new technology as it came out and as they were provided with samples sometimes even um and that was interesting, but eventually that turned into them effectively how do I put this nicely? Putting words around spec sheets from manufacturers and that wasn't really the value that I was looking for. I'm not interested in knowing how many gigahertz a certain new processor has because I can read that basically on the box. It says it on the box. You don't have to tell me that this is now a 3 gigahertz processor or like it says it on the box. Thank you. And I had like a key moment when I was buying a joystick back in the days for a computer game and I didn't know what force feedback was and that's effectively like you have like a different resistance and it might like move and vibrate the device if there's like any shaking happening in the surroundings and I didn't know what that was and it said on the box it has force feedback. And so I went to someone who worked at the shop and I anticipated them to be like an expert on the topic. Someone like, "So, this says force feedback, what does that mean? " And he literally said to me "Oh, that means that this joystick has force feedback. " — Right? And this is funny, but I'm seeing this a lot in articles and on websites that they're effectively not giving me any context. They're just explaining what I can kind of glimpse and gather from the information that is right in front of me and I think AI makes that easier. Like you don't have to spend as much time to kind of like rattle off the spec sheets into a more readable human conversational form, but chatbots do that. So, you don't necessarily have to do that on your website anymore. But maybe you have tested it and you found it to be particularly good for your use case or particularly unfit and then you can share this insight that AI doesn't have. It doesn't know. It hasn't used the technology. It doesn't know this. But you do. So, you're the expert and I might be coming if you're using your electronics the way that I use them, I might be interested in your opinion. I might not be interested in this other person's opinion because they are using their electronics differently, but that's fine because there are other people who are using their electronics the same way as they do, just not me. So, I think there's still enough space online for different outlets and people and opinions and experiences. But I think we have to increase the level of our content to be useful and interesting for humans from humans to humans and I don't think AI is going to take that away. I bridge that. Yeah, I I absolutely agree. I get often kind of nervous like when I see like the kind of AI style uh reports like you know, obviously internally we want to use these tools like they're helping us. They're helping me understand the documentation more easily. Okay, let me ask questions like you know, Notebook LM has been a fascinating tool uh that can like in a couple minutes explain a complicated thing. So, yeah, I do believe there is still a need for the human touch on top of all of that. I do think we need to understand the capabilities of the tools, but in the end us providing the value, us making sure that yes, we're bringing something to the table and I think that's like where we want to focus. But yeah. Are you using like the coding tools? Interestingly enough, yes. And that's exactly where this stuff comes in so handily. So, the code base in Google is huge because it has a lot of stuff and you've seen it yourself. And just a couple of days ago we stumbled upon a specific piece of code and it was going through like lots of layers of indirection and abstraction to do something and we had a hypothesis where this is going in the end, but we didn't know. So, we asked our internal tool is like, "So, we found this thing that does this thing, but where does the information actually go? " So it was basically like we found this method that tells us how big an image is. But where does this information come from? Does it have to download it or does it use like the image and index for this? And we could have found that ourselves by going like

20-30 minutes through abstraction layer after abstraction layer to finally get to where it's coming from. Or we just asked the system and it's like, "Oh, this is coming from here. " And that was the right spot and we're like, "Oh, yeah. Okay, so it comes from where we expect it to come from. Cool, that's good to know. " So it is useful. It does help. It makes things faster, right? Doesn't replace us making the effort of figuring out if what we're doing makes sense in the first place and if it takes the right trade-offs and if it's the right choice, those things I think are not that automatable or AI-able yet. Yet. — Maybe yet. That might change, right? But um I think these tools are useful, but yeah, you're absolutely right. It depends on how you use these tools. But yeah, and on top of that I think there's always risk of introducing a bug that you don't understand and so on. So uh I think that the whole discussion of how is like the software of the future, you know, is going to be maintained by the AI and will will, you know, remain human maintainable or like understandable. Right now you know, you still have like a bunch of people who can understand what's going on. Mhm. We'll see how that's going to evolve and will like the system that is fully AI run and AI automated be uh become at some point in the future more um will have an edge over the current system architecture or style of building systems. We'll see all that, but um I think it's important for now at least for all of us like in the engineering side to lean into the tools and make sure we we continue using them and they're be capable with them. And I would love to hear from you all out there. What do you think? Are you using AI for something that you wouldn't have expected before you tried it out or are you skeptical? Have you made good experiences? Have you made bad experiences with AI? I'm just curious how you all out there are experiencing this shift and this time of exploration basically. Anyway, thank you so much, Nikola, for being here. I think that was really interesting. We touched upon so many interesting things from how we are running experiments to how AI evolved at Google into the thinking behind AI or using AI mode and uh thank you so much for your time and thanks so much for being here. Thank you, Martin. It was pleasure joining in the podcast. And all of you out there, if you like to hear more of this, please do subscribe. We are on all your podcast platforms out there and we're looking forward to hear from you. So leave us a comment, leave us a like, leave us a subscription and uh talk to you soon. Bye-bye. Bye, everybody. We've been having fun with these podcast episodes. I hope you, the listener, have found them both entertaining and insightful, too. Feel free to drop us a note on LinkedIn or chat with us at one of our next events we go to. If you have any thoughts, let us know and of course, do not forget to like and subscribe. Thank you so much for listening and goodbye. —