# The Rise of Automated Fraud: Architecting Security Against "AI Slop"

## Метаданные

- **Канал:** InfoQ
- **YouTube:** https://www.youtube.com/watch?v=7dtTMFOzTJk
- **Дата:** 21.05.2026
- **Длительность:** 49:02
- **Просмотры:** 295

## Описание

As generative AI hits exponential scale, engineering teams face a massive shift in how they must handle fraud, identity, and trust boundaries.

In this InfoQ video, Shuman Ghosemajumder (Co-Founder of Reken, former Head of Global Product for Trust & Safety at Google) breaks down how bad actors use generative AI to scale phishing, defeat security training, bypass CAPTCHAs, and weaponize "model collapse" against enterprise systems. 

Learn why traditional authentication boundaries are failing and how to transition your infrastructure to a robust Zero-Trust model to combat AI-fueled automation.

⏱️ Video Timestamps (For Navigation)
00:00 - The Illusion of Ubiquitous AI (From Radio Shack to Toothbrushes)
02:15 - The Gell-Mann Amnesia Effect: Cognitive Biases in Tech
03:40 - Sora, Deepfakes, and the Legal Battleground of Copyrighted Models
05:30 - The Explosion of "AI Slop" on Social Feeds & Peer-Reviewed Journals
07:22 - Why LLMs Fail at Basic Math (Stop Using ChatGPT as a Calculator)
08:50 - Model Collapse in Action: How Chatbots Generate Hallucinated Data Loops
10:45 - Real-Time Deepfakes: The $25M Hong Kong CFO Video Scam
12:00 - Inside a Dark Web Credential Stuffing Attack (Sentry MBA Deep Dive)
13:40 - Why CAPTCHA is Dead (AI's 99.8% Solve Rate vs. Human Friction)
15:55 - The Scaling Advantage: Why Cybercriminals Love Customer Support Tech
18:10 - Fallacies of Security Training and Deepfake Detection
20:25 - Transitioning to Zero-Trust Architecture & Cyber Fusion Centers
22:15 - 3 Pillars of AI Risk: Infrastructure, Business Models, and Communication Channels
25:02 - The Future of Human Augmentation and Co-Intelligence

🔗 Transcript available on InfoQ:   https://bit.ly/42NNQjC      
   
#SoftwareArchitecture #Cybersecurity #ZeroTrust #GenerativeAI #AppSec

## Содержание

### [0:00](https://www.youtube.com/watch?v=7dtTMFOzTJk) The Illusion of Ubiquitous AI (From Radio Shack to Toothbrushes)

— I don't know about all of you, but I have been interested in AI for as long as I can remember. This is the very first book that I read on AI. It was purchased by my dad for me when I was 10 years old from RadioShack, as you can see, and it had no mention of phrases like machine learning or data, and certainly no mention of concepts like transformer models. So, the industry as a whole has been thinking about AI for a very long time, and there's been so much transformation over the course of the last several decades, but in particular, in just the last few years. You've probably noticed that everything is AI now. I I've had my teams walk through the show floor of some of the big security conferences and take some pictures of the booths that are advertising what these companies do now, and here's a collage of what it looks like. There are so many different ways to be able to say that your product is AI-enabled or AI-boosted or AI-infused is one of my favorites. I bought a toothbrush recently that has artificial intelligence in it. I don't know exactly where the artificial intelligence is, but apparently that makes it a better toothbrush. One of the best illustrations that I've seen of the ubiquity of AI in society today and throughout the industry is this cartoon that basically summarizes what is going on with everyone generating content using AI, and then saying, "Oh, that's way too much content. Why don't we summarize that using AI as well, and that'll save us a lot of time. " And you've probably seen some of these incredible statistics that AI infrastructure building is now even larger than plans for buildings that humans are going to be occupying. But there are some holes in this story. For example, we may be seeing patterns where AI use drops precipitously when students

### [2:15](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=135s) The Gell-Mann Amnesia Effect: Cognitive Biases in Tech

aren't using it for their primary application. Now, you may be wondering as a result of all of this, where exactly is AI leading us? And I have good news. There is a definitive answer to where AI is leading us, and it was produced uh a few months ago from the Financial Times. They put some data together, and what they concluded was that AI is either going to be our salvation, or it's going to destroy all of civilization, or something in between. Definitely one of those three things and nothing else. So, AI is a great and terrible term. I think that uh part of the problem associated with it is that it sounds like it's doing so much. And of course, we can blame Alan Turing for making us so excited about this area in the first place when he published his seminal paper Computing Machinery and Intelligence. Like, what an amazing title for an academic paper, by the way. That's like going back hundreds of years, and if Isaac Newton published a paper called math. Like, just being able

### [3:40](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=220s) Sora, Deepfakes, and the Legal Battleground of Copyrighted Models

to have something that's so broad is only possible when the field doesn't exist yet yet, and you're inventing it. And so, that's exactly what Alan Turing did. He proposed the idea that is now called the Turing test, and he proposed in the context of what he called the imitation game. Would it be possible to create a computer system that's so powerful that it would imitate a human being successfully to another human observer? And the Economist a number of years later was very skeptical about this. Well, one of my favorite phrases from their dismissal of Turing's idea was that there is no practical reason to create machine intelligence as indistinguishable from human ones. Basically, there's no reason that we need to have the entire field of AI. But my favorite is the second part of the quote that people are in plentiful supply. Should a shortage arise, there are proven and popular methods for creating more of them. But, you know, don't be too hard on the Economist. As Yogi Berra said, "Making predictions is hard, especially about the future. " So, there are many confusing terms that are associated with AI and I think that what this has resulted in is people generally conflating it with AGI. So, you look at all of these terms like machine learning and deep learning and artificial intelligence and artificial general intelligence and I would say that people are most informed about AI and their feelings about AI not from any sort of technical source, but from science fiction and from stories that they have consumed over the course of their lives from society as a whole. And that's extremely confusing for folks and it results in whatever technology

### [5:30](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=330s) The Explosion of "AI Slop" on Social Feeds & Peer-Reviewed Journals

seems sufficiently advanced today, as Arthur C. Clarke would say, fooling us into thinking that they are the magic of AGI. And the one thing that we have to realize is that none of us are quite as smart as we think we might be in every single field. So, George Carlin, I think, really illustrated this best. And I think that the problem is that we're able to see through hallucinations, and errors when AI is generating content in our own field. But, as soon as we ask AI a question that is outside of our area of expertise, all of a sudden it's much harder for us to be able to see those hallucinations. And there's a principle that describes this, and it's called the Gell-Mann Amnesia effect. It's named after Nobel laureate Murray Gell-Mann, who would read an article in the newspaper about physics, and he would rant and rave about how terrible the article was and how many errors there were, and then he would turn the page and read an article about finance or about politics and assume that reporter had done their homework, and clearly everything was accurate in that. And this is something that we all fall into in terms of a cognitive bias. And AI today, generative AI in particular, is so great at being able to fool us. So, when we first saw generative adversarial networks come into view, we saw fairly harmless applications like sticking Nicolas Cage's face in places that it shouldn't be. And it didn't seem like it was that realistic or that convincing, so people weren't that alarmed by it. But, then we started to see a little more sophistication arise. So, let's say that

### [7:22](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=442s) Why LLMs Fail at Basic Math (Stop Using ChatGPT as a Calculator)

you take someone who is a really good Tom Cruise impersonator, like Miles Fisher, and now you add generative AI on top of that. Because he can nail his voice and his mannerisms, we were able to see, just a few years ago, very realistic-looking Tom Cruise deep fakes that really started to catch everyone's imagination. And just a couple of months ago, we saw where this has led us, which is Sora being launched by OpenAI and allowing anyone to create highly realistic videos. For example, Mark Cuban, if you were at QCon, might say something like this. Hey QCon AI conference, this video's totally fake. Don't believe anything on the internet anymore. Mark Cuban gave all Sora users the ability to use his image and his voice to create whatever videos they want. So, you can now basically puppeteer Mark Cuban to say whatever you'd like. And users of Sora immediately jumped on this, and they started to use all kinds of copyrighted characters, which that there were no guardrails against at the time to be able to create videos that many of the copyright holders had a big problem with. So, the copyright holders quickly responded. And recently, just in the last week, Disney announced a

### [8:50](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=530s) Model Collapse in Action: How Chatbots Generate Hallucinated Data Loops

billion-dollar deal with OpenAI while simultaneously suing Google for not having a deal with them, effectively. And this is something that leads a lot of people to asking the question, how important is copyrighted content to generative AI as a whole? Well, I have an illustration for you. If we go to a generative AI model that is reliant on copyrighted content and compare it to a model that isn't using copyrighted content, what does it look like? For example, if we go to Midjourney, which has very few guardrails, especially historically, against the use of copyrighted content, and we give it the prompt Chewbacca reading, you get something that looks highly realistic in terms of a representation of the character Chewbacca from Star Wars sitting there reading. Now, if we go to Adobe Firefly, which was specifically trained without copyrighted content, what you get can be at best charitably described as homemade Chewbacca. So, this is essentially the trade-off that we have in terms of all of our generative AI models. If there's copyrighted content in there, we can get something on the left. Without the copyrighted content, we're going to get much less satisfying results. And in just the last few months, there was an uproar over the idea of an AI-generated actress, Tilly Norwood, and we had actors and directors and filmmakers from all over the world decrying the idea of an automaton that didn't have any of the humanity associated with human actors representing the same types of emotions and acting on screen. And the idea

### [10:45](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=645s) Real-Time Deepfakes: The $25M Hong Kong CFO Video Scam

types of acting on screen. And the idea was, "How are you going to have the same level of creativity, the same imaginative ideas that we see in film and TV without the humans themselves? " And to that, I would say, "Here's another way of looking at it. Here is the level of creativity and imagination that humans have actually produced for many years on the Hallmark Channel. " So, when you start to see the pattern of how humanity has been producing a lot of content, you start to realize that there are a lot of things that AI models can extrapolate and interpolate from in order to be able to really threaten jobs that are being performed by humans today. And so, you know, maybe we're not talking about the uh threatening of somebody like a Tom Hanks or a Julia Roberts, but there are lots of working actors and directors and screenwriters that are producing a certain level of work that AI can kind of run with to some extent. And this is one of the reasons that in just the last couple of days, Merriam-Webster declared slop to be the word of the year. Now, I take one exception to the definition that they provided of slop

### [12:00](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=720s) Inside a Dark Web Credential Stuffing Attack (Sentry MBA Deep Dive)

which is saying that it's low-quality content that's produced by AI. And the problem with saying that it's low-quality is that it sounds like you're going to be able to easily identify it. It sounds like something that is not competing with human content, and the reality is there is a lot of content that's already out there that people can't distinguish from human-generated content that's coming from AI. For example, you've probably never seen a gorilla wrestling with a python before. I certainly hadn't. And when this came up on YouTube Shorts, I was immediately amazed by this, and I was thinking, "When did this happen? Where did this happen? " It turns out it never happened at all. There's an entire YouTube Shorts channel dedicated to pythons and gorillas and other primates fighting, and because it's so amazing to look at, it has millions of views. In fact, there are thousands of channels that are producing AI-generated content on YouTube, on TikTok, on and on many other platforms. So, we may be thinking about AI slop in the context of well, in a few years' time, this is something that we're going to have to worry about, and we should try and educate ourselves in terms of how we're going to be able to identify that content. The reality is we're consuming it already. In tests that we've done, I'd say about 20% of the content on the default feed on YouTube Shorts as well as on TikTok is already AI generated. And one of the amazing things that's happening at the same time

### [13:40](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=820s) Why CAPTCHA is Dead (AI's 99.8% Solve Rate vs. Human Friction)

is that a lot of real content from movies and TV shows that had nothing to do with AI-related production are actually going through filters that make them look more AI generated. So, that in turn makes it harder to tell the difference between real content and AI-generated content in the future. When you go to YouTube or you go to other websites and you view ads, many of those ads are now AI generated as well. So, if we take a look at uh YouTube in particular, one of the things that we see is so many ads that are AI generated representing celebrities like Oprah and Ben Carson and many others that Oprah actually had to put out a statement saying this is not me. I'm not endorsing these random pharmaceutical products or these random home remedies, but they look and sound exactly like Oprah and out of the context of understanding that this is an AI generated video, the vast majority of people simply consume that content and they don't realize what's going on. If you Google Tiananmen Square Tank Man selfie, you literally get hundreds of copies of this image that went viral, which of course is AI generated. There was no such selfie. There were no selfie cameras that were available at the time to be able to take a picture like this. And yet, future generations Googling this will probably assume that this was a real photo that was taken because it's not only in Google search results, it's at the top of Google search results. One exercise that I did with a group of university presidents was I went to Nature's website, Nature being one of the most prestigious academic journals in the world. And I did a search for artificial intelligence on the theory that anyone who is engaged with AI in any form is probably an AI enthusiast and using other AI tools. And so I pulled up this paper about cervical cancer and ran it through GPT-0. And what we discovered was that the abstract and the first paragraph came back as 100% AI-generated.

### [15:55](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=955s) The Scaling Advantage: Why Cybercriminals Love Customer Support Tech

So that leads to the question, how much AI-generated content is in peer-reviewed journals already and we just don't realize it and what exactly does it mean? Because it doesn't necessarily mean that this content is being generated wholesale by AI. It could simply mean that it was cleaned up from a grammatical perspective by AI. It could simply mean that the AI was used to be able to generate a first draft and then that was fact-checked by humans. But the problem is we don't know. And what we can see is the widespread use of AI tools. And this brings me to one of my favorite quotes from Winston Churchill that a lie gets halfway around the world before the truth has a chance to get its pants on. And my favorite thing about this quote is that Winston Churchill never actually said it. And this is a great illustration of a learning that came from MIT Professor Sinan Aral where they did a study that showed that lies spread six times as fast on social media as the truth does. And you might ask yourself, why is that the case? But it actually kind of makes sense that if you're the purveyor of a lie, then you're trying to propagate it as widely as possible, as quickly as possible. But if you consume that lie and you spot that it's actually false information, it can often outrage you and then you share it with a bunch of your friends saying, "Look at this lie. " And so everyone on both sides of it is actually engaged in making it go as far and as wide as possible. And this is one of the reasons that social media to a large extent has just turned into a giant rage circus. And so I created a framework that describes what happens with this type of misinformation and fraud that basically goes through these stages for any type of content. And what it consists of is in the first stage, it requires a great deal of effort and talent and resources to be able to create a convincing fake. So in the case of the Nicholas Cages

### [18:10](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=1090s) Fallacies of Security Training and Deepfake Detection

face going into different movie clips, that was an example of a way of being able to automate something that previously would require Hollywood special effects. But it wasn't something that anyone could do. It actually required many days of computing power in order to be able to do effectively at the time. But all of that has now changed. You can not only go to a system like Sora and generate a video in a matter of maybe about 60 to 120 seconds on average. But you can go to Grok and give it a single frame of a video or a still image and it will generate a video of that image for you in under 60 seconds in a lot of cases. And so that's bringing us with video and audio to stage three, where you can have one individual or entity producing vast amounts of content. And we're already there when it comes to text. We we've seen it in the context of websites that are low quality, that are trying to attract AdSense viewers and monetize through AdSense. This is something that we saw 20 years ago at Google. And that's something which has only become more sophisticated with the introduction of generative AI. So, when we think about how generative AI is used in this context, we can also realize that misinformation that's presented can be extremely subtle. So, when you go to chat GPT, even the most recent models of it, and you ask it fairly simple counting questions, like for example, I asked it, "How many J's are there in the last name Ghost of Tsushima? " Which I assume is something that no one has ever asked before on the internet. It comes back with a very confident answer and shows its work. And you can see here that it's not even pointing at letters. It's pointing at the spaces between the letters while it's miscounting. So, if there's one thing that you take away from this talk, it's please stop using chat GPT or any

### [20:25](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=1225s) Transitioning to Zero-Trust Architecture & Cyber Fusion Centers

generative AI as a calculator. It may seem like it's able to do amazing things from a math perspective. It's able to score really high on the International Math Olympiad questions. But in reality, it is just simulating those behaviors and it's not actually doing any real computing. If you ask it to create a highly detailed diagram, that diagram at first glance can look extremely impressive. But then you start to notice that there are a few things that it has misunderstood. Now, since we're in the New York Academy of Medicine, let's take the example of something that might be a bit more serious. Would you be interested in using an AI system that was performing vibe surgery? There are real consequences to getting certain details wrong. And when we look at where this information comes from, one of the things that we see is that AI is getting its facts from some unexpected sources. Wikipedia, of course, is full of a lot of high-quality information, but it's also misinformation. There are hoaxes that have lived on Wikipedia for years before they were discovered. And of course, Reddit is full of misinformation, and yet it appears to be one of the top sources for content that generative AI models are trained on. Here's one thing that we found, which was another subtle form of misinformation. This is an article that I wrote in my column for Inc. Magazine. Within an hour of it being published, we saw this article appear on an Argentinian website in Spanish.

### [22:15](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=1335s) 3 Pillars of AI Risk: Infrastructure, Business Models, and Communication Channels

And it is basically taking the original article and translating it into Spanish and adding a few more keywords to the prompt in order to generate what looks like coverage of that article. And this has been done so convincingly that this publication is actually a Google News source. So, anyone reading that article in Spanish would have no idea that this is actually a form of AI slop that has been generated and isn't an original article. One of the things that I was fascinated by recently was a venture capitalist reached out to us and this is someone that I've known for a while and said we'd love to have a conversation about Recon. And when I went into the conversation, they said, "Well, we feel like we have a pretty good idea of what you do. " And I said, "Really? Because we're in stealth and we haven't shared what we do. So, how exactly do you have this idea? " And the answer was they had gone to a chatbot and asked, "What does Recon do? " And it had gotten an answer. And so, I did the same thing. And the chatbot came back with a bunch of things that are kind of in our space. And I was really curious, where did it get this not quite correct information? And of course, one of the innovations that we've seen in the last couple of years is the introduction of chatbots like Perplexity that have sources for the content that they're generating. And that produces a great deal of trust because how on earth could the content be wrong if it actually has a source that it's coming from on the web? And indeed, in this case, there was a source. So, I was curious, what was the source that they had for this misinformation about our company? And it turns out the source was an AI-generated website. So, this is an example of model collapse in action in a way that people wouldn't realize. So, if you're not familiar with the subject matter, because how could you be? You're asking the question, "Tell me about a company that I have no information about. " Then all of this is going to look highly credible from the original website which is being cited to the AI-generated summary. And that's how the VC got the wrong idea and that's how a lot of people can get the wrong idea about a lot of things. And where this leads us to from a risk perspective and a security perspective is that generative AI is really the ultimate cybercriminal tool. A few years ago, we saw generative AI being used to be able to clone voices. And that was in a fairly simplistic form. And this has only gotten more sophisticated. In the past year, we had the Arup case where an employee of this

### [25:02](https://www.youtube.com/watch?v=7dtTMFOzTJk&t=1502s) The Future of Human Augmentation and Co-Intelligence

engineering firm in Hong Kong got onto a Zoom call and had a conversation with several of their colleagues, and then agreed to transfer $25 million at the request of their CFO, only to discover afterwards that it wasn't their CFO that they were talking to or any of their colleagues. It was a bunch of real-time deepfake Zoom uh representations of them. And so, this technology is only going to get more advanced. And we started hearing about these kinds of examples a few years ago. And the reason that this is such a threat is because AI at its core is really just automation. It's really the ultimate form of automation, and cybercriminals are constantly automating. Let me give you an illustration of how cybercriminals have been automating for the last 10 years in a way that people don't have a lot of visibility into. When people think of cybercriminals, they often still think of someone in a hoodie in their parents' basement trying to figure out a way into a server, and that's not what cybercrime has looked like for two decades. Instead, it's highly commoditized and federated, and there are organizations of cybercriminals that work with each other in order to be able to create high levels of automation. So, every single time you see a big data breach announced, you probably think, "Oh, I need to go to that website where the data breach happened and change my password. " But that's only the beginning of what happens with those data breaches. Those usernames and passwords then become part of a corpus that cybercriminals use to be able to breach completely unrelated websites and accounts. And they do that through something called a credential stuffing attack. Here's a piece of software that we discovered in the dark web called Century MBA. It looks like just a standard Windows application, but this is actually an application built by a specialized group of cybercriminals for other cybercriminals to be able to plug into botnets that they commandeer in order to create large-scale credential stuffing attacks. Here's what it looks like on a real-life website. So, if you're familiar with what web traffic generally looks like, especially on a retail website, you're used to seeing this diurnal periodicity of users accessing the web more when they're awake than when they're asleep. So, this is what a typical week might look like on any given website, but this is actual data from one of the largest retailers in the world. Now, when we went into their website initially, we did not see this pattern. Instead, we saw a pattern that looked like this. And that was a bit weird. We're not seeing organic users behaving the way that organic users typically do. And when we started to distinguish between the automated traffic and the human traffic, we saw that the human traffic was actually there, but the vast majority of the traffic was actually automated. On a 24/7 basis, they were getting hit with cybercriminals who were automatically taking stolen usernames and passwords from other websites and testing them against their login form to see which ones were valid. Because of the fact that users constantly reuse their passwords, there was typically about a 1 to 2% success rate that they would have, which would allow them to take over thousands of accounts on mass. And this is something that we see in basically every single industry. So, what's going on behind the scenes is a very high level of automation that the vast majority of society had no idea about, and generative AI now allows us to automate this even further. Now, you might be wondering, "Well, aren't there protections that are available to be able to protect against bot-related activity? " And there've been a number of attempts that have been tried in the past. The most famous being CAPTCHA, which you may not realize is actually an acronym that stands for Completely Automated Public Turing Test to tell computers and humans apart. And CAPTCHAs have been trying to distinguish humans from bots for many years. And a few years ago, Google did a study because they were wondering, "How have CAPTCHAs fared? How have they ended up in terms of their efficacy? " And when we look at all of the time that's spent solving CAPTCHA, that's a really important question, because we want to know that this is an effective mechanism and that it's worthwhile to engage with. So, what Google found was that humans had dropped down to a 33% solve rate with the typical synthetic distorted text CAPTCHAs that we see all throughout the internet. But, AI had actually skyrocketed in terms of its ability. Machine learning-based OCR had a 99. 8% solve rate. And so, this difference has only gotten exacerbated in the following years. By the way, there are cybercriminal services that are specialized in just helping other cybercriminals solve CAPTCHAs. So, some of these services offer group discounts and all kinds of customer support, because they want to be good cybercriminal businesses to the other cybercriminals. And if you take CAPTCHAs today and you run them through generative AI systems, they pose no barrier whatsoever for them. And so, if you're using CAPTCHA today on your website, you're basically doing the exact opposite of what you should be doing. You're introducing friction for real users, and it's no barrier at all for cybercriminals. So, what exactly does this mean in terms of automation for cybercriminals? Now, you might have heard about the IRS phone scams that affected more than 400,000 people in the United States over the span of just a few years. In the vast majority of those cases, in order to be able to steal any money from individuals, the victims had to have a conversation with someone in the cybercriminals' effectively call center. And that's expensive, that's risky for the cybercriminals to operate, and they had to do that because that was the only way to be able to convince people to transfer their money. Now, one of the things that you may have seen is a seminal study from Stanford in just the last year that showed that when they analyzed a bunch of different applications of generative AI, the most promising application was in customer support. So, across the board, there was customer support productivity improvements that came by using generative AI tools, and those uh productivity enhancements were the greatest for the least experienced customer support folks. For cybercriminals, this is just an amazing opportunity because while legitimate enterprise has to deal with hallucinations and errors and figuring out, what do we do about the shortcomings of generative AI in the context of customer support, for cybercriminals, those hallucinations are actually features instead of bugs. Because they're trying to tell a gullible victim a believable story. And so, generative AI as it works today is out of the box usable by cybercriminals to be able to automate what they previously could not automate, which was essentially their last mile problem of how do we generate realistic audio, realistic video in order to be able to con people more effectively. As a result of this, folks like Geoffrey Hinton decided that he needed to leave Google in order to spread the word about how dangerous these scams are as enabled by AI. Warren Buffett, when he was asked at his annual meeting about the greatest possible growth industry, he made a comment about how AI-enabled scams are probably going to be the greatest growth industry that he's ever seen, but unfortunately he's not going to be able to invest in that. And another trend that we've seen is the democratization of AI. There's been this idea for a few years now that AI is extremely expensive to create. That you have to have billions of dollars or, if you believe some of the hyper-scalers, hundreds of billions of dollars or even trillions of dollars in order to be able to create more effective AI. But as soon as Deepfake launched, we saw the contradiction of that and we started to see that you can have very inexpensive systems, including models trained by cybercriminals, that can be extremely effective, in many cases even more effective than the models that are produced in more expensive ways. Now, the key thing to realize here is that you don't even need to have the greatest AI in the world in order to be able to fool people effectively. There are all kinds of examples of people being fooled at scale without that much sophistication in the technology that enables it. So, what do we do about all of this? There's lots of advice out there. Lots of fortune cookie wisdom. Um there are ideas like you should create a secret password with your family. And that's actually something that I think is a good exercise. It's good to have the conversation about what would you do if you received a phone call and it sounded exactly like a friend or a family member. And having a secret password is a way of being able to imagine yourself in that situation and try and navigate it if it ever does arise. But, the reality is if you're in that situation, the fraudster knows exactly what buttons to push. And if they simulate your loved one's voice saying, "I can't remember the password. " because they're in such a stressful situation, that's going to convince a lot of people. And fraudsters don't need to be successful 100% of the time. As we were discussing with credential stuffing attacks, even a 2% success rate is enough for cybercriminals to be able to create multi-billion dollar businesses. So, there are a lot of different drawbacks that are associated with some of the standard practices that are out there. And one of the most common practices is fishing training or security training. Again, something that you should do and that everyone does. But, there's been a lot of academic research and, you know, anecdotal and quantitative information that we've seen that shows that fishing training doesn't actually prevent people from clicking on fishing links or engaging with social engineering attempts, especially if they're realistic enough, especially if they're contextually targeted based on generative AI that can customize the message that's sent out to everyone in your organization that they're trying to socially engineer. And there are a variety of different techniques that fall into this bucket of not working all that well. So, we talked about CAPTCHA, we talked about um uh security training, deep fake detection is another area that a lot of people have a lot of fascination about, but the problem with deep fake detection is twofold. Again, it's something that's helpful, but it's not a solution for a couple of reasons. So, one it's really difficult to be able to definitively say that something is generated by a malicious deep fake model versus something being generated by AI that is modifying an image or video or audio for other purposes. Every single time Apple launches a new phone, they talk about the multiple layers of AI that every image gets processed through, that every video gets processed through, and there are very few images and videos these days that don't go through some form of AI processing before they go online. Google, when it launched Nano Banana, was effectively encouraging everyone to modify their photos using generative AI. So, we're seeing a great deal of generative AI content that isn't necessarily created for malicious purposes. But, of course, the other problem that's associated with deep fake detection is the fact that you can't detect all of the deep fake models that cybercriminals might be using. And so, if you analyze a particular image or video, and it comes back with an assessment that says, "There's a 50% chance that 40% of this content is AI-generated. " How exactly do you operationalize that? So, what are techniques that work more effectively? Multi-factor authentication is something which is very effective. Behavioral know your customer operations, where you actually study the behavior of an account, of a device, of an individual in order to be able to flag it for anomalies or patterns, that can be something that's highly effective. One of the things that has been an interesting trend for me to observe in the security industry in the last 10 years is the rise of the idea of zero trust security. And the basic idea here is you shouldn't have an authentication gate where when the user passes that gate, they're now given full access and you can completely trust them to perform whatever actions they want. Because you could end up in a scenario like I was describing before where a user's password is stolen and now that authentication step has actually been commandeered by a cybercriminal and now you have to watch everything that happens post-authentication in order to discover if that account is being abused in some way that you didn't foresee. So this was an epiphany for the security industry and they called it zero trust. And one of the reasons that this was so interesting for me was because this has always been the mindset of the fraud industry. From a fraud perspective, you never trust any account or any device. You're constantly looking at the data that's associated with the behavior of those entities and the good news is that message is now spread throughout the entire industry. In fact, we're seeing a lot of collaboration between fraud teams and infosec teams in order to be able to take all of the data that might be available to a web application or a mobile application or an enterprise and figure out how to be able to use that to spot those patterns and anomalies and identify where abuse may be occurring. And uh a lot of these integrated efforts are called cyber fusion centers, and I think that it's a fantastic trend uh to be able to see. There are three areas overall in organizations uh from a cybersecurity perspective that are impacted by AI in different ways. So, when we think about infrastructure security, the primary impact that people are worried about is how AI enables cybercriminals to be able to discover and exploit vulnerabilities at scale. So, AI is fantastic at helping us solve completeness problems. Even though it's not doing any real thinking, what it is capable of doing is taking all of the data that's out there, including from Reddit and Wikipedia and so on, and telling us the things that we might not be thinking of in a particular context. So, if you use AI as a cybercriminal to be able to analyze a web application for vulnerabilities, it can give that cybercriminal a much more complete list of things to probe for than if you didn't have such a tool. And so, that's the risk from an infrastructure perspective. From a business model perspective and a trust and safety perspective, what we see is account abuse. And being able to automate user actions in a way that websites and mobile apps might not have anticipated. So, we're now seeing this tension playing out in that there are a number of AI-enabled browsers that have been launched from OpenAI and Perplexity and others. And Gartner came out with a statement recently saying, "You as an organization should block all AI-enabled browsers. " And one of the reasons for doing that is the risk that's associated with automation writ large being possible on your website and what cybercriminals can do with that. The third area, of course, is your channels of communication. And regardless of how much you lock down your business model and infrastructure, you always have to keep those doors open in order to be able to communicate within your organization and with the external world. And that introduces the opportunity for cyber criminals to be able to use AI to socially engineer your employees, your customers, your executives, folks all throughout your supply chain in ways that you never would have conceived of in the past because it simply wasn't possible at scale. Well, one of the analogies that I used to be able to describe the scale that cyber criminals can operate at with automation technologies is we don't have good intuition for how that works because we think in real-world terms. When we think about locking our houses, we often think, "Well, what can I do to make myself less of a target and how can I secure my particular house? " But in the case of cyber criminals on the internet, they're not focusing on you in particular. There doesn't have to be anything special about you or your organization or your website. They can attack everyone simultaneously. Imagine if a robber could break into every house in a community at the same time. If they could rob every bank in a city at the same time. That's what's possible with automation, and there were previously blocks that prevented cyber criminals from performing that last mile where they wouldn't be able to branch to human and actually have someone speak to the victim on the phone realistically in the past, but now generative AI enables that. So, we need to think differently about protecting those communication channels against those types of attacks. What we ultimately need is good AI to be able to fight against that bad AI. We need scale on our side to be able to deal with the scale that's coming in from the cyber criminal side. And there's good news in that regard. There's another study from MIT from Professor Tom Malone where he was looking at all the different ways that AI can help different work processes within organizations. And what he found was that humans and AI combined would outperform humans alone. So there's a great opportunity for us to be able and some people have called this co-intelligence. human augmentation. Regardless of what terminology ends up being the standard, I think we're going to see a great deal of products and services that combine humans and AI in order to be able to make better decisions and especially to help humans make better decisions. In fact, I think that this is one of the long-term trends that we're going to see in terms of how do we use generative AI throughout our organizations and even in our personal lives. So a lot of people look at AI, conflate it with AGI like I was mentioning before, and they think that AI is going to do our thinking for us. I think that we're going to rapidly realize that that's not the case, but generative AI can be a fantastic brainstorming partner. partner for bouncing ideas off of to help refine our thinking. And so this is an early research finding that kind of illustrates the direction that I think we're moving towards where humans and AI are going to be working together in a very conscious way where we're making intentional choices to bring AI into a conversation as opposed to simply outsourcing an entire work process or our thinking to it. What I want to leave you with is this quote from William Gibson that I love that I think is illustrative of so many different technologies throughout society, but AI in particular. The future is already here, it's just not evenly distributed. And what this means for me in this context is that we already have examples of the most dangerous uses of AI that we could possibly conceive of. There are uses of AI that are even more dangerous than the things that I talked about in this presentation today, but the reason that society isn't completely freaking out about them is because they're not affecting everyone yet. So, we have the opportunity to identify where these risks start to emerge, and at the same time look for uses of AI that are extremely beneficial both in the security industry as well as throughout all of our operations, and then extrapolate from that to figure out how does this scale? How do we use this throughout an organization? all of society? And so, what I always tell my teams is that we need to monitor all of the advancements in any new technology so that we can identify something that could potentially help us, but we don't want to fall in love with the technology. We don't want to use it for its own sake. There are a lot of organizations, as you saw in that slide with AI-enabled and AI-infused security, many organizations that are simply saying, "We've added AI to our product, and that's why you should buy our toothbrush. " And in reality, you shouldn't use something just because it's AI. You should use a product because it makes your life better in some capacity. And I think that that's the trend that we're going to see overall. But the only way to be able to improve your organization or your product quicker than your competitors is to be able to realize and recognize those opportunities before everyone else. And that's really what uh the challenge is for all of us and and one of the cool things about being at a conference like uh QCon AI because you get to see what the cutting edge uh looks like today. So, uh I'm going to be around uh for uh questions if you want to grab me afterwards. My Because I'm in security, my social media policy is uh I only add people that I've met with in person. But we're all in person here, so come and grab me and uh you know, let let's have a conversation. Thank you very much. —

---
*Источник: https://ekstraktznaniy.ru/video/51768*