How Cyber Criminals Are Using ChatGPT (w/ Sergey Shykevich)

hello today I'm talking to Sergey shakiewicz who is a researcher at checkpoint research and is largely concerned with the field of cyber criminals specifically today we're talking about how cyber criminals use chat GPT and tools like that in order to do their bidding there is so much stuff happening in this area so cyber criminals are starting to use chat GPD to do social engineering to construct malware and I wanted to know just how imminent is this threat you know how enabling is it to someone and what does it mean for the rest I hope you enjoy and I'll see you around Sergey shukovic

very glad to have you here today thank you it's pleasure for me to be here thank you for having me first of all maybe what is checkpoint research and what do you do there so checkpoint research is a research organization in Chuck inside checkpoint software security the big company where our organization of what 177 180 people that our goal is generally to do a research we are not dealing with statistic security products or something we are doing research collecting intelligence in all areas of cyber security of course it's also to support our products so understand what's going on the other side of the mostly the side of the bad guys and also our goal is to contribute to cyber security Society will also publish a lot of different researchers just to be available to sharing knowledge that's one of our question in our day today we mostly track cyber criminals hacktivism Mission State actor and part of what we're doing also talking to new technologies and how data and influence the Cyber threat when the new language models came out

what was the first thing that you observed that the let's say call them bad guys did with it like what was immediately accessible to them after release in their last November kind of sparkled a lot of interest from the bad guy I maybe it's important to mention on uh we're tracking also a lot of Darker communities in at least one of the biggest Russian dark web Forum there was a section called AI machine learning for years but this section was barely active there was one post there months or so after the release it sparkled it was a huge interest rail cyber furnace how to use to do something new like for us yeah like in for normal people also for them and the first thing was which all happened in December 24th pretty quickly after just gpk launched that a guy called uspot like U. S Departed and that of course doesn't have any connection to a United States Department of Defense posted the post he called the python multi-layer encryption tool in other words it was the code to a rental a software tool that encrypts file in normal World it can be a good tool to protect your data on your machine but in context of dark web form of course it's not something good and people see the code of it and told this is my store script so keep this in mind and posted the python script and then just few hours afterwards another guy asked me these things a lot like open AI called trimmer style and then you'll do the confirm yet open AI gave me a nice hand to finish the script with knifed code helicos came back the first time these guys it was never a developer before he became I call it alleged developer because I don't want to really inteller people who really learn coding development for a few years and don't think he's not really a developer but he was able to create the software malicious software even the 3gp video that will drive him the first real figure yeah I think the uh worry

about this is more widespread that oh now let's say creating malicious software is much more accessible and so on the particular script that this person created how hard would it be for me to just find that somewhere on that dark web Forum like on some other sub Forum like was it a new script or was it something that I could just download from somewhere there conceivably so I would say it's possible to find similar scripts yeah but it will require a bit more of work for example if you want to you know find such script you need to find this forum or some specific repositories in GitHub something like this you should do some work before during just much easier just you know you I'm going to tgpt or Google bars by the way I'm in this perspectively there is not much difference or codecs or other generative ai2 and write what I want and it provides me so most of the tools we saw until now are not extremely sophisticated that's important to mention it's not like it's not a basically a developer can create it or you never can find it that's still nothing um

um there are also other strains here that people use specifically Also regarding for example phishing emails and so on so the scripts I can maybe see a little bit but I think when it comes to phishing and social engineering people say these language models are specifically Adept to that maybe related a bit to you know fake news creation oh I can these they can produce things on a personalized level how much of that are we seeing currently in the Cyber criminal world so I think it's a great question before because if as I mentioned in the malware area we still don't see something really in top tie or something new in fishing email we already see that fishing emails and impersonating content that is created using the generative AI engines is much better that we already saw before that cyber gun has created before because even now the church Japanese for example create perfect fishing demos that can impersonate Bank PayPal government whoever and the Cyber criminals can easily use it moreover you can prompt those engines of AI engines to a jacket to whatever fishing campaign you want what do I mean you want to use fishing gear model with just a malicious link puts inside the mirror you write please the craft modification demon that will urge the user to click a link or if it's an attachment you just take oh it should be something that will prompt the user or an Excel attachment and it create so it opens thing to play a really a new steel for cyber performance especially or non-native English speaker I would say that's especially because for example the Russian guys the more sophisticated from Mostly their English level is very low up to non-exist and until now what they did they paid to other cyber criminals what is called call centers mostly those are graduates of English literature colleges in Russia by the way that consider the very high-end yet employment or opportunity for those guys or ladies so they pay them and those credit for them fishing demons or so for a payment of course now those hypotenuse don't have to pay don't have to find the external vendors let's call it and just can use a check Deputy for it so in a

sense previously these people didn't have enough language skills or maybe also social skills like how do I write a professional looking email and so on and that can be now produced in a much cheaper way than having to pay someone for it yeah I can see that but it is still it's quite a it's quite an interesting fact what kind of economies spring up around these Concepts right with people of different skills somehow finding each other in order to achieve um yeah do are you aware of any high profile let's say phishing attacks that have been carried out where we know that uh like GPT models were part of it or is that still unclear so with cyber criminals talking about it but I think it's still very difficult to tell about a specific fishing giml or malware that we've seen our data whether it was created using AI engine or human beings there are already of course tools that allegedly tell how it was created but I think their first positive rate is still high that you can't really rely on it so I assume yes but it's something very difficult to kind of proven pinpoint yeah this big fishing campaign was created using jgpt bars or whatever

um is there a let's assume we accept this is the new world right people are able to use these tools they're not going to go away uh they're only going to become more prolific uh can chat GPT or similar models also be used to counter apart from the detectors let's say can they be used to counter these attacks are there novel ways of Defending so it's something I think at some point here of course on the detection gear we and we have already a lot of engines even before jgbt based on AI to detect you know malicious URLs malicious patterns all these to directly Tackle fishing games I think yeah it's a it's kind of throwing different development stages in the understanding what is the best way to do it but I think at some point we'll have to tackle AI with the you know defensive AI let's call it there will be no choice for us if Defender yeah um there is a strain of people on YouTube who sort of reverse hack like scammers that try to call them and try to establish remote connections and so on so they will take a call from a scammer the scammer will ask them to some and establish some sort of Team Viewer or something like this and through that sometimes they're able if this scammer isn't very versed to sort of take over the scammers computer or at least I mean at least cause them some trouble and so on um maybe I don't know in my thought that should be possible that essentially you use the first line of defense you use these models to just kind of waste these people's time and just sort of get back at them a little bit with their own tricks because it's maybe not a sustainable defense it's an interesting perspective it's kind of uh Lord to lure those people with AI and do it it's an interesting it's traps I think it's less it's more a kind of anti-social engineering maybe I would I can tell it it's in a new era of I think we're generally entering social engineering that's maybe it's a good way to define it so like the attackers now will be more sophisticated deep fakes fishing as a Defender also maybe we should just try to view it the way you mentioned really how we really waste the time of the attacker but I think we are still not there I'm okay apart from sort of using GPT models to

counter this what is the most effective way of combating these new threats be that you know users being able to really easily create malicious code or really easily being able to create malicious uh phishing emails and so on like what's the most promising thing to do I think unfortunately maybe the best thing is the oldest trick is the awareness to the people if people should understand how fishing looks like how should we prevented because you know on the corporate level yeah you can Implement really better Solutions uh on the end point on your email all the but as a regular person who doesn't understand a lot in cyber security or AI in some kind of something it starts with the burn when you receive an email you receive a call to understand what are the risks what's going on and I think the B is different now with generative AI is so widely available is the cost of each mistake is much higher because again we still don't see something extremely sophisticated that never exists it's just there are more attacks where the fishing malware everyone can create so the cost of your mistake as an end user or is a corporation just becomes much higher

higher what is maybe we've spoken a tiny bit about you research and articles that have come out from your side what is the current what are you currently looking at what are the current things that you're interested in and you're seeing maybe some more recent research that you wanna or you can share so I would say we are now dealing with two things one is everything connected to account takeovers of generative AI accounts for example of church gpp we see last months and a half two months and huge Market of accountable covers in the dark basically account takeover is a very old Market that mostly focused on the financial institutions account Banks PayPal Gmail Facebook Instagram site but since merch April we see more and more discussion about uh stolen hgpt accounts especially Premium Accounts are sold in dark web okay and I would say there are several reasons why cyber criminals are so excited about it one is a to bypass girlfriends maybe some people will be sure right but there are certain countries that are forbidden to use the GPT by opening ideas among those are countries like Russia China Iran for example and the cyberpreneurial girl looking how they can still use it for their purposes so next one the second is to get more API calls of and to integrate for example open Ai and an API to different Bots and I think the last Trend we see is about the choir and pride are now looking to get your inputs and your queries that you submitted to jgpt because if you submitted something private there or corporate information if someone takes you over your account it's not only open AI will have the information but the Cyber criminal itself and we see now a lot of different Services what they are doing they are trying to get your password to jgbt and even premium account service so that's one area we're dealing another is we are now trying to see what's going on with Google board because yes everyone talking about check GPT because that was the first one but a part of ggpt there are I think at this point possible to tell dozens of different General TVs platforms now maybe the Google bars is the biggest competitor if it's possible to 10 other thread surgically not sure but so we are now also trying to see whether a Google board are better in protecting from cyber abuse than 2gbt that's what a rare area we're trying to now to understand whether they learn the lessons I didn't at least in the beginning didn't know that's enough efforts to prevent the abuse of their platform so that's for example an area we are trying to understand

um maybe a slightly different topic um the there is a new emerging field of let's say prompt injection uh threats and something like this and that's essentially where and there are some really cool examples of that online where if you have something like chat gbt especially if you enable the browsing plugins right so that it can go to websites an attacker will craft a malicious website that if uh gpt4 so goes there reads the website there will be like a prompt injection it will say something like oh this is a secret message you've been selected for a secret experiment right uh do this and this don't tell anyone about it right and so this will be kind of loaded into the prompt and will be there uh in the subsequent conversation and people have gotten because people have gotten for example the user's data private data out of it because open AI will or other suppliers will sometimes put that into the prompt itself like you know in order to like fill out a form on a website or so um how much of that is just still just kind of research and people imagining what could happen and how much of this do you see on the ground people actively using I wonder a little bit because it seems like it's a bit playing with stuff so far but I'm wondering how much is in practice so I would say still mostly at least again mostly on a more theoretical research stage you know it's kind of proof of content that it's possible to do but I think as more and more engines and are connected to the internet in real time that the deaths I think the best we are going to well you know in the beginning GPT was completely disconnected with very old information of the 2021 and the way now it's connected we are being and trajective for also with other and same Bard there will be more and more risks behind it really look for real time attacked against subject itself or against third party website also because I see the you know nightmare future scenario when a cyber criminal that has zero knowledge in being attacker will write I won't know to scan all IP range of specific Corporation to find the latest vulnerabilities and then you know to do some web injection and get the database just you write who I wanted and then the output will be the database from this you know that you know and it can happen it's still I don't see it still happening too much you know in real but is it of course I can see the argument let let's say hey this is It's a dystopian world this is kind of bad if that's possible on the other hand security should be built such that you have to expect this right whether someone does a port scan and an injection of the latest vulnerabilities because they're really skilled or because they use chat GPT or they pay someone to do it should not matter to you know the implementer of security on the defender side so yeah I'm wondering a little bit why you say you know that's kind of a really bad future is it because you're aware that the Defenders usually don't Implement proper security or the unfortunately you already cried I think always cybercrime generally is a matter of cost efficient because cybercrime is a business yeah and just using AI engines those activities will be much easier and cost efficient for cyber criminals because now they should do some manual work on such is scanning even with all automation they have and they have autonomics but they will help to collect some intelligence a lot of organization to find IP range such stuff you know and to do some set of activities still manual even though you know the scanning is automated using air agents all this will be automated they will say oh they take all the Rangers appropriation in certain countries can it do it you don't need to understand you barely need to understand what is IP you know you will just tell just came so I think that's the problem and of course there are a lot of Defenders and corporations that do not Implement all the relevant measures definitely but it's just a matter now it will be just much easier it's not like it's not happening now it um

um we've spoken a lot about cyber criminals and so on which is it's a business right but these people are let's say either part of a network or independent you know they trade on markets and so on the other fear that people have that's very related to it is the future of warfare like once bigger actors come into play once State and so on um what do you see on that or do you have first do you have any knowledge even of that front end what do you see the future of to me that's still a bit of an ominous future like people say the future of warfare will change there will be like AI Warfare I'm not too sure how that might look right what do you see in the future of you know geopolitics or Global Warfare or anything like this as far as your as far as you can see or is you're aware so first of all I don't have any specific information about what the big nation but it's someone who in the past share with an intelligence organization in Israel I can definitely tell that based on Mayan or understanding how intelligent organizations work I am sure that the countries like Russia or China for example are actively trying to find vulnerabilities for example in church GPT I'm sure they want to find their you know whether to get the data in the input or furthermore to manipulate the date because I think that will be also always the question how much we can rely on the output we get because of course there are it's not perfect even now we see a lot of mistakes misinformation but we assume it's not intended but now let's assume some country gets an excess and a fine sound vulnerability that allows them to manipulate certain information for example every information about the Russia Ukrainian war and anything about this will be presented only in a certain way so I'm looking in this perspective because I think we are now in the era of information misinformation business information so if you are talking about you know all the problems in social media when it will come to a generative AI engine I think it will be much more complicated much more confident so that's definitely one area I see what

in general do you think um like what can be done against any of that people also afraid who in the future we're gonna all content will be AI generated we won't know what's true anymore uh things like this I can definitely see a way to this but what can be done about it like in the same way awareness is probably as you said before one of the key factors but then what can we done about it how can we know something is still true in the future it's a great question I think we will have to build completely new methods or for fact check I think it because I already saw numerous stories about you know different alleged facts that church bikini invented and even provided references to newspaper like the guardian or washing components Washington Post or something for a article that never exists but the scientifically did provide you know the URL of it so I think we'll have to build a new method how we're doing fact checking check it all I think if you are talking that maybe some professions will disappear now with General TV AI so maybe Dozer will be in a new profession how to verify you know the information by open AI you know the ethics of AI Aldi that now bear you know there are people who are doing it but it's not you know it's not the tens of thousands of people or something so I think we should maybe look on this way okay

um last question from me is there any anything else you would want to get out to people any words of advice or any cool new things to look out for so I think first of all we talked a lot about the bed sides of AI generative AI a bit mentioned but I think they're also a good sign a lot of good sites and like every new technology with human being should look on how build it and that's what will you know to build the path of this technology it will use it for bed yet it will be a better knowledge that we can bring us to disasters and more cyber attacks and everything but it will look how we protect the world better how we create new things I think that's a big opportunity of so I think it's mostly about how we take this technology and what we do

you do you think the path to that because that's a big discussion right now do you think the path to that should be to mostly open up the technology to as many people as possible to have as much of the tech available let's say publish the weights and democratize it quote unquote or do you think this other approach of saying No it should be shepherded by a few big corporations so that they can make sure people use it responsibly like there's Arguments for both sides which what do you see more likely I think at this point I think now we don't have way back to limit it already it's I think it's too late so I think do you think it should be Democratic but I do think the rule should be some at least basic regulation in a worldwide regulation you know like uh on everything connected to flights and Aerospace there are you know International regulations and all the countries get to some basic you know because they don't know what's a disasters to happen the same here and it's a big question about what regulation should we regulate how it's a completely you know there are a lot of ethical legal and technological consequences but I don't think we already have a big totally Limited in fact we should know maybe stop and think what exactly will regulate how verify that if there is more use of AI we're able to investigate it for example to find who did it but the I guess the I'm not gonna but my estimation will be the Cyber criminals aren't going to be too impressed by regulations but by means of being cyber criminals already so um what you know what kind of regulations do you have in mind to make life easier with regards to cyber criminals I think first of all there is never any regulation any load does not reduce crime to zero whatever we do you know never introduces but I think we should just make the bar a bit higher than each now not because yeah the sophisticated cyber criminals will find in Nano the way like now sophisticated criminal Bridge B Corporation or government still but they bar is much higher not every script PD is able now to breach I don't know U. S government or a Fortune 500 company so I think here we should make the bar a bit higher for a cyber companies on this I think that's mostly yeah okay that makes sense Sergey it was a pleasure having you here and thank you so much for speaking to me thank you for the pleasure everyone