Is open source safe? Featuring Mixture of Experts

Hello and welcome to a very special episode of Security Intelligence. I'm your host Matt Kazinski and today we have a crossover episode with our sister show Mixture of Experts IBM's AI podcast repping. Today we have Martin Keane, master inventor, and Gabe Goodart, chief architect AI open innovation. And in the security intelligence corner, folks, it's Jeff Kroom, distinguished engineer, master inventor, data and AI security. And we're doing this little crossover today because we want to tackle questions about open source. How do we enjoy the unique benefits of open source in AI and elsewhere while navigating some of its unique risks? Gabe, I will start with you. Tell us how you feeling about open source. I've got to be the person you're referencing on Mixture of Experts that's all in all the time on uh open source. That's it's where I live. It's what I do. So, come at me. Let's see what we got here. — I appreciate that. Martin, how about you? Where do you land on the scale from open source is great to open source is terrible. — Also a big open source fan and gave, you know, fair warning when it comes to cyber security people like Jeff here. No matter what you tell them you're doing and how good it is and useful it is, he will find a way to tell you that this thing is dangerous, it's reckless, and that they've solved this problem 30 years ago and nobody's listening. — Absolutely. And you're welcome. I have no doubt. I can't wait. — Jeeoff, is that what you're going to tell us? Where do you fall on open source? — Well, I almost feel like I should because we'd get better ratings if we had a food fight here. Uh it if we all just, you know, join hands and sing Kumbaya. I don't know that anybody's going to care. So I'll take uh cyber security people are contrarians by nature. We have to be. I mean we're always looking at everybody else is looking at how a system will work. Our job is to consider how it will fail and then try to prevent that from happening. So uh in general, believe it or not, now I'll say this and then we'll just wipe this off of the recording. I'm actually in favor of open source. What I'm not in favor of is some of the over grand grandiose kind of claims that are made for it when it comes to cyber security. So that's what I would uh would tap the brakes on. — That makes perfect sense to me. And let's circle back around then to Gabe because Gabe, you're our kind of probably strongest soldier in in open source's corner. We all kind of dig it, but you've said, "Look, I'm all for it. " What are the benefits? Why does it why you such a champion for it? Tell us a little bit about that. In this era of AI, we have never been closer between the science and things that are genuinely usable. And I think that is happening in large part because of open source. Um, and because the science itself, the math, the tensors are actually remarkably close to something that's usable as opposed to needing a whole lot of layers between those and something that people can get utility out of. uh and because of that I think open-source relative to most other innovation waves is where the vast majority of the actual innovation is happening because science by its very nature is open and because the science is so close to utility um it's actually very tightly coupled with the source code that's generating utility. So I think that's why open source is such a key driving force in the AI revolution we're sitting in today. Um, now Jeff 100% agree. Open source done wrong is terrible and there are so many projects out there. In fact, um, a close collaborator of mine is working in a project that he just moved over to. Um, and having come from a very principled, uh, open- source project that managed security contributions, trustworthiness very carefully into this new project, it he's telling me about all the smells he's got and all of the vibe code Spidey sense going off right and left that this is not a secure project. So I believe strongly that open source can and should be a force for good in this world of AI, but it is not a cart blanch solution to everything. So — and we're having this conversation about open source at a time when there's a lot of debate around questions of who gets access to what models and when. Right. Martin, I'm wondering how you see open source fitting into this question about model access and who gets access to what. Do you think that's a kind of guiding light for us in terms of who can see things and when? I don't What are your thoughts there? — Right, we we've sort of got this comparison between the Frontier models that are released by labs that are closed models and up until now they've been quite democratized that they're available to everybody. You sign up, you pay your $20 a month, you get to use the latest, smartest model. And we're just starting to see now with bottles that are coming out that that's no longer the case. That you have to be on the list. either in the consortium of companies or you have to be uh you know at least approved by the lab to use it. So that of course is not the case at all with an open-source model that you can you know

find on hugging face or wherever it is that you take a model out and you run it on whatever hardware you want to run it on. So we're actually seeing I think a bigger and bigger divergence now where some of these frontier models are really super advanced but also very locked down as to who has access to them. But you know if history is anything to go by the open source models they will they'll catch up to where these frontier models are today pretty quickly and they're not going to have the same restrictions on them. So it's going to be a really interesting time for these models. Jeff, do you have any concerns on your end when it comes to model access in the sense that the bad guys can get their hands on this stuff? Uh, maybe just as soon as the good guys can and with open source models, maybe they can see even more of it. Is that something that you ever get worried about or how do you feel there? — It's a thing for sure. Uh, there's a part of me, by the way, even though I'm a cyber security guy, I'm also have this side hustle of being an adjunct professor uh at NC State University. Well, the academic in me, which is just a small part, I'll have to admit, but that part is yelling out information wants to be free. So, you know, we don't want to keep this stuff in the hands of just a few. The cyber security side of me also is reminding myself that security through obscurity is not an effective model either. So, the idea that I'm going to keep secret information and that's going to make my system more secure is not going to work. You know, we have this idea in cryptography that's called Kirkoff's principle where it basically says that the only thing about a crypto system that should be secret are the keys. In other words, the algorithms, the generation of keys, you know, all of the where they're stored, all none of that should be a secret. The only thing secret are the keys themselves. Because if the algorithm or the cipher itself is uh is something that's secret, well, probably it just means not enough people have looked at it and really taken it through its paces and therefore it's got vulnerabilities in it that we just haven't shaken out. So, on the one hand, the argument that putting this stuff out there, uh the argument a thousand eyes means that we've got a thousand eyes out there inspecting all of this and they'll find where the problems are. That sounds good in theory. It's a little naive when we start talking about systems that are as large as the ones we're talking about because a thousand eyes wouldn't be nearly enough to look at a billion parameters. So, you know, that the idea that we can just by making it open source that it becomes more secure, that's the part where I push back. Um, I remember a debate I had with a colleague 25 years ago. He was arguing that Linux was by definition more secure than any other operating system because it was open- source and because the way the kernel was designed, you couldn't have malware on it. Well, how did that age? I mean, we've had countless examples of malware in Linux. Linux is a good example of a system that is securable, but in and of itself is not necessarily secure. And the same thing those same lessons can be applied to AI. They can be securable if we have more people that know about it. But just because everybody knows about it is no guarantee because this it's too big. It's scaled too large for enough competent eyes to really uh analyze all these things. But the bottom line is trying to keep the information secret. Eventually it leaks out. Eventually it always leaks out. So the idea that we'll keep it just for us, you know, we'll just keep a secret. The four of us will know this. We'll not tell anybody else. Uh, somebody's gonna have loose lips. Somebody is going to have it on their system and somebody's going to hack into their system and then they're going to get a copy anyway. So, that can't be what we're leaning on when it comes to security is what I would say. — A thousand eyes being on a thing doesn't necessarily mean that thing is like totally free of vulnerabilities and we found every vulnerability. But then my question becomes, you know, the thousand eyes might not make sure that every vulnerability is closed, but it does also lend itself to like Gabe said, uh, innovation, right? And I was wondering, Gabe, if you could talk about that angle of things in terms of, okay, when you have more hands on a model, we can do more things with it. Could you walk us through some of that a little bit? — Yeah, absolutely. So before we dive into what more hands on a model mean, I want to just draw a very important distinction that I think often gets missed in this conversation. It's the open- source of the code that implements the systems that we are all relying on today. And then it's the open weights of the models and the architectures of those models as defined by their configurations. Now those things are coupled because those architectures and weights have to get loaded up by software and actually executed somewhere. But um in and of itself, the weights of the models being open um h has a whole different set of

pluses and minuses than the software itself being open. So I want to focus on the software for a minute because fundamentally this is no different than exactly what you're saying about Linux, right? It is a securable ecosystem. It is not secure by default, right? And the same thing with Kubernetes, with literally any other large software system that's a composition of a different bunch of different open source projects. Um, everyone has its attack surface, a intelligent implementation of that system needs to have those attack surfaces addressed meticulously and it needs to have a good policy for staying up to date when new vulnerabilities are discovered. So in that sense, the AI stack is just software. Now the models themselves pose an interesting different challenge. So I think one of the big security arguments in favor of closed source models is that the software and the model are both essentially uh inaccessible without whatever software layer is sitting in front of that. And that allows the authors of that system to not only manage the vulnerabilities of the software itself, but manage the potential risks that the model's trained capabilities would expose because they can put guardrails in front of it. on the back half. They can detect usage patterns that would indicate malfeence. They can do whatever they want at that software layer to try to mitigate the harm that the model's capabilities would have. Once those model weights are out in the open, even if the model has trained into it some rejection uh capabilities, the first thing that everyone on the internet's going to do is obliterate that model and take all those rejection capabilities out, right? Everyone's gotten really smart about figuring out, okay, what is the right vector that I have to, you know, tweak on in the embedding space to just crank out those rejection samples and now I've got a free-for-all model that can do whatever I want it to. So model weights in the open do have that inherent danger that somebody will get at whatever is underneath like the base training level the sort of unfiltered training with they can remove whatever you know post- training has been done on those models pretty effectively and so that's certainly a risk so I think um you know that doesn't mitigate the fact that this will get out there and that the science and the innovation still wants to push forward because the innovation front of it is that once somebody puts a model out there it's not us the weights. It's the architecture and it's the software that implements that architecture. We've seen this in a recent round of innovation around the attention mechanism and using linear attention. We've got these great new models um coming out that scale much better with long context and that's because this was all done with open science and people piggybacked off each other and thought hey how can I take this interesting idea of a recurrent linear layer and I can hybridize that with some attention layers and oh maybe I could even tweak on that linear layer to use a different you know bunch of matrix m so all of this is building on itself with the innovation front so you it's hard you I don't think you can have both at the same time release how the model works and also don't run the risk of those models becoming obliterated and people understanding how they work. Um, so again, it's sort of it's not a one-sizefits-all with the open weights models because uh we certainly need to then mitigate against those threats proactively that the moral capabilities, you know, expose. — And Martin, I saw you kind of nodding along there. Did you want to jump in there? You want to add what what's your thoughts here? When we think of open-source and AI, we're often thinking of, oh, open-source AI models. So, you know, Mistral or Llama or Deepseek or something like that. Uh, those openw weight models. But, as Gabe says, there's, you know, there's a these are built on the foundation of software and existing projects that are already open source. But I think you could even go one step further than that. And really the argument here is not should you adopt open- source models versus the closed source foundation models from the frontier labs. Um because there's an extra layer here which is to say that the models themselves are working with all sorts of open-source capabilities as well. So if we think about MCP for example, model context protocol that allows a model to go out and basically invoke another service and that came from a frontier lab that was anthropic that came out with that and then donated that to the Linux Foundation. And that means that today I can take one of my open source models and I can use MCP and I can go invoke whatever it is I want to invoke, you know, an MCP server. But I could also use a closed source model and do that exact same thing as well. Same with skill. md. So there's an agent skill spec that is now an open standard that defines how these things are actually defined, how you tell an agent how to do something. And again that is an open standard that works with openw weight models and also the foundation models as well. So there really is no getting away from the fact that open source is foundational to everything in AI now even if we're talking about models that were actually frontier closed models as well.

— Now speaking of the kind of nuances of open source right we we've mentioned the ways it can be incredibly beneficial lend itself to innovation and then we've also mentioned some of the risks that come along with that. And I want to shift our gears a little bit then to that side of things. How do we manage those risks? And Jeff, I want to start with you and I specifically want to pose you the question because I know you've talked about it before. Is this a place where AI can help us secure our AI? And if so, how? What are your thoughts there? — Yeah. So, you're talking about nuance and then you asked the guy with the sledgehammer to come in and talk about nuance. So, all right, let me go. This is one of those things actually it is a very nuanced question and the people that are all 100% this is the way to go or 100% that's are probably missing a big part of the argument here because it's not as simple as that. I'll say I'm never going to be in favor of a proprietary thing saying that it's more secure because it's proprietary proprietary. What I'm going to say is the reverse is not necessarily true that it's more secure because it was open source. So designed in a secure way. Uh and you put the right kind of controls in place and hopefully open source will allow us to discover those kinds of things. Going back to the model discussion, look, one of the big things in security has always been this word, this concept of trust. You know, can I trust the system? Can I trust that you're who you claim to be when you're trying to do this transaction? Can I trust the stuff that I wrote to the database is still the same thing that I wrote to it a day ago or somebody has changed it? Can I trust that someone hasn't made off with all of our confidential information? One of the things about trust when you're talking about trustworthy AI is transparency. How can I trust something that's not transparent? If I don't know what's going into it, then that's not trust. That's just blind faith. And that's not a technical, you know, pillar of trust is blind faith. You know, I think because it's been exposed to sunlight, therefore surely some smart people must have found all the problems. Again, when we look at some of these bugs that have been latent in open-source code that have been around for decades means all the smart people in the world still didn't find that. But guess what's going to be able to find it? AI. because AI will be able to look at that source code and be able to identify these things. But I'm going to say even though that's kind of top of mind because of what's happening in the news right now, I'm going to say this is just a variation on a theme. We've already had this issue if you want to call it that previously. You could give me just the executable code with no source code, the most proprietary thing that's ever happened, and I run it through a decompiler and it's going to give me back machine language code. And then if I want to, I could find vulnerabilities in that. Or I could use an LLM to translate that machine code into original higher level language and its equivalents and then I run that through a vulnerability scanner. So even if you only in other words it's not just because the source code is hidden that it's secure because I can essentially uh define what the source code would have been I can reverse engineer the source code not maybe not line per line but function by function what it does and that has existed that kind of capability has existed before we were even talking about generative AI so this just puts it on steroids it just amps up what that is but uh for a lot of people I think they've come to the party late and think this is a brand new threat that we've never seen before. Nope, we've seen it before. Um, now it's just a little bit different. — See, I knew you'd say that, Jeff, that this was a problem that was solved years ago. We just weren't listening. — Of course. Of course, if you all would just listen to me. Yeah. — All the cyber security issues in the world would be fixed if we all just listen to Jeff Kroom. We all know this. Gabe, I want to swing around to you now and kind of ask you, you know, for your thoughts on either what Jeff just said. I saw you just raised your hand, so I don't know if you had something you wanted to jump in with there, but go ahead. What's your thoughts here, Gabe? — Sorry, I I'm dealing with a bunch of internet lag, so I had to raise my hand to jump in the conversation here. I thought he was just waving. — I really want to tie together two things that Martin and Jeff said because I think there was actually a really interesting point that you guys brought up um in combination, which is Jeff, you mentioned that everything is all about trust and where that trust boundary is. And Martin, you mentioned the fact that there's actually a whole lot going into these model systems even beyond the open source software that's running them and the open weights or closed weights models that are actually, you know, doing the math. And that is the context. And I think this is something that, you know, all of the big ability to reverse engineer binaries is great. It's on steroids. Okay, we got to be careful of that. But what I am even more scared of is that there is a view where especially where you have agents operating with autonomy, the agent loop is essentially a code interpreter and

the code is literally any text you pass through it. And so we're all very familiar with this concept of, you know, having to sandbox your untrusted code. Well, now the internet is your untrusted code. like literally anything on the internet can instruct your interpreter which is your agent loop to go do whatever it can and depending on what tools that agent loop has access to there is a lot of damage that can be done and so uh I think this is a net new vulnerability that um is very ripe for the security eye about that trust lens right because I think people are used to sort of implicitly trusting textual unstructured data because it's not actually ever getting directly executed. It's being filtered through some kind of mechanism that's either going to programmatically extract signal or manual, you know, it's going to go to eyeballs and eyeballs are going to extract that signal. It's going to be a human brain. But when it's actually triggering the reasoning that then takes action in an agent, that becomes a huge attack surface. And we've seen that immediately with OpenClaw and other like all bets are off nonsecure uh systems and again I will point out that OpenClaw is a securable system but it's very not secure by default. Um and uh there are already everybody in their uh uncle is trying to claim that they have the secure implementation of openclaw and most of those things are focusing on some of the security problems like sandboxing the execution, managing the tool permissions and curating a trusted set of skills. But what I haven't seen anybody talk about is managing the wild world of untrusted data that's going to get slurped in at runtime and what that's going to actually do to prompt injection. So if anybody can solve that problem, I am all ears because that's the one that keeps me up at night. — Let me get this straight. Gabe, are you suggesting that not everything on the internet is true? — Because I I'm taking notes. — Absolutely. — All right. I learned something today. In fact, some things are deliberately untrue. I would have never considered that. — And if look, if we knew how to solve that, this would be the most amazing podcast ever published. Everyone would love us because we solved the prompt injection problem. No one has solved that yet. Martin though, I want to move on to you and just start get your thoughts as we start wrapping up here in terms of all right, so we want to move forward with our open-source uh in AI, whatever that might look like. We want to use it responsibly, but what needs to be top of mind as we're doing that? like what do we need to be thinking about to make sure we're getting this right? What were your thoughts there? — Yeah, so in addition to the model itself, context problem which Gabe is talking about there, we've also got the problem of interpretability with these models that a model is made available but large language models are generally not interpretable in the same way that other open-source software is. So, as Jeff points to, we can reverse engineer stuff. We can go look at the machine code that was generated from uh a particular open source project, figure out what it's doing, why it's doing it. We cannot do that with large language models because there are so many connections, all of these model weights that there is no good way yet to interpret how a model is going to take an input and come out with its output. So, how do we know that we can actually trust a given model when we can't actually go in there and look to see how it's made? So, I think that just adds an extra thing to be a little bit concerned about. But that is a concern that applies to open weight models and to the closed ones as well because even those foundation models, there are labs doing all sorts of interpretability studies on that stuff, but nobody really knows how these things actually fundamentally work. Just a little plug for another one of our friends on Mixture of Experts. Uh I actually Chris Haye has been pinging me on the side with his wild crazy ideas and he has actually figured out how to unbox some of these open weights models and treat them as a knowledge graph database and actually get that explanability out of them which I think is really cool. Um, so that is a bit of a the col point in the column of open weights models is that you actually can start unboxing what those weight matrices are doing and what impact they have on the final output because you can tweak and poke and turn and change a little weight here and there and see what the change is. And you can actually get quite a bit more interpretability out of that box. But of course, that comes at the cost that somebody could also go in and do exactly that same process and make them do much worse things. So, uh, it's a double-edged sword there. — I expect you to fix the hallucination problem then in the next couple of weeks now that you have this capability. — Oh, boy. No pressure. No pressure at all. — Folks, I wish I really wish we could keep going, but unfortunately we are out of time for today. Obviously, there's so much to say on this subject and I just hope we kind of got across to

folks. Uh, but I want to thank our panelists. It's Gabe and Martin and Jeff for being here. I want to thank the viewers and the listeners. Thank you to our producers. Subscribe to Security Intelligence and Mixture of Experts wherever podcasts are found. And stay safe out there.