Security-driven Rapid Release - Pwn2Own Documentary (Part 4)

Manfred Paul successfully exploited Firefox on stage at Pwn2Own. And in the last video, we learned about how he discovered the issue and some technical details about his bug. So with that knowledge, let's head to the disclosure room where Mozilla hears the exploit details for the first time. This USB stick was just sold for $50,000. What could be that valuable? It's not a crypto wallet, no stolen data, and no state secrets. Just a few lines of code. Beautiful but dangerous code. A zero day exploit in a back room of a hotel in Berlin. The trade went down. They are the ones who bought it. sold it. And they will make it worthless within hours. This is the story of 2 Firefox zero days, the world's most prestigious hacking competition, and what it takes to keep millions of users safe. We already have been in the disclosure room before and the process was basically the same. After we got settled in, Freddy gets handed the USB stick with the exploit files on it to share it with the team offsite. And in parallel, Christian wants to talk to Manfred to learn about the exploit. While Freddy is looking at it, uh maybe you can explain to me what you have in store today. If you watched the previous video, you should be quite familiar with this bug. So, can you understand and recognize some of the things Manfred is about to explain? Like I always do like kind of mathy bugs. This one, the structure is literally called math space. So it was like, so it's in the bounds check elimination, right? So it's a JIT bug. Yeah. And it's like basically an overflow bug, but I mean the bug isn't that the overflow can happen because it's about like additions that are supposed to be wrapping and um but they are treated by this exact linear sum. You can do it on like an integer definition and uh it exacts like okay this looks like a sum of this kind. So if you do like an expression like this then what happens is these additions get truncated because of the OR, the OR gets like eliminated away. So you have like these unchecked wrapping additions but because it still passes unknown to this math space thing, it just assumes okay the 2nd index is like 5 larger than the 1st one. So it extends the 1st bounds check by 5. And if you have now have like a really large array like 2 to the power of 31 or larger then you can actually have like indices there that like the range goes above 2 to the power of 31 and the range check passes but the actual addition underflows to something negative. Okay. Yeah, it makes sense. I hope his explanation was familiar to you from the last video. But more important, Freddy just got the files ready here. He attached the exploit files to the bug tracker and announced it on Slack. PoC attachment is up. So engineers offsite can also have a look at it. And Christian is surprised that this issue wasn't found through fuzzing because the basic building block looks like code a fuzzer would generate. I wonder why fuzzing did not find this particular because I mean it's it looks sort of complicated but it doesn't look complicated enough for me to immediately say like I know why fuzzing wouldn't find this, right? I mean maybe the large array, right? Because it's we have specific code for doing all of this because large arrays have always been a problem and like 1 subtlety is that you if you just do this it will still be inbounds like differentially you could maybe find it uh it will still return something when it should return undefined but the problem is then like this um in int32 to intptr it has this logic where is okay it only uses for like a load index and it's like we can't be negative and it doesn't do actual sign extension and so then this big in constructor also uses like the same int32 extension and but because all users have to be like caught here um it won't set like the can be negative to false so yeah that will be interesting for Christian to analyze why fuzzing didn't find it and then maybe improve their fuzzer to catch something like this before it ends up in an exploit.

But more important to Manfred and ZDI, the organizers of Pwn2Own, is to know whether this is a known bug or not because that will decide whether Manfred will get the prize money of $50,000. I I'm fairly certain we can say already it's not a duplicate because we already checked beforehand if we have any kind of JIT bugs open. We usually don't. We don't keep we don't like keeping these around, right? Yeah. Yeah, makes sense. I mean I guess it should be like a pretty easy fix because in my opinion like the correct thing is to just add like this infinite math space. But um of course I don't know like the performance side of things if there spikes anything or something like that. By the way there's 1 detail about the Firefox exploit and the Pwn2Own competition that I have not really mentioned yet. Browsers nowadays have a sandbox. This basically means website rendering and JavaScript execution runs in a process that is locked down. For example, it's not allowed to read or write files or execute another program. But of course, if you want to download a file or even write cookies to disk, you need to be able to access files. So, this is done indirectly. The content process just tells the Firefox main process to do that for them. This means if you have an exploit here in the renderer, it doesn't give you full access to the system. Usually you need a 2nd exploit to compromise the sandbox, which means in a full browser compromise there are at least 2 vulnerabilities involved. But at Pwn2Own we have only always seen 1 exploit, and that's because both the exploit from Eduardo and Tao as well as Manfred's exploits were renderer-only exploits. If you look into the detailed rules of Pwn2Own, you can see here that for each browser, there are multiple options and both entries had no sandbox escape and just targeted renderer-only. So in theory, they shouldn't be able to pop calculator. The sandbox would prevent them from executing another process. And that's why when you target renderer-only in the competition, Firefox gets started with the MOZ_DISABLE_CONTENT_SANDBOX flag. This disables the sandbox and allows the exploit to launch the calculator. I just thought this is a small but important detail to mention for anybody not very familiar with browser exploits. It's really impressive to find such exploits, but there are still a lot of work to get the whole way full browser compromise. And that just shows that sandboxes are working. They really make it harder. And so for Mozilla, this is a very important line of defense. And in fact, they actually invested time to improve the sandbox over the years. And that's also why Mozilla wants to use this opportunity at Pwn2Own to ask these security researchers about the sandbox whether it made it harder for them. I think we discussed this briefly before that you didn't try to look for a sandbox escape or you I mean really shortly but like there was not a lot of time and like especially with like the changes that were recently it was like didn't think I had enough time to write. So, you didn't try to find one or you didn't found something that was close to a sandbox escape? I mean, I think I looked at it for like 1 or 2 days or so, but like so, if you feel like you have something that would lead you somewhere, even if it's hella interesting, I don't think I have anything there. I have like 1 small thing in the renderer that I might like report uh separately. But report yeah, it's like a different thing. It's like I don't know if it's just correctness or we know there are some shortcomings in the recent work we've done that we still have to deploy right despite the major um major milestone. So if you would keep looking I think let us know. Specifically we do have a bounty an expert mitigation bounty that says if you can provide additional information even if there isn't a first but like even if you don't have a first but you can somehow mitigate um bypass all mitigations in the parent process without a content process might definitely also interest or if you can just yeah bypass the CSP without actually having an injection point that would also just be super boring. Maybe you notice Christian is really focused and typing a lot. So what is he up to? Yeah, you bisecting. I can't actually reproduce it in a shell yet. Okay. The proof of concept does not reproduce in any way. Did you add the begin? Yeah, at the beginning. What would be the symptom? Would it be a crash? Would it trigger an assertion? Oh no, it should just Oh, I think it should crash. Christian is using JS Shell, which is the JavaScript interpreter as a standalone process, not the whole browser. And he cannot reproduce the issue on the screen from Freddy. You can see the backtrace of the crash in the browser. So obviously the exploit should work. So what's going on? Did you do Spectre mitigations off? And uh mitigation? Oh, Spectre mitigations. Yeah. And the JS Shell have Spectre mitigations by default like the browser doesn't. Yeah. Now I get a crash. Okay. With Spectre mitigations turned off, Christian was able to reproduce it in JS Shell and we were also slowly wrapping up. On the way out, Christian and Freddy started talking in German

and I overheard them mentioning the Spectre mitigations again. It turns out the browser itself runs without Spectre mitigations. So, if JS Shell has it enabled by default, and you couldn't reproduce the issue with it turned on, then maybe that's the explanation why fuzzing JS Shell didn't find it. But that's just my theory. Those are the things Christian will look into later. This time, we didn't go back to the office and sat down right in the hotel lobby at the conference. By the way, it's also Saturday, but that doesn't matter much for Mozilla because they are using Pwn2Own as a fire drill to practice when a real in-the-wild zero day gets discovered. So, they are not waiting until Monday. Freddy being the incident manager again handles some of the organizational stuff like preparing the Mozilla advisory announcements, but then also spend some time looking into the exploit to better understand it. At some point also Manfred joins us eating lunch and we talked a bit more about the bug which was really great for Freddy and Christian because his insights are very valuable and appreciated. Uh so what we also did is we looked at uh could this bug be found by other means by automated means that we use internally like fuzzing for example. While this was not the case for yesterday's bug, today's bug looks a bit more promising in terms of we could find this with fuzzing. We found out that the bug has been around for 4 years. So the question arises why did no fuzzing tool find this even though we have lots of fuzzing tools running on this. So there will be in the aftermath there will be a deeper investigation. Are there any kind of factors that prevent current fuzzing tools from finding this particular issue and potentially other issues. So this is a standard procedure that we're going to investigate. How can we improve our tools uh to potentially cover this one as well? Any hunches what might have been why did fuzzers get unlucky or do you have a intuition for what maybe could be I have an intuition right now so we have multiple fuzzing tools the test case is not super complicated but it's also not super trivial um the tool that would be mostly suitable for this case would be Fuzzilli uh because this is a JIT optimization bug and Fuzzilli was actually written to target these kinds of bugs uh the problem is that in debug builds this test case takes quite long to actually run and uh this actually exceeds Fuzzilli's default runtime. In an optimized build, you get a crash much faster, but we actually don't have Fuzzilli running with purely optimized builds right now. We do have some other fuzzers running in the setup. But we mostly rely on debug builds because they often give us much more value in terms of uh assertions and these are also not pure debug builds. We use debug-optimized builds. They are usually very comparable in terms of performance. We still have to investigate why the debug-optimized build is so much slower in this particular case. That I don't know. But this could be play a role why Fuzzilli wouldn't have found it. Let's recap the timeline really quick. The exploit was shown on stage around 12pm. 20 minutes later, we got into the disclosure room. So around 12:30, the time started ticking for Mozilla to create the patch. Jan, the JavaScript engineer who fixed yesterday's issue, also mentioned that this bug looks quite old and seems to go back around 4 years. But then, not even 1 hour later, Jan said that the fix could be very simple for now by disabling optimization. If we can collect some data and confirm that optimization is not very important for performance, that could be a good emergency fix. Shortly after, Jan pushed exactly this change, an early return in this linear sum function. In this special modulo case, we've shared the write up and the exploit with everyone involved. We have a patch up that's waiting for review. It already went through a security review. We believe this is fine to land when it has been reviewed by someone else from JavaScript. Uh we started writing security advisories for TBD but I guess tomorrow um for new versions of Firefox 138. 0. 4 and the corresponding version numbers for extended support release which I don't know from the top of my head. Um those are also ready to land. So I think we just really wait for the patches to be reviewed, to be built, and then um yeah QA to test the new build before we actually call it a new version of Firefox. It seems like it's less of an issue than what we saw yesterday in terms of bringing up a reasonable patch that is that looks safe. Um, it looks like there's an optimization that we can just try to remove for the time being with an early return, which seems to be a very low-risk patch in terms of security. We will still do some shallow fuzzing for the time that we have while QA is looking at it as well. Um, but I think it should be much smoother than yesterday where we have to do a variant analysis and look for similar issues. Yeah, since I mean Freddy already mentioned that the patch is very small. It's an early return, disables an optimization, there's practically no impact on any other system except performance, and uh performance we're going to measure of course. Uh but for a security fix that's likely acceptable uh for now

and we can uh check out what we do about performance uh in a regular release cycle. Okay, it seems like they are going for this emergency fix now. But that is not all that has to be done for release. Here's the Mozilla Wiki page about the release process. Under normal circumstances, the code committed in main leads to a Firefox nightly build every 12 hours. Every 4 weeks, those changes get merged into beta. And if then beta turns out to be stable for 4 weeks and the final build is validated by quality assurance, we get a release. So it can take 4 to 8 weeks for a regular patch to be pushed out as an update. But an incident is different. Mozilla calls this a chemspill. Chemspill is a term used to describe a security-driven rapid release. But still they must make sure to not break Firefox by pushing out a broken release. So incident process is at this point really just patches landed. So next step is review the patch um make sure that it lands in CI runs through all of the tests. Then we have um test builds which QA can test and once all of that has happened we'll ship the release to all of our users. I'm thinking that will maybe be tomorrow. Uh so in completely independent of the uh of of this release uh cycle uh we actually discovered that since yesterday evening I believe or yesterday night uh there is an issue in the synchronization of 1 of our repositories. This has nothing to do with the security fix that we're deploying but it's of course causing issues now in the release process. Uh so people are working on fixing this synchronization so we can actually uh deploy uh the fixes to release. Right now it's not a critical issue though because we are still waiting for review of the 2nd patch and as long as we don't have the review for the 2nd patch, we can't land anything anyway. So we assume that these this work will now run in parallel. It has been escalated and we will likely fix the repository issue before we actually get the review for the patch. Now that everything is going towards a release, Dianna Smith, a senior release manager at Mozilla, also joins the discussion. And if you read these Slack messages, you can get a sense for that creating a new release is not trivial. But eventually, the patch got reviewed and we are 1 step closer to the release. At this point, both of the bugs have been fixed. We've handed over most of the tasks to release management and our folks in North America. Um, I believe we've already gone through CI and completed all of the testing. There will be a bit of manual testing and QA from our side and I guess we can just call it sort of done from my perspective and I'm hoping we'll get to release uh late at night my time or early tomorrow. Yeah. And there was not much else we could do anymore. It was now around 7pm and people in the US time zone took over. At home, I kept an eye on the Slack communications. And as somebody who has never really worked in software engineering, this was really interesting to observe, especially seeing this complexity behind the release and everything it affects, how many people are involved and so forth. And so, let's have another look at the timeline. At 7pm, a release build got started. Nightly builds with the patches were already available. ESR builds, the extended support builds, also got started. And in parallel, there was extensive testing and QA. And then just 30 minutes before midnight, the new version got released. A new Chemspill security release 138. 0. 4. Basically, this means for Manfred's exploits, it took Mozilla less than 12 hours from hearing about the zero day to pushing out a release. I think that might be a new record. The year before it was 21 hours that they got even recognized for. Here's their advisory with both exploits. Edoardo Binn and Taoan from Palo Alto Networks with an out-of-bounds access when resolving promise objects and Manfred Paul with an out-of-bounds access in the linear sums optimization. Anyway, all that is left to do now is I open Firefox and there it was. It's downloading the update and then the famous button appears restart to update Firefox. Wow, what a journey. I feel so privileged that I got this opportunity to be allowed into the disclosure room. So, thanks so much, ZDI, for trusting me and thank you Freddy from Mozilla who had this idea originally. I hope this was also interesting for you. I would really love to hear about what you learned from this entire series. So leave a comment below. Thank you so much for watching and here are a few last words by Freddy. So if any of this has been interesting uh sometimes we're hiring careers. mozilla. org. We also have a blog attackanddefense. dev where we post internals for security researchers, hackers, uh bug bounty hunters, and yeah, we do have a bug bounty program. I believe we pay nicely for bugs. We don't need exploits. Come check it out. Have you checked out our online training platform, hextree. io. In this crazy new world of AI and agents, you might think that you don't really need education anymore because the LLM can just hack a website for you.

But even if I strongly believe that is the future, that is the case, I still think it's extremely important to have a knowledge foundation because only with a strong knowledge foundation and good understanding of technologies and all these different systems, you are able to actually prompt the LLMs and the agents effectively and you will be able to realize when they make mistakes. So I still think it's very important to educate yourself and don't become lazy because of AI. Our courses on Hextree are basically micro courses. They are very short and each individual video is very short because we don't want to waste your time with an 1-hour long boring screen recording. Also, our courses focus on fundamentals. And by that I don't mean the beginner entry level stuff. I mean actually foundation knowledge to build up to really complex understandings of systems. So you really need knowledge and especially foundational knowledge to use these tools effectively. And I hope with our courses at hextree. io we can teach you exactly that. ```