Germany Was Just Deleted From The Internet

Germany was temporarily deleted from the internet. I don't mean that literally, of course, butde, the tldd, tople domain. That thing you see at the end of a URL like. com. net. gov, or in this case, a CCTL, a country code tople domain like au for Australia, for the United Kingdom, JP for Japan, or in this case, the problem. de. de DE was almost completely inaccessible for a period of a few hours from around about 10:40 p. m. to about 200 a. m. Germany time. So, it didn't cause any major disruptions amongst most people. However, for the night owls out there, yeah, this was a pretty big deal. In the biggest duck up since Germany was internet, the root DNS forde is unavailable. Everything is down here. I'm sitting in a train and can't show my ticket, but the conductor also can't check it. Pretty much everything that used was completely broken, completely inaccessible. I feel kind of bad. You know what? I feel very bad for any of the CIS admins who got a very angry call late at night because whatever corporate website you're managing was no longer working. But there is one very important question with two very different interpretations. How did this problem happen? As in, I thought TLD were just some letters at the end of a URL. How do they just stop working? And on the technical side, what actually caused the problem here? As put by the one and only Jeff Gailing, it was DNS. Now, that's a good surface level answer, but what does that actually mean? What actually happened here? DNS stands for domain name system. When you want to connect to a website, typically you enter what is known as a URL. This is a string of characters that indicates the resource you want to access. For example, www. youtube. com, the site you're probably on right now. This is a URL. But on the back end, this is not the language the machines are speaking. They speak the language of IP addresses. So if we go and ping this, we can see the IP of 142. 250. 124. 93. However, YouTube being a very large website, if we go and run that again, we get a different address 192. 178. 187. 91. Let's try again. that is different from the original one. So, a third IP address. Unless you've got it behind something like Cloudflare, your random little blog probably only has a single IP address. But when we are looking at a big website like YouTube, there is a reason they want to have multiple and you could just enter the IP address and connect directly to the site. But there's probably a good reason they have multiple of them. So there needs to be a system that can map the YouTube URL to that series of IP addresses that at least in part there are other things that go into it as well is DNS. One of the initial ideas here is this was a problem with the name server. This is a part of DNS the domain name system and Cloudflare has a really good post on this. A name server is a type of DNS server. It is the server that stores all DNS records for a domain, including A records, MX records, or CNAME records. What these are out of the scope for this video, but I will leave this page linked down below if you want to go check it out for yourself. Almost all domains rely on multiple name servers to increase reliability. If one name server goes down or is unavailable, DNS queries can go to another one. Typically, there is one name server and several secondary name servers which store exact copies of the DNS records in the primary server. Updating the primary server will trigger an update of the secondary name servers as well. Whilst it was an initial theory there was a problem with the name server, this would also cause the site to be inaccessible. That ultimately ended up not being true. Instead, it was a problem with something known as DNS SEC, domain name system security extensions. Whilst it might seem like they do, TLD don't just exist in the ether. You can't just say, "Oh, I'm going to use. com or. net oru orjp or lii or in this case. de. " All of these tld have some sort of body that is managing that TLD. All of them do it in various different ways, but there is some sort of body that manages who uses the TLD, what the rules are, how it operates, all of this stuff. In the case of DE, this is done by Denick.

They operate out of Frankfurt, Germany. And what some technical users started noticing, likely some of the CIS admins who are awake very late at night trying to deal with this problem, is they were seeing malformed RIGs. These are cryptographic signatures assigned to these. de domains. So the point of DNS sec again that is domain name system security extensions is basically to ensure that the DNS records you're seeing are authentic. They've not been messed with in any way. So there have been cases of things like DNS hijacking where you basically redirect the user to a page of the hacker's choosing. So you think you're going to www. youtube. com and there is a man in the middle and they redirect you over to you've been pawned hackers domain whatever place they want to send you to and maybe they make it look like YouTube oh you're supposed to run a command or whatever it is they want to do they've now got you on their website. Also there is DNS spoofing where it's basically the same idea. You think you're going to www. youtube youtube. com and what they do is change out the IP address to where you're going. So you still think you're at YouTube but they've changed where the destination is and these are your textbook definition of a maninthe-middle attack. Now there are various other pieces of technology that stop these attacks from happening. DNS sec is just one of them and the way it does this is by providing cryptographic signatures those RR sigs and public keys DNS key to compare the signature against. So if the signature is invalid then you're obviously seeing an invalid domain. Something has been done here. Someone has messed with this. For you as a user with a computer to actually make use of DNS, you need something known as a DNS resolver. Now, most people are using the one provided by their ISP. Maybe you're using something like 1. 1. 1. 1. This is provided by Cloudflare. Or maybe you're even going and hosting your own. But for DNS SEC to function, there has to be a source of truth. Each DNS zone is trusted because it's verified by its parent. But there has to be somewhere where that stops. There has to be this is how we know this is valid and in the case ofde that is with Denick. What appears to have happened here is through a bad update through a bad config change Denick basically started generating malformed signatures those RR sigs those cryptographic signatures that are verifying that this DNS record is a legitimate DNS record. What's interesting here though is those public keys, those DNS keys, those appear to be correct. Nothing was wrong there. It was just the signatures themselves that were wrong. This resulted in any DNS sex sign domain being basically inaccessible because if the signature is wrong, then you can't use the public key to validate the signature because it's always going to return this is a wrong signature. But it wasn't just the individual domains themselves that were failing. There arede domains that are not DNS sex signed. However, they were in a DNS zone that was DNS sexigned. These zones have a DS record. And again, if that record cannot be validated, the whole zone is now invalid. This resulted in basically everything failing to resolve because pretty much every sensible DNS resolver is using DNS SEC if DNS SEC is available. So pretty much all of them were just failing to resolve the signatures. Even though the sites themselves working perfectly fine, their hosting was fine, the servers were fine, all of that stuff was good, the DNSX signature was failing. So it just assumed, oh something is wrong here. This is not valid. Someone is messing with this. You're not going there. But some D domains were still accessible during the situation. That part is not yet fully understood, but is most likely due to caching. Whilst this was all going on, Denick published a status page DNS sect disruption affecting D domains. Denk Eg is currently experiencing a disruption in its DNS service for DEU domains. As a result, all DNSX signed DU domains are currently affected in their reachability. Now, as we discussed, this was not entirely true. It was also affecting zones as well with domains which were not DNS sexed. If you'd like to dive more into that specific topic, there is this great thread over on

Masttodon. I'll leave it linked down below. Be sure to go and check it out. Whilst the name web and the way we interpret the web might seem like this big decentralized thing and in many ways it really is at the foundations of this technology are these very centralized pillars and if these pillars go down things go really badly. Now in the case ofde in the case of Denk this is a very rare occurrence so rare in fact that people initially weren't actually blaming Denk they were blaming Cloudflare because you know Cloudflare has had uh quite a few outages over the past year or so. In this specific case though, Cloudflare is entirely innocent, but they did go and publish something on their status page, resolution issues forde domains. Basically, what they decided to do was temporarily disable DNSC validation forde domains with their 1. 1. 1. 1 resolver. So, if you tried to connect to any of those domains, basically it wouldn't validate the signature and would just allow you to connect to it. Certainly not a perfect or a long-term solution, but with the goal of getting things functioning, getting websites back online, this was a good temporary thing to do. There's one thing which I wanted to save for the end. Something which is probably not related, I assume. I hope. But this took place on the 5th. 7 days from now. There is a scheduled outage by Denick. Do I think this is related? No. However, the timing is very amusing. With all that being said, my love goes out to the CIS admins out there who got called very late at night to fix a problem that was not your problem to fix because you had nothing to do with it, but your boss didn't know that. So, you still had to come into work very late at night. So, get some uh get some claps, get some W's in the chat for all of the CIS admins who had to stay up very late to fix a problem that they couldn't fix. Anyway, if you like the video, go like the video. Let me know your thoughts down below. Were you one of the CIS admins who got called very late at night to fix this problem? I would love to know. So, if you like the video, go like the video. Go subscribe as well. And if you really like the video and you want to become one of these amazing people over here, check out the Patreon scribe barrap linked in the description down below. That's going to be it for me. And don't let the DNS see you. time ain't playing at your best. If it don't involve money, then I don't accept. something like this.