# How to Build a 100% Private AWS EKS Cluster (Client VPN + Local Route 53 DNS)

## Метаданные

- **Канал:** Anton Putra
- **YouTube:** https://www.youtube.com/watch?v=Zv4c4YC-aAM

## Содержание

### [0:00](https://www.youtube.com/watch?v=Zv4c4YC-aAM) Segment 1 (00:00 - 05:00)

In this video, I’ll help you set up a fully private EKS cluster with only a private endpoint and use OpenVPN to not only connect to EKS but also resolve private Route 53 hosted zones. For example, we’ll deploy Grafana to a Kubernetes cluster with a private ingress controller. This will enable your team members to securely access private applications and services using a private DNS and private IP addresses only. Additionally, we’ll deploy Postgres on an EC2 instance, and I’ll show you how to access it from your laptop using a private IP address and DNS via OpenVPN and what security group rules need to be created. Usually, you have multiple public and private subnets, and for convenience, you create an EKS cluster that has a public endpoint so that you can connect to Kubernetes to manage, deploy your application, etc. And you would use private EC2 instances as Kubernetes nodes deployed in private subnets to run your applications. And if you need to expose anything to the internet, you would use ingress or just a regular load balancer service that you would create as a public elastic load balancer in the public subnet so anyone on the internet can access it. Well, sometimes you might have a requirement to not expose any public endpoints to the internet for security reasons, so you would create an EKS control plane with only a private endpoint, which is inconvenient because now to access and manage your applications, you have to do it only within the VPC— maybe from the bastion host or a private AWS workspace. Well, you can also create a client VPN and connect to that endpoint directly from your laptop. Another common use case for the client VPN is to be able to deploy private applications like internal Grafana dashboards and access them using private DNS and private IP addresses from your laptop. Also, we will deploy a private ingress and Grafana, and I’ll explain how to securely connect to them from your local browser using private DNS. And finally, it’s common to create private databases or messaging systems in private subnets and use a client VPN to connect to them—for example, for debugging or just development. So, the way it works is that we will create an OpenVPN server EC2 instance with a public static IP address, which will be located in a public subnet—like a bastion host, for example. Then, we’ll create a self-signed certificate authority, generate server and client certificates and profiles, and use them to connect to OpenVPN. When you start OpenVPN, it will create a virtual network, and when you connect to that server, your client will get a private virtual address. When, for example, you want to connect to a private EC2 instance, that private IP address will be routed to the EC2's private IP, and then you will be able to connect to any private EC2 instance and EKS inside your VPC. Additionally, since our client's traffic will be routed through the OpenVPN private IP, we can use the OpenVPN security group as a source when we create firewalls. Also, if you enable DNS in your VPN, you’ll get an AWS DNS resolver at a specific address. It’s the VPC CIDR range plus 2. For example, if I were to create a VPC with a 10. 0. 0. 0/16 CIDR block, your DNS resolver will be located at 10. 0. 0. 2. That means I can push that IP address to the client and be able to resolve any private DNS names within the VPC, as well as public DNS names like google. com. So, being able to resolve private DNS names like postgres. example. pvt and connect to your private applications like Grafana is very useful and secure, rather than exposing them to the internet. You can also automate the generation of OpenVPN profiles with single sign-on, as well as revoke certificates for people who have left the company. If you need any help, you can reach out to me. You can find more information in the video description. Now, there is a managed AWS Client VPN service, and there is always a trade-off between convenience and cost. For the managed service, you would pay $0. 10 per hour per subnet and $0. 05 per hour for each connected client.

### [5:00](https://www.youtube.com/watch?v=Zv4c4YC-aAM&t=300s) Segment 2 (05:00 - 10:00)

For example, if you had 3 private subnets in 3 different availability zones, which is common, you would have to pay $55 per month just to bootstrap your Client VPN. And if you had, say, 10 team members who use the VPN 8 hours a day, 5 days a week, you would additionally pay $92, which would be around $150 per month. On the other hand, you can set up your own OpenVPN and pay as little as $15 per month, or even lower, like $6 for a micro instance. Also, you would be subject to a data transfer fee of around $0. 09 per gigabyte, and your total per month might be between $15 and $20 per month. These are just rough estimates, but I think you get the point. Let me know if you are interested in the managed AWS Client VPN service, and I can create a similar tutorial for you. Let’s start. You can find the link to the GitHub repo in the video description. I suggest following along and applying the same Terraform files as I do in the video. First of all, we have local variables. Next, we have the traditional providers file. VPC with an additional Elastic IP address for the NAT gateway, which is optional, and I also use a Terraform module for simplicity. Next, we have a security group for OpenVPN. Port 22 is optional—you can use AWS Session Manager—but you need to open the default 1194 port for OpenVPN itself, and let’s use UDP. But if you really want to hide your traffic, you can use 443 and TCP instead. And finally, we have an EC2 instance based on a vanilla Ubuntu image. For EC2, you really want to preallocate an Elastic IP and associate it with your OpenVPN EC2 instance; otherwise, you would have to update all the configuration files if AWS were to take that IP during a reboot or if you just shut it down when you are not using it. And don’t forget to provide your own SSH key for the EC2. Alright, let’s go ahead and apply Terraform. Let’s see what we have so far. We have a VPC with 4 subnets: 2 private and 2 public. Next, we have a couple of static IP addresses, one for the NAT gateway and the second one for OpenVPN. And finally, we have an EC2 instance for OpenVPN. For now, it’s just plain Ubuntu, but you can automate this using Packer or even just a user data script. Alright, now let’s go ahead and SSH to the EC2 instance. Before we start, let’s just update and upgrade the Ubuntu packages. All the commands I run in the video, you can find in the README file—link in the video description. You can use the default Ubuntu repo to install OpenVPN, but it’s going to be outdated, so let’s add the key first, and then the official OpenVPN repository. Let’s update the index and see if we have the latest OpenVPN version now. Alright, now we have installed the latest OpenVPN. Next, we need a tool to generate a certificate authority, as well as to issue certificates for the OpenVPN server and our clients, which are your team members. Usually with OpenVPN, we use Easy-RSA, but you can use other tools as well, like CFSSL. Let’s unpack the archive and move it to the /etc directory under OpenVPN. We also want to make Easy-RSA executable and create a soft link. At this point in time, we have all the necessary components, and we can start bootstrapping our OpenVPN server. First, we need to create a CA or self-signed certificate authority. To configure it, let’s create a vars file and provide some options. You can update the email and org, but it’s just metadata—it will not play any role. Now we can actually create the CA. Also, you don’t really need to match the common name to your domain; there is no domain name verification on the certificate. We will just use the IP address to connect to this server and provide a client certificate to perform authentication. As you can see, we have the public certificate and the private key. Now we have the CA, and we can start using it to issue certificates for the OpenVPN server and clients. First, let’s generate a certificate for the OpenVPN server. It’s a two-step process: first, we generate a certificate request, and then we need to sign it using our certificate authority. To improve security, we need to generate

### [10:00](https://www.youtube.com/watch?v=Zv4c4YC-aAM&t=600s) Segment 3 (10:00 - 15:00)

another static key, which is used to add an additional layer of authentication to the TLS. The next important step is to enable IPv4 forwarding, since OpenVPN will act as a NAT. We also need to create iptables rules specifically for NAT translation. We need to find the default network interface and use it to create NAT rules. So, this OpenVPN server will create a virtual network 10. 8. 0. 0/24, and each client would get an IP address from this range, and then that IP is translated to the EC2 private IP on that default network interface. And that’s actually why we can use the EC2 security group to allow VPN clients to access specific EC2 instances in our private network. Finally, let’s save all those rules we just created. Next, let’s create the OpenVPN server configuration. You can find it in my repo as well. The most important part is to push the correct routes. So here, for example, I push all my private subnets, and additionally, I push the DNS resolver, which is your VPC CIDR + 2. In my previous video, I also explain how to revoke a private certificate—for example, from someone who just left your company. Let’s go ahead and copy it to the server. And since we used the "nobody" user and group in the config, let’s quickly check that we have this user and group. Now let’s start OpenVPN. If for some reason it failed, you can check the status, and also you can tail the logs and pretty quickly find the issue. Alright, now we can generate a certificate for the client. First, we generate a key pair, and then we sign it. You can find an OpenVPN example profile in the repo as well. Let’s use it as a base. First of all, we need to replace the public IP address, which will be the public static IP of our EC2. If you use Linux, you might need to potentially uncomment one of those blocks to resolve DNS. Next is the CA certificate. Then the client certificate, which is required for authentication. The private key of the certificate. And lastly, the TA key. Now on Mac or Linux, we can check the current routes we have. So far, I don’t have any route to the AWS VPC and any private subnet. Let’s go ahead and tail the OpenVPN server logs just to see if we can connect to it. On Mac, you can use the Tunnelblick OpenVPN client, or you can use any other OpenVPN client. To create an OpenVPN profile, you just double-click on it, or you can use the CLI as well. Now we can click connect and immediately see logs on the server. For example, you can see your public IP address, as well as a private one assigned by OpenVPN, in the logs. Now we can check our local routes again, and you can see that we have a few more routes to the AWS VPC private subnets. And in the DNS settings—either on Mac, Linux, or Windows—you can find that now all your DNS queries are sent to the AWS resolver

### [15:00](https://www.youtube.com/watch?v=Zv4c4YC-aAM&t=900s) Segment 4 (15:00 - 20:00)

via a private route. That means now we can resolve all private Route 53 hosted zones locally. If you try to resolve any public domain, your query is also sent to the AWS resolver, which is expected. Just be careful and do not create private and public hosted zones with the same domain—you’ll get DNS split horizon, which is just inconvenient. Next, let’s quickly create a Postgres EC2 instance just to demonstrate a common use case for OpenVPN; it’s not going to take a lot of time. First of all, we need to create a dedicated security group for Postgres. And now the most important part: to allow local clients that use OpenVPN to connect to Postgres, we use the OpenVPN security group as a source. So for any EC2 instance in private subnets, to open the firewall, you just use the same OpenVPN security group and just change the ports and protocols. In this example, I just want to be able to ping my Postgres. If you want to connect to Postgres, you would use 5432 and TCP instead. Next, let’s create an EC2 instance for Postgres and use the security group. Next, we’ll create a private hosted zone; it can be anything—it’s common to use local, internal, or pvt just to indicate it’s a private zone. And finally, let’s create a DNS record for that EC2 instance. That’s all—we can go ahead and apply Terraform. Alright, now we have a Postgres EC2 instance, and it does not have a public IP address—just a private one. Take note of that IP: 10. 0. 13. 238. And as you can see, we have the local route that would allow us to reach that EC2 instance via our OpenVPN server. So now, since we opened the firewall, we can ping this IP address from the laptop connected to OpenVPN. Now let me show you the Route 53 private hosted zone. So we have a private DNS record that resolves to a private IP address inside the VPC. And we can test it locally as well, and all the DNS queries will be sent to the AWS resolver—and that’s how we can resolve the Postgres DNS to an IP. And we can just ping it or connect to that Postgres from the laptop, which is very convenient for development. Alright, time for EKS. First, we have the EKS control plane itself. By the way, I have a full course on EKS here on YouTube. In the VPC block, we only specify that we want a private endpoint. That means you can access the EKS control plane only within the VPC. Now, an important part here is actually to capture the EKS security group—we’ll use it in OpenVPN since we’re creating a fully private EKS cluster. Next, we create a security rule to allow our OpenVPN server to connect to the EKS private endpoint. Then we create EKS nodes. Private NGINX Ingress controller, which would use only a private load balancer that you can access only within the VPC. And finally, we’ll deploy Grafana and use that private ingress. I’ll use the default controller to create a load balancer, which by default creates a Classic Load Balancer, and we use an annotation to make it private. And in Grafana, we just use the same domain we created for Postgres and the same private hosted zone. And specify the private ingress class. Let’s apply Terraform. It will take up to 20 minutes to create EKS and deploy all the components. So this is our private endpoint. If you try to resolve it, you’ll get a private IP address—just make sure that those IP addresses are within the routes you pushed to the VPN client. Next, we can update the local Kubernetes config, and let’s see if we can connect to it. Great, it works, and we have Kubernetes nodes. Let’s check the load balancer that was created for the ingress. You can find it in AWS just to make sure it is a private load balancer. And if you try to resolve it, you’ll get IP addresses from your private

### [20:00](https://www.youtube.com/watch?v=Zv4c4YC-aAM&t=1200s) Segment 5 (20:00 - 21:00)

subnets. So it’s not possible to access this load balancer from the internet. The last thing we need is to create a DNS record for Grafana. For simplicity, we can just do it manually. It’s going to be a CNAME record that points to the ingress load balancer. Let’s wait for a few minutes and try to resolve it. Now we can open a browser and paste the Grafana DNS, and you should be able to access Grafana. The username is admin and the password is devops123. And that’s how you connect to the private hosted zones. If you need any help, you can reach out to me. You can find more information in the video description.

---
*Источник: https://ekstraktznaniy.ru/video/38514*