Small Technical Blog
DNS Configuration for K8s
Goal:
Configure DNS, so applications running on the K8S cluster are reachable from the internet and TLS-protected
Setting up a home made kubernetes cluster is quite straightforward. However, for deploying applications or services accessible from the internet, the configuration capabilities of the standard provider’s internet boxes are usually too limited.
In particular I had the issue of the internal host resolution. My internet box wasn’t able to properly route requests to the own external IP address from inside the internal network. And not even in the advanced configuration was possible to configure static host mapping.
I wanted to deploy a web application (cozy) on my cluster. In order to have this working, I needed the following:
- A FQDN example.com resolvable from the internet
- A wildcard SSL certificate covering both example.com and *.example.com
- A port forwarding rule for the HTTP and HTTPS ports to the K8S cluster
- A fix IP or a DynDNS service identifying the host IP where the application is running
- An own DNS server with a static host or an entry pointing to the internal IP of the FQDN
- For certmanager and Let’s Encrypt, a DNS provider supporting the DNS01 solver mechanism
The easiest solution for all this mess would have been to replace the provider’s internet box through a router of my own. I had a EdgeRouter X SFP I once bought, but never had time to set up, lying around. However short ago my internet provider replaced the internet box through a new one, featuring 10 GB internet. The EdgeRouter does not support the 10GB internet connection, so to replace the box wasn’t an option anymore.
So I decided to use the EdgeRouter to work only as a small internal DNS server.
NOTE: in this post I’m refering to the EdgeRouter as “the router” even if it’s not going to be used as one. The actual router for the host network is what I’m refering to as the “internet box”.
I followed the instructions in the quick start guide to do a factory reset of the router: press reset button, connect the power cable and wait until a moment until the lights stop blinking.
To connect to the router, first I disconnected the wifi on my laptop.
Then I connected the eth0 port directly to my laptop with a network cable and configured a fix IP on the eth0 port of the laptop as follows:
sudo ifconfig eth0 192.168.1.4 netmask 255.255.255.0 up
After this I was able to access the router on the browser under 192.168.1.1.
Before proceeding with any configuration, I performed a firmware upgrade on the device.
First of all, I had to configure the switch interface.
The router has following interfaces:
- Two external interfaces: eth0 (normal ethernet port) and eth5 (the SFP port)
- One internal interface switch with bridged ports eth1, eth2, eth3 and eth3
On the Dashboard, for each interface there is a drop-down button labeled Actions located on the right. For switch0, after selecting Config, I chose to manually define the IP address and set it to 192.168.1.140 / 24.
Next thing is to deactivate the DHCP server, so the router does not try to assign IP addresses to other devices in the network. We want the router to act just as a small server and not really as a router. The internet box keeps acting as the DHCP server and router for the home network.
Under Wizards, I went to the Basic Setup. There, under LAN ports, I deactivated the DHCP Server, unchecking the box Enable the DHCP server.
On the same wizard, in the User setup section, I changed the default user setting up a new user and password.
After this I pressed Apply to apply the changes and restart the router.
Once the router has restarted, I disconnected the router from my laptop.
I connected the eth1 port to a switch serving my home network and rebooted (disconnecting the power cord and connecting it again). After this, the router was accessible on the browser under 192.168.1.140.
Next step is the effective DNS configuration. The idea was to configure a public name server (as Google) and to manually override static some host mappings to have the right DNS resolution from inside the internal network for my applications on the K8s cluster.
-
In order for the router to find the default gateway, under Routing I added a new static route with description internetbox, destination 0.0.0.0 / 0, next hop 192.168.1.1 (which is the IP of the internet box) and interface switch0.
-
In the Config Tree under system there is a System parameters section. There I set up the the name-server to 127.0.0.1. With this the router will now act as the name server.
-
Under service, dns and forwarding, I configured listen-on as switch0 and as name-servers 8.8.8.8 and 8.8.8.4 (the Google ones).
The EdgeOS was now ready to act as the DNS server for the home network.
NOTE: regarding step 1, another way to do the same is to configure the default gateway in the system configuration.
The whole point of setting up the own DNS server is to be able to add some static host mappings for IP resolution inside of the network.
Under Config Tree and system I found the static-host-mapping section. Under host-name I configured the hostname and the aliases I needed for the application:
Hostname: cozy.example.com
IP: 192.168.1.131
Aliases:
home.cozy.example.com
drive.cozy.example.com
photos.cozy.example.com
settings.cozy.example.com
store.cozy.example.com
mailhog.cozy.example.com
The IP is the one of the worker’s node on the K8s cluster. The hostnames and aliases are the ones needed by the first application I was intending to deploy in K8s:
After I had my DNS server up und running, the remaining thing to do was to change the DNS server configuration on the internet box to feature the DNS server I just set up on the EdgeRouter. In the network settings configuration, I just had to change from automatic to manual and configure the IP of the router, which was 192.168.1.140.
I also added a static lease so the router would always get the same IP address.
I don’t have a fix IP. I still needed a FQDN.
The bright side of having to keep the provider’s box is that it comes with a DynDNS server working out of the box, so you don’t need to register a dedicated account for this. So I activated the feature and I got following hostname: example.myproviderbox.country
To make the application accessible from the internet, I had to:
- Modify the default firewall configuration to allow inbound traffic to port 443
- Create a port forwarding rule to forward all requests to port 443 to the internal IP address 192.168.1.131 (which was the address of the worker node where nginx was running).
I have a Cloudfare account and a registered domain. In cloudfare, I set up an alias by adding a CNAME DNS entry, pointing to the internet name I got from the DynDNS:
CNAME example.com example.myproviderbox.country
CNAME *.example.com example.com
The other thing I needed from Cloudfare was an API token for DNS01 challenge resolution for Let’s Encrypt.
Tokens can be created under User Profile > API Tokens > API Tokens. I used the recommended settings:
Permissions:
Zone - DNS - Edit
Zone - Zone - Read
Zone Resources:
Include - All Zones
With these preparations, my home network was ready to host and route requests to applications deployed on the K8s cluster.