Ingress DNS

DNS service for ingress controllers running on your minikube server

Overview

Problem

When running minikube locally, you may want to run your services on an ingress controller so that you don’t have to use minikube tunnel or NodePorts to access your services. While NodePort might be okay in a lot of circumstances, an ingress is necessary to test some features. Ingress controllers are great because you can define your entire architecture in something like a helm chart, and all your services will be available.

However, for minikube, there is an additional challenge. Your ingress controller relies on DNS, so local DNS names like myservice.test will have to resolve to your minikube ip. The only real way to do this is to add an entry for every service in your /etc/hosts file. This gets messy for obvious reasons. For each service you are running that each has its own DNS entry, you will need to configure it manually. Even if you automate it, you then need to rely on the host operating system for storing configurations instead of storing them in your cluster. To make it worse, these configurations have to be constantly maintained and updated as services are added, removed, and renamed. I call it the /etc/hosts pollution problem.

Solution

What if you could just access your local services magically without having to edit your /etc/hosts file? Well, now you can. The ingress-dns addon acts as a DNS service that runs inside your Kubernetes cluster. All you have to do is install the service and add the minikube ip as a DNS server on your host machine. Each time the DNS service is queried, an API call is made to the Kubernetes master service for a list of all the ingresses. If a match is found for the name, a response is given with an IP address matching minikube ip. For example, with a minikube ip of 192.168.99.169 and an ingress rule for myservice.test configured in the cluster, a DNS query from the host would produce:

#bash:~$ nslookup myservice.test $(minikube ip)
Server:		192.168.99.169
Address:	192.168.99.169#53

Non-authoritative answer:
Name:	myservice.test $(minikube ip)
Address: 192.168.99.169

Installation

1Start minikube

minikube start

2Enable the addons

minikube addons enable ingress
minikube addons enable ingress-dns

3Add the `minikube ip` as a DNS server

On Linux, you should identify your domain name resolver configuration, and update its configuration accordingly. To that end, look at the first lines of /etc/resolv.conf:

  • if it mentions resolvconf, resolution is likely handled by resolvconf,
  • if it is # Generated by NetworkManager, resolution is handled by NetworkManager,
  • if it similar to # This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8), resolution is handled by systemd-resolved.

Start minikube, and apply the configuration below matching your system configuration.

Linux OS with resolvconf

Update the file /etc/resolvconf/resolv.conf.d/base to have the following contents.

search test
nameserver 192.168.99.169
timeout 5

Replace 192.168.99.169 with the output of minikube ip.

If your Linux OS uses systemctl, run the following commands.

sudo resolvconf -u
systemctl disable --now resolvconf.service

If your Linux OS does not use systemctl, run the following commands.

# TODO add supporting docs for Linux OS that do not use `systemctl`

See https://linux.die.net/man/5/resolver

Linux OS with NetworkManager

NetworkManager can run integrated caching DNS server - dnsmasq plugin and can be configured to use separate nameservers per domain.

Edit /etc/NetworkManager/NetworkManager.conf and enable dns=dnsmasq by adding:

[main]
dns=dnsmasq

Also see dns= in NetworkManager.conf.

Configure dnsmasq to handle domain names ending with .test:

sudo mkdir -p /etc/NetworkManager/dnsmasq.d/
echo "server=/test/$(minikube ip)" | sudo tee /etc/NetworkManager/dnsmasq.d/minikube.conf

Restart Network Manager:

systemctl restart NetworkManager.service

Ensure your /etc/resolv.conf contains only single nameserver:

cat /etc/resolv.conf | grep nameserver
nameserver 127.0.0.1

Linux OS with systemd-resolved

Run the following commands to add the minikube DNS for .test domains:

sudo mkdir -p /etc/systemd/resolved.conf.d
sudo tee /etc/systemd/resolved.conf.d/minikube.conf << EOF
[Resolve]
DNS=$(minikube ip)
Domains=~test
EOF
sudo systemctl restart systemd-resolved

Create a file in /etc/resolver/minikube-test with the following contents.

domain test
nameserver 192.168.99.169
search_order 1
timeout 5

Replace 192.168.99.169 with your minikube ip.

If you have multiple minikube IPs, you must configure a file for each.

See https://www.unix.com/man-page/opendarwin/5/resolver/

Note that the port feature does not work as documented.

Open Powershell as Administrator and execute the following.

Add-DnsClientNrptRule -Namespace ".test" -NameServers "$(minikube ip)"

The following will remove any matching rules before creating a new one. This is useful for updating the minikube ip.

Get-DnsClientNrptRule | Where-Object {$_.Namespace -eq '.test'} | Remove-DnsClientNrptRule -Force; Add-DnsClientNrptRule -Namespace ".test" -NameServers "$(minikube ip)"

4(optional) Configure in-cluster DNS server to resolve local DNS names inside cluster

Sometimes it’s useful to access other applications inside cluster via ingress and by their local DNS name - microservices/APIs/tests. In such case ingress-dns addon should be used by in-cluster DNS server - CoreDNS to resolve local DNS names.

Edit your CoreDNS config

kubectl edit configmap coredns -n kube-system

and add block for your local domain

    test:53 {
            errors
            cache 30
            forward . 192.168.99.169
    }

Replace 192.168.99.169 with your minikube ip.

The final ConfigMap should look like:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }    
...
    }
    test:53 {
            errors
            cache 30
            forward . 192.168.99.169
    }
kind: ConfigMap
metadata:
...

See https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/

Testing

1Add the test ingress

kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/master/deploy/addons/ingress-dns/example/example.yaml

Note: Minimum Kubernetes version for the example ingress is 1.19

2Confirm that DNS queries are returning A records

nslookup hello-john.test $(minikube ip)
nslookup hello-jane.test $(minikube ip)

3Confirm that domain names are resolving on the host OS

ping hello-john.test
ping hello-jane.test

Expected results:

PING hello-john.test (192.168.99.169): 56 data bytes
64 bytes from 192.168.99.169: icmp_seq=0 ttl=64 time=0.361 ms
PING hello-jane.test (192.168.99.169): 56 data bytes
64 bytes from 192.168.99.169: icmp_seq=0 ttl=64 time=0.262 ms

4Curl the example server

curl http://hello-john.test
curl http://hello-jane.test

Expected results:

Hello, world!
Version: 1.0.0
Hostname: hello-world-app-557ff7dbd8-64mtv
Hello, world!
Version: 1.0.0
Hostname: hello-world-app-557ff7dbd8-64mtv

Known issues

.localhost always resolves to the loopback address

.localhost will often resolve to the loopback address (see RFC 2606 and RFC 6761), so it can’t be used for minikube ip. Instead use .test, .example, or .invalid

.local is a reserved TLD

Do not use .local as this is a reserved TLD for mDNS and bind9 DNS servers

Mac OS

mDNS reloading

Each time a file is created or a change is made to a file in /etc/resolver you may need to run the following to reload Mac OS mDNS resolver.

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist

TODO

  • Add a service that runs on the host OS which will update the files in /etc/resolver automatically
  • Start this service when running minikube addons enable ingress-dns and stop the service when running minikube addons disable ingress-dns

Contributors

Images used in this plugin

Image Source Owner
ingress-nginx ingress-nginx Kubernetes ingress-nginx
minikube-ingress-dns minikube-ingress-dns Cryptex Labs