Last week I spun up a kind cluster to test an operator change. Inside the cluster, a pod could resolve kubernetes.default.svc.cluster.local but not google.com. I spent longer than I care to admit on this before realizing it was my fault.

The setup

kind uses docker containers as nodes. CoreDNS runs inside. Pods resolve via CoreDNS, which forwards unknown names to the node’s resolver. The node here is the docker container, which uses whatever /etc/resolv.conf docker sets up for it.

Repro:

kubectl run -it debug --image=busybox:1.36 --restart=Never -- sh
# / # nslookup kubernetes.default.svc.cluster.local
# Address: 10.96.0.1
# / # nslookup google.com
# ;; connection timed out; no servers could be reached

Internal resolution works. External does not.

First layer: CoreDNS config

I pulled the configmap:

kubectl -n kube-system get configmap coredns -o yaml

Relevant part:

data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
    }

forward . /etc/resolv.conf means “for anything I don’t handle, use the resolvers listed in /etc/resolv.conf on the CoreDNS pod”. The pod inherits the node’s resolv.conf through a kubelet mechanism. Let’s see what that is.

Second layer: resolv.conf in the CoreDNS pod

kubectl -n kube-system exec coredns-xxx -- cat /etc/resolv.conf
# search kube-system.svc.cluster.local svc.cluster.local cluster.local
# nameserver 10.96.0.10
# options ndots:5

Wait, that points at CoreDNS itself. That would be a loop. But CoreDNS specifically does not use its own resolv.conf in the forwarding sense; forward . /etc/resolv.conf in CoreDNS resolves at config parse time to the upstreams listed. Let me check the node’s resolv.conf:

docker exec kind-control-plane cat /etc/resolv.conf
# nameserver 127.0.0.11
# options ndots:0

127.0.0.11 is docker’s embedded resolver. This is normal for docker containers.

Third layer: embedded resolver not forwarding

Docker’s embedded resolver at 127.0.0.11 forwards to the host’s resolvers. It reads them from the docker daemon’s daemon.json or, if not configured there, from the host’s /etc/resolv.conf at docker startup. On my laptop, the host’s resolv.conf looked like:

cat /etc/resolv.conf
# nameserver 192.168.1.1

192.168.1.1 is my home router. Which is up. But when I dig @192.168.1.1 google.com from my laptop, it worked. So why didn’t it work from inside the kind container?

Fourth layer: firewall

I was running a firewall on my laptop that blocked traffic from docker’s bridge networks to anything on my LAN except for a specific allow list. The allow list did not include 192.168.1.1:53 because I had never bothered. External DNS queries from kind’s container to my router were being silently dropped.

sudo iptables -S | grep docker
# -A DOCKER-USER -s 172.18.0.0/16 -d 192.168.0.0/16 -j DROP

There it is. The DOCKER-USER chain, where I had dropped LAN-to-docker traffic years ago for a different reason.

The fix

Instead of weakening the firewall, I pointed docker’s embedded resolver at a public DNS:

// /etc/docker/daemon.json
{
  "dns": ["1.1.1.1", "9.9.9.9"]
}

Restart docker. Recreate the kind cluster. The container’s /etc/resolv.conf now shows:

docker exec kind-control-plane cat /etc/resolv.conf
# nameserver 1.1.1.1
# nameserver 9.9.9.9
# options ndots:0

External DNS works. Internal still works because CoreDNS is still answering cluster.local.

What I wish I had done first

There is a CoreDNS debug plugin that logs every forward, and also a log directive that logs all queries. For future debugging I now add this to my kind cluster’s CoreDNS config:

.:53 {
    log
    errors
    # ...
    forward . /etc/resolv.conf
    cache 30
}

With log, I would have seen immediately that external queries reached CoreDNS, got forwarded to the node’s resolver, and then timed out. The loop “is it CoreDNS, is it the node, is it the host?” is faster to navigate with one side talking.

Also, kubectl exec debug-pod -- dig @127.0.0.1 google.com from inside the pod with dig available tells you quickly whether DNS forwarding is broken at CoreDNS (returns SERVFAIL/REFUSED) or upstream (timeout).

Reflection

Kind is great for trying things locally, but it’s not a hermetic environment. Everything in the cluster eventually punches back out to the host’s DNS. If your host DNS is weird (firewalls, split-horizon, a VPN routing DNS elsewhere, an ad blocker that blocks resolution for specific names), your kind pods will inherit that weirdness.

For local cluster development I now always set explicit dns in docker’s daemon.json. Public resolvers are fine for development purposes. If your laptop routes DNS through a corporate VPN, you will want your VPN’s resolver in there or things will be confusing.

Related: see my post on ndots:5 and the DNS tax in Kubernetes for more DNS pain at cluster scale.