Fix for kubernetes on DigitalOcean Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 11.111.11.1 11.111.11.2 11.111.11.1"
(example IP addresses used)
After ignoring the error for a bit, I finally had to absolutely solve it due to being unable to deploy a prometheus chart inside my kubernetes cluster.
This isn't really a step-by-step but more an accumulation of resources used/found to finally help solve my problem
TL;DR
- Modify
/etc/netplan/50-cloud-init.yaml
removing some entries from IFACE.nameservers.addressnetplan apply
to apply changesresolvctl
to see changes to DNS Servers stanza for the IFACE modified- Review
/run/systemd/resolve/resolv.conf
to see less nameservers
- Modify
/etc/systemd/resolved.conf.d/DigitalOcean.conf
either mving/commenting it outsystemctl restart systemd-resolved.service
to apply changesresolvctl
to see changes to DNS Servers stanza for Global modified- Review
/run/systemd/resolve/resolv.conf
to see less nameservers
- (Optionally) Using
resolvectl
commandresolvectl dns IFACE DNS_ADDRESS
to update the DNS servers that appear inresolvectl
and/run/systemd/resolve/stub-resolv.conf
resolvectl dns IFACE
to view the nameservers for the interface- This is similar to editing the
/etc/netplan
files, but most likely will not persist. - To persist these changes, edit the netplan file
I won't recommend how you modify these files, whether you want to try googles 8.8.8.8
8.8.4.4
, or attempting to use only the global entry and completely removing the entries from the cloud-init.yaml
file etc. Thats more on the individual to work with and test. The only goal really is lowering your nameservers in /run/systemd/resolve/resolv.conf
to 3 or below, how its done is up to the reader.
The bottom titled "My thoughts" is a formatted version of above with slightly more explanation (since Im not really knowledgeable on the subject of DNS). Everything else below is kind of all over copy+pasting of links and some notes on them with only a bit of coherency as I worked backwards after finally getting the solution.
https://wiki.archlinux.org/title/systemd-networkd
Under section 2 Configuration files
Configuration files are located in /usr/lib/systemd/network/, the volatile runtime network directory /run/systemd/network/ and the local administration network directory /etc/systemd/network/. Files in /etc/systemd/network/ have the highest priority.
https://askubuntu.com/questions/1128536/how-to-make-persistent-changes-to-etc-resolv-conf-ubuntu-18-10
https://linuxize.com/post/how-to-set-dns-nameservers-on-ubuntu-18-04/
https://ubuntu.com/server/docs/network-configuration
netplan apply
I found the file I needed under /run/systemd/network/10-netplan-eth0.network
I think this only gets read on system boot/restart
Using resolvectl dns IFACE DNS_ADDRESS
will add the DNS entrys to both /run/systemd/resolve/resolv.conf
and youll see it under resolvectl
I think I saw this command somwhere else but I was determined to find a config but if youre fed up, issuing the first command
will set the nameservers for the eth0
interface
https://gist.github.com/brasey/fa2277a6d7242cdf4e4b7c720d42b567?permalink_comment_id=4417602#gistcomment-4417602
https://discussion.fedoraproject.org/t/systemd-resolved-duplicate-entries/31676/3 /etc/systemd/resolved.conf.d/DigitalOcean.conf
https://unix.stackexchange.com/questions/328131/how-to-troubleshoot-dns-with-systemd-resolved
/etc/systemd/resolved.conf
/etc/systemd/resolved.conf.d/
/lib/systemd/resolv.conf
/run/systemd/resolve/resolv.conf
/run/systemd/resolve/stub-resolv.conf
/var/run/systemd/resolve/resolv.conf
/var/run/systemd/resolve/stub-resolv.conf
/run/systemd/network/*
/run/systemd/network/10-netplan-IFACE.network
/etc/netplan/50-cloud-init.yaml
systemctl restart systemd-resolved.service
systemctl status systemd-resolved.service
resolvectl dns IFACE
DNS_SERVER
resolvectl flush-caches
systemd-analyze cat-config systemd/resolved.conf
systemctl restart network
DO NOT run these
ip addr flush eth0
ip addr flush eth1
The bottom will have a quick list of items to do to fix it however you choose, below is some additional info gathered.
Whatever is in the /etc/netplan/50-cloud-init.yaml
gets reflected in /run/systemd/network/10-netplan-IFACE.network
files which then populates /run/systemd/resolve/resolv.conf
on system boot/restart as far as im aware.
You can manually apply these changes using netplan apply
For each IFACE.network file that has a DNS
entry (or under nameservers.addresses in /etc/netplan/*), that will add a nameserver
entry into /run/systemd/resolve/resolv.conf
and you can see these entries listed under each individual IFACE in resolvectl
The entries you see in resolvectl
can be modified using resolvectl dns IFACE DNS_ADDRESS
as well (to persist modify the cloud-init.yaml
file) which will also alter /run/systemd/resolve/resolv.conf
nameserver entries (the main culprit of the error)
Under /etc/systemd/resolved.conf.d/DigitalOcean.conf
theres also entries that populate the top Global
entry in the resolvectl
command and add additional entries to /run/systemd/resolve/resolv.conf
(again the culprit).
After modifying/mving /etc/systemd/resolved.conf.d/DigitalOcean.conf
, you can issue systemctl restart systemd-resolved.service
to see the changes reflected in resolvectl
.
If using a private network on DigitalOcean this will cause an additional 2 DNS entries.
So 2 from eth0, 2 from eth1, and 2 from Global for a total of 6, when the max available is 3.
The top TL;DR of this file gives some basic/simple instructions on what you can do to accomplish that.