Fix for kubernetes on DigitalOcean Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 11.111.11.1 11.111.11.2 11.111.11.1" (example IP addresses used)
After ignoring the error for a bit, I finally had to absolutely solve it due to being unable to deploy a prometheus chart inside my kubernetes cluster.
This isn't really a step-by-step but more an accumulation of resources used/found to finally help solve my problem
TL;DR
- Modify
/etc/netplan/50-cloud-init.yamlremoving some entries from IFACE.nameservers.addressnetplan applyto apply changesresolvctlto see changes to DNS Servers stanza for the IFACE modified- Review
/run/systemd/resolve/resolv.confto see less nameservers
- Modify
/etc/systemd/resolved.conf.d/DigitalOcean.confeither mving/commenting it outsystemctl restart systemd-resolved.serviceto apply changesresolvctlto see changes to DNS Servers stanza for Global modified- Review
/run/systemd/resolve/resolv.confto see less nameservers
- (Optionally) Using
resolvectlcommandresolvectl dns IFACE DNS_ADDRESSto update the DNS servers that appear inresolvectland/run/systemd/resolve/stub-resolv.confresolvectl dns IFACEto view the nameservers for the interface- This is similar to editing the
/etc/netplanfiles, but most likely will not persist. - To persist these changes, edit the netplan file
I won't recommend how you modify these files, whether you want to try googles 8.8.8.8 8.8.4.4, or attempting to use only the global entry and completely removing the entries from the cloud-init.yaml file etc. Thats more on the individual to work with and test. The only goal really is lowering your nameservers in /run/systemd/resolve/resolv.conf to 3 or below, how its done is up to the reader.
The bottom titled "My thoughts" is a formatted version of above with slightly more explanation (since Im not really knowledgeable on the subject of DNS). Everything else below is kind of all over copy+pasting of links and some notes on them with only a bit of coherency as I worked backwards after finally getting the solution.
https://wiki.archlinux.org/title/systemd-networkd
Under section 2 Configuration files
Configuration files are located in /usr/lib/systemd/network/, the volatile runtime network directory /run/systemd/network/ and the local administration network directory /etc/systemd/network/. Files in /etc/systemd/network/ have the highest priority.
https://askubuntu.com/questions/1128536/how-to-make-persistent-changes-to-etc-resolv-conf-ubuntu-18-10
https://linuxize.com/post/how-to-set-dns-nameservers-on-ubuntu-18-04/
https://ubuntu.com/server/docs/network-configuration
netplan apply
I found the file I needed under /run/systemd/network/10-netplan-eth0.network
I think this only gets read on system boot/restart
Using resolvectl dns IFACE DNS_ADDRESS will add the DNS entrys to both /run/systemd/resolve/resolv.conf and youll see it under resolvectl
I think I saw this command somwhere else but I was determined to find a config but if youre fed up, issuing the first command
will set the nameservers for the eth0 interface
https://gist.github.com/brasey/fa2277a6d7242cdf4e4b7c720d42b567?permalink_comment_id=4417602#gistcomment-4417602
https://discussion.fedoraproject.org/t/systemd-resolved-duplicate-entries/31676/3 /etc/systemd/resolved.conf.d/DigitalOcean.conf
https://unix.stackexchange.com/questions/328131/how-to-troubleshoot-dns-with-systemd-resolved
/etc/systemd/resolved.conf
/etc/systemd/resolved.conf.d/
/lib/systemd/resolv.conf
/run/systemd/resolve/resolv.conf
/run/systemd/resolve/stub-resolv.conf
/var/run/systemd/resolve/resolv.conf
/var/run/systemd/resolve/stub-resolv.conf
/run/systemd/network/*
/run/systemd/network/10-netplan-IFACE.network
/etc/netplan/50-cloud-init.yaml
systemctl restart systemd-resolved.service
systemctl status systemd-resolved.service
resolvectl dns IFACE DNS_SERVER
resolvectl flush-caches
systemd-analyze cat-config systemd/resolved.conf
systemctl restart network
DO NOT run these
ip addr flush eth0
ip addr flush eth1
The bottom will have a quick list of items to do to fix it however you choose, below is some additional info gathered.
Whatever is in the /etc/netplan/50-cloud-init.yaml gets reflected in /run/systemd/network/10-netplan-IFACE.network files which then populates /run/systemd/resolve/resolv.conf on system boot/restart as far as im aware.
You can manually apply these changes using netplan apply
For each IFACE.network file that has a DNS entry (or under nameservers.addresses in /etc/netplan/*), that will add a nameserver entry into /run/systemd/resolve/resolv.conf and you can see these entries listed under each individual IFACE in resolvectl
The entries you see in resolvectl can be modified using resolvectl dns IFACE DNS_ADDRESS as well (to persist modify the cloud-init.yaml file) which will also alter /run/systemd/resolve/resolv.conf nameserver entries (the main culprit of the error)
Under /etc/systemd/resolved.conf.d/DigitalOcean.conf theres also entries that populate the top Global entry in the resolvectl command and add additional entries to /run/systemd/resolve/resolv.conf (again the culprit).
After modifying/mving /etc/systemd/resolved.conf.d/DigitalOcean.conf, you can issue systemctl restart systemd-resolved.service to see the changes reflected in resolvectl.
If using a private network on DigitalOcean this will cause an additional 2 DNS entries.
So 2 from eth0, 2 from eth1, and 2 from Global for a total of 6, when the max available is 3.
The top TL;DR of this file gives some basic/simple instructions on what you can do to accomplish that.