Skip to content

Instantly share code, notes, and snippets.

@adelton

adelton/README.md

Last active April 17, 2025 21:32
Show Gist options
  • Save adelton/2c7d26e439111e49fadddfb33606dc69 to your computer and use it in GitHub Desktop.
Save adelton/2c7d26e439111e49fadddfb33606dc69 to your computer and use it in GitHub Desktop.
Download ZIP
NetworkManager and Oracle Cloud instances and IPv6
Raw
README.md

AlmaLinux-9-OCI-9.5-20250205.aarch64

[opc@test-ipv6-almalinux ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:00:17:08:52:c7 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 10.1.1.99/24 brd 10.1.1.255 scope global dynamic noprefixroute eth0
       valid_lft 86372sec preferred_lft 86372sec
    inet6 2603:c020:c009:fb41:d630:6a20:5baa:ab1d/128 scope global dynamic noprefixroute 
       valid_lft 89972sec preferred_lft 86372sec
    inet6 fe80::17ff:fe08:52c7/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[opc@test-ipv6-almalinux ~]$ nmcli d
DEVICE  TYPE      STATE                   CONNECTION       
eth0    ethernet  connected               Wired Connection 
lo      loopback  connected (externally)  lo               
[opc@test-ipv6-almalinux ~]$ nmcli c
NAME              UUID                                  TYPE      DEVICE 
Wired Connection  5808906a-39e9-424f-bc7a-fa443d1490b2  ethernet  eth0   
lo                307ed731-6a9b-41b4-a449-53143b479cb6  loopback  lo     
eth0              3e8f2f5f-8a9e-45cc-9385-8e4344dab151  ethernet  --     
System eth0       5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03  ethernet  --     

[root@test-ipv6-almalinux ~]# head -5 /run/NetworkManager/system-connections/default_connection.nmconnection
# Created by nm-initrd-generator

[connection]
id=Wired Connection
uuid=5808906a-39e9-424f-bc7a-fa443d1490b2

[root@test-ipv6-almalinux ~]# head -5 /etc/NetworkManager/system-connections/eth0.nmconnection
[connection]
id=eth0
uuid=3e8f2f5f-8a9e-45cc-9385-8e4344dab151
type=ethernet
autoconnect-priority=-100

That System eth0 seems to come from

[root@test-ipv6-almalinux ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Created by cloud-init automatically, do not edit.
#
AUTOCONNECT_PRIORITY=120
BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=02:00:17:08:52:c7
MTU=9000
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

so let's get rid of that first:

[root@test-ipv6-almalinux ~]# rm -f /etc/sysconfig/network-scripts/ifcfg-eth0

After changing for example attached VNIC's assigned IPv6 addresses,

[root@test-ipv6-almalinux ~]# nmcli c up 'Wired Connection'
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/3)

refreshes the IP address via DHCPv6.

We have routes

[opc@test-ipv6-almalinux ~]$ ip -6 route
::1 dev lo proto kernel metric 256 pref medium
2603:c020:c009:fb41:c484:6069:66a3:c3b9 dev eth0 proto kernel metric 100 pref medium
2603:c020:c009:fb41::/64 dev eth0 proto ra metric 100 pref medium
fe80::/64 dev eth0 proto kernel metric 1024 pref medium
default via fe80::200:17ff:fec9:7bbd dev eth0 proto ra metric 100 pref medium

When we add second VNIC with an IPv6 address, routes are

[opc@test-ipv6-almalinux ~]$ ip -6 route
::1 dev lo proto kernel metric 256 pref medium
2603:c020:c009:fb41:c484:6069:66a3:c3b9 dev eth0 proto kernel metric 100 pref medium
2603:c020:c009:fb41::/64 dev eth0 proto ra metric 100 pref medium
2603:c020:c009:fb42:6e78:9cfc:5b2a:bbc2 dev eth1 proto kernel metric 101 pref medium
2603:c020:c009:fb42::/64 dev eth1 proto ra metric 101 pref medium
fe80::/64 dev eth0 proto kernel metric 1024 pref medium
fe80::/64 dev eth1 proto kernel metric 1024 pref medium
default via fe80::200:17ff:fec9:7bbd dev eth0 proto ra metric 100 pref medium
default via fe80::200:17ff:fec9:7bbd dev eth1 proto ra metric 101 pref medium

When for example pinging the eth1's address 2603:c020:c009:fb42:6e78:9cfc:5b2a:bbc2, the return packets go though eth0. Therefore the VNIC of eth0 needs to be edited to set Skip Source/Destination Check: Yes.

Also note that the Wired Connection from that nm-initrd-generator's /run/NetworkManager/system-connections/default_connection.nmconnection is the same for both ethernet devices.

[opc@test-ipv6-almalinux ~]$ nmcli d
DEVICE  TYPE      STATE                   CONNECTION       
eth0    ethernet  connected               Wired Connection 
eth1    ethernet  connected               Wired Connection 
lo      loopback  connected (externally)  lo               
[opc@test-ipv6-almalinux ~]$ nmcli c
NAME              UUID                                  TYPE      DEVICE 
Wired Connection  b405caff-6004-4e4b-91a1-782639f050e2  ethernet  eth0   
Wired Connection  b405caff-6004-4e4b-91a1-782639f050e2  ethernet  eth1   
lo                e1298dfd-468b-449f-9b65-60b85c6c0de9  loopback  lo     
eth0              3e8f2f5f-8a9e-45cc-9385-8e4344dab151  ethernet  --

We might want to try to configure separate connections for the devices with

[root@test-ipv6-almalinux ~]# cat /etc/NetworkManager/system-connections/eth0.nmconnection
[connection]
id=eth0
type=ethernet
interface-name=eth0

[ethernet]

[ipv4]
method=auto

[ipv6]
method=auto

[root@test-ipv6-almalinux ~]# cat /etc/NetworkManager/system-connections/eth1.nmconnection
[connection]
id=eth1
type=ethernet
interface-name=eth1

[ethernet]

[ipv4]
method=auto

[ipv6]
method=auto

[root@test-ipv6-almalinux ~]# chmod 600 /etc/NetworkManager/system-connections/*

But even after reboot it does not work because the Wired Connection from initrd is defined in /run and takes precedence:

[opc@test-ipv6-almalinux ~]$ nmcli c
NAME              UUID                                  TYPE      DEVICE 
Wired Connection  e5595aec-853d-41bb-ae28-0c970888f214  ethernet  eth0   
Wired Connection  e5595aec-853d-41bb-ae28-0c970888f214  ethernet  eth1   
lo                a25c0844-98a7-4d43-89ee-0ba7c948713e  loopback  lo     
eth0              7ba00b1d-8cdd-30da-91ad-bb83ed4f7474  ethernet  --     
eth1              5eb4da26-5d1d-30a5-8747-80181ed055fa  ethernet  --

The solution is to blacklist any such connections:

[root@test-ipv6-almalinux ~]# cat /etc/NetworkManager/conf.d/no-not-keep-configuration.conf
[device-eth]
match-device=interface-name:eth*
keep-configuration=no
allowed-connections=except:origin:nm-initrd-generator

which gets us

[opc@test-ipv6-almalinux ~]$ nmcli d
DEVICE  TYPE      STATE                   CONNECTION 
eth0    ethernet  connected               eth0       
eth1    ethernet  connected               eth1       
lo      loopback  connected (externally)  lo         
[opc@test-ipv6-almalinux ~]$ nmcli c
NAME  UUID                                  TYPE      DEVICE 
eth0  7ba00b1d-8cdd-30da-91ad-bb83ed4f7474  ethernet  eth0   
eth1  5eb4da26-5d1d-30a5-8747-80181ed055fa  ethernet  eth1   
lo    4d8846a7-5dd2-4507-8149-62e216ec6f93  loopback  lo     
Wired Connection    f6c6fc71-42bc-477a-946b-5efc1d4fb60b  ethernet  --

Let's get rid of that "Wired Connection":

[root@test-ipv6-almalinux ~]# mkdir /etc/systemd/system/NetworkManager.service.d
[root@test-ipv6-almalinux ~]# cat /etc/systemd/system/NetworkManager.service.d/remove-initrd-wired-connection.conf
[Service]
ExecStartPre=-/usr/bin/mv -vf /run/NetworkManager/system-connections/default_connection.nmconnection /run/NetworkManager/system-connections/default_connection.nmconnection.removed

And time to get rid of cloud-init as well, I guess:

[root@test-ipv6-almalinux ~]# touch /etc/cloud/cloud-init.disabled
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment