Skip to content

Instantly share code, notes, and snippets.

@noteed

noteed/docker-ovs.md

Last active December 29, 2023 07:07
Show Gist options
  • Save noteed/8656989 to your computer and use it in GitHub Desktop.
Save noteed/8656989 to your computer and use it in GitHub Desktop.
Download ZIP
Docker - Open vSwitch setup
Raw
docker-ovs.md

Both machine report:

# ps ax | grep docker
837 ?        Sl     0:00 /usr/bin/docker -d -b=none

Node 1

root@node-1:~# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 04:01:10:10:5d:01 brd ff:ff:ff:ff:ff:ff
    inet 95.85.54.71/24 brd 95.85.54.255 scope global eth0
    inet6 fe80::601:10ff:fe10:5d01/64 scope link 
       valid_lft forever preferred_lft forever
3: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether da:e5:e3:9c:9e:3e brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
    inet6 fe80::d8e5:e3ff:fe9c:9e3e/64 scope link 
       valid_lft forever preferred_lft forever
17: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether ca:89:a2:3d:6c:47 brd ff:ff:ff:ff:ff:ff
    inet 172.16.42.1/24 scope global docker0
    inet6 fe80::4cea:2ff:fedc:c57a/64 scope link 
       valid_lft forever preferred_lft forever
18: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master docker0 state DOWN 
    link/ether ca:89:a2:3d:6c:47 brd ff:ff:ff:ff:ff:ff


root@node-1:~# ip r s
default via 95.85.54.1 dev eth0 
10.0.3.0/24 dev lxcbr0  proto kernel  scope link  src 10.0.3.1 
95.85.54.0/24 dev eth0  proto kernel  scope link  src 95.85.54.71 
172.16.42.0/24 dev docker0  proto kernel  scope link  src 172.16.42.1


root@node-1:~# brctl show
bridge name	bridge id		STP enabled	interfaces
docker0		8000.ca89a23d6c47	no		br0
lxcbr0		8000.000000000000	no


root@node-1:~# ovs-vsctl show
1510f4dc-8b5a-48f8-97ce-a2e233a70c0d
    Bridge "br0"
        Port "gre0"
            Interface "gre0"
                type: gre
                options: {remote_ip="188.226.138.185"}
        Port "br0"
            Interface "br0"
                type: internal
    ovs_version: "1.9.0"


root@node-1:~# ping 188.226.138.185
PING 188.226.138.185 (188.226.138.185) 56(84) bytes of data.
64 bytes from 188.226.138.185: icmp_req=1 ttl=63 time=0.426 ms
64 bytes from 188.226.138.185: icmp_req=2 ttl=63 time=0.440 ms
^C
--- 188.226.138.185 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.426/0.433/0.440/0.007 ms


root@node-1:~# ping 172.16.42.2
PING 172.16.42.2 (172.16.42.2) 56(84) bytes of data.
From 172.16.42.1 icmp_seq=1 Destination Host Unreachable
From 172.16.42.1 icmp_seq=2 Destination Host Unreachable
From 172.16.42.1 icmp_seq=3 Destination Host Unreachable
^C
--- 172.16.42.2 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3014ms
pipe 3

Node 2

root@node-2:~# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 04:01:10:10:f0:01 brd ff:ff:ff:ff:ff:ff
    inet 188.226.138.185/24 brd 188.226.138.255 scope global eth0
    inet6 fe80::601:10ff:fe10:f001/64 scope link 
       valid_lft forever preferred_lft forever
3: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 2e:1a:a6:13:53:77 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
    inet6 fe80::2c1a:a6ff:fe13:5377/64 scope link 
       valid_lft forever preferred_lft forever
17: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 42:d5:ad:4b:82:40 brd ff:ff:ff:ff:ff:ff
    inet 172.16.42.2/24 scope global docker0
    inet6 fe80::58c1:35ff:feb5:5ec8/64 scope link 
       valid_lft forever preferred_lft forever
18: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master docker0 state DOWN 
    link/ether 42:d5:ad:4b:82:40 brd ff:ff:ff:ff:ff:ff


root@node-2:~# ip r s
default via 188.226.138.1 dev eth0 
10.0.3.0/24 dev lxcbr0  proto kernel  scope link  src 10.0.3.1 
172.16.42.0/24 dev docker0  proto kernel  scope link  src 172.16.42.2 
188.226.138.0/24 dev eth0  proto kernel  scope link  src 188.226.138.185


root@node-2:~# brctl show
bridge name	bridge id		STP enabled	interfaces
docker0		8000.42d5ad4b8240	no		br0
lxcbr0		8000.000000000000	no


root@node-2:~# ovs-vsctl show
7635f0dd-d245-430c-9dbf-02dba456cc88
    Bridge "br0"
        Port "br0"
            Interface "br0"
                type: internal
        Port "gre0"
            Interface "gre0"
                type: gre
                options: {remote_ip="95.85.54.71"}
    ovs_version: "1.9.0"


root@node-2:~# ping 95.85.54.71
PING 95.85.54.71 (95.85.54.71) 56(84) bytes of data.
64 bytes from 95.85.54.71: icmp_req=1 ttl=63 time=0.460 ms
^C
--- 95.85.54.71 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.460/0.460/0.460/0.000 ms


root@node-2:~# ping 172.16.42.1
PING 172.16.42.1 (172.16.42.1) 56(84) bytes of data.
From 172.16.42.2 icmp_seq=1 Destination Host Unreachable
From 172.16.42.2 icmp_seq=2 Destination Host Unreachable
From 172.16.42.2 icmp_seq=3 Destination Host Unreachable
^C
--- 172.16.42.1 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3014ms
pipe 3
Raw
install.sh
# For lxc-docker.
sh -c "echo deb http://get.docker.io/ubuntu docker main /etc/apt/sources.list.d/docker.list"
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
apt-get update
apt-get install -q -y vim openvswitch-switch bridge-utils
# This Gist.
wget https://gist.github.com/noteed/8656989/raw/cf7e18c338901b9eb8ae6522796ebca8728c4e36/iptables-rules.sh
wget https://gist.github.com/noteed/8656989/raw/48567131827ef888149d5c04c51c7a89e1237f73/shared-docker-network.sh
sh shared-docker-network.sh
sh iptables-rules.sh
apt-get install -q -y lxc-docker
echo 'DOCKER_OPTS="-b=none"' >> /etc/default/docker
service docker restart
Raw
iptables-rules.sh
# Enable NAT
iptables -t nat -A POSTROUTING -s 172.16.42.0/24 ! -d 172.16.42.0/24 -j MASQUERADE
# Accept incoming packets for existing connections
iptables -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# Accept all non-intercontainer outgoing packets
iptables -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
# By default allow all outgoing traffic
iptables -A FORWARD -i docker0 -o docker0 -j ACCEPT
Raw
shared-docker-network.sh
# From http://goldmann.pl/blog/2014/01/21/connecting-docker-containers-on-multiple-hosts/
# The 'other' host
REMOTE_IP=188.226.138.185
# Name of the bridge
BRIDGE_NAME=docker0
# Bridge address
BRIDGE_ADDRESS=172.16.42.1/24
# Deactivate the docker0 bridge
ip link set $BRIDGE_NAME down
# Remove the docker0 bridge
brctl delbr $BRIDGE_NAME
# Delete the Open vSwitch bridge
ovs-vsctl del-br br0
# Add the docker0 bridge
brctl addbr $BRIDGE_NAME
# Set up the IP for the docker0 bridge
ip a add $BRIDGE_ADDRESS dev $BRIDGE_NAME
# Activate the bridge
ip link set $BRIDGE_NAME up
# Add the br0 Open vSwitch bridge
ovs-vsctl add-br br0
# Create the tunnel to the other host and attach it to the
# br0 bridge
ovs-vsctl add-port br0 gre0 -- set interface gre0 type=gre options:remote_ip=$REMOTE_IP
# Add the br0 bridge to docker0 bridge
brctl addif $BRIDGE_NAME br0
@adityashanbhag
Copy link

I am trying to get a multiple host docker setup and have followed you blog post "http://goldmann.pl/blog/2014/01/21/connecting-docker-containers-on-multiple-hosts/" and also this gist

Based on that I am able to ping the containers from each host [and across the host], so network connectivity between the different containers spread over the 2 hosts seems to work properly. [PS: My Host is Ubuntu 14.04 and my docker version is Docker version 1.0.1, build 990021a

After that I setup some containers and here is what I observe, kindly let me know if I am missing something or is this a limitation of the setup.

When I try ssh from one container from Host A to another container on Host B, or just try to ssh from Host A to any container on Host B, the ssh seems to be stuck. A typical output looks like this

ssh -v zookeeper01
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /home/admin/.ssh/config
debug1: /home/admin/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to zookeeper01 [10.0.5.1] port 22.
debug1: Connection established.
debug1: identity file /home/admin/.ssh/id_rsa type -1
debug1: identity file /home/admin/.ssh/id_rsa-cert type -1
debug1: identity file /home/admin/.ssh/id_dsa type -1
debug1: identity file /home/admin/.ssh/id_dsa-cert type -1
debug1: identity file /home/admin/.ssh/id_ecdsa type -1
debug1: identity file /home/admin/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/admin/.ssh/id_ed25519 type -1
debug1: identity file /home/admin/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent

After that it is stuck at that point.

More importantly I see the same issues on different application similar to SSH.

you response and any feedback/guideline would be really great and appreciated.

@remotesyssupport
Copy link

Recently got a mail from Caleb Crane who faced the same issue and apparently has solved the issue. Am quoting it here, so that the rest of us can use it if faced with the issue. The solution and all credits go to Caleb Crane.

"
I got it working in my environment. It turned out to be necessary to set the MTU to 1420 in the containers. GRE doesn’t support fragmentation so when ssh tried to send a frame at 1500 bytes the client wasn’t receiving the entire thing.
"

@mingfang
Copy link

mingfang commented Feb 8, 2015

I can confirm that --mtu=1420 is required.

@SemanticBeeng
Copy link

see this great tutorial for setting up a "MULTI-HOST DOCKER NETWORK" : https://wiredcraft.com/blog/multi-host-docker-network/

It is also being specific about MTU.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment