scyto

Using VirtioFS backed by CephFS for bind mounts

This is currently a work-in-progress documentation - rough notes for me, maybe missing a lot or wrong

The idea is to replace GlusterFS running inside the VM with storage on my cephfs cluster. This is my proxmox cluster and it runs both the storage and is the hypervisor for my docker VMs.

Other possible approaches:

• ceph fuse clien in VM to mount cephFS or CephRBD over IP
• use of ceph docker volume plugin (no useable version of this yet exists but it is being worked on)

Assumptions:

scyto

Install Debian

non graphical, SSH and basic tools only

apt-get install nano sudo nfs samba-common

usermod -aG sudo [your-username]

switch to username all commands thereafter use sudo when needed

add contrib sources (why?)

scyto

Auto Label

This container puts a label on each machine based on a config that matches service names. If the service is running the label gets a 1 and if the label get as 0 its not running. this can be used with constraints to either locate serices on a node with another service OR make sure a service doesn't land on a node with another service

I would love to find a better version of this that does this without the need for the manual config file (you can use a file bindmount instead of a config if you prefer)

Swarm Consideration

State is all read-only in a config

scyto

Portception - deploying portainer with portainer in a swarm

No one should be like scyto, no one should do this..... be prepared to see your portainer disappear in a puff of smoke if you get this wrong

Prep

1. This assumes all nodes are manager nodes
2. This assumes you already have agents managed as s stack / swarm service via portainer (see my other not recommended stack)
3. this assumes you have the portainer bind mounts on some shared medium (ceph, gluster, NFS, SMB - if you run it on one of the last two don't blame me if it corrupts)
4. my suggestiton is get your non-managed portainer working with your shared storage before you go any further
5. BACKUP ALL YOUR STACKS / SECRETS AND CONFIGS - WORST CASE YOU CAN RECREATE EVERY STACK / SECRET / CONFIG BY HAND FAIRLY QUICKLY

scyto

Enable Dual Stack (IPv4 and IPv6) OpenFabric Routing

this gist is part of this series

This assumes you are running Proxmox 8.2 and that the line source /etc/network/interfaces.d/* is at the end of the interfaces file (this is automatically added to both new and upgraded installations of Proxmox 8.2).

This changes the previous file design thanks to @NRGNet for the suggestions to move thunderbolt settings to a file in /etc/network/interfaces.d it makes the system much more reliable in general, more maintainable esp for folks using IPv4 on the private cluster network (i still recommend the use of the IPv6 FC00 network you will see in these docs)

This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node

scyto

Introduction

This one is the one that has to work, even more so the domain controllers. This is what my swarm looks like

you may want to read from the bottom up as later migrations are where i had the process more locked and less experimentation

The plan

So the plan is as follows (and is based on my experience with home assistant oddlye enough)

1. Backup node 1 VM with synology hyper-v backup

scyto

Migrating Home Assistant OVA VM from Hyper-V to Proxmox

Now that i have nailed the qm disk import command and given all linux kernel have the virtio drivers in them after 5.6 this should be a breeze!

Export

Export VHD from Hyper-V into share proxmox can see (tbh at this point if you don't know how...)

Create VM on proxmox

I created a 4GB VM with no disks at all andthe virtio network. Make sure you connect it to a live bridge or hass will hand at starting network manager I added a TPM drive

scyto

Migration Steps

Intro

I Use windows admin center - it is a sever 2019 no gui install. I am uising these generic instructions to import so won't document in detail.

Learning from the disaster moving my DCs where i went the hardway (backup and restore - which did work) this time i will use the right versions of the disk import command qm disk import [...]

Driver install

I had real issues with driver install - while i ran the installer and everything seemed to install it didn't

scyto

Don't be like scyto

Don't do all the backup and restoring crap below

Only reason vhdx import wouldn't work is becuause i mis-documented the command as qm import.. when it is qm disk import

I could easily have imported the vhdx all along.... learn my lesson padawan

original gist content

Migrating Domain Controller 1 from Hyper-V to Proxmox by using Synology Backup

Why? Well it turns out long lived VHDX's often error on import with qm and never import.

scyto

Don't be like scyto

Don't do all the restoring crap below

Only reason vhdx import wouldn't work is becuause i mis-documented the command as qm import.. when it is qm disk import

I could easily have imported the vhdx all along.... learn my lesson padawan

original gist content

Random Notes (stream of real-time conciousness) on Migrating Windows Server Core 2019 based AD domain controler

tl;dr it worked - but due to an issue with the disk I had to use the synology bare metal restore into the VM and then use the disk shuffle approach i outlined in the parent gist to this one