grok-harvester-vm-dpdk-argo

https://x.com/i/grok/share/u7qUed0FwgrMDIA7nMJl11iLL
coverity https://x.com/i/grok/share/eZ60X6fI3c7i6ZUutW0eeDAm4

netbox: https://x.com/i/grok/share/kBw2S30rCgA3mZ5VMKtESNdVS

https://x.com/i/grok/share/UZUEebJmIIqhEFdSZDUZYukDe
"Configuring HCI Harvester for VLAN 1076": https://x.com/i/grok/share/motnb7lGG6VLPnop4m9ctsxVi
"Ansible Script for PXE Boot via BMC LAN": https://x.com/i/grok/share/Z4Cxy97XDEjGbWMAnJEptkSln
"Ansible Script for PXE Boot via BMC": https://x.com/i/grok/share/8dU03Skr2MCECoGw7SJsscXsC
"Harvester Cluster Networking and Hugepages Setup": https://x.com/i/grok/share/uWufwtXjrgOg11yc3Q64qk9ke


I) I have a Harvester Cluster, RKE2 up and running and accessible under ip address 10.123.235.200/22. I have also Rancher standalone deployed as well as Argo stack.
Everything is running on Intel based CPU (Xeon) with VT-d enabled and ready to use Intel E810 network cards in each node with SRIOV support enabled.
I want to deploy a 3 node ephemeral Kubernetes cluster on Virtual Machines provisioned on demand, that uses custom build of DPDK and Ice drivers for E810 hardware network interfaces from host I a way that I could urilize it as Virtual NIC and/or VF driver inside the VM.
I need the best known method to do it in an automated way, for example using ansible, helm or just argo. The purpose of this is to test Intel ICE drivers on top of witch MTL and MCM from https://www.github.com/OpenVisualCloud are being run.

II) Reiterate above, but focus on NIC part utilizing Intel Ethernet Operator and/or SR-IOV Network Device Plugin for Kubernetes.

I have broken some of my linux directories probably (like /usr, /lib, /sbin) and now trying to type any command ends in file not found (but my ssh connection was not broken then - after rebooting the OS it boots up to lopading ice driver and then fails (file not found + kernel panic). What is the best way to fix it? How can I recover the system files
[,,.]
The last 2 lines goes like Begin: Running /scripts/init-bottom... done. run-init: can't execute '/sbin/init': no such file or directory
[...]
I have managed to mount the partitions inside other running ubuntu 24.04 on second drive. /dev/mapper/ubuntu--vg-ubuntu--lv on /mnt/reconstruct/c type ext4 (rw,relatime) /dev/nvme0n1p1 on /mnt/reconstruct/a type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro) /dev/nvme0n1p2 on /mnt/reconstruct/b type ext4 (rw,relatime) When I try to do the chroot to /mnt/reconstruct/c where I have mounted the lkinux drive of broken system I am getting error: root@bcs-cicd-2:/mnt/reconstruct# chroot /mnt/reconstruct/c chroot: failed to run command '/bin/bash': No such file or directory
[...]
SO i've done: root@bcs-cicd-2:/# cp /bin/bash /mnt/reconstruct/c/bin/ root@bcs-cicd-2:/# sudo mount --bind /dev /mnt/reconstruct/c/dev sudo mount --bind /proc /mnt/reconstruct/c/proc sudo mount --bind /sys /mnt/reconstruct/c/sys sudo mount --bind /run /mnt/reconstruct/c/run and root@bcs-cicd-2:/mnt/reconstruct# chroot /mnt/reconstruct/c chroot: failed to run command '/bin/bash': No such file or directory root@bcs-cicd-2:/mnt/reconstruct# chroot /mnt/reconstruct/c chroot: failed to run command '/bin/bash': No such file or directory root@bcs-cicd-2:/mnt/reconstruct# sudo chroot /mnt/reconstruct/c /bin/bash chroot: failed to run command ‘/bin/bash’: No such file or directory

#!/bin/bash
# On the working Ubuntu 24.04 system with broken system mounted at /mnt/reconstruct/c

# Ensure critical dirs exist
mkdir -p /mnt/reconstruct/c/{bin,lib,lib64,sbin,usr/bin,usr/lib,usr/sbin,etc,dev,proc,sys,run}

# Copy essential binaries and libs from working system
cp -a /bin/bash /mnt/reconstruct/c/bin/
cp -a /lib/x86_64-linux-gnu/{libc.so.6,libdl.so.2,libtinfo.so.6,libselinux.so.1} /mnt/reconstruct/c/lib/
cp -a /lib64/ld-linux-x86-64.so.2 /mnt/reconstruct/c/lib64/

# Bind mount essential filesystems
mount --bind /dev /mnt/reconstruct/c/dev
mount --bind /dev/pts /mnt/reconstruct/c/dev/pts
mount --bind /proc /mnt/reconstruct/c/proc
mount --bind /sys /mnt/reconstruct/c/sys
mount --bind /run /mnt/reconstruct/c/run

# Copy resolver config
cp /etc/resolv.conf /mnt/reconstruct/c/etc/

# Chroot and fix
chroot /mnt/reconstruct/c /bin/bash <<EOF
# Update package cache
apt-get update

# Reinstall core packages
apt-get install --reinstall coreutils bash init systemd

# Repair broken packages
apt-get install -f

# Reinstall critical system components
apt-get install --reinstall linux-image-$(uname -r) grub2

# Regenerate initramfs
update-initramfs -u -k all

# Update grub
update-grub

# Exit chroot
exit
EOF

# Unmount binds
umount /mnt/reconstruct/c/{dev/pts,dev,proc,sys,run}

# Reboot the broken system
reboot

#!/bin/bash

# If chroot still fails, use debootstrap to rebuild base system
debootstrap --arch=amd64 noble /mnt/reconstruct/c http://archive.ubuntu.com/ubuntu/

# Repeat bind mounts and chroot steps above

#!/bin/bash
# If kernel panic persists, boot with live USB, mount root, and reinstall kernel
mount /dev/mapper/ubuntu--vg-ubuntu--lv /mnt
mount /dev/nvme0n1p2 /mnt/boot
mount /dev/nvme0n1p1 /mnt/boot/efi
for i in /dev /dev/pts /proc /sys /run; do mount --bind $i /mnt$i; done
chroot /mnt
apt-get install --reinstall linux-image-$(uname -r)
update-initramfs -u -k all
update-grub
exit
for i in /mnt/dev/pts /mnt/dev /mnt/proc /mnt/sys /mnt/run; do umount $i; done
umount /mnt/boot/efi /mnt/boot /mnt
reboot