Alex Ellis alexellis

Practical fixes that actually work, drawn from the same NVIDIA forum threads, Reddit reports, GitHub issues, and dual-Spark recipes people are using right now.

1. Kill the most common progressive leak (multi-node / TP=2)

Add these two environment variables to every vLLM launch (or docker-compose / start script):

UCX_MEM_MMAP_HOOK_MODE=none
UCX_RCACHE_MAX_UNRELEASED=1024

Rebuilt llama.cpp:

llama.cpp/build/bin/llama-server \
  -hf unsloth/GLM-4.7-Flash-GGUF:UD-Q4_K_XL \
  --jinja --threads -1 --ctx-size 96768 \
  --temp 1.0 --top-p 0.95 --min-p 0.01 --fit on \
  --flash-attn off

This final prompt ended up having to be so specific, even giving the implementation detail as the model was so dumb.

Prompt:

We're buying licenses from slicervm.com for their microVM product.

Find out the pricing for 2x home edition licenses and 5x commercial ones the page
is https://slicervm.com/pricing

From my blog post: The Internet is my computer

Equinix Metal is my computer

Run hosted VSCode on Equinix Metal's huge: AMD Epyc instances with 64GB RAM and 24 Cores, coupled with a bonded 2 x 10 Gbps uplink to the Internet.

Provision your Ubuntu 20.04 LTS server using the dashboard and add your SSH key
Once you have the public IP, log in over SSH

Installing OpenFaaS with libkrun-based isolation on Fedora 33

Setting up the system

Revert to cgroup v1

The version of containerd that OpenFaaS will download during the installation doesn't support cgroup v2, so we need to configure systemd to avoid using the Unified cgroup Hierarchy, and reboot the system:

	config:
	host_groups:
	- name: coral
	userdata_file: ./userdata.sh
	storage: image
	storage_size: 25G
	count: 1
	vcpu: 2
	ram_gb: 4
	network:

	# Example from https://docs.slicervm.com/getting-started/walkthrough/

	# Then under the hostgroup instead of userdata:
	# userdata_file: ./userdata.sh


	# Then add a DNAT rule from the host to the microVM via iptables (ask ChatGPT for this!)

	# Or simply add a static route on your router via your Slicer host to the microVM - easy to do
	# in most router admin interfaces, then also set the default DNS to the VM IP there too.

	package main

	import (
	"context"
	"flag"
	"fmt"
	"os"
	"reflect"
	"time"

	#! /bin/sh

	# Alex Ellis 18th Nov 2020
	# Updated for the 1U unit, which has 7 fans

	logo () {
	echo ""
	echo " ▄██▄"
	echo " ▄█ █▄"
	echo " ▄█ █▄"

Alex Ellis alexellis

1. Kill the most common progressive leak (multi-node / TP=2)

Equinix Metal is my computer

Installing OpenFaaS with libkrun-based isolation on Fedora 33

Setting up the system

Revert to cgroup v1

Resources

Side-project thread

GitLab 5-minute rails