So say that you have been running your TrueNAS SCALE 24.04 or 23.10 or older happily with your TrueCharts apps installed (as happy as you could be with that buggy mess of a project).

You either stuck with 23.10 Cobia, because you didn't want to bother doing yet another TrueCharts troublesome migration where they ~~removed~~ did something with PVC rearrangement and expected you to re-setup all your apps yet again (!), or you upgraded to 24.04 Fangtooth either while ignoring these instructions and apps still kept on working or running the PVC migration script as intended.

You kept your apps config storages to PVC as that was a simple default. _{(You start to grow a certain regret of not using host path mounts for everything from the beginning, but that's all water in drain now.)}

Now you want to backup that data so that you can move to Docker-based apps in upcoming 24.10 Electric Eel release flawlessly and gracefully, and be happy everafter.

This tutorial and a bunch of pointers here is for you then! Read and follow it carefully and (hopefully) you won't get lost.

Steps

Do note that these are just steps I figured out with basic research and trial and error. Some things might not be the best practices and I don't take responsibility for some Kubernetes nerds ending up with a heart attack after reading it. The method works though, so in my book it's valid.

Important

Do these steps with all your apps that you want to back up running (and hopefully keep them idle).

My methods here are simple but rely on the apps actually running and taking advantage of an existing runtime PVC mount, without having to bother with complicated commands to do it manually. Just take advantage of what we have already.

So don't shut off your apps before doing this. Start them up.

All commands assume you're running from root shell (do sudo su - or something if not). I also assume you only ran TrueCharts apps, so I don't focus on official TrueNAS apps, but some steps will apply to them too (apart from data backup, I have no idea how iXvolume works, as I never used these apps).

I also logically assume you're doing these steps right before an upgrade to Electric Eel, since otherwise your backed up apps data would get stale if you put it off.

Replace /mnt/tank/mybackup with the path you're backing up to. Ideally have that path be part of a dataset without ACLs enabled (as is by default), otherwise you may be unable to backup ownership information of files+dirs.

0. In-app backups

Before you start, some apps out there with web admin UI panels have some built-in option of exporting and downloading a backup of database/config. If any of your apps do happen to have such a feature, use it and download such a backup as an extra measure as well.

1. App config screenshots

Firstly save screenshots of all your apps' configs in the UI. Yes, we'll save some extra data about your apps to the JSON files later, but looking at the visual configuration will prove to be easier for the most part, so just do it and thank yourself later. Use the extension FireShot to take the screenshots, as it works and you won't be stitching screenshots together manually like a moron.

These will allow you to view things like what you chose for particular app storage type on a glance, including custom mounts that you'll need to re-add later (this tutorial doesn't cover backing up host mounts, as I assume they're safe on your pool somewhere already, just take note of them).

2. Dump some helpful info to text files

mkdir -p /mnt/tank/mybackup/k3s-info/
k3s kubectl get pvc -A | sort -u | awk '{print $1}' | tail -n +2 | sort -u | xargs -I {} sh -c 'k3s kubectl -n {} describe deploy > /mnt/tank/mybackup/k3s-info/{}.txt'

The above will save some per-app information from Kubernetes runtime. Might be a bit verbose and hard to follow, but this contains your full underlying app config, ports (note: only internal ports here, not exposed ones, so keep the screenshots above handy as well), ENV, mounts. This maybe come useful for you, for example to copy-paste ENV vars to new place afterwards.

Now we'll back up a list of our apps together with their PVC mount names and IDs:

k3s kubectl get pvc -A | sort -u | awk '{print "\t" $1 "\t" $2 "\t" $4}' | column -t | tee /mnt/tank/mybackup/truenas_pvc_names.txt

Now in the file /mnt/tank/mybackup/truenas_pvc_names.txt you should have something like this:

#    NAMESPACE            NAME                                                                            VOLUME
1)   ix-firefox           firefox-config                                                                  pvc-8c0f36f9-13f9-4890-a914-660ee7a39f5d
2)   ix-gitea             db-gitea-postgresql-0                                                           pvc-76bf480b-d305-4d77-b2f5-e19d5b1e8b2f
3)   ix-gitea             gitea-data                                                                      pvc-3387d272-6c3a-4411-872a-8ab16cdbb1ea
4)   ix-jellyfin          jellyfin-config                                                                 pvc-a1e4b06e-adec-4085-8f79-0dc42e8771c3
5)   ix-jellyfin          jellyfin-transcode                                                              pvc-afe8dc5f-ac55-45ee-8b65-c22eebe54a13
6)   ix-librespeed        librespeed-config                                                               pvc-90bd4a63-4231-4bac-a247-03ba3900a13d
7)   ix-paperless-ngx     data-paperless-ngx-redis-0                                                      pvc-e4012924-a839-46f1-b1f4-6c21260a238b
8)   ix-paperless-ngx     paperless-ngx-consume                                                           pvc-750af274-5234-49bf-86b9-60b4d2ae9454
9)   ix-paperless-ngx     paperless-ngx-data                                                              pvc-0848c5e0-f6c7-4922-9262-13c965b7e861

Warning

If you have mount names prefixed with db-, such as db-gitea-postgresql-0, it's likely they're TrueCharts' special thing with their special setup/handling of databases. In other words, you may have some extra trouble with trying to restore/deal with this data. If you see it at this point and care about said app, I'd recommend to pause here and figure out what's the best way to backup that separately: some apps have built-in database export feature, use that if available. Otherwise I recommend trying to figure out how to take an SQL dump from within container shell just to be safe.

Thankfully this doesn't apply to most apps, and I didn't really care about this Gitea instance I had, so I'm happily moving on, yolo.

3. Backup the PVC data

Let's create a tree-like directory structure for all of our mounts. Assuming you have a table like above, you'd do:

mkdir /mnt/tank/mybackup/apps-pvc-data
mkdir -p /mnt/tank/mybackup/apps-pvc-data/ix-firefox/firefox-config
mkdir -p /mnt/tank/mybackup/apps-pvc-data/ix-gitea/{db-gitea-postgresql-0,gitea-data}
mkdir -p /mnt/tank/mybackup/apps-pvc-data/ix-jellyfin/{jellyfin-config,jellyfin-transcode}
mkdir -p /mnt/tank/mybackup/apps-pvc-data/ix-librespeed/librespeed-config
mkdir -p /mnt/tank/mybackup/ix-paperless-ngx/{data-paperless-ngx-redis-0,paperless-ngx-consume,paperless-ngx-data}

Now it's time for some manual labour of copying the PVCs one by one.

Repeat the below to copy every single item from the NAME column into your backup. For example, for the two Jellyfin datasets, we'll run these two commands (since Jellyfin has these two mounts on PVC):

rsync -ra $(find /var/lib/kubelet/pods/ -name 'pvc-a1e4b06e-adec-4085-8f79-0dc42e8771c3')/mount/ /mnt/tank/mybackup/apps-pvc-data/ix-jellyfin/jellyfin-config/
rsync -ra $(find /var/lib/kubelet/pods/ -name 'pvc-afe8dc5f-ac55-45ee-8b65-c22eebe54a13')/mount/ /mnt/tank/mybackup/apps-pvc-data/ix-jellyfin/jellyfin-transcode/

Replace ix-jellyfin, jellyfin-config/jellyfin-transcode, pvc-xxx with your values from the table. Run one command for each row. Double-check you replaced the three things in the command properly with your row from the table. Afterwards I recommend sanity-check of ls -la /mnt/tank/mybackup/apps-pvc-data/ix-jellyfin/jellyfin-config/ to ensure your app's mount data is there as it should.

(in rsync, -r stands for recursive, -a ensures that chmod, owner and links will also be copied over)

4. Do full backup of `/var/lib/kubelet/pods/`

This is where your mounted PVCs are located, as you could judge from the commands we ran above. We want to back that full thing up for future reference just in case we messed up and omitted something.

tar -acvf /mnt/tank/mybackup/var-lib-kubelet-pods.tar.zst /var/lib/kubelet/pods/

Also don't bother trying to back up ix-applications dataset with tar, by default from filesystem you only have access to the binaries of your containers and that's gonna be useless. The underlying PVCs there are not mounted there and thus not accessible. They're in subdirs of /var/lib/kubelet/pods/ (again: only for currently running apps!!!), which we took care of backing up in the previous step.

5. Final look back

It won't hurt now to be safe and once again review your screenshots from step 1. Look over the "Persistence" section and each item. Ensure you have safely backed up every entry there. The PVC entires should have been backed up to the dirs we created and probably non-empty. The host paths should be safe on your pools. Same with network mounts. Everything should be okay, but this is the last chance to catch any mistakes.

6. ☢️ Nuke

Now comes the scary part. When it's time for upgrade to Electric Eel, we want to nuke the old ix-applications dataset. Otherwise post-upgrade the iX scripts will attempt to do their own upgrades/conversions, which were designed FOR OFFICIAL APPS ONLY. There's a chance things will implode if you were to keep TrueCharts apps on there.

Note

Nuking this should be fiiiine at this point, since we already did two methods of backups above. But if you're paranoid, you can always figure out how to replicate/rename the dataset somewhere else for some time instead of junking it. Just make sure to Unset Pool before doing that (Unset Pool doesn't actually remove it, only makes the apps disappear from the UI and Kubernetes to leave it alone).

Alternatively if you want to rely on built-in scripts to convert your official apps, do remove just TrueCharts apps and keep only the official ones, then you might keep the dataset and cross your fingers.

If you only have TrueCharts apps and wish to proceed with the act of destruction:

Shut down all apps
Remove all apps
Press Configuration -> Unset Pool, confirm
Now you should have a screen saying you have no apps installed.
Time to go back to terminal to nuke ix-applications dataset:

Figure out where's your ix-applications:

zfs list | grep ix-applications | head -n 1

For example: tank/ix-applications (assuming your parent dataset is tank and ix-applications was created as its direct child)

Now do:

zfs unmount -f tank/ix-applications
zfs destroy -f tank/ix-applications

Here we go, the junk was purged. Now we can upgrade.

(you can run the listing command again to ensure ix-applications is actually gone)

Life in Electric Eel ⚡🐍

How we'll re-setup our backed up and new apps.

0. Preparation

Structure

Now I'll share my preferred structure for hosting apps data, which will help you keep things organized in one place, in logical structure, and in a way that's easy to backup one way or another (replication, snapshot, tar, you name it).

TrueNAS now creates a dataset ix-apps in the parent dataset that you choose, let's say in our case this will again be tank. We don't touch their dataset, instead we'll create our own one that will keep our app data in a similar tree structure to our backup that we previously took.

So in my case I'll go into the UI -> Datasets, and I'll create a dataset called apps-data under tank dataset. I'll change the Dataset Preset option to Apps. I now end up with my empty folder at /mnt/tank/apps-data.

Here I create a directory for each of my apps. So let's focus on the Jellyfin example from above.

mkdir -p /mnt/tank/apps-data/jellyfin/{cache,config}
chown -R 568:568 /mnt/tank/apps-data/jellyfin/{cache,config}

Jellyfin just so happens to allow configuring arbitrary user ID and GID, so we also set them to 568.

Now I can copy over my data from the backup:

rsync -ra /mnt/tank/mybackup/apps-pvc-data/ix-jellyfin/jellyfin-config/ /mnt/tank/apps-data/jellyfin/config/

Turns out the available mounts are different. It seems "cache" folder wasn't mounted before, so I don't copy anything to it. I don't create a directory for transcode, as I'll want to keep that in a temporary directory instead, available in the UI configuration as Temporary (Temporary directory created on the disk) (or tmpfs (Temporary directory created on the RAM)).

So you end up with a tree like this:

apps-data/
 \-- jellyfin/
      \-- cache/
      \-- config/
 \-- immich/
      \-- postgres/
      \-- compose.yml
      \-- .env

I hope you get the idea. The compose.yml files is what we'll use for Custom App ran with Docker Compose, and we'll put them in root dir of our app's folder.

Of course you don't have to keep all mounts there! If you add extra custom mounts, or some predefined app mounts, that are huge in size, you're free to point those particular mounts elsewhere, for example having apps-data with your configs and caches on SSD pool, and something like actual media storage for Jellyfin on a HDD pool.

So in case of Jellyfin you have to create a new mount with arbitrary mount path in the container such as /medialibrary, then point it to your media library wherever it is, then add /medialibrary as library scan path in Jellyfin admin panel).

Permissions

Note that TrueCharts allowed you to select user, gid and fs id of the apps. It was often a recommendation to use 568 everywhere for apps user and group. Official apps and Docker images you'll use with Custom App don't have a uniform way of doing that. Some apps ask you to input the UID+GID in ENV vars (which may then be exposed in TrueNAS app config, or possible to add manually), with Custom Apps you'll be able to add user: 568:568 line to the YAML file, but that always won't work properly.

In other words, in some cases your app will end up running on a different user and group than it did before. If it runs on root, then that's not really a problem since root can do anything with files. If it runs under some arbitrary user, such as 1000:0 for ElasticSearch, you may need to chown -R on the mounted paths to the expected owner to let the app run properly. If in doubt, simply refer to the documentation of the app you're trying to run, they'll usually explain the exact expectations they have in this regard.

You now have two paths for figuring out how to run your apps in the new Docker-flavored reality:

1. TrueNAS official apps

Their official catalog is growing and most of the app configurations are sane. They're quickly adding new apps and fixing reported issues with existing ones. It's not a bad option at all for simpler/default setups at all. Do note that if you had advanced configs such as some crazy VPNs and routing and Traefik and whatnot, then maybe not.

Also note that there's no super-obvious ways of backing up your primary app configs with this method, as you set it up in the UI, so maybe write down your custom settings somewhere or take screenshots of config panel from time to time.

But also remember that network-wise you can turn on "host mode" and then just directly instruct your app to bind to some IP or interface right on your host, without all the abstractions.

Otherwise just set your app up, fill in the details correctly. Reference your screenshots or .txt files to figure out what ports and ENV vars you used. Add the same extra mounts you had. Point your correct app storage paths to the dirs you've set up in step 0 (don't use iXvolume, don't repeat the PVC fiasco again, feel free to only use iXvolume or temp dir for ephemeral temporary data that can be junked at any time (but also please double-check that your app's hypothetical /cache directory really can be junked at any time with no consequences, I know instances of apps that keep rather important data there counter-intuitively; if in doubts, err on the safe side)).

Tip

Note that apps in the "stable" train (great minority of them) run on user 473:473. Apps in "enterprise" or "community" trains run on 568:568 aka the apps:apps user, just as TrueCharts app did. You may need to adjust permissions for datasets of apps like Minio.

Now if you didn't mess up mimicking the configuration, finish up creating the app and start it up, and that's it, it should work. The most common issues you'll run into are permission issues, you'll need to take a look at them individually every time.

2. Docker Compose apps

TrueNAS allows you to create a custom Docker Compose app now pretty easily. All you have to do is go to: Apps -> Discover Apps -> Custom App. You then enter a name and paste contents of Compose file.

Unfortunately out-of-the-box, at the moment of writing this article, the custom apps have a few problems:

cannot reference complimentary files with relative paths, project work-dir is set arbitrarily for you
cannot provide .env file with environmental variables to use
in current RC1 a bug: cannot edit app's Compose file after it was already created (this is fixed in stable .0 branch already)
cannot easily backup the config file as it's in a different place from your apps dataset
cannot easily reference a Dockerfile to build with relative location

But there's a neat trick around all these limitations, courtery of @sfatula that was described here: https://forums.truenas.com/t/electric-eel-how-i-am-using-dockerfile-env-files-compose-files/15252

In short, create your compose.yml file under your apps-data, for example at /mnt/tank/apps-data/immich/compose.yml. Yes, nowadays the new convention is to name these files just compose.yml instead of docker-compose.yml, but that doesn't really matter for us.

Now create a new custom app, enter its name (such as immich), and in the compose contents put:

include:
  - /mnt/tank/apps-data/immich/compose.yml

And that's it! You can use your linked compose.yml file like you normally would (for contents, refer to documentation of the app you want to install obviously). If you update it, simply hit Edit and Save under the app to recreate it ^{(in bugged RC1 you remove the app and re-create it with the same snippet)}.

For .env files, you may need to use {env.PORKBUN_API_KEY} expansion instead of ${PORKBUN_API_KEY} (TODO: that claim remains to be tested). You may also need to specify env_file: .env explicitly under your containers to let them see it too.

If you need the app to run on a different user and it doesn't have environment variables to control how it runs, you may try adding user: 568:568 directive to the compose file under the containers, but it's not always guaranteed to work!

Note

Just rememer to not run docker compose inside of your dir where you put the compose.yml file. The workdir of your Custom App is different, so messing with docker compose directly in that dir will show you nothing running or spawn duplicates and then spew errors about duplicated ports etc.

You have to remember to not habitually do that if you're coming from a system where you managed the Docker Compose stacks directly from the CLI. For how to use docker compose now, read on below.

Bonus tip

Now I'll sell you my own tip, courtesy of me.

If you want to use the docker compose command to aid you in some tasks:

First check list of projects with docker compose ls. It will print out a table of NAME, STATUS, CONFIG FILES. The names are in format of ix-<HowYouNamedYourAppInUI>
Now run a command like docker compose -p ix-minio ps

Tada 🎉. Very neat. You can use it for some things if you can't be bothered to do them in the UI or factually can't do them there, and it's cleaner than messing up with the containers individually. As for why you can't instead enter the workdir of Compose project like usually with stacks or use the -f option to specify the runtime path of Compose file: don't ask me.

Closing words

I wrote these steps down around a week after a successful migration done in more-less this way, on 24.10 RC-1. Feel free to share any feedback in comments. I found my fair share of useful tips from people in this community, so now it's my time to give back.

p0358/truecharts_pvc_backup.md

Steps

0. In-app backups

1. App config screenshots

2. Dump some helpful info to text files

3. Backup the PVC data

4. Do full backup of `/var/lib/kubelet/pods/`

5. Final look back

6. ☢️ Nuke

Life in Electric Eel ⚡🐍

0. Preparation

Structure

Permissions

1. TrueNAS official apps

2. Docker Compose apps

Bonus tip

Closing words

p0358 commented Mar 11, 2025

cjshrader commented Apr 16, 2025

mgillman commented Apr 25, 2025

p0358/truecharts_pvc_backup.md

Steps

0. In-app backups

1. App config screenshots

2. Dump some helpful info to text files

3. Backup the PVC data

4. Do full backup of /var/lib/kubelet/pods/

5. Final look back

6. ☢️ Nuke

Life in Electric Eel ⚡🐍

0. Preparation

Structure

Permissions

1. TrueNAS official apps

2. Docker Compose apps

Bonus tip

Closing words

p0358 commented Mar 11, 2025

cjshrader commented Apr 16, 2025

mgillman commented Apr 25, 2025

4. Do full backup of `/var/lib/kubelet/pods/`