Guide for migrating data in Pocketbase v23 upgrade

pocketbase_v23_data_migration_guide.md

Upgrading to v23

Upgrading to Pocketbase v23 was a large undertaking due to the amount of fundamental changes to extending as a framework. The library contains several excellent resources, including the GitHub Release and Pocketbase Upgrade Guide, that help with the upgrade. However, they do not explain how to handle upgrading deployed/production instances, which is a bit more tricky depending on the server host. This guide deals with upgrading a Pocketbase application deployed on Pockethost, an excellent and affordable hosting service for Pocketbase apps.

Official Upgrade Guide

The Pocketbase Upgrade Guide provides an excellent guide to the major breaking changes. However, there were a few assorted changes not explicitly documented (and there may be others still).

Email templates no longer have ACTION_URL

Email templates have been moved to per-auth collection (from global settings), and no longer have a customizable ACTION_URL parameter. Instead, the email template itself can be modified (as previously) to compose the APP_URL variable with the required frontend path.

Per-collection token settings

Similarly to email templates being moved to per-auth collection (from global settings), so to have the token settings (expiries, etc).

Upgrading Migrations

While the official guide provides an excellent path through the upgrade, it is targeted at local/development environments, or perhaps even deployments where the developer has full SSH access. However, for developments where this is not the case (such as Pockethost), additional steps are necessary for properly handling migrations.

While it theoretically would be possible to manually update the existing migrations to the new Pocketbase v23 approach, which is necessary for setting up new environments in the future, it is hardly practical. Instead, the upgrade guide recommends deleting existing migrations and generating a new collections snapshot (migration).

Collapsing Migrations

Collapsing already-deployed migrations is generally not a best practice, as it can result in unexpected results in both the current deployments as well as future deployments (in new environments). However, there are cases where this is necessary, such as when a migration tool itself is overhauled (as is the case with Pocketbase v23). When collapsing migrations is necessary, caution is required to ensure proper results across development, deployed, and future environments!

Thankfully, collection snapshot migrations (by default) will not delete additional tables/fields (ie. present in database but not migration) when applying the snapshot migration against an existing database. Therefore, if the snapshot migration is applied to target deployment relatively soon after generating, there should be no issues/changes.

However, any previous manual changes/migrations (ie. data seeding, manipulation, etc) that are not captured in the generated (collapsed) snapshot will need to be manually handled (for setting up new databases in future). Since these changes have already been applied in current deployments, their migration must be "skipped" in current deployments (to avoid duplicates, etc). Similarly, the _migrations table will technically contain a list of migrations that no longer exist (since they are rolled up into new snapshot migration). Both of these "issues" can be handled by manually updating the deployed SQLite data.db file.

Warning

Prior to collapsing migrations, ensure all pb_data directories are backed up (local and remote)!

Migration Preparation

1. Ensure local Pocketbase binary is <v23
2. Ensure development and deployed Pocketbase collections structure is effectively identical
   • Changes between data environments could cause inconsistencies later
3. Delete existing local migration files (for upcoming history sync)
   • These migrations will be replaced by upcoming "collapsed" collection migration
4. Copy deployed pb_data directory to local environment
   • Using deployed (production, etc) data ensures most accurate upgrade experience!
5. Upgrade local Pocketbase binary to v23
6. Run pocketbase serve to apply built-in upgrade migrations (then stop server)
7. Run pocketbase migrate collections to generate "collapsed" migration (ie. current collection state)
   • Must be run after applying built-in v23 upgrade migrations to properly generate updated collection state!
8. Run pocketbase serve again to apply/track this migration (then stop server)
   • Applying snapshot migrations (by default) will not remove "unrecognized" data/tables/fields
9. Run pocketbase migrate history-sync to remove database tracking of now-deleted migrations
   • This removes any records in _migrations table that no longer have an associated migration file
   • Technically this is optional, as extra records have no negative impact, but still recommended (less confusing long-term)
10. Create new migration(s) with required settings initialization, seed data, or modifications (optional)
    • While dev and deployed environments should have this already, future created databases would not otherwise!
11. Manually update data.db to insert these data "initialization" migrations into _migrations table
    • ⚠️ Must be done before running Pocketbase server again (to avoid applying migration)!
    • Must manually track these migrations to avoid executing them again on existing deployments!
    • Use a tool like sqlite3 to modify the data.db file (_migrations table)
    • Use an applied timestamp after the previous applied migration (or the previous migration for simplicity)
      • Timestamp for applied is the (13-digit) UTC epoch seconds value
12. Run pocketbase serve again to ensure data "initialization" migrations were not applied again
13. Test thoroughly!

Migrating Deployments

After preparing the migration process locally, and generating an updated set of migrations, it is time to handle migrating any existing deployments (ie. production). If there is no shell access to the server, any data.db updates will need to be applied locally, then re-uploaded (which this guide covers). Similarly, some servers may require creating a new instance entirely for the upgrade to v23!

1. Stop remote deployment instance!
   • Must stop deployed instances to avoid data corruption!
2. Upgrade deployed Pocketbase instance to v23
3. Ensure deployment instance is stopped after upgrade!
4. Download deployed data.db file to local environment
5. Ensure previous migrations were removed from local environment
6. Run pocketbase migrate history-sync to remove database tracking of previous migrations
7. Move (temporarily) all migrations except snapshot migration out of pb_migrations
   • Initialization migrations should not be applied in next step (will be handled manually)!
8. Run pocketbase serve to apply built-in upgrade migrations and snapshot migration (then stop server)
9. Manually update data.db to insert any data "initialization" migrations into _migrations table
   • ⚠️ Must be done before copying other migrations back into pb_migrations (to avoid applying migration)!
10. Move all data initialization migrations back into pb_migrations
11. Run pocketbase serve again to ensure data "initialization" migrations were not applied again
12. Test thoroughly! (with local server)
13. Remove old deployment logs.db file (no longer used)
14. Remove old deployment migration files
15. Upload modified data.db to deployment environment
16. Upload collapsed migration (and any data initialization migrations)
17. Start remote deployment instance
18. Test thoroughly! (with deployment server)