oc mirror manual archive recovery procedure

oc-mirror-manual-archive-recovery.md

oc-mirror v2 Manual Archive Recovery

When a large oc-mirror v2 archive has been transferred into a disconnected environment and only a few images failed to push to the container registry, you can manually push them from the local cache without re-running the full transfer.

How It Works

The oc-mirror v2 cache at ~/.oc-mirror/.cache/docker/ is a native Docker Distribution v2 filesystem layout. Blobs (manifests, configs, layers) are stored under registry/v2/blobs/sha256/<2-char-prefix>/<full-hash>/data, and repository metadata lives under registry/v2/repositories/.

Three options are provided below depending on what tooling is available in your disconnected environment.

---

Option 1: Temporary Local Registry (requires the registry:2 container image)

The registry:2 container expects the Docker Distribution v2 layout at /var/lib/registry, which is exactly what the oc-mirror cache provides.

1. Start a Temporary Local Registry

podman run -d --rm --name temp-mirror \
  -p 5555:5000 \
  -v ~/.oc-mirror/.cache/docker:/var/lib/registry:z \
  registry:2

2. Copy the Image to the Target Registry

By digest (preserves the exact image reference):

skopeo copy --all \
  docker://localhost:5555/rhoai/odh-operator-bundle@sha256:cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a \
  docker://YOUR_REGISTRY/rhoai/odh-operator-bundle@sha256:cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a \
  --dest-tls-verify=true \
  --dest-creds='USERNAME:PASSWORD'

By tag (oc-mirror v2 stores digest-based tags with dashes instead of colons):

skopeo copy \
  docker://localhost:5555/rhoai/odh-operator-bundle:sha256-cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a \
  docker://YOUR_REGISTRY/rhoai/odh-operator-bundle:sha256-cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a \
  --dest-tls-verify=true \
  --dest-creds='USERNAME:PASSWORD'

3. Stop the Temporary Registry

podman stop temp-mirror

---

Option 2: Skopeo dir: Transport Only (no container runtime needed)

If you do not have the registry:2 container image available (common in strict disconnected environments), you can assemble a skopeo-compatible dir: directory directly from the cache blobs and push with skopeo copy alone.

The dir: transport expects a flat directory containing:

• manifest.json — the image manifest
• One file per blob (config and layers) named by its digest (sha256:<hash>)

All of these files already exist in the cache under blobs/sha256/<2-char-prefix>/<full-hash>/data. The steps below symlink them into the layout skopeo expects.

Manual Steps

CACHE=~/.oc-mirror/.cache/docker/registry/v2
IMAGE_DIGEST=cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a

# Create a temporary staging directory
STAGING=$(mktemp -d)

# Symlink the manifest
ln -s "$CACHE/blobs/sha256/${IMAGE_DIGEST:0:2}/$IMAGE_DIGEST/data" \
  "$STAGING/manifest.json"

# Parse the manifest and symlink every referenced blob (config + layers)
python3 -c "
import json, sys, os

manifest = json.load(open(sys.argv[1]))
staging  = sys.argv[2]
cache    = sys.argv[3]

blobs = [manifest['config']['digest']]
blobs += [layer['digest'] for layer in manifest.get('layers', [])]

for digest in blobs:
    algo, hashval = digest.split(':', 1)
    src = os.path.join(cache, 'blobs', algo, hashval[:2], hashval, 'data')
    dst = os.path.join(staging, digest)
    os.symlink(src, dst)
    print(f'  {digest} -> {src}')
" "$STAGING/manifest.json" "$STAGING" "$CACHE"

# Push to the target registry
skopeo copy \
  dir:"$STAGING" \
  docker://YOUR_REGISTRY/rhoai/odh-operator-bundle:sha256-$IMAGE_DIGEST \
  --dest-tls-verify=true \
  --dest-creds='USERNAME:PASSWORD'

# Clean up
rm -rf "$STAGING"

As a Reusable Script

Save this as push-from-cache.sh and call it for each failed image:

#!/usr/bin/env bash
set -euo pipefail

CACHE="${OC_MIRROR_CACHE:-$HOME/.oc-mirror/.cache/docker/registry/v2}"
DEST_REGISTRY="${1:?Usage: $0 <dest-registry> <namespace/repo> <sha256-digest>}"
REPO="${2:?}"
IMAGE_DIGEST="${3:?}"

STAGING=$(mktemp -d)
trap 'rm -rf "$STAGING"' EXIT

# Symlink manifest
ln -s "$CACHE/blobs/sha256/${IMAGE_DIGEST:0:2}/$IMAGE_DIGEST/data" \
  "$STAGING/manifest.json"

# Symlink config + layer blobs
python3 -c "
import json, os, sys
manifest = json.load(open(sys.argv[1]))
staging, cache = sys.argv[2], sys.argv[3]
for entry in [manifest['config']] + manifest.get('layers', []):
    algo, h = entry['digest'].split(':', 1)
    os.symlink(os.path.join(cache, 'blobs', algo, h[:2], h, 'data'),
               os.path.join(staging, entry['digest']))
" "$STAGING/manifest.json" "$STAGING" "$CACHE"

skopeo copy \
  dir:"$STAGING" \
  "docker://$DEST_REGISTRY/$REPO:sha256-$IMAGE_DIGEST" \
  --dest-tls-verify=true

echo "Pushed $REPO@sha256:$IMAGE_DIGEST to $DEST_REGISTRY"

Example usage:

./push-from-cache.sh registry.example.com:5000 \
  rhoai/odh-operator-bundle \
  cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a

Handling Multi-Architecture (Manifest List) Images

If the image digest points to a manifest list (OCI index) rather than a single manifest, the manifest will contain a list of platform-specific manifests instead of a config and layers field. You will need to push each platform manifest separately or use an OCI layout instead. Check the manifest type:

python3 -c "
import json, sys
m = json.load(open(sys.argv[1]))
print(m.get('mediaType', 'unknown'))
" ~/.oc-mirror/.cache/docker/registry/v2/blobs/sha256/${IMAGE_DIGEST:0:2}/$IMAGE_DIGEST/data

• application/vnd.docker.distribution.manifest.v2+json — single image, the steps above work directly.
• application/vnd.docker.distribution.manifest.list.v2+json or application/vnd.oci.image.index.v1+json — manifest list; push each platform digest individually using the same process.

---

Option 3: JFrog CLI with Temporary Local Registry (pushing to Artifactory)

If your disconnected environment uses JFrog Artifactory as its container registry and you have the jf (JFrog CLI) available instead of or in addition to skopeo, you can serve the oc-mirror cache with a temporary registry:2 container and use the JFrog CLI to push images into Artifactory.

Prerequisites

• podman (or docker) to run the registry:2 container
• The registry:2 container image available locally
• jf CLI installed (JFrog CLI docs)
• A Docker-type repository in Artifactory configured to accept pushes

1. Configure the JFrog CLI

If not already configured, set up a server connection and authenticate:

jf c add my-artifactory \
  --url=https://YOUR_ARTIFACTORY_HOST/artifactory \
  --user=USERNAME \
  --password=PASSWORD \
  --interactive=false

jf c use my-artifactory

2. Start a Temporary Local Registry

Same as Option 1 — serve the oc-mirror cache as a local registry:

podman run -d --rm --name temp-mirror \
  -p 5555:5000 \
  -v ~/.oc-mirror/.cache/docker:/var/lib/registry:z \
  registry:2

3. Pull the Image from the Local Registry

Use podman to pull the image from the temporary registry into local container storage:

podman pull --tls-verify=false \
  localhost:5555/rhoai/odh-operator-bundle:sha256-cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a

4. Re-tag the Image for Artifactory

Tag the image with the Artifactory destination. The tag format is <ARTIFACTORY_HOST>/<DOCKER_REPO_KEY>/<namespace>/<image>:<tag>:

podman tag \
  localhost:5555/rhoai/odh-operator-bundle:sha256-cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a \
  YOUR_ARTIFACTORY_HOST/YOUR_DOCKER_REPO/rhoai/odh-operator-bundle:sha256-cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a

5. Push with the JFrog CLI

Use jf docker push to push the image. This command wraps podman push / docker push but adds Artifactory-specific metadata (build info collection, property propagation):

jf docker push \
  YOUR_ARTIFACTORY_HOST/YOUR_DOCKER_REPO/rhoai/odh-operator-bundle:sha256-cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a

To attach the push to a build for traceability in Artifactory:

jf docker push \
  YOUR_ARTIFACTORY_HOST/YOUR_DOCKER_REPO/rhoai/odh-operator-bundle:sha256-cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a \
  --build-name="oc-mirror-recovery" \
  --build-number="1"

# Publish the build info to Artifactory
jf rt build-publish oc-mirror-recovery 1

6. Clean Up

podman stop temp-mirror
podman rmi \
  localhost:5555/rhoai/odh-operator-bundle:sha256-cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a \
  YOUR_ARTIFACTORY_HOST/YOUR_DOCKER_REPO/rhoai/odh-operator-bundle:sha256-cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a

Batch Push Script for JFrog CLI

Save as push-to-artifactory.sh for pushing multiple failed images:

#!/usr/bin/env bash
set -euo pipefail

ARTIFACTORY_HOST="${1:?Usage: $0 <artifactory-host> <docker-repo-key> <namespace/image> <sha256-digest> [digest2 ...]}"
DOCKER_REPO="${2:?}"
IMAGE="${3:?}"
shift 3

# Start temp registry if not already running
if ! podman inspect temp-mirror &>/dev/null; then
  podman run -d --rm --name temp-mirror \
    -p 5555:5000 \
    -v ~/.oc-mirror/.cache/docker:/var/lib/registry:z \
    registry:2
  echo "Started temporary registry on localhost:5555"
  sleep 2
fi

for DIGEST in "$@"; do
  TAG="sha256-${DIGEST}"
  SRC="localhost:5555/${IMAGE}:${TAG}"
  DST="${ARTIFACTORY_HOST}/${DOCKER_REPO}/${IMAGE}:${TAG}"

  echo "Processing ${IMAGE}@sha256:${DIGEST} ..."

  podman pull --tls-verify=false "$SRC"
  podman tag "$SRC" "$DST"
  jf docker push "$DST" \
    --build-name="oc-mirror-recovery" \
    --build-number="$(date +%s)"
  podman rmi "$SRC" "$DST"

  echo "Pushed $DST"
done

echo "Done. Run 'podman stop temp-mirror' when finished."

Example usage:

./push-to-artifactory.sh artifactory.example.com my-docker-local \
  rhoai/odh-operator-bundle \
  cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a

For multiple digests in one run:

./push-to-artifactory.sh artifactory.example.com my-docker-local \
  rhoai/odh-operator-bundle \
  cb6c88c74e68dbc55c9c976e251da2a38bf97feb2cf3ad0719f458a98f05352a \
  aabbccdd... \
  eeff0011...

Artifactory-Specific Notes

• Repository type: The target must be a local Docker repository in Artifactory (not remote or virtual) to accept pushes.
• jf docker vs jf rt docker-push: Both work; jf docker push is the modern syntax. If you are on an older JFrog CLI version, use jf rt docker-push <image> <repo-key>.
• Podman compatibility: The JFrog CLI uses docker by default. If using podman, set the environment variable export JFROG_CLI_CONTAINER_MANAGER=podman or ensure podman is aliased as docker.
• Self-signed certs: If Artifactory uses a self-signed certificate, configure podman to trust it by placing the CA cert in /etc/containers/certs.d/YOUR_ARTIFACTORY_HOST/ca.crt.

---

Finding the Repository Path

The repository path used in the skopeo copy source corresponds to the subdirectory structure under the cache's repositories/ directory. For example:

.cache/docker/registry/v2/repositories/rhoai/odh-operator-bundle/

maps to localhost:5555/rhoai/odh-operator-bundle.

You can list all available repositories with:

ls ~/.oc-mirror/.cache/docker/registry/v2/repositories/

And list tags for a specific repository once the temp registry is running:

skopeo list-tags docker://localhost:5555/rhoai/odh-operator-bundle --tls-verify=false

Notes

• Self-signed certs: If the target registry uses a self-signed certificate, add --dest-tls-verify=false or point to the CA with --dest-cert-dir=/path/to/certs.
• Source TLS: The temporary registry listens on plain HTTP. Localhost is typically allowed as insecure by default, but if skopeo complains add --src-tls-verify=false.
• Multiple images: Leave the temporary registry running and issue multiple skopeo copy commands before stopping it.
• Signature images: oc-mirror v2 also stores .sig tagged images alongside the main image. Check the _manifests/tags/ directory for signature tags if you need to push those as well.