Skip to content

Instantly share code, notes, and snippets.

@mdsumner
Last active April 11, 2025 22:52
Show Gist options
  • Save mdsumner/e3c8720ffd3d3c732b782cbf04d416c1 to your computer and use it in GitHub Desktop.
Save mdsumner/e3c8720ffd3d3c732b782cbf04d416c1 to your computer and use it in GitHub Desktop.
## a manifest array in a virtual Zarr store contains references like this (for this file it is >90000 references
##  because 28x51*1500x3600 in 1x1x300x300 blocks)

#'1.39.0.7': {'path': 'https://thredds.nci.org.au/thredds/fileServer/gb6/BRAN/BRAN2020/daily/ocean_salt_1993_01.nc', 
#  'offset': 302129687, 'length': 48186}, 


## so let's unpack one block
path <- "https://thredds.nci.org.au/thredds/fileServer/gb6/BRAN/BRAN2020/daily/ocean_salt_1993_01.nc"
library(gdalraster)
## we could use generic url connections but I'm not sure how to do that, GDAL VSIFile makes it easy
vsi <- new(VSIFile, sprintf("/vsicurl/%s", path))
vsi$rewind() ## ensure we are at start of file, then seek to the offset
vsi$seek(302129687, SEEK_SET)
cmp_bytes <- vsi$read(48186)

## decompress the bytes 
bytes <- memDecompress(cmp_bytes, "gzip")
length(bytes)  ## 1x1x300x300 (*2)
## rearrange because they were shuffled
bytes <- c(t(matrix(bytes, ncol = 2)))  ## "shuffle"
## unpack the 2-byte integers
vals <- readBin(bytes, what = "integer", size = 2L, signed = T, n = 300 * 300, endian = "little")
vals[vals == -32768] <- NA

## make a picture
image(matrix(vals, 300))

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment