Skip to content

Hybrid stores

A pyramid doesn't have to duplicate the source data. In a hybrid store, the original files serve as level 0, and only the coarser levels are written as new data. VirtualiZarr writes references to the source chunks into an Icechunk repository as group 0, and write builds levels 1..N from it. For background, see Hybrid Icechunk stores for serverless web mapping.

level 0 ยท virtual references to air.nc chunks level 1 written by topozarr level 2 written by topozarr pyramid.write(session.store, mode='a', levels=[1, 2])

A NetCDF source

Start with a chunked NetCDF file. This stands in for the archive you already have:

from pathlib import Path

import xarray as xr
import xproj  # noqa: F401

ds = xr.tutorial.open_dataset("air_temperature").drop_encoding()
ds.to_netcdf("air.nc", engine="netcdf4", encoding={"air": {"chunksizes": (100, 25, 53)}})

Write level 0 as references

Open the file as a virtual dataset and write its chunk references to group 0. The repository needs permission to read chunks from the source location:

import icechunk
from obspec_utils.registry import ObjectStoreRegistry
from obstore.store import LocalStore
from virtualizarr import open_virtual_dataset
from virtualizarr.parsers import HDFParser

here = Path.cwd()
prefix = here.as_uri() + "/"
vds = open_virtual_dataset(
    (here / "air.nc").as_uri(),
    parser=HDFParser(),
    registry=ObjectStoreRegistry({prefix: LocalStore(prefix=here)}),
    loadable_variables=["lat", "lon", "time"],
)

# let the repo read chunks from files under `prefix`
config = icechunk.RepositoryConfig.default()
config.set_virtual_chunk_container(
    icechunk.VirtualChunkContainer(prefix, icechunk.local_filesystem_store(str(here)))
)
repo = icechunk.Repository.create(
    icechunk.local_filesystem_storage("air.icechunk"),
    config=config,
    authorize_virtual_chunk_access={prefix: None},
)
session = repo.writable_session("main")
vds.vz.to_icechunk(session.store, group="0")
session.commit("virtual level 0")
print(vds.air.data)

ManifestArray<shape=(2920, 25, 53), dtype=float64, chunks=(100, 25, 53)>

Write the overviews

Plan the pyramid from the same data, then write only levels 1 and 2 with mode="a". Level 1 is coarsened from the virtual level 0, and write also adds the CRS attributes to it:

from topozarr import create_pyramid

pyramid = create_pyramid(
    ds.proj.assign_crs(spatial_ref="EPSG:4326"), levels=3, x_dim="lon", y_dim="lat"
)
session = repo.writable_session("main")
pyramid.write(session.store, mode="a", levels=[1, 2])
session.commit("overviews")

dt = xr.open_datatree(repo.readonly_session("main").store, engine="zarr", consolidated=False)
for level in sorted(dt.children):
        print(level, dict(dt[level].sizes), dt[level].attrs["proj:code"])

0 {'time': 2920, 'lat': 25, 'lon': 53} EPSG:4326
1 {'time': 2920, 'lat': 12, 'lon': 26} EPSG:4326
2 {'time': 2920, 'lat': 6, 'lon': 13} EPSG:4326

On S3, the steps are the same. Use S3 storage and an S3 virtual chunk container instead.

Render the hybrid store with zarr-layer

zarr-layer can render a hybrid store directly in the browser. It reprojects the data on the fly, so there's no tile server and no duplicate copy of the data. Read more in Hybrid Icechunk stores for serverless web mapping.

A hybrid store in the browser, rendered with zarr-layer. From the blog post.