Dask

Pyramid.write runs on a local thread pool. To use Dask instead, for example on a distributed cluster, call as_datatree() to get a lazy DataTree and write it with to_zarr.

A lazy source

Open air_temperature with one Dask chunk per year:

import xarray as xr
import xproj  # noqa: F401

ds = xr.tutorial.open_dataset("air_temperature", chunks={"time": 730}).drop_encoding()
ds = ds.proj.assign_crs(spatial_ref="EPSG:4326")
ds.air

Write with Dask

Plan the pyramid as usual. as_datatree() returns every level as a lazy DataTree, and pyramid.encoding holds the recommended chunks and shards. The shards are sized to divide evenly into the Dask chunks, so no rechunking is needed:

from topozarr import create_pyramid

pyramid = create_pyramid(ds, levels=2, x_dim="lon", y_dim="lat")
lazy = pyramid.as_datatree()
lazy.to_zarr("air-dask.zarr", zarr_format=3, consolidated=False, mode="w", encoding=pyramid.encoding)
lazy

To run on a cluster, start a dask.distributed.Client before calling to_zarr.