Skip to content

How it works

This page describes what topozarr does under the hood: how it plans a pyramid, how it writes one, and how each coarsening method combines cells.

Plan, then write

create_pyramid is lazy. It returns a Pyramid plan and writes nothing.

ds create_pyramid() Pyramid level_templates encoding attrs plan · nothing written .write(store) thread pool · reads source once .as_datatree() lazy DataTree → to_zarr (Dask)
Field Holds
level_templates One xr.Dataset per level
encoding Chunk and shard sizes for each variable at each level
attrs Root metadata: multiscales, proj, spatial (layout)

There are two ways to turn the plan into data:

  • Pyramid.write (default) uses a local thread pool and reads the source only once.
  • Pyramid.as_datatree returns a lazy xr.DataTree built with xarray.coarsen, for use with Dask. You then call to_zarr with pyramid.encoding.

Write path

write splits the source into level 0 tiles, each covering whole shards of levels 0..k. Each worker:

  1. Writes its tile to level 0.
  2. For each level 1..k, reduces the previous level's block with the Rust kernel (topozarr_core.block_reduce), then writes the result.

This way, levels 0..k never re-read from the store or share buffers between workers.

0 1 2 = k 3 4 level tile 1 tile 2 tile 3 tile 4 one worker 1 write tile to level 0 reduce, then write 2 each level 1..k: above k: reduced from the stored level below
How k is chosen

k is the deepest level whose tile still fits in max_region_bytes and leaves at least one tile per worker. Each level after k is reduced from the previous level, which is already written.

Coarsening methods

2 × 2 window 2 6 1 7 one cell at the next level 4 mean (2+6+1+7) / 4 7 max largest 1 min smallest 16 sum 2+6+1+7 2 nearest top-left cell
Method Each output cell is Use for
mean (default) Window mean. Integers round half to even Continuous data
max, min Window maximum or minimum Peaks and extents
sum Window sum Counts and totals
nearest Top-left cell of the window Categorical data (class codes, masks)
  • Missing values: NaN and _FillValue are skipped. A window with only missing values gives 0 for sum and the fill value (or NaN) for the other methods. nearest ignores both and copies the cell as is.
  • Dtypes: u8, u16, i16, i32, i64, f32, and f64. Integers stay integers (unlike xarray.coarsen, which promotes them to float).
  • Shape: matches xarray.coarsen(boundary="trim"), so partial windows at the trailing edges are dropped.
5 × 5 hatched cells are dropped 2 × 2