Convert NetCDF To Zarr - Polar Hackathon

This notebook converts the ice-temperature NetCDF example into a consolidated Zarr store. Zarr is useful for multidimensional arrays because chunked object storage lets clients read selected depths, windows, or variables without downloading one monolithic file.

from pathlib import Path


def find_repo_root(start: Path = Path.cwd()) -> Path:
    """Find the repository root when the kernel starts in a subfolder."""
    start = start.resolve()
    for path in (start, *start.parents):
        if (path / ".git").exists() or (path / "downloaded_data").exists():
            return path
    return start


REPO_ROOT = find_repo_root()
DATA_DIR = REPO_ROOT / "downloaded_data"
DATA_DIR.mkdir(exist_ok=True)

import numpy as np
import rioxarray
import xarray as xr

NC_FILE = DATA_DIR / "SM_TEST_MIR_ITUDP4_20130101T000000_20141231T000000_200_001_0.nc"
ZARR_STORE = DATA_DIR / "SM_TEST_MIR_ITUDP4_20130101T000000_20141231T000000_200_001_0.zarr"
CHUNKS = {'depth': 2, 'y':5063, 'x':5673}

Open With Explicit Chunks And CRS Metadata¶

ds = xr.open_dataset(NC_FILE, chunks=CHUNKS, decode_coords="all")
source_crs = ds.attrs.get("srid")
if source_crs:
    ds = ds.rio.write_crs(source_crs)
    if {"x", "y"}.issubset(ds.dims):
        ds = ds.rio.set_spatial_dims(x_dim="x", y_dim="y")
        ds = ds.rio.write_coordinate_system()
ds

Write Consolidated Zarr¶

Consolidated metadata stores array metadata in one place, which avoids many small metadata reads from object storage.

write_kwargs = dict(mode="w", consolidated=True)
ds.to_zarr(ZARR_STORE, zarr_format=2, write_empty_chunks=False, **write_kwargs)

<xarray.backends.zarr.ZarrStore at 0x705c8a911a80>