Zarr v2 vs v3: What Actually Changed, and When It Matters

If you store large arrays — microscopy volumes, spatial omics tensors, climate grids — you have probably written Zarr without thinking much about which version of the spec you were writing. That was a safe habit for years. It is less safe now: v2 and v3 differ in ways that change how a dataset performs on object storage, and OME-Zarr made a hard break between them.

Here is what actually changed, and how to decide.

The problem v3 was built to solve

On a local disk, a chunked array is a directory of files and nobody counts them. On S3 or GCS, every chunk is an object, and object count is a first-class cost: it drives request charges, listing latency, and the time to open a dataset at all.

That creates a bind. Small chunks give good random access — you read a 2 MB tile instead of a 200 MB slab — but a terabyte-scale array with 1 MB chunks means on the order of a million objects. Large chunks keep the object count sane but force every reader to pull far more bytes than it needs.

In v2 you simply pick a side. Most of v3’s design follows from refusing to.

Metadata: three files become one

In v2, every node carries up to three dotfiles:

my_array/
  .zarray      ← shape, chunks, dtype, compressor
  .zattrs      ← user attributes
  0.0          ← chunk
  0.1

In v3, one document describes the node completely:

my_array/
  zarr.json    ← everything, including attributes
  c/0/0        ← chunk
  c/0/1

{
  "zarr_format": 3,
  "node_type": "array",
  "shape": [1024, 1024],
  "data_type": "float32",
  "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [256, 256]}},
  "chunk_key_encoding": {"name": "default"},
  "codecs": [{"name": "bytes", "configuration": {"endian": "little"}}],
  "fill_value": 0.0,
  "dimension_names": ["y", "x"],
  "attributes": {}
}

This is not just tidiness. Opening a deep hierarchy over HTTP meant up to three requests per node in v2; v3 makes it one. node_type removes the guesswork about whether a prefix is an array or a group, and dimension_names is finally part of the spec rather than an xarray convention smuggled through _ARRAY_DIMENSIONS.

Two other details in that snippet are easy to miss. Data types are named ("float32") instead of NumPy typestrings ("<f4"), so endianness is no longer welded into the dtype — it moved to the bytes codec, which makes the format readable without a NumPy-shaped mental model. And chunk keys are now c/0/0 by default rather than 0.0, which keeps chunks from colliding with metadata in a flat key space.

Codecs: a pipeline instead of a compressor

v2 has filters (a list, applied first) and compressor (exactly one):

{"filters": [{"id": "delta", "dtype": "<i4"}],
 "compressor": {"id": "blosc", "cname": "zstd", "clevel": 5}}

v3 has one ordered codecs list, where each codec declares what it transforms:

{"codecs": [
  {"name": "transpose", "configuration": {"order": [1, 0]}},
  {"name": "bytes", "configuration": {"endian": "little"}},
  {"name": "blosc", "configuration": {"cname": "zstd", "clevel": 5}}
]}

The stages are typed — array → array, then exactly one array → bytes, then bytes → bytes — so the pipeline is checkable rather than conventional. Memory layout follows: v2’s top-level order: "F" is gone, replaced by a transpose codec, which is the same idea expressed once instead of twice.

The cost is verbosity. A v2 compressor spec that fit on one line is now three objects. In exchange, a reader can validate a pipeline it has never seen, and unknown extensions fail loudly instead of being silently ignored — v3 replaces v2’s “skip what you don’t recognize” with explicit must_understand semantics.

Sharding: the change that actually matters

Everything above is better engineering. Sharding is the part that changes what is possible.

A shard is a single storage object holding many chunks, plus an index. Readers fetch individual chunks with byte-range requests; writers produce one object instead of hundreds.

chunks = (128, 128)      shards = (2048, 2048)
→ 256 chunks per shard, one object

That breaks the bind described at the top. Access granularity is now set by the chunk size; object count is set by the shard size; and you choose them independently. A 1 TB array that would have been a million objects becomes a few thousand, without making readers download more than they asked for.

One constraint decides whether this fits your workflow: in the general case, writes happen a whole shard at a time. Some codec and store combinations allow writing inner chunks individually, but you should not plan on it. Sharding suits write-once, read-many data — exactly the archive pattern for imaging and sequencing — and suits incremental patching of a live array much less well.

Everything else, briefly

v3 also formalizes the store interface — declared readable / writeable / listable capabilities, plus partial reads and writes — and adds storage transformers that sit between the codec pipeline and storage. Most users will never touch either directly, but they are why sharding could be specified as a codec rather than bolted on.

So which should you write?

Write v2 when compatibility is the binding constraint. It is not deprecated, and zarr-python 3 reads and writes it happily via zarr_format=2. If your data will be opened by a long tail of tools, older pipelines, or a language binding that has not finished v3 support, v2 remains the option that simply works everywhere. For a few hundred chunks on a local disk, v3 buys you very little.

Write v3 when the dataset lives on object storage and is large. The decisive question is whether you are about to create hundreds of thousands of objects. If you are, sharding is not a nice-to-have — it is the difference between a dataset that opens in a second and one that costs real money to list. The metadata consolidation and named dtypes are welcome; sharding is the reason to migrate.

The awkward case is data that is both large and frequently updated in place. The whole-shard write constraint bites there, and a v2 layout with moderate chunks may still be the pragmatic answer.

A note for the imaging crowd

If you work with OME-Zarr, this is not an abstract choice. OME-Zarr 0.5 is a hard break: it moves to Zarr v3, and a hierarchy may not mix versions. OME metadata now lives under a dedicated ome key in the attributes, and dimension_names in the Zarr metadata must match the axis names in the OME metadata — the two descriptions of the same dimensions are no longer allowed to drift.

RFC-2, which specified all this, was accepted by its reviewers in September 2024, and sharding was an explicit motivation for the move. If you are producing whole-slide or volumetric data destined for cloud storage, that is the version to target — with the caveat that mixed-version hierarchies are prohibited, so migration is a whole-dataset operation, not a gradual one.

Sources: the Zarr v3 specification, the Zarr project’s V2/V3 comparison, ZEP 2 (sharding codec), the zarr-python 3 release notes, and OME-NGFF RFC-2.