Skip to content

Remote Dataset Access

This example shows how to access remote datasets. This can be done directly using RemoteDataset.open(), or listing all available datasets via RemoteDataset.list().

from textwrap import wrap

import webknossos as wk


def main() -> None:
    # Remote datasets are read-only, but can be used similar to normal datasets:
    l4_sample_dataset = wk.RemoteDataset.open("l4_sample")

    # Print information of the public l4dense_motta_et_al_demo dataset:
    print(l4_sample_dataset.url)
    print("\n  ".join(["Description:"] + wrap(l4_sample_dataset.description or "")))
    print("Layers:", ", ".join(l4_sample_dataset.layers))
    print("Tags:", ", ".join(l4_sample_dataset.tags))

    # List all accessible remote datasets via get_remote_datasets():
    own_remote_datasets = wk.RemoteDataset.list()

    # Print the first 10 dataset names from your organization:
    print()
    print(f"First 10 datasets (of {len(own_remote_datasets)}) for own organization:")
    for dataset_name in sorted(own_remote_datasets)[:10]:
        print("*", dataset_name)

    # Set demo tag of l4_sample dataset:
    l4_sample_dataset.tags = ("demo",)

    # List all accessible demo datasets:
    remote_demo_datasets = wk.RemoteDataset.list(tags="demo")
    print("Remote demo datasets:", list(remote_demo_datasets))
    assert l4_sample_dataset in remote_demo_datasets.values()


if __name__ == "__main__":
    main()

Access Modes

The image data of a remote dataset can be reached in three ways, expressed by RemoteAccessMode:

Mode Description
ZARR_STREAMING The WEBKNOSSOS datastore re-serves the data as Zarr. Works everywhere and is the default. It is also the only mode that can read annotations.
PROXY_PATH The WEBKNOSSOS datastore proxies the bytes of the underlying storage, preserving its data format.
DIRECT_PATH The underlying storage is read directly. This is the fastest option, but it requires that your machine has access to that storage (and credentials for it).

RemoteDataset.open(access_mode=...) sets the default that all mags inherit. Individual mags can override it, so a dataset can be read directly where that works and through the datastore where it does not:

import webknossos as wk

ds = wk.RemoteDataset.open("https://webknossos.org/datasets/scalable_minds/l4_sample_dev/view")
layer = ds.get_layer("color")

mag1 = layer.get_mag(1, access_mode=wk.RemoteAccessMode.DIRECT_PATH)
mag2 = layer.get_mag(2, access_mode=wk.RemoteAccessMode.PROXY_PATH)

Each mag knows all of its available paths, regardless of how it is currently accessed, via mag.paths, a dict[RemoteAccessMode, UPath]. A mode is missing from the dict if it isn't available for that mag (e.g. DIRECT_PATH when the server doesn't expose it):

mag = layer.get_mag(1)
print(mag.paths)
# {RemoteAccessMode.ZARR_STREAMING: ..., RemoteAccessMode.PROXY_PATH: ..., RemoteAccessMode.DIRECT_PATH: ...}
print(mag.paths.get(wk.RemoteAccessMode.DIRECT_PATH))  # e.g. s3://bucket/dataset/color/1, or None if not exposed
print(mag.data_format)  # what this mag actually serves

Only the direct path is stored in the dataset properties; the other paths in mag.paths are computed from the datastore URL.

Metadata (layer bounding boxes, view configurations, mags, attachments, ...) can be written back to the server under any access mode, as long as the dataset's properties stem from the WEBKNOSSOS api — which is the case unless the dataset is viewed through an annotation, or its data source is unusable. Reading and writing metadata is independent of which access mode is used to read the image data itself.