webknossos.dataset.dataset
¶
Classes:
-
Dataset
–A dataset is the entry point of the Dataset API.
-
RemoteDataset
–Representation of a dataset on the webknossos server, returned from
Dataset.open_remote()
.
Dataset
¶
Dataset(dataset_path: Union[str, PathLike], voxel_size: Optional[Tuple[float, float, float]] = None, name: Optional[str] = None, exist_ok: bool = _UNSET, *, voxel_size_with_unit: Optional[VoxelSize] = None, scale: Optional[Tuple[float, float, float]] = None, read_only: bool = False)
A dataset is the entry point of the Dataset API. An existing dataset on disk can be opened or new datasets can be created.
A dataset stores the data in .wkw
files on disk with metadata in datasource-properties.json
.
The information in those files are kept in sync with the object.
Each dataset consists of one or more layers (webknossos.dataset.layer.Layer), which themselves can comprise multiple magnifications (webknossos.dataset.mag_view.MagView).
When using Dataset.open_remote()
an instance of the RemoteDataset
subclass is returned.
Creates a new dataset and the associated datasource-properties.json
.
If the dataset already exists and exist_ok is set to True,
it is opened (the provided voxel_size and name are asserted to match the existing dataset).
Currently, exist_ok=True
is the deprecated default and will change in future releases.
Please use Dataset.open
if you intend to open an existing dataset and don't want/need the creation behavior.
scale
is deprecated, please use voxel_size
instead.
Classes:
-
ConversionLayerMapping
–Strategies for mapping file paths to layers, for use in
Methods:
-
add_copy_layer
–Copies the data at
foreign_layer
which belongs to another dataset to the current dataset. -
add_fs_copy_layer
–Copies the files at
foreign_layer
which belongs to another dataset -
add_layer
–Creates a new layer called
layer_name
and adds it to the dataset. -
add_layer_for_existing_files
– -
add_layer_from_images
–Creates a new layer called
layer_name
with magmag
fromimages
. -
add_layer_like
– -
add_remote_layer
–Adds a layer of another dataset to this dataset.
-
add_symlink_layer
–Creates a symlink to the data at
foreign_layer
which belongs to another dataset. -
announce_manual_upload
–Announces a manual dataset upload to webknossos. This is useful when users with access
-
calculate_bounding_box
–Calculates and returns the enclosing bounding box of all data layers of the dataset.
-
compress
–Compresses all mag views in-place that are not yet compressed.
-
copy_dataset
–Creates a new dataset at
new_dataset_path
and copies the data from the current dataset toempty_target_ds
. -
create
–Deprecated, please use the constructor
Dataset()
instead. -
delete_layer
–Deletes the layer from the
datasource-properties.json
and the data from disk. -
download
–Downloads a dataset and returns the Dataset instance.
-
downsample
–Downsamples all layers that are not yet downsampled.
-
from_images
–This method imports image data in a folder or from a file as a
-
get_color_layer
–Deprecated, please use
get_color_layers()
. -
get_color_layers
–Returns all color layers.
-
get_layer
–Returns the layer called
layer_name
of this dataset. The return type iswebknossos.dataset.layer.Layer
. -
get_or_add_layer
–Creates a new layer called
layer_name
and adds it to the dataset, in case it did not exist before. -
get_or_create
–Deprecated, please use the constructor
Dataset()
instead. -
get_remote_datasets
–Returns a dict of all remote datasets visible for selected organization, or the organization of the logged in user by default.
-
get_segmentation_layer
–Deprecated, please use
get_segmentation_layers()
. -
get_segmentation_layers
–Returns all segmentation layers.
-
open
–To open an existing dataset on disk, simply call
Dataset.open("your_path")
. -
open_remote
–Opens a remote webknossos dataset. Image data is accessed via network requests.
-
shallow_copy_dataset
–Create a new dataset at the given path. Link all mags of all existing layers.
-
trigger_reload_in_datastore
–This method is used for datasets that were uploaded manually
-
upload
–Uploads this dataset to WEBKNOSSOS.
Attributes:
-
default_view_configuration
(Optional[DatasetViewConfiguration]
) – -
layers
(Dict[str, Layer]
) –Getter for dictionary containing all layers.
-
name
(str
) – -
path
(Path
) – -
read_only
(bool
) – -
scale
(Tuple[float, float, float]
) –Deprecated, use
voxel_size
instead. -
voxel_size
(Tuple[float, float, float]
) – -
voxel_size_with_unit
(VoxelSize
) –
default_view_configuration
property
writable
¶
default_view_configuration: Optional[DatasetViewConfiguration]
ConversionLayerMapping
¶
Bases: Enum
Strategies for mapping file paths to layers, for use in
Dataset.from_images
for the map_filepath_to_layer_name
argument.
If none of the strategies fit, the mapping can also be specified by a callable.
Attributes:
-
ENFORCE_LAYER_PER_FILE
–Enforce a new layer per file. This is useful for 2D
-
ENFORCE_LAYER_PER_FOLDER
–Combine all files in a folder into one layer.
-
ENFORCE_LAYER_PER_TOPLEVEL_FOLDER
–The first folders of the input path are each converted to one layer.
-
ENFORCE_SINGLE_LAYER
–Combines all found files into a single layer. This is only
-
INSPECT_EVERY_FILE
–Like
INSPECT_SINGLE_FILE
, but the strategy -
INSPECT_SINGLE_FILE
–The first found image file is opened. If it appears to be
ENFORCE_LAYER_PER_FILE
class-attribute
instance-attribute
¶
ENFORCE_LAYER_PER_FILE = 'enforce_layer_per_file'
Enforce a new layer per file. This is useful for 2D images that should be converted to 2D layers each.
ENFORCE_LAYER_PER_FOLDER
class-attribute
instance-attribute
¶
ENFORCE_LAYER_PER_FOLDER = 'enforce_layer_per_folder'
Combine all files in a folder into one layer.
ENFORCE_LAYER_PER_TOPLEVEL_FOLDER
class-attribute
instance-attribute
¶
ENFORCE_LAYER_PER_TOPLEVEL_FOLDER = 'enforce_layer_per_toplevel_folder'
The first folders of the input path are each converted to one layer. This might be useful if multiple layers have stacks of 2D images, but parts of the stacks are in different folders.
ENFORCE_SINGLE_LAYER
class-attribute
instance-attribute
¶
ENFORCE_SINGLE_LAYER = 'enforce_single_layer'
Combines all found files into a single layer. This is only useful if all images are 2D.
INSPECT_EVERY_FILE
class-attribute
instance-attribute
¶
INSPECT_EVERY_FILE = 'inspect_every_file'
Like INSPECT_SINGLE_FILE
, but the strategy
is determined for each image file separately.
INSPECT_SINGLE_FILE
class-attribute
instance-attribute
¶
INSPECT_SINGLE_FILE = 'inspect_single_file'
The first found image file is opened. If it appears to be
a 2D image, ENFORCE_LAYER_PER_FOLDER
is used,
if it appears to be 3D, ENFORCE_LAYER_PER_FILE
is used.
This is the default mapping.
add_copy_layer
¶
add_copy_layer(foreign_layer: Union[str, Path, Layer], new_layer_name: Optional[str] = None, chunk_shape: Optional[Union[Vec3IntLike, int]] = None, chunks_per_shard: Optional[Union[Vec3IntLike, int]] = None, data_format: Optional[Union[str, DataFormat]] = None, compress: Optional[bool] = None, executor: Optional[Executor] = None) -> Layer
Copies the data at foreign_layer
which belongs to another dataset to the current dataset.
Additionally, the relevant information from the datasource-properties.json
of the other dataset are copied too.
If new_layer_name is None, the name of the foreign layer is used.
add_fs_copy_layer
¶
add_fs_copy_layer(foreign_layer: Union[str, Path, Layer], new_layer_name: Optional[str] = None) -> Layer
Copies the files at foreign_layer
which belongs to another dataset
to the current dataset via the filesystem. Additionally, the relevant
information from the datasource-properties.json
of the other dataset
are copied too. If new_layer_name is None, the name of the foreign
layer is used.
add_layer
¶
add_layer(layer_name: str, category: LayerCategoryType, dtype_per_layer: Optional[DTypeLike] = None, dtype_per_channel: Optional[DTypeLike] = None, num_channels: Optional[int] = None, data_format: Union[str, DataFormat] = DEFAULT_DATA_FORMAT, bounding_box: Optional[NDBoundingBox] = None, **kwargs: Any) -> Layer
Creates a new layer called layer_name
and adds it to the dataset.
The dtype can either be specified per layer or per channel.
If neither of them are specified, uint8
per channel is used as default.
Creates the folder layer_name
in the directory of self.path
.
WKW layers can only be added to datasets on local file systems.
The return type is webknossos.dataset.layer.Layer
.
This function raises an IndexError
if the specified layer_name
already exists.
add_layer_for_existing_files
¶
add_layer_for_existing_files(layer_name: str, category: LayerCategoryType, **kwargs: Any) -> Layer
add_layer_from_images
¶
add_layer_from_images(images: Union[str, FramesSequence, List[Union[str, PathLike]]], layer_name: str, category: Optional[LayerCategoryType] = 'color', data_format: Union[str, DataFormat] = DEFAULT_DATA_FORMAT, mag: Union[int, str, list, tuple, ndarray, Mag] = Mag(1), chunk_shape: Optional[Union[Vec3IntLike, int]] = None, chunks_per_shard: Optional[Union[int, Vec3IntLike]] = None, compress: bool = False, *, topleft: VecIntLike = zeros(), swap_xy: bool = False, flip_x: bool = False, flip_y: bool = False, flip_z: bool = False, dtype: Optional[DTypeLike] = None, use_bioformats: Optional[bool] = None, channel: Optional[int] = None, timepoint: Optional[int] = None, czi_channel: Optional[int] = None, batch_size: Optional[int] = None, allow_multiple_layers: bool = False, max_layers: int = 20, truncate_rgba_to_rgb: bool = True, executor: Optional[Executor] = None, chunk_size: Optional[Union[Vec3IntLike, int]] = None) -> Layer
Creates a new layer called layer_name
with mag mag
from images
.
images
can be one of the following:
- glob-string
- list of paths
pims.FramesSequence
instance
Please see the pims docs for more information.
This method needs extra packages like tifffile or pylibczirw. Please install the respective extras,
e.g. using python -m pip install "webknossos[all]"
.
Further Arguments:
category
:color
by default, may be set to "segmentation"data_format
: by default wkw files are written, may be set to "zarr"mag
: magnification to use for the written datachunk_shape
,chunks_per_shard
,compress
: adjust how the data is stored on disktopleft
: set an offset in Mag(1) to start writing the data, only affecting the outputswap_xy
: set toTrue
to interchange x and y axis before writing to diskflip_x
,flip_y
,flip_z
: set toTrue
to reverse the respective axis before writing to diskdtype
: the read image data will be convertoed to this dtype usingnumpy.ndarray.astype
use_bioformats
: set toTrue
to only use the pims bioformats adapter directly, needs a JVM, set toFalse
to forbid using the bioformats adapter, by default it is tried as a last optionchannel
: may be used to select a single channel, if multiple are availabletimepoint
: for timeseries, select a timepoint to use by specifying it as an int, starting from 0czi_channel
: may be used to select a channel for .czi images, which differs from normal color-channelsbatch_size
: size to process the images (influences RAM consumption), must be a multiple of the chunk-size z-axis for uncompressed and the shard-size z-axis for compressed layers, default is the chunk-size or shard-size respectivelyallow_multiple_layers
: set toTrue
if timepoints or channels may result in multiple layers being added (only the first is returned)max_layers
: only applies ifallow_multiple_layers=True
, limits the number of layers added via different channels or timepointstruncate_rgba_to_rgb
: only applies ifallow_multiple_layers=True
, set toFalse
to write four channels into layers instead of an RGB channelexecutor
: pass aClusterExecutor
instance to parallelize the conversion jobs across the batches
add_remote_layer
¶
add_remote_layer(foreign_layer: Union[str, UPath, Layer], new_layer_name: Optional[str] = None) -> Layer
Adds a layer of another dataset to this dataset.
The relevant information from the datasource-properties.json
of the other dataset is copied to this dataset.
Note: If the other dataset modifies its bounding box afterwards, the change does not affect this properties
(or vice versa).
If new_layer_name is None, the name of the foreign layer is used.
add_symlink_layer
¶
add_symlink_layer(foreign_layer: Union[str, Path, Layer], make_relative: bool = False, new_layer_name: Optional[str] = None) -> Layer
Creates a symlink to the data at foreign_layer
which belongs to another dataset.
The relevant information from the datasource-properties.json
of the other dataset is copied to this dataset.
Note: If the other dataset modifies its bounding box afterwards, the change does not affect this properties
(or vice versa).
If make_relative is True, the symlink is made relative to the current dataset path.
If new_layer_name is None, the name of the foreign layer is used.
Symlinked layers can only be added to datasets on local file systems.
announce_manual_upload
classmethod
¶
announce_manual_upload(dataset_name: str, organization: str, initial_team_ids: List[str], folder_id: str, token: Optional[str] = None) -> None
Announces a manual dataset upload to webknossos. This is useful when users with access to the file system of the datastore want to upload a dataset manually. It creates an entry in the database and sets the access rights accordingly.
calculate_bounding_box
¶
calculate_bounding_box() -> NDBoundingBox
Calculates and returns the enclosing bounding box of all data layers of the dataset.
compress
¶
compress(executor: Optional[Executor] = None) -> None
Compresses all mag views in-place that are not yet compressed.
copy_dataset
¶
copy_dataset(new_dataset_path: Union[str, Path], voxel_size: Optional[Tuple[float, float, float]] = None, chunk_shape: Optional[Union[Vec3IntLike, int]] = None, chunks_per_shard: Optional[Union[Vec3IntLike, int]] = None, data_format: Optional[Union[str, DataFormat]] = None, compress: Optional[bool] = None, args: Optional[Namespace] = None, executor: Optional[Executor] = None, *, voxel_size_with_unit: Optional[VoxelSize] = None, chunk_size: Optional[Union[Vec3IntLike, int]] = None, block_len: Optional[int] = None, file_len: Optional[int] = None) -> Dataset
Creates a new dataset at new_dataset_path
and copies the data from the current dataset to empty_target_ds
.
If not specified otherwise, the voxel_size
, chunk_shape
, chunks_per_shard
and compress
of the current dataset
are also used for the new dataset.
WKW layers can only be copied to datasets on local file systems.
create
classmethod
¶
create(dataset_path: Union[str, PathLike], voxel_size: Tuple[float, float, float], name: Optional[str] = None) -> Dataset
Deprecated, please use the constructor Dataset()
instead.
delete_layer
¶
delete_layer(layer_name: str) -> None
Deletes the layer from the datasource-properties.json
and the data from disk.
download
classmethod
¶
download(dataset_name_or_url: str, organization_id: Optional[str] = None, sharing_token: Optional[str] = None, webknossos_url: Optional[str] = None, bbox: Optional[BoundingBox] = None, layers: Union[List[str], str, None] = None, mags: Optional[List[Mag]] = None, path: Optional[Union[PathLike, str]] = None, exist_ok: bool = False) -> Dataset
Downloads a dataset and returns the Dataset instance.
dataset_name_or_url
may be a dataset name or a full URL to a dataset view, e.g.https://webknossos.org/datasets/scalable_minds/l4_sample_dev/view
If a URL is used,organization_id
,webknossos_url
andsharing_token
must not be set.organization_id
may be supplied if a dataset name was used in the previous argument, it defaults to your current organization from thewebknossos_context
. You can find yourorganization_id
here.sharing_token
may be supplied if a dataset name was used and can specify a sharing token.webknossos_url
may be supplied if a dataset name was used, and allows to specify in which webknossos instance to search for the dataset. It defaults to the url from your currentwebknossos_context
, using https://webknossos.org as a fallback.bbox
,layers
, andmags
specify which parts of the dataset to download. If nothing is specified the whole image, all layers, and all mags are downloaded respectively.path
andexist_ok
specify where to save the downloaded dataset and whether to overwrite if thepath
exists.
downsample
¶
downsample(sampling_mode: SamplingModes = ANISOTROPIC, coarsest_mag: Optional[Mag] = None, executor: Optional[Executor] = None) -> None
Downsamples all layers that are not yet downsampled.
from_images
classmethod
¶
from_images(input_path: Union[str, PathLike], output_path: Union[str, PathLike], voxel_size: Optional[Tuple[float, float, float]] = None, name: Optional[str] = None, *, map_filepath_to_layer_name: Union[ConversionLayerMapping, Callable[[Path], str]] = INSPECT_SINGLE_FILE, z_slices_sort_key: Callable[[Path], Any] = natsort_keygen(), voxel_size_with_unit: Optional[VoxelSize] = None, layer_name: Optional[str] = None, layer_category: Optional[LayerCategoryType] = None, data_format: Union[str, DataFormat] = DEFAULT_DATA_FORMAT, chunk_shape: Optional[Union[Vec3IntLike, int]] = None, chunks_per_shard: Optional[Union[int, Vec3IntLike]] = None, compress: bool = False, swap_xy: bool = False, flip_x: bool = False, flip_y: bool = False, flip_z: bool = False, use_bioformats: Optional[bool] = None, max_layers: int = 20, batch_size: Optional[int] = None, executor: Optional[Executor] = None) -> Dataset
This method imports image data in a folder or from a file as a
WEBKNOSSOS dataset. The image data can be 3D images (such as multipage
tiffs) or stacks of 2D images. In case of multiple 3D images or image
stacks, those are mapped to different layers. The exact mapping is handled
by the argument map_filepath_to_layer_name
, which can be a pre-defined
strategy from the enum ConversionLayerMapping
, or a custom callable, taking
a path of an image file and returning the corresponding layer name. All
files belonging to the same layer name are then grouped. In case of
multiple files per layer, those are usually mapped to the z-dimension.
The order of the z-slices can be customized by setting
z_slices_sort_key
.
If a layer_name is set, this name is used if a single layer is created. Otherwise the layer_name is used as a common prefix for all layers.
The category of layers (color
vs segmentation
) is determined
automatically by checking if segmentation
is part of the path.
The category decision is evaluated and corrected after data import with a
data driven approach. Alternatively, a category can be enforced by passing
layer_category
.
Further arguments behave as in add_layer_from_images
, please also
refer to its documentation.
For more fine-grained control, please create an empty dataset and use
add_layer_from_images
.
get_color_layer
¶
get_color_layer() -> Layer
Deprecated, please use get_color_layers()
.
Returns the only color layer. Fails with a RuntimeError if there are multiple color layers or none.
get_layer
¶
get_layer(layer_name: str) -> Layer
Returns the layer called layer_name
of this dataset. The return type is webknossos.dataset.layer.Layer
.
This function raises an IndexError
if the specified layer_name
does not exist.
get_or_add_layer
¶
get_or_add_layer(layer_name: str, category: LayerCategoryType, dtype_per_layer: Optional[DTypeLike] = None, dtype_per_channel: Optional[DTypeLike] = None, num_channels: Optional[int] = None, data_format: Union[str, DataFormat] = DEFAULT_DATA_FORMAT, **kwargs: Any) -> Layer
Creates a new layer called layer_name
and adds it to the dataset, in case it did not exist before.
Then, returns the layer.
For more information see add_layer
.
get_or_create
classmethod
¶
get_or_create(dataset_path: Union[str, Path], voxel_size: Tuple[float, float, float], name: Optional[str] = None) -> Dataset
Deprecated, please use the constructor Dataset()
instead.
get_remote_datasets
staticmethod
¶
get_remote_datasets(organization_id: Optional[str] = None, tags: Optional[Union[str, Sequence[str]]] = None) -> Mapping[str, RemoteDataset]
Returns a dict of all remote datasets visible for selected organization, or the organization of the logged in user by default.
The dict contains lazy-initialized RemoteDataset
values for keys indicating the dataset name.
import webknossos as wk
print(sorted(wk.Dataset.get_remote_datasets()))
ds = wk.Dataset.get_remote_datasets(
organization_id="scalable_minds"
)["l4dense_motta_et_al_demo"]
get_segmentation_layer
¶
get_segmentation_layer() -> SegmentationLayer
Deprecated, please use get_segmentation_layers()
.
Returns the only segmentation layer. Fails with a IndexError if there are multiple segmentation layers or none.
get_segmentation_layers
¶
get_segmentation_layers() -> List[SegmentationLayer]
Returns all segmentation layers.
open
classmethod
¶
open(dataset_path: Union[str, PathLike]) -> Dataset
To open an existing dataset on disk, simply call Dataset.open("your_path")
.
This requires datasource-properties.json
to exist in this folder. Based on the datasource-properties.json
,
a dataset object is constructed. Only layers and magnifications that are listed in the properties are loaded
(even though there might exist more layers or magnifications on disk).
The dataset_path
refers to the top level directory of the dataset (excluding layer or magnification names).
open_remote
classmethod
¶
open_remote(dataset_name_or_url: str, organization_id: Optional[str] = None, sharing_token: Optional[str] = None, webknossos_url: Optional[str] = None) -> RemoteDataset
Opens a remote webknossos dataset. Image data is accessed via network requests.
Dataset metadata such as allowed teams or the sharing token can be read and set
via the respective RemoteDataset
properties.
dataset_name_or_url
may be a dataset name or a full URL to a dataset view, e.g.https://webknossos.org/datasets/scalable_minds/l4_sample_dev/view
If a URL is used,organization_id
,webknossos_url
andsharing_token
must not be set.organization_id
may be supplied if a dataset name was used in the previous argument, it defaults to your current organization from thewebknossos_context
. You can find yourorganization_id
here.sharing_token
may be supplied if a dataset name was used and can specify a sharing token.webknossos_url
may be supplied if a dataset name was used, and allows to specify in which webknossos instance to search for the dataset. It defaults to the url from your currentwebknossos_context
, using https://webknossos.org as a fallback.
shallow_copy_dataset
¶
shallow_copy_dataset(new_dataset_path: Union[str, PathLike], name: Optional[str] = None, make_relative: bool = False, layers_to_ignore: Optional[Iterable[str]] = None) -> Dataset
Create a new dataset at the given path. Link all mags of all existing layers. In addition, link all other directories in all layer directories to make this method robust against additional files e.g. layer/mappings/agglomerate_view.hdf5. This method becomes useful when exposing a dataset to webknossos. Only datasets on local filesystems can be shallow copied.
trigger_reload_in_datastore
classmethod
¶
trigger_reload_in_datastore(dataset_name: str, organization: str, token: Optional[str] = None) -> None
This method is used for datasets that were uploaded manually to a webknossos datastore. It can not be used for local datasets. After a manual upload of a dataset, the datasets properties are updated automatically after a few minutes. To trigger a reload of the datasets properties manually, use this method.
upload
¶
upload(new_dataset_name: Optional[str] = None, layers_to_link: Optional[List[Union[LayerToLink, Layer]]] = None, jobs: Optional[int] = None) -> RemoteDataset
Uploads this dataset to WEBKNOSSOS.
The new_dataset_name
parameter allows to assign a specific name for the dataset.
layers_to_link
allows to add (or override) a layer in the uploaded dataset, so that
it links to a layer of an existing dataset in WEBKNOSSOS. That way, already existing
layers don't need to be uploaded again.
If supplied, the jobs
parameter will determine the number of simultaneous chunk uploads. Defaults to 5.
Returns the RemoteDataset
upon successful upload.
RemoteDataset
¶
RemoteDataset(dataset_path: UPath, dataset_name: str, organization_id: str, sharing_token: Optional[str], context: ContextManager)
Bases: Dataset
Representation of a dataset on the webknossos server, returned from Dataset.open_remote()
.
Read-only image data is streamed from the webknossos server using the same interface as Dataset
.
Additionally, metadata can be set via the additional properties below.
Do not call manually, please use Dataset.open_remote()
instead.
Classes:
-
ConversionLayerMapping
–Strategies for mapping file paths to layers, for use in
Methods:
-
add_copy_layer
–Copies the data at
foreign_layer
which belongs to another dataset to the current dataset. -
add_fs_copy_layer
–Copies the files at
foreign_layer
which belongs to another dataset -
add_layer
–Creates a new layer called
layer_name
and adds it to the dataset. -
add_layer_for_existing_files
– -
add_layer_from_images
–Creates a new layer called
layer_name
with magmag
fromimages
. -
add_layer_like
– -
add_remote_layer
–Adds a layer of another dataset to this dataset.
-
add_symlink_layer
–Creates a symlink to the data at
foreign_layer
which belongs to another dataset. -
announce_manual_upload
–Announces a manual dataset upload to webknossos. This is useful when users with access
-
calculate_bounding_box
–Calculates and returns the enclosing bounding box of all data layers of the dataset.
-
compress
–Compresses all mag views in-place that are not yet compressed.
-
copy_dataset
–Creates a new dataset at
new_dataset_path
and copies the data from the current dataset toempty_target_ds
. -
create
–Deprecated, please use the constructor
Dataset()
instead. -
delete_layer
–Deletes the layer from the
datasource-properties.json
and the data from disk. -
download
–Downloads a dataset and returns the Dataset instance.
-
downsample
–Downsamples all layers that are not yet downsampled.
-
explore_and_add_remote
– -
from_images
–This method imports image data in a folder or from a file as a
-
get_color_layer
–Deprecated, please use
get_color_layers()
. -
get_color_layers
–Returns all color layers.
-
get_layer
–Returns the layer called
layer_name
of this dataset. The return type iswebknossos.dataset.layer.Layer
. -
get_or_add_layer
–Creates a new layer called
layer_name
and adds it to the dataset, in case it did not exist before. -
get_or_create
–Deprecated, please use the constructor
Dataset()
instead. -
get_remote_datasets
–Returns a dict of all remote datasets visible for selected organization, or the organization of the logged in user by default.
-
get_segmentation_layer
–Deprecated, please use
get_segmentation_layers()
. -
get_segmentation_layers
–Returns all segmentation layers.
-
open
–Do not call manually, please use
Dataset.open_remote()
instead. -
open_remote
–Opens a remote webknossos dataset. Image data is accessed via network requests.
-
shallow_copy_dataset
–Create a new dataset at the given path. Link all mags of all existing layers.
-
trigger_reload_in_datastore
–This method is used for datasets that were uploaded manually
-
upload
–Uploads this dataset to WEBKNOSSOS.
Attributes:
-
allowed_teams
(Tuple[Team, ...]
) – -
default_view_configuration
(Optional[DatasetViewConfiguration]
) – -
description
(Optional[str]
) – -
display_name
(Optional[str]
) – -
folder
(RemoteFolder
) – -
is_public
(bool
) – -
layers
(Dict[str, Layer]
) –Getter for dictionary containing all layers.
-
metadata
(DatasetMetadata
) – -
name
(str
) – -
path
– -
read_only
(bool
) – -
scale
(Tuple[float, float, float]
) –Deprecated, use
voxel_size
instead. -
sharing_token
(str
) – -
tags
(Tuple[str, ...]
) – -
url
(str
) – -
voxel_size
(Tuple[float, float, float]
) – -
voxel_size_with_unit
(VoxelSize
) –
default_view_configuration
property
writable
¶
default_view_configuration: Optional[DatasetViewConfiguration]
ConversionLayerMapping
¶
Bases: Enum
Strategies for mapping file paths to layers, for use in
Dataset.from_images
for the map_filepath_to_layer_name
argument.
If none of the strategies fit, the mapping can also be specified by a callable.
Attributes:
-
ENFORCE_LAYER_PER_FILE
–Enforce a new layer per file. This is useful for 2D
-
ENFORCE_LAYER_PER_FOLDER
–Combine all files in a folder into one layer.
-
ENFORCE_LAYER_PER_TOPLEVEL_FOLDER
–The first folders of the input path are each converted to one layer.
-
ENFORCE_SINGLE_LAYER
–Combines all found files into a single layer. This is only
-
INSPECT_EVERY_FILE
–Like
INSPECT_SINGLE_FILE
, but the strategy -
INSPECT_SINGLE_FILE
–The first found image file is opened. If it appears to be
ENFORCE_LAYER_PER_FILE
class-attribute
instance-attribute
¶
ENFORCE_LAYER_PER_FILE = 'enforce_layer_per_file'
Enforce a new layer per file. This is useful for 2D images that should be converted to 2D layers each.
ENFORCE_LAYER_PER_FOLDER
class-attribute
instance-attribute
¶
ENFORCE_LAYER_PER_FOLDER = 'enforce_layer_per_folder'
Combine all files in a folder into one layer.
ENFORCE_LAYER_PER_TOPLEVEL_FOLDER
class-attribute
instance-attribute
¶
ENFORCE_LAYER_PER_TOPLEVEL_FOLDER = 'enforce_layer_per_toplevel_folder'
The first folders of the input path are each converted to one layer. This might be useful if multiple layers have stacks of 2D images, but parts of the stacks are in different folders.
ENFORCE_SINGLE_LAYER
class-attribute
instance-attribute
¶
ENFORCE_SINGLE_LAYER = 'enforce_single_layer'
Combines all found files into a single layer. This is only useful if all images are 2D.
INSPECT_EVERY_FILE
class-attribute
instance-attribute
¶
INSPECT_EVERY_FILE = 'inspect_every_file'
Like INSPECT_SINGLE_FILE
, but the strategy
is determined for each image file separately.
INSPECT_SINGLE_FILE
class-attribute
instance-attribute
¶
INSPECT_SINGLE_FILE = 'inspect_single_file'
The first found image file is opened. If it appears to be
a 2D image, ENFORCE_LAYER_PER_FOLDER
is used,
if it appears to be 3D, ENFORCE_LAYER_PER_FILE
is used.
This is the default mapping.
add_copy_layer
¶
add_copy_layer(foreign_layer: Union[str, Path, Layer], new_layer_name: Optional[str] = None, chunk_shape: Optional[Union[Vec3IntLike, int]] = None, chunks_per_shard: Optional[Union[Vec3IntLike, int]] = None, data_format: Optional[Union[str, DataFormat]] = None, compress: Optional[bool] = None, executor: Optional[Executor] = None) -> Layer
Copies the data at foreign_layer
which belongs to another dataset to the current dataset.
Additionally, the relevant information from the datasource-properties.json
of the other dataset are copied too.
If new_layer_name is None, the name of the foreign layer is used.
add_fs_copy_layer
¶
add_fs_copy_layer(foreign_layer: Union[str, Path, Layer], new_layer_name: Optional[str] = None) -> Layer
Copies the files at foreign_layer
which belongs to another dataset
to the current dataset via the filesystem. Additionally, the relevant
information from the datasource-properties.json
of the other dataset
are copied too. If new_layer_name is None, the name of the foreign
layer is used.
add_layer
¶
add_layer(layer_name: str, category: LayerCategoryType, dtype_per_layer: Optional[DTypeLike] = None, dtype_per_channel: Optional[DTypeLike] = None, num_channels: Optional[int] = None, data_format: Union[str, DataFormat] = DEFAULT_DATA_FORMAT, bounding_box: Optional[NDBoundingBox] = None, **kwargs: Any) -> Layer
Creates a new layer called layer_name
and adds it to the dataset.
The dtype can either be specified per layer or per channel.
If neither of them are specified, uint8
per channel is used as default.
Creates the folder layer_name
in the directory of self.path
.
WKW layers can only be added to datasets on local file systems.
The return type is webknossos.dataset.layer.Layer
.
This function raises an IndexError
if the specified layer_name
already exists.
add_layer_for_existing_files
¶
add_layer_for_existing_files(layer_name: str, category: LayerCategoryType, **kwargs: Any) -> Layer
add_layer_from_images
¶
add_layer_from_images(images: Union[str, FramesSequence, List[Union[str, PathLike]]], layer_name: str, category: Optional[LayerCategoryType] = 'color', data_format: Union[str, DataFormat] = DEFAULT_DATA_FORMAT, mag: Union[int, str, list, tuple, ndarray, Mag] = Mag(1), chunk_shape: Optional[Union[Vec3IntLike, int]] = None, chunks_per_shard: Optional[Union[int, Vec3IntLike]] = None, compress: bool = False, *, topleft: VecIntLike = zeros(), swap_xy: bool = False, flip_x: bool = False, flip_y: bool = False, flip_z: bool = False, dtype: Optional[DTypeLike] = None, use_bioformats: Optional[bool] = None, channel: Optional[int] = None, timepoint: Optional[int] = None, czi_channel: Optional[int] = None, batch_size: Optional[int] = None, allow_multiple_layers: bool = False, max_layers: int = 20, truncate_rgba_to_rgb: bool = True, executor: Optional[Executor] = None, chunk_size: Optional[Union[Vec3IntLike, int]] = None) -> Layer
Creates a new layer called layer_name
with mag mag
from images
.
images
can be one of the following:
- glob-string
- list of paths
pims.FramesSequence
instance
Please see the pims docs for more information.
This method needs extra packages like tifffile or pylibczirw. Please install the respective extras,
e.g. using python -m pip install "webknossos[all]"
.
Further Arguments:
category
:color
by default, may be set to "segmentation"data_format
: by default wkw files are written, may be set to "zarr"mag
: magnification to use for the written datachunk_shape
,chunks_per_shard
,compress
: adjust how the data is stored on disktopleft
: set an offset in Mag(1) to start writing the data, only affecting the outputswap_xy
: set toTrue
to interchange x and y axis before writing to diskflip_x
,flip_y
,flip_z
: set toTrue
to reverse the respective axis before writing to diskdtype
: the read image data will be convertoed to this dtype usingnumpy.ndarray.astype
use_bioformats
: set toTrue
to only use the pims bioformats adapter directly, needs a JVM, set toFalse
to forbid using the bioformats adapter, by default it is tried as a last optionchannel
: may be used to select a single channel, if multiple are availabletimepoint
: for timeseries, select a timepoint to use by specifying it as an int, starting from 0czi_channel
: may be used to select a channel for .czi images, which differs from normal color-channelsbatch_size
: size to process the images (influences RAM consumption), must be a multiple of the chunk-size z-axis for uncompressed and the shard-size z-axis for compressed layers, default is the chunk-size or shard-size respectivelyallow_multiple_layers
: set toTrue
if timepoints or channels may result in multiple layers being added (only the first is returned)max_layers
: only applies ifallow_multiple_layers=True
, limits the number of layers added via different channels or timepointstruncate_rgba_to_rgb
: only applies ifallow_multiple_layers=True
, set toFalse
to write four channels into layers instead of an RGB channelexecutor
: pass aClusterExecutor
instance to parallelize the conversion jobs across the batches
add_remote_layer
¶
add_remote_layer(foreign_layer: Union[str, UPath, Layer], new_layer_name: Optional[str] = None) -> Layer
Adds a layer of another dataset to this dataset.
The relevant information from the datasource-properties.json
of the other dataset is copied to this dataset.
Note: If the other dataset modifies its bounding box afterwards, the change does not affect this properties
(or vice versa).
If new_layer_name is None, the name of the foreign layer is used.
add_symlink_layer
¶
add_symlink_layer(foreign_layer: Union[str, Path, Layer], make_relative: bool = False, new_layer_name: Optional[str] = None) -> Layer
Creates a symlink to the data at foreign_layer
which belongs to another dataset.
The relevant information from the datasource-properties.json
of the other dataset is copied to this dataset.
Note: If the other dataset modifies its bounding box afterwards, the change does not affect this properties
(or vice versa).
If make_relative is True, the symlink is made relative to the current dataset path.
If new_layer_name is None, the name of the foreign layer is used.
Symlinked layers can only be added to datasets on local file systems.
announce_manual_upload
classmethod
¶
announce_manual_upload(dataset_name: str, organization: str, initial_team_ids: List[str], folder_id: str, token: Optional[str] = None) -> None
Announces a manual dataset upload to webknossos. This is useful when users with access to the file system of the datastore want to upload a dataset manually. It creates an entry in the database and sets the access rights accordingly.
calculate_bounding_box
¶
calculate_bounding_box() -> NDBoundingBox
Calculates and returns the enclosing bounding box of all data layers of the dataset.
compress
¶
compress(executor: Optional[Executor] = None) -> None
Compresses all mag views in-place that are not yet compressed.
copy_dataset
¶
copy_dataset(new_dataset_path: Union[str, Path], voxel_size: Optional[Tuple[float, float, float]] = None, chunk_shape: Optional[Union[Vec3IntLike, int]] = None, chunks_per_shard: Optional[Union[Vec3IntLike, int]] = None, data_format: Optional[Union[str, DataFormat]] = None, compress: Optional[bool] = None, args: Optional[Namespace] = None, executor: Optional[Executor] = None, *, voxel_size_with_unit: Optional[VoxelSize] = None, chunk_size: Optional[Union[Vec3IntLike, int]] = None, block_len: Optional[int] = None, file_len: Optional[int] = None) -> Dataset
Creates a new dataset at new_dataset_path
and copies the data from the current dataset to empty_target_ds
.
If not specified otherwise, the voxel_size
, chunk_shape
, chunks_per_shard
and compress
of the current dataset
are also used for the new dataset.
WKW layers can only be copied to datasets on local file systems.
create
classmethod
¶
create(dataset_path: Union[str, PathLike], voxel_size: Tuple[float, float, float], name: Optional[str] = None) -> Dataset
Deprecated, please use the constructor Dataset()
instead.
delete_layer
¶
delete_layer(layer_name: str) -> None
Deletes the layer from the datasource-properties.json
and the data from disk.
download
classmethod
¶
download(dataset_name_or_url: str, organization_id: Optional[str] = None, sharing_token: Optional[str] = None, webknossos_url: Optional[str] = None, bbox: Optional[BoundingBox] = None, layers: Union[List[str], str, None] = None, mags: Optional[List[Mag]] = None, path: Optional[Union[PathLike, str]] = None, exist_ok: bool = False) -> Dataset
Downloads a dataset and returns the Dataset instance.
dataset_name_or_url
may be a dataset name or a full URL to a dataset view, e.g.https://webknossos.org/datasets/scalable_minds/l4_sample_dev/view
If a URL is used,organization_id
,webknossos_url
andsharing_token
must not be set.organization_id
may be supplied if a dataset name was used in the previous argument, it defaults to your current organization from thewebknossos_context
. You can find yourorganization_id
here.sharing_token
may be supplied if a dataset name was used and can specify a sharing token.webknossos_url
may be supplied if a dataset name was used, and allows to specify in which webknossos instance to search for the dataset. It defaults to the url from your currentwebknossos_context
, using https://webknossos.org as a fallback.bbox
,layers
, andmags
specify which parts of the dataset to download. If nothing is specified the whole image, all layers, and all mags are downloaded respectively.path
andexist_ok
specify where to save the downloaded dataset and whether to overwrite if thepath
exists.
downsample
¶
downsample(sampling_mode: SamplingModes = ANISOTROPIC, coarsest_mag: Optional[Mag] = None, executor: Optional[Executor] = None) -> None
Downsamples all layers that are not yet downsampled.
explore_and_add_remote
classmethod
¶
explore_and_add_remote(dataset_uri: Union[str, PathLike], dataset_name: str, folder_path: str) -> RemoteDataset
from_images
classmethod
¶
from_images(input_path: Union[str, PathLike], output_path: Union[str, PathLike], voxel_size: Optional[Tuple[float, float, float]] = None, name: Optional[str] = None, *, map_filepath_to_layer_name: Union[ConversionLayerMapping, Callable[[Path], str]] = INSPECT_SINGLE_FILE, z_slices_sort_key: Callable[[Path], Any] = natsort_keygen(), voxel_size_with_unit: Optional[VoxelSize] = None, layer_name: Optional[str] = None, layer_category: Optional[LayerCategoryType] = None, data_format: Union[str, DataFormat] = DEFAULT_DATA_FORMAT, chunk_shape: Optional[Union[Vec3IntLike, int]] = None, chunks_per_shard: Optional[Union[int, Vec3IntLike]] = None, compress: bool = False, swap_xy: bool = False, flip_x: bool = False, flip_y: bool = False, flip_z: bool = False, use_bioformats: Optional[bool] = None, max_layers: int = 20, batch_size: Optional[int] = None, executor: Optional[Executor] = None) -> Dataset
This method imports image data in a folder or from a file as a
WEBKNOSSOS dataset. The image data can be 3D images (such as multipage
tiffs) or stacks of 2D images. In case of multiple 3D images or image
stacks, those are mapped to different layers. The exact mapping is handled
by the argument map_filepath_to_layer_name
, which can be a pre-defined
strategy from the enum ConversionLayerMapping
, or a custom callable, taking
a path of an image file and returning the corresponding layer name. All
files belonging to the same layer name are then grouped. In case of
multiple files per layer, those are usually mapped to the z-dimension.
The order of the z-slices can be customized by setting
z_slices_sort_key
.
If a layer_name is set, this name is used if a single layer is created. Otherwise the layer_name is used as a common prefix for all layers.
The category of layers (color
vs segmentation
) is determined
automatically by checking if segmentation
is part of the path.
The category decision is evaluated and corrected after data import with a
data driven approach. Alternatively, a category can be enforced by passing
layer_category
.
Further arguments behave as in add_layer_from_images
, please also
refer to its documentation.
For more fine-grained control, please create an empty dataset and use
add_layer_from_images
.
get_color_layer
¶
get_color_layer() -> Layer
Deprecated, please use get_color_layers()
.
Returns the only color layer. Fails with a RuntimeError if there are multiple color layers or none.
get_layer
¶
get_layer(layer_name: str) -> Layer
Returns the layer called layer_name
of this dataset. The return type is webknossos.dataset.layer.Layer
.
This function raises an IndexError
if the specified layer_name
does not exist.
get_or_add_layer
¶
get_or_add_layer(layer_name: str, category: LayerCategoryType, dtype_per_layer: Optional[DTypeLike] = None, dtype_per_channel: Optional[DTypeLike] = None, num_channels: Optional[int] = None, data_format: Union[str, DataFormat] = DEFAULT_DATA_FORMAT, **kwargs: Any) -> Layer
Creates a new layer called layer_name
and adds it to the dataset, in case it did not exist before.
Then, returns the layer.
For more information see add_layer
.
get_or_create
classmethod
¶
get_or_create(dataset_path: Union[str, Path], voxel_size: Tuple[float, float, float], name: Optional[str] = None) -> Dataset
Deprecated, please use the constructor Dataset()
instead.
get_remote_datasets
staticmethod
¶
get_remote_datasets(organization_id: Optional[str] = None, tags: Optional[Union[str, Sequence[str]]] = None) -> Mapping[str, RemoteDataset]
Returns a dict of all remote datasets visible for selected organization, or the organization of the logged in user by default.
The dict contains lazy-initialized RemoteDataset
values for keys indicating the dataset name.
import webknossos as wk
print(sorted(wk.Dataset.get_remote_datasets()))
ds = wk.Dataset.get_remote_datasets(
organization_id="scalable_minds"
)["l4dense_motta_et_al_demo"]
get_segmentation_layer
¶
get_segmentation_layer() -> SegmentationLayer
Deprecated, please use get_segmentation_layers()
.
Returns the only segmentation layer. Fails with a IndexError if there are multiple segmentation layers or none.
get_segmentation_layers
¶
get_segmentation_layers() -> List[SegmentationLayer]
Returns all segmentation layers.
open
classmethod
¶
open(dataset_path: Union[str, PathLike]) -> Dataset
Do not call manually, please use Dataset.open_remote()
instead.
open_remote
classmethod
¶
open_remote(dataset_name_or_url: str, organization_id: Optional[str] = None, sharing_token: Optional[str] = None, webknossos_url: Optional[str] = None) -> RemoteDataset
Opens a remote webknossos dataset. Image data is accessed via network requests.
Dataset metadata such as allowed teams or the sharing token can be read and set
via the respective RemoteDataset
properties.
dataset_name_or_url
may be a dataset name or a full URL to a dataset view, e.g.https://webknossos.org/datasets/scalable_minds/l4_sample_dev/view
If a URL is used,organization_id
,webknossos_url
andsharing_token
must not be set.organization_id
may be supplied if a dataset name was used in the previous argument, it defaults to your current organization from thewebknossos_context
. You can find yourorganization_id
here.sharing_token
may be supplied if a dataset name was used and can specify a sharing token.webknossos_url
may be supplied if a dataset name was used, and allows to specify in which webknossos instance to search for the dataset. It defaults to the url from your currentwebknossos_context
, using https://webknossos.org as a fallback.
shallow_copy_dataset
¶
shallow_copy_dataset(new_dataset_path: Union[str, PathLike], name: Optional[str] = None, make_relative: bool = False, layers_to_ignore: Optional[Iterable[str]] = None) -> Dataset
Create a new dataset at the given path. Link all mags of all existing layers. In addition, link all other directories in all layer directories to make this method robust against additional files e.g. layer/mappings/agglomerate_view.hdf5. This method becomes useful when exposing a dataset to webknossos. Only datasets on local filesystems can be shallow copied.
trigger_reload_in_datastore
classmethod
¶
trigger_reload_in_datastore(dataset_name: str, organization: str, token: Optional[str] = None) -> None
This method is used for datasets that were uploaded manually to a webknossos datastore. It can not be used for local datasets. After a manual upload of a dataset, the datasets properties are updated automatically after a few minutes. To trigger a reload of the datasets properties manually, use this method.
upload
¶
upload(new_dataset_name: Optional[str] = None, layers_to_link: Optional[List[Union[LayerToLink, Layer]]] = None, jobs: Optional[int] = None) -> RemoteDataset
Uploads this dataset to WEBKNOSSOS.
The new_dataset_name
parameter allows to assign a specific name for the dataset.
layers_to_link
allows to add (or override) a layer in the uploaded dataset, so that
it links to a layer of an existing dataset in WEBKNOSSOS. That way, already existing
layers don't need to be uploaded again.
If supplied, the jobs
parameter will determine the number of simultaneous chunk uploads. Defaults to 5.
Returns the RemoteDataset
upon successful upload.
- Get Help
- Community Forums
- Email Support