syconn.backend package
syconn.backend.base module
- class syconn.backend.base.BTBase(identifier, cache_decomp=False, read_only=True, disable_locking=False)[source]
Bases:
syconn.backend.base.StorageBaseBTBase is a subclass of StorageBase that provides an interface for data storage and retrieval in a BTree-like structure. It is designed to work with compressed data and provides methods for data caching and decompression.
- identifier
Identifier for the BTBase instance.
- Type
str
- cache_decomp
Flag indicating whether to cache decompressed arrays.
- Type
bool
- read_only
Flag indicating whether the BTBase instance is read-only.
- Type
bool
- disable_locking
Flag indicating whether to disable locking.
- Type
bool
- class syconn.backend.base.FSBase(inp_p, cache_decomp=False, read_only=True, max_delay=100, timeout=1000, disable_locking=True, max_nb_attempts=100)[source]
Bases:
syconn.backend.base.StorageBaseThis class is a customized dictionary that stores compressed numpy arrays. The compression process happens in the background, providing an intuitive user interface. The ‘cache_decomp’ kwarg can be enabled to cache decompressed arrays, saving decompression time when accessing items frequently.
- pull(source=None)[source]
Fetches data from the specified source.
- Parameters
source (str, optional) – The source location. Defaults to None.
- push(dest=None)[source]
Pushes data to the specified destination.
- Parameters
dest (str, optional) – The storage destination. Defaults to None.
syconn.backend.storage module
- class syconn.backend.storage.AttributeDict(inp_p, **kwargs)[source]
Bases:
syconn.backend.base.FSBaseThis class is a general-purpose dictionary class that inherits from the StorageClass. It is used to store attributes of segmentation objects in the SyConn toolkit. The attributes are stored in a dictionary format which allows for easy access and manipulation of the data.
- class syconn.backend.storage.BinarySearchStore(fname, id_array=None, attr_arrays=None, overwrite=False, n_shards=None, rdcc_nbytes=5242880)[source]
Bases:
objectA data structure to store properties (values) of a corresponding ID array (keys). This class uses a binary search internally, which uses a sorted representation of keys and values to enable sparse look-ups with a much lower memory complexity than python dictionaries. The maximum ID is the last element of the id_array attribute.
- Parameters
fname (str) – The file name.
id_array (Optional[np.ndarray]) – An unsorted ID array.
attr_arrays (Optional[Dict[str, np.ndarray]]) – Unsorted attribute arrays, must have the same ordering as the ID array.
overwrite (bool) – If True, overwrite existing array files. Defaults to False.
n_shards (Optional[int]) – The number of shards/chunks the ID and attribute arrays are split into. Defaults to None.
rdcc_nbytes (int) – The size of h5 chunks in bytes. Default is 5 MiB.
- get_attributes(obj_ids, attr_key)[source]
Queries attributes of the given object IDs. Note that this will not raise an Exception if an ID does not exist in the store, as the lookup uses binary search.
- Parameters
obj_ids (np.ndarray) – The object IDs to query.
attr_key (str) – The value type obtained from the store.
- Returns
The value array.
- Return type
np.ndarray
- property id_array: numpy.ndarray
Returns the flat ID array.
- Returns
The flat ID array.
- Return type
np.ndarray
- property n_shards: int
Returns the number of shards/chunks the ID and attribute arrays are split into.
- Returns
The number of shards.
- Return type
int
- class syconn.backend.storage.CompressedStorage(inp, **kwargs)[source]
Bases:
syconn.backend.base.FSBaseThis class is a customized dictionary used to store compressed numpy arrays. The compression and decompression processes happen in the background, providing an intuitive user interface. It also has an optional ‘cache_decomp’ kwarg to cache decompressed arrays for faster access when items are frequently accessed.
- class syconn.backend.storage.MeshStorage(inp, load_colarr=False, compress=False, **kwargs)[source]
Bases:
syconn.backend.base.FSBaseThis class is a customized dictionary designed to store compressed numpy arrays. It provides an intuitive user interface where compression happens in the background. It also has an option, ‘cache_decomp’, to cache decompressed arrays for faster access and saving decompressing time. This class is primarily used for storing mesh data.
- class syconn.backend.storage.SkeletonStorage(inp, **kwargs)[source]
Bases:
syconn.backend.base.FSBaseThis class is designed to store skeleton dictionaries as compressed numpy arrays. The keys of the dictionaries are “nodes”, “diameters”, and “edges”. This class is primarily used for storing skeleton data.
- syconn.backend.storage.VoxelStorage(inp, **kwargs)[source]
This function is an alias for the VoxelStorageDyn class. It initializes a VoxelStorageDyn object and returns it.
- Parameters
inp – The input path where the dictionary is stored.
**kwargs – Additional keyword arguments.
- Returns
A VoxelStorageDyn object.
- class syconn.backend.storage.VoxelStorageClass(inp, **kwargs)[source]
Bases:
syconn.backend.storage.VoxelStorageLThis class is a customized dictionary used to store compressed numpy arrays representing voxel masks. The compression and decompression processes happen in the background, providing an intuitive user interface. It also has an optional feature, ‘cache_decomp’, to cache decompressed arrays for faster access. This class does not provide any locking mechanism.
- class syconn.backend.storage.VoxelStorageDyn(inp, voxel_mode=True, voxeldata_path=None, **kwargs)[source]
Bases:
syconn.backend.storage.CompressedStorageA class that extends the CompressedStorage class to provide a storage mechanism for voxel data. This class does not store the voxels explicitly, but the information necessary to query the voxels of an object. It operates in two modes depending on the voxel_mode flag. If voxel_mode is True, the getter method operates on the underlying data set to retrieve voxels of an object and the __setitem__ method throws a RuntimeError. The __getitem__ method will return a list of 3D binary cubes with ones at the object’s locations (key: object ID). Note: The item ID has to match the object ID in the segmentation. If voxel_mode is False, __getitem__ and __setitem__ allow manipulation of the object’s bounding boxes. In this case, voxeldata_path has to be given or already be existent in the loaded dictionary. The __setitem__ method requires the object ID as key and a 3D array with all bounding boxes defining the object (N, 2, 3). Those bounding boxes are then used to query the object voxels. The bounding box is expected to be two 3D coordinates which define the lower and the upper limits.
- get_boundingdata(item)[source]
Gets the bounding data for a given item from the storage. The old voxel_mode is stored and voxel_mode is set to False. The bounding data for the item is returned and voxel_mode is set back to the old voxel_mode.
- Parameters
item (int) – The item to get the bounding data for. Object ID.
- Return type
List[ndarray]- Returns
The bounding data for the item. This is a list of bounding boxes (in voxels; xyz).
- get_voxel_cache(key)[source]
Gets the voxel cache for a given key from the storage. The voxel cache must have been added to the store via the set_voxel_cache method. This implementation operates independently of the get_voxeldata method. The voxel cache for the key is returned from the _cache_dc attribute.
- Parameters
key (int) – Segment ID to get the voxel cache for.
- Returns
The voxel cache for the key.
- get_voxel_data_cubed(item)[source]
Gets the voxel data for a given item from the storage as a dense 3D array. The voxel data and the minimum offset are obtained for the item. The size is calculated as the maximum extent minus the minimum offset. A voxel array of zeros with the size as the shape is created. For each binary array and offset, the corresponding slice of the voxel array is set to the binary array. The voxel array and the minimum offset are returned.
- Parameters
item (int) – The item to get the voxel data for. This is the Object ID.
- Return type
Tuple[ndarray,ndarray]- Returns
The voxel data for the item as a dense 3D array and the minimum offset, which is the cube offset in voxels (xyz).
- get_voxeldata(item)[source]
Gets the voxel data for a given item from the storage. The voxel data is a list of 3D binary masks with the respective offsets (in voxels). The old voxel_mode is stored and voxel_mode is set to True. If voxeldata_path is None, a ValueError is raised. voxeldata_path is used to create a KnossosDataset which is set as the voxeldata attribute. The voxel data for the item is returned and voxel_mode is set back to the old voxel_mode.
- Parameters
item (int) – The item to get the voxel data for. This is the object ID.
- Return type
Tuple[List[ndarray],List[ndarray]]- Returns
The voxel data for the item. This is a list of 3D binary masks and offsets (in voxels; xyz).
- get_voxelmask_offset(item, overlap=0)[source]
Gets the voxel mask offset for a given item from the storage. If voxel_mode is True, the voxel mask offset is calculated for each bounding box of the item. Otherwise, the superclass’s __getitem__ method is called with the item as the argument.
- Parameters
item (int) – The item to get the voxel mask offset for.
overlap (int, optional) – The overlap for the voxel mask. Defaults to 0.
- Returns
The voxel mask offset for the item.
- increase_object_size(item, value)[source]
Increases the size of an object in the storage by a given value. If voxel_mode is True, a warning is logged. The size of the object in the internal dictionary is increased by the value.
- Parameters
item – The object to increase the size for.
value – The value to increase the size by.
- iter_voxelmask_offset(item, overlap=0)[source]
Returns an iterator over the voxel mask offsets for a given item from the storage. For each bounding box of the item, the voxel mask offset is calculated and yielded.
- Parameters
item (int) – The item to iterate over the voxel mask offsets for.
overlap (int, optional) – The overlap for the voxel mask. Defaults to 0.
- Return type
Iterator[Tuple[ndarray,ndarray]]- Returns
An iterator over the voxel mask offsets for the item.
- keys()[source]
Gets the keys of the objects in the storage. The keys are the keys in the internal dictionary that are either strings that can be converted to integers or are not strings.
- Returns
The keys of the objects in the storage.
- object_repcoord(item)[source]
Gets the representative coordinate of an object from the storage. If voxel_mode is False, a warning is logged. If the item is not in the internal dictionary, a KeyError is raised. Otherwise, the representative coordinate of the object is returned.
- Parameters
item – The object to get the representative coordinate for.
- Returns
The representative coordinate of the object.
- object_size(item)[source]
Gets the size of an object from the storage. If voxel_mode is False, a warning is logged. If the item is not in the internal dictionary, a KeyError is raised. Otherwise, the size of the object is returned.
- Parameters
item – The object to get the size for.
- Returns
The size of the object.
- push()[source]
Pushes the changes to the storage. If the _cache_dc attribute has more than 0 items, its push method is called. The superclass’s push method is then called.
- set_object_repcoord(item, value)[source]
Sets the representative coordinate of an object in the storage to a given value. If voxel_mode is True, a warning is logged. The representative coordinate of the object in the internal dictionary is set to the value.
- Parameters
item – The object to set the representative coordinate for.
value – The value to set the representative coordinate to.
- set_voxel_cache(key, voxel_coords)[source]
Sets the voxel cache for a given key in the storage to a given array of voxel coordinates. This method operates independently of the __setitem__ method. The voxel coordinates are set for the key in the _cache_dc attribute.
- Parameters
key (int) – The key to set the voxel cache for.
voxel_coords (np.ndarray) – The array of voxel coordinates to set the voxel cache to.
- class syconn.backend.storage.VoxelStorageL(inp, **kwargs)[source]
Bases:
syconn.backend.base.FSBaseThis class is a customized dictionary used to store compressed numpy arrays representing voxel masks. The compression and decompression processes happen in the background, providing an intuitive user interface. It also has an optional feature, ‘cache_decomp’, to cache decompressed arrays for faster access, saving decompressing time.
- append(key, voxel_mask, offset)[source]
Appends a voxel mask and its corresponding offset to the given key in the dictionary. The voxel mask is compressed and stored in the dictionary.
- Parameters
key (
int) – The key of the item to append to in the dictionary.voxel_mask (
ndarray) – The voxel mask to append.offset (
ndarray) – The corresponding offset to append.
- class syconn.backend.storage.VoxelStorageLazyLoading(path, overwrite=False)[source]
Bases:
objectThis class is a variant of the VoxelStorage class that uses lazy loading via numpy npz files. It is designed to handle the storage of voxel data in a memory-efficient manner. The class does not support modification of npz storages once written, similar to VoxelStorage. It also only supports integer keys, which are internally converted to strings as required by npz, and then always converted back to integers for external use (e.g.
keys). When opening an existing npz file, the close method should be called (close).