syconn.handler package

Submodules

• syconn.handler.basics module

• syconn.handler.compression module

• syconn.handler.prediction module

• syconn.handler.config module

• syconn.handler.logger module

syconn.handler.basics module

class syconn.handler.basics.DelayedInterrupt(signals)

Bases: object

syconn.handler.basics.chunkify(lst, n)

Splits a list or array into a specified number of approximately equal-sized chunks.

This function divides a list or array into n chunks, where n is the minimum of the specified number and the length of the list. Each chunk contains consecutive elements from the original list.

Parameters

• lst (Union[list, ndarray]) – The list or numpy array to be chunked.

• n (int) – The desired number of chunks.

Examples

>>> chunkify(np.arange(10), 2)
>>> chunkify(np.arange(10), 100)

Return type

List[list]

Returns

A list of chunks. Length is np.min([n, len(lst)]).

syconn.handler.basics.chunkify_successive(l, n)

Yield successive n-sized chunks from l.

This generator function divides a list into chunks of size n and yields each chunk in turn, ensuring that all elements in the list are presented in these chunks.

Parameters

• l – The list to be chunked.

• n – The size of each chunk.

Yields

A generator of successive n-sized chunks of the list l.

syconn.handler.basics.chunkify_weighted(lst, n, weights)

Splits a list into weighted sub-lists.

This function divides a list into n sub-lists based on the provided weights. The weights are not necessarily used for sorting; they determine the distribution of elements across the sub-lists.

Parameters

• lst – The list to be chunked.

• n – The number of chunks to create.

• weights – An array of weights corresponding to the elements of lst.

Returns

A list of n chunks, where each chunk is a sublist of the original list.

syconn.handler.basics.convert_keys_byte2str(dc)

Converts byte string keys in a dictionary to regular strings.

This function recursively traverses a dictionary and converts all keys that are byte strings to regular strings.

Parameters

dc – The dictionary with byte string keys.

Returns

The dictionary with all keys converted to regular strings.

syconn.handler.basics.coordpath2anno(coords, scaling=None, add_edges=True)

Creates a SkeletonAnnotation from a path of coordinates.

This function generates a SkeletonAnnotation object from a list of coordinates, optionally scaling them and adding edges between consecutive nodes. The assumption is made that the coordinates are scaled and in order for edge creation.

Parameters

• coords – A numpy array of scaled coordinates.

• scaling – A tuple representing the scaling factors for each coordinate axis. If not provided, the global scaling parameters are used.

• add_edges – A boolean indicating whether to add edges between consecutive nodes.

Returns

A SkeletonAnnotation object representing the skeleton formed by the coordinates.

syconn.handler.basics.crop_bool_array(arr)

Crops a bool array to its True region.

This function finds the bounding box of the True region in a 3D boolean array and returns the cropped array along with the offset of the crop.

Parameters

arr – 3D boolean array The array to be cropped.

Returns

A tuple containing the cropped 3D boolean array and a list representing the offset of the crop in the format [x_min, y_min, z_min].

syconn.handler.basics.data2kzip(kzip_path, fpaths, fnames_in_zip=None, force_overwrite=True, verbose=False)

Writes multiple files to a k.zip archive and optionally removes original files.

This function adds multiple files to a k.zip archive and can optionally overwrite existing files with the same names in the archive. If force_overwrite is set to True, it will overwrite files. After adding the files to the archive, it removes the original files specified by fpaths if not contradicted by the calling code.

Parameters

• kzip_path (str) – The path to the k.zip archive.

• fpaths – A list of paths to the files to be added to the archive.

• fnames_in_zip – A list of names for the files within the archive. If not provided, the original file names are used.

• force_overwrite – A boolean indicating whether to overwrite existing files in the archive with the same names.

• verbose – A boolean indicating whether to print progress information.

Returns

None.

syconn.handler.basics.flatten(x)

Recursively flattens a nested iterable into a flat list.

This function replaces the deprecated compiler.ast.flatten by performing recursive flattening. It takes a nested iterable (e.g., list of lists of lists) and flattens it into a single list containing all the non-iterable elements. Originally shared under Public domain code: https://stackoverflow.com/questions/16176742/python-3-replacement-for- deprecated-compiler-ast-flatten-function

Parameters

x – The nested iterable to be flattened.

Returns

A flat list containing all the non-iterable elements of the input.

syconn.handler.basics.flatten_list(lst)

Flattens a list of lists into a single list.

This function takes a list of lists and concatenates their elements into a single list, preserving the order of elements, similar to np.concatenate.

Parameters

lst – A list of lists to be flattened.

Returns

A single list containing all the elements of the sublists.

syconn.handler.basics.get_filepaths_from_dir(directory, ending=('k.zip',), recursively=False, exclude_endings=False, fname_includes=())

Retrieves file paths with specific endings from a directory.

This function collects all file paths from a given directory that have specified endings. It can search recursively and include or exclude files based on the endings and substrings in the filenames.

Parameters

• directory – The directory to search for files.

• ending – A tuple, list, or string specifying the file endings to include.

• recursively – A boolean indicating whether to search subdirectories.

• exclude_endings – A boolean indicating whether to exclude files with the specified endings.

• fname_includes – A string or list of substrings that must be included in the filenames.

Returns

A list of strings representing the paths to files that match the specified criteria.

syconn.handler.basics.get_paths_of_skelID(id_list, traced_skel_dir)

Gather paths to kzip of skeletons with ID in id_list

Parameters

• id_list – list of str skeleton ID’s

• traced_skel_dir – str directory of mapped skeletons

Returns: list of str

paths of skeletons in id_list. If a skeleton ID does not have a corresponding file, None is returned in its place.

syconn.handler.basics.get_skelID_from_path(skel_path)

Extracts the skeleton ID from a file path.

This function parses the file path of a skeleton file to extract the skeleton ID, which is represented as an integer.

Parameters

skel_path – str The file path of the skeleton.

Returns

The ID of the skeleton.

Return type

int

syconn.handler.basics.group_ids_to_so_storage(ids, params, significant_digits=5)

Groups IDs and corresponding parameters for storage optimization.

This function creates a dictionary where keys are strings representing the last significant_digits of each ID, and values are lists of IDs that share the same key. It also groups corresponding parameters in the same manner.

Parameters

• ids – A list of integer IDs to be grouped.

• params – A list of parameters corresponding to each ID.

• significant_digits – The number of digits from the end of the ID to use for grouping.

Returns

A list containing the dictionary of grouped IDs and dictionaries of grouped parameters.

syconn.handler.basics.kd_factory(kd_path, channel='jpg')

Initializes a KnossosDataset at the given kd_path.

This function attempts to initialize a KnossosDataset by searching for configuration files in the specified path. It prioritizes pyk.conf files and attempts to handle different scenarios where the configuration might be located.

Notes

• Prioritizes pyk.conf files.

Parameters

• kd_path (str) – The file system path where the KnossosDataset configuration is expected to be found.

• channel (str) – The channel to use for the dataset. This argument is currently not used in the function.

Returns

An initialized KnossosDataset object. If the initialization fails due to missing configuration files, a ValueError is raised.

Raises

ValueError – If no configuration file can be found at the specified path.

syconn.handler.basics.load_pkl2obj(path)

Deserializes and loads an object from a pickle file.

This function reads a pickle file from the specified path and returns the deserialized object.

Parameters

path (str) – The path to the pickle file.

Returns

The object deserialized from the pickle file.

syconn.handler.basics.majority_element_1d(arr)

Finds the most frequent element in a 1D array.

This function returns the element that appears most frequently in the provided array. If multiple elements have the same highest frequency, the function returns the first one encountered.

Parameters

arr – np.array

Returns

scalar - The most frequent element in the array.

syconn.handler.basics.parse_cc_dict_from_g(g)

Parses connected components from a graph object and returns a dictionary mapping each component ID to its associated node IDs.

Parameters

g (networkx.Graph) – The graph object containing connected components.

Returns

A dictionary where each key is a component ID and the corresponding value is a

set of node IDs belonging to that component.

Return type

dict

syconn.handler.basics.parse_cc_dict_from_kml(kml_path)

Parses connected components from a Knossos mergelist file specified by the path and returns a dictionary mapping each component ID to its associated node IDs.

Parameters

kml_path (str) – The file path to the Knossos mergelist file.

Returns

A dictionary where each key is a component ID and the corresponding value is

a list of node IDs belonging to that component. The information regarding the numpy array in the generated docstring has been replaced with ‘list’ from the old docstring to resolve the conflict in the return type information.

Return type

dict

syconn.handler.basics.parse_cc_dict_from_kzip(k_path)

Parses connected components from a Knossos mergelist text file within a zip archive and returns a dictionary mapping each component ID to its associated node IDs.

Parameters

k_path (str) – The file path to the zip archive containing the Knossos mergelist file.

Returns

A dictionary where each key is a component ID and the corresponding value is a

numpy array of node IDs belonging to that component.

Return type

dict

syconn.handler.basics.prase_cc_dict_from_txt(txt)

Parse connected components from a Knossos mergelist text file.

Parameters

txt (str or bytes) – The content of a Knossos mergelist text file.

Returns

A mapping of each component ID to its associated node IDs.

Return type

dict

syconn.handler.basics.read_mesh_from_zip(zip_fname, fname_in_zip)

Reads a PLY mesh file from a zip archive.

This function extracts and reads the vertex and face data from a PLY file stored within a zip archive. Currently, it does not support reading normals.

Parameters

• zip_fname – The path to the zip file.

• fname_in_zip – The name of the PLY file within the zip archive.

Returns

A tuple containing three np.array objects for indices, vertices, and normals (the latter is not supported and will be None).

syconn.handler.basics.read_meshes_from_zip(zip_fname, fnames_in_zip)

Reads multiple PLY mesh files from a zip archive.

This function extracts and reads the vertex and face data from multiple PLY files stored within a zip archive. Currently, it does not support reading normals or other additional data.

Parameters

• zip_fname – The path to the zip file containing PLY files.

• fnames_in_zip – A list of filenames of the PLY files within the zip.

Returns

Three numpy arrays containing the vertices, faces, and a placeholder for normals (which is currently set to None) for each PLY file.

syconn.handler.basics.read_txt_from_zip(zip_fname, fname_in_zip)

Reads a text file from a zip archive.

This function extracts and reads the contents of a text file stored within a zip archive.

Parameters

• zip_fname (str) – The path to the zip file.

• fname_in_zip (str) – The name of the text file within the zip archive.

Returns

The content of the text file.

Return type

bytes

syconn.handler.basics.remove_from_zip(zipfname, *filenames)

Removes specified files from a zip archive.

This function deletes files with the given names from a zip archive.

Parameters

• zipfname – The path to the zip archive.

• *filenames – A variable number of strings representing the names of the files to be removed from the archive.

Returns

None.

syconn.handler.basics.safe_copy(src, dest, safe=True)

Copies file and optionally throws an exception if destination exists.

This function copies a file from the specified source path (src) to the destination path (dest). If the safe parameter is True, the function checks if the destination file exists and throws an exception to prevent overwriting. If safe is False, the file is copied and any existing file at the destination is replaced.

Note: Credit to Misandrist on Stackoverflow for the original implementation

date 03/31/17.

Parameters

• src (str) – The source file path.

• dest (str) – The destination file path.

• safe (bool) – If True, raise an exception if the destination file exists. If False, allows overwriting of the destination file.

Returns

None.

syconn.handler.basics.str_delta_sec(seconds)

Converts a time duration in seconds to a human-readable string, omitting time units that are zero.

Examples

>>> sec = 2 * 24 * 3600 + 12 * 3600 + 5 * 60 + 1
>>> str_rep = str_delta_sec(sec)
>>> assert str_rep == '2d:12h:05min:01s'
>>> assert str_delta_sec(4 * 3600 + 20 * 60 + 10) == '4h:20min:10s'

Parameters

seconds (int) – The time duration in seconds, e.g., result of a time delta.

Returns

A human-readable string representation of the time duration, formatted as

’Xd:Xh:XXmin:XXs’ where X represents non-zero time units, e.g. ‘2d:12h:05min:01s’ for sec = 1 + 5 * 60 + 12 * 3600 + 2 * 24 * 3600.

Return type

str

syconn.handler.basics.switch_array_entries(this_array, entries)

Switches two specified entries in an array.

This function swaps the values at two specified indices in the given array.

Parameters

• this_array – The array in which the entries will be switched.

• entries – A list or tuple containing two indices whose corresponding values in this_array will be swapped.

Returns

The array with the two entries switched.

syconn.handler.basics.temp_seed(seed)

A context manager for temporarily setting the random seed within a block of code to ensure reproducibility of random operations. (From https://stackoverflow.com/questions/49555991/can-i-create-a-local-numpy-random-seed)

Parameters

seed (int) – The seed value to set for random number generation.

Returns

This context manager does not return any value but ensures that the

random state is reset to its original state after the block of code is executed.

Return type

None

syconn.handler.basics.texts2kzip(kzip_path, texts, fnames_in_zip, force_overwrite=False)

Writes multiple text strings to files within a k.zip archive.

This function creates or updates a k.zip archive by adding multiple text files with the specified contents. It can optionally overwrite existing files if the ‘force_overwrite’ parameter is set to True.

Parameters

• kzip_path (str) – The path to the k.zip archive.

• texts (List[str]) – A list of text contents to write to the files within the archive.

• fnames_in_zip (List[str]) – A list of names for the files to be created or updated within the archive, indicating the name of the file when added to the zip.

• force_overwrite (bool) – A boolean indicating whether to overwrite existing files with the same names.

Returns

None.

syconn.handler.basics.write_data2kzip(kzip_path, fpath, fname_in_zip=None, force_overwrite=False)

Writes a file to a k.zip archive.

This function adds a file to a k.zip archive, optionally overwriting an existing file with the same name.

Parameters

• kzip_path (str) – The path to the k.zip archive.

• fpath (str) – The path to the file to be added to the archive.

• fname_in_zip (str) – The name of the file within the archive. If not provided, the original file name is used.

• force_overwrite (bool) – A boolean indicating whether to overwrite an existing file with the same name.

Returns

None.

syconn.handler.basics.write_obj2pkl(path, objects)

Writes object to pickle file.

This function writes a given object to a pickle file at the specified path.

Parameters

• path (str) – Destination.

• objects (object) – The object to be serialized and written to the file.

Returns

None.

syconn.handler.basics.write_txt2kzip(kzip_path, text, fname_in_zip, force_overwrite=False)

Writes a text string to a file within a k.zip archive.

This function creates or updates a k.zip archive by adding a text or byte file with the specified content. It can optionally overwrite existing files if force_overwrite is set to True.

Parameters

• kzip_path – The path to the k.zip archive.

• text – The text or bytes content to write to the file within the archive.

• fname_in_zip – The name of the file to be created or updated within the archive.

• force_overwrite – A boolean indicating whether to overwrite existing files with the same name in the archive.

Returns

None.

syconn.handler.compression module

syconn.handler.compression.arrtolz4string(arr)

Converts (multi-dimensional) array to an LZ4 compressed byte string.

This function takes a NumPy array or a list and compresses it using the LZ4 algorithm, returning the compressed data as a byte string. If the input is a list, it is first converted to a NumPy array. If the array is empty, an empty byte string is returned. In case of an OverflowError during compression, a warning is logged, and the ‘arrtolz4string_list’ function is used instead.

Parameters

arr (ndarray) – The NumPy array to be compressed.

Return type

bytes

Returns

A byte string containing the LZ4 compressed data.

syconn.handler.compression.arrtolz4string_list(arr)

Converts a NumPy array to a list of LZ4 compressed byte strings.

This function is similar to ‘arrtolz4string’ but returns a list of compressed byte strings instead of a single byte string, to handle large arrays that cannot be compressed in a single step. If the input is a list, it is first converted to a NumPy array. If the array is empty, a list containing a single empty byte string is returned.

Parameters

arr (ndarray) – The NumPy array or a list to be compressed.

Return type

List[bytes]

Returns

A list of LZ4 compressed byte strings.

syconn.handler.compression.load_from_h5py(path, hdf5_names=None, as_dict=False)

Loads data from an HDF5 file.

This function reads data from an HDF5 file and returns it either as a list of NumPy arrays or as a dictionary mapping dataset names to their corresponding arrays. If ‘hdf5_names’ is None, all datasets in the file are loaded. If ‘as_dict’ is True, the data is returned as a dictionary; otherwise, it is returned as a list.

Parameters

• path (str) – The file path of the HDF5 file.

• hdf5_names (Optional[Iterable[str]]) – An iterable of dataset names to load; if None, all datasets are loaded.

• as_dict (bool) – Whether to return the data as a dictionary.

Return type

Union[Dict[str, ndarray], List[ndarray]]

Returns

The data stored at path either as a list of NumPy arrays (ordering as hdf5_names) or as a dictionary.

syconn.handler.compression.load_lz4_compressed(p, shape=(-1, 20, 2, 128, 256), dtype=<class 'numpy.float32'>)

Loads a NumPy array from an LZ4 compressed file.

This function reads LZ4 compressed data from a file and decompresses it to reconstruct the original NumPy array. The shape and data type of the original array must be provided. In cases of decompression issues, possibly due to overflow in Python 2, it attempts to recursively load and concatenate split parts of the array to return the complete array.

Parameters

• p (str) – The file path of the LZ4 compressed file.

• shape (Tuple[int]) – The shape of the original array, provided as a tuple.

• dtype (dtype) – The data type of the original array, provided as a type.

Returns

The decompressed NumPy array as np.array.

syconn.handler.compression.lz4string_listtoarr(str_lst, dtype=<class 'numpy.float32'>, shape=None)

Converts a list of LZ4 compressed byte strings back to a NumPy array.

This function takes a list of byte strings, each compressed using the LZ4 algorithm, and concatenates their decompressed data to reconstruct the original NumPy array. If the input is already a NumPy array, it is returned as is. If the list is empty, an empty NumPy array is returned. If the shape is provided, the decompressed data is reshaped accordingly.

Parameters

• str_lst (Union[List[bytes], ndarray]) – A list of LZ4 compressed byte strings or a NumPy array. If numpy array, do nothing.

• dtype (dtype) – The data type of the original array. If not provided, the dtype inferred from the compressed data will be used.

• shape (Optional[Tuple[int]]) – The shape of the original array, used for reshaping the decompressed data into the specified shape. If not provided, the shape is inferred from the compressed data.

Return type

ndarray

Returns

A 1d numpy array reconstructed from the compressed data.

syconn.handler.compression.lz4stringtoarr(string, dtype=<class 'numpy.float32'>, shape=None)

Converts an LZ4 compressed byte string back to a NumPy array.

This function decompresses a byte string that was previously compressed using the LZ4 algorithm and reconstructs the original NumPy array. The data type and shape of the original array must be provided. If the input string is empty, an empty NumPy array is returned. If the shape is provided, the decompressed data is reshaped accordingly.

Parameters

• string (bytes) – The LZ4 compressed byte string to be decompressed.

• dtype (dtype) – The data type of the original array, to ensure data integrity.

• shape (Optional[Tuple[int]]) – The original array’s shape, used to reshape the decompressed data.

Returns

An N-dimensional numpy array reconstructed from the compressed data.

syconn.handler.compression.save_lz4_compressed(p, arr, dtype=<class 'numpy.float32'>)

Saves a NumPy array as an LZ4 compressed file.

This function compresses a NumPy array using the LZ4 algorithm and saves the compressed data to a file. If an OverflowError or ValueError occurs during compression, the array is split into two halves and the function is called recursively to save each half separately, addressing issues specifically related to python2.

Parameters

• p (str) – The file path where the compressed data will be saved.

• arr (ndarray) – The NumPy array to be compressed and saved.

• dtype (dtype) – The data type to which the array should be cast before compression.

syconn.handler.compression.save_to_h5py(data, path, hdf5_names=None, overwrite=False, compression=True)

Saves data to an HDF5 file.

This function writes data to an HDF5 file, using dataset names provided in ‘hdf5_names’ if ‘data’ is a list. If ‘data’ is a dictionary, its keys are used as dataset names. If ‘overwrite’ is True, existing files at the specified path are overwritten. If ‘compression’ is True, gzip compression is used, which is beneficial for sparse and ordered data.

Parameters

• data (Union[Dict[str, ndarray], List[ndarray]]) – The data to be saved, either as a list of NumPy arrays or as a dictionary. If list, ‘hdf5_names’ must be provided and have the same length as ‘data’.

• path (str) – Forward-slash separated path to the file where the HDF5 file will be saved.

• hdf5_names (Optional[List[str]]) – The names of the datasets to be saved; required if ‘data’ is a list and must be the same length as ‘data’.

• overwrite (bool) – Whether to overwrite existing files.

• compression (bool) – Whether to use gzip compression, recommended for sparse and ordered data.

Raises

• TypeError – If ‘data’ is a list and ‘hdf5_names’ is not provided.

• ValueError – If the length of ‘hdf5_names’ does not match the length of ‘data’.

syconn.handler.prediction module

syconn.handler.prediction.binarize_labels(labels, foreground_ids, target_labels=None)

Transforms label array to binary label array (0=background, 1=foreground) or to the labels provided in target_labels by mapping the foreground IDs accordingly.

Parameters

• labels (np.ndarray) – Input label array.

• foreground_ids (Iterable[int]) – IDs to be considered as foreground.

• target_labels (Optional[Iterable[int]], optional) – Target labels for mapping foreground IDs. Defaults to None if not provided.

Returns

Transformed label array.

Return type

np.ndarray

syconn.handler.prediction.certainty_estimate(inp, is_logit=False)

Estimates the certainty of (independent) predictions of the same sample:

1. If is_logit is True, apply softmax on the input.

2. Sum the evidence per class and (re-)normalize.

3. Compute the entropy, scale it with the maximum entropy (equal probabilities) and subtract it from 1 to get the certainty.

Parameters

• inp (ndarray) – 2D array of prediction results (N: number of samples, C: Number of classes).

• is_logit (bool) – If True, applies softmax(inp, axis=1).

Return type

float

Returns

Certainty measure based on the entropy of a set of (independent) predictions.

syconn.handler.prediction.chunk_pred(ch, model, debug=False)

Helper function to write chunks to disk.

This function writes chunks of predicted data to disk during the prediction process. It handles individual chunks, storing the output of the prediction made by the given model.

Parameters

• ch (chunky.Chunk) – The chunk of data to be predicted (Chunk).

• model (torch.nn.Module) – The model or the path to the model used for prediction (str or model object).

• debug (bool) – A flag indicating whether to run in debug mode (bool).

Returns

None. The function writes data to disk as a side effect.

syconn.handler.prediction.create_h5_from_kzip(zip_fname, kd_p, foreground_ids=None, overwrite=True, raw_data_offset=75, debug=False, mag=1, squeeze_data=True, target_labels=None, apply_mops_seg=None)

Create .h5 files for elektronn3 (zyx) input. Only supports binary labels (0=background, 1=foreground).

Examples

Suppose your k.zip file contains the segmentation GT with two segmentation IDs 1, 2 and is stored at zip_fname. The corresponding KnossosDataset is located at kd_p. The following code snippet will create an .h5 file in the folder of zip_fname with the raw data (additional offset controlled by raw_data_offset) and the label data (either binary or defined by target_labels) with the keys raw and label respectively:

create_h5_from_kzip(d_p=kd_p, raw_data_offset=75, zip_fname=zip_fname, mag=1, foreground_ids=[1, 2], target_labels=[1, 2])

Parameters

• zip_fname (str) – Path to the annotated kzip file.

• kd_p (str) – Path to the underlying raw data stored as KnossosDataset.

• foreground_ids (Optional[Iterable[int]], optional) – IDs to be converted to foreground (1). Defaults to None, meaning all non-zero are foreground.

• overwrite (bool, optional) – If True, overwrites existing h5 files. Defaults to False.

• raw_data_offset (int, optional) – Number of voxels for additional raw offset. Defaults to 75. Set to 0 if debug is True.

• debug (bool, optional) – If True, adds a ‘debug’ suffix and adapts bit depths. Defaults to False.

• mag (int, optional) – Magnification level of the data. Defaults to 1.

• squeeze_data (bool, optional) – If True, squeezes label and raw data. Defaults to False.

• target_labels (Optional[Iterable[int]], optional) – Target labels for mapping foreground IDs. Must be set if foreground_ids is specified. Defaults to None.

• apply_mops_seg (Optional[List[str]], optional) – Morphological operations to apply to segmentation. Defaults to None.

syconn.handler.prediction.create_h5_gt_file(fname, raw, label, foreground_ids=None, target_labels=None, debug=False, apply_mops_seg=None)

Creates an h5 file for ELEKTRONN input from raw and label arrays, supporting only binary labels (0=background, 1=foreground). For creating true negative cubes, set foreground_ids=[] to indicate no foreground. If foreground_ids is None, all non-zero values are treated as foreground.

Parameters

• fname (str) – Path where the h5 file will be saved.

• raw (np.ndarray) – Raw data array.

• label (np.ndarray) – Label data array for binary classification.

• foreground_ids (Optional[Iterable[int]], optional) – IDs to be treated as foreground. If None, all non-zero values are considered foreground. Defaults to None.

• target_labels (Optional[Iterable[int]], optional) – If set, foreground_ids must also be set. Each ID in foreground_ids will be mapped to the corresponding label in target_labels. Defaults to None.

• debug (bool, optional) – If True, stores labels and raw as uint8 ranging from 0 to 255 for debugging purposes. Defaults to False.

• apply_mops_seg (Optional[List[str]], optional) – List of string identifiers for morphological operations to be applied to segmentation. Defaults to None.

syconn.handler.prediction.dense_predicton_helper(raw, predictor, is_zyx=False, return_zyx=False)

Assists with the prediction of dense volumes.

This function helps with the prediction of dense volumes by handling the transformation of data formats and calling the model’s prediction method.

Parameters

• raw (ndarray) – The input data array in CXYZ format.

• predictor (Predictor) – The model which performs the inference. Requires predictor.predict.

• is_zyx – A flag indicating if the input data is already in ZYX format.

• return_zyx – A flag indicating if the output data should be in ZYX format.

Return type

ndarray

Returns

The inference result in CXYZ format as uint8 ranging from 0 to 255.

syconn.handler.prediction.dense_predictor(args)

Transforms volumes and performs model predictions.

This function is responsible for transforming volumes by switching XYZ and ZYX order before passing them to the model for prediction. The function may also transform the predicted volumes back if necessary.

Parameters

args –

Tuple( chunk_ids: list

List of chunk IDs in the chunk dataset.

kd_p: str

Path to the Knossos dataset configuration file.

cd_p: str

Destination folder for the chunk dataset containing prediction.

model_p: str

Path to the model used for predictions.

offset: Variable type and description from the old docstring. chunk_size: Variable type and description from the old docstring. … Additional parameters related to the prediction process.

Returns

None. The function is used for its side effects of writing predictions to disk.

syconn.handler.prediction.get_celltype_model_e3()

Retrieves the elektronn3 model trained for cell type prediction. Unlike standard e3 InferenceModel instances which employ naive_view_normalization_new, this model applies view normalization in a distinct downstream inference method (predict_sso_celltype) due to the additional scalar inputs which should not undergo normalization.

Returns

The trained InferenceModel for cell type prediction, which handles view normalization differently to accommodate additional scalar inputs.

syconn.handler.prediction.get_glia_model_e3()

Retrieves the elektronn3 model for glia prediction, typically trained with naive_view_normalization_new.

Returns

The trained InferenceModel for glia prediction.

syconn.handler.prediction.get_knn_tnet_embedding_e3()

Retrieves the K-Nearest Neighbors classifier trained on triplet network embeddings.

Returns

The trained KNeighborsClassifier instance.

syconn.handler.prediction.get_myelin_cnn()

Retrieves the elektronn3 model trained to predict binary myelin-in class.

Returns

The trained Inference model for myelin prediction.

syconn.handler.prediction.get_pca_tnet_embedding_e3()

Retrieves the PCA transformation trained on triplet network embeddings.

Returns

The trained PCA instance. This instance is fitted to the specific embeddings generated by the triplet network, allowing for dimensionality reduction and visualization of high-dimensional data.

syconn.handler.prediction.get_semseg_axon_model()

Retrieves the elektronn3 model trained for semantic segmentation of axons.

Returns

The trained InferenceModel for semantic segmentation of axons.

syconn.handler.prediction.get_semseg_spiness_model()

Retrieves the elektronn3 model trained for semantic segmentation of spines.

Returns

The trained InferenceModel for semantic segmentation of spines.

syconn.handler.prediction.get_tripletnet_model_e3()

Retrieves the elektronn3 model trained with naive_view_normalization_new.

Returns

The trained InferenceModel used for comparison tasks in a triplet network configuration.

syconn.handler.prediction.int2str_converter(label, gt_type)

Converts an integer label into a semantic string based on the specified ground truth type. The conversion is specific to the domain of connectomics, where labels represent various cellular structures or cell types.

Parameters

• label (int) – An integer representing the label to be converted.

• gt_type (str) – A string specifying the ground truth type. It determines the mapping from integer labels to semantic strings. Examples include ‘spgt’ for spines, ‘axgt’ for cell compartments, ‘ctgt’ for cell types, ‘ctgt_v2’, and ‘ctgt_j0251’ for different versions of cell type ground truths.

Return type

str

Returns

A string representing the semantic label corresponding to the input integer. If the label does not match any known category, a value indicating an unknown label is returned (commonly -1 or “N/A”).

Raises

ValueError – If the provided ground truth type is not recognized by the function.

syconn.handler.prediction.knn_clf_tnet_embedding(fold, fit_all=False)

Trains a K-Nearest Neighbors classifier on triplet network embeddings.

This method assumes embedding for GT views has been created already in ‘fold’ and stored in l_train_%d.npy / l_valid_%d.npy files.

Parameters

• fold (str) – The directory containing the embeddings and labels.

• fit_all (bool) – A flag indicating whether to fit the classifier on all data.

Returns

The trained KNeighborsClassifier instance.

syconn.handler.prediction.load_gt_from_kzip(zip_fname, kd_p, raw_data_offset=75, verbose=False, mag=1)

Loads ground truth from zip file, generated with Knossos. Corresponding dataset config file is located at kd_p.

Parameters

• zip_fname (str) – Path to the zip file containing ground truth data.

• kd_p (str or List[str]) – Path or list of paths to the Knossos dataset configuration file(s).

• raw_data_offset (int or np.array) – Number of voxels for additional raw offset, i.e., the offset for the raw data will be label_offset - raw_data_offset, while the raw data volume will be label_volume + 2*raw_data_offset. It will use ‘kd.scaling’ to account for dataset anisotropy if scalar or a list of length 3 has to be provided for a custom x, y, z offset. Defaults to 75 if not provided.

• verbose (bool, optional) – Enables verbose output. Defaults to False.

• mag (int) – Magnification level of the data. Defaults to 1 if not provided.

Returns

A tuple containing two numpy arrays:

• raw data (float32) normalized by 1/255.

• label data (uint16)

Return type

tuple

syconn.handler.prediction.naive_view_normalization(d)

Performs a simple normalization of the input data for backward compatibility.

Parameters

d – The input data to be normalized.

Returns

The normalized data.

syconn.handler.prediction.naive_view_normalization_new(d)

Performs a simple normalization on the input data.

Parameters

d – The input data to be normalized.

Returns

The normalized data.

syconn.handler.prediction.overlaycubes2kzip(dest_p, vol, offset, kd_path)

Writes segmentation volume to kzip.

Parameters

• dest_p (str) – Path to the destination k.zip file.

• vol (np.ndarray) – Segmentation or prediction volume as an unsigned integer array in XYZ order.

• offset (np.ndarray) – Offset of the volume in the dataset.

• kd_path (str) – Path to the Knossos dataset configuration file.

Returns

A numpy array with shape [Z, X, Y] following the segmentation

or prediction volume.

Return type

np.ndarray

syconn.handler.prediction.parse_movement_area_from_zip(zip_fname)

Parses the MovementArea (bounding box of labeled volume) from an annotation.xml file within a (k.)zip file.

Parameters

zip_fname (str) – Path to the zip file containing the annotation.xml file.

Returns

Array representing the Movement Area with shape [2, 3].

Return type

np.ndarray

syconn.handler.prediction.pca_tnet_embedding(fold, n_components=3, fit_all=False)

Performs PCA on triplet network embeddings.

Parameters

• fold – The directory containing the embeddings and labels.

• n_components – The number of principal components to keep.

• fit_all – A flag indicating whether to fit the PCA on all data.

Returns

The trained PCA instance. Assumes embedding for GT views has been created already in ‘fold’ and put into l_train_%d.npy / l_valid_%d.npy files.

syconn.handler.prediction.pred_dataset(*args, **kwargs)

syconn.handler.prediction.predict_dense_to_kd(kd_path, target_path, model_path, n_channel, target_names=None, target_channels=None, channel_thresholds=None, log=None, mag=1, overlap_shape_tiles=(40, 40, 20), cube_of_interest=None, overwrite=False, cube_shape_kd=None)

Performs dense prediction on a Knossos dataset and writes the results to new KnossosDatasets.

This function runs predictions on the entire Knossos dataset specified by kd_path. The predictions are written to new KnossosDatasets with names specified by target_names in the directory target_path. If no threshold is provided and there is only one channel per target name, the KnossosDataset will contain a probability map in the raw channel as uint8 (0..255). Otherwise, classification results will be written to the overlay channel.

Notes

• Requires a significant amount of GPU memory (at least 12GB).

• The resulting KnossosDatasets do not use pyknossos configurations.

• The GPU memory usage should be adjustable from the configuration or determined automatically.

Parameters

• kd_path (str) – Path to the KnossosDataset configuration file of the raw data.

• target_path (str) – Destination directory for the output KnossosDatasets containing predictions.

• model_path (str) – Path to the elektronn3 model used for predictions. Loaded via the Predictor.

• n_channel (int) – Number of channels predicted by the model.

• target_names (Optional[Iterable[str]]) – Names of the target KnossosDatasets, e.g. target_names=[‘synapse_fb’, ‘synapse_type’]. Defaults to [‘pred’]. Length must match with target_channels.

• target_channels (Optional[Iterable[Iterable[int]]]) – Channel IDs in the prediction for each target KnossosDataset, e.g. target_channels=[(1, 2)] if the prediction has two foreground labels. Defaults to [[ix for ix in range(n_channel)]]. Length must match with target_names.

• channel_thresholds (Optional[Iterable[Union[float, Any]]]) – Thresholds for channels. If None and the number of channels for the target KnossosDataset is 1, probabilities are stored. Otherwise, defaults to 0.5.

• log (Optional[Logger]) – Logger for output messages.

• mag (int) – Magnification level of the data.

• overlap_shape_tiles (Tuple[int, int, int]) –

  Overlap in voxels [XYZ] used for each tile predicted during inference. Properties such as chunk size and tile count may additionally influence the overlap:

  chunk_size = np.array([1024, 1024, 256], dtype=np.int32)  # XYZ
  n_tiles = np.array([4, 4, 16])
  tile_shape = (chunk_size / n_tiles).astype(np.int32)
  # final input shape must be multiple of tile_shape
  overlap_shape = tile_shape // 2
  

• cube_of_interest (Optional[Tuple[ndarray]]) – Bounding box of the volume of interest (min and max coordinate in voxels) in the respective magnification level (see kwarg mag).

• overwrite (bool) – Whether to overwrite existing KnossosDatasets.

• cube_shape_kd (Optional[Tuple[int]]) – Cube shape for storing sub-volumes in KnossosDataset on the file system.

syconn.handler.prediction.predict_h5(h5_path, m_path, clf_thresh=None, mfp_active=False, gpu_ix=0, imposed_patch_size=None, hdf5_data_key=None, data_is_zxy=True, dest_p=None, dest_hdf5_data_key='pred', as_uint8=True)

Predicts data from an h5 file using a specified predictive model.

Parameters

• h5_path (str) – Path to the h5 file containing raw data.

• m_path (str) – Path to the predictive model.

• clf_thresh (float, optional) – Classification threshold. If None, no thresholding is applied.

• mfp_active (bool) – Flag to activate max-fragment pooling. Defaults to False.

• gpu_ix (int, optional) – GPU index for model prediction. Defaults to 0.

• imposed_patch_size (tuple, optional) – Imposed patch size for the model.

• hdf5_data_key (str, optional) – Key for raw data in the h5 file. If None, the first entry is used.

• data_is_zxy (bool) – Flag to indicate data order. If False, data is assumed to be in [X, Y, Z] format.

• dest_p (str) – Destination path for the prediction output.

• dest_hdf5_data_key (str, optional) – Key for predicted data in the output h5 file. Defaults to ‘pred’.

• as_uint8 (bool, optional) – Flag to store prediction as uint8. Defaults to True.

syconn.handler.prediction.predict_kzip(kzip_p, m_path, kd_path, clf_thresh=0.5, mfp_active=False, dest_path=None, overwrite=False, gpu_ix=0, imposed_patch_size=None)

Predicts data contained in a k.zip file using a specified predictive model.

Parameters

• kzip_p (str) – Path to the k.zip file containing raw data cube information.

• m_path (str) – Path to the predictive model.

• kd_path (str) – Path to the Knossos dataset configuration file.

• clf_thresh (float, optional) – Classification threshold. If not specified, the function uses a default value.

• mfp_active (bool, optional) – Flag to activate max-fragment pooling. Defaults to False. When set to False, the prediction uses standard processing.

• dest_path (str, optional) – Destination folder path. If None, the folder of k.zip is used. Defaults to None.

• overwrite (bool, optional) – Flag to enable overwriting of existing files. Defaults to False.

• gpu_ix (int, optional) – GPU index for model prediction. Defaults to 0.

• imposed_patch_size (tuple, optional) – Imposed patch size for the model. If not specified, the model determines the patch size.

Returns

The function returns predictions for the data in the specified k.zip file. If a destination path is provided, it will save the predictions there, otherwise, it saves in the folder of the k.zip file. The function supports GPU prediction, overwriting existing files, and custom patch sizes when specified.

syconn.handler.prediction.prediction_helper(raw, model, override_mfp=True, imposed_patch_size=None)

Helper function for predicting raw volumes (range: 0 to 255; uint8). This function predicts raw volumes and converts the coordinate format from XYZ to the ELEKTRONN-specific ZXY before returning the prediction in the original XYZ format. Ensure the imposed patch size is specified in ZXY.

Parameters

• raw – np.array The raw volume data in XYZ format.

• model – str or model object The path to the model (.mdl) or the model object itself.

• override_mfp – bool A flag indicating whether to override max fragment pooling.

• imposed_patch_size – tuple The patch size imposed on the model, in ZXY format.

Returns: np.array

The prediction data in XYZ format.

syconn.handler.prediction.str2int_converter(comment, gt_type)

Converts semantic string labels into integer labels based on the ground truth type.

Parameters

• comment (str) – The semantic string label to be converted.

• gt_type (str) – The type of ground truth, which determines the conversion logic.

Return type

int

Returns

The integer label corresponding to the semantic string label.

syconn.handler.prediction.to_knossos_dataset(kd_p, kd_pred_p, cd_p, model_p, imposed_patch_size, mfp_active=False)

Converts a chunk dataset to a Knossos dataset using a model for prediction.

This function is deprecated and will be replaced by predict_dense_to_kd. It predicts the entire or partial Knossos dataset and writes the results to a Knossos dataset.

Parameters

• kd_p – The path to the Knossos dataset configuration file.

• kd_pred_p – The path to the Knossos dataset directory which will contain the prediction.

• cd_p – The destination folder for the chunk dataset containing the prediction.

• model_p – The path to the ELEKTRONN2 model.

• imposed_patch_size – The patch size of the model.

• mfp_active – A flag indicating whether max-fragment pooling is active.

Returns

None. The function is used for its side effects of writing predictions to disk.

syconn.handler.prediction.views2tripletinput(views)

Converts input views to the format required by a triplet network.

Parameters

views – The input views to be converted.

Returns

The converted views in the format required by the triplet network.

syconn.handler.prediction.xyz2zxy(vol)

Swaps axes to ELEKTRONN convention ([M, .., X, Y, Z] -> [M, .., Z, X, Y]).

Parameters

vol (np.ndarray) – Input volume in the shape [M, .., X, Y, Z].

Returns

Reordered volume in the shape [M, .., Z, X, Y].

Return type

np.ndarray

syconn.handler.prediction.xyz2zyx(vol)

Swaps axes to ELEKTRONN convention ([M, .., X, Y, Z] -> [M, .., Z, X, Y]).

Parameters

vol (ndarray) – np.array [M, .., X, Y, Z]

Returns: np.array [M, .., Z, X, Y]

Return type

ndarray

syconn.handler.prediction.zxy2xyz(vol)

Swaps axes to ELEKTRONN convention, converting a volume from ZXY to XYZ order.

Parameters

vol (np.ndarray) – Input volume with shape [M, .., Z, X, Y].

Returns

The volume rearranged to shape [M, .., X, Y, Z].

Return type

np.ndarray

syconn.handler.prediction.zyx2xyz(vol)

Swaps axes to ELEKTRONN convention ([M, .., Z, X, Y] -> [M, .., X, Y, Z]).

Parameters

vol (ndarray) – np.array [M, .., Z, X, Y]

Returns: np.array [M, .., X, Y, Z]

Return type

ndarray

syconn.handler.config module

class syconn.handler.config.DynConfig(wd=None, log=None, fix_config=False)

Bases: syconn.handler.config.Config

Enables dynamic and SyConn-wide update of working directory ‘wd’ and provides an interface to all working directory dependent parameters.

Notes

• Due to sync. checks it is favorable to not use __getitem__() inside loops.

Examples

To initialize a working directory at the beginning of your script, run:

from syconn import global_params
global_params.wd = '~/SyConn/example_cube1/'
cfg = global_params.config  # this is the `DynConfig` object

property allow_mesh_gen_cells: bool

If True, meshes are not provided for cell supervoxels and will be computed from scratch, see use_new_meshing.

Return type

bool

Returns

A boolean indicating if mesh generation for cell supervoxels is permitted.

property allow_ssv_skel_gen: bool

Determines whether skeleton generation for cell supervoxels is allowed, or if provided a priori. A naive sampling procedure is currently employed.

Return type

bool

Returns

A boolean indicating if skeleton generation for cell supervoxels is permitted, matching the configuration specified in the config.yml file.

property astrocyte_svagg_list_path: str

Retrieves the path to the agglomeration list for astrocytes.

Return type

str

Returns

Path to agglomeration list (list of SV IDs for every cell).

astrocyte_svgraph_path()

Retrieves the path to the astrocyte supervoxel graph.

Return type

str

Returns

The path to the astrocyte supervoxel graph.

property asym_label: Optional[int]

Retrieves the label used for asymmetric synapses.

Return type

Optional[int]

Returns

The label for asymmetric synapses, if available.

property batchjob_script_folder: str

Retrieves the path to the folder containing batchjob scripts.

Return type

str

Returns

The path to the batchjob script folder.

property default_conf: syconn.handler.config.Config

Loads the default configuration from the config.yml file if it has not been loaded already.

Return type

Config

Returns

The default configuration object.

property entries

Retrieves the current configuration entries, checking for updates to the working directory before returning.

Returns

A dictionary of the current configuration entries.

property init_svgraph_path: str

Retrieves the path to the initial region adjacency graph (RAG).

Return type

str

Returns

Path to initial RAG.

property kd_asym_path: str

Retrieves the path to the synaptic asym. type probability map stored as a KnossosDataset.

Return type

str

Returns

Path to synaptic asym. type probability map stored as KnossosDataset.

property kd_er_path: str

Retrieves the path to the ER probability map or binary predictions stored as a KnossosDataset.

Return type

str

Returns

Path to ER probability map or binary predictions stored as KnossosDataset.

property kd_golgi_path: str

Retrieves the path to the Golgi probability map or binary predictions stored as a KnossosDataset.

Return type

str

Returns

Path to Golgi probability map or binary predictions stored as KnossosDataset.

property kd_mi_path: str

Retrieves the path to the mitochondria probability map or binary predictions stored as a KnossosDataset.

Return type

str

Returns

Path to mitochondria probability map or binary predictions stored as KnossosDataset.

property kd_organelle_seg_paths: Dict[str, str]

Retrieves the paths to KnossosDataset of available cellular organelle segmentations.

Return type

Dict[str, str]

Returns

Dictionary containing the paths to KnossosDataset of available cellular organelles defined in global_params.config[‘process_cell_organelles’].

property kd_organelles_paths: Dict[str, str]

Retrieves the paths to KnossosDataset of available cellular organelle probability maps.

Return type

Dict[str, str]

Returns

Dictionary containing the paths to KnossosDataset of available cellular organelles specified in global_params.config[‘process_cell_organelles’].

property kd_seg_path: str

Retrieves the path to the cell supervoxel segmentation KnossosDataset.

Return type

str

Returns

Path to cell supervoxel segmentation KnossosDataset.

property kd_sj_path: str

Retrieves the path to the synaptic junction probability map or binary predictions stored as a KnossosDataset.

Return type

str

Returns

Path to synaptic junction probability map or binary predictions stored as KnossosDataset.

property kd_sym_path: str

Retrieves the path to the synaptic sym. type probability map stored as a KnossosDataset.

Return type

str

Returns

Path to synaptic sym. type probability map stored as KnossosDataset.

property kd_vc_path: str

Retrieves the path to the vesicle cloud probability map or binary predictions stored as a KnossosDataset.

Return type

str

Returns

Path to vesicle cloud probability map or binary predictions stored as KnossosDataset.

property model_dir: str

Retrieves the path to the directory containing the classification models.

Return type

str

Returns

Path to model directory.

property mpath_axonsem: str

Retrieves the path to the model trained on detecting axon, terminal boutons, en-passant boutons, dendrites, and somata via 2D projections.

Return type

str

Returns

Path to model trained on detecting axon, terminal boutons and en-passant, dendrites and somata via 2D projections.

property mpath_celltype_e3: str

Retrieves the path to the model trained on predicting cell types from multi-view sets.

Return type

str

Returns

Path to model trained on prediction cell types from multi-view sets.

property mpath_celltype_pts: str

Retrieves the path to the model trained on predicting cell types from point data.

Return type

str

Returns

Path to model trained on prediction cell types from point data.

property mpath_compartment_pts: str

Returns: Path to model trained on detecting axon, terminal and en-passant boutons, dendritic shaft, spine head and neck, and soma from point data.

Return type

str

property mpath_er: str

Retrieves the path to the model trained on identifying cell parts occupied by ER within 3D EM raw data.

Return type

str

Returns

Path to model trained on identifying cell parts occupied by ER within 3D EM raw data.

property mpath_glia_e3: str

Retrieves the path to the model trained to classify local 2D projections into glia vs. neuron.

Return type

str

Returns

Path to model trained to classify local 2D projections into glia vs. neuron (img2scalar).

property mpath_glia_pts: str

Retrieves the path to the point-based model trained to classify local 2D projections into glia vs. neuron.

Return type

str

Returns

Path to point-based model trained to classify local 2D projections into glia vs. neuron.

property mpath_golgi: str

Retrieves the path to the model trained on identifying cell parts occupied by Golgi Apparatus within 3D EM raw data.

Return type

str

Returns

Path to the Golgi identification model.

property mpath_mivcsj: str

Retrieves the path to the model trained on identifying synapse types (symmetric vs. asymmetric) within 3D EM raw data.

Return type

str

Returns

The path to the synapse type identification model.

property mpath_myelin: str

Retrieves the path to the model trained on identifying myelinated cell parts within 3D EM raw data.

Return type

str

Returns

Path to model trained on identifying myelinated cell parts within 3D EM raw data.

property mpath_spiness: str

Retrieves the path to the model trained on detecting spine head, neck, dendritic shaft, and other (soma and axon) via 2D projections for semantic segmentation.

Return type

str

Returns

The path to the spine detection model.

property mpath_syn_rfc: str

Retrieves the path to the Random Forest Classifier model for synapse classification.

Return type

str

Returns

The path to the synapse classification RFC model.

property mpath_syn_rfc_fallback: str

Path to rfc model created with sklearn==0.21.0 for synapse classification.

Return type

str

Returns

The path to the fallback synapse classification RFC model.

property mpath_syntype: str

Returns the path to the model trained on identifying synapse types (symmetric vs. asymmetric) within 3D EM raw data.

Return type

str

Returns

The path to the synapse type identification model.

property mpath_tnet: str

Retrieves the path to the tCMN model, an encoder network of local cell morphology trained via triplet loss.

Return type

str

Returns

Path to tCMN - an encoder network of local cell morphology trained via triplet loss.

property mpath_tnet_pts: str

Retrieves the path to the encoder network of local cell morphology trained via triplet loss on point data.

Return type

str

Returns

The path to the encoder network model for point data.

property mpath_tnet_pts_wholecell: str

Returns: Path to an encoder network of local cell morphology trained via triplet loss on point data.

Return type

str

property ncore_total: int

Calculates the total number of cores available for processing.

Return type

int

Returns

The total number of cores.

property neuron_svagg_list_path: str

Retrieves the path to the agglomeration list for neurons.

Return type

str

Returns

Path to agglomeration list (list of SV IDs for every cell).

property neuron_svgraph_path: str

Retrieves the path to the neuron supervoxel graph.

Return type

str

Returns

The path to the neuron supervoxel graph.

property ngpu_total: int

Calculates the total number of GPUs available for processing.

Return type

int

Returns

The total number of GPUs.

property prior_astrocyte_removal: bool

Determines whether an astrocyte separation procedure should be initiated to create a astrocyte-separated RAG (see glia/neuron_svgraph.bz2 and glia/astrocyte_svgraph.bz2).

Return type

bool

Returns

Value stored in config.yml.

property pruned_svagg_list_path: str

Pruned SV lists. All cells or cell fragments with bounding box diagonal of less than global_params.config[‘min_cc_size_ssv’] are filtered.

Return type

str

Returns

Path to pruned agglomeration list (list of SV IDs for every cell) after size filtering.

property pruned_svgraph_path: str

Retrieves the path to the pruned supervoxel graph after size filtering. It excludes cells or cell fragments with a bounding box diagonal smaller than the threshold defined by global_params.config['min_cc_size_ssv'].

Return type

str

Returns

The path to the pruned supervoxel graph after size filtering.

property qsub_work_folder: str

Retrieves the directory where intermediate batchjob results are stored.

Return type

str

Returns

Path to the batchjob work directory.

property sym_label: Optional[int]

Retrieves the label used for symmetric synapses.

Return type

Optional[int]

Returns

The label for symmetric synapses, if available.

property syntype_available: bool

Determines whether synaptic types are available as a KnossosDataset.

Return type

bool

Returns

Value stored at the config.yml file indicating if synaptic types are available for use in matrix generation.

property temp_path: str

Retrieves the path to the temporary directory used to store data caches.

Return type

str

Returns

Path to temporary directory used to store data caches.

property use_kimimaro: bool

Controls if skeletons should be generated with kimimaro.

Return type

bool

Returns

The value stored in the config.yml file, indicating if the kimimaro algorithm is to be used for skeleton generation.

property use_new_meshing: bool

Determines whether to use new, dense meshing computed distributed on 3D sub-cubes. If False, meshes are computed sparsely, i.e., per object/supervoxel.

Return type

bool

Returns

A boolean indicating if new meshing should be used, as stored in the config.yml file.

property use_new_renderings_locs: bool

Determines whether to use new rendering locations for faster computation and closer proximity to the neuron surface.

Return type

bool

Returns

Value stored at the config.yml file.

property use_new_subfold: bool

Determines whether to use a new subfolder hierarchy for storing objects with similar IDs.

Return type

bool

Returns

Value stored in config.yml indicating if the new subfolder hierarchy should be used.

property use_onthefly_views: bool

Determines whether to generate views for cell type prediction on the fly.

Return type

bool

Returns

A boolean indicating if on-the-fly view generation should be used for cell type prediction, as configured in the config.yml file.

property use_point_models: bool

Determines whether to use point cloud based models instead of multi-views.

Return type

bool

Returns

Value stored at the config.yml file indicating if point cloud based models should be used.

property working_dir

Retrieves the current working directory, checking for updates before returning.

Returns

Path to the current working directory.

syconn.handler.config.generate_default_conf(working_dir, scaling, syntype_avail=True, use_new_renderings_locs=True, kd_seg=None, kd_sym=None, kd_asym=None, kd_sj=None, kd_mi=None, kd_vc=None, kd_er=None, kd_golgi=None, init_svgraph_path='', prior_astrocyte_removal=True, use_new_meshing=True, allow_mesh_gen_cells=True, use_new_subfold=True, force_overwrite=False, key_value_pairs=None)

Generates the default SyConn configuration file, including paths to KnossosDatasets of e.g. cellular organelle predictions/probability maps and the cell supervoxel segmentation, general settings for OpenGL (egl vs osmesa), the scheduling system (SLURM vs QSUB vs None), and various parameters for processing the data. See SyConn/scripts/example_run/start.py for an example. init_svgraph_path can be set specifically in the config-file which is optional. By default, it is set to init_svgraph_path = working_dir + “rag.bz2”. SyConn then will require an edge list of the supervoxel graph, see also SyConn/scripts/example_run/start.py. Writes the file config.yml to working_dir after adapting the attributes as given by the method input. This file can also only contain the values of attributes which should differ from the default config at SyConn/syconn/handlers/config.yml. SyConn refers to the latter if a parameter cannot be found in the config file inside the currently active working directory.

The default config content is located at SyConn/syconn/handler/config.yml

Parameters

• working_dir (str) – Folder of the working directory.

• scaling (Union[Tuple, ndarray]) – Voxel size in NM.

• syntype_avail (bool) – If True, synapse objects will contain additional type property (symmetric vs asymmetric).

• use_new_renderings_locs (bool) – If True, uses new heuristic for generating rendering locations.

• kd_seg (Optional[str]) – Path to the KnossosDataset which contains the cell segmentation.

• kd_sym (Optional[str]) – Path to the symmetric type prediction.

• kd_asym (Optional[str]) – Path to the asymmetric type prediction.

• kd_sj (Optional[str]) – Path to the synaptic junction predictions.

• kd_mi (Optional[str]) – Path to the mitochondria predictions.

• kd_vc (Optional[str]) – Path to the vesicle cloud predictions.

• kd_er (Optional[str]) – Path to the ER predictions.

• kd_golgi (Optional[str]) – Path to the Golgi-Apparatus predictions.

• init_svgraph_path (str) – Path to the initial supervoxel graph.

• prior_astrocyte_removal (bool) – If True, applies astrocyte separation before analyzing cell reconstructions.

• use_new_meshing (bool) – If True, uses new meshing procedure based on zmesh.

• allow_mesh_gen_cells (bool) – If True, meshing of cell supervoxels will be permitted.

• use_new_subfold (bool) – If True, similar object IDs will be stored in the same storage file.

• force_overwrite – Will overwrite existing config.yml file.

• key_value_pairs (Optional[List[tuple]]) – List of key-value pairs used to modify attributes in the config file.

syconn.handler.config.initialize_logging(log_name, log_dir=None, overwrite=True)

Initializes and configures a logger with the given name. If a log directory is provided, it will create a file handler and ignore the disable_file_logging state from the global configuration. For import processing steps, individual loggers can be defined (e.g., proc, reps).

Parameters

• log_name (str) – Name of the logger to be initialized.

• log_dir (Optional[str]) – Specific directory for log files. If provided, a file handler is created regardless of the global configuration.

• overwrite (bool) – If True, the existing log file will be overwritten.

Returns

A configured logger instance.

syconn.handler.logger module