syconn.handler package

syconn.handler.basics module

class syconn.handler.basics.DelayedInterrupt(signals)[source]

Bases: object

syconn.handler.basics.chunkify(lst, n)[source]

Splits list into np.min([n, len(lst)]) sub-lists.

Parameters

  • lst (Union[list, ndarray]) –

  • n (int) –

Examples

>>> chunkify(np.arange(10), 2)
>>> chunkify(np.arange(10), 100)
Return type

List[list]

Returns

List of chunks. Length is np.min([n, len(lst)]).

syconn.handler.basics.chunkify_successive(l, n)[source]

Yield successive n-sized chunks from l.

syconn.handler.basics.chunkify_weighted(lst, n, weights)[source]

splits list into n sub-lists according to weights.

Parameters

  • lst – list

  • n – int

  • weights – array

Returns:

syconn.handler.basics.convert_keys_byte2str(dc)[source]
syconn.handler.basics.coordpath2anno(coords, scaling=None, add_edges=True)[source]

Creates skeleton from scaled coordinates, assume coords are in order for edge creation.

Parameters

  • coords – np.array scaled cooridnates

  • scaling – tuple

  • add_edges – bool

Returns: SkeletonAnnotation

syconn.handler.basics.crop_bool_array(arr)[source]

Crops a bool array to its True region

Parameters

arr – 3d bool array array to crop

Returns: d bool array, list

cropped array, offset

syconn.handler.basics.data2kzip(kzip_path, fpaths, fnames_in_zip=None, force_overwrite=True, verbose=False)[source]

Write files to k.zip. Finally removes files at fpaths.

Parameters

  • kzip_path (str) – str

  • fpaths – List[str]

  • fnames_in_zip – List[str] name of file when added to zip

  • force_overwrite – bool

  • verbose – bool

Returns:

syconn.handler.basics.flatten(x)[source]

Replacement for compiler.ast.flatten - this performs recursive flattening in comparison to the function above. Public domain code: https://stackoverflow.com/questions/16176742/ python-3-replacement-for-deprecated-compiler-ast-flatten-function

Parameters

x –

Returns: flattend x

syconn.handler.basics.flatten_list(lst)[source]

Flattens list of lists. Same ordering as np.concatenate

Parameters

lst – list of lists

Returns: list

syconn.handler.basics.get_filepaths_from_dir(directory, ending=('k.zip',), recursively=False, exclude_endings=False, fname_includes=())[source]

Collect all files with certain ending from directory.

Parameters

  • directory – str path to lookup directory

  • ending – tuple/list/str ending(s) of files

  • recursively – boolean add files from subdirectories

  • exclude_endings – bool filenames with endings defined in endings will not be added

  • fname_includes – str or list file names with this substring(s) will be added

Returns: list of str

paths to files

syconn.handler.basics.get_paths_of_skelID(id_list, traced_skel_dir)[source]

Gather paths to kzip of skeletons with ID in id_list

Parameters

  • id_list – list of str skeleton ID’s

  • traced_skel_dir – str directory of mapped skeletons

Returns: list of str

paths of skeletons in id_list

syconn.handler.basics.get_skelID_from_path(skel_path)[source]

Parse skeleton ID from filename.

Parameters

skel_path – str path to skeleton

Returns: int

skeleton ID

syconn.handler.basics.group_ids_to_so_storage(ids, params, significant_digits=5)[source]
syconn.handler.basics.kd_factory(kd_path, channel='jpg')[source]

Initializes a KnossosDataset at the given kd_path.

Notes

  • Prioritizes pyk.conf files.

Parameters

  • kd_path (str) – Path to the KnossosDataset.

  • channel (str) – Channel which to use. Currently not used.

Returns:

syconn.handler.basics.load_pkl2obj(path)[source]

Loads pickle file of object

Parameters

path – str path of source file

Returns:

syconn.handler.basics.majority_element_1d(arr)[source]

Returns most frequent element in ‘arr’.

Parameters

arr – np.array

Returns: scalar

syconn.handler.basics.parse_cc_dict_from_g(g)[source]
syconn.handler.basics.parse_cc_dict_from_kml(kml_path)[source]

Parse connected components from knossos mergelist text file

Parameters

kml_path – str

Returns: dict

syconn.handler.basics.parse_cc_dict_from_kzip(k_path)[source]
Parameters

k_path – str

Returns: dict

syconn.handler.basics.prase_cc_dict_from_txt(txt)[source]

Parse connected components from knossos mergelist text file

Parameters

txt – str or bytes

Returns: dict

syconn.handler.basics.read_mesh_from_zip(zip_fname, fname_in_zip)[source]

Read ply file from zip. Currently does not support normals!

Parameters

  • zip_fname – str

  • fname_in_zip – str

Returns: np.array, np.array, np.array

syconn.handler.basics.read_meshes_from_zip(zip_fname, fnames_in_zip)[source]

Read ply files from zip. Currently does not support normals!

Parameters

  • zip_fname – str

  • fnames_in_zip – str

Returns: np.array, np.array, np.array

syconn.handler.basics.read_txt_from_zip(zip_fname, fname_in_zip)[source]

Read text file from zip.

Parameters

  • zip_fname – str

  • fname_in_zip – str

Returns: bytes

syconn.handler.basics.remove_from_zip(zipfname, *filenames)[source]

Removes filenames from zipfile

Parameters

  • zipfname – str Path to zipfile

  • *filenames – list of str files to delete

Returns:

syconn.handler.basics.safe_copy(src, dest, safe=True)[source]

Copies file and throws exception if destination exists. Taken from Misandrist on Stackoverflow (03/31/17).

Parameters

  • src – str path to source file

  • dest – str path to destination file

  • safe – bool If False, copies file with replacement

Returns:

syconn.handler.basics.str_delta_sec(seconds)[source]

String time formatting - omits time units which are zero.

Examples

>>> sec = 2 * 24 * 3600 + 12 * 3600 + 5 * 60 + 1
>>> str_rep = str_delta_sec(sec)
>>> assert str_rep == '2d:12h:05min:01s'
>>> assert str_delta_sec(4 * 3600 + 20 * 60 + 10) == '4h:20min:10s'
Parameters

seconds (int) – Number of seconds, e.g. result of a time delta.

Return type

str

Returns

String representation, e.g. '2d:12h:05min:01s' for sec = 1 + 5 * 60 + 12 * 3600 + 2 * 24 * 3600.

syconn.handler.basics.switch_array_entries(this_array, entries)[source]
syconn.handler.basics.temp_seed(seed)[source]

From https://stackoverflow.com/questions/49555991/can-i-create-a-local-numpy-random-seed

Parameters

seed –

Returns:

syconn.handler.basics.texts2kzip(kzip_path, texts, fnames_in_zip, force_overwrite=False)[source]

Write strings to files in k.zip.

Parameters

  • kzip_path – str

  • texts – List[str]

  • fnames_in_zip – List[str] name of file when added to zip

  • force_overwrite – bool

Returns:

syconn.handler.basics.write_data2kzip(kzip_path, fpath, fname_in_zip=None, force_overwrite=False)[source]

Write file to k.zip.

Parameters

  • kzip_path – str

  • fpath – str

  • fname_in_zip – str name of file when added to zip

  • force_overwrite – bool

Returns:

syconn.handler.basics.write_obj2pkl(path, objects)[source]

Writes object to pickle file

Parameters

  • path – str Destination.

  • objects – object

Returns:

syconn.handler.basics.write_txt2kzip(kzip_path, text, fname_in_zip, force_overwrite=False)[source]

Write string to file in k.zip.

Parameters

  • kzip_path – str

  • text – str or bytes

  • fname_in_zip – str name of file when added to zip

  • force_overwrite – bool

Returns:

syconn.handler.compression module

syconn.handler.compression.arrtolz4string(arr)[source]

Converts (multi-dimensional) array to list of lz4 compressed strings.

Parameters

arr (ndarray) – Input array.

Return type

bytes

Returns

lz4 compressed string.

syconn.handler.compression.arrtolz4string_list(arr)[source]

Converts (multi-dimensional) array to list of lz4 compressed strings.

Parameters

arr (ndarray) – Input array.

Return type

List[bytes]

Returns

lz4 compressed string.

syconn.handler.compression.load_from_h5py(path, hdf5_names=None, as_dict=False)[source]

Loads data from a h5py File.

Parameters

  • path (str) – Path to .h5 file.

  • hdf5_names (Optional[Iterable[str]]) – If None, all keys will be loaded.

  • as_dict (bool) – If True, returns a dictionary.

Return type

Union[Dict[str, ndarray], List[ndarray]]

Returns

The data stored at path either as list of arrays (ordering as hdf5_names) or as dictionary.

syconn.handler.compression.load_lz4_compressed(p, shape=(-1, 20, 2, 128, 256), dtype=<class 'numpy.float32'>)[source]

Shape must be known in order to load (multi-dimensional) array from binary string. Due to overflow in python2 added recursive loading.

Parameters

  • p (str) – path to lz4 file

  • shape (Tuple[int]) – tuple

  • dtype (dtype) – type

Returns: np.array

syconn.handler.compression.lz4string_listtoarr(str_lst, dtype=<class 'numpy.float32'>, shape=None)[source]

Converts lz4 compressed strings to array.

Parameters

  • str_lst (Union[List[bytes], ndarray]) – Binary string representation of the array. If numpy array, do nothing.

  • dtype (dtype) – Data type of the serialized array.

  • shape (Optional[Tuple[int]]) – Shape of the serialized array.

Return type

ndarray

Returns

1d numpy array.

syconn.handler.compression.lz4stringtoarr(string, dtype=<class 'numpy.float32'>, shape=None)[source]

Converts lz4 compressed string to 1d array.

Parameters

  • string (bytes) – Serialized array.

  • dtype (dtype) – Data type of original array.

  • shape (Optional[Tuple[int]]) – Shape of original array.

Returns

N-dimensional numpy array.

syconn.handler.compression.save_lz4_compressed(p, arr, dtype=<class 'numpy.float32'>)[source]

Saves array as lz4 compressed string. Due to overflow in python2 added error handling by recursive splitting.

Parameters

  • p (str) – Path to the destination file.

  • arr (ndarray) – Numpy array.

  • dtype (dtype) – Data type in which the array should be stored.

syconn.handler.compression.save_to_h5py(data, path, hdf5_names=None, overwrite=False, compression=True)[source]

Saves data to h5py File.

Parameters

  • data (Union[Dict[str, ndarray], List[ndarray]]) – If list, hdf5_names has to be set.

  • path (str) – Forward-slash separated path to file.

  • hdf5_names (Optional[List[str]]) – Keys used to store arrays in data. Has to be the same length as data.

  • overwrite (bool) – Determines whether existing files are overwritten.

  • compression (bool) – If True, compression='gzip' is used which is recommended for sparse and ordered data.

syconn.handler.prediction module

syconn.handler.prediction.binarize_labels(labels, foreground_ids, target_labels=None)[source]

Transforms label array to binary label array (0=background, 1=foreground) or to the labels provided in target_labels by mapping the foreground IDs accordingly.

Parameters

  • labels (ndarray) – np.array

  • foreground_ids (Iterable[int]) – iterable

  • target_labels (Optional[Iterable[int]]) – Iterable labels used for mapping foreground IDs.

Returns: np.array

syconn.handler.prediction.certainty_estimate(inp, is_logit=False)[source]
Estimates the certainty of (independent) predictions of the same sample:

  1. If is_logit is True, Generate pseudo-probabilities from the input using softmax.

  2. Sum the evidence per class and (re-)normalize.

  3. Compute the entropy, scale it with the maximum entropy (equal probabilities) and subtract it from 1.

Parameters

  • inp (ndarray) – 2D array of prediction results (N: number of samples, C: Number of classes)

  • is_logit (bool) – If True, applies softmax(inp, axis=1).

Return type

float

Returns

Certainty measure based on the entropy of a set of (independent) predictions.

syconn.handler.prediction.chunk_pred(ch, model, debug=False)[source]

Helper function to write chunks.

Parameters

  • ch (chunky.Chunk) – Chunk

  • model (torch.nn.Module) – str or model object

  • debug (bool) – bool

Returns:

syconn.handler.prediction.create_h5_from_kzip(zip_fname, kd_p, foreground_ids=None, overwrite=True, raw_data_offset=75, debug=False, mag=1, squeeze_data=True, target_labels=None, apply_mops_seg=None)[source]

Create .h5 files for elektronn3 (zyx) input. Only supports binary labels (0=background, 1=foreground).

Examples

Suppose your k.zip file contains the segmentation GT with two segmentation IDs 1, 2 and is stored at kzip_fname. The corresponding KnossosDataset is located at kd_path . The following code snippet will create an .h5 file in the folder of kzip_fname with the raw data (additional offset controlled by raw_data_offset) and the label data (either binary or defined by target_labels) with the keys raw and label respectively:

create_h5_from_kzip(d_p=kd_path, raw_data_offset=75,
zip_fname=kzip_fname, mag=1, foreground_ids=[1, 2],
target_labels=[1, 2])
Parameters

  • zip_fname (str) – Path to the annotated kzip file.

  • kd_p (str) – Path to the underlying raw data stored as KnossosDataset.

  • foreground_ids (Optional[Iterable[int]]) – IDs which have to be converted to foreground, i.e. 1. Everything else is considered background (0). If None, everything except 0 is treated as foreground.

  • overwrite (bool) – If True, will overwrite existing .h5 files

  • raw_data_offset (int) – Number of voxels used for additional raw offset, i.e. the offset for the raw data will be label_offset - raw_data_offset, while the raw data volume will be label_volume + 2*raw_data_offset. It will use ‘kd.scaling’ to account for dataset anisotropy if scalar or a list of length 3 hast to be provided for a custom x, y, z offset.

  • debug (bool) – If True, file will have an additional ‘debug’ suffix and raw_data_offset is set to 0. Also their bit depths are adatped to be the same.

  • mag (int) – Data mag. level.

  • squeeze_data (int) – If True, label and raw data will be squeezed.

  • target_labels (Optional[Iterable[int]]) – If set, foreground_ids must also be set. Each ID in foreground_ids will be mapped to the corresponding label in target_labels.

  • apply_mops_seg (Optional[List[str]]) – List of string identifiers for ndimage morphological operations.

syconn.handler.prediction.create_h5_gt_file(fname, raw, label, foreground_ids=None, target_labels=None, debug=False, apply_mops_seg=None)[source]

Create .h5 files for ELEKTRONN input from two arrays. Only supports binary labels (0=background, 1=foreground). E.g. for creating true negative cubes set foreground_ids=[] to be an empty list. If set to None, everything except 0 is treated as foreground.

Parameters

  • fname (str) – str Path where h5 file should be saved

  • raw (ndarray) – np.array

  • label (ndarray) – np.array

  • foreground_ids (Optional[Iterable[int]]) – iterable ids which have to be converted to foreground, i.e. 1. Everything else is considered background (0). If None, everything except 0 is treated as foreground.

  • target_labels (Optional[Iterable[int]]) – Iterable If set, foreground_ids must also be set. Each ID in foreground_ids will be mapped to the corresponding label in target_labels.

  • debug (bool) – bool will store labels and raw as uint8 ranging from 0 to 255

  • apply_mops_seg (Optional[List[str]]) – List of string identifiers for ndimage morphological operations.

syconn.handler.prediction.dense_predicton_helper(raw, predictor, is_zyx=False, return_zyx=False)[source]
Parameters

  • raw (ndarray) – The input data array in CXYZ.

  • predictor (Predictor) – The model which performs the inference. Requires predictor.predict.

  • is_zyx –

  • return_zyx –

Return type

ndarray

Returns

The inference result in CXYZ as uint8 between 0..255.

syconn.handler.prediction.dense_predictor(args)[source]

Volumes are transformed by XYZ <-> ZYX before they are passed to the model.

Parameters

args –

Tuple( chunk_ids: list

list of chunks in chunk dataset

kd_pstr

path to knossos dataset .conf file

cd_pstr

destination folder for chunk dataset containing prediction

model_pstr

path to model

offset : chunk_size: … )

Returns:

syconn.handler.prediction.get_celltype_model_e3()[source]

Those networks are typically trained with naive_view_normalization_new Unlike the other e3 InferenceModel instances, here the view normalization is applied in the downstream inference method (predict_sso_celltype)

because the celltype model also gets scalar values as input which should not be normalized.

syconn.handler.prediction.get_glia_model_e3()[source]

Those networks are typically trained with naive_view_normalization_new

syconn.handler.prediction.get_knn_tnet_embedding_e3()[source]

OUTDATED

syconn.handler.prediction.get_myelin_cnn()[source]

elektronn3 model trained to predict binary myelin-in class.

Returns

The trained Inference model.

syconn.handler.prediction.get_pca_tnet_embedding_e3()[source]

OUTDATED

syconn.handler.prediction.get_semseg_axon_model()[source]
syconn.handler.prediction.get_semseg_spiness_model()[source]
syconn.handler.prediction.get_tripletnet_model_e3()[source]

Those networks are typically trained with naive_view_normalization_new

syconn.handler.prediction.int2str_converter(label, gt_type)[source]

TODO: remove redundant definitions. Converts integer label into semantic string.

Parameters

  • label (int) – int

  • gt_type (str) – str e.g. spgt for spines, axgt for cell compartments or ctgt for cell type

Returns: str

Return type

str

syconn.handler.prediction.knn_clf_tnet_embedding(fold, fit_all=False)[source]

Currently it assumes embedding for GT views has been created already in ‘fold’ and put into l_train_%d.npy / l_valid_%d.npy files.

Parameters

  • fold – str

  • fit_all – bool

Returns:

syconn.handler.prediction.load_gt_from_kzip(zip_fname, kd_p, raw_data_offset=75, verbose=False, mag=1)[source]

Loads ground truth from zip file, generated with Knossos. Corresponding dataset config file is located at kd_p.

Parameters

  • zip_fname – str

  • kd_p – str or List[str]

  • raw_data_offset – int or np.array number of voxels used for additional raw offset, i.e. the offset for the raw data will be label_offset - raw_data_offset, while the raw data volume will be label_volume + 2*raw_data_offset. It will use ‘kd.scaling’ to account for dataset anisotropy if scalar or a list of length 3 hast to be provided for a custom x, y, z offset.

  • verbose – bool

  • mag – int Data mag. level.

Returns: np.array, np.array

raw data (float32) (multiplied with 1/255.), label data (uint16)

syconn.handler.prediction.naive_view_normalization(d)[source]
syconn.handler.prediction.naive_view_normalization_new(d)[source]
syconn.handler.prediction.overlaycubes2kzip(dest_p, vol, offset, kd_path)[source]

Writes segmentation volume to kzip.

Parameters

  • dest_p (str) – str path to k.zip

  • vol (ndarray) – np.array Segmentation or prediction (unsigned integer, XYZ).

  • offset (ndarray) – np.array

  • kd_path (str) – str

Returns: np.array [Z, X, Y]

syconn.handler.prediction.parse_movement_area_from_zip(zip_fname)[source]

Parse MovementArea (e.g. bounding box of labeled volume) from annotation.xml in (k.)zip file.

Parameters

zip_fname (str) – str

Returns: np.array

Movement Area [2, 3]

Return type

ndarray

syconn.handler.prediction.pca_tnet_embedding(fold, n_components=3, fit_all=False)[source]

Currently it assumes embedding for GT views has been created already in ‘fold’ and put into l_train_%d.npy / l_valid_%d.npy files.

Parameters

  • fold – str

  • n_components – int

  • fit_all – bool

Returns:

syconn.handler.prediction.pred_dataset(*args, **kwargs)[source]
syconn.handler.prediction.predict_dense_to_kd(kd_path, target_path, model_path, n_channel, target_names=None, target_channels=None, channel_thresholds=None, log=None, mag=1, overlap_shape_tiles=(40, 40, 20), cube_of_interest=None, overwrite=False, cube_shape_kd=None)[source]

Helper function for dense dataset prediction. Runs predictions on the whole knossos dataset located at kd_path. Prediction results will be written to KnossosDatasets called target_names at target_path. If no threshold and only one channel per target_names is given, the resulting KnossosDataset will contain a probability map in the raw channel as uint8 (0..255). Otherwise the classification results will be written to the overlay channel.

Notes

  • Has a high GPU memory requirement (minimum 12GB). Does should be controllable from the config or determined automatically.

  • Resulting KnossosDatasets currently do not use pyknossos confs.

Parameters

  • kd_path (str) – Path to KnossosDataset .conf file of the raw data.

  • target_path (str) – Destination directory for the output KnossosDataset(s) which contain the prediction(s).

  • model_path (str) – Path to elektronn3 model for predictions. Loaded via the Predictor.

  • n_channel (int) – Number of channels predicted by the model.

  • target_names (Optional[Iterable[str]]) – Names of target knossos datasets, e.g. target_names=['synapse_fb', 'synapse_type']. Defaults to ['pred']. Length must match with target_channels.

  • target_channels (Optional[Iterable[Iterable[int]]]) – Channel_ids in prediction for each target knossos data set e.g. target_channels=[(1, 2)] if prediction has two foreground labels. Defaults to [[ix for ix in range(n_channel)]]. Length must match with target_names.

  • channel_thresholds (Optional[Iterable[Union[float, Any]]]) – Thresholds for channels: If None and number of channels for target kd is 1: probabilities are stored. Else: 0.5 as default e.g. channel_thresholds=[None,0.5,0.5].

  • log (Optional[Logger]) – Logger.

  • mag (int) – Data magnification level.

  • overlap_shape_tiles (Tuple[int, int, int]) –

    [WIP] Overlap in voxels [XYZ] used for each tile predicted during inference. Currently the following chunk/tile properties are used additionally (overlap_shape is the per-chunk overlap):

    chunk_size = np.array([1024, 1024, 256], dtype=np.int32)  # XYZ
n_tiles = np.array([4, 4, 16])
tile_shape = (chunk_size / n_tiles).astype(np.int32)
# the final input shape must be a multiple of tile_shape
overlap_shape = tile_shape // 2

  • cube_of_interest (Optional[Tuple[ndarray]]) – Bounding box of the volume of interest (minimum and maximum coordinate in voxels in the respective magnification (see kwarg mag).

  • overwrite (bool) – Overwrite existing KDs.

  • cube_shape_kd (Optional[Tuple[int]]) – Cube shape used to store sub-volumes in KnossosDataset on the file system.

syconn.handler.prediction.predict_h5(h5_path, m_path, clf_thresh=None, mfp_active=False, gpu_ix=0, imposed_patch_size=None, hdf5_data_key=None, data_is_zxy=True, dest_p=None, dest_hdf5_data_key='pred', as_uint8=True)[source]

Predicts data from h5 file. Assumes raw data is already float32.

Parameters

  • h5_path – str path to h5 containing the raw data

  • m_path – str path to predictive model

  • clf_thresh – float classification threshold, if None, no thresholding

  • mfp_active – False

  • gpu_ix – int

  • imposed_patch_size – tuple

  • hdf5_data_key – str if None, it uses the first entry in the list returned by ‘load_from_h5py’

  • data_is_zxy – bool if False, it will assumes data is [X, Y, Z]

  • dest_p – str

  • dest_hdf5_data_key – str

  • as_uint8 – bool

Returns:

syconn.handler.prediction.predict_kzip(kzip_p, m_path, kd_path, clf_thresh=0.5, mfp_active=False, dest_path=None, overwrite=False, gpu_ix=0, imposed_patch_size=None)[source]

Predicts data contained in k.zip file (defined by bounding box in knossos)

Parameters

  • kzip_p – str path to kzip containing the raw data cube information

  • m_path – str path to predictive model

  • kd_path – str path to knossos dataset

  • clf_thresh – float classification threshold

  • mfp_active – False

  • dest_path – str path to destination folder, if None folder of k.zip is used.

  • overwrite – bool

  • gpu_ix – int

  • imposed_patch_size – tuple

Returns:

syconn.handler.prediction.prediction_helper(raw, model, override_mfp=True, imposed_patch_size=None)[source]

Helper function for predicting raw volumes (range: 0 to 255; uint8). Will change X, Y, Z to ELEKTRONN format (Z, X, Y) and returns prediction in standard format [X, Y, Z]. Imposed patch size has to be given in Z, X, Y!

Parameters

  • raw – np.array volume [X, Y, Z]

  • model – str or model object path to model (.mdl)

  • override_mfp – bool

  • imposed_patch_size – tuple in Z, X, Y FORMAT!

Returns: np.array

prediction data [X, Y, Z]

syconn.handler.prediction.str2int_converter(comment, gt_type)[source]
Return type

int

syconn.handler.prediction.to_knossos_dataset(kd_p, kd_pred_p, cd_p, model_p, imposed_patch_size, mfp_active=False)[source]
Parameters

  • kd_p –

  • kd_pred_p –

  • cd_p –

  • model_p –

  • imposed_patch_size –

  • mfp_active –

Returns:

syconn.handler.prediction.views2tripletinput(views)[source]
syconn.handler.prediction.xyz2zxy(vol)[source]

Swaps axes to ELEKTRONN convention ([M, .., X, Y, Z] -> [M, .., Z, X, Y]).

Parameters

vol (ndarray) – np.array [M, .., X, Y, Z]

Returns: np.array [M, .., Z, X, Y]

Return type

ndarray

syconn.handler.prediction.xyz2zyx(vol)[source]

Swaps axes to ELEKTRONN convention ([M, .., X, Y, Z] -> [M, .., Z, X, Y]).

Parameters

vol (ndarray) – np.array [M, .., X, Y, Z]

Returns: np.array [M, .., Z, X, Y]

Return type

ndarray

syconn.handler.prediction.zxy2xyz(vol)[source]

Swaps axes to ELEKTRONN convention ([M, .., Z, X, Y] -> [M, .., X, Y, Z]).

Parameters

vol (ndarray) – np.array [M, .., Z, X, Y]

Returns: np.array [M, .., X, Y, Z]

Return type

ndarray

syconn.handler.prediction.zyx2xyz(vol)[source]

Swaps axes to ELEKTRONN convention ([M, .., Z, X, Y] -> [M, .., X, Y, Z]).

Parameters

vol (ndarray) – np.array [M, .., Z, X, Y]

Returns: np.array [M, .., X, Y, Z]

Return type

ndarray

syconn.handler.config module

class syconn.handler.config.DynConfig(wd=None, log=None, fix_config=False)[source]

Bases: syconn.handler.config.Config

Enables dynamic and SyConn-wide update of working directory ‘wd’ and provides an interface to all working directory dependent parameters.

Notes

  • Due to sync. checks it is favorable to not use __getitem__() inside loops.

Examples

To initialize a working directory at the beginning of your script, run:

from syconn import global_params
global_params.wd = '~/SyConn/example_cube1/'
cfg = global_params.config  # this is the `DynConfig` object
property allow_mesh_gen_cells: bool

If True, meshes are not provided for cell supervoxels and will be computed from scratch, see use_new_meshing.

Return type

bool

property allow_ssv_skel_gen: bool

Controls whether cell supervoxel skeletons are provided a priori or can be computed from scratch. Currently this is done via a naive sampling procedure.

Return type

bool

Returns

Value stored at the config.yml file.

property astrocyte_svagg_list_path: str

Astrocyte SV lists.

Return type

str

Returns

Path to agglomeration list (list of SV IDs for every cell).

astrocyte_svgraph_path()[source]

Astrocyte SV graph.

Return type

str

Returns

Path to neuron SV graph.

property asym_label: Optional[int]
Return type

Optional[int]

property batchjob_script_folder: str
Return type

str

property default_conf: syconn.handler.config.Config

Load default config.yml if necessary.

Return type

Config

property entries

Entries stored in the config.yml file.

Returns

All entries.

property init_svgraph_path: str

Returns: Path to initial RAG.

Return type

str

property kd_asym_path: str

Returns: Path to synaptic asym. type probability map stored as KnossosDataset.

Return type

str

property kd_er_path: str

Returns: Path to ER probability map or binary predictions stored as KnossosDataset.

Return type

str

property kd_golgi_path: str

Returns: Path to Golgi probability map or binary predictions stored as KnossosDataset.

Return type

str

property kd_mi_path: str

Returns: Path to mitochondria probability map or binary predictions stored as KnossosDataset.

Return type

str

property kd_organelle_seg_paths: Dict[str, str]

KDs of subcell. organelle segmentations.

Return type

Dict[str, str]

Returns

Dictionary containing the paths to KnossosDataset of available cellular organelles global_params.config['process_cell_organelles'].

property kd_organelles_paths: Dict[str, str]

KDs of subcell. organelle probability maps

Return type

Dict[str, str]

Returns

Dictionary containing the paths to KnossosDataset of available cellular containing global_params.config['process_cell_organelles'].

property kd_seg_path: str

Returns: Path to cell supervoxel segmentation KnossosDataset.

Return type

str

property kd_sj_path: str

Returns: Path to synaptic junction probability map or binary predictions stored as KnossosDataset.

Return type

str

property kd_sym_path: str

Returns: Path to synaptic sym. type probability map stored as KnossosDataset.

Return type

str

property kd_vc_path: str

Returns: Path to vesicle cloud probability map or binary predictions stored as KnossosDataset.

Return type

str

property model_dir: str

Returns: Path to model directory.

Return type

str

property mpath_axonsem: str

Returns: Path to model trained on detecting axon, terminal boutons and en-passant, dendrites and somata via 2D projections.

Return type

str

property mpath_celltype_e3: str

Returns: Path to model trained on prediction cell types from multi-view sets.

Return type

str

property mpath_celltype_pts: str

Returns: Path to model trained on prediction cell types from point data.

Return type

str

property mpath_compartment_pts: str

Returns: Path to model trained on detecting axon, terminal and en-passant boutons, dendritic shaft, spine head and neck, and soma from point data.

Return type

str

property mpath_er: str

Returns: Path to model trained on identifying cell parts occupied by ER within 3D EM raw data.

Return type

str

property mpath_glia_e3: str

Returns: Path to model trained to classify local 2D projections into glia vs. neuron (img2scalar).

Return type

str

property mpath_glia_pts: str

Returns: Path to point-based model trained to classify local 2D projections into glia vs. neuron.

Return type

str

property mpath_golgi: str

Returns: Path to model trained on identifying cell parts occupied by Golgi Apparatus within 3D EM raw data.

Return type

str

property mpath_mivcsj: str

Returns: Path to model trained on identifying synapse types (symmetric vs. asymmetric) within 3D EM raw data.

Return type

str

property mpath_myelin: str

Returns: Path to model trained on identifying myelinated cell parts within 3D EM raw data.

Return type

str

property mpath_spiness: str

Returns: Path to model trained on detecting spine head, neck, dendritic shaft, and other (soma and axon) via 2D projections (-> semantic segmentation).

Return type

str

property mpath_syn_rfc: str
Return type

str

property mpath_syn_rfc_fallback: str

Path to rfc model created with sklearn==0.21.0

Return type

str

property mpath_syntype: str

Returns: Path to model trained on identifying synapse types (symmetric vs. asymmetric) within 3D EM raw data.

Return type

str

property mpath_tnet: str

Returns: Path to tCMN - an encoder network of local cell morphology trained via triplet loss.

Return type

str

property mpath_tnet_pts: str

Returns: Path to an encoder network of local cell morphology trained via triplet loss on point data.

Return type

str

property mpath_tnet_pts_wholecell: str

Returns: Path to an encoder network of local cell morphology trained via triplet loss on point data.

Return type

str

property ncore_total: int
Return type

int

property neuron_svagg_list_path: str

Neuron SV lists.

Return type

str

Returns

Path to agglomeration list (list of SV IDs for every cell).

property neuron_svgraph_path: str

Neuron SV graph.

Return type

str

Returns

Path to neuron SV graph.

property ngpu_total: int
Return type

int

property prior_astrocyte_removal: bool

If True astrocyte separation procedure will be initiated to create a astrocyte-separated RAG (see glia/neuron_svgraph.bz2 and glia/astrocyte_svgraph.bz2).

Return type

bool

Returns

Value stored in config.yml.

property pruned_svagg_list_path: str

Pruned SV lists. All cells or cell fragments with bounding box diagonal of less than global_params.config['min_cc_size_ssv'] are filtered.

Return type

str

Returns

Path to pruned agglomeration list (list of SV IDs for every cell) after size filtering.

property pruned_svgraph_path: str

Pruned SV graph. All cells or cell fragments with bounding box diagonal of less than global_params.config['min_cc_size_ssv'] are filtered.

Return type

str

Returns

Path to pruned SV graph after size filtering.

property qsub_work_folder: str

Directory where intermediate batchjob results are stored.

Return type

str

Returns

Path to directory.

property sym_label: Optional[int]
Return type

Optional[int]

property syntype_available: bool

Synaptic types are available as KnossosDataset. Will be used during the matrix generation.

Return type

bool

Returns

Value stored at the config.yml file.

property temp_path: str

Returns: Path to temporary directory used to store data caches.

Return type

str

property use_kimimaro: bool

Controls if skeletons should be generated with kimimaro Returns: value stores in config.yml file

Return type

bool

property use_new_meshing: bool

Use new, dense meshing (zmesh) computed distributed on 3D sub-cubes. If False meshes are computed sparsely, i.e. per object/supervoxel.

Return type

bool

Returns

Value stored at the config.yml file.

property use_new_renderings_locs: bool

Use new rendering locations which are faster to computed and are located closer to the neuron surface.

Return type

bool

Returns

Value stored at the config.yml file.

property use_new_subfold: bool

Use new subfolder hierarchy where objects with similar IDs are stored in the same file.

Return type

bool

Returns

Value stored in config.yml.

property use_onthefly_views: bool

Generate views for cell type prediction on the fly.

Return type

bool

Returns

Value stored at the config.yml file.

property use_point_models: bool

Use point cloud based models instead of multi-views.

Return type

bool

Returns

Value stored at the config.yml file.

property working_dir

Returns: Path to working directory.

syconn.handler.config.generate_default_conf(working_dir, scaling, syntype_avail=True, use_new_renderings_locs=True, kd_seg=None, kd_sym=None, kd_asym=None, kd_sj=None, kd_mi=None, kd_vc=None, kd_er=None, kd_golgi=None, init_svgraph_path='', prior_astrocyte_removal=True, use_new_meshing=True, allow_mesh_gen_cells=True, use_new_subfold=True, force_overwrite=False, key_value_pairs=None)[source]

Generates the default SyConn configuration file, including paths to KnossosDatasets of e.g. cellular organelle predictions/prob. maps and the cell supervoxel segmentation, general settings for OpenGL (egl vs osmesa), the scheduling system (SLURM vs QSUB vs None) and various parameters for processing the data. See SyConn/scripts/example_run/start.py for an example. init_svgraph_path can be set specifically in the config-file which is optional. By default it is set to init_svgraph_path = working_dir + "rag.bz2". SyConn then will require an edge list of the supervoxel graph, see also SyConn/scripts/example_run/start.py. Writes the file config.yml to working_dir after adapting the attributes as given by the method input. This file can also only contain the values of attributes which should differ from the default config at SyConn/syconn/handlers/config.yml. SyConn refers to the latter if a parameter cannot be found in the config file inside the currently active working directory.

Examples

The default config content is located at SyConn/syconn/handler/config.yml

Parameters

  • working_dir (str) – Folder of the working directory.

  • scaling (Union[Tuple, ndarray]) – Voxel size in NM.

  • syntype_avail (bool) – If True, synapse objects will contain additional type property (symmetric vs asymmetric).

  • use_new_renderings_locs (bool) – If True, uses new heuristic for generating rendering locations.

  • kd_seg (Optional[str]) – Path to the KnossosDataset which contains the cell segmentation.

  • kd_sym (Optional[str]) – Path to the symmetric type prediction.

  • kd_asym (Optional[str]) – Path to the asymmetric type prediction.

  • kd_sj (Optional[str]) – Path to the synaptic junction predictions.

  • kd_mi (Optional[str]) – Path to the mitochondria predictions.

  • kd_vc (Optional[str]) – Path to the vesicle cloud predictions.

  • kd_er (Optional[str]) – Path to the ER predictions.

  • kd_golgi (Optional[str]) – Path to the Golgi-Apparatus predictions.

  • init_svgraph_path (str) – Path to the initial supervoxel graph.

  • prior_astrocyte_removal (bool) – If True, applies astrocyte separation before analysing cell reconstructions.

  • use_new_meshing (bool) – If True, uses new meshing procedure based on zmesh.

  • allow_mesh_gen_cells (bool) – If True, meshing of cell supervoxels will be permitted.

  • use_new_subfold (bool) – If True, similar object IDs will be stored in the same storage file.

  • force_overwrite – Will overwrite existing config.yml file.

  • key_value_pairs (Optional[List[tuple]]) – List of key-value pairs used to modify attributes in the config file.

syconn.handler.config.initialize_logging(log_name, log_dir=None, overwrite=True)[source]

Logger for each package module. For import processing steps individual logger can be defined (e.g. proc, reps).

Parameters

  • log_name (str) – Name of the logger.

  • log_dir (Optional[str]) – Set log_dir specifically. Will then create a filehandler and ignore the state of global_params.config['disable_file_logging'] state.

  • overwrite (bool) – Overwrite previous log file.

Returns

The logger.

syconn.handler.logger module