syconn.exec package

Submodules

• syconn.exec.exec_init module

• syconn.exec.exec_inference module

• syconn.exec.exec_skeleton module

• syconn.exec.exec_syns module

syconn.exec.exec_init module

syconn.exec.exec_init.init_cell_subcell_sds(chunk_size=None, n_folders_fs=10000, n_folders_fs_sc=10000, max_n_jobs=None, load_cellorganelles_from_kd_overlaycubes=False, transf_func_kd_overlay=None, cube_of_interest_bb=None, overwrite=False)

Convert binary class segmentation mask of sub-cellular structure predictions into an isntance segmentation using connected components / watershed. Subsequently, the properties of sub-cellular structures (voxel count, coordinate, bounding box, mesh, ..) and their associations with cell fragments (calculating the overlap between every sub-cellular structure and cell fragment instance) are extracted.

Parameters

• chunk_size (Optional[Tuple[int, int, int]]) – Size of the cube which are processed by each worker.

• n_folders_fs (int) – Number of folders used to create the folder structure in the resulting SegmentationDataset for the cell supervoxels (version='sv').

• n_folders_fs_sc (int) – Number of folders used to create the folder structure in the resulting SegmentationDataset for the cell organelle supervxeols (e.g. version='mi').

• max_n_jobs (Optional[int]) – Number of parallel jobs.

• load_cellorganelles_from_kd_overlaycubes (bool) –

• transf_func_kd_overlay (Optional[Dict[Any, Callable]]) – Transformation applied on the prob. map or segmentation data.

• cube_of_interest_bb (Union[tuple, ndarray, None]) – Bounding of the (sub-) volume of the dataset which is processed (minimum and maximum coordinates in mag1 voxels, XYZ).

• overwrite – If True, will overwrite existing data.

syconn.exec.exec_init.kd_init(co, chunk_size, transf_func_kd_overlay, load_cellorganelles_from_kd_overlaycubes, cube_of_interest_bb, log)

Replaced by a single call of generate_subcell_kd_from_proba().

Initializes a per-object segmentation KnossosDataset for the given supervoxel type co based on an initial prediction which location has to be defined in the config.yml file for the co object, e.g. 'kd_mi' for co='mi' (see generate_default_conf()). Results will be stored as a KnossosDataset at “{}/knossosdatasets/{}_seg/”.format(global_params.config.working_dir, co). Appropriate parameters have to be set inside the config.yml file, see generate_subcell_kd_from_proba() or generate_default_conf() for more details.

Examples

Was used to process sub-cellular structures independently:

ps = [Process(target=kd_init, args=[co, chunk_size, transf_func_kd_overlay,

load_cellorganelles_from_kd_overlaycubes, cube_of_interest_bb, log]) for co in global_params.config[‘process_cell_organelles’]]

for p in ps:

p.start() time.sleep(5)

for p in ps:

p.join() if p.exitcode != 0:

raise Exception(f’Worker {p.name} stopped unexpectedly with exit code {p.exitcode}.’)

Parameters

• co – Type of cell organelle supervoxels, e.g ‘mi’ for mitochondria or ‘vc’ for vesicle clouds.

• chunk_size – Size of the cube which are processed by each worker.

• transf_func_kd_overlay (Optional[Callable]) – Transformation applied on the prob. map or segmentation data.

• load_cellorganelles_from_kd_overlaycubes (bool) –

• cube_of_interest_bb (Tuple[ndarray]) – Bounding of the (sub-) volume of the dataset which is processed.

• log (Logger) – Logger.

syconn.exec.exec_init.run_create_neuron_ssd(apply_ssv_size_threshold=False, ncores_per_job=1, overwrite=False)

Creates a SuperSegmentationDataset with version=0 at the currently active working directory based on the SV graph at neuron_svgraph_path. In case astrocyte splitting is active, this will be the SV graph after astrocyte removal, if it was disabled it is identical to pruned_svgraph.bz2.

Parameters

• apply_ssv_size_threshold (bool) – Apply filter with minimum bounding box diagonal. This is usually not needed as the filter is applied either in run_create_rag() (prior_astrocyte_removal=False) or during the astrocyte separation.

• ncores_per_job (int) – Number of cores per worker for save_dataset_deep().

• overwrite (bool) –

Notes

• This is a memory intensiv step, consider increasing ncores_per_job.

• Requires init_cell_subcell_sds() and optionally run_astrocyte_splitting().

• Networkx requires a lot of memory for >1e9 edges, graph_tool and igraph are not usable for this either. Currently the work-around is to not use the graph information when storing the cell SV graph, but only the connected component as a graph with N-1 edges (N nodes). [TODO]

Returns:

syconn.exec.exec_init.run_create_rag(graph_node_dtype=None)

If global_params.config.prior_astrocyte_removal==True:

stores pruned RAG at global_params.config.pruned_svgraph_path, required for all glia removal steps. run_astrocyte_splitting() will finally store the neuron SV graph.

else:

stores pruned SV graph at pruned_svgraph_path, required by run_create_neuron_ssd().

Parameters

graph_node_dtype – Defaults to np.uint64.

syconn.exec.exec_init.sd_init(co, max_n_jobs, log=None)

Initialize SegmentationDataset of given supervoxel type co.

Parameters

• co (str) – Cellular organelle identifier (e.g. ‘mi’, ‘vc’, …).

• max_n_jobs (int) – Number of parallel jobs.

• log (Optional[Logger]) – Logger.

syconn.exec.exec_inference module

syconn.exec.exec_inference.run_astrocyte_prediction()

Predict astrocyte supervoxels based on the img2scalar CMN.

Notes

Requires init_cell_subcell_sds() and run_astrocyte_rendering().

syconn.exec.exec_inference.run_astrocyte_prediction_pts(max_n_jobs_gpu=None)

Predict astrocyte and neuron supervoxels with point cloud based convolutional networks.

Notes

• post-processing currently requires locking. In order to prevent locking, an additional map-reduce step is required to write the final probas of all SVs in a “per-storage” (per chunk attribute dict) fashion.

Parameters

max_n_jobs_gpu (Optional[int]) –

Notes

Requires init_cell_subcell_sds().

syconn.exec.exec_inference.run_astrocyte_splitting()

Uses the pruned RAG at global_params.config.pruned_svgraph_path (stored as edge list .bz2 file) which is computed in init_cell_subcell_sds() to split astrocyte fragments from neuron reconstructions and separate those and entire glial cells from the neuron supervoxel graph.

Stores neuron SV graph at neuron_svgraph_path which is then used by run_create_neuron_ssd().

Notes

Requires init_cell_subcell_sds(), run_astrocyte_rendering() and run_astrocyte_prediction().

syconn.exec.exec_inference.run_cell_embedding(max_n_jobs=None)

Infer cell embeddings for all neuron reconstructions base on triplet-loss trained cellular morphology learning network (tCMN). The point based model is trained with the pts_loader_scalar (used for celltypes). Multi-views functionality is not implemented.

Parameters

max_n_jobs (Optional[int]) – Number of parallel jobs.

Notes

Requires run_create_neuron_ssd(), run_neuron_rendering() and run_skeleton_generation().

syconn.exec.exec_inference.run_celltype_prediction(max_n_jobs_gpu=None)

Run the celltype inference based on the img2scalar CMN.

Parameters

max_n_jobs_gpu (Optional[int]) – Number of parallel GPU jobs.

Notes

Requires run_create_neuron_ssd() and run_neuron_rendering().

syconn.exec.exec_inference.run_morphology_embedding(max_n_jobs=None)

Infer local morphology embeddings for all neuron reconstructions base on triplet-loss trained cellular morphology learning network (tCMN). The point based model is trained with the pts_loader_scalar (used for celltypes)

Parameters

max_n_jobs (Optional[int]) – Number of parallel jobs.

Notes

Requires run_create_neuron_ssd(), run_neuron_rendering() and run_skeleton_generation().

syconn.exec.exec_inference.run_semsegaxoness_prediction(max_n_jobs_gpu=None)

Infer and map semantic segmentation of the 2D projections onto the cell reconstruction mesh (ssv.label_dict('vertex')) via semseg_of_sso_nocache. The following skeleton attributes are generated by semsegaxoness2skel and available in skeleton:

• “axoness”: Vertex predictions mapped to skeleton (see global_params.config['compartments']['map_properties_semsegax'].

• “axoness_avg10000”: Sliding window average along skeleton (10um traversal length).

• “axoness_avg10000_comp_maj”: Majority vote on connected components after removing the soma.

Parameters

max_n_jobs_gpu (Optional[int]) – Number of parallel GPU jobs.

syconn.exec.exec_inference.run_semsegspiness_prediction(max_n_jobs_gpu=None)

Will store semantic spine labels inside``ssv.label_dict(‘vertex’)[‘spiness]``.

Parameters

max_n_jobs_gpu (Optional[int]) – Number of parallel GPU jobs. Used for the inference.

syconn.exec.exec_skeleton module

syconn.exec.exec_skeleton.map_myelin_global(max_n_jobs=None)

Stand-alone myelin mapping to cell reconstruction skeletons. See kwarg map_myelin in run_skeleton_generation() for a mapping right after skeleton generation.

Parameters

max_n_jobs (Optional[int]) – Number of parallel jobs.

syconn.exec.exec_skeleton.run_kimimaro_skeletonization(max_n_jobs=None, map_myelin=None, cube_size=None, cube_of_interest_bb=None, ds=None, ncores_skelgen=2)

Generate the cell reconstruction skeletons with the kimimaro tool. functions are in proc.sekelton, GSUB_kimimaromerge, QSUB_kimimaroskelgen

Parameters

• max_n_jobs (Optional[int]) – Number of parallel jobs.

• map_myelin (Optional[bool]) – Map myelin predictions at every skeleton['nodes'] in skeleton.

• cube_size (Optional[ndarray]) – Cube size used within each worker. This should be as big as possible to prevent un-centered skeletons in cell compartments with big diameters. In mag 1 voxels.

• cube_of_interest_bb (Optional[tuple]) – Partial volume of the data set. Bounding box in mag 1 voxels: (lower coord, upper coord)

• ds (Optional[ndarray]) – Downsampling.

• ncores_skelgen (int) – Number of cores used during skeleton generation.

syconn.exec.exec_skeleton.run_skeleton_generation(cube_of_interest_bb=None, map_myelin=None, ncores_skelgen=2)

Parameters

• cube_of_interest_bb (Union[tuple, ndarray, None]) – Partial volume of the data set. Bounding box in mag 1 voxels: (lower coord, upper coord)

• map_myelin (Optional[bool]) – Map myelin predictions at every skeleton['nodes'] in skeleton.

• ncores_skelgen (int) – Number of cores used during skeleton generation.

syconn.exec.exec_skeleton.run_skeleton_generation_fallback(max_n_jobs=None, map_myelin=None)

Generate the cell reconstruction skeletons.

Parameters

• max_n_jobs (Optional[int]) – Number of parallel jobs.

• map_myelin (Optional[bool]) – Map myelin predictions at every skeleton['nodes'] in skeleton.

syconn.exec.exec_syns module

syconn.exec.exec_syns.run_cs_ssv_generation(n_folders_fs=10000, overwrite=False)

Create agglomerated contact site objects between cells. For this, ‘cs’ objects need to be extracted.

Parameters

• n_folders_fs (int) – Number of folders used to create the folder structure in each SegmentationDataset.

• overwrite (bool) –

syconn.exec.exec_syns.run_matrix_export()

Export the matrix as a .csv file at the connectivity_matrix folder of the currently active working directory. Also collects the following synapse properties from prior analysis steps:

• ‘partner_axoness’: Cell compartment type (axon: 1, dendrite: 0, soma: 2, en-passant bouton: 3, terminal bouton: 4) of the partner neurons.

• ‘partner_spiness’: Spine compartment predictions (0: dendritic shaft, 1: spine head, 2: spine neck, 3: other) of both neurons.

• ‘partner_spineheadvol’: Spinehead volume in µm^3 of pre- and post-synaptic partners.

• ‘partner_celltypes’: Celltype of the both neurons.

• ‘latent_morph’: Local morphology embeddings of the pre- and post- synaptic partners.

Examples

See SegmentationDataset for examples.

syconn.exec.exec_syns.run_spinehead_volume_calc()

Calculate spine head volumes based on a watershed segmentation which is run on 3D spine label masks propagated from cell surface predictions. Spine head volumes are stored in the SSV attribute dictionary with the key partner_spineheadvol in µm^3.

Subsequent call to collect_properties_from_ssv_partners() will add this property to the attribute dict of all syn_ssv. Calling syconn.proc.sd_proc.dataset_analysis() accordingly collects all syn_ssv properties and makes them available as numpy arrays. These two steps are performed in run_matrix_export().

syconn.exec.exec_syns.run_syn_generation(chunk_size=(512, 512, 512), n_folders_fs=10000, max_n_jobs=None, cube_of_interest_bb=None, overwrite=False, transf_func_sj_seg=None)

Run the synapse generation. Will create SegmentationDataset objects with the following versions:

• ‘cs’: Contact site objects between supervoxels.

• ‘syn’: Objects representing the overlap between ‘cs’ and the initial synaptic junction predictions. Note: These objects effectively represent synapse fragments between supervoxels.

• ‘syn_ssv’: Final synapse objects. Agglomerated ‘syn’ objects based on the supervoxel graph.

• ‘cs_ssv’: Final contact site objects. Agglomerated ‘cs’ objects based on the supervoxel graph.

  Only processed if ['cell_contacts']['generate_cs_ssv'] is set to True in the config.

Parameters

• chunk_size (Optional[Tuple[int, int, int]]) – The size of processed cubes.

• n_folders_fs (int) – Number of folders used to create the folder structure in each SegmentationDataset.

• max_n_jobs (Optional[int]) – Number of parallel jobs.

• cube_of_interest_bb (Union[tuple, ndarray, None]) – Defines the bounding box of the cube to process. By default this is set to (np.zoers(3); kd.boundary).

• overwrite (bool) –

• transf_func_sj_seg (Optional[Callable]) – Method that converts the cell organelle segmentation into a binary mask of background vs. sj foreground.