syconn.proc package

Submodules

• syconn.proc.general module

• syconn.proc.graphs module

• syconn.proc.image module

• syconn.proc.mapping module

• syconn.proc.meshes module

• syconn.proc.rendering module

• syconn.proc.sd_proc module

• syconn.proc.skel_based_classifier module

• syconn.proc.skel_based_classifier_helper module

• syconn.proc.ssd_proc module

• syconn.proc.stats module

syconn.proc.general module

syconn.proc.general.crop_bool_array(arr)

Crops a bool array to its True region

Parameters

arr – 3d bool array to crop

Returns

• 3d bool array

• list

• cropped array

• offset

syconn.proc.general.cut_array_in_one_dim(array, start, end, dim)

Cuts an array along a dimension

Parameters

• array (np.array) –

• start (int) –

• end (int) –

• dim (int) –

Returns

np.array

Return type

array

syconn.proc.general.dense_matrix(sv, edge_size)

Get dense matrix representation of coordinates

Parameters

• sv (np.array) –

• edge_size (int) –

Returns

np.array

syconn.proc.general.timeit(func)

syconn.proc.graphs module

syconn.proc.graphs.bfs_smoothing(vertices, vertex_labels, max_edge_length=120, n_voting=40)

Smooth vertex labels by applying a majority vote on a BFS subset of nodes for every node in the graph Parameters

Parameters

• vertices – np.array N, 3

• vertex_labels – np.array N, 1

• max_edge_length – float maximum distance between vertices to consider them connected in the graph

• n_voting – int Number of collected nodes during BFS used for majority vote

Returns: np.array

smoothed vertex labels

syconn.proc.graphs.chunkify_contiguous(l, n)

Yield successive n-sized chunks from l. https://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks

syconn.proc.graphs.coordpath2anno(coords, scaling=None)

Creates skeleton from scaled coordinates, assume coords are in order for edge creation.

Parameters

• coords (ndarray) – np.array

• scaling (Optional[ndarray]) – np.ndarray

Returns: SkeletonAnnotation

Return type

SkeletonAnnotation

syconn.proc.graphs.create_ccsize_dict(g, bbs, is_connected_components=False)

Calculate bounding box size of connected components.

Parameters

• g (Graph) – Supervoxel graph.

• bbs (dict) – Bounding boxes (physical units).

• is_connected_components (bool) – If graph g already is connected components. If False, nx.connected_components is applied.

Return type

dict

Returns

Look-up which stores the connected component bounding box for every single node in the input Graph g.

syconn.proc.graphs.create_graph_from_coords(coords, max_dist=6000, force_single_cc=True, mst=False)

Generate skeleton from sample locations by adding edges between points with a maximum distance and then pruning the skeleton using MST. Nodes will have a ‘position’ attribute.

Parameters

• coords (ndarray) – Coordinates.

• max_dist (float) – Add edges between two nodes that are within this distance.

• force_single_cc (bool) – Force that the tree generated from coords is a single connected component.

• mst (bool) – Compute the minimum spanning tree.

Return type

Graph

Returns

Networkx graph. Edge between nodes (coord indices) using the ordering of coords, i.e. the edge (1, 2) connects coordinate coord[1] and coord[2].

syconn.proc.graphs.draw_glia_graph(G, dest_path, min_sv_size=0, ext_glia=None, iterations=150, seed=0, glia_key='glia_probas', node_size_cap=inf, mcmp=None, pos=None)

Draw graph with nodes colored in red (glia) and blue) depending on their class. Writes drawing to dest_path.

Parameters

• G – nx.Graph

• dest_path – str

• min_sv_size – int

• ext_glia – dict keys: node in G, values: number indicating class

• iterations –

• seed – int Default: 0; random seed for layout generation

• glia_key – str

• node_size_cap – int

• mcmp – color palette

• pos –

Returns:

syconn.proc.graphs.eucl_dist(a, b)

syconn.proc.graphs.get_glia_paths(g, glia_dict, node2ccsize_dict, min_cc_size_neuron, node2ccsize_dict_glia, min_cc_size_glia)

Currently not in use, Refactoring needed Find paths between neuron type SV grpah nodes which contain glia nodes.

Parameters

• g – nx.Graph

• glia_dict –

• node2ccsize_dict –

• min_cc_size_neuron –

• node2ccsize_dict_glia –

• min_cc_size_glia –

Returns:

syconn.proc.graphs.get_glianess_dict(seg_objs, thresh, glia_key, nb_cpus=1, use_sv_volume=False, verbose=False)

syconn.proc.graphs.glia_loader_helper(args)

syconn.proc.graphs.glia_path_length(glia_path, glia_dict, write_paths=None)

Get the path length of glia SV within glia_path. Assumes single connected glia component within this path. Uses the mesh property of each SegmentationObject to build a graph from all vertices to find shortest path through (or more precise: along the surface of) glia. Edges between non-glia vertices have negligible distance (0.0001) to ensure shortest path along non-glia surfaces.

Parameters

• glia_path – list of SegmentationObjects

• glia_dict – dict Dictionary which keys the SegmentationObjects in glia_path and returns their glia prediction

• write_paths – bool

Returns: float

Shortest path between neuron type nodes in nm

syconn.proc.graphs.merge_nodes(G, nodes, new_node)

FOR UNWEIGHTED, UNDIRECTED GRAPHS ONLY

syconn.proc.graphs.nxGraph2kzip(g, coords, kzip_path)

syconn.proc.graphs.remove_glia_nodes(g, size_dict, glia_dict, return_removed_nodes=False)

Calculate distance weights for shortest path analysis or similar, based on glia and size vertex properties and removes unsupporting glia nodes.

Parameters

• g – Graph

• size_dict –

• glia_dict –

• return_removed_nodes – bool

Returns: list of list of nodes

Remaining connected components of type neuron

syconn.proc.graphs.split_glia(sso, thresh, clahe=False, pred_key_appendix='')

Split SuperSegmentationObject into glia and non glia SegmentationObjects.

Parameters

• sso – SuperSegmentationObject

• thresh – float

• clahe – bool

• pred_key_appendix – str Defines type of glia predictions

Returns: list, list (of SegmentationObject)

Neuron, glia nodes

syconn.proc.graphs.split_glia_graph(nx_g, thresh, clahe=False, nb_cpus=1, pred_key_appendix='')

Split graph into glia and non-glua CC’s.

Parameters

• nx_g – nx.Graph

• thresh – float

• clahe – bool

• nb_cpus – int

• pred_key_appendix – str

• verbose – bool

Returns: list, list

Neuron, glia connected components.

syconn.proc.graphs.split_subcc(g, max_nb, verbose=False, start_nodes=None)

Creates subgraph for each node consisting of nodes until maximum number of nodes is reached.

Parameters

• g – Graph

• max_nb – int

• verbose – bool

• start_nodes – iterable node ID’s

Returns: dict

syconn.proc.graphs.split_subcc_join(g, subgraph_size, lo_first_n=1)

Creates a subgraph for each node consisting of nodes until maximum number of nodes is reached.

Parameters

• g (Graph) – Supervoxel graph

• subgraph_size (int) – Size of subgraphs. The difference between subgraph_size and lo_first_n defines the supervoxel overlap.

• lo_first_n (int) – Leave out first n nodes: will collect subgraph_size nodes starting from center node and then omit the first lo_first_n nodes, i.e. not use them as new starting nodes.

Returns:

Return type

List[List[Any]]

syconn.proc.graphs.stitch_skel_nx(skel_nx, n_jobs=1)

Stitch connected components within a graph by recursively adding edges between the closest components.

Parameters

• skel_nx (Graph) – Networkx graph. Nodes require ‘position’ attribute.

• n_jobs (int) – Number of jobs used for query of cKDTree.

Return type

Graph

Returns

Single connected component graph.

syconn.proc.graphs.svgraph2kzip(ssv, kzip_path)

Writes the SV graph stored in ssv.edgelist_path to a kzip file. The representative coordinate of a SV is used as the corresponding node location.

Parameters

• ssv (SuperSegmentationObject) – Cell reconstruction object.

• kzip_path (str) – Path to the output kzip file.

syconn.proc.graphs.write_sopath2skeleton(so_path, dest_path, scaling=None, comment=None)

Writes very simple skeleton, each node represents the center of mass of a SV, and edges are created in list order.

Parameters

• so_path – list of SegmentationObject

• dest_path – str

• scaling – np.ndarray or tuple

• comment – str

Returns:

syconn.proc.image module

syconn.proc.image.apply_clahe(arr, clipLimit=4.0, tileGridSize=(8, 8), ret_normalized=True)

If cv2 is available applies clahe filter on array.

Parameters

• arr – np.array

• clipLimit (float) –

• tileGridSize (tuple) – tuple of int

• ret_normalized (bool) –

Returns

np.array

syconn.proc.image.apply_clahe_plain(arr, clipLimit, tileGridSize)

syconn.proc.image.apply_equalhist(arr)

If cv2 is available applies histogram normalization on array.

Parameters

arr – np.array

Returns:

syconn.proc.image.apply_morphological_operations(vol, morph_ops, mop_kwargs=None)

Applies morphological operations on the input volume. String identifier in the morph_ops list must match scipy.ndimage functions.

Parameters

• vol (ndarray) – Input array (3D).

• morph_ops (List[str]) – List with string identifier.

• mop_kwargs (Optional[dict]) – Keyword arguments for the called morphological operation(s).

Return type

ndarray

Returns

Processed volume.

syconn.proc.image.apply_pca(sv, pca=None)

Apply principal component analysis and return rotated supervoxel

Parameters

• sv – np.array [N x 3] super voxel

• pca – PCA prefitted pca

Returns

super voxel coordinates rotated in principle component system

syconn.proc.image.conn_comp(sv, max_dist)

syconn.proc.image.createCLAHE([clipLimit[, tileGridSize]]) → retval

. @brief Creates a smart pointer to a cv::CLAHE class and initializes it. . . @param clipLimit Threshold for contrast limiting. . @param tileGridSize Size of grid for histogram equalization. Input image will be divided into . equally sized rectangular tiles. tileGridSize defines the number of tiles in row and column.

syconn.proc.image.equalizeHist(src[, dst]) → dst

. @brief Equalizes the histogram of a grayscale image. . . The function equalizes the histogram of the input image using the following algorithm: . . - Calculate the histogram f$Hf$ for src . . - Normalize the histogram so that the sum of histogram bins is 255. . - Compute the integral of the histogram: . f[H’_i = sum _{0 le j < i} H(j)f] . - Transform the image using f$H’f$ as a look-up table: f$texttt{dst}(x,y) = H’(texttt{src}(x,y))f$ . . The algorithm normalizes the brightness and increases the contrast of the image. . . @param src Source 8-bit single channel image. . @param dst Destination image of the same size and type as src .

syconn.proc.image.fast_check_sing_comp(sv, max_dist=5)

Fast check if super voxel is single connected component by subsampling

Parameters

• sv – np.array

• max_dist (int) –

Returns

True if single connected component

Return type

bool

syconn.proc.image.find_contactsite(coords_a, coords_b, max_hull_dist=1)

Computes contact sites between supver voxels and returns contact site voxel

Parameters

• coords_a – np.array

• coords_b – np.array

• max_hull_dist (int) – Maximum distance between voxels in coods_a and coords_b

Returns

contact site coordinates

Return type

np.array

syconn.proc.image.get_aniso_struct(scaling)

Get kernel for morphology operations; cross-like with aniso dilations in the xy plane.

Parameters

scaling (Union[tuple, ndarray]) – Voxel size in nm.

Returns

Kernel taking into account the voxel size.

syconn.proc.image.multi_dilation(overlay, n_dilations, use_find_objects=False, background_only=True)

Wrapper function for dilation

Parameters

• overlay –

• n_dilations –

• use_find_objects –

• background_only –

Returns:

syconn.proc.image.multi_dilation_backgroundonly(overlay, n_dilations, mop_kwargs=None)

Same as multi_dilation(), but processes each object in overlay independently. In addition, changes only apply to the background (0). E.g. objects will not dilate into other objects.

Parameters

• overlay – 3D volume of type uint.

• n_dilations – Number of dilations.

• mop_kwargs – Additional keyword arguments passed to mop_func.

Returns

Dilated overlay.

syconn.proc.image.multi_mop(mop_func, overlay, n_iters, use_find_objects=False, mop_kwargs=None, verbose=False)

Generic function for binary morphological image operations with multi-label content.

Currently supported operations:

• scipy.ndimage.binary_dilation, scipy.ndimage.binary_erosion, scipy.ndimage.binary_closing, scipy.ndimage.binary_fill_holes.

Parameters

• mop_func –

• overlay –

• n_iters –

• use_find_objects –

• mop_kwargs –

• verbose –

Returns:

syconn.proc.image.multi_mop_backgroundonly(mop_func, overlay, iterations, mop_kwargs=None)

Same as multi_mop(), but processes each object in overlay independently. In addition, changes only apply to the background (0). E.g. objects will not dilate into other objects. The original regions/segmentation is only affected in case of erosion.

Notes

• For binary_closing it is advised to pass structure=np.ones((2, 2, 2)) in order to fill gaps at the array boundaries. See https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.ndimage.morphology.binary_closing.html for an example.

Parameters

• mop_func – One of binary_closing, binary_dilation, binary_erosion, binary_fill_holes (see scipy.ndimage).

• overlay – 3D volume of type uint.

• iterations – Number of iterations.

• mop_kwargs – Additional keyword arguments passed to mop_func.

Returns

Volume processed by the given morphological operation.

syconn.proc.image.normalize_img(img, max_val=255)

Parameters

• img – np.array

• max_val – int or float

Returns

np.array

Normalized image

syconn.proc.image.normalize_vol(sv, edge_size, center_coord)

returns cube with given edge size and sv centered at center coordinate

Parameters

• sv – np.array [N x 3] coordinates of voxels in supervoxel

• edge_size – int edge size of returned cube

• center_coord – np.array

Returns

np.array

syconn.proc.image.remove_outlier(sv, edge_size)

Removes outlier in array sv beyond [0, edge_sizes]

Parameters

• sv – np.array

• edge_size (int) –

Returns

np.array

syconn.proc.image.rgb2gray(rgb)

syconn.proc.image.single_conn_comp(sv, max_dist=2, ref_coord=None, return_bool=False)

Returns single connected component of coordinates.

Parameters

• sv – np.array

• max_dist (int) –

• ref_coord – np.array

• return_bool (bool) –

Returns

np.array

syconn.proc.image.single_conn_comp_img(img, background=1.0)

Returns connected component in image which is located at the center. TODO: add ‘max component’ option

Parameters

• img – np.array

• background (float) –

Returns

np.array

syconn.proc.mapping module

syconn.proc.mapping.crop_box_to_bndry(offset, box_size, bndry)

Restricts box_size and offset to valid values, i.e. within an upper limit (bndry) and a lower limit (0, 0, 0).

Parameters

• offset (np.array) –

• box_size (np.array | list) –

• bndry (np.array) –

Returns

Valid box size and offset

Return type

np.array

syconn.proc.mapping.get_glia_coverage(seg, neuron_ids, glia_ids, max_dist, scale)

Computes the glia coverage of neurons in a segmentation volume. Neurons and glia are treated as two classes and coverage is defined as neuron boundary voxels close (within max_dist) to the glia boundary.

Parameters

• seg (np.array) –

• neuron_ids (list) –

• glia_ids (list) –

• max_dist (int|float) –

• scale (np.array) –

Returns

Number and fraction of neuron boundary voxels close to glia boundary

Return type

int | float

syconn.proc.mapping.map_glia_fraction(so, box_size=None, min_frag_size=10, overwrite=True)

Map glia properties within subvolume to SegmentationObject (cs). Requires attribute ‘neuron_partners’.

Parameters

• so – SegmentationObject

• box_size (np.array) – size in voxels (XYZ), default: (500, 500, 250)

• min_frag_size (int) –

• overwrite (bool) –

syconn.proc.meshes module

class syconn.proc.meshes.MeshObject(object_type, indices, vertices, normals=None, color=None, bounding_box=None)

Bases: object

property bounding_box

property colors

property normals

property normals_resh

perform_pca_rotation()

Rotates vertices into principal component coordinate system.

renormalize_vertices(bounding_box=None)

Renomralize, i.e. substract mean and divide by max. extent, vertices using either center and max. distance from self.vertices or given from keyword argument bounding_box.

Parameters

bounding_box – tuple center, scale (applied as follows: self.vert_resh / scale)

Returns:

retransform_external_coords(coords)

transform_external_coords(coords)

Parameters

coords – np.array

Returns: np.array

transformed coordinates

property vert_resh

property vertices_scaled

syconn.proc.meshes.calc_contact_syn_mesh(segobj, voxel_dc=None, **gen_kwgs)

Parameters

• segobj (segmentation.SegmentationObject) –

• voxel_dc (Union[VoxelStorageLazyLoading, VoxelStorageDyn, None]) –

• **gen_kwgs –

Returns:

Return type

Union[List[ndarray], List[List[ndarray]]]

syconn.proc.meshes.calc_rot_matrices(coords, vertices, edge_length, nb_cpus=1)

# Optimization comment: bottleneck is now ‘get_rotmatrix_from_points’

Fits a PCA to local sub-volumes in order to rotate them according to its main process (e.g. x-axis will be parallel to the long axis of a tube)

Parameters

• coords (ndarray) – Center coordinates [M x 3]

• vertices (ndarray) – Vertices [N x 3]

• edge_length (Union[float, int]) – Spatial extent of box used to querying vertices for the PCA fit (used for the view alignment).

• nb_cpus (int) – Number of CPUs.

Returns: Flattened OpenGL rotation matrix (Fortran ordering).

Return type

ndarray

syconn.proc.meshes.compartmentalize_mesh(ssv, pred_key_appendix='')

Splits SuperSegmentationObject mesh into axon, dendrite and soma. Based on axoness prediction of SV’s contained in SuperSuperVoxel ssv.

Parameters

• ssv (super_segmentation_object.SuperSegmentationObject) – SuperSegmentationObject

• pred_key_appendix – str Specific version of axoness prediction

Returns: np.array

Majority label of each face / triangle in mesh indices; Triangle faces are assumed. If majority class has n=1, majority label is set to -1.

syconn.proc.meshes.find_meshes(chunk, offset, pad=0, ds=None, scaling=None, meshing_props=None)

Find meshes within a segmented cube. The offset is given in voxels. Mesh vertices are scaled according to global_params.config['scaling'].

Parameters

• chunk (ndarray) – Cube which is processed.

• offset (ndarray) – Offset of the cube in voxels.

• pad (int) – Pad chunk array with mode ‘edge’.

• ds (Union[list, tuple, ndarray, None]) – Downsampling array in xyz. Default: No downsampling.

• scaling (Union[list, tuple, ndarray, None]) – Voxel size.

• meshing_props (Optional[dict]) – Keyword arguments used in zmesh.Mesher.get_mesh.

Return type

Dict[int, List[ndarray]]

Returns

The mesh of each segmentation ID in the input chunk. Vertices are in nm!

syconn.proc.meshes.gen_mesh_voxelmask(voxel_iter, scale, vertex_size=10, boundary_struct=None, depth=10, compute_connected_components=True, voxel_size_simplify=None, min_vert_num=200, overlap=1, verbose=False, nb_neighbors=20, std_ratio=2.0)

Parameters

• voxel_iter (Iterator[Tuple[ndarray, ndarray]]) – Iterator of binary voxel mask (3D cube) and cube offset (in voxels).

• scale (ndarray) – Size of voxels in mask_list in nm (x, y, z).

• vertex_size (float) – In nm. Resolution used to simplify mesh.

• boundary_struct (Optional[ndarray]) – Connectivity of kernel used to determine boundary

• depth (int) – http://www.open3d.org/docs/latest/tutorial/Advanced/surface_reconstruction.html#Poi sson-surface-reconstruction : “An important parameter of the function is depth that defines the depth of the octree used for the surface reconstruction and hence implies the resolution of the resulting triangle mesh. A higher depth value means a mesh with more details.”

• compute_connected_components (bool) – Compute connected components of mesh. Return list of meshes.

• voxel_size_simplify (Optional[float]) – Voxel size in nm when applying simplify_vertex_clustering. Defaults to vertex_size.

• min_vert_num (int) – Minimum number of vertices of the connected component meshes (only applied if compute_connected_components=True).

• overlap (int) – Overlap between adjacent masks in mask_list.

• verbose (bool) – Extra stdout output.

• nb_neighbors (int) – Number of neighbors used to calculate distance mean and standard deviation. See http://www.open3d.org/docs/latest/tutorial/Advanced/pointcloud_outlier_removal.html#Statistical-outlier-removal

• std_ratio (float) – Standard deviation of distance between points used as threshold for filtering. See http://www.open3d.org/docs/latest/tutorial/Advanced/pointcloud_outlier_removal.html#Statistical-outlier-removal

Notes: Use voxel_iter with cubes that have 1-voxel-overlap to guarantee that segmentation instance boundaries

that align with the 3D array border are identified correctly.

Return type

Union[List[ndarray], List[List[ndarray]]]

Returns

Flat Index/triangle, vertex and normals array of the mesh. List[ind, vert, norm] if compute_connected_components=True.

syconn.proc.meshes.get_object_mesh(obj, ds, mesher_kwargs=None)

Get object mesh from object voxels using marching cubes. Boundary artifacts are minimized by using a single 3D mask array of the object.

Notes

This method is not suited for large objects as it creates a single 3D binary mask of the object.

Parameters

• obj (segmentation.SegmentationObject) – SegmentationObject.

• ds (Union[tuple, list, ndarray]) – Magnitude of downsampling for each axis.

• mesher_kwargs (Optional[dict]) – Keyword arguments parsed to ‘find_meshes’ method.

Returns

vertices [N, 1], indices [M, 1], normals [M, 1]

syconn.proc.meshes.get_random_centered_coords(pts, nb, r)

Parameters

• pts – np.array coordinates

• nb – int number of center of masses to be returned

• r – int radius of query_ball_point in order to get list of points for center of mass

Returns: np.array

coordinates of randomly located center of masses in pts

syconn.proc.meshes.merge_meshes(ind_lst, vert_lst, nb_simplices=3)

Combine several meshes into a single one.

Parameters

• ind_lst – list of np.array [N, 1]

• vert_lst – list of np.array [N, 1]

• nb_simplices – int Number of simplices, e.g. for triangles nb_simplices=3

Returns: np.array, np.array

syconn.proc.meshes.merge_meshes_incl_norm(ind_lst, vert_lst, norm_lst, nb_simplices=3)

Combine several meshes into a single one.

Parameters

• ind_lst – List[np.ndarray] array shapes [M, 1]

• vert_lst – List[np.ndarray] array shapes [N, 1]

• norm_lst – List[np.ndarray] array shapes [N, 1]

• nb_simplices – int Number of simplices, e.g. for triangles nb_simplices=3

Returns: [np.array, np.array, np.array]

syconn.proc.meshes.merge_someshes(sos, nb_simplices=3, nb_cpus=1, color_vals=None, cmap=None, alpha=1.0, use_new_subfold=True)

Merge meshes of SegmentationObjects. This will cache SegmentationObject.

Parameters

• sos (Iterable[ForwardRef]) – SegmentationObjects are used to get .mesh, N x 1

• nb_simplices (int) – Number of simplices, e.g. for triangles nb_simplices=3

• nb_cpus (int) – int

• color_vals (Optional[Iterable[float]]) – Color values for every mesh, N x 4 (rgba). No normalization!

• cmap (Optional[Iterable[tuple]]) – matplotlib colormap

• alpha (float) – float

• use_new_subfold (bool) –

Returns: np.array, np.array [, np.array]

indices, vertices (scaled) [, colors]

syconn.proc.meshes.mesh2obj_file(dest_path, mesh, color=None, center=None, scale=None)

Writes mesh to .obj file.

Parameters

• dest_path (str) – Path to file.

• mesh (List[ndarray]) – Flat arrays of indices (triangle faces), vertices and normals.

• color (Union[int, ndarray, None]) – Color as int or numpy array (rgba).

• center (Optional[ndarray]) – Subtracts center from original vertex locations.

• scale (Optional[float]) – Multiplies vertex locations after centering.

Returns:

syconn.proc.meshes.mesh_area_calc(mesh)

Parameters

mesh – meshobject

Returns: float

Mesh area in um^2

syconn.proc.meshes.mesh_chunk(args)

syconn.proc.meshes.mesh_creator_sso(ssv, segobjs=('sv', 'mi', 'sj', 'vc'))

Cache meshes of specified SegmentationObjects.

Parameters

• ssv (super_segmentation_object.SuperSegmentationObject) – SuperSegmentationObject.

• segobjs (Iterable[str]) – Types of SegmentationObjects.

Returns:

syconn.proc.meshes.write_mesh2kzip(k_path, ind, vert, norm, color, ply_fname, force_overwrite=False, invert_vertex_order=False)

Writes mesh as .ply’s to k.zip file.

Parameters

• k_path – str path to zip

• ind – np.array

• vert – np.array

• norm – np.array

• color – tuple or np.array rgba between 0 and 255

• ply_fname – str

• force_overwrite – bool

• invert_vertex_order – bool Invert the vertex order.

Returns:

syconn.proc.meshes.write_meshes2kzip(k_path, inds, verts, norms, colors, ply_fnames, force_overwrite=True, verbose=True, invert_vertex_order=False)

Writes meshes as .ply’s to k.zip file.

Parameters

• k_path – str path to zip

• inds – list of np.array

• verts – list of np.array

• norms – list of np.array

• colors – list of tuple or np.array rgba between 0 and 255

• ply_fnames – list of str

• force_overwrite – bool

• verbose – bool

• invert_vertex_order – bool Invert the vertex order.

Returns:

syconn.proc.rendering module

syconn.proc.rendering.get_sso_view_dc(sso, verbose=False)

Extracts views from sampled positions in SSO for each SV.

Parameters

• sso (SuperSegmentationObject) – SuperSegmentationObject

• verbose (bool) – bool

Return type

dict

Returns

Dictionary with sso id as key and lz4 compressed view array (see :py:func:~syconn.handler.compression.arrtolz4string).

syconn.proc.rendering.load_rendering_func(func_name)

Return type

Callable

syconn.proc.rendering.render_mesh(mo, **kwargs)

Render super voxel raw views located at randomly chosen center of masses in vertex cloud.

Parameters

• mo (MeshObject) – Mesh.

• **kwargs – Keyword arguments pass to multi_view_mesh() call.

Return type

ndarray

Returns

View array.

syconn.proc.rendering.render_mesh_coords(coords, ind, vert, **kwargs)

Render raw views located at given coordinates in mesh Returns ViewContainer list if dest_dir is None, else writes views to dest_path.

Parameters

• coords (ndarray) – Rendering locations.

• ind (ndarray) – Mesh indices/faces [N, 1]

• vert (ndarray) – Mesh vertices [M, 1]

• **kwargs – Keyword arguments passed to _render_mesh_coords().

Returns: Views at each coordinate.

Return type

ndarray

syconn.proc.rendering.render_sampled_sso(sso, ws=None, verbose=False, woglia=True, return_rot_mat=False, overwrite=True, add_cellobjects=None, index_views=False, return_views=False, cellobjects_only=False, rot_mat=None, view_key=None)

Renders for each SV views at sampled locations (number is dependent on SV mesh size with scaling fact) from combined mesh of all SV.

Parameters

• sso (SuperSegmentationObject) – SuperSegmentationObject

• ws (Optional[Tuple[int, int]]) – Window size in pixels (y, x). Default: See config.yml or custom configs in the working directory.

• verbose (bool) – Log additional information.

• woglia (bool) – Store views with “without glia” identifier, i.e. flag the views as being created after the glia separation.

• return_rot_mat (bool) – Return rotation matrices.

• add_cellobjects (Union[bool, Iterable[str], None]) – Default: (‘sj’, ‘vc’, ‘mi’)

• overwrite (bool) – If True, do not skip existing views.

• index_views (bool) – Also render index views.

• return_views (bool) – Return view arrays.

• cellobjects_only (bool) – Only render cell objects.

• rot_mat (Optional[ndarray]) – Rotation matrix array for every rendering location [N, 4, 4].

• view_key (Optional[str]) – String identifier for storing view arrays. Only needed if return_views=False.

Returns

None; View array; View array and rotation matrices; rotation matrices.

Return type

Depending on return_views and return_rot_mat

syconn.proc.rendering.render_sso_coords(sso, coords, add_cellobjects=None, verbose=False, clahe=False, ws=None, cellobjects_only=False, wire_frame=False, nb_views=None, comp_window=None, rot_mat=None, return_rot_mat=False)

Render views of SuperSegmentationObject at given coordinates.

Parameters

• sso (SuperSegmentationObject) – SuperSegmentationObject

• coords (ndarray) – Rendering locations [N, 3].

• add_cellobjects (Union[bool, Iterable[str], None]) – Default: (‘sj’, ‘vc’, ‘mi’). This ordering determines the channel order of the view array.

• verbose (bool) – Log additional information.

• clahe (bool) – Use clahe to enahnce view contrast.

• ws (Optional[Tuple[int]]) – Window size in pixels (y, x). Default: See config.yml or custom configs in the working directory.

• cellobjects_only (bool) – Only render cell objects.

• wire_frame (bool) – Render the mesh as a wire frame.

• nb_views (Optional[int]) – Number of views. Default: See config.yml or custom configs in the working directory.

• comp_window (Optional[float]) – Window size in nm. the clipping box during rendering will have an extent of [comp_window, comp_window / 2, comp_window]. Default: 8 um.

• rot_mat (Optional[ndarray]) – Rotation matrix array for every rendering location [N, 4, 4].

• return_rot_mat (bool) – Return rotation matrices, e.g. if not provided via rot_mat, this output can be provided for other rendering calls.

Returns

[len(coords), 4 (default: cell outline + number of cell objects), nb_views, y, x].

Return type

Resulting views rendered at each location. Output shape

syconn.proc.rendering.render_sso_coords_index_views(sso, coords, verbose=False, ws=None, rot_mat=None, nb_views=None, comp_window=None, return_rot_matrices=False)

Uses per-face color via flattened vertices (i.e. vert[ind] -> slow!). This was added to be able to calculate the surface coverage captured by the views.

Parameters

• sso (SuperSegmentationObject) – SuperSegmentationObject

• coords (ndarray) – Rendering locations [N, 3].

• verbose (bool) – Log additional information.

• ws (Optional[Tuple[int, int]]) – Window size in pixels (y, x). Default: See config.yml or custom configs in the working directory.

• rot_mat (Optional[ndarray]) – np.array

• nb_views (Optional[int]) – Number of views. Default: See config.yml or custom configs in the working directory.

• comp_window (Optional[float]) – Window size in nm. the clipping box during rendering will have an extent of [comp_window, comp_window / 2, comp_window]. Default: 8 um.

• return_rot_matrices (bool) – Return rotation matrices, e.g. if not provided via rot_mat, this output can be provided for other rendering calls.

Returns

[len(coords), 1, nb_views, y, x].

Return type

Resulting index views rendered at each location. Output shape

syconn.proc.rendering.render_sso_coords_label_views(sso, vertex_labels, coords, verbose=False, ws=None, rot_mat=None, nb_views=None, comp_window=None, return_rot_matrices=False)

Render views with vertex colors corresponding to vertex labels.

Parameters

• sso (SuperSegmentationObject) –

• vertex_labels (ndarray) – np.array vertex labels [N, 1]. Ordering and length have to be the same as vertex array of SuperSegmentationObject (len(sso.mesh[1]) // 3).

• coords (ndarray) – Rendering locations [N, 3].

• verbose (bool) – Log additional information.

• ws (Optional[Tuple[int, int]]) – Window size in pixels (y, x). Default: See config.yml or custom configs in the working directory.

• rot_mat (Optional[ndarray]) – np.array

• nb_views (Optional[int]) – Number of views. Default: See config.yml or custom configs in the working directory.

• comp_window (Optional[float]) – Window size in nm. the clipping box during rendering will have an extent of [comp_window, comp_window / 2, comp_window]. Default: 8 um.

• return_rot_matrices (bool) – Return rotation matrices, e.g. if not provided via rot_mat, this output can be provided for other rendering calls.

Returns

[len(coords), 1, nb_views, y, x].

Return type

Resulting label views rendered at each location. Output shape

syconn.proc.rendering.render_sso_coords_multiprocessing(ssv, n_jobs, rendering_locations=None, verbose=False, render_kwargs=None, view_key=None, render_indexviews=True, return_views=True)

Parameters

• ssv (SuperSegmentationObject) – SuperSegmentationObject

• n_jobs (int) – int number of parallel jobs running on same node of cluster

• rendering_locations (Optional[ndarray]) – array of locations to be rendered if not given, rendering locations are retrieved from the SSV’s SVs. Results will be stored at SV locations.

• verbose (bool) – bool flag to show the progress of rendering.

• render_kwargs (Optional[dict]) – dict

• view_key (Optional[str]) – str

• render_indexviews (bool) – bool

• return_views (bool) – If False and rendering_locations is None, views will be saved on supervoxel level.

Return type

Optional[ndarray]

Returns

Array of views after rendering of locations or None.

syconn.proc.rendering.write_sv_views_chunked(svs, views, part_views, view_kwargs, disable_locking=False)

Parameters

• svs (List[ForwardRef]) – SegmentationObjects

• views (ndarray) – View array.

• part_views (ndarray) – Cumulated number of views -> indices of start and end of SV views in views array.

• view_kwargs (dict) –

• disable_locking (bool) – Usually required as SVs are stored in chunks and rendered distributed on many processes.

Returns:

syconn.proc.sd_proc module

syconn.proc.sd_proc.convert_nvox2ratio_mapdict(map_dc)

convert number of overlap voxels of each subcellular structure object inside the mapping dicts to each cell SV (subcell ID -> cell ID -> number overlap vxs) to fraction.

syconn.proc.sd_proc.dataset_analysis(sd, recompute=True, n_jobs=None, compute_meshprops=False)

Analyze SegmentationDataset and extract and cache SegmentationObjects attributes as numpy arrays. Will only recognize dict/storage entries of type int for object attribute collection.

Parameters

• sd – SegmentationDataset of e.g. cell supervoxels (‘sv’).

• recompute – Whether or not to (re-)compute key information of each object (rep_coord, bounding_box, size).

• n_jobs – Number of jobs.

• compute_meshprops – Compute mesh properties. Will also calculate meshes (sparsely) if not available.

syconn.proc.sd_proc.init_sos(sos_dict)

syconn.proc.sd_proc.invert_mdc(mapping_dict)

Inverts mapping dict to: cell ID -> subcell ID -> value (ratio or voxel count)

syconn.proc.sd_proc.map_subcell_extract_props(kd_seg_path, kd_organelle_paths, n_folders_fs=1000, n_folders_fs_sc=1000, n_chunk_jobs=None, n_cores=1, cube_of_interest_bb=None, chunk_size=None, log=None, overwrite=False)

Extracts segmentation properties for each SV in cell and subcellular segmentation. Requires KDs at kd_seg_path and kd_organelle_paths.

• Step 1: Extract properties (representative coordinate, bounding box, size) and overlap voxels locally (orgenelles <-> cell segmentation).

• Step 2: Write out combined results for each SV object.

Parameters

• kd_seg_path (str) –

• kd_organelle_paths (dict) –

• n_folders_fs (int) –

• n_folders_fs_sc (int) –

• n_chunk_jobs (Optional[int]) –

• n_cores (int) –

• cube_of_interest_bb (Optional[tuple]) –

• chunk_size (Union[tuple, ndarray, None]) –

• log (Optional[Logger]) –

• overwrite –

Returns:

syconn.proc.sd_proc.merge_map_dicts(map_dicts)

Merge map dictionaries in-place. Values will be stored in first dictionary

Parameters

map_dicts –

Returns:

syconn.proc.sd_proc.merge_meshes_dict(m_storage, tmp_dict)

Merge meshes dictionaries:

m_storage: list dictionary tmp_dict: list dictionary {obj_id: [faces, vertices, normals]}

syconn.proc.sd_proc.merge_meshes_single(m_storage, obj_id, tmp_dict)

Merge meshes dictionaries: m_storage: objec of type MeshStorage tmp_dict: list dictionary

syconn.proc.sd_proc.merge_prop_dicts(prop_dicts, offset=None)

Merge property dicts in-place. All values will be stored in the first dict.

syconn.proc.sd_proc.mesh_proc_chunked(working_dir, obj_type, nb_cpus=None)

Caches the meshes for all SegmentationObjects within the SegmentationDataset with object type ‘obj_type’.

Parameters

• working_dir – str Path to working directory

• obj_type – str Object type identifier, like ‘sj’, ‘vc’ or ‘mi’

• nb_cpus – int Default is 20.

Returns:

syconn.proc.sd_proc.multi_probas_saver(args)

syconn.proc.sd_proc.predict_sos_views(model, sos, pred_key, nb_cpus=1, woglia=True, verbose=False, raw_only=False, single_cc_only=False, return_proba=False)

Parameters

• model –

• sos –

• pred_key –

• nb_cpus –

• woglia –

• verbose –

• raw_only –

• single_cc_only –

• return_proba –

Returns:

syconn.proc.sd_proc.predict_views(model, views, ch, pred_key, single_cc_only=False, verbose=False, return_proba=False, nb_cpus=1)

Will not be written to disk if return_proba is True.

Parameters

• model – nn.Model

• views – np.array

• ch – List[SegmentationObject]

• pred_key – str

• single_cc_only – bool

• verbose – bool

• return_proba – bool

• nb_cpus – int

Returns:

Return type

Optional[List[ndarray]]

syconn.proc.sd_proc.sos_dict_fact(svixs, version=None, scaling=None, obj_type='sv', working_dir=None, create=False)

syconn.proc.skel_based_classifier module

syconn.proc.skel_based_classifier_helper module

syconn.proc.ssd_assembly module

syconn.proc.ssd_assembly.init_ssd_from_kzips(dir_path)

syconn.proc.ssd_assembly.init_sso_from_kzip(path, load_as_tmp=True, sso_id=None)

Initializes cell reconstruction from k.zip file. The k.zip needs the following content:

• Mesh files: ‘sv.ply’, ‘mi.ply’, ‘sj.ply’, ‘vc.ply’

• meta dict: ‘meta.pkl’

• [Optional] Rendering locations: ‘sample_locations.pkl’ (currently broekn to use .npy, fixed in python 3.7)

• [Optional] Supervoxel graph: ‘rag.bz2’

• [Optional] Skeleton representation: ‘skeleton.pkl’

• [Optional] attribute dict: ‘attr_dict.pkl’

Parameters

• path – str Path to kzip which contains SSV data

• load_as_tmp – bool If True then working_dir and version_dict in meta.pkl dictionary is not passed to SSO constructor, instead all version will be set to ‘tmp’ and working directory will be None. Used to process SSO independent on working directory.

• sso_id – int ID of SSV, if not given looks for the first scalar occurrence in path

Returns: SuperSegmentationObject

syconn.proc.ssd_proc module

syconn.proc.ssd_proc.aggregate_segmentation_object_mappings(ssd, obj_types, n_jobs=None, nb_cpus=None)

Populates SSV attributes mapping_{obj_type}_ids and mapping_{obj_type}_ratios for each element in obj_types; every element must exist in ssd.version_dict.

Parameters

• ssd (SuperSegmentationDataset) – SuperSegmentationDataset

• obj_types (List[str]) – List of object identifiers which are mapped to the cells, e.g. [‘mi’, ‘sj’, ‘vc’].

• n_jobs (Optional[int]) – Number of jobs.

• nb_cpus (Optional[int]) – Cores per job when using BatchJob or the total number of jobs used if single node multiprocessing.

syconn.proc.ssd_proc.apply_mapping_decisions(ssd, obj_types, n_jobs=None, nb_cpus=None)

Populates SSV attributes {obj_type} for each element in`obj_types`, every element must exist in ssd.version_dict and in ssd.config['cell_objects'] (see also config.yml in the working directory and/or in syconn/handler/config.yml).

Requires prior execution of aggregate_segmentation_object_mappings().

Parameters

• ssd (SuperSegmentationDataset) – SuperSegmentationDataset.

• obj_types (List[str]) – List of object identifiers which are mapped to the cells, e.g. [‘mi’, ‘sj’, ‘vc’].

• n_jobs (Optional[int]) –

• nb_cpus (Optional[int]) – cpus per job when using BatchJob or the total number of jobs used if single node multiprocessing.

syconn.proc.ssd_proc.get_available_ssv_ids(ssd, n=2)

syconn.proc.ssd_proc.init_ssv(ssv_id, sv_ids, ssd)

Initializes an SuperSegmentationObject and caches all relevant data. Cell organelles and supervoxel SegmentationDatasets must be initialized.

Return type

SuperSegmentationObject

syconn.proc.ssd_proc.map_synssv_objects(synssv_version=None, log=None, nb_cpus=None, n_jobs=None, syn_threshold=None)

Map syn_ssv objects and merge their meshes for all SSO objects contained in SSV SuperSegmentationDataset.

Notes

• Stores meshes with keys: ‘syn_ssv’ and ‘syn_ssv_sym’, syn_ssv_asym (if synapse type is available). This may take a while.

Parameters

• synssv_version (Optional[str]) – String identifier.

• n_jobs – Number of jobs.

• log (Optional[Logger]) – Logger.

• nb_cpus – Number of cpus for local multi-processing.

• syn_threshold – Probability threshold applied during the mapping of syn_ssv objects.

syconn.proc.ssd_proc.map_synssv_objects_thread(args)

syconn.proc.ssd_proc.mesh_proc_ssv(working_dir, version=None, ssd_type='ssv', nb_cpus=None)

Caches the SSV meshes locally with 20 cpus in parallel.

Parameters

• working_dir (str) – str Path to working directory.

• version (Optional[str]) – str version identifier, like ‘spgt’ for spine ground truth SSD. Defaults to the SSD of the cellular SSVs.

• ssd_type (str) – str Default is ‘ssv’.

• nb_cpus (Optional[int]) – int Default is cpu_count().

Returns:

syconn.proc.ssd_proc.split_ssv(ssv, splitted_sv_ids)

Splits an SuperSegmentationObject into two.

Return type

Tuple[SuperSegmentationObject, SuperSegmentationObject]

syconn.proc.stats module

class syconn.proc.stats.FileTimer(working_dir, overwrite=False, add_detail_vols=False)

Bases: object

ContextDecorator for timing. Stores the results as dict in a pkl file.

Examples

The script SyConn/examples/start.py uses FileTimer to track the execution time of several major steps of the analysis. The results are written as dict to the file ‘.timing.pkl’ in the working directory. The timing data can be accessed after the run to by initializing FileTimer with the output file:

ft = FileTimer(path_to_timings_pkl) # this is a dict with the step names as keys and the timings in seconds as values print(ft.timings)

property dataset_mm3: float

Returns: Data set size in cubic mm.

Return type

float

property dataset_nvoxels: float

Returns: Data set size in giga voxels.

Return type

float

property dataset_shape: float

Returns: Data set size in giga voxels.

Return type

float

prepare_report()

Return type

str

prepare_vol_info()

start(step_name)

stop()

syconn.proc.stats.array2xls(dest_p, arr)

syconn.proc.stats.cluster_summary(train_d, train_l, valid_d, valid_l, fold, prefix='', pca=None, return_valid_pred=False)

Create clustering summary and save results to folder.

Parameters

• train_d –

• train_l –

• valid_d –

• valid_l –

• fold (str) – destination folder

syconn.proc.stats.fscore(rec, prec, beta=1.0)

Calculates f-score with beta value

Parameters

• rec (np.array) – recall

• prec (np.array) – precision

• beta (float) – weighting of precision

Returns

f-score

Return type

np.array

syconn.proc.stats.hist(vals, labels=None, dest_path=None, axis_labels=None, x_lim=None, y_lim=None, y_log_scale=False, ls=22, color=None, **kwargs)

syconn.proc.stats.model_performance(proba, labels, model_dir=None, prefix='', n_labels=3, fscore_beta=1, target_names=None, add_text='')

syconn.proc.stats.model_performance_predonly(y_pred, y_true, model_dir=None, prefix='', target_names=None, labels=None)

syconn.proc.stats.plot_pr(precision, recall, title='', r=[0.67, 1.01], legend_labels=None, save_path=None, nbins=5, colorVals=None, xlabel='Recall', ylabel='Precision', l_pos='lower left', legend=True, r_x=[0.67, 1.01], ls=22, xtick_labels=())

syconn.proc.stats.projection_pca(ds_d, ds_l, dest_path, pca=None, colors=None, do_3d=True, target_names=None)

Parameters

• ds_d (np.array) – data in feature space, e.g. (#data, #feature)

• ds_l – sparse labels, i.e. (#data, 1)

• dest_path (str) – file name of plot

• pca – PCA prefitted PCA object to use to prject data of ds_d

syconn.proc.stats.projection_tSNE(ds_d, ds_l, dest_path, colors=None, target_names=None, do_3d=False, cmap_ident='prism', **tsne_kwargs)

Parameters

• ds_d (np.array) – data in feature space, e.g. (#data, #feature)

• ds_l – sparse labels, i.e. (#data, 1)

• dest_path (str) – file name of plot

• pca – PCA prefitted PCA object to use to prject data of ds_d