syconn.mp package

syconn.mp.batchjob_utils module

syconn.mp.batchjob_utils.batchjob_enabled()[source]

Checks if active batch processing system is actually working.

Return type

bool

Returns

True if either SLURM or QSUB is active.

syconn.mp.batchjob_utils.batchjob_fallback(params, name, n_cores=1, suffix='', script_folder=None, python_path=None, remove_jobfolder=False, show_progress=True, log=None, overwrite=False, job_folder=None)[source]

# TODO: utilize log and error files (‘path_to_err’, path_to_log’) Fallback method in case no batchjob submission system is available.

Parameters

  • list[Any] (params) –

  • name (str) –

  • n_cores (int) – CPUs per job.

  • suffix (str) –

  • script_folder (str) –

  • python_path (str) –

  • remove_jobfolder (bool) –

  • show_progress (bool) –

  • log – Logger Logger.

  • overwrite –

  • job_folder –

Returns

Path to output.

Return type

str

syconn.mp.batchjob_utils.batchjob_script(params, name, batchjob_folder=None, n_cores=1, additional_flags='', suffix='', job_name='default', script_folder=None, max_iterations=10, python_path=None, disable_batchjob=False, use_dill=False, remove_jobfolder=False, log=None, sleep_time=None, show_progress=True, overwrite=False, exclude_nodes=None)[source]

Submits batch jobs to process a list of parameters params with a python script on the specified environment (either None, SLURM or QSUB; run global_params.config['batch_proc_system'] to get the active system).

Notes

  • The memory available for each job is coupled to the number of cores per job (n_cores).

Parameters

  • params (list) – List of all parameter sets to be processed.

  • name (str) – Name of batch job submitted via the batch processing system.

  • batchjob_folder (Optional[str]) – Directory which contains all submission relevant files, e.g. bash scripts, logs, output files, .. Defaults to "{}/{}_folder{}/".format(global_params.config.qsub_work_folder, name, suffix).

  • n_cores (int) – Number of cores used for each job.

  • additional_flags (str) – Used to set additional parameters for each job. To allocate one GPU for each worker use: additional_flags=--gres=gpu:1.

  • suffix (str) – Suffix added to batchjob_folder.

  • job_name (str) – Name of the jobs submitted via the batch processing system. Defaults to a random string of 8 letters.

  • script_folder (Optional[str]) – Directory where to look for the script which is executed. Looks for QSUB_{name}.py.

  • max_iterations (int) – Maximum number of retries of failed jobs.

  • python_path (Optional[str]) – Path to python binary.

  • disable_batchjob (bool) – Use single node multiprocessing.

  • use_dill (bool) – Use dill to enable pickling of lambda expressions. remove_jobfolder: Remove batchjob_folder after successful termination.

  • remove_jobfolder (bool) –

  • log (Optional[Logger]) – Logger.

  • sleep_time (Optional[int]) – Sleep duration before checking batch job states again.

  • show_progress (bool) – Only used if disabled_batchjob=True.

  • overwrite (bool) –

  • exclude_nodes (Optional[list]) – Nodes to exclude during job submission.

syconn.mp.batchjob_utils.delete_jobs_by_name(job_name)[source]

Deletes a group of jobs that have the same name

Parameters

job_name (str) – job_name as shown in qstats

Returns:

syconn.mp.batchjob_utils.fallback_exec(cmd_exec)[source]

Helper function to execute commands via subprocess.Popen.

syconn.mp.batchjob_utils.jobstates_slurm(job_name, start_time, max_retry=10)[source]

Generates a dictionary which stores the state of every job belonging to job_name.

Parameters

  • job_name (str) –

  • start_time (str) – The following formats are allowed: MMDD[YY] or MM/DD[/YY] or MM.DD[.YY], e.g. datetime.datetime.today().strftime("%m.%d").

  • max_retry (int) – Number of retries for sacct SLURM query if failing (5s sleep in-between).

Returns

job ID, value: state)

Return type

Dictionary with the job states. (key

syconn.mp.batchjob_utils.nodestates_slurm()[source]

Generates a dictionary which stores the state of every job belonging to job_name.

Args:

Returns

job ID, value: state dict)

Return type

Dictionary with the node states. (key

syconn.mp.batchjob_utils.number_of_running_processes(job_name)[source]

Calculates the number of running jobs using qstat/squeue

Parameters

job_name (str) – job_name as shown in qstats

Returns

number of running jobs

Return type

nb_jobs (int)

syconn.mp.batchjob_utils.restart_nodes_daemon()[source]

Only support gce. [WIP]

Returns:

syconn.mp.mp_utils module

syconn.mp.mp_utils.multi_helper_obj(args)[source]

Generic helper emthod for multiprocessed jobs. Calls the given object method.

Parameters

args (Iterable) – object, method name, optional: kwargs

Returns:

syconn.mp.mp_utils.parallel_process(array, function, n_jobs, use_kwargs=False, front_num=0, show_progress=True, use_dill=False)[source]
From http://danshiebler.com/2016-09-14-parallel-progress-bar/

A parallel version of the map function with a progress bar.

Parameters

  • array (array-like) – An array to iterate over.

  • function (function) – A python function to apply to the elements of array n_jobs (int, default=16): The number of cores to use

  • use_kwargs (boolean, default=False) – Whether to consider the elements of array as dictionaries of keyword arguments to function

  • front_num (int, default=3) – The number of iterations to run serially before kicking off the parallel job. Useful for catching bugs.

  • n_jobs (int) –

  • show_progress (bool) – show progress

  • use_dill (bool) –

Return type

list

Returns

[function(array[0]), function(array[1]), …]

syconn.mp.mp_utils.start_multiprocess(func, params, debug=False, verbose=False, nb_cpus=None)[source]
Parameters

  • func (callable) – function

  • params (list) – function parameters

  • debug (bool) –

  • verbose (bool) –

  • nb_cpus (int) –

Returns

list of function returns

Return type

result (list)

syconn.mp.mp_utils.start_multiprocess_imap(func, params, debug=False, verbose=False, nb_cpus=None, show_progress=True, ignore_cpu_cnt=False, desc=None, use_dill=False)[source]
Parameters

  • func (Callable) –

  • params –

  • debug –

  • verbose –

  • nb_cpus –

  • show_progress –

  • ignore_cpu_cnt –

  • desc (Optional[str]) – Task description. Used for progress bar.

  • use_dill (bool) –

Returns

list of function returns.

syconn.mp.mp_utils.start_multiprocess_obj(func_name, params, debug=False, verbose=False, nb_cpus=None)[source]

Args: func_name (str): params (list): List[List]

each element in params must be object with attribute func_name (+ optional: kwargs)

debug (bool): verbose (bool): nb_cpus (int):

Returns

list of function returns

Return type

result (list)