bout_runners.submitter.slurm_submitter¶

Contains the SLURM submitter class.

Classes

SLURMSubmitter(job_name, store_directory, …)

The SLURM submitter class.

class bout_runners.submitter.slurm_submitter.SLURMSubmitter(job_name: str, store_directory: pathlib.Path, submission_dict: Optional[Dict[str, Optional[str]]] = None, processor_split: Optional[bout_runners.submitter.processor_split.ProcessorSplit] = None)[source]¶

The SLURM submitter class.

Examples

>>> submitter = SLURMSubmitter(job_name, store_path)
>>> submitter.submit_command("echo 'Hello'")
>>> submitter.wait_until_completed()
>>> submitter.std_out
Hello

Attributes

__sacct_starttimestr: Time to search from in sacct

Methods

_wait_for_std_out_and_std_err()	Wait until the process completes if a process has been started
extract_job_id(std_out)	Return the job_id
get_return_code(sacct_str)	Return the exit code if any
get_state(sacct_str)	Return the state from sacct
structure_time_to_slurm_format(time_str)	Structure the time string to a SLURM time string
completed()	Return the completed status
create_submission_string(command, waiting_for)	Return the PBS script as a string
get_sacct()	Return the trace from `sacct`
reset()	Reset released, waiting_for and status dict

Set the member data.

Parameters

job_namestr or None

Name of the job If None, a timestamp will be given as job_name

store_directoryPath or None

Directory to store the script If None, the caller directory will be used as the store directory

submission_dictNone or dict of str of None or str

Dict containing optional submission options One the form

>>> {'walltime': None or str,
...  'account': None or str,
...  'queue': None or str,
...  'mail': None or str}

These options will not be used if the submission_dict is None

processor_splitProcessorSplit or None

Object containing the processor split If None, default values will be used

Attributes

job_id: Return the process id.
job_name: Set the properties of self.job_name.
released: Return whether the job has been released to the cluster.
return_code: Return the return code.
std_err: Return the standard error.
std_out: Return the standard output.
store_dir: Set the properties of self.store_dir.
waiting_for: Return the waiting for list as a tuple.

Methods

`add_waiting_for`(waiting_for_id)	Add a waiting for id to the waiting for list.
`completed`()	Return the completed status.
`create_submission_string`(command, waiting_for)	Return the PBS script as a string.
`errored`([raise_error])	Return True if the process errored.
`extract_job_id`(std_out)	Return the job_id.
`get_days_hours_minutes_seconds_from_str`(time_str)	Return days, hours, minutes, seconds from the string.
`get_return_code`(sacct_str)	Return the exit code if any.
`get_sacct`()	Return the result from `sacct`.
`get_state`(sacct_str)	Return the state from sacct.
`kill`()	Kill a job if it exists.
`raise_error`()	Raise and error from the subprocess in a clean way.
`release`()	Release job if held.
`reset`()	Reset released, waiting_for and status dict.
`structure_time_to_slurm_format`(time_str)	Structure the time string to a SLURM time string.
`submit_command`(command)	Submit a command.
`wait_until_completed`([raise_error])	Wait until the process has completed.
`write_python_script`(path, function[, args, …])	Write python function to file.

add_waiting_for(waiting_for_id: Union[str, None, Iterable[str]]) → None¶

Add a waiting for id to the waiting for list.

This will waiting for list will be written to the submission string upon creation

Parameters

waiting_for_idNone or list of str: Id to the job waiting for

completed() → bool[source]¶

Return the completed status.

Returns

bool: Whether the job has completed

create_submission_string(command: str, waiting_for: Tuple[str, …]) → str[source]¶

Return the PBS script as a string.

Parameters

commandstr: The command to submit
waiting_fortuple of str: Tuple of ids that this job will wait for

Returns

job_scriptstr: The script to be submitted

errored(raise_error: bool = False) → bool¶

Return True if the process errored.

Parameters

raise_errorbool: Whether or not to raise errors

Returns

bool: True if the process returned a non-zero code

static extract_job_id(std_out: Optional[str]) → str[source]¶

Return the job_id.

Parameters

std_outstr or None: The standard output from the local submitter which submits the job

Returns

job_idstr: The job id

Raises

RuntimeError: If the job_id cannot be found

static get_days_hours_minutes_seconds_from_str(time_str: str) → Tuple[int, int, int, int]¶

Return days, hours, minutes, seconds from the string.

Parameters

time_strstr

Must be on the format

>>> 'hh:mm:ss'

or

>>> 'd-hh:mm:ss'

Returns

daysint: Number of days in the time string
hoursint: Number of hours in the time string
minutesint: Number of minutes in the time string
secondsint: Number of seconds in the time string

Raises

ValueError: If the string is malformatted

static get_return_code(sacct_str: str) → Optional[int][source]¶

Return the exit code if any.

Parameters

sacct_strstr: Trace obtained from the sacct command

Returns

return_codeNone or int: Return code obtained from the cluster

Notes

Assumes the job line is the third line of sacct_str (the two others being headers), and that the “ExitCode” is the last column in sacct

get_sacct() → str[source]¶

Return the result from sacct.

Returns

sacct_strstr: The string obtained from sacct An empty string is will be returned if no job_id exist

static get_state(sacct_str: str) → Optional[str][source]¶

Return the state from sacct.

Parameters

sacct_strstr: Trace obtained from the sacct command

Returns

statusNone or str: Status code obtained from the cluster

Notes

Assumes the job line is the third line of sacct_str (the two others being headers), and that the “State” is the second last column before “ExitCode” in sacct

property job_id¶

Return the process id.

Returns

self._status[“job_id”]int or None: The process id if a process has been called, else None

property job_name¶

Set the properties of self.job_name.

Returns

str: The job name

kill() → None¶: Kill a job if it exists.

raise_error() → None¶

Raise and error from the subprocess in a clean way.

Raises

RuntimeError: If an error was caught

release() → None¶: Release job if held.

property released¶

Return whether the job has been released to the cluster.

Returns

bool: True if the job is not held in the cluster

reset() → None[source]¶: Reset released, waiting_for and status dict.

property return_code¶: Return the return code.

property std_err¶

Return the standard error.

Returns

self._status[“std_err”]str or None: The standard error None if the process has not completed

property std_out¶

Return the standard output.

Returns

self._status[“std_out”]str or None: The standard output None if the process has not completed

property store_dir¶

Set the properties of self.store_dir.

Returns

Path: Path to the store directory

static structure_time_to_slurm_format(time_str: str) → str[source]¶

Structure the time string to a SLURM time string.

Parameters

time_strstr: Must be on the format >>> ‘hh:mm:ss’ or >>> ‘d-hh:mm:ss’

Returns

str: The time string formatted as >>> ‘hh:mm:ss’

submit_command(command: str) → None¶

Submit a command.

Parameters

commandstr: Command to submit

Notes

All submitted jobs are held Release with self.release See [1] for details

References

1: https://community.openpbs.org/t/ignoring-finished-dependencies/1976

wait_until_completed(raise_error: bool = True) → None¶

Wait until the process has completed.

Parameters

raise_errorbool: Whether or not to raise errors

property waiting_for¶

Return the waiting for list as a tuple.

Returns

tuple of str: The waiting for list as a tuple

static write_python_script(path: pathlib.Path, function: Callable, args: Optional[Tuple[Any, …]] = None, kwargs: Optional[Dict[str, Any]] = None) → None¶

Write python function to file.

Parameters

pathPath: Absolute path to store the python file which holds the function and its arguments
functionfunction: The function to call
argstuple: The positional arguments
kwargsdict: The keyword arguments