parsl.executors.base.ParslExecutor

class parsl.executors.base.ParslExecutor(*, monitoring_messages: Queue[Tuple[MessageType, Dict[str, Any]]] | None = None, run_dir: str = '.', run_id: str | None = None)

Executors are abstractions that represent available compute resources to which you could submit arbitrary App tasks.

This is an abstract base class that only enforces concrete implementations of functionality by the child classes.

Can be used as a context manager. On exit, calls self.shutdown() with no arguments and re-raises any thrown exception.

In addition to the listed methods, a ParslExecutor instance must always have a member field:

label: str - a human readable label for the executor, unique

with respect to other executors.

remote_monitoring_radio: RadioConfig describing how tasks on this executor

should report task resource status

An executor may optionally expose:

storage_access: List[parsl.data_provider.staging.Staging] - a list of staging

providers that will be used for file staging. In the absence of this attribute, or if this attribute is None, then a default value of parsl.data_provider.staging.default_staging will be used by the staging code.

Typechecker note: Ideally storage_access would be declared on executor __init__ methods as List[Staging] - however, lists are by default invariant, not co-variant, and it looks like @typeguard cannot be persuaded otherwise. So if you’re implementing an executor and want to @typeguard the constructor, you’ll have to use List[Any] here.

The DataFlowKernel will set this attribute before calling .start(), if monitoring is enabled:

monitoring_messages: Optional[Queue[TaggedMonitoringMessage]] - an executor

can send messages to the monitoring hub by putting them into this queue.

__init__(*, monitoring_messages: Queue[Tuple[MessageType, Dict[str, Any]]] | None = None, run_dir: str = '.', run_id: str | None = None)

Methods

__init__(*[, monitoring_messages, run_dir, ...])
monitor_resources()	Should resource monitoring happen for tasks on running on this executor?
shutdown()	Shutdown the executor.
start()	Start the executor.
submit(func, resource_specification, *args, ...)	Submit.

Attributes

label
run_dir	Path to the run directory.
run_id	UUID for the enclosing DFK.

label: str = 'undefined'

monitor_resources() → bool

Should resource monitoring happen for tasks on running on this executor?

Parsl resource monitoring conflicts with execution styles which do not directly use a process tree - for example, the ThreadPoolExecutor and the MPIExecutor.

This function allows resource monitoring to be disabled per executor implementation.

property run_dir: str

Path to the run directory.

property run_id: str | None

UUID for the enclosing DFK.

abstractmethod shutdown() → None

Shutdown the executor.

Executors should call super().shutdown() as part of their overridden implementation.

start() → None

Start the executor.

By default, this does nothing, but this method should be overridden to perform any spin-up operations (for example: starting thread pools).

abstractmethod submit(func: Callable, resource_specification: Dict[str, Any], *args: Any, **kwargs: Any) → Future

Submit.

The executor can optionally set a parsl_executor_task_id attribute on the Future that it returns, and in that case, parsl will log a relationship between the executor’s task ID and parsl level try/task IDs.