flowserv.controller.worker.docker module

Implementation of a workflow step engine that uses the local Docker daemon to execute workflow steps.

class flowserv.controller.worker.docker.DockerWorker(variables: Optional[Dict] = None, env: Optional[Dict] = None, identifier: Optional[str] = None, volume: Optional[str] = None)

Bases: flowserv.controller.worker.base.ContainerWorker

Container step engine that uses the local Docker deamon to execute the commands in a workflow step.

run(step: flowserv.model.workflow.step.ContainerStep, env: Dict, rundir: str) → flowserv.controller.serial.workflow.result.ExecResult

Execute a list of commands from a workflow steps synchronously using the Docker engine.

Stops execution if one of the commands fails. Returns the combined result from all the commands that were executed.

Parameters

  • step (flowserv.controller.serial.workflow.ContainerStep) – Step in a serial workflow.

  • env (dict, default=None) – Default settings for environment variables when executing workflow steps. May be None.

  • rundir (string) – Path to the working directory of the workflow run that this step belongs to.

Return type

flowserv.controller.serial.workflow.result.ExecResult

flowserv.controller.worker.docker.NOTEBOOK_DOCKER_WORKER = 'nbdocker'

Default Dockerfile for created papermill containers.

class flowserv.controller.worker.docker.NotebookDockerWorker(env: Optional[Dict] = None, identifier: Optional[str] = None, volume: Optional[str] = None)

Bases: flowserv.controller.worker.base.Worker

Execution engine for notebook steps in a serial workflow.

exec(step: flowserv.model.workflow.step.NotebookStep, context: Dict, store: flowserv.volume.fs.FileSystemStorage) → flowserv.controller.serial.workflow.result.ExecResult

Execute a given notebook workflow step in the current workflow context.

The notebook engine expects a file system storage volume that provides access to the notebook file and any other aditional input files.

Parameters

  • step (flowserv.model.workflow.step.NotebookStep) – Notebook step in a serial workflow.

  • context (dict) – Dictionary of variables that represent the current workflow state.

  • store (flowserv.volume.fs.FileSystemStorage) – Storage volume that contains the workflow run files.

Return type

flowserv.controller.serial.workflow.result.ExecResult

flowserv.controller.worker.docker.docker_build(name: str, requirements: List[str]) → Tuple[str, List[str]]

Build a Docker image from a standard Python image with papermill and the given requirements installed.

Returns the identifier of the created image.

Parameters

  • name (string) – Name for the created image (derived from the workflow step name).

  • requirements (list of string) – List of requirements that will be written to a file requirements.txt and installed inside the created Docker image.

Return type

string, list of string

flowserv.controller.worker.docker.docker_run(image: str, commands: List[str], env: Dict, rundir: str, result: flowserv.controller.serial.workflow.result.ExecResult) → flowserv.controller.serial.workflow.result.ExecResult

Helper function that executes a list of commands inside a Docker container.

Parameters

  • image (string) – Identifier of the Docker image to run.

  • commands (string or list of string) – Commands that are executed inside the Docker container.

  • result (flowserv.controller.serial.workflow.result.ExecResult) – Result object that will contain the run outputs and status code.

Return type

flowserv.controller.serial.workflow.result.ExecResult