flowserv.volume.manager module

Manager for storage volumes. The volume manager is associated with a workflow run. It maintains information about the files and directories that are available to the workflow run in the virtual workflow environment.

flowserv.volume.manager.DefaultVolume(basedir: str) → flowserv.volume.manager.VolumeManager

Helper method to create a volume manager with a single file system store as the default store.

Parameters

basedir (str) – Base directory for the created file system store.

Return type

flowserv.volume.manager.VolumeManager

class flowserv.volume.manager.VolumeManager(stores: List[Dict], files: Optional[Dict[str, List[str]]] = None)

Bases: object

The volume manager maintains information about storage volumes and the files that are available to the workers during workflow execution at each volume. The volume manager is the main component that maintains a virtual runtime environment in which all workers have access to their required input files.

The manager also acts as a factory for volume stores. When the manager is instantiated all storage volumes are specified via their dictionary serialization. The respective volume instances are only created when they are first accessed.

get(identifier: str) → flowserv.volume.base.StorageVolume

Get the instance for the storage volume with the given identifier.

identifier: str

Unique storage volume identifier.

Return type

flowserv.volume.base.StorageVolume

prepare(store: flowserv.volume.base.StorageVolume, inputs: List[str], outputs: List[str])

Prepare the storage volume for a worker.

Ensures that the input files that are needed by the worker are available in their latest version at the given volume store.

Raises a ValueError if a specified input file does not exist.

Parameters

  • store (flowserv.volume.base.StorageVolume) – Storage volume that is being prepared.

  • inputs (list of string) – Relative path (keys) of required input files for a workflow step.

  • outputs (list of string) – Relative path (keys) of created output files by a workflow step.

update(store: flowserv.volume.base.StorageVolume, files: List[str])

Update the availability index for workflow files.

The update method is used by a worker to signal the successful execution of a workflow step. The given files specify the output files that were generated by the worker. The store identifier references the volume store that now contains the latest version of these files.

Raises a ValueError if the specified storage volume does not exist.

Parameters

  • store (flowserv.volume.base.StorageVolume) – Storage volume that contains the latest versions for the given files.

  • files (list of str) – List of relative path (keys) for output files that were generated by a successful workflow step.

flowserv.volume.manager.exact_match(s1: str, s2: str) → bool

Test if two strings are exact matches.

Parameters

  • s1 (string) – Left side string of the comparison.

  • s2 (string) – Right side string of the comparison.

Return type

bool

flowserv.volume.manager.prefix_match(value: str, prefix: str) → bool

Test of the given string value starts with a given prefix.

Parameters

  • value (str) – Value for which the prefix is evaluated.

  • prefix (str) – Prefix string that is tested for the given value.

Return type

bool