flowserv.volume.base module

Base classes for workflow runtime storage volumes.

class flowserv.volume.base.IOBuffer(buf: IO)

Bases: flowserv.volume.base.IOHandle

Implementation of the file object interface for bytes IO buffers.

open() → IO

Get the associated BytesIO buffer.

Return type

io.BytesIO

size() → int

Get size of the file in the number of bytes.

Return type

int

class flowserv.volume.base.IOHandle

Bases: object

Wrapper around different file objects (i.e., files on disk or files in object stores). Provides functionality to load file content as a bytes buffer and to write file contents to disk.

abstract open() → IO

Get file contents as a BytesIO buffer.

Return type

io.BytesIO

Raises

flowserv.error.UnknownFileError –

abstract size() → int

Get size of the file in the number of bytes.

Return type

int

class flowserv.volume.base.StorageVolume(identifier: Optional[str] = None)

Bases: object

The runtime storage volume provides access to a file system-like object for storing and retrieving files and folders that are required or produced by a workflow step.

Storage volumes are used to provide a copy of the required run files for a workflow step. Each volume has a unique identifier that is used to keep track which files and file versions are available in the volume.

abstract close()

Close any open connection and release all resources when workflow execution is done.

copy(src: Union[str, List[str]], store: flowserv.volume.base.StorageVolume, dst: Optional[str] = None, verbose: Optional[bool] = False) → List[str]

Copy the file or folder at the source path of this storage volume to the given storage volume.

The source path is relative to the base directory for the workflow run.

Returns the list of files that were copied.

Parameters

  • src (string or list of string) – Relative source path(s) for downloaded files and directories.

  • store (flowserv.volume.base.StorageValue) – Storage volume for destination files.

  • dst (string, default=None) – Destination folder for downloaded files.

  • verbose (bool, default=False) – Print information about source and target volume and the files that are being copied.

Return type

list of string

abstract delete(key: str) → int

Delete file or folder with the given key.

Parameters

key (str) – Path to a file object in the storage volume.

abstract describe() → str

Get short descriptive string about the storage volume for display purposes.

Return type

str

abstract erase()

Erase the storage volume base directory and all its contents.

abstract get_store_for_folder(key: str, identifier: Optional[str] = None) → flowserv.volume.base.StorageVolume

Get storage volume for a sob-folder of the given volume.

Parameters

  • key (string) – Relative path to sub-folder. The concatenation of the base folder for this storage volume and the given key will form te new base folder for the returned storage volume.

  • identifier (string, default=None) – Unique volume identifier.

Return type

flowserv.volume.base.StorageVolume

abstract load(key: str) → flowserv.volume.base.IOHandle

Load a file object at the source path of this volume store.

Returns a file handle that can be used to open and read the file.

Parameters

key (str) – Path to a file object in the storage volume.

Return type

flowserv.volume.base.IOHandle

abstract mkdir(path: str)

Create the directory with the given (relative) path and all of its parent directories.

Does not raise an error if the directory exists.

Parameters

path (string) – Relative path to a directory in the storage volume.

abstract store(file: flowserv.volume.base.IOHandle, dst: str)

Store a given file object at the destination path of this volume store.

Parameters

  • file (flowserv.volume.base.IOHandle) – File-like object that is being stored.

  • dst (str) – Destination path for the stored object.

abstract to_dict() → Dict

Get dictionary serialization for the storage volume.

The returned serialization can be used by the volume factory to generate a new instance of this volume store.

Return type

dict

abstract walk(src: str) → List[Tuple[str, flowserv.volume.base.IOHandle]]

Get list of all files at the given source path.

If the source path references a single file the returned list will contain a single entry. If the source specifies a folder the result contains a list of all files in that folder and the subfolders.

Parameters

src (str) – Source path specifying a file or folder.

Return type

list of tuples (str, flowserv.volume.base.IOHandle)

flowserv.volume.base.copy_files(src: Union[str, List[str]], source: flowserv.volume.base.StorageVolume, dst: str, target: flowserv.volume.base.StorageVolume, verbose: Optional[bool] = False) → List[str]

Copy files and folders at the source path (path) of a given source storage volume to the destination path (path) of a target storage volume.

Returns the list of files that were copied.

Parameters

  • src (str or list of string) – Path specifying the source file(s) or folder(s).

  • source (flowserv.volume.base.StorageValue) – Storage volume for source files.

  • dst (string) – Destination path for copied files.

  • target (flowserv.volume.base.StorageValue) – Storage volume for destination files.

  • verbose (bool, default=False) – Print information about source and target volume and the files that are being copied.

Return type

list of string