flowserv.volume.fs module

File system workflow storage volume. Maintains workflow run files in a folder on the local file system.

class flowserv.volume.fs.FSFile(filename: str, raise_error: Optional[bool] = True)

Bases: flowserv.volume.base.IOHandle

Implementation of the IO object handle interface for files that are stored on the file system.

open() → IO

Get file contents as a BytesIO buffer.

Return type

io.BytesIO

Raises

flowserv.error.UnknownFileError –

size() → int

Get size of the file in the number of bytes.

Return type

int

flowserv.volume.fs.FStore(basedir: str, identifier: Optional[str] = None) → Dict

Get configuration object for a file system storage volume.

Parameters

  • basedir (string) – Google Cloud Storage bucket identifier.

  • identifier (string, default=None) – Optional storage volume identifier.

Return type

dict

class flowserv.volume.fs.FileSystemStorage(basedir: str, identifier: Optional[str] = None)

Bases: flowserv.volume.base.StorageVolume

The file system storage volume provides access to workflow run files that are maintained in a run directory on the local file system.

close()

The file system runtime manager has no connections to close or resources to release.

delete(key: str)

Delete file or folder with the given key.

Parameters

key (str) – Path to a file object in the storage volume.

describe() → str

Get short descriptive string about the storage volume for display purposes.

Return type

str

erase()

Erase the storage volume base directory and all its contents.

static from_dict(doc) → flowserv.volume.fs.FileSystemStorage

Get file system storage volume instance from dictionary serialization.

Parameters

doc (dict) – Dictionary serialization as returned by the to_dict() method.

Return type

flowserv.volume.fs.FileSystemStorage

get_store_for_folder(key: str, identifier: Optional[str] = None) → flowserv.volume.base.StorageVolume

Get storage volume for a sob-folder of the given volume.

Parameters

  • key (string) – Relative path to sub-folder. The concatenation of the base folder for this storage volume and the given key will form te new base folder for the returned storage volume.

  • identifier (string, default=None) – Unique volume identifier.

Return type

flowserv.volume.base.StorageVolume

load(key: str) → flowserv.volume.base.IOHandle

Load a file object at the source path of this volume store.

Returns a file handle that can be used to open and read the file.

Parameters

key (str) – Path to a file object in the storage volume.

Return type

flowserv.volume.base.IOHandle

mkdir(path: str)

Create the directory with the given (relative) path and all of its parent directories.

Does not raise an error if the directory exists.

Parameters

path (string) – Relative path to a directory in the storage volume.

path(*args) → pathlib.Path

Get a file system path object for a file or directory that is given by a list of path components relative to the base directory of the storage volume.

Parameters

args (list of string) – List of path components that are joined with the base directory of the storage volume to generate the path object.

Return type

pathlib.Path

store(file: flowserv.volume.base.IOHandle, dst: str)

Store a given file object at the destination path of this volume store.

Parameters

  • file (flowserv.volume.base.IOHandle) – File-like object that is being stored.

  • dst (str) – Destination path for the stored object.

to_dict() → Dict

Get dictionary serialization for the storage volume.

The returned serialization can be used by the volume factory to generate a new instance of this volume store.

Return type

dict

walk(src: str) → List[Tuple[str, flowserv.volume.base.IOHandle]]

Get list of all files at the given source path.

If the source path references a single file the returned list will contain a single entry. If the source specifies a folder the result contains a list of all files in that folder and the subfolders.

Parameters

src (str) – Source path specifying a file or folder.

Return type

list of tuples (str, flowserv.volume.base.IOHandle)

flowserv.volume.fs.walkdir(dirname: str, prefix: str, files: List[Tuple[str, flowserv.volume.base.IOHandle]]) → List[Tuple[str, flowserv.volume.base.IOHandle]]

Recursively add all files in a given source folder to a file upload list. The elements in the list are tuples of file object and relative target path.

Parameters

  • dirname (string) – Path to folder of the local file system.

  • prefix (string) – Relative destination path for all files in the folder.

  • files (list of (string, flowserv.volume.base.IOHandle)) – Pairs of file objects and their relative target path for upload to a file store.