flowserv.volume.gc module

Implementation of the flowserv.volume.base.GCVolume for the use of Google Cloud File Store buckets.

For testing the GCBucket the Google Cloud credentials have to be configured. Set up authentication by creating a service account and setting the environment variable GOOGLE_APPLICATION_CREDENTIALS. See the documentation for more details: https://cloud.google.com/storage/docs/reference/libraries#setting_up_authentication

flowserv.volume.gc.GCBucket(bucket: str, prefix: Optional[str] = None, identifier: Optional[str] = None) → Dict

Get configuration object for Google Cloud storage volume.

Parameters

  • bucket (string) – Google Cloud Storage bucket identifier.

  • prefix (string, default=None) – Key-prefix for all files.

  • identifier (string, default=None) – Optional storage volume identifier.

Return type

dict

class flowserv.volume.gc.GCFile(client: flowserv.volume.gc.GCClient, bucket_name: str, key: str)

Bases: flowserv.volume.base.IOHandle

Implementation of the file object interface for files that are stored on Google Cloud Storage buckets.

open() → IO

Get file contents as a BytesIO buffer.

Return type

io.BytesIO

size() → int

Get size of the file in the number of bytes.

Return type

int

class flowserv.volume.gc.GCVolume(bucket_name: str, prefix: Optional[str] = None, identifier: Optional[str] = None)

Bases: flowserv.volume.base.StorageVolume

Implementation of the storage volume class for Google Cloud File Store buckets.

close()

The Google Cloud client resource does not need to be closed.

delete(key: str)

Delete file or folder with the given key.

Parameters

key (str) – Path to a file object in the storage volume.

delete_objects(keys: Iterable[str])

Delete objects with the given identifier.

Parameters

keys (iterable of string) – Unique identifier for objects that are being deleted.

describe() → str

Get short descriptive string about the storage volume for display purposes.

Return type

str

erase()

Erase the storage volume base directory and all its contents.

static from_dict(doc) → flowserv.volume.gc.GCVolume

Get Google Cloud storage volume instance from dictionary serialization.

Parameters

doc (dict) – Dictionary serialization as returned by the to_dict() method.

Return type

flowserv.volume.gc.GCVolume

get_store_for_folder(key: str, identifier: Optional[str] = None) → flowserv.volume.base.StorageVolume

Get storage volume for a sob-folder of the given volume.

Parameters

  • key (string) – Relative path to sub-folder. The concatenation of the base folder for this storage volume and the given key will form te new base folder for the returned storage volume.

  • identifier (string, default=None) – Unique volume identifier.

Return type

flowserv.volume.base.StorageVolume

load(key: str) → flowserv.volume.base.IOHandle

Load a file object at the source path of this volume store.

Returns a file handle that can be used to open and read the file.

Parameters

key (str) – Path to a file object in the storage volume.

Return type

flowserv.volume.base.IOHandle

mkdir(path: str)

Create the directory with the given (relative) path and all of its parent directories.

For bucket stores no directories need to be created prior to accessing them.

Parameters

path (string) – Relative path to a directory in the storage volume.

query(filter: str) → Iterable[str]

Get identifier for objects that match a given prefix.

Parameters

filter (str) – Prefix query for object identifiers.

Return type

iterable of string

store(file: flowserv.volume.base.IOHandle, dst: str)

Store a given file object at the destination path of this volume store.

Parameters

  • file (flowserv.volume.base.IOHandle) – File-like object that is being stored.

  • dst (str) – Destination path for the stored object.

to_dict() → Dict

Get dictionary serialization for the storage volume.

The returned serialization can be used by the volume factory to generate a new instance of this volume store.

Return type

dict

walk(src: str) → List[Tuple[str, flowserv.volume.base.IOHandle]]

Get list of all files at the given source path.

If the source path references a single file the returned list will contain a single entry. If the source specifies a folder the result contains a list of all files in that folder and the subfolders.

Parameters

src (str) – Source path specifying a file or folder.

Return type

list of tuples (str, flowserv.volume.base.IOHandle)

flowserv.volume.gc.GC_STORE = 'gc'

Type alias for Google Cloud storage bucket objects.

flowserv.volume.gc.get_google_client()

Helper method to get instance of the Google Cloud Storage client. This method was added to support mocking for unit tests.