flowserv.volume.s3 module
Implementation of the flowserv.volume.base.GCVolume for the
use of AWS S3 buckets.
When using the S3Bucket the AWS credentials have to be configured. See the documentation for more details: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
- flowserv.volume.s3.S3Bucket(bucket: str, prefix: Optional[str] = None, identifier: Optional[str] = None) Dict
Type identifier for storage volume serializations.
- class flowserv.volume.s3.S3File(bucket: flowserv.volume.s3.S3Bucket, key: str)
Bases:
flowserv.volume.base.IOHandleImplementation of the file object interface for files that are stored on S3 object bucktes.
- open() IO
Get file contents as a BytesIO buffer.
- Return type
io.BytesIO
- size() int
Get size of the file in the number of bytes.
- Return type
int
- class flowserv.volume.s3.S3Volume(bucket_id: str, prefix: Optional[str] = None, identifier: Optional[str] = None)
Bases:
flowserv.volume.base.StorageVolumeImplementation of the bucket interface for AWS S3 buckets.
- close()
The AWS S3 bucket resource does not need to be closed.
- delete(key: str)
Delete file or folder with the given key.
- Parameters
key (str) – Path to a file object in the storage volume.
- describe() str
Get short descriptive string about the storage volume for display purposes.
- Return type
str
- erase()
Erase the storage volume base directory and all its contents.
- static from_dict(doc) flowserv.volume.s3.S3Volume
Get S3 bucket storage volume instance from dictionary serialization.
- Parameters
doc (dict) – Dictionary serialization as returned by the
to_dict()method.- Return type
- get_store_for_folder(key: str, identifier: Optional[str] = None) flowserv.volume.base.StorageVolume
Get storage volume for a sob-folder of the given volume.
- Parameters
key (string) – Relative path to sub-folder. The concatenation of the base folder for this storage volume and the given key will form te new base folder for the returned storage volume.
identifier (string, default=None) – Unique volume identifier.
- Return type
- load(key: str) flowserv.volume.base.IOHandle
Load a file object at the source path of this volume store.
Returns a file handle that can be used to open and read the file.
- Parameters
key (str) – Path to a file object in the storage volume.
- Return type
- mkdir(path: str)
Create the directory with the given (relative) path and all of its parent directories.
For bucket stores no directories need to be created prior to accessing them.
- Parameters
path (string) – Relative path to a directory in the storage volume.
- query(filter: str) Iterable[str]
Get identifier for objects that match a given prefix.
- Parameters
filter (str) – Prefix query for object identifiers.
- Return type
iterable of string
- store(file: flowserv.volume.base.IOHandle, dst: str)
Store a given file object at the destination path of this volume store.
- Parameters
file (flowserv.volume.base.IOHandle) – File-like object that is being stored.
dst (str) – Destination path for the stored object.
- to_dict() Dict
Get dictionary serialization for the storage volume.
The returned serialization can be used by the volume factory to generate a new instance of this volume store.
- Return type
dict
- walk(src: str) List[Tuple[str, flowserv.volume.base.IOHandle]]
Get list of all files at the given source path.
If the source path references a single file the returned list will contain a single entry. If the source specifies a folder the result contains a list of all files in that folder and the subfolders.
- Parameters
src (str) – Source path specifying a file or folder.
- Return type
list of tuples (str, flowserv.volume.base.IOHandle)