flowserv.volume.ssh module

Workflow storage manager that uses a SSH client to connect to a remote server where run files are maintained.

class flowserv.volume.ssh.RemoteStorage(client: flowserv.util.ssh.SSHClient, remotedir: str, identifier: Optional[str] = None)

Bases: flowserv.volume.base.StorageVolume

File storage volume that connects to a remote server via sftp.

close()

Close the SSH connection when workflow execution is done.

delete(key: str)

Delete file or folder with the given key.

Parameters

key (str) – Path to a file object in the storage volume.

describe() → str

Get short descriptive string about the storage volume for display purposes.

Return type

str

erase()

Erase the storage volume base directory and all its contents.

static from_dict(doc) → flowserv.volume.ssh.RemoteStorage

Get remote storage volume instance from dictionary serialization.

Parameters

doc (dict) – Dictionary serialization as returned by the to_dict() method.

Return type

flowserv.volume.ssh.RemoteStorage

get_store_for_folder(key: str, identifier: Optional[str] = None) → flowserv.volume.base.StorageVolume

Get storage volume for a sob-folder of the given volume.

Parameters

  • key (string) – Relative path to sub-folder. The concatenation of the base folder for this storage volume and the given key will form te new base folder for the returned storage volume.

  • identifier (string, default=None) – Unique volume identifier.

Return type

flowserv.volume.base.StorageVolume

load(key: str) → flowserv.volume.base.IOHandle

Load a file object at the source path of this volume store.

Returns a file handle that can be used to open and read the file.

Parameters

key (str) – Path to a file object in the storage volume.

Return type

flowserv.volume.base.IOHandle

mkdir(path: str)

Create the directory with the given (relative) path and all of its parent directories.

For bucket stores no directories need to be created prior to accessing them.

Parameters

path (string) – Relative path to a directory in the storage volume.

store(file: flowserv.volume.base.IOHandle, dst: str)

Store a given file object at the destination path of this volume store.

Parameters

  • file (flowserv.volume.base.IOHandle) – File-like object that is being stored.

  • dst (str) – Destination path for the stored object.

to_dict() → Dict

Get dictionary serialization for the storage volume.

The returned serialization can be used by the volume factory to generate a new instance of this volume store.

Return type

dict

walk(src: str) → List[Tuple[str, flowserv.volume.base.IOHandle]]

Get list of all files at the given source path.

If the source path references a single file the returned list will contain a single entry. If the source specifies a folder the result contains a list of all files in that folder and the subfolders.

Parameters

src (str) – Source path specifying a file or folder.

Return type

list of tuples (str, flowserv.volume.base.IOHandle)

class flowserv.volume.ssh.SFTPFile(filename: str, client: flowserv.util.ssh.SSHClient)

Bases: flowserv.volume.base.IOHandle

Implementation of the IO object handle interface for files that are stored on a remote file system.

open() → IO

Get file contents as a BytesIO buffer.

Return type

io.BytesIO

Raises

flowserv.error.UnknownFileError –

size() → int

Get size of the file in the number of bytes.

Return type

int

flowserv.volume.ssh.Sftp(remotedir: str, hostname: str, port: Optional[int] = None, timeout: Optional[float] = None, look_for_keys: Optional[bool] = False, sep: Optional[str] = '/', identifier: Optional[str] = None) → Dict

Get configuration object for a remote server storage volume that is accessed via sftp.

Parameters

  • remotedir (string) – Base directory for stored files on the remote server.

  • hostname (string) – Server to connect to.

  • port (int, default=None) – Server port to connect to.

  • timeout (float, default=None) – Optional timeout (in seconds) for the TCP connect.

  • look_for_keys (bool, default=False) – Set to True to enable searching for discoverable private key files in ~/.ssh/.

  • sep (string, default=’/’) – Path separator used by the remote file system.

  • identifier (string, default=None) – Unique storage volume identifier.

Return type

dict

flowserv.volume.ssh.sftp_mkdir(client: paramiko.sftp_client.SFTPClient, dirpath: str)

Create a directory on the remote server.

client: paramiko.SFTPClient

SFTP client.

dirpath: string

Path to the created directory on the remote server.