Allow client to interact with remote PBS server

[bschroeter]

The current client is designed for local execution, however, there is a solid case for making it interact with a remote server.

This would likely be implemented like this...

client = ClientFactory().get_client(host="xxx.xxx.xxx.xxx") # where the host will default to localhost

Other considerations:

• There would need to be a private method that is triggered to test the SSH connection on instantiation.
• The remote host would need to be assessed for the appropriate client type based on the available scheduler.
• The job script would need to be copied to the remote server prior to execution
• The scheduler commands and directives will need to be safely inserted into a remote SSH command.

This is a reasonable package of work. Target v2.0.

[access-hive-bot]

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/poster-connecting-evaluation-tools-across-the-lsm-community-me-org-client-hpcpy/3485/12

[frizwi]

Hi Ben, you may find this package useful https://www.fabfile.org/ for the ssh connections. I use this in scripts to submit and monitor jobs on Gadi. It has very occasional timeout issues but mostly works really well.

[bschroeter]

Hi Ben, you may find this package useful https://www.fabfile.org/ for the ssh connections. I use this in scripts to submit and monitor jobs on Gadi. It has very occasional timeout issues but mostly works really well.

Excellent suggestion. Thanks!

[bschroeter]

Hi @frizwi, I'm going through the backlog of issues on this one and having some trouble getting the Connection object to authenticate with Gadi.

Are you able to share a snippet of your code (redacted of course) with which you've been able to establish a connection?

Thanks in advance.

[bschroeter]

Hi @frizwi, I'm going through the backlog of issues on this one and having some trouble getting the Connection object to authenticate with Gadi.

Are you able to share a snippet of your code (redacted of course) with which you've been able to establish a connection?

Thanks in advance.

Disregard, I've figured it out. :)

[bschroeter]

For future reference when we get up to this bit:

from paramiko.client import SSHClient
from paramiko import RSAKey
from pathlib import Path
import os

nci_user = os.getenv("NCI_USER")
nci_host = os.getenv("NCI_HOST")

key_file = Path.home() / ".ssh/id_rsa"
known_hosts = Path.home() / ".ssh/known_hosts"

pkey = RSAKey.from_private_key_file(key_file)
client = SSHClient()
client.load_host_keys(filename=known_hosts)

client.connect(
    nci_host,
    pkey=pkey,
    look_for_keys=False,
    allow_agent=False,
    username=nci_user,
)

stdin, stdout, stderr = client.exec_command("echo $HOME")

print(stdout.read())

I assume we can pass similar details into the connect_kwargs of fabric to achieve a higher-level abstraction.

Then, from within the base client class of HPCPy, we can intercept the shell call to execute over ssh instead.

Theoretically...