Secure FTP

[pohmelie]

After #36 I read about FTPS, SFTP and FTP over SSH.

• SFTP is SSH extension and we should ignore it

• FTPS is extension of FTP (https://tools.ietf.org/html/rfc2228)

• FTP over SSH is (as I realized) just tunneling and have problems(?) with data connections

• implicit ftps mode (Add implicit FTPS support #81)

• explicit ftps mode

[rsichnyi]

FTP over SSH

yes, it's just tunneling...

i was thinking of FTPS - actually this might be quite simple, but as usually the main problem is finding free time for doing things (especially with a full-time job)

[creatorrr]

👍 Any updates on this? Happy to help out.

[pohmelie]

@creatorrr, I tried out ssl module:

import asyncio
import ssl


req = b"GET / HTTP/1.1\r\nHost: www.python.org\r\n\r\n"


async def flush(outgoing, writer):
    if outgoing.pending:
        writer.write(outgoing.read())
        await writer.drain()


async def foo():
    BLOCK_SIZE = 8192
    # ssl_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
    ssl_context = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH)
    ssl_context.check_hostname = False

    incoming = ssl.MemoryBIO()
    outgoing = ssl.MemoryBIO()
    ssl_object = ssl_context.wrap_bio(incoming, outgoing)

    reader, writer = await asyncio.open_connection("python.org", 443)
    while True:
        try:
            ssl_object.do_handshake()
            break
        except ssl.SSLWantReadError:
            await flush(outgoing, writer)
            incoming.write(await reader.read(BLOCK_SIZE))

    ssl_object.write(req)
    await flush(outgoing, writer)
    data = None
    while True:
        try:
            data = ssl_object.read(BLOCK_SIZE)
            print(data)
        except ssl.SSLWantReadError:
            await flush(outgoing, writer)

        await flush(outgoing, writer)
        data = await reader.read(BLOCK_SIZE)
        # print("->", data)
        if not data:
            break
        incoming.write(data)

    writer.close()


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(foo())

But it's pretty annoying… also, I was thinking about who will need this, since I did not find any FTPS server online. Feel free to implement this functionality is you have enough courage and time 😉

[creatorrr]

Haha, thanks for looking into this, @pohmelie! I will try and see if I can get things to work and create a patch.

[oleksandr-kuzmenko]

I did not find any FTPS server online

#81 — works with my private FTPS.

cc @pohmelie

[pohmelie]

Unfortunately, there is an asyncio bug, which will come out when your path io is slower than network io. Solution for this (except wait for fix) is own wrapper, like present above.

[markshhsu]

Unfortunately, there is an asyncio bug, which will come out when your path io is slower than network io. Solution for this (except wait for fix) is own wrapper, like present above.

@pohmelie, for the example code you provided above, I import uvloop and the example code can run without any errors, seems uvloop can fix this issue?

[pohmelie]

@markshhsu, it looks like yes.

[Midnighter]

Just to note that I'm failing to connect to an SFTP server.

Trying to form a connection like this:

    async with aioftp.Client.context(
        host=settings.host,
        port=settings.port,
        user=settings.username,
        password=settings.password.get_secret_value(),
        ssl=True,
        socket_timeout=10,
        path_timeout=10,
    ) as client:
        logger.info("Connected.")
        logger.info(str(await client.list()))

which results in this error:

    async with aioftp.Client.context(
  File "~/miniconda3/envs/fetch/lib/python3.8/contextlib.py", line 171, in __aenter__
    return await self.gen.__anext__()
  File "~/miniconda3/envs/fetch/lib/python3.8/site-packages/aioftp/client.py", line 1199, in context
    await client.connect(host, port)
  File "~/miniconda3/envs/fetch/lib/python3.8/site-packages/aioftp/client.py", line 604, in connect
    await super().connect(host, port)
  File "~/miniconda3/envs/fetch/lib/python3.8/site-packages/aioftp/client.py", line 131, in connect
    reader, writer = await self._open_connection(host, port)
  File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/streams.py", line 52, in open_connection
    transport, _ = await loop.create_connection(
  File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "~/miniconda3/envs/fetch/lib/python3.8/ssl.py", line 944, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1124)

If I set the ssl argument to ssl.SSLContext(ssl.PROTOCOL_TLS) then I see a message

Using selector: EpollSelector

but nothing happens and I have to interrupt.

I can connect to the server with the same information when using pysftp and disabling the SSH known hosts check.

    cnx_opts = pysftp.CnOpts()
    cnx_opts.hostkeys = None
    with pysftp.Connection(
        host=settings.host,
        port=settings.port,
        username=settings.username,
        password=settings.password.get_secret_value(),
        cnopts=cnx_opts,
    ) as sftp:
        logger.info("Connected.")
        logger.info(sftp.pwd)
        logger.info(sftp.listdir())

Do you have any advice on what I'm doing wrong?

[pohmelie]

@Midnighter, since you have success with pysftp lib, I think you are trying to do SFTP, but it is unrelated to aioftp and ftp protocol. aioftp have partial FTPS support though. This information is in first post of this issue.

[Midnighter]

🤔 indeed the distinction between SFTP and FTP over SSH (which you demonstrated in your comment) was not clear to me. Thank you for the heads up.

[maulberto3]

FWIW, tried with ssl.create_default_context(), server seems to start, client can't connect...

[pohmelie]

@maulberto3 need more context.

[ghost]

Started working on a raw test implementation for FTPES, but unfortunately didn't get it working. Here is my attempt in case anybody wants to look into it and may have a suggestion or even solution.

[pohmelie]

@pantierra
As a doc said (https://docs.python.org/3/library/asyncio-eventloop.html?highlight=start_tls#asyncio.loop.start_tls)

Return a new transport instance, that the protocol must start using immediately after the await. The transport instance passed to the start_tls method should never be used again.

I think that is the problem

[ghost]

Good point! It looks like we would need for Python v3.11 for this, which includes this PR that makes things much easier: python/cpython#91453.

[antonio-hickey]

Any updates on explicit ftps mode?

[pohmelie]

@antonio-hickey
No updates. It's just on contributors shoulders. I'm not interested in this mode and not ready to try to implement this, since it's PITA, definitely.

[antonio-hickey]

@pohmelie
Ah ok, do you know of anyone in specific working on this? I'd like to help get this feature added.

[pohmelie]

@antonio-hickey
AFAIK nobody even tried. So you can be the first.

[sammichaels]

Hi.

I added a quick implementation of explicit TLS (requires Python >= 3.11): 86a6a8c.

Pass in explicit_tls=True for Client() or Client.context() and the connection will automatically be upgraded prior to login. Alternatively, call client.upgrade_to_tls() at any point to enable TLS. The data channel is automatically TLS encrypted if the command channel has been upgraded. Downgrading back to clear text with CCC or REIN commands is not supported.

For backwards compatibility, passing ssl=True or ssl=yourcontext will continue to do implicit TLS. If you specify both explicit_tls=True and ssl=yourcontext, the TLS upgrade will use your context (specifying ssl=True or leaving it None will use the default context).

Use default SSL context:

aioftp.Client("localhost", explicit_tls=True)

Use custom SSL context to bypass self-signed cert errors:

import ssl
sslcontext = ssl.create_default_context()
sslcontext.check_hostname = False
sslcontext.verify_mode = ssl.CERT_NONE
aioftp.Client("localhost", ssl=sslcontext, explicit_tls=True)

I haven't done much testing other than getting this to work within my own project, and I'm not sure how you'd like to handle the tests for it since you've disabled SSL tests, but I figure I'd start the discussion.

[pohmelie]

@sammichaels, sounds great! I think it is time to jump onto 3.11 and use recent fixes for ssl and your code to allow explicit switch. I will try to migrate project to modern technologies before this (pyproject.toml, black, ruff, bump to 3.11+, etc.) and restore old implicit tests. Then we can move forward with your approach. Thank you!

[sammichaels]

@pohmelie I'll continue updating the fork with my changes as there's much more to do, like error handling and SSL session reuse. Glad to hear you're considering adding explicit support!

[pohmelie]

@sammichaels I moved codebase to pyproject.toml, black, ruff and pre-commit tools. Minimal version bumped to 3.11.

[elpablete]

Hey @sammichaels Need help with the fork? I woul be willing to help so we can make a PR

[sammichaels]

@elpablete Hi!

I guess I've forgotten to actually submit the PR. I've been running TLS for some time now in production. Let me make sure I have the production code committed to the fork and I'll do a PR.