Small fixes for fsspec backend, Enable fsspec tests for s3fs and smb #247
Conversation
painebot
force-pushed
the
tkp/fsspectests
branch
from
c43feb2 to
93308c2
Compare
painebot
force-pushed
the
tkp/fsspectests
branch
from
93308c2 to
3ae0a9c
Compare
| @@ -85,7 +85,7 @@ jobs: | |||
|
|
|||
| - name: Test | |||
| run: make coverage | |||
| if: matrix.os == 'ubuntu-latest' | |||
| if: matrix.os != 'windows-latest' | |||
There was a problem hiding this comment.
runs local filesystem tests on mac, I validated docker tests (samba and s3) locally but unfortunately no more docker support on mac arm on GHA due to licensing issues.
| # Fix any differences due to root markers | ||
| path = path.replace("//", "/") | ||
|
|
||
| # TODO better carveouts |
There was a problem hiding this comment.
we should consider a more rigorous mechanism for determining backend-specific behavior. i didn't want to just check s3:// because theoretically there could be other registered s3 backends that are not s3fs.
| if PurePosixPath(path).name.startswith("."): | ||
| return True | ||
|
|
||
| # TODO PyFilesystem implementation does more, perhaps we should as well |
| if self._fs.__class__.__name__.startswith("S3"): | ||
| # need to make a file temporarily | ||
| # use the convention of a hidden file | ||
| self._fs.touch(f"{path}/.s3fskeep") |
There was a problem hiding this comment.
folders aren't real in s3, i adopted the convention many s3 backends do which is to force the creation of a "directory" by creating a file. e.g. on backblaze its called .bzEmpty.
| @@ -23,7 +25,8 @@ | |||
|
|
|||
| test_root_osfs = "osfs_local" | |||
|
|
|||
| test_url_s3 = "http://127.0.0.1/" | |||
| # test_url_s3 = "http://127.0.0.1/" | |||
There was a problem hiding this comment.
there was a reason for this at some point, but i validated this against s3 tests on my local mac.
|
|
||
|
|
||
| @pytest.mark.skipif(which("docker") is None, reason="requires docker") |
There was a problem hiding this comment.
see note above, tested locally on mac against docker but cannot use docker in arm macs on GHA.
#238 (comment)
Fixes a few small things, I suspect all of these were because i started from the fsspec implementation that had bitrotted a bit (due to our slowness in getting to it).