Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backend sub module #172

Merged
merged 4 commits into from
Jan 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,18 @@ audbackend
|tests| |coverage| |docs| |python-versions| |license|

Manage file storage on different backends.
At the moment we support:

* Artifactory_ with ``audbackend.Artifactory``
* local file system with ``audbackend.FileSystem``
At the moment we support
the following backends:

* Artifactory_ with ``audbackend.backend.Artifactory``
* local file system with ``audbackend.backend.FileSystem``

And the following interfaces
to access files on a backend:

* unversioned with ``audbackend.interface.Unversioned``
* versioned with ``audbackend.interface.Versioned``

Have a look at the installation_ instructions.

Expand Down
9 changes: 5 additions & 4 deletions audbackend/__init__.py
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
from audbackend import backend
from audbackend import interface
from audbackend.core.api import access
from audbackend.core.api import available
from audbackend.core.api import create
from audbackend.core.api import delete
from audbackend.core.api import register
from audbackend.core.backend import Backend
from audbackend.core.backend.base import Base as Backend # legacy
from audbackend.core.backend.filesystem import FileSystem # legacy
from audbackend.core.errors import BackendError
from audbackend.core.filesystem import FileSystem
from audbackend.core.repository import Repository

# Import optional backends
# Import optional backends (legacy)
try:
from audbackend.core.artifactory import Artifactory
from audbackend.core.backend.artifactory import Artifactory
except ImportError: # pragma: no cover
pass

Expand Down
8 changes: 8 additions & 0 deletions audbackend/backend/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
from audbackend.core.backend.base import Base
from audbackend.core.backend.filesystem import FileSystem

# Import optional backends
try:
from audbackend.core.backend.artifactory import Artifactory
except ImportError: # pragma: no cover
pass
18 changes: 9 additions & 9 deletions audbackend/core/api.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
import typing

from audbackend.core import utils
from audbackend.core.backend import Backend
from audbackend.core.filesystem import FileSystem
from audbackend.core.backend.base import Base
from audbackend.core.backend.filesystem import FileSystem
from audbackend.core.interface.base import Base as Interface
from audbackend.core.interface.versioned import Versioned

Expand All @@ -18,7 +18,7 @@ def _backend(
name: str,
host: str,
repository: str,
) -> Backend:
) -> Base:
r"""Get backend instance."""
if name not in backend_registry:
raise ValueError(
Expand Down Expand Up @@ -89,7 +89,7 @@ def access(

Examples:
>>> access('file-system', 'host', 'repo')
audbackend.core.interface.versioned.Versioned('audbackend.core.filesystem.FileSystem', 'host', 'repo')
audbackend.core.interface.versioned.Versioned('audbackend.core.backend.filesystem.FileSystem', 'host', 'repo')

""" # noqa: E501
backend = _backend(name, host, repository)
Expand All @@ -98,7 +98,7 @@ def access(
return interface(backend, **interface_kwargs)


def available() -> typing.Dict[str, typing.List[Backend]]:
def available() -> typing.Dict[str, typing.List[Base]]:
r"""List available backend instances.

Returns a dictionary with
Expand All @@ -114,7 +114,7 @@ def available() -> typing.Dict[str, typing.List[Backend]]:
>>> list(available())
['artifactory', 'file-system']
>>> available()['file-system'][0]
('audbackend.core.filesystem.FileSystem', 'host', 'repo')
('audbackend.core.backend.filesystem.FileSystem', 'host', 'repo')

""" # noqa: E501
result = {}
Expand Down Expand Up @@ -174,7 +174,7 @@ def create(

Examples:
>>> create('file-system', 'host', 'repository')
audbackend.core.interface.versioned.Versioned('audbackend.core.filesystem.FileSystem', 'host', 'repository')
audbackend.core.interface.versioned.Versioned('audbackend.core.backend.filesystem.FileSystem', 'host', 'repository')

""" # noqa: E501
backend = _backend(name, host, repository)
Expand Down Expand Up @@ -222,7 +222,7 @@ def delete(

def register(
name: str,
cls: typing.Type[Backend],
cls: typing.Type[Base],
):
r"""Register backend class.

Expand All @@ -245,7 +245,7 @@ def register(

# Register optional backends
try:
from audbackend.core.artifactory import Artifactory
from audbackend.core.backend.artifactory import Artifactory
register('artifactory', Artifactory)
except ImportError: # pragma: no cover
pass
Empty file.
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import audeer

from audbackend.core import utils
from audbackend.core.backend import Backend
from audbackend.core.backend.base import Base


def _artifactory_path(
Expand Down Expand Up @@ -110,7 +110,7 @@ def _download(
pbar.update(n_data)


class Artifactory(Backend):
class Artifactory(Base):
r"""Backend for Artifactory.

Looks for the two environment variables
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from audbackend.core.errors import BackendError


class Backend:
class Base:
r"""Backend base class.

Derive from this class to implement a new backend.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@
import audeer

from audbackend.core import utils
from audbackend.core.backend import Backend
from audbackend.core.backend.base import Base


class FileSystem(Backend):
class FileSystem(Base):
r"""Backend for file system.

Args:
Expand Down
8 changes: 4 additions & 4 deletions audbackend/core/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@
import audbackend


class DoctestFileSystem(audbackend.FileSystem):
class DoctestFileSystem(audbackend.backend.FileSystem):

def __repr__(self) -> str:
name = 'audbackend.core.filesystem.FileSystem'
name = 'audbackend.core.backend.filesystem.FileSystem'
return str((name, self.host, self.repository))

def _date(
Expand Down Expand Up @@ -45,7 +45,7 @@ def prepare_docstring_tests(doctest_namespace):

# backend

backend = audbackend.Backend('host', 'repo')
backend = audbackend.backend.Base('host', 'repo')
doctest_namespace['backend'] = backend

interface = audbackend.interface.Base(backend)
Expand Down Expand Up @@ -84,6 +84,6 @@ def prepare_docstring_tests(doctest_namespace):

audbackend.delete('file-system', 'host', 'repo')
audbackend.delete('file-system', 'host', 'repo-unversioned')
audbackend.register('file-system', audbackend.FileSystem)
audbackend.register('file-system', audbackend.backend.FileSystem)

os.chdir(current_dir)
Empty file.
4 changes: 2 additions & 2 deletions audbackend/core/interface/base.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import typing

from audbackend.core.backend import Backend
from audbackend.core.backend.base import Base as Backend


class Base:
Expand Down Expand Up @@ -38,7 +38,7 @@ def backend(self) -> Backend:

Examples:
>>> interface.backend
('audbackend.core.backend.Backend', 'host', 'repo')
('audbackend.core.backend.base.Base', 'host', 'repo')

"""
return self._backend
Expand Down
3 changes: 1 addition & 2 deletions audbackend/core/interface/versioned.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
import audeer

from audbackend.core import utils
from audbackend.core.backend import Backend
from audbackend.core.errors import BackendError
from audbackend.core.interface.base import Base

Expand All @@ -20,7 +19,7 @@ class Versioned(Base):

def __init__(
self,
backend: Backend,
backend: Base,
):
super().__init__(backend)

Expand Down
24 changes: 24 additions & 0 deletions docs/api-src/audbackend.backend.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
audbackend.backend
==================

.. automodule:: audbackend.backend

Currently the following
backends are supported:

.. autosummary::
:toctree:
:nosignatures:

Artifactory
FileSystem

Users can implement their own
backend by deriving from
:class:`audbackend.backend.Base`.

.. autosummary::
:toctree:
:nosignatures:

Base
5 changes: 1 addition & 4 deletions docs/api-src/audbackend.interface.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,7 @@ audbackend.interface
.. automodule:: audbackend.interface

To access the files on a backend users
can choose between two interfaces
:class:`audbackend.interface.Unversioned`
or
:class:`audbackend.interface.Versioned`:
can use the following interfaces:

.. autosummary::
:toctree:
Expand Down
34 changes: 14 additions & 20 deletions docs/api-src/audbackend.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,32 +3,26 @@ audbackend

.. automodule:: audbackend

A backend is a generic interface
:mod:`audbackend`
provides an abstract layer
for storing and accessing files
on a host.

Currently the following backends
are shipped with :mod:`audbackend`:
This involves two components:

.. autosummary::
:toctree:
:nosignatures:

Artifactory
FileSystem

Users can implement their own
backend by deriving from
:class:`audbackend.Backend`.

.. autosummary::
:toctree:
:nosignatures:
1. A *backend* that
implements file operations
on a specific storing device
(:mod:`audbackend.backend`).

Backend
2. An *interface* that
passes user requests
to a backend
(:mod:`audbackend.interface`).

In addition to the backends
the following classes and functions are available.
Additionally,
the following classes
and functions are available.

.. autosummary::
:toctree:
Expand Down
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
:hidden:

api/audbackend
api/audbackend.backend
api/audbackend.interface
genindex

Expand Down
5 changes: 3 additions & 2 deletions docs/install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,16 @@ To install :mod:`audbackend` run:
$ pip install audbackend

By default,
only the :class:`audbackend.FileSystem` backend will be installed.
only the :class:`audbackend.backend.FileSystem`
backend will be installed.
To install all backends run:

.. code-block:: bash

$ pip install audbackend[all]

or select single backends,
e.g. :class:`audbackend.Artifactory`
e.g. :class:`audbackend.backend.Artifactory`

.. code-block:: bash

Expand Down
Loading