Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: pepkit/pepdbagent
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.7.1
Choose a base ref
...
head repository: pepkit/pepdbagent
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref

Commits on Feb 1, 2024

  1. Fixed pepkit/pephub#283

    khoroshevskyi committed Feb 1, 2024
    Copy the full SHA
    9100a7a View commit details

Commits on Feb 2, 2024

  1. Fixed #121

    khoroshevskyi committed Feb 2, 2024
    Copy the full SHA
    be8df69 View commit details
  2. Added new exceptions

    khoroshevskyi committed Feb 2, 2024
    Copy the full SHA
    82e3c97 View commit details
  3. Merge pull request #123 from pepkit/dev

    release 0.7.2
    khoroshevskyi authored Feb 2, 2024
    Copy the full SHA
    f5257a1 View commit details

Commits on Feb 5, 2024

  1. Fixed pop undate

    khoroshevskyi committed Feb 5, 2024
    Copy the full SHA
    e785fd4 View commit details
  2. Merge pull request #124 from pepkit/dev

    Fixed pop undate
    khoroshevskyi authored Feb 5, 2024
    Copy the full SHA
    149e18c View commit details
  3. Copy the full SHA
    9b489ce View commit details
  4. Copy the full SHA
    10366c6 View commit details

Commits on Feb 6, 2024

  1. Copy the full SHA
    f650134 View commit details

Commits on Feb 7, 2024

  1. Added coverage

    khoroshevskyi committed Feb 7, 2024
    Copy the full SHA
    3a386c2 View commit details
  2. updated actions

    khoroshevskyi committed Feb 7, 2024
    Copy the full SHA
    d77caff View commit details
  3. updated actions

    khoroshevskyi committed Feb 7, 2024
    Copy the full SHA
    7a333f3 View commit details
  4. updated requirements

    khoroshevskyi committed Feb 7, 2024
    Copy the full SHA
    ef7376b View commit details
  5. updated python version

    khoroshevskyi committed Feb 7, 2024
    Copy the full SHA
    ba342ce View commit details
  6. Copy the full SHA
    82bcd20 View commit details
  7. 1 python version

    khoroshevskyi committed Feb 7, 2024
    Copy the full SHA
    9e470bd View commit details
  8. updated key

    khoroshevskyi committed Feb 7, 2024
    Copy the full SHA
    52ad937 View commit details
  9. Added coverage

    khoroshevskyi committed Feb 7, 2024
    Copy the full SHA
    477a8bd View commit details
  10. updated readme

    khoroshevskyi committed Feb 7, 2024
    Copy the full SHA
    916255b View commit details
  11. Copy the full SHA
    034cfde View commit details
  12. updated readme 2

    khoroshevskyi committed Feb 7, 2024
    Copy the full SHA
    d36f82c View commit details

Commits on Feb 8, 2024

  1. pr updates

    Co-authored-by: Donald Campbell <125581724+donaldcampbelljr@users.noreply.github.com>
    khoroshevskyi and donaldcampbelljr committed Feb 8, 2024
    Copy the full SHA
    3cce0a9 View commit details
  2. Merge pull request #126 from pepkit/dev

    Release 0.7.3
    khoroshevskyi authored Feb 8, 2024
    Copy the full SHA
    895eb86 View commit details
  3. updated coverage

    khoroshevskyi committed Feb 8, 2024
    Copy the full SHA
    211d1de View commit details
  4. fixed forking

    khoroshevskyi committed Feb 8, 2024
    Copy the full SHA
    3ac1343 View commit details
  5. updated exceptions

    khoroshevskyi committed Feb 8, 2024
    Copy the full SHA
    2d20f47 View commit details

Commits on Feb 9, 2024

  1. fixed error in stats

    khoroshevskyi committed Feb 9, 2024
    Copy the full SHA
    7d79df9 View commit details

Commits on Feb 23, 2024

  1. fixed 129, 130

    khoroshevskyi committed Feb 23, 2024
    Copy the full SHA
    dc93470 View commit details
  2. Copy the full SHA
    ab42a07 View commit details
  3. updated changelog date

    khoroshevskyi committed Feb 23, 2024
    Copy the full SHA
    a937b06 View commit details
  4. updated requirements

    khoroshevskyi committed Feb 23, 2024
    Copy the full SHA
    74f2fd3 View commit details

Commits on Feb 26, 2024

  1. updated version

    khoroshevskyi committed Feb 26, 2024
    Copy the full SHA
    020b6e3 View commit details
  2. Merge pull request #131 from pepkit/dev

    Release 0.8.0
    khoroshevskyi authored Feb 26, 2024
    Copy the full SHA
    723e2d2 View commit details

Commits on Feb 28, 2024

  1. Update README.md

    khoroshevskyi authored Feb 28, 2024
    Copy the full SHA
    d4431da View commit details

Commits on Jun 12, 2024

  1. some work on efficiency

    khoroshevskyi committed Jun 12, 2024
    Copy the full SHA
    b9b0250 View commit details

Commits on Jun 13, 2024

  1. Copy the full SHA
    64d7840 View commit details

Commits on Jun 14, 2024

  1. some changes

    khoroshevskyi committed Jun 14, 2024
    Copy the full SHA
    23c8f90 View commit details

Commits on Jun 17, 2024

  1. Copy the full SHA
    8260645 View commit details

Commits on Jun 19, 2024

  1. working updates

    khoroshevskyi committed Jun 19, 2024
    Copy the full SHA
    4decc28 View commit details
  2. fixed inserting rows

    khoroshevskyi committed Jun 19, 2024
    Copy the full SHA
    2365fab View commit details

Commits on Jun 20, 2024

  1. work on cleaning

    khoroshevskyi committed Jun 20, 2024
    Copy the full SHA
    c872ec6 View commit details
  2. Copy the full SHA
    7a0a170 View commit details
  3. restructuring

    khoroshevskyi committed Jun 20, 2024
    Copy the full SHA
    8eeaf20 View commit details

Commits on Jun 21, 2024

  1. fixing tests

    khoroshevskyi committed Jun 21, 2024
    Copy the full SHA
    c26e74b View commit details
  2. sample add fix

    khoroshevskyi committed Jun 21, 2024
    Copy the full SHA
    7e3b041 View commit details
  3. fixed all tests

    khoroshevskyi committed Jun 21, 2024
    Copy the full SHA
    a05704c View commit details
  4. added bug handling

    khoroshevskyi committed Jun 21, 2024
    Copy the full SHA
    41c6a95 View commit details

Commits on Jun 24, 2024

  1. Copy the full SHA
    45e52f1 View commit details
  2. cleaning

    khoroshevskyi committed Jun 24, 2024
    Copy the full SHA
    70f94fb View commit details
  3. Copy the full SHA
    3f0c05e View commit details
Showing with 5,531 additions and 1,937 deletions.
  1. +52 −0 .github/workflows/cli-coverage.yml
  2. +3 −6 .github/workflows/pytest.yml
  3. +132 −9 README.md
  4. +0 −140 docs/README.md
  5. +44 −2 docs/changelog.md
  6. +2 −2 docs/db_tutorial.md
  7. +0 −125 manual_tests.py
  8. +1 −0 pepdbagent/__init__.py
  9. +1 −1 pepdbagent/_version.py
  10. +4 −0 pepdbagent/const.py
  11. +196 −20 pepdbagent/db_utils.py
  12. +74 −0 pepdbagent/exceptions.py
  13. +118 −3 pepdbagent/models.py
  14. +39 −19 pepdbagent/modules/annotation.py
  15. +197 −29 pepdbagent/modules/namespace.py
  16. +791 −142 pepdbagent/modules/project.py
  17. +38 −34 pepdbagent/modules/sample.py
  18. +641 −0 pepdbagent/modules/schema.py
  19. +26 −10 pepdbagent/modules/user.py
  20. +75 −67 pepdbagent/modules/view.py
  21. +24 −17 pepdbagent/pepdbagent.py
  22. +69 −3 pepdbagent/utils.py
  23. +2 −1 requirements/requirements-all.txt
  24. +3 −1 requirements/requirements-dev.txt
  25. +4 −3 setup.py
  26. +3 −3 tests/README.md
  27. +83 −84 tests/conftest.py
  28. +69 −0 tests/schemas/namespace1/2.0.0.yaml
  29. +77 −0 tests/schemas/namespace1/2.1.0.yaml
  30. +47 −0 tests/schemas/namespace2/bedboss.yaml
  31. +25 −0 tests/schemas/namespace2/bedbuncher.yaml
  32. +59 −0 tests/schemas/namespace2/bedmaker.yaml
  33. +325 −0 tests/test_annotation.py
  34. +195 −0 tests/test_namespace.py
  35. +0 −1,216 tests/test_pepagent.py
  36. +301 −0 tests/test_project.py
  37. +305 −0 tests/test_project_history.py
  38. +235 −0 tests/test_samples.py
  39. +223 −0 tests/test_schema.py
  40. +49 −0 tests/test_tar_meta.py
  41. +628 −0 tests/test_updates.py
  42. +231 −0 tests/test_views.py
  43. +140 −0 tests/utils.py
52 changes: 52 additions & 0 deletions .github/workflows/cli-coverage.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: test coverage

on:
push:
branches: [master, dev]

jobs:
cli-coverage-report:
strategy:
matrix:
python-version: [ "3.11" ]
os: [ ubuntu-latest ] # can't use macOS when using service containers or container jobs
r: [ release ]
runs-on: ${{ matrix.os }}
services:
postgres:
image: postgres
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: pass8743hf9h23f87h437
POSTGRES_DB: pep-db
POSTGRES_HOST: localhost
ports:
- 5432:5432
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5

steps:
- uses: actions/checkout@v2

- uses: actions/setup-python@v1
with:
python-version: '3.10'

- name: Install dev dependancies
run: if [ -f requirements/requirements-dev.txt ]; then pip install -r requirements/requirements-dev.txt; fi

- run: pip install .

- name: Run tests
run: coverage run -m pytest

- name: build coverage
run: coverage html -i

- run: smokeshow upload htmlcov
env:
SMOKESHOW_GITHUB_STATUS_DESCRIPTION: Coverage {coverage-percentage}
SMOKESHOW_GITHUB_COVERAGE_THRESHOLD: 50
SMOKESHOW_GITHUB_CONTEXT: coverage
SMOKESHOW_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SMOKESHOW_GITHUB_PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
SMOKESHOW_AUTH_KEY: ${{ secrets.SMOKESHOW_AUTH_KEY }}
9 changes: 3 additions & 6 deletions .github/workflows/pytest.yml
Original file line number Diff line number Diff line change
@@ -10,7 +10,7 @@ jobs:
pytest:
strategy:
matrix:
python-version: ["3.8", "3.11"]
python-version: ["3.8", "3.12"]
os: [ubuntu-latest] # can't use macOS when using service containers or container jobs
r: [release]
runs-on: ${{ matrix.os }}
@@ -19,7 +19,7 @@ jobs:
image: postgres
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: docker
POSTGRES_PASSWORD: pass8743hf9h23f87h437
POSTGRES_DB: pep-db
POSTGRES_HOST: localhost
ports:
@@ -33,12 +33,9 @@ jobs:
with:
python-version: ${{ matrix.python-version }}

- name: Install dev dependancies
- name: Install dev dependencies
run: if [ -f requirements/requirements-dev.txt ]; then pip install -r requirements/requirements-dev.txt; fi

- name: Install test dependancies
run: if [ -f requirements/requirements-test.txt ]; then pip install -r requirements/requirements-test.txt; fi

- name: Install package
run: python -m pip install .

141 changes: 132 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,138 @@
# pepdbagent
<h1 align="center">pepdbagent</h1>

[![PEP compatible](https://pepkit.github.io/img/PEP-compatible-green.svg)](https://pep.databio.org/)
![Run pytests](https://github.com/pepkit/pepdbagent/workflows/Run%20pytests/badge.svg)
[![pypi-badge](https://img.shields.io/pypi/v/pepdbagent?color=%2334D058)](https://pypi.org/project/pepdbagent)
[![pypi-version](https://img.shields.io/pypi/pyversions/pepdbagent.svg?color=%2334D058)](https://pypi.org/project/pepdbagent)
[![Coverage](https://coverage-badge.samuelcolvin.workers.dev/pepkit/pepdbagent.svg)](https://coverage-badge.samuelcolvin.workers.dev/redirect/pepkit/pepdbagent)
[![Github badge](https://img.shields.io/badge/source-github-354a75?logo=github)](https://github.com/pepkit/pepdbagent)


---

**Documentation**: <a href="https://pep.databio.org" target="_blank">https://pep.databio.org</a>

**Source Code**: <a href="https://github.com/pepkit/pepdbagent" target="_blank">https://github.com/pepkit/pepdbagent</a>

---

`pepdbagent` is a Python library and toolkit that gives a user-friendly
interface to connect, upload, update and retrieve information from pep-db.
interface to connect, upload, update and retrieve information from the pep database. This library is designed to work
with PEPhub, but it can be used for any other purpose.

`pepdbagent` creates a connection to the database and creates table schemas for the PEPhub database if necessary.
Core database is `postgres` database, but it can be easily extended to other relational databases.
To use `pepdbagent`, you need to have a database instance running with its credentials.
If the version of the database schema is not compatible with the version of `pepdbagent`, it will throw an exception.

## Installation
To install `pepdbagent` use this command:
```
pip install pepdbagent
```
or install the latest version from the GitHub repository:
```
pip install git+https://github.com/pepkit/pepdbagent.git
```

---
## Overview:

The pepdbagent provides a core class called **PEPDatabaseAgent**. This class has 4 modules, divided
to increase readability, maintainability, and user experience of pepdbagent, which are:

**The `pepdbagent` consists of 6 main modules:**
- <u>Namespace</u>: Includes methods for searching namespaces, retrieving statistics, and fetching information.
- <u>Project</u>: Provides functionality for retrieving, uploading, updating, and managing projects.
- <u>Annotation</u>: Offers features for searching projects in the database and namespaces, retrieving annotations, and other related information.
- <u>Sample</u>: Handles the creation, modification, and deletion of samples, without modification of the entire project.
- <u>View</u>: Manages the creation, modification, and deletion of views for specific projects.
- <u>User</u>: Contains user-related information such as favorites and other user-related data.

## Example:

#### Instiantiate a PEPDatabaseAgent object and connect to database:

```python
import pepdbagent
# 1) By providing credentials and connection information:
agent = pepdbagent.PEPDatabaseAgent(user="postgres", password="docker", )
# 2) or By providing connection string:
agent = pepdbagent.PEPDatabaseAgent(dsn="postgresql://postgres:docker@localhost:5432/pep-db")
```

#### Example of usage of the pepdbagent modules:

```python
import peppy

prj_obj = peppy.Project("sample_pep/basic/project_config.yaml")

# create a project
namespace = "demo"
name = "basic_project"
tag = None
agent.project.create(prj_obj, namespace, name, tag)

update_dict = {"is_private" = True}
# after creation of the dict, update record by providing update_dict and namespace, name and tag:
agent.project.update(update_dict, namespace, name, tag)
```


#### Annotation example:


The `.annotation` module provides an interface to PEP annotations.
PEP annotations refers to the information *about* the PEPs (or, the PEP metadata).
Retrieved information contains: [number of samples, submission date, last update date,
is private, PEP description, digest, namespace, name, tag]

```python

```python
# Get annotation of one project:
agent.annotation.get(namespace, name, tag)

# Get annotations of all projects from db:
agent.annotation.get()

# Get annotations of all projects within a given namespace:
agent.annotation.get(namespace='namespace')

# Search for a project with partial string matching, either within namespace or entire database
# This returns a list of projects
agent.annotation.get(query='query')
agent.annotation.get(query='query', namespace='namespace')

# Get annotation of multiple projects given a list of registry paths
agent.annotation.get_by_rp(["namespace1/project1:tag1", "namespace2/project2:tag2"])

# By default get function will retrun annotations for public projects,
# To get annotation including private projects admin list should be provided.
# admin list means list of namespaces where user has admin rights
# For example:
agent.annotation.get(query='search_pattern', admin=['databio', 'ncbi'])
```

**pep-db** is a postgres database created for storing [PEPs](http://pep.databio.org/en/latest/).
It is a backend database for PEPhub.

Before using pepdbagent, you should install or have access to a pep-db instance.
#### Namespace
The `.namespace` module contains search namespace functionality that helps to find namespaces in database
and retrieve information: `number of samples`, `number of projects`.

To run a pep-db instance, you can use this tutorial:
- [pep-db installation](./docs/db_tutorial.md)
Example:
```python
# Get info about namespace by providing query argument. Then pepdbagent will
# search for a specified pattern of namespace in database.
agent.namespace.get(query='Namespace')

Then, follow the `pepdbagent` tutorial here:
- [pedbagent](./docs/README.md)
# By default all get functions will return namespace information for public projects,
# To get information with private projects, admin list should be provided.
# admin list means list of namespaces where user has admin rights
# For example:
agent.namespace.get(query='search_pattern', admin=['databio', 'geo', 'ncbi'])
```
For more information, developers should use `pepdbagent pytest` as documentation due to its natural language syntax and the
ability to write tests that serve as executable examples.
This approach not only provides detailed explanations but also ensures that code examples are kept
up-to-date with the latest changes in the codebase.
140 changes: 0 additions & 140 deletions docs/README.md

This file was deleted.

Loading