Merge pull request #24 from pepkit/dev

Release v0.1.0

.github/workflows/pytest.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        python-version: ["3.6", "3.9", "3.10"]
+        python-version: ["3.8", "3.9", "3.10"]
         os: [ubuntu-latest]
 
     services:

.github/workflows/python-publish.yml

@@ -0,0 +1,31 @@
+# This workflows will upload a Python Package using Twine when a release is created
+# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
+
+name: Upload Python Package
+
+on:
+  release:
+    types: [created]
+
+jobs:
+  deploy:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.x'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install setuptools wheel twine
+    - name: Build and publish
+      env:
+        TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
+        TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+      run: |
+        python setup.py sdist bdist_wheel
+        twine upload dist/*

LICENSE.txt

@@ -0,0 +1,9 @@
+Copyright 2017 Nathan Sheffield
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

MANIFEST.in

@@ -0,0 +1,3 @@
+include LICENSE.txt
+include requirements/*
+include README.md

README.md

@@ -1,74 +1,15 @@
-# pepagent + pep_db
+# pepdbagent
 
-Database and pep_db_agent for storing and processing pep projects
+pepdbagent is a python library and toolkit that gives a user user-friendly 
+interface to connect and retrieve information from pep-db.
 
----
-## How to create pep_db:
+It provides a various comprehensive functions that enables user to retrieve 
+projects, annotations, sets and other information from pep-db.
 
-Complete instruction can be found here: [pep_db](pep_db)
+**pep-db** is and postgres database created for storing [PEPs](http://pep.databio.org/en/latest/). 
+**pep-db** is a backend database for PEPhub. It enables storing huge projects and provides fast speed.
 
----
-## How to use pepagent
-1) Create connection with DB:
-```python
-# 1) By providing credentials and connection information:
-projectDB = PepAgent(user="postgres", password="docker",)
-# 2) or By providing connection string:
-projectDB = PepAgent("postgresql://postgres:docker@localhost:5432/pep-base-sql")
-```
-
-2) Add new project to the DB
-```python
-# initiate peppy Project
-pep_project = peppy.Project("/sample_pep/subtable3/project_config.yaml")
-# use upload_project function to add this project to the DB
-projectDB.upload_project(pep_project, namespace = "Test", anno={"project": "annotation_dict"})  
-# additionally you can specify name of the project
-
-```
-
-3) Get list of available namespaces:
-```python
-list_of_namespaces = projectDB.get_namespaces()
-print(list_of_namespaces)
-```
-
-4) Get project
-```python
-# Get project by id:
-pr_ob = projectDB.get_project(id=3)
-print(pr_ob.samples)
-
-# Get project by registry
-pr_ob = projectDB.get_project(registry='Test/subtable3')
-print(pr_ob.samples)
-
-# Get project by namespace and name
-pr_ob = projectDB.get_project(namespace='Test', name='subtable3')
-print(pr_ob.samples)
-
-# Get project by digest
-pr_ob = projectDB.get_project(digest='1495b8d5b586ab71c9f3a30dd265b3c3')
-print(pr_ob.samples)
-```
-
-5) Get list of all available projects in the namespace
-```python
-# Get project by id:
-projects_list = projectDB.get_project_list('Test')
-print(projects_list)
-```
-
-6) Get annotation about single project or projects:
-```python
-# Get dictionary of annotation for multiple projects by namespace
-projects_anno_list = projectDB.get_anno(namespace='Test')
-# Get dictionary of annotation for 1 project by id 
-projects_anno_list = projectDB.get_anno(id='5')
-# Get dictionary of annotation for 1 project by digest
-projects_anno_list = projectDB.get_anno(digest='1495b8d5b586ab71c9f3a30dd265b3c3')
-# Get dictionary of annotation for 1 project by registry
-projects_anno_list = projectDB.get_anno(digest='Test/subtable3')
-
-```
+Here you can find more information with tutorials:
 
+- [pep-db installation](./docs/db_tutorial.md)
+- [pedbagent](./docs/tutorial.md)

docs/changelog.md

@@ -3,6 +3,6 @@
 This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format.
 
 
-## [0.1.0] -- 2022-07-19
+## [0.1.0] -- 2022-08-18
 
 - 🎉 first release! 

docs/db_tutorial.md

@@ -0,0 +1,17 @@
+# pep_db
+
+### pep_db installation: 
+
+
+0) Go to [pep_db](../pep_db)  directory and then run the following lines
+1) Build the docker: `docker build -t pep-db ./`
+2) Run the docker: `docker run --name pep-db -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=docker -p 5432:5432 -d pep-db`
+3) Start it: `docker start pep-db`
+
+Now db is installed
+
+### How to connect to the docker:
+
+`docker exec -it 65f bash`
+
+`psql -U postgres -d pep-db`

docs/tutorial.md

@@ -0,0 +1,162 @@
+# pepagent + pep_db
+
+pepdbagent package for processing and uploading retrieving pep projects using python
+
+---
+##  Brief pephubdb tutorial:
+
+0) Import Connection from pepdbagent:
+```python
+from pepdbagent import Connection
+```
+
+1) Create connection with DB:
+```python
+# 1) By providing credentials and connection information:
+projectDB = Connection(user="postgres", password="docker",)
+# 2) or By providing connection string:
+projectDB = Connection("postgresql://postgres:docker@localhost:5432/pep-db")
+```
+
+2) Add new project to the DB
+```python
+# initiate peppy Project
+import peppy
+pep_project = peppy.Project("/sample_pep/subtable3/project_config.yaml")
+# use upload_project function to add this project to the DB
+projectDB.upload_project(pep_project, namespace = "Test", status = "approved", description = "ocean dream", anno={"additional": "annotation"})  
+# additionally you can specify name and tag of the project
+
+# update project*
+
+projectDB.update_project(pep_project, namespace = "Test", anno={"enot": "annotation_dict"})  
+# additionally you can specify name and tag of the project
+
+```
+* If you want to update project you should specify all annotation fields, otherwise they will be empty
+
+3) Get list of projects in namespace:
+```python
+list_of_namespaces = projectDB.get_namespace_info(namespace="King")
+print(list_of_namespaces)
+
+```
+
+4) Get list of available namespaces:
+
+```python
+list_of_namespaces = projectDB.get_namespaces_info_by_list()
+print(list_of_namespaces)
+# To get list with with just names of namespaces set: names=True
+# otherwise you will get list with namespaces with information about all projects
+```
+
+5) Get project
+
+```python
+
+# Get project by registry
+pr_ob = projectDB.get_project_by_registry(registry_path='Test/subtable3')
+print(pr_ob.samples)
+
+# Get project by registry
+pr_ob = projectDB.get_project_by_registry(registry_path='Test/subtable3:this_is_tag')
+print(pr_ob.samples)
+
+# Get project by namespace and name
+pr_ob = projectDB.get_project(namespace='Test', name='subtable3')
+print(pr_ob.samples)
+
+# Get project by namespace and name
+pr_ob = projectDB.get_project(namespace='Test', name='subtable3', tag='this_is_tag')
+print(pr_ob.samples)
+
+```
+
+4) Get list of projects
+
+```python
+# Get projects by tag
+pr_ob = projectDB.get_projects_in_namespace(tag='new_tag')
+print(pr_ob.samples)
+
+# Get projects by namespace
+pr_ob = projectDB.get_projects_in_namespace(namespace='King')
+print(pr_ob.samples)
+
+# Get projects by namespace and tag
+pr_ob = projectDB.get_projects_in_namespace(namespace='King', tag='taggg')
+print(pr_ob.samples)
+
+# Get projects by list of registry paths
+pr_ob = projectDB.get_projects_in_list(registry_paths=['Test/subtable3:default', 'Test/subtable3:bbb'])
+print(pr_ob.samples)
+
+# Get all the projects
+pr_ob = projectDB.get_project_all()
+print(pr_ob.samples)
+
+```
+
+5) Get annotation about single project:
+
+```python
+
+# Get dictionary of annotation for 1 project by registry
+projects_anno_list = projectDB.get_project_annotation_by_registry(registry='Test/subtable3:this_is_tag')
+# if tag is not set default tag will be set
+projects_anno_list = projectDB.get_project_annotation(namespace='Test/subtable3')
+
+# As a return value user will get `Annotation` class object. There is two options to retrieve data:
+#1) Using object as simple dict:
+projects_anno_list["status"]
+#2) Using .key ; Available keys:
+projects_anno_list.registry  # to know what project annotation is it 
+projects_anno_list.status
+projects_anno_list.description
+projects_anno_list.last_update
+projects_anno_list.n_samples
+
+```
+
+6) Get annotations namespace or all namespaces:
+
+```python
+# Get dictionary of annotation for specific namespace
+namespace_anno = projectDB.get_namespace_annotation(namespace='Test')
+
+# Get dictiionary of annotations for all namespaces
+namespace_anno_all = projectDB.get_namespace_annotation()
+```
+
+
+7) Check project existance:
+
+```python
+# by name and namespace:
+projectDB.project_exists(namespace="nn", name="buu")
+
+# by name and namespace and tag:
+projectDB.project_exists(namespace="nn", name="buu", tag='dog')
+
+# by registry path:
+projectDB.project_exists_by_registry(registry_path='nn/buu/dog')
+
+```
+
+
+8) Check project status:
+
+```python
+# by name and namespace and tag:
+# Get dictionary of annotation for specific namespace
+projectDB.project_status(namespace="nn", name="buu", tag='dog')
+
+# by registry path:
+projectDB.project_status_by_registry(registry_path='nn/buu/dog')
+```
+
+9) Get registry paths of all the projects by digest:
+```python
+projectDB.get_registry_paths_by_digest(digest='sdafsgwerg243rt2gregw3qr24')
+```

pep_db/Dockerfile

@@ -1,5 +1,5 @@
 FROM postgres
 ENV POSTGRES_USER postgres
 ENV POSTGRES_PASSWORD docker
-ENV POSTGRES_DB pep-base-sql
+ENV POSTGRES_DB pep-db
 COPY pep_db.sql /docker-entrypoint-initdb.d/

pep_db/pep_db.sql

@@ -6,11 +6,13 @@ SET client_encoding = 'LATIN1';
 
 
 CREATE TABLE projects (
-    id BIGSERIAL NOT NULL PRIMARY KEY,
+    id BIGSERIAL NOT NULL,
     namespace TEXT NOT NULL,
     name TEXT NOT NULL,
+    tag TEXT NOT NULL,
     digest TEXT NOT NULL,  -- shoud be changed to CHARACTER
-    project_value json NOT NULL,
-    anno_info json  -- annotation information
+    project_value jsonb NOT NULL,
+    anno_info jsonb,  -- annotation information
+    CONSTRAINT id PRIMARY KEY (namespace, name, tag)
 );