Merge pull request #60 from databio/dev

Added bedboss and bedhost and bbconf

docs/bedboss/how-to-configure.md

@@ -1,11 +1,11 @@
-# How to create bedbase config file (for bedstat)
+# How to create bedbase config file
 
 ### Bedbase config file is yaml file with 4 parts:
-- path to output files
-- database credentials 
-- qdrant credentials (can be skipped for if indexing is not needed)
-- server information 
-- remote info (can be skipped for bedboss)
+- paths and vector models
+- relational database credentials
+- qdrant credentials
+- server information
+- remote info
 
 ### Example:
 ```yaml
@@ -14,7 +14,7 @@ path:
   pipeline_output_path: /data/outputs
   bedstat_dir: outputs/bedstat_output
   bedbuncher_dir: outputs/bedbuncher_output
-  region2vec: databio/r2v-ChIP-atlas-hg38
+  region2vec: databio/r2v-ChIP-atlas-hg38-v2
   vec2vec: databio/v2v-MiniLM-v2-ATAC-hg38
   text2vec: sentence-transformers/all-MiniLM-L6-v2
 database:
@@ -23,8 +23,11 @@ database:
   password: $POSTGRES_PASSWORD
   user: $POSTGRES_USER
   name: bedbase
+  bed_table: bedfiles
+  bedset_table: bedsets
+  relationship_table: bedset_bedfiles
   dialect: postgresql
-  driver: psycopg
+  driver: psycopg2
 qdrant:
   host: $QDRANT_HOST
   port: 6333
@@ -35,26 +38,9 @@ server:
   port: 8000
 remotes:
   http:
-    prefix: http://data.bedbase.org/
-    description: HTTP compatible path
-  s3:
-    prefix: s3://data.bedbase.org/
-    description: S3 compatible path
-access_methods:
-  http:
-    type: "https"
-    description: HTTP compatible path
     prefix: https://data2.bedbase.org/
+    description: HTTP compatible path
   s3:
-    type: "s3"
-    description: S3 compatible path
     prefix: s3://data2.bedbase.org/
-  local:
-    type: "https"
-    description: How to serve local files.
-    prefix: /static/
-```
-
-Download example bedbase configuration file here: <a href="../bedbase_configuration.yaml" download>Example bedbase configuration file</a>
-
-.
+    description: S3 compatible path
+```

docs/bedboss/how-to-create-database.md

@@ -1,9 +1,14 @@
-# How to create bedbase database
+# How to create BEDbase database
 
-To run bedstat, bedbuncher and bedmbed we need to create postgres database.
+To run bedboss and upload data to the database we need to create postgres database, or use existing one.
 
+---
+### To create local database:
 We are initiating postgres db in docker.
-If you don't have docker installed, you can install it with `sudo apt-get update && apt-get install docker-engine -y`.
+If you don't have docker installed, you can install it with 
+```bash
+sudo apt-get update && apt-get install docker-engine -y
+```
 
 Now, create a persistent volume to house PostgreSQL data:
 
@@ -16,7 +21,9 @@ docker run -d --name bedbase-postgres -p 5432:5432 \
   -e POSTGRES_PASSWORD=bedbasepassword \
   -e POSTGRES_USER=postgres \
   -e POSTGRES_DB=postgres \
-  -v postgres-data:/var/lib/postgresql/data postgres:13
+  -v postgres-data:/var/lib/postgresql/data \
+  postgres:13
 ```
 
 Now we have created docker and can run pipelines.
+To connect to the database, change your credentials in the `bedbase_config.yaml` file.

docs/bedboss/how-to-develop.md

@@ -0,0 +1 @@
+### 🚧 docs in progress! Stay tuned for updates. We're working hard to bring you valuable content soon!

docs/bedboss/how-to-install-r-dependencies.md

@@ -1,12 +1,7 @@
 # How to install R dependencies
 
-1. Install R: <https://cran.r-project.org/bin/linux/ubuntu/fullREADME.html>
-2. Download this script: <a href="../installRdeps.R" download>Install R dependencies</a>
-3. Install dependencies by running this command in your terminal: 
-
-    ```
-    Rscript installRdeps.R
-    ```
-
-4. Run `bash_requirements_test.sh` to check if everything was installed correctly (located in test folder: 
-[Bash requirement tests](https://github.com/bedbase/bedboss/blob/68910f5142a95d92c27ef53eafb9c35599af2fbd/test/bash_requirements_test.sh))
+0. Install bedboss
+1. Install R: https://cran.r-project.org/bin/linux/ubuntu/fullREADME.html
+2. Download this script: [installRdeps.R](https://github.com/databio/bedboss/blob/dev/scripts/installRdeps.R)
+3. Install dependencies by running this command in your terminal: ```Rscript installRdeps.R```
+4. Run `bedboss requirements-check` to check if everything was installed correctly.

docs/bedboss/tutorials/bedbuncher_tutorial.md

@@ -0,0 +1 @@
+### 🚧 Tutorial in progress! Stay tuned for updates. We're working hard to bring you valuable content soon!

docs/bedboss/tutorials/bedclassifier_tutorial.md

@@ -0,0 +1 @@
+### 🚧 Tutorial in progress! Stay tuned for updates. We're working hard to bring you valuable content soon!

docs/bedboss/tutorials/bedindex_tutorial.md

@@ -0,0 +1 @@
+### 🚧 Tutorial in progress! Stay tuned for updates. We're working hard to bring you valuable content soon!

docs/bedboss/tutorials/bedmaker_tutorial.md

@@ -0,0 +1,39 @@
+## BEDmaker
+
+The BEDmaker is a tool that allows you to convert various file types into BED format and bigBed format.
+Currently supported formats are:
+- bed 
+- bigBed
+- bigWig
+- wig
+
+Before running pipeline first, you have to install bedboss and check if bedmaker requirements are satisfied.
+To do so, you can run the next command:
+```bash
+bedboss requirements-check
+```
+
+### Run BEDmaker from command line
+```bash
+bedboss make \
+    --input-file path/to/input/file \
+    --input-type bed\
+    --output-folder path/to/output/dir \
+    --genome hg38 \
+    --sample-name sample1
+    --bigbed "path/to/bigbedfile.bigbed" # optional
+```
+
+### Run BEDmaker from within Python
+```python
+from bedboss.bedmaker.bedmaker import make_all
+
+make_all(
+    input_file="path/to/input/file",
+    input_type="bed",
+    output_folder="path/to/output/dir",
+    genome="hg38",
+    sample_name="sample1",
+    bigbed="path/to/bigbedfile.bigbed" # optional
+)
+```

docs/bedboss/tutorials/bedqc_tutorial.md

@@ -0,0 +1,32 @@
+## BEDqc
+
+BEDqc is a tool for quality control of BED files. 
+As for now, it checks:
+- maximum file size, 
+- maximum number of regions, 
+- minimum region width threshold
+
+----
+### Run BEDqc from command line
+```bash
+bedboss qc \
+    --bedfile path/to/bedfile.bed \
+    --outfolder path/to/output/dir \
+```
+
+---
+
+Run BEDqc from within Python
+```python
+from bedboss import bedqc
+
+bedqc.run_bedqc(
+    bedfile="path/to/bedfile.bed",
+    outfolder="path/to/output/dir"
+    max_file_size=1000000, # optional
+    max_number_of_regions=1000, # optional
+    min_region_width=10, # optional
+)
+```
+
+If file won't pass the quality control, it will raise an error. and add this information to the log file.