Merge pull request #11 from SD2E/development

Jupyter Upgrade

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 All notable changes to this project are to be documented in this file.
 
+## Version 0.3.1
+
+ - Added `PYTHONUSERBASE` variable, which points at `tacc-work/jupyter_packages`, and allows for persistent local pip installs
+ - Added pyemd
+ - Added graphviz, python-graphviz, cmake to support DSGRN
+ - FIXED git funtionality by setting
+   - `GIT_COMMITTER\_NAME=$JUPYTERHUB\_USER`
+   - `GIT_COMMITTER\_EMAIL=${JUPYTERHUB\_USER}@sd2e.org`
+ - REMOVED the hardcoding of `MPLBACKEND` to `Agg`
+
 ## Version 0.3.0
 
  - Refactored images to start from taccsciapps/jupyteruser-base

CONTRIBUTING.md

@@ -1,40 +1,96 @@
 # Contribute to jupyteruser-sd2e
 
-Have an idea for what our base Jupyter Notebooks environment needs in it? Here's how to make it happen!
+If you find that the SD2E jupyter environment lacks some
+
+ - tools
+ - libraries
+ - packages
+ - examples
+
+that the community would benefit from, here is a guide on getting it incorporated.
 
 ## Propose work
 
-* Raise an issue:
-    * Describing a proposed change/update/improvement/fix (we provide a nice template).
-    * If you have one (even just an idea) provide an implementation plan
-    * Include thoughts about how to test the new feature, as well as how it will be communicated to end users
+* Raise an issue \([link](https://github.com/SD2E/jupyteruser-sd2e/issues)\):
+  * Describing a proposed change/update/improvement/fix (we provide a nice template).
+    * List and link the software \(required\)
+      * boost >= 1.58 - (http://www.boost.org/users/download/)
+      * python boost >= 0.1
+    * List any known dependencies
+      * C++
+  * If you have one (even just an idea) provide an implementation plan
+    * `apt-get install foo`
+    * `pip install foo`
+  * Let us know how we can verify the software installation
+    * Sample commands
+    * Example notebooks
 
 ## Contribute work
 
-* To contribute, fork this repo and pull it down locally
-* Checkout a new branch (myFeature) off `development` named after the Github issue documenting the bug you're fixing or feature you're adding
+While not necessary, you can also locally develop and test your own feature for integration. When you are satisfied, you can then submit a pull request for integration into the main SD2E repository.
+
+### Fork the repository
+
+While the project is open, we restrict the ability to commit code. Begin by forking
+
+![fork](https://upload.wikimedia.org/wikipedia/commons/3/38/GitHub_Fork_Button.png)
+
+our repository to your own user account on github. This is necessary so we can see the remote repository to pull your changes from. Github cannot pull from your laptop.
+
+### Clone YOUR repository
+
+Clone a local copy of **your** repository
+
+```
+git clone https://github.com/[your_name_hear]/jupyteruser-sd2e.git
+```
+
+and then make a new branch (`myFeature`) for your feature based on the `development` branch.
+
 ```
 git checkout -b myFeature development
 ```
-* In `images/custom`:
-  * Edit the `Dockerfile` to add your software
-  * Add example files and notebooks
-* Clear local image cache
+
+### Incorporate your changes
+
+Please add any external example or configuration files in a descriptive folder in
+
+```
+images/custom/
 ```
-sudo make clean
+
+and add a new section at the end of the main `Dockerfile` in
+
 ```
-* Build the new image
+images/custom/Dockerfile
+```
+
+for your code immediately **before** the permissiosn section.
+
+### Build and Test your changes
+
+Build your modified container from the **root** `jupyteruser-sd2e` directory using the (hopefully) helpful make commands
+
 ```
+sudo make clean
 sudo make build sd2e
 ```
-* Test the new image
+
+If your build succeeds, you can then test with
+
 ```
 sudo make test sd2e
 ```
-* Push your branch to your Github fork when you're ready for integration
-* Submit a Pull Request against the SD2E `development` branch (template provided once again)
-* Your PR will be reviewed:
-  * A Docker image will be built from your code and will be available in about an hour on the test Jupyter server: https://jupyter-staging.sd2e.org/
+
+Visit your local jupyter environment using either the localhost URL if on your own computer, or the external URL if testing in the cloud.
+
+### Submit pull request
+
+When done testing, commit and push your changes to your personal fork of the repository. Then, you can create a pull request against the SD2E `development` branch. Once again, we provide a template to help populate the necessary information about your changes.
+
+After we review and accept your pull request:
+  * A Docker image will be built from your code and will be available in about an hour on the test Jupyter server
+    * https://jupyter-staging.sd2e.org/
   * Both the SD2E development team and you must review and validate
     * The updated/new functionality
     * Completeness and correctness of provided documenation

Makefile

@@ -53,38 +53,50 @@ test: docker
 	case $$TARGET in
 	base)
 		build/build_jupyteruser.sh test images/base
+		;;
 	sd2e)
 		build/build_jupyteruser.sh test images/custom
+		;;
 	singularity)
 		build/build_jupyteruser.sh test images/singularity
+		;;
 	*)
 		$(MAKE) help
+		;;
 	esac
 
 stage: docker
 	TARGET=$(filter-out $@,$(MAKECMDGOALS))
 	case $$TARGET in
 	base)
 		build/build_jupyteruser.sh stage images/base
+		;;
 	sd2e)
 		build/build_jupyteruser.sh stage images/custom
+		;;
 	singularity)
 		build/build_jupyteruser.sh stage images/singularity
+		;;
 	*)
 		$(MAKE) help
+		;;
 	esac
 
 release: docker
 	TARGET=$(filter-out $@,$(MAKECMDGOALS))
 	case $$TARGET in
 	base)
 		build/build_jupyteruser.sh release images/base
+		;;
 	sd2e)
 		build/build_jupyteruser.sh release images/custom
+		;;
 	singularity)
 		build/build_jupyteruser.sh release images/singularity
+		;;
 	*)
 		$(MAKE) help
+		;;
 	esac
 
 clean: docker

README.md

@@ -2,15 +2,23 @@
 
 ## What it Gives You
 
-* Jupyter Notebook server (v4.0.x or v3.2.x, see tag)
+* Jupyter Notebook \(v5.2.0\)server
 * Conda Python 3.4.x and Python 2.7.x environments
-* pandas, matplotlib, scipy, seaborn, scikit-learn, numba pre-installed
-* R-essentials (including an R kernel)
-* Plotly, igraph, and networkx for graphs and plots
+* Pre-installed python packages (highlights)
+  * pandas
+  * matplotlib 2.1.2
+  * numpy, scipy
+  * seaborn
+  * scikit-learn
+  * numba
+  * pyemd
+* R-kernel
+* Plotly, igraph, networkx, graphviz for graphs and plots
 * MIT's [Open Probabilistic Programming Stack](http://probcomp.org/)
 * [Common Lisp kernel and stack](http://www.sbcl.org/)
 * Bioconda and Bioconductor
 * BioPython
+* Git
 
 ### SD2E Specific Features
 

VERSION

@@ -1 +1 @@
-0.3.0
+0.3.1

build/build_jupyteruser.sh

@@ -145,11 +145,11 @@ function pushImage {
 			fi
 		fi
 	fi
-	if askTrue "Do you want to tag ${IMG}:${VERSION} as ${3}?"; then
-		ed "docker tag ${IMG}:${VERSION} ${IMG}:${3}"
-		docker tag ${IMG}:${VERSION} ${IMG}:${3}
-		ed "docker push ${IMG}:${3}"
-		docker push ${IMG}:${3}
+	if askTrue "Do you want to tag ${IMG}:${VERSION} as ${2}?"; then
+		ed "docker tag ${IMG}:${VERSION} ${IMG}:${2}"
+		docker tag ${IMG}:${VERSION} ${IMG}:${2}
+		ed "docker push ${IMG}:${2}"
+		docker push ${IMG}:${2}
 	fi
 }
 

images/base/Dockerfile

@@ -1,7 +1,7 @@
 #Image: sd2e/jupyteruser-base
-#Version: 0.1.0
+#Version: 0.1.1
 
-FROM taccsciapps/jupyteruser-base:latest
+FROM taccsciapps/jupyteruser-base:0.1.2
 LABEL origin="https://bitbucket.org/gzynda/jupyter/src/40adbad7b10b34de715d5a86aec3fcf65e724b70/notebooks/base/jupyter-notebook/Dockerfile?at=CIC-425-image-redesign&fileviewer=file-view-default"
 
 ###############################################
@@ -11,20 +11,26 @@ USER root
 
 # Image release
 ARG image_version=development
+ENV NB_GID=G-819382
 RUN echo "${image_version}" > /etc/sd2e_jupyteruser-base-release && \
 chmod a+r /etc/sd2e_jupyteruser-base-release && \
 # Make sure Jupyter group == SD2E-Community == 819382
-groupadd --gid 819382 G-819382 && \
-usermod -a -G G-819382 jupyter && \
-usermod -g G-819382 jupyter && \
+groupadd --gid 819382 $NB_GID && \
+usermod -a -G $NB_GID jupyter && \
+usermod -g $NB_GID jupyter && \
 # Jupyterhub correctly sets JUPYTERHUB_USER
 echo 'PS1="$JUPYTERHUB_USER@\h:\w\$ "' >> /home/jupyter/.bashrc && \
 # allow users to add kernels
-chmod 777 /opt/conda/share/jupyter/kernels
-
+chmod 777 /opt/conda/share/jupyter/kernels && \
 # Environment variables
-ENV WORK /home/jupyter/tacc-work
-ENV SD2_DATA /home/jupyter/sd2e-community
+echo "export JPY_USER=\$JUPYTERHUB_USER\n\
+export WORK=/home/jupyter/tacc-work\n\
+export SD2_DATA=/home/jupyter/sd2e-community\n\
+export PYTHONUSERBASE=/home/jupyter/tacc-work/jupyter_packages\n" >> /home/jupyter/.bashrc
+# Add activation sripts
+ADD src/activate-python.sh /opt/conda/etc/conda/activate.d/activate-python.sh
+ADD src/activate-python.sh /opt/conda/envs/python2/etc/conda/activate.d/activate-python.sh
+RUN find /opt/conda -type d -name activate.d -exec chmod -R a+rx {} \;
 
 # Add readme
 ADD docs/SD2E_README.md /home/jupyter/SD2E_README.md
@@ -51,26 +57,26 @@ cd /root && \
 	rm -rf sd2e-jupyter-ascending && \
 # Ubuntu Packages
 apt-get update && \
-    apt-get install -y tree less && \
+    # jq and bsdmainutils->column is required for abaco CLI
+    apt-get install -y tree less jq rsync bsdmainutils && \
     #apt-get install -y tree less openjdk-8-jre ca-certificates-java && \
-    apt-get clean && \
+    docker-clean
 # Replace stock Agave CLI with SD2E-customized tooling
-curl -L https://raw.githubusercontent.com/sd2e/sd2e-cli/master/install/install.sh | sh && \
-	rm -rf /opt/cli && \
+ADD src/configuration.rc configuration.rc
+RUN git clone https://github.com/SD2E/sd2e-cli.git && \
+	mv configuration.rc sd2e-cli/ && cd sd2e-cli && \
+	git submodule update --init --recursive && make && \
+	make install && rm -rf /opt/cli && rm -rf $HOME/sd2e-cli && \
 	mv /home/jupyter/sd2e-cloud-cli /opt/cli && \
+	chmod -R a+rX /opt/cli && chmod -R a+x /opt/cli/bin && \
 # Spark 2.2.1 and pyspark
 #cd /usr/local && \
 #        curl http://apache.claz.org/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz | tar -xzf - && \
 #        mv spark-2.2.1-bin-hadoop2.7 spark && \
 #        conda install --yes pyspark && \
 #        conda install -n python2 --yes pyspark && \
-# Clean conda cache
-conda clean -tipsy && \
-# Clean temporary files
-apt-get clean && conda clean -tipsy && \
-bash -c "for dir in /tmp/* /var/tmp/* /home/jupyter/.cache/pip /root/.cache/pip; do [ -e $dir ] && rm -rf $dir; done"
-
-ENV SPARK_HOME=/usr/local/spark
+docker-clean
+#ENV SPARK_HOME=/usr/local/spark
 
 ###############################################
 # Permissions

images/base/src/activate-python.sh

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+export PYTHONUSERBASE=/home/jupyter/tacc-work/jupyter_packages

images/base/src/configuration.rc

@@ -0,0 +1,56 @@
+# SD2E TACC.Cloud variables
+#
+# Human-readable name of your TACC Cloud API tenant
+
+# Source this before running any make commands
+
+# Human-readable name for your project
+export TENANT_NAME="DARPA SD2E"
+
+# Tenant shortname (see https://tacc.cloud/tenants for options)
+export TENANT_KEY=sd2e
+
+# Namespace for custom executable assets
+export TENANT_NS=sd2e
+
+# TACC Community project (for access to default resources)
+export TENANT_PROJECT=SD2E-Community
+
+# Docker
+# Dockerfile for building the CLI image
+export TENANT_DOCKERFILE=Dockerfile
+# DockerHub virtual organization
+export TENANT_DOCKER_ORG=sd2e
+# Docker image tag
+export TENANT_DOCKER_TAG=cloud-cli
+# Docker image version
+# This comes from the VERSION file
+
+# Destination repository
+# Repository and branch for the tenant-specific project
+export TENANT_CLI_REPO=https://github.com/SD2E/sd2e-cli
+export TENANT_CLI_BRANCH=master
+
+# Optional: Set a remote repo to populate the docs/ directory
+export CLI_DOCS=
+export CLI_DOCS_BRANCH=
+export CLI_DOCS_DEST=docs
+
+# Optional: Set a remote repo to populate the extras directory
+export CLI_EXTRAS_REPO=
+export CLI_EXTRAS_BRANCH=
+export CLI_EXTRAS_DEST=extras
+
+# Dont' change below this line unless you know what you're doing!
+
+# Submodules
+export CLI_BASE_REPO=https://bitbucket.org/tacc-cic/cli
+export CLI_BASE_BRANCH=master
+export CLI_BASE_DEST=tacc-cli-base
+
+export ABACO_BASE_REPO=https://github.com/TACC-Reactors/abaco-cli.git
+export ABACO_BASE_BRANCH=master
+export ABACO_BASE_DEST=abaco-cli
+
+# Make support
+export MAKE_OBJ=sd2e-cloud-cli