Merge pull request #103 from bird-house/setup-cruft-to-refresh-from-c…

…ookiecutter

Setup cruft to refresh from cookiecutter.

This PR is part of issue bird-house/cookiecutter-birdhouse#61, rolling out cruft as a mechanism to easily update cookiecutter template changes to existing project.

Require cookiecutter-birdhouse PR bird-house/cookiecutter-birdhouse#89

Noticeable changes:

* Docs testing also enabled in Travis-CI
* Use `make develop` in Travis-CI instead of manual install
* ReadTheDocs build changed to use minimal `environment-docs.yml` and mock import

Travis-CI build working: https://travis-ci.org/github/bird-house/emu/builds/694623148
RtD build working: https://readthedocs.org/projects/emu/builds/11184100/

.codacy.yml

@@ -1,4 +1,8 @@
 ---
+engines:
+ pylint:
+   enabled: true
+   python_version: 3
 exclude_paths:
   - 'tests/**'
   - 'docs/source/conf.py'

.cruft.json

@@ -0,0 +1,36 @@
+{
+    "template": "https://github.com/bird-house/cookiecutter-birdhouse.git",
+    "commit": "73eee18ca4a16763335d18ed9512cc617fbcc777",
+    "skip": [
+        "emu/processes/wps_say_hello.py",
+        "tests/test_wps_hello.py",
+        "docs/source/notebooks/example.ipynb",
+        "environment.yml",
+        "requirements.txt",
+        "AUTHORS.rst",
+        "CHANGES.rst",
+        "docs/source/notebooks/index.rst",
+        "docs/source/processes.rst",
+        "emu/processes/__init__.py",
+        "tests/test_wps_caps.py"
+    ],
+    "context": {
+        "cookiecutter": {
+            "full_name": "Carsten Ehbrecht",
+            "email": "ehbrecht@dkrz.de",
+            "github_username": "bird-house",
+            "project_name": "Emu",
+            "project_slug": "emu",
+            "project_repo_name": "emu",
+            "project_readthedocs_name": "emu",
+            "project_short_description": "WPS processes for testing and demos.",
+            "version": "0.11.1",
+            "open_source_license": "Apache Software License 2.0",
+            "http_port": "5000",
+            "_copy_without_render": [
+                "{{cookiecutter.project_slug}}/templates/*.cfg"
+            ],
+            "_template": "https://github.com/bird-house/cookiecutter-birdhouse.git"
+        }
+    }
+}

.github/CONTRIBUTING.md

@@ -2,5 +2,5 @@
 
 Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
 
-Please read the Birdhouse [Contributer Guide](http://birdhouse.readthedocs.io/en/latest/contributing.html)
-and the [Cookiecutter Documentation](http://emu.readthedocs.io/en/latest/) to get started.
+Please read the Birdhouse [Developer Guide](https://birdhouse.readthedocs.io/en/latest/dev_guide.html)
+and the [Emu Documentation](https://emu.readthedocs.io/en/latest/) to get started.

.gitignore

@@ -61,12 +61,15 @@ bootstrap-buildout.py
 bootstrap.py
 #generated by buildout
 
+*.pid
+
 # sphinx
 #docs/Makefile
 docs/make.bat
 docs/doctrees/
 docs/html/
 docs/build/
+docs/source/output-sanitize.cfg
 
 # External Sources
 #src/external
@@ -85,3 +88,6 @@ testdata.json
 *.a
 *.mod
 *.out
+
+# Merge conflict
+*.orig

.readthedocs.yml

@@ -0,0 +1,27 @@
+# .readthedocs.yml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/source/conf.py
+
+# Build documentation with MkDocs
+#mkdocs:
+#  configuration: mkdocs.yml
+
+# Optionally build your docs in additional formats such as PDF and ePub
+formats: all
+
+# Optionally set the version of Python and requirements required to build your docs
+#python:
+#  version: 3.6
+
+conda:
+  environment: environment-docs.yml
+
+build:
+  image: stable

.travis.yml

@@ -1,35 +1,40 @@
 dist: xenial   # required for Python >= 3.7
 language: python
+os:
+  - linux
 python:
   - "3.6"
   - "3.7"
-os:
-  - linux
 #  - osx
 sudo: false
+before_install:
+  # Useful for debugging Travis CI environment
+  - printenv
 install:
   # Python 3.x is default
   - wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
   - bash miniconda.sh -b -p $HOME/miniconda
   - export PATH="$HOME/miniconda/bin:$PATH"
   - hash -r
   - conda config --set always_yes yes --set changeps1 no
+  - conda install setuptools
   - conda update -q conda
   # Useful for debugging any issues with conda
   - conda info -a
   # Prepare env with Python version
-  - conda create -n emu python=$TRAVIS_PYTHON_VERSION
+  - conda create -n emu -c conda-forge python=$TRAVIS_PYTHON_VERSION
   # Update now the env with our environment
   - conda env update -f environment.yml
   - source activate emu
-  # Packages for testing
-  - conda install pytest flake8
-  # Install WPS
-  - python setup.py install
+  # Packages for testing, generating docs and installing WPS
+  - make develop
 before_script:
   # Start WPS service on port 5000 on 0.0.0.0
   - emu start --daemon --bind-host 0.0.0.0 --port 5000
   - sleep 2
 script:
   - pytest -v -m 'not online' tests/
+  # No notebooks in emu yet.
+  #- make test-notebooks
   - flake8
+  - make docs

LICENSE.txt

@@ -1,5 +1,4 @@
-
-                                 Apache License
+Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/
 
@@ -187,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright [yyyy] [name of copyright owner]
+   Copyright [2020] [Carsten Ehbrecht]
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
@@ -200,3 +199,4 @@
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
+

Makefile

@@ -2,6 +2,15 @@
 APP_ROOT := $(abspath $(lastword $(MAKEFILE_LIST))/..)
 APP_NAME := emu
 
+WPS_URL = http://localhost:5000
+
+# Used in target refresh-notebooks to make it looks like the notebooks have
+# been refreshed from the production server below instead of from the local dev
+# instance so the notebooks can also be used as tutorial notebooks.
+OUTPUT_URL = https://pavics.ouranos.ca/wpsoutputs
+
+SANITIZE_FILE := https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/raw/master/notebooks/output-sanitize.cfg
+
 # Bumpversion 'dry' config
 # if 'dry' is specified as target, any bumpversion call using 'BUMP_XARGS' will not apply changes
 BUMP_XARGS ?= --verbose --allow-dirty
@@ -33,7 +42,9 @@ help:
 	@echo "\nTesting targets:"
 	@echo "  test              to run tests (but skip long running tests)."
 	@echo "  test-all          to run all tests (including long running tests)."
+	@echo "  test-notebooks    to verify Jupyter Notebook test outputs are valid."
 	@echo "  lint              to run code style checks with flake8."
+	@echo "  refresh-notebooks to verify Jupyter Notebook test outputs are valid."
 	@echo "\nSphinx targets:"
 	@echo "  docs              to generate HTML documentation with Sphinx."
 	@echo "\nDeployment targets:"
@@ -78,15 +89,15 @@ restart: stop start
 
 .PHONY: status
 status:
-	@echo "Show status ..."
+	@echo "Showing status ..."
 	@-bash -c "$(APP_NAME) status"
 
 .PHONY: clean
 clean: clean-build clean-pyc clean-test ## remove all build, test, coverage and Python artifacts
 
 .PHONY: clean-build
 clean-build:
-	@echo "Remove build artifacts ..."
+	@echo "Removing build artifacts ..."
 	@-rm -fr build/
 	@-rm -fr dist/
 	@-rm -fr .eggs/
@@ -97,21 +108,22 @@ clean-build:
 
 .PHONY: clean-pyc
 clean-pyc:
-	@echo "Remove Python file artifacts ..."
+	@echo "Removing Python file artifacts ..."
 	@-find . -name '*.pyc' -exec rm -f {} +
 	@-find . -name '*.pyo' -exec rm -f {} +
 	@-find . -name '*~' -exec rm -f {} +
 	@-find . -name '__pycache__' -exec rm -fr {} +
 
 .PHONY: clean-test
 clean-test:
-	@echo "Remove test artifacts ..."
+	@echo "Removing test artifacts ..."
 	@-rm -fr .pytest_cache
 
 .PHONY: clean-dist
 clean-dist: clean
-	@echo "Run 'git clean' ..."
-	@git diff --quiet HEAD || echo "There are uncommited changes! Not doing 'git clean' ..."
+	@echo "Running 'git clean' ..."
+	@git diff --quiet HEAD || echo "There are uncommitted changes! Aborting 'git clean' ..."
+	## do not use git clean -e/--exclude here, add them to .gitignore instead
 	@-git clean -dfx
 
 ## Test targets
@@ -126,18 +138,35 @@ test-all:
 	@echo "Running all tests (including slow and online tests) ..."
 	@bash -c 'pytest -v tests/'
 
+.PHONY: notebook-sanitizer
+notebook-sanitizer:
+	@echo "Copying notebook output sanitizer ..."
+	@-bash -c "curl -L $(SANITIZE_FILE) -o $(CURDIR)/docs/source/output-sanitize.cfg --silent"
+
+.PHONY: test-notebooks
+test-notebooks: notebook-sanitizer
+	@echo "Running notebook-based tests"
+	@bash -c "env WPS_URL=$(WPS_URL) pytest --nbval --verbose $(CURDIR)/docs/source/notebooks/ --sanitize-with $(CURDIR)/docs/source/output-sanitize.cfg --ignore $(CURDIR)/docs/source/notebooks/.ipynb_checkpoints"
+
 .PHONY: lint
 lint:
 	@echo "Running flake8 code style checks ..."
 	@bash -c 'flake8'
 
+.PHONY: refresh-notebooks
+refresh-notebooks:
+	@echo "Refresh all notebook outputs under docs/source/notebooks"
+	@bash -c 'for nb in $(CURDIR)/docs/source/notebooks/*.ipynb; do WPS_URL="$(WPS_URL)" jupyter nbconvert --to notebook --execute --ExecutePreprocessor.timeout=60 --output "$$nb" "$$nb"; sed -i "s@$(WPS_URL)/outputs/@$(OUTPUT_URL)/@g" "$$nb"; done; cd $(APP_ROOT)'
+
 ## Sphinx targets
 
 .PHONY: docs
 docs:
 	@echo "Generating docs with Sphinx ..."
 	@-bash -c '$(MAKE) -C $@ clean html'
-	@echo "open your browser: open file://$(APP_ROOT)/docs/build/html/index.html"
+	@echo "Open your browser to: file:/$(APP_ROOT)/docs/build/html/index.html"
+	## do not execute xdg-open automatically since it hangs travis and job does not complete
+	@echo "xdg-open $(APP_ROOT)/docs/build/html/index.html"
 
 ## Deployment targets
 
@@ -149,7 +178,7 @@ bump:
 
 .PHONY: dist
 dist: clean
-	@echo "Builds source and wheel package ..."
+	@echo "Building source and wheel package ..."
 	@-python setup.py sdist
 	@-python setup.py bdist_wheel
-	ls -l dist
+	@-bash -c 'ls -l dist/'

README.rst

@@ -1,8 +1,8 @@
 Emu
-===
+===============================
 
 .. image:: https://img.shields.io/badge/docs-latest-brightgreen.svg
-   :target: http://emu.readthedocs.org/en/latest/?badge=latest
+   :target: http://emu.readthedocs.io/en/latest/?badge=latest
    :alt: Documentation Status
 
 .. image:: https://travis-ci.org/bird-house/emu.svg?branch=master
@@ -25,6 +25,37 @@ Emu
 Emu (the bird)
   *Emus are curious birds who are known to follow and watch other animals and humans. Emus do not sleep continuously at night but in several short stints sitting down. [..].* (`Wikipedia <https://en.wikipedia.org/wiki/Emu>`_).
 
+WPS processes for testing and demos.
+
 Emu is a Python package with some test proccess for Web Processing Services (WPS). Currently it is using the `PyWPS 4.x <http://pywps.org/>`_ server.
 
-Full `documentation <http://emu.readthedocs.io/en/latest/>`_ is available on ReadTheDocs or in the docs directory.
+Documentation
+-------------
+
+Learn more about Emu in its official documentation at
+https://emu.readthedocs.io.
+
+Submit bug reports, questions and feature requests at
+https://github.com/bird-house/emu/issues
+
+Contributing
+------------
+
+You can find information about contributing in our `Developer Guide`_.
+
+Please use bumpversion_ to release a new version.
+
+License
+-------
+
+Free software: Apache Software License 2.0
+
+Credits
+-------
+
+This package was created with Cookiecutter_ and the `bird-house/cookiecutter-birdhouse`_ project template.
+
+.. _Cookiecutter: https://github.com/audreyr/cookiecutter
+.. _`bird-house/cookiecutter-birdhouse`: https://github.com/bird-house/cookiecutter-birdhouse
+.. _`Developer Guide`: https://emu.readthedocs.io/en/latest/dev_guide.html
+.. _bumpversion: https://emu.readthedocs.io/en/latest/dev_guide.html#bump-a-new-version

docker-compose.yml

@@ -1,6 +1,6 @@
 version: '3'
 services:
-  emu:
+  wps:
     # http://localhost:5000/wps
     build: .
     image: birdhouse/emu:dev