fixing up bugs for server settings

Dockerfile

@@ -3,19 +3,23 @@ ENV PYTHONUNBUFFERED 1
 RUN apt-get update && apt-get install -y \
     libopenblas-dev \
     gfortran \
+    pkg-config \
+    libxml2-dev \
+    libxmlsec1-dev \
     libhdf5-dev \
     libgeos-dev \
     build-essential \
     openssl \
-    nginx \
-    wget
+    wget \
+    vim
 
 RUN pip install cython
 RUN pip install numpy
 RUN pip install scikit-learn pandas h5py matplotlib
 RUN pip install uwsgi
 RUN pip install Django==1.11.2
 RUN pip install social-auth-app-django
+RUN pip install social-auth-core[saml]
 RUN pip install djangorestframework
 RUN pip install django-filter
 RUN pip install django-taggit

README.md

@@ -37,8 +37,8 @@ DEIDENTIFY_RESTFUL=True
 
 If this variable is False, we skip this task, and images are instead sent to the next task (or tasks) to send them to different storage. If True, the images are first put in the queue to be de-identified, and then upon receival of the identifiers, then they are put into the same queues to be sent to storage. These functions can be modified to use different endpoints, or do different replacements in the data:
 
- - The function `get_identifiers` under [images/tasks.py](sendit/apps/images/tasks.py) should take in a series ID, and use that series to look up images, and send a RESTful call to some API point to return fields to replace in the data. The JSON response should be saved to an `SeriesIdentifiers` object along with a pointer to the Series.
- - The function `replace_identifers` also under [images/tasks.py](sendit/apps/images/tasks.py) should then load this object, do whatever work is necessary for the data, and then put the data in the queue for storage.
+ - The function `get_identifiers` under [main/tasks.py](sendit/apps/main/tasks.py) should take in a series ID, and use that series to look up images, and send a RESTful call to some API point to return fields to replace in the data. The JSON response should be saved to an `SeriesIdentifiers` object along with a pointer to the Series.
+ - The function `replace_identifers` also under [main/tasks.py](sendit/apps/main/tasks.py) should then load this object, do whatever work is necessary for the data, and then put the data in the queue for storage.
 
 You might want to tweak both of the above functions depending on your call endpoint, the response format (should be json as it goes into a jsonfield), and then how it is used to deidentify the data.
 

docs/watcher.md

@@ -0,0 +1,30 @@
+# Watcher
+The watcher is implemented as a [pyinotify daemon](https://github.com/seb-m/pyinotify/wiki) that is controlled via the [manage.py](../manage.py). If you are more interested about how this module works, it uses [inotify](https://pypi.python.org/pypi/inotify) that comes from the linux Kernel. Specifically, this means that you can start and stop the daemon with the following commands (from inside the Docker image):
+
+```
+python manage.py start_watcher
+python manage.py stop_watcher
+
+```
+
+The functions are stored in the [management/commands](../sendit/apps/watcher/management/commands) folder within the watcher app. This organization is standard for adding a command to `manage.py`, and is done by way of instantiating Django's [BaseCommand](https://docs.djangoproject.com/en/1.11/howto/custom-management-commands/#django.core.management.BaseCommand):
+
+```
+from django.core.management.base import (
+    BaseCommand, 
+    CommandError
+)
+```
+
+For better understanding, you can look at the code itself, for each of [watcher_start.py](../sendit/apps/watcher/management/commands/watcher_start.py) and [watcher_stop.py](../sendit/apps/watcher/management/commands/watcher_stop.py).
+
+## Basic Workflow
+The basic workflow of the watcher is the following:
+
+1. The user starts the daemon.
+2. The process id is stored at `watcher.pid` in the application base folder at `/code`. Generally, we always check for this file first to stop a running process, stop an old process, or write a new pid.
+3. Logs for error (watcher.err) and output (watcher.out) are written under [logs](../logs). We likely (might?) want to clear / update these every so often, in case they get really big.
+4. The daemon responds to all events within the `/data` folder of the application. When this happens, we trigger a function that:
+   - checks for a complete series folder
+   - if complete, adds to database and starts processing
+

manage.py

@@ -0,0 +1,10 @@
+#!/usr/bin/env python
+import os
+import sys
+
+if __name__ == "__main__":
+    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "sendit.settings")
+
+    from django.core.management import execute_from_command_line
+
+    execute_from_command_line(sys.argv)

sendit/celery.py

@@ -3,7 +3,7 @@
 from celery import Celery
 from django.conf import settings
 from celery.schedules import crontab
-from whatisit.settings import (
+from sendit.settings import (
     INSTALLED_APPS,
     BROKER_URL
 )

sendit/settings/applications.py

@@ -8,14 +8,15 @@
     'django.contrib.messages',
     'django.contrib.staticfiles',
     'django_user_agents',
+    'sendit.apps.base',
     'sendit.apps.main',
     'sendit.apps.watcher',
     'sendit.apps.api',
     'sendit.apps.users',
 ]
 
 THIRD_PARTY_APPS = [
-    'social.apps.django_app.default',
+    'social_django',
     'crispy_forms',
     'opbeat.contrib.django',
     'djcelery',

sendit/settings/main.py

@@ -67,8 +67,8 @@
                 'django.template.context_processors.request',
                 'django.contrib.auth.context_processors.auth',
                 'django.contrib.messages.context_processors.messages',
-                'social.apps.django_app.context_processors.backends',
-                'social.apps.django_app.context_processors.login_redirect',
+                'social_django.context_processors.backends',
+                'social_django.context_processors.login_redirect',
                 'sendit.apps.main.context_processors.domain_processor', #custom context processor
                 'sendit.apps.main.context_processors.disqus_processor', #custom context processor
             ],
@@ -152,8 +152,8 @@
 # Static files (CSS, JavaScript, Images)
 # https://docs.djangoproject.com/en/1.9/howto/static-files/
 
-MEDIA_ROOT = '/var/www/images'
-MEDIA_URL = '/images/'
+MEDIA_ROOT = '/var/www/data'
+MEDIA_URL = '/data/'
 STATIC_ROOT = '/var/www/static'
 STATIC_URL = '/static/'
 

sendit/settings/queue.py

@@ -18,7 +18,7 @@
 CELERY_QUEUES = (
     Queue('default', Exchange('default'), routing_key='default'),
 )
-CELERY_IMPORTS = ('sendit.apps.images.tasks',
+CELERY_IMPORTS = ('sendit.apps.main.tasks',
                   'sendit.apps.users.tasks', )
 
 CELERY_RESULT_BACKEND = 'redis://%s:%d/%d' %(REDIS_HOST,REDIS_PORT,REDIS_DB)