design_document.md

Design Document

Environment

VMs

• Main CMS portal
  • Public Facing portal
  • Admin portal (behind VPN)
• Main API
• DB (MySQL vs Postgres vs NoSQL)
  • MySql
    • MySQl lighter?
    • Probably better supported by some 3rth party libraries.
  • Postgres
    • https://docs.djangoproject.com/en/1.10/ref/contrib/postgres/
    • ArrayField
    • JSONField
    • FullText Search
  • NoSQL (Mongo)
    • https://github.com/django-nonrel
    • Still an independent branch.
• Data (Per request)
• Dev Portal (Just two levels)

Django

• Settings
  • Session store in memcache.
  • Do not push SECRET_KEY to a public repo.
  • Security settings:
    • https://docs.djangoproject.com/en/1.10/topics/security/
    • https://www.ponycheckup.com/
  • Folder Structure:

   + project_name
     + docker
       - Dockerfile
       - docker-compose.yml
     + conf
       + keys
         - dummy.crt
         - dummy.key
       + server
         - nginx.conf
         - uwsgi.ini
         - project_name.service
         - error.html
       + git_hooks
         - pre-commit
         - pre-push
       - project_name.env
     + bin
       - admin_script.sh
       - colorize.sh
     + portal
       + apps
         + django_app
       + libs
         + custom_library 
       + utils
         + custom_util
         - celery.py
       + static
       + templates
       + settings
         - common_settings.py
         - local_settings.py
         - elasticsearch_settings.py
         - celery_settings.py
     - .gitignore
     - README.py
     - requirements.txt
     - manage.py
  

  • pre-push example: https://trello.com/c/xmww8iU7/620-git-hooks
  • colorize examples: https://github.com/xirdneh/misc-scripts/tree/master/awk/colorized/django
  • admin scripts examples: https://github.com/DesignSafe-CI/portal/tree/master/bin

Celery

• Rabbit AMQP
• Flower

Django-Channels

• Redis
• Webhook callbacks
• Websockets
• Data Binding from the backend

ElasticSearch

• Analyzers
• Data Structure
• Processing before indexing:
  • Domain Lemmatisation
  • Topic extraction
  • Tagging

Permissions

• Different types of users
• Are we gonna have different access levels for the API?
• Portal permissions vs Agave permissions.

Notifications

• Single application to handle notifications.
  • When bootstrapping AngularJS form elements must have an action attribute or handled by some JS code.
• (Front-end) Handle any kind of notification. e.g. Agave notifications or notifications sent to users.
• Rely on Django-Channels to process external notifications.
• Should act only as a service layer and then hand out the processing to different apps, libs, utils, etc...
• Front-end and back-end should work similarly. As in just acting as a service layer.
• Format:
  • Type (for mapping to the correct module/service. e.g. data, jobs, projects)
  • Operation (for user configuration, logging and mapping. e.g. file_sharing, job_submit, data_publication)
  • Message (for user display)
  • Content (unmodified object resultant of the operation)
• How to display notifications list?
  • API call to draw insight from different contents?

Overall Structure

• Framework agnostic library
  • Converts agavepy into a friendly object oriented library (agavepy_dsl)
  • Gives plenty of room for extensibility.
• Django-app
  • Views and models necessary to work with django
  • Extensibility is paramount.
• Front-end
  • Services that know how to talk to its corresponding api.
  • Directives to show corresponding data.
  • Providers to be able to customize look and feel.
  • Anular style guide: https://github.com/johnpapa/angular-styleguide

Testing

• Backend tests (without calling any external service)
• Front end tests (without calling any backend service)
• Functionality tests (backend and frontend without calling any external service)
• Integration tests (careful with this ones)

Logging

• A logging format should be defined.
• When running localhost should log to a local logfile.
• Format:
  • Username
  • SessionID
  • IP
  • Operation pertaining to the request
  • Extra data of each step.
• DEBUG for actual debugging, INFO for metrics?

Development workflow

• Each module should be developed following the Overall Structure points.
• Each structure point should be develope with its corresponding test suite.
• Logging features for each structure point should be evaluated before marked as finished.
• Documentation should for each structure point should be revise before marked as finished.
• Corresponding Splunk reports might be created before marked as finished.