design_document.md

Design Document

Environment

VMs

• Main CMS portal
  • Public Facing portal
  • Admin portal (behind VPN)
• Main API
• DB (MySQL vs Postgres)
• Data
• Dev Portal

Django

• Settings
  • Session store in memcache
  • Folder Structure:

   + project_name
     + docker
       - Dockerfile
       - docker-compose.yml
     + conf
       + keys
         - dummy.crt
         - dummy.key
       + server
         - nginx.conf
         - uwsgi.ini
         - project_name.service
         - error.html
       - project_name.env
     + bin
       - admin_script.sh
     + portal
       + apps
         + django_app
       + libs
         + custom_library 
       + utils
         + custom_util
       + static
       + templates
       + settings
         - common_settings.py
         - local_settings.py
         - elasticsearch_settings.py
         - celery_settings.py
     - .gitignore
     - README.py
     - requirements.txt
     - manage.py
  

Celery

• Rabbit AMQP
• Flower

Django-Channels

• Redis
• Use for Webhooks callbacks
• Use for Websockets

ElasticSearch

• Analyzers
• Data Structure

Permissions

• Different types of users
• Are we gonna have different access levels for the API?
• Portal permissions vs Agave permissions.

Notifications

• Single application to handle notifications.
• Handle any kind of notification. e.g. Agave notifications or notifications sent to users.
• Rely on Django-Channels to process external notifications.
• Should act only as a service layer and then hand out the processing to different apps, libs, utils, etc...
• Front-end and back-end should work similarly. As in just acting as a service layer.

Overall Structure

• Framework agnostic library
  • Converts agavepy into a friendly object oriented library (agavepy_dsl)
  • Gives plenty of room for extensibility.
• Django-app
  • Views and models necessary to work with django
  • Extensibility is paramount.
• Front-end
  • Services that know how to talk to its corresponding api.
  • Directives to show corresponding data.
  • Providers to be able to customize look and feel.

Testing

• Backend tests (without calling any external service)
• Front end tests (without calling any backend service)
• Functionality tests (backend and frontend without calling any external service)
• Integration tests (careful with this ones)

Logging

• A logging format should be defined.
• When running localhost should log to a local logfile.

Development workflow

• Each module should be developed following the Overall Structure points.
• Each structure point should be develope with its corresponding test suite.
• Logging features for each structure point should be evaluated before marked as finished.
• Documentation should for each structure point should be revise before marked as finished.
• Corresponding Splunk reports might be created before marked as finished.