ejokeeffe

#hide
%matplotlib inline
import pandas as pd
import seaborn as sns
from scipy.stats import gamma
import matplotlib.pyplot as plt
from matplotlib import dates as mdates
import numpy as np

sns.set()

ejokeeffe

#hide
%matplotlib inline
import seaborn as sns
import matplotlib.pyplot as plt

sns.set()
sns.set_context('talk',font_scale=0.7, rc={"lines.linewidth": 1.5})
sns.set_style("whitegrid")
sns.set_color_codes("dark")
sns.set_palette(sns.color_palette("Set1", n_colors=8, desat=.5))

ejokeeffe

#!/bin/bash
#
# Script to setup a Elastic Beanstalk AMI with geospatial libraries and postGIS
#
# sh aws_ami_prep.sh > aws_ami_prep.log 2>&1 &

# Go to ec2-user home directory
cd /home/ec2-user

# yum libraries

ejokeeffe

Introduction

This is just a wee play to see how to set luigi up on aws. This was driven by own misunderstanding of how luigi works and the need to get it set up on AWS. Most, if not all, of what's covered below will seem ridiculuously obvious to people.

AWS Setup

Launch an aws instance, droppping this code into the user_data field in advanced settings. You need to add your own environment name and reference the luigi setup script for git to pull down and run, which is basically adding the code below to the base of the script.

ejokeeffe

Quick Instructions

These instructions are not exhaustive, check out the links at the bottom for a more detailed treatment.

I'm assuming that you have the conda environment setup and activated on a linux ec2-instance on aws and you're currently on the shell. Run the following:

conda install jupyter
jupyter notebook --generate-config

Then you need to generate the password. Open ipython and type the following:

ejokeeffe

sudo: false
language: python
virtualenv:
  system_site_packages: true
services:
env:
  matrix:
    - DISTRIB="conda" PYTHON_VERSION="3.4" COVERAGE="true"
    - DISTRIB="conda" PYTHON_VERSION="3.5" COVERAGE="false"
addons:

ejokeeffe

Set up the AWS Instance Launch AWS instance under a security group with HTTP (port 80) and (port 22)ssh access. Additionally, as you'll likely be running a test site using django - open up the 8000 port.

Update the installer, sudo yum update -y.

Install python using miniconda following these instructions. Create the environment, also installing pip, conda env create -n django --yes python=3.5 pip and activate it source activate django.

Set up the database I tend to set up a separate postgres instance in in RDS, rather than install on the instance. Go go to RDS on AWS and click Launch DB Instance selecting postgres.

ejokeeffe

from sklearn import linear_model
from scipy import stats
import numpy as np


class LinearRegression(linear_model.LinearRegression):
    """
    LinearRegression class after sklearn's, but calculate t-statistics
    and p-values for model coefficients (betas).
    Additional attributes available after .fit()