fb-prophet transformation for aws lambda

fb_prophet_chalice.py

# based on https://github.com/ryansb/sklearn-build-lambda

$ mkdir ~/.aws
$ cat >> ~/.aws/config
[default]
aws_access_key_id=YOUR_ACCESS_KEY_HERE
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
region=YOUR_REGION (such as us-west-2, us-west-1, etc)


rm -rf /srv/python/chalice
virtualenv --python=/usr/bin/python2.7 /srv/python/chalice
source /srv/python/chalice/bin/activate

pip install --upgrade pip
pip install -r ~/PycharmProjects/xxx/requirements/requirements.txt
pip install chalice

cd "$VIRTUAL_ENV/local/lib/python2.7/site-packages"
du -sh
du -sh * | sort -rh | head -10
du -xh -d 1 * | sort -rh | head -20

du -xh -d 1 pystan/* | sort -rh | head -20
du -xh -d 1 scipy/* | sort -rh | head -20
du -xh -d 1 fbprophet/* | sort -rh | head -20


# libdir="$VIRTUAL_ENV/lib64/python2.7/site-packages/lib/"
# mkdir -p $VIRTUAL_ENV/lib64/python2.7/site-packages/lib || true
# cp /usr/lib64/atlas/* $libdir
# cp /usr/lib64/libquadmath.so.0 $libdir
# cp /usr/lib64/libgfortran.so.3 $libdir


echo "venv original size $(du -sh $VIRTUAL_ENV | cut -f1)"
find "$VIRTUAL_ENV/local/lib/python2.7/site-packages" -name "*.so" | xargs strip
echo "venv stripped size $(du -sh $VIRTUAL_ENV | cut -f1)"
find "$VIRTUAL_ENV/local/lib/python2.7/site-packages" -name "test" | xargs rm -rf
find "$VIRTUAL_ENV/local/lib/python2.7/site-packages" -name "tests" | xargs rm -rf
echo "venv tests removed size $(du -sh $VIRTUAL_ENV | cut -f1)"
rm -rf "$VIRTUAL_ENV/local/lib/python2.7/site-packages/pystan/stan/src"
rm -rf "$VIRTUAL_ENV/local/lib/python2.7/site-packages/pystan/stan/lib/stan_math/lib"
echo "venv source codes removed size $(du -sh $VIRTUAL_ENV | cut -f1)"
find "$VIRTUAL_ENV/local/lib/python2.7/site-packages" -name "*.py" | xargs python -m py_compile
find "$VIRTUAL_ENV/local/lib/python2.7/site-packages" -name "*.py" | xargs rm -f
echo "venv py files compiled size $(du -sh $VIRTUAL_ENV | cut -f1)"
find "$VIRTUAL_ENV/local/lib/python2.7/site-packages" -name "*.json" | xargs rm -f
echo "json files removed size $(du -sh $VIRTUAL_ENV | cut -f1)"

"""
import os
import ctypes

for d, _, files in os.walk('lib'):
    for f in files:
        if f.endswith('.a'):
            continue
        ctypes.cdll.LoadLibrary(os.path.join(d, f))
"""


# pushd $VIRTUAL_ENV/lib/python2.7/site-packages/ && zip -r -9 -q /tmp/partial-venv.zip * ; popd
# pushd $VIRTUAL_ENV/local/python2.7/site-packages/ && zip -r -9 --out /outputs/venv.zip -q /tmp/partial-venv.zip * ; popd
# echo "site-packages compressed size $(du -sh /outputs/venv.zip | cut -f1)"

# pushd $VIRTUAL_ENV && zip -r -q /outputs/full-venv.zip * ; popd
# echo "venv compressed size $(du -sh /outputs/full-venv.zip | cut -f1)"

# pushd $VIRTUAL_ENV && zip -r -q /outputs/full-venv.zip * ; popd

# zip -r -q /outputs/full-venv.zip

Hey! I followed your way of removing these extra files but on Python 3.6, and it keeps returning this error when I test in Lambda Function:

File "/var/task/pystan/init.py", line 9, in
from pystan.api import stanc, stan
File "/var/task/pystan/api.py", line 13, in
import pystan._api # stanc wrapper
ModuleNotFoundError: No module named 'pystan._api'

Any chances you would know how to solve this issue?

I found in pystan/_api.cpp, it has:
"include_dirs": [
"./pystan",
"pystan/stan/src",
"pystan/stan/lib/stan_math/",
"pystan/stan/lib/stan_math/lib/eigen_3.3.3",
"pystan/stan/lib/stan_math/lib/boost_1.64.0",
"pystan/stan/lib/stan_math/lib/cvodes_2.9.0/include"
],
Is this error raised because of doing the following command?
rm -rf "$VIRTUAL_ENV/local/lib/python2.7/site-packages/pystan/stan/src"
rm -rf "$VIRTUAL_ENV/local/lib/python2.7/site-packages/pystan/stan/lib/stan_math/lib"

Note to self:
facebook/prophet#179
stan-dev/math#545
https://medium.com/@maebert/machine-learning-on-aws-lambda-5dc57127aee1

got it working finally. Thanks a lot for the help! Here's all the steps I took:
https://medium.com/@amacenov/how-to-get-fbprophet-work-on-aws-lambda-c3a33a081aaf

got it working finally. Thanks a lot for the help! Here's all the steps I took:
https://medium.com/@amacenov/how-to-get-fbprophet-work-on-aws-lambda-c3a33a081aaf

Thanks for your instructions.
Followed your step to run the docker container
docker run --rm -it -v “$PWD”:/var/task lambci/lambda:build-python3.6 bash
Then I created the virtual env (venv), activate venv and upgrade pip. But I'm getting error when installing the fbprophet. This looks like an issue with installing pystan dependency as part of the fbprophet installation. Just wondering if you need to install any gcc libraries within the docker container before you install the fbprophet

pip install fbprophet --no-cache

'    ERROR: Complete output from command /var/task/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-16ok_rcd/fbprophet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ikxedp7w/install-record.txt --single-version-externally-managed --compile --install-headers /var/task/venv/include/site/python3.6/fbprophet:
    ERROR: running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/fbprophet
    creating build/lib/fbprophet/stan_model
    DIAGNOSTIC(S) FROM PARSER:
    Warning: left-hand side variable (name=cp_idx) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
    Warning: left-hand side variable (name=m_pr) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
    
    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_861b75c6337e237650a61ae58c4385ef NOW.
    error: command 'gcc' failed with exit status 4
    ----------------------------------------
ERROR: Command "/var/task/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-16ok_rcd/fbprophet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ikxedp7w/install-record.txt --single-version-externally-managed --compile --install-headers /var/task/venv/include/site/python3.6/fbprophet" failed with error code 1 in /tmp/pip-install-16ok_rcd/fbprophet/

got it working finally. Thanks a lot for the help! Here's all the steps I took:
https://medium.com/@amacenov/how-to-get-fbprophet-work-on-aws-lambda-c3a33a081aaf

Thanks for your instructions.
Followed your step to run the docker container
docker run --rm -it -v “$PWD”:/var/task lambci/lambda:build-python3.6 bash
Then I created the virtual env (venv), activate venv and upgrade pip. But I'm getting error when installing the fbprophet. This looks like an issue with installing pystan dependency as part of the fbprophet installation. Just wondering if you need to install any gcc libraries within the docker container before you install the fbprophet

pip install fbprophet --no-cache

'    ERROR: Complete output from command /var/task/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-16ok_rcd/fbprophet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ikxedp7w/install-record.txt --single-version-externally-managed --compile --install-headers /var/task/venv/include/site/python3.6/fbprophet:
    ERROR: running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/fbprophet
    creating build/lib/fbprophet/stan_model
    DIAGNOSTIC(S) FROM PARSER:
    Warning: left-hand side variable (name=cp_idx) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
    Warning: left-hand side variable (name=m_pr) occurs on right-hand side of assignment, causing inefficient deep copy to avoid aliasing.
    
    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_861b75c6337e237650a61ae58c4385ef NOW.
    error: command 'gcc' failed with exit status 4
    ----------------------------------------
ERROR: Command "/var/task/venv/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-16ok_rcd/fbprophet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ikxedp7w/install-record.txt --single-version-externally-managed --compile --install-headers /var/task/venv/include/site/python3.6/fbprophet" failed with error code 1 in /tmp/pip-install-16ok_rcd/fbprophet/

Try pystan==2.18
Also: https://medium.com/@marc.a.metz/docker-run-rm-it-v-pwd-var-task-lambci-lambda-build-python3-7-bash-c7d53f3b7eb2