Notes on installing pytorch-mask-rcnn

Following https://github.com/multimodallearning/pytorch-mask-rcnn#installation

Setup

Using an anaconda3 Python 3.5 install
CUDA 9 support on GTX 1060
PyTorch from https://github.com/peterjc123/pytorch-scripts (one with GPU support/CUDA 9)

Caveats

Using Anaconda3 instead of Miniconda3
For nms build: nvcc -c -o nms_kernel.cu.o nms_kernel.cu -x cu -Xcompiler -fPIC -ccbin "c:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin -arch=sm_61
updated path to include c:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin
Set the following environmental vars (can set in the System Variables as well):
- set INCLUDE=\Program Files (x86)\Windows Kits\10\Include\10.0.10240.0\ucrt according to this SO post
- set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
- set "VS150COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build"
- set DISTUTILS_USE_SDK=1
- set LIB=\Program Files (x86)\Windows Kits\10\Lib\10.0.10240.0\um\x86;\Program Files (x86)\Windows Kits\10\Lib\10.0.10240.0\ucrt\x86
- set PATH=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin;%PATH%
call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvarsall.bat" x86 -vcvars_ver=14.12

Coco API build

git clone: https://github.com/cocodataset/cocoapi and python setup.py install in PythonAPI library
In Git Bash linked pycocotools with: ln -s cocoapi/PythonAPI/pycocotools/ pycocotools
To get this PythonAPI to compile:

First go to cocoapi\PythonAPI\setup.py and change line 14 from:
extra_compile_args=['-Wno-cpp', '-Wno-unused-function', '-std=c99'],
to
extra_compile_args={'gcc': ['/Qstd=c99']},
as pointed out in this issue: CharlesShang/FastMaskRCNN#173

From: cocodataset/cocoapi#51

michhar/notes_pytorch_maskrcnn_windows.md