MISP Docker Dev Env

This is a short guide on how to set up a development environment for MISP using MISP/misp-docker docker images.

Prerequisites

Install docker and docker-compose.
Clone MISP project repo: git clone --recurse-submodules [email protected]:MISP/MISP.git
Clone MISP/misp-docker repo: git clone https://github.com/MISP/misp-docker.git
Clone MISP/PyMISP repo: git clone https://github.com/MISP/PyMISP.git

From now on we assume that projects are cloned in the /home/myuser directory.

Docker dev env setup

By default MISP/misp-docker uses a local clone of the codebase inside the misp-core docker container. By using a docker-compose.override.yml file we can make it map the MISP repo we checked out in our host machine instead.

This way we can make local changes on our host machine and see the changes reflected in the container.

Configuration

Copy the template.env to .env
Customize .env based on your needs (optional step)

IMPORTANT: Set up the ADMIN_KEY setting in the .env file.

Mapping our local MISP repo

Create docker-compose.override.yml file in /home/myuser/misp-docker/ directory.

docker-compose.override.yml

version: '3'
services:
  misp:
    volumes:
      - "/home/myuser/MISP/app/composer.json:/var/www/MISP/app/composer.json"
      - "/home/myuser/MISP/app/phinx.php:/var/www/MISP/app/phinx.php"
      - "/home/myuser/MISP/app/Console:/var/www/MISP/app/Console"
      - "/home/myuser/MISP/app/Controller:/var/www/MISP/app/Controller"
      - "/home/myuser/MISP/app/View:/var/www/MISP/app/View"
      - "/home/myuser/MISP/app/Model:/var/www/MISP/app/Model"
      - "/home/myuser/MISP/app/webroot:/var/www/MISP/app/webroot"
      - "/home/myuser/MISP/app/Locale:/var/www/MISP/app/Locale"
      - "/home/myuser/MISP/app/Lib/Dashboard:/var/www/MISP/app/Lib/Dashboard"
      - "/home/myuser/MISP/app/Lib/EventReport:/var/www/MISP/app/Lib/EventReport"
      - "/home/myuser/MISP/app/Lib/Export:/var/www/MISP/app/Lib/Export"
      - "/home/myuser/MISP/app/Lib/Tools:/var/www/MISP/app/Lib/Tools"
      - "/home/myuser/PyMISP:/var/www/MISP/app/PyMISP"
      - "./core/files/enable_xdebug.sh:/custom/files/customize_misp.sh"
    extra_hosts:
      - "host.docker.internal:host-gateway"

Share MISP directory group ownership with your host user:

sudo usermod -a -G www-data righel
chgrp righel /var/www
chmod g+rwxs /var/www

XDebug and Docker

A few tweaks are required if you want to debug the php-fpm process running inside the misp docker container.

Add a new entrypoint to the examples folder:

For MISP 2.5.x (PHP>=8.2): core/files/enable_xdebug.sh

#!/bin/bash
echo "INIT | Installing XDebug ..."

# remove lock files
rm /var/lib/apt/lists/lock

# Install XDebug
apt-get update
apt-get install -y --no-install-recommends php-xdebug &&
    apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/*

cat >/etc/php/8.2/fpm/conf.d/20-xdebug.ini <<EOL
zend_extension=xdebug
xdebug.mode = debug
xdebug.client_port = 9999
xdebug.client_host = host.docker.internal
xdebug.idekey = VSCODE
xdebug.xdebug.start_with_request = yes
xdebug.discover_client_host = 1
EOL

echo "INIT | Restarting php-fpm ..."
/etc/init.d/php8.2-fpm restart

For MISP 2.4.x (PHP=7.4): core/files/enable_xdebug.sh

#!/bin/bash
echo "INIT | Installing XDebug ..."

# remove lock files
rm /var/lib/apt/lists/lock

# Install XDebug
apt-get update
apt-get install -y --no-install-recommends php7.4-xdebug &&
    apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/*

# Enable XDebug
cat >/etc/php/7.4/fpm/conf.d/20-xdebug.ini <<EOL
zend_extension=xdebug
xdebug.mode = debug
xdebug.client_port = 9999
xdebug.client_host = host.docker.internal
xdebug.idekey = VSCODE
xdebug.xdebug.start_with_request = yes
xdebug.discover_client_host = 1
EOL

echo "INIT | Restarting php-fpm ..."
/etc/init.d/php7.4-fpm restart

Set enable_xdebug.sh to be executable:
```
chmod +x core/files/enable_xdebug.sh
```

Add a new volume to docker-compose.override.yml and the extra_host key:

  volumes:
    - "/home/myuser/MISP/app/:/var/www/MISP/app/"
    - "./examples/enable_xdebug.sh:/custom-entrypoint.sh"
  extra_hosts:
    - "host.docker.internal:host-gateway"

Configure your IDE to listen incoming XDebug connections on the port 9999.

Example for Visual Studio Code:

launch.json

  {
      "version": "0.2.0",
      "configurations": [
          {
              "name": "Listen for Xdebug",
              "type": "php",
              "request": "launch",
              "port": 9999,
              "pathMappings": {
                  "/var/www/MISP/app": "${workspaceRoot}/app",
              },
          },
      ]
  }

CLI Debug

For debugging MISP shell.

$ export XDEBUG_MODE=debug XDEBUG_SESSION=1

`misp-modules` debugging

TODO

Running MISP dockerized

Start dockerized MISP in deattached mode:

cd /home/myuser/misp-docker
docker-compose build misp-core
docker-compose up -d
docker-compose exec -T --user www-data misp-core bash -c "app/Console/cake Live 1"

After docker-compose spins up the containers you should be able to browse your local dockerized MISP instance here:

https://localhost

Update MISP

On the host machine (/home/myuser/MISP)

git submodule update --init --recursive

docker-compose exec -T --user www-data misp-core bash -c "app/Console/cake Admin updateMISP"

Run MISP tests

In the host machine:

cd /home/myuser/MISP
export HOST=localhost
export AUTH="The ADMIN_KEY you defined in /home/myuser/misp-docker/.env"
python tests/testlive_comprehensive_local.py -v
python tests/testlive_sync.py -v
python tests/testlive_security.py -v
...

Run PyMISP tests

In the host machine, create the keys.py in the /home/myuser/PyMISP/tests/

With the following content:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

url = "https://localhost"
key = "The ADMIN_KEY you defined in /home/myuser/misp-docker/.env"
verifycert = False

Run the tests:

# from the host machine
docker-compose exec misp-core bash
cd app/PyMISP
# inside the misp-docker docker container
# create virtual env and install python dependencies
apt install python3-virtualenv
python3.11 -m virtualenv -p python3 ./venv
. ./venv/bin/activate
export PYTHONPATH=$PYTHONPATH:./app/files/scripts
pip install -r requirements.txt
pip install -r requirements-dev.txt

# run the tests
python -m pytest -v --durations=0 tests/testlive_comprehensive.py
...

Check MISP logs

Application logs are accesible in your host in /home/myuser/misp-docker/logs.

For MISP container logs:

cd /home/myuser/misp-docker
docker-compose logs -f misp

Applying code changes

First, read the MISP project contributing guide.

Fork MISP repository via GitHub UI.

Go to your local clone of MISP project and create a new branch for your fix/feature:

cd /home/myuser/MISP
git remote add myuser [email protected]:myuser/MISP.git
git checkout develop
git checkout -b fix-annoying-bug

Do your code changes.
Keep track of the files you modified using git status and git diff, only add the files you modified to the commits.
Commit your changes, example: git commit -m "fix: remove typo in user view email label"
Push changes to your remote: git push myuser
Check everything is working as expected and then create the pull request via GitHub UI. Always use the develop branch as target for the merge. Add a good description of why you want to get this merged, what issue solves or how to use the feature you want to add providing use-cases if possible.

righel/misp-docker-development.md