script to gather secrets from h.vault

README.md

Vault Template Processor Script

DISCLAIMER

This is a script made because having a good system to handle secrets in pipeline is apparently too hard to implement. It is given as is will probably bug if you push it a bit too much. I will probably never maintain this script. so use it at your own risk.

Overview

This Bash script automates the process of retrieving secrets from HashiCorp Vault and using them to generate configuration files from templates. It searches for files with the .vaulttpl extension in the current directory, replaces Vault placeholders with actual secrets, and outputs the processed files without the .vaulttpl extension.

You can have it to generate a self contained script that will encapsulate all secrets found and encrypt them using AES. Useful when git forges does not care about security of users.

Prerequisites

• Vault CLI: The script requires the HashiCorp Vault CLI to be installed and accessible in the system's PATH.
• Vault Access: You must have access to a Vault instance and authentication credentials.
• OpenSSL: You might need openssl in case you want to use the hydrator system
• Environment Variables:
  • VAULT_ADDR: Specifies the Vault server address.
  • VAULT_TOKEN: If not set, the script will attempt to log in using LDAP authentication.
  • VAULT_USER and VAULT_PASSWORD: Required for LDAP authentication if VAULT_TOKEN is not provided.
  • VAULT_TEMPLATE_KEY: (optional) password of the generated hydrate.sh script that will encapsulate secrets in a single encrypted file
  • VAULT_TEMPLATE_HYDRATOR_PATH: (default=secrets/hydrate.sh)

Usage

1. Ensure Vault CLI is installed and configured.
2. Set the required environment variables for Vault authentication.
3. Run the script:

   vault-template

4. The script will:
   • Authenticate to Vault (if VAULT_TOKEN is not already set).
   • Search for all .vaulttpl files in the current directory.
   • Replace Vault placeholders with actual secret values.
   • Generate the final configuration files without the .vaulttpl extension.

Functions

• vget <path> <field>: Retrieves a specific field from a secret stored in Vault.
• escape <char>: Escapes a given character in a string.
• escape_dollar_sign: Escapes dollar signs ($) to prevent unintended variable substitution.

Example

If you have a template file named config.json.vaulttpl:

{
  "database_password": "$(vget secret/data/db password)"
}

After running the script, it will generate config.json with the actual secret value from Vault.

Notes

• The script uses set -e -o pipefail to ensure it exits immediately if any command fails.
• Templates are processed using eval, so ensure input files are trusted to avoid command injection risks.

Troubleshooting

• Authentication Issues: Ensure VAULT_USER and VAULT_PASSWORD are correctly set if using LDAP authentication.
• Permission Denied: Verify that your Vault token has access to the required secrets.
• Missing Dependencies: Ensure the Vault CLI is installed and accessible.

This script simplifies the management of secret-based configurations, making it easier to maintain secure applications.

Usage in bitbucket-pipelines

• Ensure you have some templates defined like described before in a folder named secrets at the root of your project

• Ensure you have a repository variable only accessible by admin named VAULT_TEMPLATE_KEY with a random password

• Add a step to gather secrets in your pipeline definition:

      - step:
          name: Gather secrets
          image: <your image>
          runs-on:
            - self.hosted
            - linux
            - docker
          artifacts:
            - secrets/hydrate.sh
          script:
            - export VAULT_USER=$LDAP_USER
            - export VAULT_PASSWORD=$LDAP_PASSWORD
            - vault-template
      - step:
          name: test
          image: alpine
          script:
            - apk add openssl
            - ./secrets/hydrate.sh 

Dockerfile

FROM hashicorp/vault:1.18

ENV VAULT_USER=""
ENV VAULT_PASSWORD=""

COPY ./vault-template.sh /usr/local/bin/vault-template

RUN apk add --no-cache bash jq openssl \
  && chmod a+x /usr/local/bin/vault-template

vault-template.sh

#!/bin/bash

set -e -o pipefail

export VAULT_ADDR=${VAULT_ADDR:-"https://vault.x.net"}
VAULT_TEMPLATE_HYDRATOR_PATH=${VAULT_TEMPLATE_HYDRATOR_PATH:-"secrets/hydrate.sh"}

if [ -z $VAULT_TOKEN ]; then
  vault login -method=ldap \
    username=$VAULT_USER password=$VAULT_PASSWORD
fi

function vget {
  local path="$1"
  local field="$2"
  vault kv get -field="$field" "$path"
}

function escape {
   sed -e "s/$1/\\\\$1/g"
}
function escape_dollar_sign {
   sed -e 's/\$/\\$/g'
}

if [ ! -z $VAULT_TEMPLATE_KEY ]; then
  cat <<EOF > $VAULT_TEMPLATE_HYDRATOR_PATH
#!/usr/bin/env bash
set -e -o pipefail
EOF
chmod a+x $VAULT_TEMPLATE_HYDRATOR_PATH
fi 

function add_encrypted {
  if [ -z $VAULT_TEMPLATE_KEY ]; then return 0; fi
  local src="$1"

  cat <<EOF >> $VAULT_TEMPLATE_HYDRATOR_PATH

echo "Decrypting file '$src'"
cat <<FILE_CONTENT | openssl enc -d -aes-256-cbc -pbkdf2 -pass "pass:\$VAULT_TEMPLATE_KEY" -in - -out /dev/stdout -base64 > "$src"
$(cat "$src" | openssl enc -aes-256-cbc -pbkdf2 -pass "pass:$VAULT_TEMPLATE_KEY" -in - -out /dev/stdout -base64)
FILE_CONTENT
chmod $(stat -c '%a' "$src") "$src"
EOF
}

for src_file in $(find . -name "*.vaulttpl"); do
  dst_file=$(echo "$src_file" | sed -e 's/\.vaulttpl$//')
  echo "Build file '$dst_file'"
  template=$(cat <<EOVAULTTPL 
cat <<EOVTPL
$(cat "$src_file")
EOVTPL
EOVAULTTPL
  )
  eval "$template" > "$dst_file"
  chmod $(stat -c '%a' "$src_file") "$dst_file"
  add_encrypted "$dst_file"
done