-
-
Save AfroThundr3007730/ba99753dda66fc4abaf30fb5c0e5d012 to your computer and use it in GitHub Desktop.
| #!/bin/bash | |
| # Imports DoD root certificates into Linux CA store | |
| # Version 0.4.1 updated 20241216 by AfroThundr | |
| # SPDX-License-Identifier: GPL-3.0-or-later | |
| # For issues or updated versions of this script, browse to the following URL: | |
| # https://gist.github.com/AfroThundr3007730/ba99753dda66fc4abaf30fb5c0e5d012 | |
| # Dependencies: curl gawk openssl unzip wget | |
| set -euo pipefail | |
| shopt -s extdebug nullglob | |
| add_dod_certs() { | |
| local bundle cert certdir file form tmpdir url update | |
| trap '[[ -d ${tmpdir:-} ]] && rm -fr $tmpdir' EXIT INT TERM | |
| # Location of bundle from DISA site | |
| url='https://public.cyber.mil/pki-pke/pkipke-document-library/' | |
| bundle=$(curl -s $url | awk -F '"' 'tolower($2) ~ /dod.zip/ {print $2}') | |
| # Set cert directory and update command based on OS | |
| [[ -f /etc/os-release ]] && source /etc/os-release | |
| if [[ ${ID:-} =~ (fedora|rhel|centos) || | |
| ${ID_LIKE:-} =~ (fedora|rhel|centos) ]]; then | |
| certdir=/etc/pki/ca-trust/source/anchors | |
| update='update-ca-trust' | |
| elif [[ ${ID:-} =~ (debian|ubuntu|mint) || | |
| ${ID_LIKE:-} =~ (debian|ubuntu|mint) ]]; then | |
| certdir=/usr/local/share/ca-certificates | |
| update='update-ca-certificates' | |
| else | |
| certdir=${1:-} && update=${2:-} | |
| fi | |
| [[ ${certdir:-} && ${update:-} ]] || { | |
| printf 'Unable to autodetect OS using /etc/os-release.\n' | |
| printf 'Please provide CA certificate directory and update command.\n' | |
| printf 'Example: %s /cert/store/location update-cmd\n' "${0##*/}" | |
| exit 1 | |
| } | |
| # Extract the bundle | |
| wget -qP "${tmpdir:=$(mktemp -d)}" "$bundle" | |
| unzip -qj "$tmpdir"/"${bundle##*/}" -d "$tmpdir" | |
| # Check for existence of PEM or DER format p7b. | |
| for file in "$tmpdir"/*_dod_{pem,der}.p7b; do | |
| # Iterate over glob instead of testing directly (SC2144) | |
| [[ -f ${file:-} ]] && form=${file%.*} && form=${form##*_} && break | |
| done | |
| [[ ${form:-} && ${file:-} ]] || { printf 'No bundles found!\n' && exit 1; } | |
| # Convert the PKCS#7 bundle into individual PEM files | |
| openssl pkcs7 -print_certs -inform "$form" -in "$file" | | |
| awk -v d="$tmpdir" \ | |
| 'BEGIN {c=0} /subject=/ {c++} {print > d "/cert." c ".pem"}' | |
| # Rename the files based on the CA name | |
| for cert in "$tmpdir"/cert.*.pem; do | |
| mv "$cert" "$certdir"/"$( | |
| openssl x509 -noout -subject -in "$cert" | | |
| awk -F '(=|= )' '{print gensub(/ /, "_", 1, $NF)}' | |
| )".crt | |
| done | |
| # Remove temp files and update certificate stores | |
| rm -fr "$tmpdir" && $update | |
| } | |
| # Only execute if not being sourced | |
| [[ ${BASH_SOURCE[0]} == "$0" ]] || return 0 && add_dod_certs "$@" |
Broken for Ubuntu. I had to make the following changes.
Replace
# Convert the PKCS#7 bundle into individual PEM files
openssl pkcs7 -print_certs -in tmp/*.pem.p7b |
awk 'BEGIN {c=0} /subject=/ {c++} {print > "cert." c ".pem"}'
with
for i in tmp/*_der.p7b; do
openssl pkcs7 -print_certs -inform DER -in $i >> certs.txt
done
for i in tmp/*_pem.p7b; do
openssl pkcs7 -print_certs -inform PEM -in $i >> certs.txt
done
cat certs.txt | awk 'BEGIN {c=0} /subject=/ {c++} {print > "cert." c ".pem"}'
The openssl command didn't like file globbing for the -in argument, and required me to tell it what type of cert with -inform. On top of that, 5 of the 6 certificates downloaded were in DER format with the last in PEM format with a different file naming standard.
@BennieCopeland The main _der.p7b bundle has the same content as the main _pem.p7b bundle, but in DER format. The ca_X_der.p7b bundles have the same content as the main DER bundle but split by issuing CA. So the only bundle needed from the zip file is the PEM format bundle. Since OpenSSL assumes PEM format by default, adding -inform X isn't necessary in this case. Your set of commands will just end up extracting the same certs several times.
Updated the wildcard file match to account for the underscore. Since they keep renaming the zip, I don't want to use a direct URL here still.
Thanks for the update on the file name format. I'll try it out and drop a note if I run into any issues.
Script is all good now. Gave it a test run today. Thank you for updating.
@AfroThundr3007730 Thanks for the information. I wasn't aware of all the duplication in the certificate files. The implementation makes a lot more sense now.
I noticed that a current release, 03NOV2023, of the PKI CA Certificate Bundles: PKCS#7 for DoD PKI Only - Version 5.13 provides a CA certificate bundle that is missing the expected "tmp/*_pem.p7b" file. I have modified my version of your file with the following adjustment to support either case.
...
unzip -qj tmp/${bundle##*/} -d tmp
# Check for existence of PEM format p7b.
if [ -f "tmp/*_dod_pem.p7b" ]; then
echo 'Found PEM formatted file, continuing extraction...'
certform="PEM"
certfile="*_dod_pem.p7b"
else
echo 'Found DER formatted file, continuing extraction and conversion...'
certform="DER"
certfile="*_dod_der.p7b"
fi
# Convert the PKCS#7 bundle into individual PEM files
openssl pkcs7 -inform ${certform} -print_certs -in tmp/${certfile} |
awk 'BEGIN {c=0} /subject=/ {c++} {print > "cert." c ".pem"}'
# Rename the files based on the CA name
...
@senterfd-jrg Thanks for the update. I coincidentally came across the same issue yesterday morning. Appreciate the fix. Tested locally and successfully imported certificates.
Added the above and retested
#!/bin/bash
# Import DoD root certificates into linux CA store
# https://gist.github.com/AfroThundr3007730/ba99753dda66fc4abaf30fb5c0e5d012
main() {
# Location of bundle from DISA site
url='https://public.cyber.mil/pki-pke/pkipke-document-library/'
bundle=$(curl -s $url | awk -F '"' 'tolower($2) ~ /dod.zip/ {print $2}')
#bundle=https://dl.dod.cyber.mil/wp-content/uploads/pki-pke/zip/certificates_pkcs7_v5-6_dod.zip
# Set cert directory and update command based on OS
source /etc/os-release
if [[ $ID =~ (fedora|rhel|centos) ||
$ID_LIKE =~ (fedora|rhel|centos) ]]; then
certdir=/etc/pki/ca-trust/source/anchors
update=update-ca-trust
elif [[ $ID =~ (debian|ubuntu|mint) ||
$ID_LIKE =~ (debian|ubuntu|mint) ]]; then
certdir=/usr/local/share/ca-certificates
update=update-ca-certificates
else
certdir=$1
update=$2
fi
[[ -n $certdir && -n $update ]] || {
echo 'Unable to autodetect OS using /etc/os-release.'
echo 'Please provide CA certificate directory and update command.'
echo 'Example: add-dod-certs.sh /cert/store/location update-cmd'
exit 1
}
# Extract the bundle
cd $certdir
wget -qP tmp $bundle
unzip -qj tmp/${bundle##*/} -d tmp
# Check for existence of PEM format p7b.
if [ -f "tmp/*_dod_pem.p7b" ]; then
echo 'Found PEM formatted file, continuing extraction...'
certform="PEM"
certfile="*_dod_pem.p7b"
else
echo 'Found DER formatted file, continuing extraction and conversion...'
certform="DER"
certfile="*_dod_der.p7b"
fi
# Convert the PKCS#7 bundle into individual PEM files
openssl pkcs7 -inform ${certform} -print_certs -in tmp/${certfile} |
awk 'BEGIN {c=0} /subject=/ {c++} {print > "cert." c ".pem"}'
# Rename the files based on the CA name
for i in *.pem; do
name=$(
openssl x509 -noout -subject -in $i |
awk -F '(=|= )' '{gsub(/ /, "_", $NF); print $NF}'
)
mv $i ${name}.crt
done
# Remove temp files and update certificate stores
rm -fr tmp
$update
}
# Only execute if not being sourced
[[ ${BASH_SOURCE[0]} == "$0" ]] && main "$@"
what are the args to this script supposed to be?
It doesn’t need any I don’t think
The script uses $1 and $2, are those not 2 arguments that get passed in?
If I recall those reference the tokens that awk produces. I will double check.
@danielcjacobs if you look in the if/else block, you'll see that those args are supposed to be set to the certdir and whatever shellscript/binary is being used to update all the cert bundles. the gist tries to provide good defaults, but you can also overwrite if you so desire.
Will this run on mac?
@senterfd-jrg Thanks for catching that. I've updated the script to incorporate it (and fix various shellcheck warnings).
@njohbillie I haven't researched how a mac imports trusted certificates. If it works similarly to Linux certificate management, then you could adapt the script to make it work, if not then you may need to look at another tool to do it.
For the bots that drive search, this works on WSL - Windows Subsystem for Linux Ubuntu nicely. Thank you @AfroThundr3007730 for saving me a bunch of time.
Looks like the naming convention changed again on the .p7b files. I was able to get it to work by modifying the following lines. This is written in a way that no longer depends on "dod" in the filenames.
From:
# Check for existence of PEM or DER format p7b.
for file in "$tmpdir"/*_dod_{pem,der}.p7b; do
# Iterate over glob instead of testing directly (SC2144)
[[ -f ${file:-} ]] && form=${file%.*} && form=${form##*_} && break
done
[[ ${form:-} && ${file:-} ]] || { printf 'No bundles found!\n' && exit 1; }
To:
# Check for existence of PEM or DER format p7b.
for file in "$tmpdir"/*.p7b; do
[[ -f ${file:-} ]] || continue
if [[ $file =~ (_pem|\.pem)\.p7b$ ]]; then
form="pem"
elif [[ $file =~ (_der|\.der)\.p7b$ ]]; then
form="der"
else
continue
fi
break
done
[[ ${form:-} && ${file:-} ]] || { printf 'No valid bundles found!\n' && exit 1; }
@nathanhoeller - Another good catch for the change to the naming convention. Upgrading to this version with your change also revealed that this new version does not replace all the space chars within the certificate's filename with "_" (underscore). I am not a fan of spaces within filenames, so I changed the 1 to "g" in the following gensub() command.
From:
awk -F '(=|= )' '{print gensub(/ /, "_", 1, $NF)}'To:
awk -F '(=|= )' '{print gensub(/ /, "_", "g", $NF)}'
Public.cyber.mil made another change and now the command to initialize bundle, near line 22, is having problems. The previous calls to initialize bundle would return a single value for the variable bundle. Now we are seeing two values. I have added the grep command to use only the line beginning with the phrase http in my file. See below.
From:
bundle=$(curl -s $url | awk -F '"' 'tolower($2) ~ /dod.zip/ {print $2}')To:
bundle=$(curl -s $url | awk -F '"' 'tolower($2) ~ /dod.zip/ {print $2}' | grep "^http")
@senterfd-jrg - I see the spaces in the filenames in /etc/ssl/certs/. It looks like it automatically added the underscores after running the update command, but I can understand wanting to add this to keep the file names consistent. Thanks for the catch! I'll be adding this to my code.
Also, I verified the issue you mentioned with the bundle URL. That's another helpful addition. Thanks!
Is this currently functional, as-is above? I see lots of suggestions for fixes, but wasn't sure if this is still being maintained.
@allebone - I use the code above with some of the modifications that the users have reported. @AfroThundr3007730 did incorporate a bunch of stuff as he mentioned. His script is a big help. The DoD site continually changes their structures which requires a modification to the script. I have come here to this chat and seen recommended updates within a day to hours of required changes based on DoD site changes.
@senterfd-jrg @AfroThundr3007730 I hope my comment wasn’t taken as a complaint or criticism. TBH, I was a little tired and on a small screen with a half-wit understanding of the GIST revisions tab. (Which I didn’t see until today)
I’m just getting started on a base install of 72 servers, and this problem plagues us! (Times a ton)
I’m not scripting savvy but if there is anything I can do to help…let me know. Or I can go back to lurking until someone wants a layman to test. :)
The format of the pem.7b file has changed with the latest PKI certificates. Update line 38 to
openssl pkcs7 -print_certs -in tmp/*pem.p7b |and script will execute without issue. For newer dev environments, it might be good to check for unzip as well.Thanks for the script. Makes life nice and easy.