Bash Note

Bash scripts and snippets I commonly use.

Powershell

Files

Look for directories with a name contains the string 'anime':

# Navigate to the desired root directory
Set-Location -Path "C:\Path\To\Your\Root\Directory"

# Recursively search for directories with names containing "anime" (case insensitive)
Get-ChildItem -Recurse -Directory | Where-Object { $_.Name -match "anime" }

# Navigate to the desired root directory
Set-Location -Path "C:\Path\To\Your\Root\Directory"

# Recursively search for directories with names containing "anime" (case insensitive)
Get-ChildItem -Recurse -Directory | Where-Object { $_.Name -match "anime" } | Select-Object FullName

Network

Show excluded port range for tcp protocol

netsh interface ipv4 show excludedportrange protocol=tcp

File management

`find` command

find -H -O3 {starting-point} -regextype 'posix-extended' -iregex '{regex}'

find -H -O3 / -path /mapr -prune -o -type f -regextype 'posix-extended' -iregex '.*wikiofspark\.txt' -print

find -H -O3 / \
  -path /mapr -prune -o \
  -newermt "2023-03-17 12:00:00" ! -newermt "2023-03-17 13:43:00" \
  -regextype 'posix-extended' -iregex '{regex}'

`tar` command

Archive while keeping the symlinks

tar -czvf archive.tar.gz /target_folder_to_backup

Archive while with the linked files

tar -czvfh archive_with_linked_files.gz /target_folder_to_backup

Remote connection related SSH | XDMCP | X11 | VNC | RDP

SSH related

View the verbose log of SSH agent

ssh -v <userID>@<ip>

SSH agent config - use exclusive config | option for different remote hosts

# ~/.ssh/config
Host github_com_aruruka
    ProxyCommand /usr/bin/nc -Xconnect -x172.24.208.1:10809 %h %p
    AddKeysToAgent yes
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_ed25519_github

ssh -v -T [email protected]

tee -a ~/.profile <<-'EOF'
# Add HTTP proxy for SSH agent
sed -ri '/ProxyCommand/d' ~/.ssh/config && \
echo -e "\tProxyCommand /usr/bin/nc -Xconnect -x`cat /etc/resolv.conf | grep nameserver | cut -d ' ' -f 2`:10809 %h %p" >> ~/.ssh/config
EOF

Add your SSH key to the ssh-agent

# start the ssh-agent in the background
$ eval `ssh-agent -s`
# > Agent pid 59566

ssh-add ~/.ssh/id_ed25519_github

tee -a ~/.profile <<-'EOF'
sshAgentPid=$(ps aux | grep -Ei ssh-agent | grep -Ev 'grep' | sed -r 's|\s+| |g' | cut -d ' ' -f 2) && \
if [[ -z ${sshAgentPid} ]]; then \
    echo "ssh-agent is not running yet..."; \
    echo "Now starting the ssh-agent..."; \
    eval `ssh-agent -s` && \
    echo "Starting ssh-agent successfully..."; \
    sleep 1; \
fi && \
ssh-add ~/.ssh/id_ed25519_github
EOF

Add SSH public key to remote host

ssh mutual trust password-free login settings

# Generate SSH keys on master node:
ssh-keygen -t ed25519 -P '' -C "[email protected]"

# Use `ssh-copy-id` to copy the public keys to worker nodes:
ssh-copy-id root@<node1> && \
ssh-copy-id root@<node2> && \
ssh-copy-id root@<node3>

# Or specify public key file to deliver:
ssh-copy-id -i ${identity_file} ${USER}@${target_host}    

# Then copy the private key to worker nodes, so the workers can log into the master without password as well:
scp /root/.ssh/id_ed25519 root@<node2>:/root/.ssh && \
scp /root/.ssh/id_ed25519 root@<node3>:/root/.ssh

# To verify:
# On every node:
timeout 2 ssh -o StrictHostKeyChecking=no root@node1 command 'echo "hello"' && \
timeout 2 ssh -o StrictHostKeyChecking=no root@node2 command 'echo "hello"' && \
timeout 2 ssh -o StrictHostKeyChecking=no root@node3 command 'echo "hello"'

Alternatively, you can:

timeout 2 ssh -o StrictHostKeyChecking=no {userName}@{remoteHost} command 'echo "hello"'
if [[ $? -ne 0 ]]; then
    echo 'public key not exist on {remoteHost}'
fi
cat ~/.ssh/id_ed25519.pub | ssh -o StrictHostKeyChecking=no -i <SSH_PRIVATE_KEY> ${USER}@${target_host} "mkdir -p ~/.ssh; cat >> ~/.ssh/authorized_keys; chmod 600 ~/.ssh/authorized_keys"

Configuring no-port-forwarding, no-agent-forwarding, no-x11-forwarding for a SSH public key:

cat ~/.ssh/authorized_keys
no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"centos\" rather than the user \"root\".';echo;sleep 10" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQe/zRZzbgpbgVFrEy45+MJbTGbZhPucF/glYD+vezV397WEYvWufuX6aTd7B2J3r/hPCndpTEpZCSBEbCaxJIzZ9LGdexlIE4Usl7K82/7tt0XSWFGYvg9YsK5Tr7uUfrMlKLuLgAmJMkMO42irrHjzNMakilgSs+HyoBhTmAbTmBFGvJ4TsGoPCnaTxHMWk/dmFrHecePAvdFF/ANlMhu0PIIOVROj2YrmTHdlWShVNyav6L+K/D3pNW601VH1EoZ80kEMIyaR2Iw+Q1W2+WnKjtjCXW/yb83UA89vGI5I4rYQNhQAi50pjts2cT4NBbRpWo2K6VE+KBHw2dT77BdLG5kLSyzOu0T7U/8Q0vE5Uw6TA0fxnPfydobPHFSVnEgf1g50HCT6l2jHbrZjwfZWF5Mq+STauqF6qasGEYrPbKbcc8Q7iGSYsE+GJVOX4jZxpE4BXmy2QZJ4TiQj7M/pCqvCtbmSVDi+DqSh38eaFJbMOb8qx3qWFvkMsnJam+/yffT/mANFSUT3XTbQGodK52YWRHK9TjVLPREAeP19opTzJnYirq3NCNOWB5M1tsRolsPCf0nwJxtnsu2aQcB4Sla6SMhJfD3tHmZWZVwHBvkPivTBba4okRzkF98khQ3ZVhMR7xzE5AYVJ1Hlta2AeuDwSyVaiLwOXCXcLyxw== kyan@kyan-5540

REMOTE HOST IDENTIFICATION HAS CHANGED!

Add correct host key in /home/shouneng/.ssh/known_hosts to get rid of this message. Offending ECDSA key in /home/shouneng/.ssh/known_hosts:7 remove with: ssh-keygen -f "/home/shouneng/.ssh/known_hosts" -R "host-10-17-102-138.coe.cloudera.com" ECDSA host key for host-10-17-102-138.coe.cloudera.com has changed and you have requested strict checking.

Remove known hosts

ssh-keygen -f "/home/shouneng/.ssh/known_hosts" -R "host-10-17-102-138.coe.cloudera.com"

shell script tips

set These four options of the command set are commonly used together:

# manner-1
set -euxo pipefail

# manner-2
set -eux
set -o pipefail

sudo execute long command:

cd /data/ && sudo -- sh -c 'su -c "pwd;whoami" appuser'

sudo -i -- bash -c 'su appuser --shell "COMMAND-A && COMMAND-B || COMMAND-C"'

cat<<'EOF' | sudo -i -- bash -c
uname -rn;
DEFAULT_INF=`ip route show | grep default | awk '{print $5}'` && \
    ip addr show $DEFAULT_INF | grep inet | awk '{ print $2; }' | sed 's/\/.*$//'
EOF

xargs concatenate stdin and a string:

ls -1 /opt/mapr/hadoop/hadoop-2.7.6/etc/hadoop | xargs -n 1 printf -- '/opt/mapr/hadoop/hadoop-2.7.6/etc/hadoop/%s\n' | xargs -n 1 grep -H -n -Ei '.*node.*manager.*'

Bash parameter expansion

Replace spaces in file names using a bash script:
```
for f in *\ *; do mv "$f" "${f// /_}"; done
```
```
for file in ./*; do mv "$file" "${file//(internal_view)/-internal_view}"; done
```
The ${f// /_} part utilizes bash's parameter expansion mechanism to replace a pattern within a parameter with supplied string. The relevant syntax is ${parameter/pattern/string}. See: https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
or http://wiki.bash-hackers.org/syntax/pe.

aruruka/bash_note.md