rsync_automation.md

Automate Data Transfers Between CNS and the HPC at NYU (Synology use case)

---

Scope:

This doc covers how to set up an environment that allows one to Rsync data to/from CNS and the HPC using: a shell script, cron job, SSH keys, and Sendmail.

For reference:

source = local (hostname.cns.nyu.edu)

destination = remote (greene.hpc.nyu.edu)

Synology (source)

---

Background:

Synology is a brand of network attached storage (NAS) running a paired-down version of Debian OS. While you can accomplish many tasks via the NAS's Disk Station Manager (DSM), it has limits. The NAS's command line interface (CLI) is NOT well documented, but the BusyBox CLI reference is analogous.

For a utility's syntax, do:

utility name --help

Prereqs:

I. Install Sendmail

The following packages are required to emit email messages:

Perl

PHP 7.4

Synology Mail Server

and may be added via DSM's Package Manager.

Note: Mail Station package appears to do the above in one (1) package.

II. Test Sendmail

Create a test email, e.g.:

vi sendmail_test.sh

#!/usr/bin/env bash

timestamp="date +'%b %d %Y %X'"

subject="Test Sendmail Utility"
from="username@domain"
to="username@gdomain"

mail_cmd="sendmail -f $from -t $to"

{
  printf "Subject:%s\n" "$subject"
  printf "%s\n" "Your mission for today: $timestamp, should you choose to accept it..."
} | $mail_cmd`

Send a test email:

`bash sendmail_test.sh`

III. Configure DNS

Manually configure DNS resolver (defaults may be enough) via DSM:

Control Panel -> Network -> Manually Configure DNS Server (enable via checkbox) ->

• Preferred DNS server: 128.122.0.71;

• Alternative DNS server: 128.122.253.46 (e.g.).

IV. Configure SSH

Note: ONLY local admins can SSH in/out directly on a Synology.

Note: The Local admin account needs a home directory, e.g.: /volume1/homes/username/ . Create one first if it doesn't exist.

Enable SSH via DSM:

Control Panel > Terminal & SNMP > SSH (enable via checkbox)

Test SSH:

ssh [email protected]

Then, from ~/:

mkdir .ssh
touch .ssh/authorized_keys

Restrict access to:

• user's home dir

  chmod 700 .

• .ssh dir

  chmod 700 .ssh

• authorized_keys

  chmod 600 .ssh/authorized_keys

Generate SSH keys:

ssh-keygen -t rsa -b 4096

Allow SSH keys:

Note: Enable telnet via DSM for a safety net.

Note: Make a backup copy of sshd_config, e.g. sshd_config.bak*

Edit sshd_config file:

sudo vi /etc/ssh/sshd_config

Remove comments here:

# PubkeyAuthentication yes
# AuthorizedKeysFile .ssh/authorized_keys

Restart SSH service

sudo synoservicectl --reload sshd

Test that SSH still works w/out keys.

Note: Disable telnet via DSM if enabled earlier.

HPC (destination)

---

Background:

"High Performance Computing (HPC) provides supercomputer access and supporting software for researchers who need powerful processing resources." -- source. This doc will reference the "greene" cluster at New York University (NYU).

Prereqs:

I. Valid account on HPC.

II. Configure SSH keys on HPC:

Get id_rsa.pub key for local admin on Synology to your dir on HPC: ~/.ssh/authorized_keys

If things are set correctly, you should be able to SSH from Syno to the HPC w/out a password prompt:

ssh [email protected]

Data Transfer (source)

---

Background:

Per the Rsync man page: "rsync -- a fast, versatile, remote (and local) file-copying tool."

For reference:

rsync --help

I. Test sample Rsync command from source to destination

Note: scripts is the local dir being transferred to the remote dir Testy_McTest:

rsync --archive --compress --verbose --human-readable scripts [email protected]:/home/netID/Testy_McTest

II. Create Rsync script

vi rsync_test.sh e.g.:

 #!/usr/bin/env bash

 timestamp="date +'%b %d %Y %X'"

 subject="Rsync Job Status"
 from="username@domain"
 to="username@gdomain"

 mail_cmd="sendmail -f $from -t $to"

 {
  printf "Subject:%s\n" "$subject"
  rsync --archive --compress --verbose --human-readable scripts [email protected]:/home/netID/Testy_McTest  
 } | $mail_cmd`

Create Cronjob (source)

I. Edit crontab, e.g.:

Note: Crontab entries in Synology must be run by root, though you can switch user (su) to a desired UID:

sudo vi /etc/crontab

0   0   *   *   *   root    /bin/su -c "/usr/local/bin/rsync_test.sh" marshki

II. Restart cron service

sudo synoservicectl --reload crond

Sample transcript:

Rsync completed successfully on: greene.hpc.nyu.edu@Mar 31 2022 12:45:13 PM.
rsync.log written to:/volume1/homes/marshki/log