Slurm-sbatch-email-with-output

Slurm-sbatch-email-with-output.sh

#!/bin/bash
#SBATCH -J MyModel
#SBATCH -n 1 # Number of cores
#SBATCH -t 1-00:00 # Runtime in D-HH:MM
#SBATCH -o JOB%j.out # File to which STDOUT will be written
#SBATCH -e JOB%j.out # File to which STDERR will be written
#SBATCH --mail-type=BEGIN
#SBATCH [email protected]

secs_to_human(){
    echo "$(( ${1} / 3600 )):$(( (${1} / 60) % 60 )):$(( ${1} % 60 ))"
}
start=$(date +%s)
echo "$(date -d @${start} "+%Y-%m-%d %H:%M:%S"): ${SLURM_JOB_NAME} start id=${SLURM_JOB_ID}\n"

### exec task here
( << replace with your task here >> ) \
&& (cat JOB$SLURM_JOB_ID.out |mail -s "$SLURM_JOB_NAME Ended after $(secs_to_human $(($(date +%s) - ${start}))) id=$SLURM_JOB_ID" [email protected] && echo mail sended) \
|| (cat JOB$SLURM_JOB_ID.out |mail -s "$SLURM_JOB_NAME Failed after $(secs_to_human $(($(date +%s) - ${start}))) id=$SLURM_JOB_ID" [email protected] && echo mail sended && exit $?)

Slurm & Sbatch 发送邮件

缘起

slurm 自带的邮件系统--mail-type=all --mail-user={}只带状态没有输出，有点鸡肋，寻求改造如neilmunday/slurm-mail (github.com)需要修改slurm.conf 的MailProg接管但要root权限，用sacct监控如AlisaGU/email_me_on_slurm (github.com)有点舍近求远，How to configure the content of slurm notification emails? - Stack Overflow给了点灵感但不完善

解决方案

结合 内部变量 以及bash命令，输出log文件并调用mailx 发送邮件

确实不错，可以成功发送邮件!
手动点赞!!

This is great! do you have an example on how logs/plots can be attached?

This is great! do you have an example on how logs/plots can be attached?

in this snippet, only default stderr&stdout log will be attached as content by cat JOB$SLURM_JOB_ID.out and pipe to mailx

for more logs/plots, you need to check how the mail client in your env supports adding attachments. as ref:https://stackoverflow.com/questions/902591/how-to-attach-a-file-using-mail-command-on-linux