Poor man's parallel execution on servers

poor_man_parallel.sh

#!/usr/bin/env bash

set -e

for N in {1..5}
do
  for M in {0.1,1,2,5,10};
  do
    # Replace ./my_prog.py with your program to run it in parallel
    ./my_prog.py -N $N -M $M > output/stdout_N${N}_M${M}.txt &
    echo $!
  done
done

## Run as:
## ./pool_man_parallel.sh > pids
## Then kill recalcitrant processes from `pids` file, if needed.

Nice one! As an alternative, you could kill all the processes by killing the pid of the parent, e.g. pkill -P <parent_pid>.

Also, if you are not sure if you have any bugs in my_prog.py, and you want the script to die and stop spawning new processes if an exception happens, you can use set -e in the beginning.

@harrymvr Thanks for the set -e idea, I've included it in the script. I didn't know about pkill -P trick, but won't it kill the shell itself?

@musically-ut pkill -P does not kill the TERM, but I think that it killed the forked processes. Let me know, because I might need to change that in my scripts too 😄

@harrymvr The PPID of the processes put in background seems to be 1, i.e. the init process:

I am not too keen on running pkill -P 1. <_<

hmmm interesting. Yeah, I would pass on that as well 😆