How to easily use multiprocessing in Python (on a SLURM cluster)

python_multiprocessing_slurm.py

import os
from multiprocessing import Pool, cpu_count

# function you want to run in parallel:
def myfunction(a, b):
  return a + b

# list of tuples to serve as arguments to function:
args = [(1, 2), (9, 11), (6, 2)]

# number of cores you have allocated for your slurm task:
number_of_cores = int(os.environ['SLURM_CPUS_PER_TASK'])
# number_of_cores = cpu_count() # if not on the cluster you should do this instead

# multiprocssing pool to distribute tasks to:
with Pool(number_of_cores) as pool:
    # distribute computations and collect results:
    results = pool.starmap(myfunction, args)
    

Thank you, very much!
It's save my day (or week, or ...)

Thank you. I wonder how to manage that number of cpus properly in the case of multiple nodes allocation and multiple tasks with a number of cpu per task ?

For instance in slurm :
#SBATCH --ntasks=4
#SBATCH --nodes=4
#SBATCH --cpus-per-task=5

Thus a total of 4*5 = 20 cpus.

I don't :) On 14 Jun 2024 at 13.47 +0200, Nicolas Mendiboure ***@***.***>, wrote: @nmendiboure commented on this gist.

…

________________________________ Thank you. I wonder how to manage that number of cpus properly in the case of multiple nodes allocation and multiple tasks with a number of cpu per task ? For instance in slurm : #SBATCH --ntasks=4 #SBATCH --nodes=4 #SBATCH --cpus-per-task=5 Thus a total of 4*5 = 20 cpus. — Reply to this email directly, view it on GitHub<https://gist.github.com/kaspermunch/64e11cd21e3953295e149e75bee51733#gistcomment-5088937> or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAI46BFR522424K6QCBQ4QLZHLJ3DBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTANJZGYZTKNZTU52HE2LHM5SXFJTDOJSWC5DF>. You are receiving this email because you authored the thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.