Add default mem_mb, disk_mb, tmpdir

[leoisl]

snakemake accepts a --default-resources option that allows us to specify the default mem_mb, disk_mb and tmpdir if a rule did not specify these directives. We can ask for these values in the cookie-cutter, and in config.yaml they can be specified using the following format:

default-resources: ["mem_mb=7777", "disk_mb=8888", "tmpdir='/hps/nobackup/research/zi/leandro/snakemake_test/temp'"]

Adding this to my config.yaml, and submitting this dummy job, which does not specify any of these directives:

rule make_file:
    output:
        "new_file.txt"
    shell:
        "touch {output}"

I got the following log:

[Tue Apr 19 11:39:19 2022]
rule make_file:
    output: new_file.txt
    jobid: 1
    resources: mem_mb=7777, disk_mb=8888, tmpdir=/hps/nobackup/research/zi/leandro/snakemake_test/temp

Job script is:

#!/bin/sh
# properties = {"type": "single", "rule": "make_file", "local": false, "input": [], "output": ["new_file.txt"], "wildcards": {}, "params": {}, "log": [], "threads": 1, "resources": {"mem_mb": 7777, "disk_mb": 8888, "tmpdir": "/hps/nobackup/research/zi/leandro/snakemake_test/temp"}, "jobid": 1, "cluster": {}}
cd /hps/nobackup/research/zi/leandro/snakemake_test && /hps/nobackup/research/zi/leandro/miniconda3/bin/python -m snakemake --snakefile '/hps/nobackup/research/zi/leandro/snakemake_test/Snakefile' 'new_file.txt' --allowed-rules 'make_file' --cores 'all' --attempt 1 --force-use-threads  --wait-for-files '/hps/nobackup/research/zi/leandro/snakemake_test/.snakemake/tmp.ftj3136o' --force --keep-target-files --keep-remote --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --skip-script-cleanup  --conda-frontend 'mamba' --wrapper-prefix 'https://github.com/snakemake/snakemake-wrappers/raw/' --latency-wait 10 --scheduler 'greedy' --scheduler-solver-path '/hps/nobackup/research/zi/leandro/miniconda3/bin' --default-resources 'mem_mb=7777' 'disk_mb=8888' "tmpdir='/hps/nobackup/research/zi/leandro/snakemake_test/temp'" --mode 2 && exit 0 || exit 1

if we instead specify mem_mb in the job:

rule make_file:
    output:
        "new_file.txt"
    resources: mem_mb=100
    shell:
        "touch {output}"

we get this log:

[Tue Apr 19 11:49:00 2022]
rule make_file:
    output: new_file.txt
    jobid: 1
    resources: mem_mb=100, disk_mb=8888, tmpdir=/hps/nobackup/research/zi/leandro/snakemake_test/temp

with this job script:

#!/bin/sh
# properties = {"type": "single", "rule": "make_file", "local": false, "input": [], "output": ["new_file.txt"], "wildcards": {}, "params": {}, "log": [], "threads": 1, "resources": {"mem_mb": 100, "disk_mb": 8888, "tmpdir": "/hps/nobackup/research/zi/leandro/snakemake_test/temp"}, "jobid": 1, "cluster": {}}
cd /hps/nobackup/research/zi/leandro/snakemake_test && /hps/nobackup/research/zi/leandro/miniconda3/bin/python -m snakemake --snakefile '/hps/nobackup/research/zi/leandro/snakemake_test/Snakefile' 'new_file.txt' --allowed-rules 'make_file' --cores 'all' --attempt 1 --force-use-threads  --wait-for-files '/hps/nobackup/research/zi/leandro/snakemake_test/.snakemake/tmp.iq82hnq_' --force --keep-target-files --keep-remote --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --skip-script-cleanup  --conda-frontend 'mamba' --wrapper-prefix 'https://github.com/snakemake/snakemake-wrappers/raw/' --latency-wait 10 --scheduler 'greedy' --scheduler-solver-path '/hps/nobackup/research/zi/leandro/miniconda3/bin' --default-resources 'mem_mb=7777' 'disk_mb=8888' "tmpdir='/hps/nobackup/research/zi/leandro/snakemake_test/temp'" --mode 2 && exit 0 || exit 1

so it indeed works

[mbhall88]

Sorry, what is the issue here?

[leoisl]

Issue is that the current default_mem_mb implemented in the profile does not work. Adding default-resources as specified in the previous post to config.yaml can make it work, and we can also even further specify default disk_mb and tmpdir, which currently is not configurable in the profile. All these 3 options can be configured by the cookie-cutter and specified in config.yaml