Institutional Profile for IRIS

[buehlere]

• base config file for running on IRIS.
  • consider no re-queue, multiple partitions, preemptive
  • considerations for multiple caches: https://www.nextflow.io/docs/latest/reference/env-vars.html
• do we want to contribute to nf-core/configs or do we want msk/configs?

[buehlere]

added config fork: https://github.com/mskcc-omics-workflows/configs

[buehlere]

We want to check with HPC if symlink-ing to scratch/ works for nextflow's work directory. If so, we can use it as the work directory for nextflow pipelines.

If not, we want to use data1/ as the work directory so large files like BAMS don't fill up scratch/. This also might be an issue due to data1/'s snapshotting? We should confirm this with HPC.

[buehlere]

We want to check with HPC if symlink-ing to scratch/ works for nextflow's work directory. If so, we can use it as the work directory for nextflow pipelines.

If not, we want to use data1/ as the work directory so large files like BAMS don't fill up scratch/. This also might be an issue due to data1/'s snapshotting? We should confirm this with HPC.

symlink-ing to scratch/ works for nextflow's work directory. We can use it as the work directory for nextflow pipelines.

[buehlere]

We were expecting to use /scratch directory on core005 as the working directory for voyager nextflow pipelines.

But /scratch doesn't get cleaned up on IRIS. Scratch only has 1 TB of storage. Where do we want job / work stores?

regular nextflow cache case:

• keep all nextflow processes intermediate files in work directory and deleted upon completion and written to output directory.

special nextflow cache case:

• nextflow store directory option: previous nextflow processes replaced with dummy files -> final nextflow process --hard-linked-> final output bam. Store directory will never be in /scratch. Work directory doesn't matter as much.

We can't hard link between core005 and core006.

Because of special caching case and group usage, we need to follow up with HPC to see about /scratch with a lot more memory.

[buehlere]

JobArray not treated as one submission. We're still stuck with 100 job limit: https://github.com/mskcc-omics-workflows/configs/blob/896471156353f5d809bf61c168c6a671cc36d2d8/conf/iris.config#L37.

Following up with HPC.

[buehlere]

JobArray not treated as one submission. We're still stuck with 100 job limit: https://github.com/mskcc-omics-workflows/configs/blob/896471156353f5d809bf61c168c6a671cc36d2d8/conf/iris.config#L37.

Following up with HPC.

This is not true. A JobArray is one submission.

Custom JobArray submit from @johnoooh:

• Graceful fallback when array is used on unsupported executors nextflow-io/nextflow#5924
• Enhance TaskArrayCollector to support configurable timeout-based array submission nextflow-io/nextflow#6465

[jpintar]

The IRIS config should be updated to follow strict syntax (cf. nf-core/configs #1020).