Unable to change time limit with custom config

[PeterWang0502]

Description of the bug

I'm running the pipeline with singularity, but for some reason, my NFCORE_RNASEQ:PREPARE_GENOME:STAR_GENOMEGENERATE ({fasta file}) reports an error caused by Process exceeded running time limit (1h). Same thing happened to NFCORE_RNASEQ:RNASEQ:FASTQ_QC_TRIM_FILTER_SETSTRANDEDNESS:BBMAP_BBSPLIT ({sample name})

I've looked into the main.nf in nf-core/rnaseq/modules/nf-core/star/genomegenerate and nf-core/rnaseq/modules/nf-core/bbmap/bbsplit/, and both showed that the tasks were labeled with 'process_high', which should put the job running time to 10+ hours.

I've also added a custom config file to target these two tasks specifically, but it did not fix the problem. Any help is appreciated.
@marchoeppner @lbeltrame @robsyme @tucano

Command used and terminal output

Command: nextflow run nf-core/rnaseq -profile test,singularity -params-file nextflow_params.yaml -c RNAseq.config -resume

$ nextflow

terminal output:

ERROR ~ Error executing process > 'NFCORE_RNASEQ:PREPARE_GENOME:STAR_GENOMEGENERATE (Homo_sapiens.GRCh38.dna_sm.primary_assembly_gfp.fasta)'

Caused by:
  Process exceeded running time limit (1h)


Command executed:

  samtools faidx Homo_sapiens.GRCh38.dna_sm.primary_assembly_gfp.fasta
  NUM_BASES=`gawk '{sum = sum + $2}END{if ((log(sum)/log(2))/2 - 1 > 14) {printf "%.0f", 14} else {printf "%.0f", (log(sum)/log(2))/2 - 1}}' Homo_sapiens.GRCh38.dna_sm.primary_assembly_gfp.fasta.fai`

  mkdir star
  STAR \
      --runMode genomeGenerate \
      --genomeDir star/ \
      --genomeFastaFiles Homo_sapiens.GRCh38.dna_sm.primary_assembly_gfp.fasta \
      --sjdbGTFfile Homo_sapiens.GRCh38.dna_sm.primary_assembly_gfp.gtf \
      --runThreadN 4 \
      --genomeSAindexNbases $NUM_BASES \
      --limitGenomeGenerateRAM 16006127360 \

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_RNASEQ:PREPARE_GENOME:STAR_GENOMEGENERATE":
      star: $(STAR --version | sed -e "s/STAR_//g")
      samtools: $(echo $(samtools --version 2>&1) | sed 's/^.*samtools //; s/Using.*$//')
      gawk: $(echo $(gawk --version 2>&1) | sed 's/^.*GNU Awk //; s/, .*$//')
  END_VERSIONS

Command exit status:
  -

Command output:
        /opt/conda/bin/STAR-avx2 --runMode genomeGenerate --genomeDir star/ --genomeFastaFiles Homo_sapiens.GRCh38.dna_sm.primary_assembly_gfp.fasta --sjdbGTFfile Homo_sapiens.GRCh38.dna_sm.primary_asse$
        STAR version: 2.7.11b   compiled: 2024-11-25T09:14:51+0000 :/opt/conda/conda-bld/star_1732525954305/work/source
  Mar 27 15:55:00 ..... started STAR run
  Mar 27 15:55:00 ... starting to generate Genome files
  Mar 27 15:55:47 ..... processing annotations GTF
  Mar 27 15:56:22 ... starting to sort Suffix Array. This may take a long time...
  Mar 27 15:56:36 ... sorting Suffix Array chunks and saving them to disk...

Command wrapper:
        /opt/conda/bin/STAR-avx2 --runMode genomeGenerate --genomeDir star/ --genomeFastaFiles Homo_sapiens.GRCh38.dna_sm.primary_assembly_gfp.fasta --sjdbGTFfile Homo_sapiens.GRCh38.dna_sm.primary_asse$
        STAR version: 2.7.11b   compiled: 2024-11-25T09:14:51+0000 :/opt/conda/conda-bld/star_1732525954305/work/source
  Mar 27 15:55:00 ..... started STAR run
  Mar 27 15:55:00 ... starting to generate Genome files
  Mar 27 15:55:47 ..... processing annotations GTF
  Mar 27 15:56:22 ... starting to sort Suffix Array. This may take a long time...
  Mar 27 15:56:36 ... sorting Suffix Array chunks and saving them to disk...

Relevant files

params file nextflow_params.yaml:
input: "/home/bow012/nextflow_RNAseq_test.csv"
outdir: "/expanse/lustre/scratch/bow012/temp_project/RNAseq_PCSD13_nextflow_results"
gtf: "/home/bow012/nextflow_reference/Homo_sapiens.GRCh38.113.gtf.gz"
fasta: "/home/bow012/nextflow_reference/Homo_sapiens.GRCh38.dna_sm.primary_assembly.fa.gz"

custom config file RNAseq.config:
process {
withName: 'NFCORE_RNASEQ:PREPARE_GENOME:STAR_GENOMEGENERATE*' {
time = 24.h
}
withName: 'NFCORE_RNASEQ:RNASEQ:FASTQ_QC_TRIM_FILTER_SETSTRANDEDNESS:BBMAP_BBSPLIT*' {
time = 24.h
}
}

System information

Nextflow version: 24.10.5.5935
Slurm is used for job submission.
Submission format:
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=32
#SBATCH --time=48:00:00
#SBATCH --mem=200GB

[pinin4fjords]

Could I just check- are you using the `slurm' executor? https://www.nextflow.io/docs/latest/executor.html#slurm

[PeterWang0502]

Yes, I am using Slurm to submit jobs on my server.

[robsyme]

Hi Peter

Can you try with configuration:

process {
  withName: 'NFCORE_RNASEQ:PREPARE_GENOME:STAR_GENOMEGENERATE.*' {
    time = 24.h
  }
  withName: 'NFCORE_RNASEQ:RNASEQ:FASTQ_QC_TRIM_FILTER_SETSTRANDEDNESS:BBMAP_BBSPLIT.*' {
    time = 24.h
  }
}

Note that the selectors use regular expressions rather than glob patterns, so you'll need .* rather than just * to match "zero or more occurences of any token"

[pinin4fjords]

Yes, I am using Slurm to submit jobs on my server.

Thank you- I did understand you were using Slurm to submit jobs. However, sometimes people submit Nextflow jobs to Slurm clusters without setting:

process.executor = 'slurm'

in configuration, with the result that all jobs end up running local to the main Nextflow job, which would bypass the mechanisms Nextflow uses to apply time limits to requests (as well as playing havock with resource usage on that machine).

I just wanted to check you were applying that configuration.

[PeterWang0502]

Hi. Thanks for pointing that out. I added process.executor = 'slurm' to my nextflow.config configuration file, and I faced another problem shown below:

ERROR ~ Error executing process > 'NFCORE_RNASEQ:PREPARE_GENOME:STAR_GENOMEGENERATE (Homo_sapiens.GRCh38.dna_sm.primary_assembly_gfp.fasta)'

Caused by:
  java.io.IOException: Cannot run program "sbatch" (in directory "/expanse/lustre/scratch/bow012/temp_project/work/e1/2a0afebc04153e2f244c1835d20710"): error=2, No such file or directory


Command executed:

  sbatch .command.run

Command exit status:
  -

Command output:
  (empty)

Work dir:
  /expanse/lustre/scratch/bow012/temp_project/work/e1/2a0afebc04153e2f244c1835d20710

Container:
  /expanse/lustre/scratch/bow012/temp_project/work/singularity/quay.io-nf-core-htslib_samtools_star_gawk-311d422a50e6d829.img

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details

I've tried to run the command with bash .command.run at the working directory, and it can run normally. I'm wondering if it's because I sent the nextflow job to a node on my cluster, and the pipeline uses sbatch .command.run on that node, but sbatch is not available on that node.
Also, is there any way to change job's time limit when it ends up running local?

[pinin4fjords]

OK, sounds like your nodes are not submission hosts. I would talk to your cluster admins, explaining this use case- using a workflow manager to distribute jobs to worker nodes. They may recommend using the head node, or a specific node for this.

I would not recommend 'local' operation, you have a cluster at your disposal and these workflows are really optimised for distributed operation where you send jobs out to nodes. time should work fine with the local executor though (see docs https://www.nextflow.io/docs/latest/executor.html#local), you'll just have to get your config right. The default config should be setting this correctly, but Rob's suggestions may also help.