Skip to content

XeniumRanger import-segmentation failed on Tower #8587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
2 tasks done
an-altosian opened this issue Jun 2, 2025 · 0 comments
Open
2 tasks done

XeniumRanger import-segmentation failed on Tower #8587

an-altosian opened this issue Jun 2, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@an-altosian
Copy link
Contributor

an-altosian commented Jun 2, 2025 •
edited
Loading

Have you checked the docs?

Description of the bug

Dear @khersameesh24 and @maxulysse,

I am trying to run the proseg subworkflow from spatialxe on Nextflow Tower, which uses the nf-core xeniumranger module to convert proseg results to a xenium bundle. I have tested that this error was persistent.

    File "/xeniumranger-xenium3.1/mro/insitu/stages/xr_select_dataset/__init__.py", line 47, in main
      os.link(str(src_path), outs.dataset)
  OSError: [Errno 95] Operation not supported: '/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/XENIUM_RANGER_CS/XR_IMPORT_TRANSCRIPT_ASSIGNMENTS/fork0/join-u008637f7fd/files/cells.zarr.zip' -> '/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/XENIUM_RANGER_CS/SELECT_CELLS_DATASET/fork0/chnk0-u008637f958/files/cells.zarr.zip'

Command used and terminal output

$ nextflow run main.nf -profile docker --input samplesheet.csv --outdir results --segmentation proseg --coordinate_based

The exit status of the task that caused the workflow execution to fail was: 1

Error executing process > 'PROSEG:XENIUMRANGER_IMPORT_SEGMENTATION (XET)

Caused by:
  Essential container in task exited


Command executed:

  xeniumranger import-segmentation \
      --id="XET" \
      --xenium-bundle="output-XET" \
       \
       \
       \
       \
      --transcript-assignment="xr-transcript-metadata.csv" \
      --viz-polygons="xr-cell-polygons.geojson" \
      --units=microns \
      --localcores=94 \
      --localmem=760 \
  
  
  cat <<-END_VERSIONS > versions.yml
  "PROSEG:XENIUMRANGER_IMPORT_SEGMENTATION":
      xeniumranger: $(xeniumranger -V | sed -e "s/xeniumranger-/- /g")
  END_VERSIONS

Command exit status:
  1

Command output:
  2025-05-29 04:45:03 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS
  2025-05-29 04:45:03 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS.fork0.join
  2025-05-29 04:51:04 [runtime] (update)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS.fork0 join_running
  2025-05-29 04:57:05 [runtime] (update)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS.fork0 join_running
  2025-05-29 05:03:06 [runtime] (update)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS.fork0 join_running
  2025-05-29 05:04:21 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS
  2025-05-29 05:40:12 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_FINALIZE_RELABEL
  2025-05-29 05:40:13 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_FINALIZE_RELABEL.fork0.chnk0.main
  2025-05-29 05:40:46 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_FINALIZE_RELABEL
  2025-05-29 06:00:29 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
  2025-05-29 06:00:29 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS.fork0.split
  2025-05-29 06:00:29 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.XR_UNZIP_TRANSCRIPTS
  2025-05-29 06:00:29 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.XR_UNZIP_TRANSCRIPTS.fork0.chnk0.main
  2025-05-29 06:00:32 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
  2025-05-29 06:00:33 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS.fork0.join
  2025-05-29 06:06:15 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
  2025-05-29 06:06:16 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS
  2025-05-29 06:06:16 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS.fork0.chnk0.main
  2025-05-29 06:06:16 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET
  2025-05-29 06:06:16 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET.fork0.chnk0.main
  2025-05-29 06:06:19 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS
  2025-05-29 06:06:20 [runtime] (failed)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET
  
  [error] Pipestance failed. Error log at:
  XETG00509__0043420__Kidney_1__20241219__000427/XENIUM_RANGER_CS/SELECT_CELLS_DATASET/fork0/chnk0-u008637f958/_errors
  
  Log message:
  Traceback (most recent call last):
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 648, in _main
      stage.main()
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 607, in main
      self._run(lambda: self._module.main(args, outs))
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 564, in _run
      cmd()
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 607, in <lambda>
      self._run(lambda: self._module.main(args, outs))
    File "/xeniumranger-xenium3.1/mro/insitu/stages/xr_select_dataset/__init__.py", line 47, in main
      os.link(str(src_path), outs.dataset)
  OSError: [Errno 95] Operation not supported: '/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/XENIUM_RANGER_CS/XR_IMPORT_TRANSCRIPT_ASSIGNMENTS/fork0/join-u008637f7fd/files/cells.zarr.zip' -> '/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/XENIUM_RANGER_CS/SELECT_CELLS_DATASET/fork0/chnk0-u008637f958/files/cells.zarr.zip'
  
  
  Waiting 6 seconds for UI to do final refresh.
  Pipestance failed. Use --noexit option to keep UI running after failure.
  
  2025-05-29 06:06:26 Shutting down.
  Saving pipestance info to "/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/XETG00509__0043420__Kidney_1__20241219__000427.mri.tgz"
  For assistance, upload this file to 10x Genomics by running:
  
  xeniumranger upload <your_email> "/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/XET.mri.tgz"

Command error:
  (more omitted..)
  2025-05-29 05:03:06 [runtime] (update)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS.fork0 join_running
  2025-05-29 05:04:21 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_SUBSAMPLE_TRANSCRIPTS
  2025-05-29 05:40:12 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_FINALIZE_RELABEL
  2025-05-29 05:40:13 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_FINALIZE_RELABEL.fork0.chnk0.main
  2025-05-29 05:40:46 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_RELABEL.XR_FINALIZE_RELABEL
  2025-05-29 06:00:29 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
  2025-05-29 06:00:29 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS.fork0.split
  2025-05-29 06:00:29 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.XR_UNZIP_TRANSCRIPTS
  2025-05-29 06:00:29 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.INSITU_COUNTER_CS.INSITU_COUNTER_CORE.XR_UNZIP_TRANSCRIPTS.fork0.chnk0.main
  Could not read cgroup stats: open /sys/fs/cgroup/memory/ecs/12cd279ff2f54605ab1483f664a883cb/46b378d1a297945e96ffed9d65d8bd76180aa4b3cb5d565f0381bc6aa6bcc577/memory.stat: no such file or directory
  tar: Removing leading `/' from member names
  2025-05-29 06:00:32 [runtime] (split_complete)  ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
  2025-05-29 06:00:33 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS.fork0.join
  2025-05-29 06:06:15 [runtime] (join_complete)   ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.XR_IMPORT_TRANSCRIPT_ASSIGNMENTS
  2025-05-29 06:06:16 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS
  2025-05-29 06:06:16 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS.fork0.chnk0.main
  2025-05-29 06:06:16 [runtime] (ready)           ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET
  2025-05-29 06:06:16 [runtime] (run:local)       ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET.fork0.chnk0.main
  2025-05-29 06:06:19 [runtime] (chunks_complete) ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.MERGE_METRICS
  2025-05-29 06:06:20 [runtime] (failed)          ID.IMPORT_SEGMENTATION.XENIUM_RANGER_CS.SELECT_CELLS_DATASET
  [error] Pipestance failed. Error log at:
  XETG00509__0043420__Kidney_1__20241219__000427/XENIUM_RANGER_CS/SELECT_CELLS_DATASET/fork0/chnk0-u008637f958/_errors
  Log message:
  Traceback (most recent call last):
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 648, in _main
      stage.main()
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 607, in main
      self._run(lambda: self._module.main(args, outs))
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 564, in _run
      cmd()
    File "/xeniumranger-xenium3.1/external/martian/adapters/python/martian_shell.py", line 607, in <lambda>
      self._run(lambda: self._module.main(args, outs))
    File "/xeniumranger-xenium3.1/mro/insitu/stages/xr_select_dataset/__init__.py", line 47, in main
      os.link(str(src_path), outs.dataset)
  OSError: [Errno 95] Operation not supported: '/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/fork0/join-u008637f7fd/files/cells.zarr.zip' -> '/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET/SELECT_CELLS_DATASET/fork0/chnk0-u008637f958/files/cells.zarr.zip'
  Waiting 6 seconds for UI to do final refresh.
  Pipestance failed. Use --noexit option to keep UI running after failure.
  2025-05-29 06:06:26 Shutting down.
  Saving pipestance info to "/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XET.mri.tgz"
  For assistance, upload this file to 10x Genomics by running:
  xeniumranger upload <your_email> "/fusion/s3/scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336/XETG.mri.tgz"
  Fusion Info:
      ami-id: ami-0161203d1ed74d5f9
      instance-id: i-05713dd0e069ef5b6
      instance-type: r6id.32xlarge
      fusion_version: 2.4.13-cd14732
      clone_namespace: false
      kernel_version: 4.14
      disk_cache_size: 49Gb
      max_open_files: 1048576

Work dir:
  s3://scratch/1B87d1PKcR8hmK/ee/a3c4ea95749be8ce0aa9463bb92336

Container:
  wave.seqera.io/wt/87676aa08cd1/nf-core/xeniumranger:3.1.1

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

Relevant files

I could run the workflow locally, but it failed on Tower with fusion turned on. The error was a Python "Operation not supported" error when XeniumRange was trying to use os.link to create a hard link. The complete error message is attached below. As I remember we encountered similar problem in nf-core scrnaseq with cellranger, here I also tag the amazing nf-core scrnaseq developers in case they have any idea. Thanks. @grst @fmalmeida @apeltzer.

System information

No response
@an-altosian an-altosian added the bug Something isn't working label Jun 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@an-altosian