Persistent MPI_Abort During Volume Rendering in Batch Mode

[yuheng-guo]

Hi all,

I'm running VisIt on a supercomputer using SLURM scheduler and encoutering persistent crashes when trying to render volume plots in batch mode.

In my script, I launch VisIt using:

export VISITDIR=/PATH_OF_VISIT_3.3.2/
export PATH=$VISITDIR/bin:$PATH
$VISITDIR/bin/visit -nn 1 -np 16 -cli -nowin -s $SOME_PATH -disableghostsfort-intersections

During the final rendering steps, VisIt crashes with:

VisIt: Warning - Although VisIt typically saves out files sequentially, some numbers are being skipped when saving out this file to avoid overwriting previous saves.
VisIt: Message - Rendering window 1...
application called MPI_Abort(comm=0x84000002, 18) - process 2
VisIt: Error - The compute engine running on a256.anvil.rcac.purdue.edu has exited abnormally.
VisIt: Error - Scalable Render Request Failed (VisItException)

It then tries to restart 2 to 3 times before exit.

Some details that I think may help:

1. When running without specifying -np, the job doesn't crash but silently skips the volume rendering step, producing an imcomplete image without error in log.
2. The data I'm plotting is large and contains multiple refinement levels.
3. Disabling ghost zones didn't seem to have an impact
4. The volume xml files may be taxing:

<Field name="resampleTarget" type="int">100000</Field>
<Field name="samplesPerRay" type="int">2000</Field>
<Field name="rendererType" type="string">RayCasting</Field>

5. I have previously run these images on another system without issues. That system runs with VisIt 3.1.2.

Thanks for any advice!

[markcmiller86]

Sorry you're running into trouble. Is there a delay between the VisIt: Message - Rendering window 1... and the abort? Or, does it happen essentially immediately after it starts?

Any chance you can run with -debug 5 on the command-line and then paste the .vlog files (from both the client machine and the server machine) here?

export VISITDIR=/PATH_OF_VISIT_3.3.2/
export PATH=$VISITDIR/bin:$PATH
$VISITDIR/bin/visit -nn 1 -np 16 -cli -nowin -s $SOME_PATH -disableghostsfort-intersections -debug 5

I wonder if you could be running out of memory. If its easy, can you maybe double or quadruple the node count but maybe keep processor count at 16?

[yuheng-guo]

I think I see the issue here. There seems to be a parsing error in my input list. I'm resubmitting my jobs now.

[yuheng-guo]

Here is the link to the full diagnostics:
https://drive.google.com/file/d/1doI7rwf0NrJyZTeC7OJU3DtqfYvZm_jq/view?usp=sharing

[markcmiller86]

Ok, thanks for those! 💪 I took a quick look and I see engine out of memory, try more processors in several of the engine log files. So, it is definitately running out of memory. I wonder....when you were doing memory study, was that on the client or the server...and, in particular, the server nodes where the engine was running? Its easy to get confused about that. So, thats why I asked.

Also, there are 3072 domains in this dataset. Thats a lot of domains to process on just 16 ranks. To get a feel for the correct number of ranks and nodes, I would use VisIt's Subset Controls (SIL) window and set the active domains to some small number...say 16 on one node at first and do a plot of just those domains on 16 ranks. Then double the number of domains that are active using the Subset controls window. Keep doubling until it fails. That is a good idea how many domains you can process on a single node. Then, just extrapolate that to 3072 domains to figure how many nodes you need. You can also back off on number of ranks per node too to give each rank more memory.

I looked and here is what I was seeing in the way of memory usage per rank...

0 10071330816
1 8572899328
2 7875002368
3 7872102400
4 7803576320
5 7583158272
6 5133049856
7 3523489792
8 3148677120
9 3169636352
10 3211161600
11 3252781056
12 3233427456
13 3246436352
14 3190788096
15 3605782528

Which totals ~78 GB. Now if a281.anvil.rcac.purdue.edu is the A subcluster of Anvil, each node has 256G of memory and 128 cores. So, I am a little perplexed by this. I am also a bit perplexed why the first half of ranks have >5G whereas the second half have <3Gigs. But, there is no reason to assume volume rendering work is about equal across domains either.

[markcmiller86]

I also think you may have better luck with a newer version of VisIt. I recall other users encountering similar behavior and some fixes for this comming in for 3.4...I think

[yuheng-guo]

Thank you these are extremely helpful. I will try with an newer VisIt and optimize my nodes and cores request. I'm pretty sure that the job runs on a server node on Anvil, but I'm not entirely sure about the memory issue.