parallel jobs on aix with poe question.

[markcmiller86]

Date: Mon, 17 Aug 2009 14:28:34 -0700
From: semeraro at ncsa.illinois.edu (Dave Semeraro)
This post was imported from the [email protected] email archive

---

Hello,

I have built visit on an AIX system and am now trying to get the paralell version to work. I have
set up a host profile for the target platform that uses the serial engine and things seem to work
ok. I can connect to the remote system, read data and draw stuff. However, I get errors when I try
to run using poe and a parallel host profile. It appears that the problem is the mpi library is not
being found. The error messages I get when I try to open a remote file are:

./visit
Running: gui -dv
Running: viewer -host 127.0.0.1 -geometry 1217x1173+27+0 -borders 24,5,5,5 -shift 27,24 -preshift
-22,0 -defer -port 5600 -dv
Running: mdserver -host 127.0.0.1 -port 5601 -dv
semeraro at bp.ncsa.uiuc.edu's password:
Running: vcl -host localhost -dir /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src -noloopback
-timeout 480 -dir /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src -sshtunneling -port 29874 -dv
-launchengine localhost -engineargs ;-par
Running: mdserver -host localhost -noloopback -timeout 480 -dir
/u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src -port 17639 -sshtunneling -dv -launchengine
localhost -engineargs ;-par
Running: engine_par -host bp-login2 -noloopback -forcestatic -timeout 480 -dir
/u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src -port 29124 -sshtunneling -dv -procs 2
exec(): 0509-036 Cannot load program
/u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src/exe/engine_par because of the following errors:
rtld: 0712-001 Symbol MPI_Recv was referenced
from module /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src/lib/libengine_par.so(), but a
runtime definition
of the symbol was not found.
rtld: 0712-001 Symbol MPI_Send was referenced
from module /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src/lib/libengine_par.so(), but a
runtime definition
of the symbol was not found.
rtld: 0712-001 Symbol MPI_Barrier was referenced
from module /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src/lib/libengine_par.so(), but a
runtime definition
of the symbol was not found.

I compiled and tested a small hello world MPI program without problems on the AIX system so I can
get MPI programs to work. Does anyone have experience getting MPI applications to work with poe in
general and poe and loadleveler in particular? I also tried to follow the directions on the wiki on
getting parallel applications to work but ran into other problems there. I dont think some of the
arguments to visit on those pages apply any more.

thanks for the help.

Dave

---

[markcmiller86]

Date: Mon, 17 Aug 2009 14:47:41 -0700
From: cyrush at llnl.gov (Cyrus D. Harrison)
This post was imported from the [email protected] email archive

---

Hi Dave,
How did you specify the include directories  & link libs when
configuring VisIt?
Did you see the expected parallel flags being passed to the compiler (I
am assuming xlc) when the '_par' libs were created?
Try 'touching' a source file for the Silo (or other db plugin) and
rebuilding and double check that the mpi libs made their way into the
linking process.

We currently use poe for launching parallel interactive jobs on our AIX
clusters @ llnl and moab for batch parallel jobs.

-Cyrus

Dave Semeraro wrote:
> Hello,
>
> I have built visit on an AIX system and am now trying to get the paralell version to work. I have
set up a host profile for the target platform that uses the serial engine and things seem to work
ok. I can connect to the remote system, read data and draw stuff. However, I get errors when I try
to run using poe and a parallel host profile. It appears that the problem is the mpi library is not
being found. The error messages I get when I try to open a remote file are:
>
>  ./visit
> Running: gui -dv
> Running: viewer -host 127.0.0.1 -geometry 1217x1173+27+0 -borders 24,5,5,5 -shift 27,24 -preshift
-22,0 -defer -port 5600 -dv
> Running: mdserver -host 127.0.0.1 -port 5601 -dv
> semeraro at bp.ncsa.uiuc.edu's password:
> Running: vcl -host localhost -dir /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src -noloopback
-timeout 480 -dir /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src -sshtunneling -port 29874 -dv
-launchengine localhost -engineargs ;-par
> Running: mdserver -host localhost -noloopback -timeout 480 -dir
/u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src -port 17639 -sshtunneling -dv -launchengine
localhost -engineargs ;-par
> Running: engine_par -host bp-login2 -noloopback -forcestatic -timeout 480 -dir
/u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src -port 29124 -sshtunneling -dv -procs 2
> exec(): 0509-036 Cannot load program
/u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src/exe/engine_par because of the following errors:
> rtld: 0712-001 Symbol MPI_Recv was referenced
>       from module /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src/lib/libengine_par.so(), but
a runtime definition
>       of the symbol was not found.
> rtld: 0712-001 Symbol MPI_Send was referenced
>       from module /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src/lib/libengine_par.so(), but
a runtime definition
>       of the symbol was not found.
> rtld: 0712-001 Symbol MPI_Barrier was referenced
>       from module /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src/lib/libengine_par.so(), but
a runtime definition
>       of the symbol was not found.
>
> I compiled and tested a small hello world MPI program without problems on the AIX system so I can
get MPI programs to work. Does anyone have experience getting MPI applications to work with poe in
general and poe and loadleveler in particular? I also tried to follow the directions on the wiki on
getting parallel applications to work but ran into other problems there. I dont think some of the
arguments to visit on those pages apply any more.
>
> thanks for the help.
>
> Dave
> --

>
>
>
>

---

[markcmiller86]

Date: Mon, 17 Aug 2009 15:02:52 -0700
From: semeraro at ncsa.illinois.edu (semeraro at ncsa.illinois.edu)
This post was imported from the [email protected] email archive

---

Cyrus,

I am in the process of doing what you suggest. I dont see any mpi libs being passed to the
engine_par process. It is funny that ldd on engine_par shows no link to mpi libs. There are no
unresolved MPI refs in the make proces either. You would think that would show up there.
I thought maybe it was an LD_LIBRARY_PATH problem but maybe not.

Dave
----- Original Message -----
From: 'Cyrus D. Harrison' <cyrush at llnl.gov>
To: 'VisIt software users community' <visit-users at email.ornl.gov>
Sent: Monday, August 17, 2009 4:47:41 PM GMT -06:00 US/Canada Central
Subject: Re: [visit-users] parallel jobs on AIX with poe question.

Hi Dave,
How did you specify the include directories  & link libs when
configuring VisIt?
Did you see the expected parallel flags being passed to the compiler (I
am assuming xlc) when the '_par' libs were created?
Try 'touching' a source file for the Silo (or other db plugin) and
rebuilding and double check that the mpi libs made their way into the
linking process.

We currently use poe for launching parallel interactive jobs on our AIX
clusters @ llnl and moab for batch parallel jobs.

-Cyrus

Dave Semeraro wrote:
> Hello,
>
> I have built visit on an AIX system and am now trying to get the paralell version to work. I have
set up a host profile for the target platform that uses the serial engine and things seem to work
ok. I can connect to the remote system, read data and draw stuff. However, I get errors when I try
to run using poe and a parallel host profile. It appears that the problem is the mpi library is not
being found. The error messages I get when I try to open a remote file are:
>
>  ./visit
> Running: gui -dv
> Running: viewer -host 127.0.0.1 -geometry 1217x1173+27+0 -borders 24,5,5,5 -shift 27,24 -preshift
-22,0 -defer -port 5600 -dv
> Running: mdserver -host 127.0.0.1 -port 5601 -dv
> semeraro at bp.ncsa.uiuc.edu's password:
> Running: vcl -host localhost -dir /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src -noloopback
-timeout 480 -dir /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src -sshtunneling -port 29874 -dv
-launchengine localhost -engineargs ;-par
> Running: mdserver -host localhost -noloopback -timeout 480 -dir
/u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src -port 17639 -sshtunneling -dv -launchengine
localhost -engineargs ;-par
> Running: engine_par -host bp-login2 -noloopback -forcestatic -timeout 480 -dir
/u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src -port 29124 -sshtunneling -dv -procs 2
> exec(): 0509-036 Cannot load program
/u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src/exe/engine_par because of the following errors:
> rtld: 0712-001 Symbol MPI_Recv was referenced
>       from module /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src/lib/libengine_par.so(), but
a runtime definition
>       of the symbol was not found.
> rtld: 0712-001 Symbol MPI_Send was referenced
>       from module /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src/lib/libengine_par.so(), but
a runtime definition
>       of the symbol was not found.
> rtld: 0712-001 Symbol MPI_Barrier was referenced
>       from module /u/home/ac/semeraro/VisitBuild/visit/visit1.11.1/src/lib/libengine_par.so(), but
a runtime definition
>       of the symbol was not found.
>
> I compiled and tested a small hello world MPI program without problems on the AIX system so I can
get MPI programs to work. Does anyone have experience getting MPI applications to work with poe in
general and poe and loadleveler in particular? I also tried to follow the directions on the wiki on
getting parallel applications to work but ran into other problems there. I dont think some of the
arguments to visit on those pages apply any more.
>
> thanks for the help.
>
> Dave
> --

>
>
>
>

---

---

[markcmiller86]

Date: Wed, 21 Oct 2009 12:38:35 -0700
From: semeraro at ncsa.illinois.edu (semeraro at ncsa.illinois.edu)
This post was imported from the [email protected] email archive

---

Hello there,

I was wondering if anyone runs visit in batch mode on a parallel platform. If so how do you do it? I
need to figure out how to run visit in batch mode on an AIX platform that uses poe and loadleveler.
POE is sort of like mpirun and loadleveler is sort of like pbs. To run a parallel hello world one
submits the following load leveler command script using the llsubmit command.

#!/usr/bin/ksh
#@ job_type = parallel
#@ environment = COPY_ALL
#@ output = hellopoe.out
#@ error = hellopoe.err
#@ class = medium
#@ tasks_per_node = 1
#@ node = 4
#@ network.MPI_LAPI = sn_all,not_shared,US
#@ wall_clock_limit = 0:55:00
#@ queue
/usr/bin/poe /u/home/ac/semeraro/Projects/Poetests/hello

This starts the hello mpi program on 4 nodes. What I would like to do is something like:

visit -l poe -la -llfile visit.ll -nowin -cli -s script.py

I am guessing at some of these options based on the hostprofile I use to start parallel jobs on the
IBM platform I am using. The host profile  for this cluster works just fine by the way.

So if anyone does something similar to what I am trying to do, even if it uses mpirun and pbs or
some other combination, please post your pbs script or whatever you use to give me a clue. There may
be several ways to do this... I dont know.

Thanks,

Dave Semeraro

---

[markcmiller86]

Date: Wed, 21 Oct 2009 12:43:54 -0700
From: hankchilds at gmail.com (Hank Childs)
This post was imported from the [email protected] email archive

---

Hi Dave,
(I'm 80% sure that I'm answering this question right)

If you can do it with a VisIt host profile, then you can do it with the
command line as well.

One way is to launch with command line arguments.   Another way is to launch
from within your Python script (that's what I do).

Method #1:
visit -cli -nowin -l mpirun -np 2 ...

Method #2:
>>> OpenComputeEngine('localhost', ('-l', 'mpirun', '-np', '2', ...))

So the gameplan would be to see what you have turned on in the host profile
and then translate those commands into the '-l''s and '-np''s.  So the rest
of your question is how to get the flags.  You can get that by running
'visit -fullhelp'.

Parallel launch options

---

Notes: All of these options are ordinarily obtained from host
profiles.
However, the command line options override anything in the
profiles.

When parallel arguments are added but the engine is not the
component being launched, -launchengine is implied.  Explicitly
add -launchengine to launch a remote parallel engine.

---

-setupenv            Use the VisIt script to set up the environment
for the engine on the compute nodes.
-par                 Run the parallel version. This option is
implied
by any of the other parallel options listed
below.
-l    <method>       Launch in parallel using the given method.
-pl   <method>       Launch only the engine in parallel as
specified.
-la   <args>         Additional arguments for the parallel launcher.
-sla  <args>         Additional arguments for the parallel
sublauncher.
-np   <# procs>      The number of processors to use.
-nn   <# nodes>      The number of nodes to allocate.
-p    <part>         Partition to run in.
-n    <name>         The parallel job name.
-b    <bank>         Bank from which to draw resources.
-t    <time>         Maximum job run time.
-machinefile <file>  Machine file.
-expedite            Makes DPCS give priority scheduling.

-icet                In scalable rendering mode, use the IceT
parallel
image compositor.

Best,
Hank

On Wed, Oct 21, 2009 at 12:38 PM, <semeraro at ncsa.illinois.edu> wrote:

>
> Hello there,
>
> I was wondering if anyone runs visit in batch mode on a parallel platform.
> If so how do you do it? I need to figure out how to run visit in batch mode
> on an AIX platform that uses poe and loadleveler. POE is sort of like mpirun
> and loadleveler is sort of like pbs. To run a parallel hello world one
> submits the following load leveler command script using the llsubmit
> command.
>
> #!/usr/bin/ksh
> #@ job_type = parallel
> #@ environment = COPY_ALL
> #@ output = hellopoe.out
> #@ error = hellopoe.err
> #@ class = medium
> #@ tasks_per_node = 1
> #@ node = 4
> #@ network.MPI_LAPI = sn_all,not_shared,US
> #@ wall_clock_limit = 0:55:00
> #@ queue
> /usr/bin/poe /u/home/ac/semeraro/Projects/Poetests/hello
>
> This starts the hello mpi program on 4 nodes. What I would like to do is
> something like:
>
> visit -l poe -la -llfile visit.ll -nowin -cli -s script.py
>
> I am guessing at some of these options based on the hostprofile I use to
> start parallel jobs on the IBM platform I am using. The host profile  for
> this cluster works just fine by the way.
>
> So if anyone does something similar to what I am trying to do, even if it
> uses mpirun and pbs or some other combination, please post your pbs script
> or whatever you use to give me a clue. There may be several ways to do
> this... I dont know.
>
> Thanks,
>
> Dave Semeraro
>
> --
> List subscription information:
> https://email.ornl.gov/mailman/listinfo/visit-users

---

[markcmiller86]

Date: Thu, 22 Oct 2009 09:09:02 -0700
From: whitlock2 at llnl.gov (Brad Whitlock)
This post was imported from the [email protected] email archive

---

It sounds to me like the issue here is that the CLI and viewer must
also run in the batch job, which is not really supported well.

On the trunk, I added an option to the engine called
'-reverse_launch' that lets you directly launch the compute engine
and then it will launch the viewer and cli instead of the more common
case where the cli runs the viewer which then runs the engine. The
newer method is geared to more easily support the case where you want
to do everything as part of your batch job. Of course, if the engine
dies so does the viewer in that case.

Brad

>Hi Dave,
>
>(I'm 80% sure that I'm answering this question right)
>
>If you can do it with a VisIt host profile, then you can do it with
>the command line as well.
>
>One way is to launch with command line arguments.   Another way is
>to launch from within your Python script (that's what I do).
>
>Method #1:
>visit -cli -nowin -l mpirun -np 2 ...
>
>Method #2:
>  >>> OpenComputeEngine('localhost', ('-l', 'mpirun', '-np', '2', ...))
>
>So the gameplan would be to see what you have turned on in the host
>profile and then translate those commands into the '-l''s and
>'-np''s.  So the rest of your question is how to get the flags.  You
>can get that by running 'visit -fullhelp'.
>
>     Parallel launch options
>
> ---------------------------------------------------------------------------
>         Notes: All of these options are ordinarily obtained from
>host profiles.
>         However, the command line options override anything in the profiles.
>
>         When parallel arguments are added but the engine is not the
>         component being launched, -launchengine is implied.  Explicitly
>         add -launchengine to launch a remote parallel engine.
>
> ---------------------------------------------------------------------------
>         -setupenv            Use the VisIt script to set up the environment
>                              for the engine on the compute nodes.
>         -par                 Run the parallel version. This option is implied
>                              by any of the other parallel options
>listed below.
>         -l    <method>       Launch in parallel using the given method.
>         -pl   <method>       Launch only the engine in parallel as specified.
>         -la   <args>         Additional arguments for the parallel launcher.
>         -sla  <args>         Additional arguments for the parallel
>sublauncher.
>         -np   <# procs>      The number of processors to use.
>         -nn   <# nodes>      The number of nodes to allocate.
>         -p    <part>         Partition to run in.
>         -n    <name>         The parallel job name.
>         -b    <bank>         Bank from which to draw resources.
>         -t    <time>         Maximum job run time.
>         -machinefile <file>  Machine file.
>         -expedite            Makes DPCS give priority scheduling.
>
>         -icet                In scalable rendering mode, use the IceT parallel
>                              image compositor.
>
>Best,
>Hank
>
>On Wed, Oct 21, 2009 at 12:38 PM,
><<mailto:semeraro at ncsa.illinois.edu>semeraro at ncsa.illinois.edu>
>wrote:
>
>
>Hello there,
>
>I was wondering if anyone runs visit in batch mode on a parallel
>platform. If so how do you do it? I need to figure out how to run
>visit in batch mode on an AIX platform that uses poe and
>loadleveler. POE is sort of like mpirun and loadleveler is sort of
>like pbs. To run a parallel hello world one submits the following
>load leveler command script using the llsubmit command.
>
>#!/usr/bin/ksh
>#@ job_type = parallel
>#@ environment = COPY_ALL
>#@ output = hellopoe.out
>#@ error = hellopoe.err
>#@ class = medium
>#@ tasks_per_node = 1
>#@ node = 4
>#@ network.MPI_LAPI = sn_all,not_shared,US
>#@ wall_clock_limit = 0:55:00
>#@ queue
>/usr/bin/poe /u/home/ac/semeraro/Projects/Poetests/hello
>
>This starts the hello mpi program on 4 nodes. What I would like to
>do is something like:
>
>visit -l poe -la -llfile visit.ll -nowin -cli -s script.py
>
>I am guessing at some of these options based on the hostprofile I
>use to start parallel jobs on the IBM platform I am using. The host
>profile  for this cluster works just fine by the way.
>
>So if anyone does something similar to what I am trying to do, even
>if it uses mpirun and pbs or some other combination, please post
>your pbs script or whatever you use to give me a clue. There may be
>several ways to do this... I dont know.
>
>Thanks,
>
>Dave Semeraro
>
>--
>List subscription information:
><https://*email.ornl.gov/mailman/listinfo/visit-
users>https://*email.ornl.gov/mailman/listinfo/visit-users
>Searchable list archives:
><https://*email.ornl.gov/pipermail/visit-users>https://*email.ornl.gov/pipermail/visit-users
>VisIt Users Wiki: <http://*visitusers.org/>http://*visitusers.org/
>Frequently Asked Questions for VisIt:
><http://*visit.llnl.gov/FAQ.html>http://*visit.llnl.gov/FAQ.html
>
>
>
>--
>List subscription information:
>https://*email.ornl.gov/mailman/listinfo/visit-users

---

======================================================================
Brad Whitlock                   Lawrence Livermore National Laboratory
whitlock2 at llnl.gov
XXXXXXXXXXXXX
======================================================================

---

[markcmiller86]

Date: Fri, 23 Oct 2009 04:50:46 -0700
From: hankchilds at gmail.com (Hank Childs)
This post was imported from the [email protected] email archive

---

Hi,

> It sounds to me like the issue here is that the CLI and viewer must also
run in the batch job, which is not really supported well.

At TACC when running on the login node, they do the following:
- run qsub to get the nodes
- when the qsub job gets scheduled, they launch VisIt (meaning the cli and
viewer)
- to launch the engine, the host profile uses 'ibrun' to launch in parallel
on all the nodes
- the result is that the viewer and cli run on the first node and the engine
runs on all the nodes (including the first node, so there is a memory pinch
there)

(Also, for those running the client somewhere besides the login node, there
are separate profiles for a combined qsub/ibrun.)

Here's an example of launching the cli & viewer on a batch node.  It seems
complex, but a lot of that is just the complexity of managing a batch
system:

# To get the nodes
qsub -l h_rt=00:30:00 -V -cwd -pe 16way 16 -q vis visit_launcher_script

# The script
% cat visit_launcher_script
visit -cli -nowin -s visscript.10.py
(There are also a lot of 'module loads' in there)

# The VisIt Python script
launchArguments = ('-np', '8', '-nn', '1', '-p', 'vis', '-la', '-V -cwd',
'-l', 'ibrun', '-t', '60')
batchHost = 'localhost'
OpenComputeEngine('%s'%batchHost,launchArguments)
Source('my_script_that_does_work_besides_launching_engines.py')

Best,
Hank

On Thu, Oct 22, 2009 at 9:09 AM, Brad Whitlock <whitlock2 at llnl.gov> wrote:

>
> On the trunk, I added an option to the engine called '-reverse_launch' that
> lets you directly launch the compute engine and then it will launch the
> viewer and cli instead of the more common case where the cli runs the viewer
> which then runs the engine. The newer method is geared to more easily
> support the case where you want to do everything as part of your batch job.
> Of course, if the engine dies so does the viewer in that case.
>
> Brad
>
>
> Hi Dave,
>
>
> (I'm 80% sure that I'm answering this question right)
>
>
> If you can do it with a VisIt host profile, then you can do it with the
> command line as well.
>
>
> One way is to launch with command line arguments.   Another way is to
> launch from within your Python script (that's what I do).
>
>
> Method #1:
>
> visit -cli -nowin -l mpirun -np 2 ...
>
>
> Method #2:
>
> >>> OpenComputeEngine('localhost', ('-l', 'mpirun', '-np', '2', ...))
>
>
> So the gameplan would be to see what you have turned on in the host profile
> and then translate those commands into the '-l''s and '-np''s.  So the rest
> of your question is how to get the flags.  You can get that by running
> 'visit -fullhelp'.
>
>
>     Parallel launch options
>
>     ---------------------------------------------------------------
> ------------
>
>         Notes: All of these options are ordinarily obtained from host
> profiles.
>
>         However, the command line options override anything in the
> profiles.
>
>
>         When parallel arguments are added but the engine is not the
>
>         component being launched, -launchengine is implied.  Explicitly
>
>         add -launchengine to launch a remote parallel engine.
>
>     ---------------------------------------------------------------
> ------------
>
>         -setupenv            Use the VisIt script to set up the environment
>
>                              for the engine on the compute nodes.
>
>         -par                 Run the parallel version. This option is
> implied
>
>                              by any of the other parallel options listed
> below.
>
>         -l    <method>       Launch in parallel using the given method.
>
>         -pl   <method>       Launch only the engine in parallel as
> specified.
>
>         -la   <args>         Additional arguments for the parallel
> launcher.
>
>         -sla  <args>         Additional arguments for the parallel
> sublauncher.
>
>         -np   <# procs>      The number of processors to use.
>
>         -nn   <# nodes>      The number of nodes to allocate.
>
>         -p    <part>         Partition to run in.
>
>         -n    <name>         The parallel job name.
>
>         -b    <bank>         Bank from which to draw resources.
>
>         -t    <time>         Maximum job run time.
>
>         -machinefile <file>  Machine file.
>
>         -expedite            Makes DPCS give priority scheduling.
>
>
>         -icet                In scalable rendering mode, use the IceT
> parallel
>
>                              image compositor.
>
>
> Best,
>
> Hank
>
>
> On Wed, Oct 21, 2009 at 12:38 PM, <semeraro at ncsa.illinois.edu> wrote:
>
>
> Hello there,
>
> I was wondering if anyone runs visit in batch mode on a parallel platform.
> If so how do you do it? I need to figure out how to run visit in batch mode
> on an AIX platform that uses poe and loadleveler. POE is sort of like mpirun
> and loadleveler is sort of like pbs. To run a parallel hello world one
> submits the following load leveler command script using the llsubmit
> command.
>
> #!/usr/bin/ksh
> #@ job_type = parallel
> #@ environment = COPY_ALL
> #@ output = hellopoe.out
> #@ error = hellopoe.err
> #@ class = medium
> #@ tasks_per_node = 1
> #@ node = 4
> #@ network.MPI_LAPI = sn_all,not_shared,US
> #@ wall_clock_limit = 0:55:00
> #@ queue
> /usr/bin/poe /u/home/ac/semeraro/Projects/Poetests/hello
>
> This starts the hello mpi program on 4 nodes. What I would like to do is
> something like:
>
> visit -l poe -la -llfile visit.ll -nowin -cli -s script.py
>
>
> I am guessing at some of these options based on the hostprofile I use to
> start parallel jobs on the IBM platform I am using. The host profile  for
> this cluster works just fine by the way.
>
> So if anyone does something similar to what I am trying to do, even if it
> uses mpirun and pbs or some other combination, please post your pbs script
> or whatever you use to give me a clue. There may be several ways to do
> this... I dont know.
>
> Thanks,
>
> Dave Semeraro
>
> --
> List subscription information:
> https://*email.ornl.gov/mailman/listinfo/visit-users

>
>
>
> --
> List subscription information: https://*
> email.ornl.gov/mailman/listinfo/visit-users

>
>
>
> --
>
> ======================================================================
> Brad Whitlock                   Lawrence Livermore National Laboratory
> whitlock2 at llnl.gov
> XXXXXXXXXXXXX
> ======================================================================
>
> --
> List subscription information:
> https://email.ornl.gov/mailman/listinfo/visit-users

---

[markcmiller86]

Date: Fri, 23 Oct 2009 08:44:40 -0700
From: semeraro at ncsa.illinois.edu (Dave Semeraro)
This post was imported from the [email protected] email archive

---

I guess I am interested in doing two things here. First I am figuring out how to launch interactive
client server sessions when the server is launched via some batch mechanism. My particular situation
is poe and loadleveler. I think I can do the first?thing by passing a loadleveler command file as a
poe argument. The loadleveler file specifies the number of nodes etc but does not specify an
executable. I have tried this and it works fine.

The second thing I would like to do is launch batch non-interactive jobs that would just execute a
python script to generate one or, probably, more images saving them on disk for later examination.
That is why I was investigating the -cli -nowin option of VisIt. I may investigate using the VisIt
python module and running a python application in batch. In the batch approach all the VisIt
components would run on the remote cluster. I get confused about how to do this because only the
engine runs under mpi in that instance. So where would the other components run? I think I will have
a look at how the TACC setup runs.

Dave
----- Original Message -----
From: 'Hank Childs' <hankchilds at gmail.com>
To: 'VisIt software users community' <visit-users at email.ornl.gov>
Sent: Friday, October 23, 2009 6:50:46 AM GMT -06:00 US/Canada Central
Subject: Re: [visit-users] parallel batch execution

Hi,

> It sounds to me like the issue here is that the CLI and viewer must also run in the batch job,
which is not really supported well.
At TACC when running on the login node, they do the following:
- run qsub to get the nodes
- when the qsub job gets scheduled, they launch VisIt (meaning the cli and viewer)
- to launch the engine, the host profile uses 'ibrun' to launch in parallel on all the nodes
- the result is that the viewer and cli run on the first node and the engine runs on all the nodes
(including the first node, so there is a memory pinch there)

(Also, for those running the client somewhere besides the login node, there are separate profiles
for a combined qsub/ibrun.)

Here's an example of launching the cli & viewer on a batch node.? It seems complex, but a lot of
that is just the complexity of managing a batch system:

# To get the nodes
qsub -l h_rt=00:30:00 -V -cwd -pe 16way 16 -q vis visit_launcher_script

# The script
% cat visit_launcher_script
visit -cli -nowin -s visscript.10.py
(There are also a lot of 'module loads' in there)

# The VisIt Python script
launchArguments = ('-np', '8', '-nn', '1', '-p', 'vis', '-la', '-V -cwd', '-l', 'ibrun', '-t', '60')
batchHost = 'localhost'
OpenComputeEngine('%s'%batchHost,launchArguments)
Source('my_script_that_does_work_besides_launching_engines.py')

Best,
Hank

On Thu, Oct 22, 2009 at 9:09 AM, Brad Whitlock < whitlock2 at llnl.gov > wrote:

On the trunk, I added an option to the engine called '-reverse_launch' that lets you directly launch
the compute engine and then it will launch the viewer and cli instead of the more common case where
the cli runs the viewer which then runs the engine. The newer method is geared to more easily
support the case where you want to do everything as part of your batch job. Of course, if the engine
dies so does the viewer in that case.

Brad

Hi Dave,

(I'm 80% sure that I'm answering this question right)

If you can do it with a VisIt host profile, then you can do it with the command line as well.

One way is to launch with command line arguments.?? Another way is to launch from within your Python
script (that's what I do).

Method #1:

visit -cli -nowin -l mpirun -np 2 ...

Method #2:

>>> OpenComputeEngine('localhost', ('-l', 'mpirun', '-np', '2', ...))

So the gameplan would be to see what you have turned on in the host profile and then translate those
commands into the '-l''s and '-np''s. ?So the rest of your question is how to get the flags. ?You
can get that by running 'visit -fullhelp'.

?? ?Parallel launch options

?? ?--------------------------------------------------------------- ------------

?????? ?Notes: All of these options are ordinarily obtained from host profiles.

?????? ?However, the command line options override anything in the profiles.

?????? ?When parallel arguments are added but the engine is not the

?????? ?component being launched, -launchengine is implied. ?Explicitly

?????? ?add -launchengine to launch a remote parallel engine.

?? ?--------------------------------------------------------------- ------------

?????? ?-setupenv?????????? ?Use the VisIt script to set up the environment

??????????? ??????????? ?????? for the engine on the compute nodes.

?????? ?-par????????? ??????? Run the parallel version. This option is implied

??????????? ??????????? ?????? by any of the other parallel options listed below.

?????? ?-l?? ?<method>?????? Launch in parallel using the given method.

?????? ?-pl?? <method>?????? Launch only the engine in parallel as specified.

?????? ?-la?? <args>???????? Additional arguments for the parallel launcher.

?????? ?-sla ?<args>???????? Additional arguments for the parallel sublauncher.

?????? ?-np?? <# procs>???? ?The number of processors to use.

?????? ?-nn?? <# nodes>???? ?The number of nodes to allocate.

?????? ?-p?? ?<part>???????? Partition to run in.

?????? ?-n?? ?<name>???????? The parallel job name.

?????? ?-b?? ?<bank>???????? Bank from which to draw resources.

?????? ?-t?? ?<time>???????? Maximum job run time.

?????? ?-machinefile <file> ?Machine file.

?????? ?-expedite?????????? ?Makes DPCS give priority scheduling.

?????? ?-icet????????? ????? ?In scalable rendering mode, use the IceT parallel

??????????? ??????????? ?????? image compositor.

Best,

Hank

On Wed, Oct 21, 2009 at 12:38 PM, < semeraro at ncsa.illinois.edu > wrote:

Hello there,

I was wondering if anyone runs visit in batch mode on a parallel platform. If so how do you do it? I
need to figure out how to run visit in batch mode on an AIX platform that uses poe and loadleveler.
POE is sort of like mpirun and loadleveler is sort of like pbs. To run a parallel hello world one
submits the following load leveler command script using the llsubmit command.

#!/usr/bin/ksh
#@ job_type = parallel
#@ environment = COPY_ALL
#@ output = hellopoe.out
#@ error = hellopoe.err
#@ class = medium
#@ tasks_per_node = 1
#@ node = 4
#@ network.MPI_LAPI = sn_all,not_shared,US
#@ wall_clock_limit = 0:55:00
#@ queue
/usr/bin/poe /u/home/ac/semeraro/Projects/Poetests/hello

This starts the hello mpi program on 4 nodes. What I would like to do is something like:

visit -l poe -la -llfile visit.ll -nowin -cli -s script.py

I am guessing at some of these options based on the hostprofile I use to start parallel jobs on the
IBM platform I am using. The host profile ?for this cluster works just fine by the way.

So if anyone does something similar to what I am trying to do, even if it uses mpirun and pbs or
some other combination, please post your pbs script or whatever you use to give me a clue. There may
be several ways to do this... I dont know.

Thanks,

Dave Semeraro

---

---

---

==================================================================== ==
Brad Whitlock?????????? ???????? Lawrence Livermore National Laboratory
whitlock2 at llnl.gov
XXXXXXXXXXXXX
======================================================================

---

-- List subscription information: https://email.ornl.gov/mailman/listinfo/visit-users Searchable
list archives: https://email.ornl.gov/pipermail/visit-users VisIt Users Wiki: http://visitusers.org/
Frequently Asked Questions for VisIt: http://visit.llnl.gov/FAQ.html

---

[markcmiller86]

Date: Fri, 23 Oct 2009 08:52:10 -0700
From: hankchilds at gmail.com (Hank Childs)
This post was imported from the [email protected] email archive

---

Hi Dave,

On Fri, Oct 23, 2009 at 8:44 AM, Dave Semeraro
<semeraro at ncsa.illinois.edu>wrote:

> I guess I am interested in doing two things here. First I am figuring out
> how to launch interactive client server sessions when the server is launched
> via some batch mechanism. My particular situation is poe and loadleveler. I
> think I can do the first thing by passing a loadleveler command file as a
> poe argument. The loadleveler file specifies the number of nodes etc but
> does not specify an executable. I have tried this and it works fine.
>
The VisIt model is that you load up a client on your local desktop and then
launch a remote job on your supercomputer.  This would correspond to the
qsub/ibrun launcher profile I mentioned in the last email.  You can find
that profile at src/svn_bin/visit-config-tacc.

> The second thing I would like to do is launch batch non-interactive jobs
> that would just execute a python script to generate one or, probably, more
> images saving them on disk for later examination. That is why I was
> investigating the -cli -nowin option of VisIt. I may investigate using the
> VisIt python module and running a python application in batch. In the batch
> approach all the VisIt components would run on the remote cluster. I get
> confused about how to do this because only the engine runs under mpi in that
> instance. So where would the other components run? I think I will have a
> look at how the TACC setup runs.
>
As I'm sure is obvious to you, this one corresponds to the setup when you
run qsub to launch VisIt and then have VisIt launch the engine using only
ibrun.

-h

---