ANSYS
These instructions are for using Ansys on the current HPCC environment that uses the SLURM scheduler.
Guidelines for scheduling parallel (MPI) jobs
Here are some guidelines for requesting resources.
Use --ntasks instead of nodes
Note that -N
or -–nodes=
will request that number of unique computers,
but what most users want is the number of tasks across nodes.
Use the number of tasks you requested instead of number of nodes for the `-t``
parameter:
-t $SLURM_NTASKS
Don't forget to request memory
Request memory per task, and since the default is to have 1 cpu per
task, you can request memory using e.g. --mem-per-cpu=1gb
Create a temporary file for node list
Inside the job, Fluent requires a file of a particular format ,and the SLURM node file doesn't work. This seems to work
# create and save a unique temporary file
FLUENTNODEFILE=`mktemp`
# fill that tmpfile with node list Fluent can use
scontrol show hostnames > $FLUENTNODEFILE
# in your fluent command, use this parameter
-cnf=$FLUENTNODEFILE
Example fluent Job script (using Intel compiler). Increase tasks and memory as needed
Ansys/Fluent job
#!/usr/bin/bash --login
# example 1 hour job with ntasks across any number of nodes
# adjust the ram and tasks as needed
#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=1
#SBATCH --ntasks=10
#SBATCH --mem-per-cpu=1gb
echo "This script is from ICER's Ansys/Fluent example"
# create host list
NODEFILE=`mktemp`
scontrol show hostnames > $NODEFILE
# Load the ansys/cfx v19.2 module
module load ANSYS
# The Input file
DEF_FILE=baseline.def # this file is something you have to provide!
cfx5solve -def $DEF_FILE -parallel -par-dist $NODEFILE -start-method "Platform MPI Distributed Parallel"> cfx5.log
After you have logged into a development nodes with an X11 terminal (or use the OnDemand desktop as described above), You may run ANSYS tools in parallel and interactively as follows.
# start a approximately 4 hour interactive job with 10 tasks. Adjust tasks and memory as needed
# you'll have 4 hours to work. You must be in a X11 terminal for this to work
salloc --ntasks=10 --cpus-per-task=1 --mem-per-cpu=1gb --time=3:59:00 --x11
# wait for log-in and then...
# load module
ml intel ANSYS
# this creates a temporary file and fills it with node list Fluent can use
NODEFILE=`mktemp`
scontrol show hostnames > $FLUENTNODEFILE
#for example, run the workbench
runwb2
# after running workbench you can start fluent directly
# note we are using Intel mpi
fluent 3ddp -t $SLURM_NTASKS -mpi=intel -cnf=$FLUENTNODEFILE -ssh
CFX5 Solver
This solver uses a different hosts file format for the par-dist parameter. The following uses an example Definition file provided by Ansys 19.2.
The batch script will adapted the par-dist file depending on how you specify tasks and tasks-per-node (the example below does not specify tasks per node). Code is taken from https://secure.cci.rpi.edu/wiki/index.php?title=CFX.
CFX5 Solver Example sbatch
#!/usr/bin/bash --login
# example 1 hour job with ntasks across any number of nodes
# adjust the ram and tasks as needed
#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=1
#SBATCH --ntasks=10
#SBATCH --mem-per-cpu=1gb
echo "This script is from ICER's Ansys/CFX5 example"
module load ANSYS
# code adapts the hosts file depending on if you use multiple nodes and the tasks-per-node option.
srun hostname -s > /tmp//hosts.$SLURM_JOB_ID
if [ "x$SLURM_NPROCS" = "x" ]; then
if [ "x$SLURM_NTASKS_PER_NODE" = "x" ];then
SLURM_NTASKS_PER_NODE=1
fi
SLURM_NPROCS=`expr $SLURM_JOB_NUM_NODES \* $SLURM_NTASKS_PER_NODE`
fi
# use ssh instead of rsh
export CFX5RSH=ssh
# format the host list for cfx
cfxHosts=`tr '\n' ',' < /tmp//hosts.$SLURM_JOB_ID`
# example file
DEF=$EBROOTANSYS/v242/CFX/examples/StaticMixer.def
# run the partitioner and solver
cfx5solve -par -par-dist "$cfxHosts" -def $DEF -part $SLURM_NPROCS -start-method "Platform MPI Distributed Parallel"
# cleanup
rm /tmp/hosts.$SLURM_JOB_ID
# output will be in a file named like StaticMixer_001.out and StaticMixer_001.res