Using → Adding IMB

AKRR: Deployment of IMB Applications Kernels on a Resource

Here the deployment of xdmod.benchmark.mpi.imb application kernel is described. This application kernel is based on Intel MPI Benchmark (IOR) which design to measure the network performance.

For further convenience of application kernel deployment lets define APPKER and RESOURCE environment variable which will contain the HPC resource name:

export RESOURCE=<resource_name>
export APPKER=imb

Installing IMB

In this section the IMB installation process will be described, see also IMB documentation for installation details ( ).

Building IMB Executables

First we need to install IMB. Below is a sample listing of commands for IMB installation:

#cd to application kernel executable directory

# obtain latest version of IMB
tar xvzf IMB-v2019.2.tar.gz
# create link 
ln -s mpi-benchmarks-IMB-v2019.2 imb

#load MPI compiler
module load intel-mpi/2018.3 intel/18.3

cd imb
export CC=mpiicc
make IMB-MPI1
make IMB-EXT

Generate Initiate Configuration File

Generate Initiate Configuration File:

On AKRR server

akrr app add -a $APPKER -r $RESOURCE

Sample output:

[INFO] Generating application kernel configuration for imb on ub-hpc
[INFO] Application kernel configuration for imb on ub-hpc is in: 

Edit Configuration File

In contrast to others application kernels IMB uses only one process per node. Therefore during configuration of IMB it is importent to ensure that only one process is run per node. 

Below is a listing of configuration file located at  ~/akrr/etc/resources/$RESOURCE/ for SLURM:


appkernel_run_env_template = """
#Load application enviroment
module load intel/13.0
module load intel-mpi/4.1.0
module list

export I_MPI_PMI_LIBRARY=/usr/lib64/

ulimit -s unlimited

#set how to run mpi applications, one process per node
RUNMPI="srun --ntasks-per-node=1 -n {akrr_num_of_nodes}"

Here srun with –ntasks-per-node=1 is used to set one process per node execution. Modify loading application enviroment to reflect mpi library with which imb was compile.

Below is an example for PBS:


#Load application enviroment
module swap mvapich2 impi/
module list

ulimit -s unlimited
#set how to run mpi applications, one process per node
cat $PBS_NODEFILE|uniq >> all_nodes
cat all_nodes
RUNMPI="mpirun -n $AKRR_NODES -machinefile all_nodes"

Here, we first generates machine file containing only one node per process and uses this file with mpirun.

Generate Batch Job Script and Execute it Manually (Optional) 

The purpose of this step is to ensure that the configuration lead to correct workable batch job script. Here first batch job script is generated with ‘ batch_job’. Then this script is executed in interactive session (this improves the turn-around in case of errors). If script fails to execute, the issues can be fixed first in that script itself and then merged to configuration file.

This step is somewhat optional because it is very similar to next step. However the opportunity to work in interactive session improve turn-around time because there is no need to stay in queue for each iteration.

First generate the script to standard output and examine it:

akrr task new --dry-run --gen-batch-job-only -n 2 -r $RESOURCE -a $APPKER

Sample output

DryRun: Should submit following to REST API (POST to scheduled_tasks) {'repeat_in': None, 'time_to_start': None, 'resource': 'ub-hpc', 'app': 'imb', 'resource_param': "{'nnodes':2}"}
[INFO] Directory /home/akrruser/akrr/log/data/ub-hpc/imb does not exist, creating it.
[INFO] Directory /home/akrruser/akrr/log/comptasks/ub-hpc/imb does not exist, creating it.
[INFO] Creating task directory: /home/akrruser/akrr/log/data/ub-hpc/imb/2019.
[INFO] Creating task directories: 
[INFO] auto_walltime_limit is on, trying to estimate walltime limit...
[INFO] There are only 0 previous run, need at least 5 for walltime limit autoset
[INFO] Below is content of generated batch job script:
#SBATCH --partition=general-compute 
#SBATCH --qos=general-compute
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --time=00:30:00
#SBATCH --output=/projects/ccrstaff/general/nikolays/huey/akrr_data/imb/2019.
#SBATCH --error=/projects/ccrstaff/general/nikolays/huey/akrr_data/imb/2019.
#SBATCH --constraint="CPU-L5520"
#SBATCH --exclusive

#Common commands
export AKRR_NODES=2
export AKRR_CORES=16
export AKRR_NETWORK_SCRATCH="/projects/ccrstaff/general/nikolays/huey/tmp"
export AKRR_LOCAL_SCRATCH="/tmp"
export AKRR_TASK_WORKDIR="/projects/ccrstaff/general/nikolays/huey/akrr_data/imb/2019."
export AKRR_APPKER_DIR="/projects/ccrstaff/general/nikolays/huey/appker"
export AKRR_AKRR_DIR="/projects/ccrstaff/general/nikolays/huey/akrr_data"

export AKRR_APPKER_NAME="imb"
export AKRR_RESOURCE_NAME="ub-hpc"
export AKRR_TIMESTAMP="2019."

export AKRR_APPKERNEL_INPUT="/projects/ccrstaff/general/nikolays/huey/appker/execs/imb/src/IMB-EXT"
export AKRR_APPKERNEL_EXECUTABLE="/projects/ccrstaff/general/nikolays/huey/appker/execs/imb/src/IMB-MPI1"

source "$AKRR_APPKER_DIR/execs/bin/akrr_util.bash"

#Populate list of nodes per MPI process
export AKRR_NODELIST=`srun -l --ntasks-per-node=$AKRR_CORES_PER_NODE -n $AKRR_CORES hostname -s|sort -n| awk '{printf "%s ",$2}' `

export PATH="$AKRR_APPKER_DIR/execs/bin:$PATH"


#run common tests

#Write some info to, JSON-Like file
akrr_write_to_gen_info "start_time" "`date`"
akrr_write_to_gen_info "node_list" "$AKRR_NODELIST"

#create working dir
mkdir workdir
cd workdir

export AKRR_APPKER_EXEC_DIR=/projects/ccrstaff/general/nikolays/huey/appker/execs/imb/src

#Load application enviroment
module load module load intel-mpi/2018.3 intel/18.3
module list

export I_MPI_PMI_LIBRARY=/usr/lib64/

ulimit -s unlimited

#set how to run mpi applications, one process per node
RUNMPI="srun --ntasks-per-node=1 -n 2"

#Generate AppKer signature /projects/ccrstaff/general/nikolays/huey/appker/execs/imb/src/IMB-MPI1 $AKRR_TASK_WORKDIR/.. >> $AKRR_APP_STDOUT_FILE /projects/ccrstaff/general/nikolays/huey/appker/execs/imb/src/IMB-EXT  $AKRR_TASK_WORKDIR/.. >> $AKRR_APP_STDOUT_FILE

#Execute AppKer
echo "Checking that running one process per node (for debugging)"
${RUNMPI} hostname

akrr_write_to_gen_info "appkernel_start_time" "`date`"
${RUNMPI} ${AKRR_APPKER_EXEC_DIR}/IMB-MPI1 -multi 0 -npmin 2 -iter 1000 >> $AKRR_APP_STDOUT_FILE 2>&1
${RUNMPI} ${AKRR_APPKER_EXEC_DIR}/IMB-EXT  -multi 0 -npmin 2 -iter 1000 >> $AKRR_APP_STDOUT_FILE 2>&1
akrr_write_to_gen_info "appkernel_end_time" "`date`"

cd ..
if [ "${AKRR_DEBUG=no}" = "no" ]
        echo "Deleting input files"
        rm -rf workdir

akrr_write_to_gen_info "end_time" "`date`"

[INFO] Removing generated files from file-system as only batch job script printing was requested

Next generate the script on resource:

akrr task new --gen-batch-job-only -n 2 -r $RESOURCE -a $APPKER
[INFO] Creating task directory: /home/akrruser/akrr/log/data/ub-hpc/imb/2019.
[INFO] Creating task directories: 
[INFO] Creating batch job script and submitting it to remote machine
[INFO] Directory huey:/projects/ccrstaff/general/nikolays/huey/akrr_data/imb does not exists, will try to create it
[INFO] Directory huey:/projects/ccrstaff/general/nikolays/huey/akrr_data/imb/2019. does not exists, will try to create it
[INFO] auto_walltime_limit is on, trying to estimate walltime limit...
[INFO] There are only 0 previous run, need at least 5 for walltime limit autoset
[INFO] Local copy of batch job script is /home/akrruser/akrr/log/data/ub-hpc/imb/2019.

[INFO] Application kernel working directory on ub-hpc is /projects/ccrstaff/general/nikolays/huey/akrr_data/imb/2019.
[INFO] Batch job script location on ub-hpc is /projects/ccrstaff/general/nikolays/huey/akrr_data/imb/2019.

The output contains the working directory for this task on remote resource. On remote resource get to that directory and submit batch script for execution.

On remote resource

#get to working directory
cd /projects/ccrstaff/general/nikolays/huey/akrr_data/imb/2019.

#run ior application kernel
bash imb.job

Examine appstdout file, which contains application kernel output (appstdout sample).

If it looks ok you can move to the next step

Perform Validation Run

On this step utility is used to validate application kernel installation on the resource. It executes the application kernel and analyses its results. If it fails the problems need to be fixed and another round of validation (as detailed above) should be performed.

akrr app validate -n 2 -r $RESOURCE -a $APPKER 

See validation output sample

If validation was executed successfully IMB is installed and can be schedule for regular execution.

Schedule regular execution of application kernel.

Now this application kernel can be submitted for regular execution:

#Perform a test run on all nodes count
akrr task new -r $RESOURCE -a $APPKER -n 2,4,8

#Start daily execution from today on nodes 2,4,8 and distribute execution time between 1:00 and 5:00
akrr task new -r $RESOURCE -a $APPKER -n 2,4,8 -t0 "01:00" -t1 "05:00" -p 1

# Run on all nodes count 20 times (default number of runs to establish baseline)
akrr task new -r $RESOURCE -a $APPKER -n 2,4,8 --n-runs 20

see Scheduling and Rescheduling Application Kernels and  Setup Walltime Limit for more details.

Up: Deployment of Application Kernels on Resource