This document describes the TeraACMiN (TA) HPC from a user perspective. You will find here most of the important informations regarding usage, resources, limitations etc. If some clarification is needed feel free to contact TA admin writing to tera-acmin@agh.edu.pl or write directly via our disscussion channel for registed users.
If you are a member of ACMiN staff just contact TA administrator.
If you are not a member of staff but you collaborate with the one or you plan to, also write to Andrzej Biborski andrzej.biborski@agh.edu.pl who is responsible for the maineance of TA.
TA conists of 96 computational nodes, 16 of them are supplied with GPU units.
Each nodes is supplied with two Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz processors. There are 8 cores on each proccessor, thus 16 cores are
available per node. Note, that those two Xeons are treated logically as single CPU thanks to SMP architecture.
Non-GPU nodes are supplied with 256GB of RAM, GPU nodes posses 128GB. All physical processors are supplied with 20480 KB of L3 cache.
To be done
InfiniBand layer connecting all nodes is configured for MPI oriented software.
You need to run ssh client on your machine. Connect to th server located at tera.acmin.agh.edu.pl
Each user may use self-developed or locally installed software. However, there are some packages already intalled, e.g.,
Also, development tools such as compilers, (gnu, Intel), libraries (OpenMPI, Boost, SciPy, LAPACK, ScaLAPACK etc.) are easily accessible for each user.
The software available for all the users is organized in modules. It is not obligatory to know this tool at expert level, in fact, you need to know command module with two important options. Before you ask, if given software is installed please try to check it by yourself. For this purpose, fing e.g. GCC module you need to type
[user@login.tera ~]$ module spider gcc
When you intend to use the given software, you need to load related module, e.g., you would like to use gnuplot for drawing some figures, then type
[user@login.tera ~]$ module load gnuplot
and the gnuplot is ready to be used.
When you plan to use softaware, which you need to build on your own locally, you also may need to load some modules (compiler, libraries to link etc.). It is strongly reccomended to do so in interactive session via queing server to avoid overloading of login server. For details see the next Section.
The queue managment system exploioted on TA is slurm. Again there are only few slurm commands you need to operate with your jobs. Jobs may be divided into "regular" and "interactive. We briefly sketch both scenarios below.
When you intend to start regular job, it is implicitly assumed that you are nat going to interact with job during its execetution. Typicaly your program run for hours/days. The most important thing is the preperation of script for your job submission. There are two main sections in the slurm job script:
You may find details of script preparation in the slurm documentation, however, below you will find the snippet which you may find useful to addapt for your needs.
#!/bin/bash
#Resources description
#SBATCH --job-name=example_job # Job name
#SBATCH --ntasks=2 # Number of MPI tasks (i.e. processes)
#SBATCH --cpus-per-task=16 # Number of cores per MPI task
#SBATCH --nodes=2 # Minimum number of nodes to be allocated
#SBATCH --ntasks-per-node=1 # Maximum number of tasks on each node
#SBATCH --time=36:00:00 # Wall time limit (days-hrs:min:sec)
#SBATCH --mem-per-cpu=8G # memory per cpu-core
#SBATCH --mem=2G # total memory per node
#SBATCH --output=log
#SBATCH --partition=tera-cpu
#job execution
module load OpenMPI
module load ScalaPACK
mpiexec ./myprogram inputfile
Note, there are limitations related to hardware, e.g., maximal value of --cpus-per-taks is 16, also number of nodes or amount of memory etc. Also, the --partition provides some limitations. Partition is just the pre-defined type of jobs to be queued and executed in slurm terminology. Each partition should be selected according to the requirements of particular job to be executed. The partitions organization on TA are given in a table below.
| Parition | time limit (days) | resources limits per partition (nodes) |
|---|---|---|
| all | 5 | 93 |
| tera-cpu | 5 | 78 |
| tera-gpu | 5 | 15 |
| tera-long | 14 | 8 |
| tera-very-long | 30 | 2 |
You may check also additional information (number of nodes which are in idle, allocated or mixed state) by sinfo command. When you have prepared your script you need to submit your job using sbatch command:
[user@login.tera ~]$ sbatch path_to_my_script/job_script.sc
you may check the state of your job afterwards
[user@login.tera ~]$ squeue -u user
when your job is in the pending state (PD) you may ask scheduler about estimated time of job start
[user@login.tera ~]$ squeue --start
Interactive job serve the possibility to run your codes, process data, compile software on the computational node. To enter the interactive job you need to run following command
[user@login.tera ~]$ srun --pty bash
in the example above, just bash shell is executed on the node allocated by slurm. If you need specify the time you will need use --time option, e.g,,
[user@login.tera ~]$ srun --pty --time=1:00:00 bash
will allocate single node for one hour.