Astronomic&Space Information and Computing Center

Astronomic&Space Information and Computing Center

The computing laboratory, as part of the Fundamental Astrometry department, was founded in 1965. The head of the laboratory was Dr.Sci Duma D.P. Subsequently the computing laboratory began to deal with measuring devices and was renamed as the Computing and Measuring Center. The head of the Center Dr.Sci Taradiy V.K. established the structure of the Center with the positions for qualified young specialists in programming and electronics engineering. The Astro/Space Information and Computing Center was founded in 1994 on the basis of the Computing and Measuring Center. The head of the ASICC became Dr.Sci. Berczik P.P.

The basic tasks of the Center are: information support of scientific researches of MAO; all kinds of information services providing; MAO-worldwide information exchange support; MAO computing equipment/facilities maintenance; GRID-cluster services.

The ASICC maintains the Observatory's local network of about 200 personal computers.

The Department Staff.

Scientific and technical group tasks are: programming and technical support of MAO scientific researches, MAO-worldwide information exchange providing, the Observatory computing equipment maintenance.

Local network of MAO NANU. Local network of MAO NANU provides information transferring with speed limit up to 1GBps. Access to the cluster facilities also use 1GBps channel. The UARNET company provides MAO NANU Internet access at present time. The Internet channel capacity increased up to 10GBps since 2012.

SERVERS

MAOLING. CPU 3.4 GHz, 3 GB RAM, 750 GB.

The main MAO server hosts MAO website, ftp-server, the gateway to the MAO local network. This server provides the access to the user's local data from any computer both from the local network and from the outside. LDAP- and DNS- servers are running on MAOLING as well.

OBERON. 2х2 GHz, 16 GB RAM, 1.5 TB.

The OBERON is used for various computational tasks due to its performance. The mirror of the NASA Astrophysics Data System is located on this PC as well.

JANUS / MAIL. 3 GHz, 3 GB RAM, 320 GB. Mail-server based on ZIMBRA software; provides the access to personal MAO mail-boxes through POP-, IMAP- та WEB-interfaces.

VIRGO4.

The server hosts UAA and Ukrainian Virtual Observatory web-sites and JDA UkrVO control and searching module.

CLUSTER OF THE MAIN ASTRONOMICAL OBSERVATORY OF NASU

The high-performance computing GRAPE/GRID cluster was set in operation in the Main Astronomical Observatory in 2007 due to the financial support of the National Academy of Sciences of Ukraine. The basic computational element of the cluster consisted of the 9 Grape6-BLX64 cards and provided about 1 Tflops floating point operations for parallel tasks such as dynamic simulation of the evolution of galaxies, galactic nuclei and star clusters.

In 2011 the cluster was upgraded. Grape6-BLX64 cards were replaced by modern graphics accelerators GeForce 8800 GTS 512, thus improving the performance of astrophysical tasks modeling to approximately 4 Tflops. The cluster computing capabilities upgrade became possible due to the assistance of the Astronomisches Rechen-Institut (ARI) am Zentrum für Astronomie der Universität Heidelberg (Germany) and directly Dr. Rainer Spurzem.

In 2013 cluster was equipped by 16 GPU GeForce GTX 660 cards which provide 3-times higher performance for astrophysical tasks compared with GeForce 8800 GTS 512. Currently, the cluster is heterogeneous and consists of 8+3 computing nodes and one manager node. The Gigabit ethernet network is used for communication. Total number of computing cores is 88. The storage system includes the disk array of 5 Tb capacity, where users home directories are located, and RaidZ array of 7.1 Tb.

Each of the first 3 computing nodes is built on the basis of the Intel Xeon 5410 processors at the HP ML350 G5 servers. Each node has two dual-core processors Intel Xeon 5410 with working frequency of 2.33 GHz, 16 Gb RAM. Other 8 nodes are based on the Intel Xeon E5420 processors. Each node has 2 quad-core processors Intel Xeon E5420, with working frequency of 2.50 GHz, 8 Gb RAM and two GPU GeForce GTX 660.

SOFTWARE

The Debian GNU/Linux 7.0 distribution which is the one of the Linux based (free and open source software) operating systems is installed in the cluster.

GNU-and Intel compilers for C/C++ and Fortran are available for users. OpenMPI 1.4.3 package is used to run parallel applications. Task queue is managed by Torque 2.4.12 and Maui 3.3.1 packages. Task queues available for users are:

The queues listed below are available for cluster users: cpu_8x6, gpu_2. The queue grid is used for computations through GRID environment. Immediate task running in this queue is prohibited.

The software package CUDA SDK (version 5.5) is installed on the cluster to perform computations using GPU. It allows one to run software that uses CUDA technology and OpenCL.

GRID

The cluster of the MAO of NASU is a part of the Ukrainian Academic GRID segment. The GRID middleware Nordugrid ARC 5.0.4 is installed on the cluster.p>

Members of the virtual organizations have an opportunity to carry out their tasks on the cluster:

• multiscale;
• sysbio;
• moldyngrid;
• virgo_ua.

To ensure the efficient operation of the cluster on GRID the runtime-environment is installed. It supports virtual organizations moldyngrid and virgo_ua:

• GROMACS-4.5.5;
• PHI-GPU;
• VIRGO/COSMOMC;
• VIRGO/GADGET;
• VIRGO/FERMI;
• VIRGO/XMM.

Using of the computing cluster is free for staff of the MAO and other research institutions under the following agreements: user submits an application and accepts the user rules.

Monitoring of the cluster:

http://golowood.mao.kiev.ua/ganglia/

Monitoring of the Ukrainian Academic GRID segment resources:

http://gridmon.bitp.kiev.ua/

http://www.nordugrid.org/monitor/loadmon.php

Any questions of cluster send to: golowood-admin(at)mao.kiev.ua.

TEST NODES

In 2011 the computing resources of the MAO have been increased by two test nodes based on processors Intel Core i5-2500K CPU (3.30 GHz), 16 Gb RAM. This upgrade became possible due to financial support from the National Academy of Sciences of Ukraine. At the beginning of 2017 one of the new nodes is equipped with a graphics accelerator Nvidia GeForce GTX Titan, and the other − Nvidia GeForce GTX 1080.

Nvidia GeForce GTX Titan
Cores:	2688 cores
Total amount of global memory:	6 Gb
GPU Clock Speed:	837-993 MHz
Theoretical Shader Processing Rate (SP):	4.5 Tflops

GeForce GTX 1080
Cores:	2560 cores
Total amount of global memory:	8 Gb
GPU Clock Speed:	1700 MHz
Theoretical Shader Processing Rate (SP):	8.7 Tflops

Instructions for running tasks on MAO cluster

To put the task to perform, user ought to prepare an executable script or run a task online. First, task is compiled, then put to a queue using the qsub programme (man qsub).

Example of how to run a serial task:

a)

~$ echo "sleep 1m" | qsub

b) using a script:
user makes a script to run a task run.sh:

~$ cat run.sh
#PBS -k oe
#PBS -m abe
#PBS -N run
#!/bin/sh
cd $PBS_O_WORKDIR
export PATH=$PATH:$PBS_O_WORKDIR
sleep 1m
exit 0
~$

Script starting:

~$ qsub -V run.sh

(be sure to specify the option -V).

To compile and run parallel tasks on the cluster user can use the GNU or Intel compilers. To choose a compiler use commandsmpi-selector and mpi-selector-menu.

To see a list of available MPI implementations use the command:

~$ mpi-selector --list
openmpi-1.4.3
openmpi-1.4.3-intel
~$

At present there are two OpenMPI implementations: openmpi-1.4.3 (GNU) and openmpi-1.4.3-intel (Intel).

To find the version of MPI user should type:

~$ mpi-selector --query
default:openmpi-1.4.3
level:user
~$

To change MPI implementation user should use the command:

~$ mpi-selector-menu
Current system default: openmpi-1.4.3
Current user default: openmpi-1.4.3

"u" and "s" modifiers can be added to numeric and "U"
commands to specify "user" or "system-wide".

1. openmpi-1.4.3
2. openmpi-1.4.3-intel
U. Unset default
Q. Quit

Selection (1-2[us], U[us], Q): 1u
Defaults already exist; overwrite them? (Y/N) y


Current system default: openmpi-1.4.3
Current user default: openmpi-1.4.3

"u" and "s" modifiers can be added to numeric and "U"
commands to specify "user" or "system-wide".

1. openmpi-1.4.3
2. openmpi-1.4.3-intel
U. Unset default
Q. Quit

Selection (1-2[us], U[us], Q): Q
~$

mpi-selector-menu changes system variables PATH and MANPATH only the next time the user logs in the shell. Therefore, after the command mpi-selector-menu user should to run bash -l or open a new shell.

Example of how to run parallel programmes:

User compiles a programme, which is written using MPI library:

for C programme:

~$ mpicc -o your_program.exe your_program.c -lm

for Fortran programme:

~$ mpif77 -o your_program.exe your_program.f -lm

Startup script for task running has been prepared run-mpi.sh:

~$ cat run-mpi.sh
#PBS -N run-mpi
#PBS -k oe
#PBS -m abe
#PBS -l nodes=8
#!/bin/sh
cd $PBS_O_WORKDIR
export PATH=$PATH:$PBS_O_WORKDIR
mpiexec -n 8 ./your_program.exe
exit 0
~$

To define a queue for task execution user should add the line to his script:

#PBS -q cpu_8x6

The cpu_8x6 queue is designed to run CPU tasks, gpu_1 and gpu_2 queues − for GPU tasks.

To run script user should type:

~$ qsub -V run-mpi.sh

(be sure to specify the option -V).

In this case, the task is running on 8 CPU that is indicated by the parameter -n 8 of the mpiexec programme and the parameter #PBS -l nodes=8.

More information about the described options and programs − man qsub, man mpiexec.

Options in the PBS script:

-k: keep the standard output (stdout) and standard error output (stderr).

-m: send e-mail message when the task starts and ends or if the task is removed by batch-system.

-N: the name of the task in the batch system.

User can check the task status in a queue by the command:

~$ qstat

(for the details − man qstat).