Portici
22 maggio 2013
The evolution of ENEAGRID/CRESCO
HPC infrastructure
G.Bracco
[email protected]
ENEA Centro Ricerche Frascati
V. Enrico Fermi 45, Frascati (ROMA)
S.Migliori,A.Quintiliani,S.Podda,R.Guadagni,F.Ambrosino,F.Beone,
M.Caporicci,P.D'Angelo,A.Funel,G.Ponti,G.Furini,A.Mariano,
G.Mencuccini,P.Ornelli,A.Perozziello,S.Pierattini,D.Abate,F.Poggi,
D.Giammattei,M.DeRosa,S.Pecoraro,F.Simoni,S.Giusepponi,
G.Guarnieri,A.Petricca,A.Rocchi,C.Sciò,A.Italiano,
A.Colavincenzo,G.Giannini
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Overview
The scientific computing resources of ENEA [The Italian National
Agency for New Technologies,Energy and Sustainable Economic
Development] are integrated in ENEAGRID infrastructure, a
production quality, service oriented system for high performance
and high throughput computing.
ENE
The main computing resources are the CRESCO clusters (Linux
x86_64), the main site is Portici (NA), the location of the original
CRESCO project (2008, Computational RESearch center for
COmplex systems), funded by the Italian Minister of Research and
the Education,in the framework of PON 2000-2006.
This presentation describes the infrastructure and illustrates its
evolution as funded by the new projects (PON 2007-2013
framework) where ENEA-UTICT, the ICT Unit of ENEA, is one of
the partners.
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Outline
• ENEAGRID
– The architecture & the computing resources
– Users and applications
– User interface and Virtual Labs
– CRESCO clusters
• New projects PON 2007-2013
– IT@CHA, TEDAT, LAMRECOR
• New infrastructures
– CRESCO3
– CRESCO4
• Conclusions
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
ENEAGRID architecture
The infrastructure is based on “mature” multiplatform software
components assuring reliability and easy administration. Web
interfaces have been developed/customized for a friendly user
environment:
– Kerberos
ENE 5 authentication
– File systems:
• AFS/OpenAFS geographic file system (HOME)
• GPFS: parallel file system (also WAN)
– Resource manager: LSF Multicluster
– User Web graphical interface:
• NX/FARO
• Jobrama: job monitoring & Accounting
– System monitoring: Zabbix
– Web management of users/projects: WARC
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
ENEAGRID: computing resources
ENEAGRID offers to the users computing resources based on
Linux x86_64 ( ~5800 cores), AIX SP5 (~256 cpu), special
systems (e.g. GPUs), virtualized hosts and distributed storage.
The resources are located in 6 sites, connect by GARR network.
ENE
ENE
ENE
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
ENEAGRID: CRESCO & Portici Site
ENEA Research Center Portici
Buildings designed by Vittorio
Gregotti,1982/86
ENE
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
ENEAGRID: users and applications
ENEAGRID users:
– serial or small scale parallel jobs: ~200 users
– large scale HPC parallel jobs: ~50 users
Application domains:
ENE
– combustion CFD
– aerospace CFD
– computational chemistry
– climate modeling
– atmospheric pollusion diffusion simulation
– nuclear technologies
– nuclear fusion physics
– bioinformatics
– ...
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
ENEAGRID: applications
Compilers: Intel, PGI, AMD Open64,...
MPI Flavours:mvapich,openmpi,Intel,...
Applications:
ENE
Abaqus, Amira, Ansys, Ansys CFX, Ansys Fluent, AVS Express Developer
Edition, COMSOL, CPMD, E-cell, Fluent, FreeMat, Gambit, Grass GIS,
Gsharp, IcemCFD, IDL, LynxPrime, Marc, Mathematica, Matlab, MeshLab,
ModeFrontier, MpCCI, MultiGen-Paradigm VegaPrime, Nastran,
OpenFOAM, OpenSceneGraph, Paraview, Patran, Prince, Quantum
Espresso, Quantum GIS, Scalasca, Scilab, Scirun, Siap, Starccm+,
StarDesign, StarView, Tgrid, Totalview, VIsit, Visual Molecular Dynamics,
Workbench, XILINX...
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
CRESCO utilization by domain of activity
Combustion CFD
621.87
ENEA
New materials
531.9
ENEA,
INFN/RM,
Numonyx
Climate modeling
357.57
ENEA,Ylichron
Nuclear fission
122.9
ENEA,ISS
HPC Support activity
103.11
ENEA
Atmospheric pollution
65.62
ENEA,AriaNet
Aerospace CFD
32.77
Avio,AAPS,UniROMA1
University
22.13
UniROMA1,CERI
Nuclear fusion
19.99
ENEA,PoliTo
Bioinformatics
18.3
ENEA, CNR-ITB, Ylichron,CNR-ISA
EFDA- Fusion
4.69
ENEA,EFDA
Industry
0.37
NICE,CETMA
ENE
INFN/NA,UniFI,UniSaUniSS,
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
FARO -
Fast Access to Remote Objects
Web access to
data, applications
and virtual
machines, a
solution based on
the integration of
NX and a java
interface
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
FARO & Virtual Labs
Thematic portals
integrating the access to
data and to the
applications for a
specific context/activity
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Virtual Labs (1)
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Virtual Labs (2)
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Virtual Labs (3)
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Virtual Labs (4)
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Virtual Labs (5)
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
FARO: 3D visualization
3D Remote Rendering
Total
displacement
(mm
)
Pos Processo
t
r
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
HPC: scalability test of various applications
Commercial code
Fluent
Open Source:
OpenFoam
OpenFoam
cpmd
Combustion
Processors
procs
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
CRESCO clusters
>Portici CRESCO1(672 cores 4U) CRESCO2(2720 cores blades HS21,GPFS)
IB CISCO 70xx, DDR Intel Clovertown,Tigerton,Nehalem,Westmere
>Casaccia CRESCOC
(192 cores, twin 1U,Supermicro)
ENE
IB Qlogic Silverstorm DDR, AMD 2427 Istanbul
>Frascati CRESCOF (480 cores twin square 2U, GPFS)
IB Qlogic 12300 QDR, AMD 6172 Magnycours
>Brindisi CRESCOB (80 cores, 4 U, GPFS)
GEthernet, Intel Tigerton
>Trisaia CRESCOT (16 cores,4U)
Intel Tigerton
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
CRESCO1/2 Clusters - DDR IB & Storage
SM (ACTIVE)
SM (standby
SFS 7000D
SFS-7000D (e)
SFS-7000D(D)
4
4
2)
4
ENE
20 TB
20 TB
20 TB
20 TB
2)
2)
4
4
Server
Server
STORAGE
Server
STORAGE
Server
STORAGE
STORAGE
2)
4
2)
4
2)
SFS 7000D
SFS 7000D
4)
4 graphic FRONT
END
SFS 7012-144-P
4 X FRONT END
GRAFICI
SFS-7024-288P
4
4
4X FRONT END
4X FRONT END
42 X 3850M2
10 X 3755
340 X HS21
SECTION 2
CRESCO2
CRESCO1
BACK-UP
4
(2)
2X GPFS NODE
2X GPFS NODE
FC
IB 36 ports
DDN S2A9900 120TB
FC
2X GPFS NODE
4
IBM DCS 9550 -180TB
Storage: IBM/DDN9550 120 TB; DDN9900 90 TB; disk servers 80 TB
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
New projects : PON 2007-2013
• PON/1 Ricerca Industriale DDR MIUR 01/Ric 18/1/2010
– IT@CHA - (application of new technologies to the conservation and
valorization of cultural and artistic heritage)
– LAMRECOR - (Advanced logistics for persons and goods)
– DIRECTFOOD - Gestione integrata filiere alimentari e canali innovativi
produttore -consumatore
• PON/2 Distretti/Laboratori Pubblico Privati DDR MIUR 713/Ric
29/10/2010
– VIS4FACTORY - Sistemi Informativi Visuali per i processi di fabbrica nel
settore dei trasporti
– DATABENC - Distretto ad ALta Tecnologia per i beni culturali nella regione
Campania
• PON/3 Infrastrutture DDR MIUR 254/Ric 18/5/2011
– TEDAT - Centro di eccellenza per le Tecnologie e la diagnostica avanzata nel
settore dei trasporti (Advanced technologies for transport,new materials for
aerospace, automotive,..)
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
PON 2007-2013: Projects started 2012
Budget for infrastructures:
• IT@CHA [Ricerca industriale] ENEA-UTICT 320 k€
• LAMRECOR [Ricerca industriale] ENEA-UTICT 548 k€
On these projects: new CRESCO3 cluster
• TEDAT [Infrastrutture] ENEA-UTICT 2.221 k€
– Portici (2.056 k€)
– Brindisi (165 k€)
On this project: new CRESCO4 cluster
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Portici: new cluster CRESCO3
In the framework of IT@CHA and LAMRECOR projects the new CRESCO3
cluster has been set up in CRESCO computer room:
84 AMD nodes, 2016 cores, Twin Square SuperMicro & Acer,
peak
Infiniband QDR QLogic 12800-040 (96 ports)
2 Racks (wide)
19.3 TFlops
The cluster is now opened for general use.
HPL Benchmark: 75 % efficiency; compilers with Interlagos support (Open64
specific flags, Intel 12 specific flags; ACML Libraries)
Storage: DDN S2A9900 600 TB; GPFS
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
CRESCO3:SMicro/Acer twinsquare QDR IB
AMD Iterlagos 6234 2.4 GHz,64 GB,24 cores,84 nodes,2016 cores
ENE
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Portici: CRESCO4 cluster
In the framework of TEDAT project the procurament of the new CRESCO4
cluster is currently under way.
An European tender has been set up, evaluated and the contract has been
awarded (E4 Company).
–
–
–
–
–
–
–
–
–
–
304 computing nodes Intel E2670, 2.6 GHz, 4864 cores
Supermicro Fat-Twin chassis (8 nodes in 4U)
101 TFlops Peak
5 Racks for computing nodes, 1 network rack
Storage: DDN S2A9900 600 TB
Infiniband QDR QLogic/Intel 12800-180 (432 ports)
New computer room
Conditioning system taking advantage of free cooling technogy
Delivered in Portici by July 2013
Final test completed October 2013
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Portici: CRESCO4 - IB network
ENE
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Computer rooms: CRESCO & CRESCO4
ENE
new
CRESCO4
computer
room
CRESCO computer
room
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
CRESCO computer room (1)
ENE
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
CRESCO computer room (2)
ENE
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
CRESCO4 computer room (1)
ENE
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
CRESCO4 computer room (2)
ENE
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
CRESCO4 computer room (3)
ENE
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
CRESCO4 computer room (4)
ENE
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
CRESCO4 computer room (5)
ENE
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Conclusions
A significant update of the ENEAGRID HPC infrastructure is currently under
way, with the new CRESCO3 and CRESCO4 clusters, funded by several PON
2007-2013 projects
ENE
The architecture of ENEAGRID provides the framework for an easy
introduction of the new computing resources both from the point of view
of the user and the administrator.
By the end of the year the integrated peak computing power of ENEAGRID
will increase of a factor 5 up to ~150 Tflops, the most powerfull system
being CRESCO4 with ~100 Tflops.
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013
Links
• www.enea.it
• www.cresco.enea.it
• www.eneagrid.enea.it
• www.afs.enea.it
G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013