BaBar Tier A
Administration
Workshop CCR, Paestum Giugno
2003
Alberto Crescente, INFN Sez. Padova
Farm BaBar – Necessary Administration
Informations
Computer Model
➢
➢
Machine's Informations
Connections Informations
➢
Computer Location
➢
Temperatures, Fans
➢
➢
Monitoring Informations
CPU Load, I/O Load
➢
Hardware Errors
➢
Messages
➢
➢
Log Files
Mail
➢
Mysql
➢
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – Used Tools
➢
Machines Database (WebDB)
Machine Informations (CPU, RAM, etc.)
•Repair Informations
•Machine Connections
•Machine Location
•
➢
Log Analizer (sysAlarm)
Syslog analizer
•
➢
Performance Monitor (perfMC)
CPU Load
•I/O Load
•Temperatures
•
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – General Data Flow
Extreme Cluster
Administration Toolkit
Configuration Files
MySQL
PHP
Language C
WebDB
to xCAT
PerfMC
web
server
Farmcfg
PHP
HTML
WebDB
Clients &
Servers
syslogs
HTML/XML
SysAlarm
RR
D
PerfMC
pooler
Management Server
syslog
SysAlarm
SNMP
Script Perl
/var/log/messag
es
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – sysAlarm Data Flow
Clients &
Servers
syslogs
Query
Management Server
syslog
MySQL
Script Perl
SysAlarm
/var/log/message
s
Workshop CCR, Paestum Giugno 2003
SysAlarm
SysAlarm
configuration
file
Alberto Crescente, servizio calcolo
Farm BaBar – sysAlarm Configuration File
logPath ./log
fileToMonitor /var/log/messages
# Specify the type of file to monitor ('syslog' or 'chslog')
monitorType syslog
debug 1
#
# You can have multiple matchLine lines
#
matchLine rx buffer
matchLine timeout waiting
matchLine System Error
#
# this is the "basename" for the logfile
#
sysAlarm sysAlarm
Workshop CCR, Paestum Giugno 2003
# Notification recipients
# email program
# Default message for the subject
#
email [email protected]
mailBin /bin/mail
emailSubject sysAlarm Notification
#
# database that will store log messages...
#
#sqlserv bbr-user
sqlserv sqlserv
sqluser xxx
sqlpwd yyy
sqldb myAlarm
sqltable syslog
#
# Choose what to do with notification messages
#
sendEmail 0
writeSql 1
Alberto Crescente, servizio calcolo
Farm BaBar – WebDB Features
➢
Clients and Servers Archive Informations
Machine Name
•CPU
•RAM
•etc.
•
➢
Assistance Calls
➢
Event Log Viewer
➢
Machines Location
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – WebDB Data Flow
MySQL
PHP Script
Extreme Cluster
Administration Toolkit
Configuration Files
WebDB to xCAT
Farmcfg
PHP
HTML
WebDB
SysAlarm
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – WebDB Home Page
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – WebDB Machine Details
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – WebDB Machine Details
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – WebDB Farms Structure
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – WebDB Rack Machines Location
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – WebDB Rack Machines Location
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – WebDB Assistence Calls
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – WebDB Syslog
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – PerfMC Features
It allows the control of all the devices
that interface with SNMP
➢It allows the execution of simultaneous requests
➢It allows a high sampling frequency
➢It doesn't affect the cluster performances
➢It permits to build a user interface thanks
to the integrated web server and PHP support.
➢
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – PerfMC Data Flow
MySQL
Language C
Farmcfg
HTML/XML
PerfMC
web
server
RRD
SNMP
Workshop CCR, Paestum Giugno 2003
PerfMC
poller
Alberto Crescente, servizio calcolo
Farm BaBar – PerfMC
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – PerfMC
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – PerfMC
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – PerfMC
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – PerfMC
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – PerfMC New Interface
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – PerfMC New Interface
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – PerfMC New Interface
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – PerfMC Future Implementations
Integration of a system alarm
that notifies malfunctions
through SMS and e-mail
➢Integration with HTTPS protocol
to crypt informations
➢Integration with the
monitoring environmental parameters
➢
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – Assistance Calls
Failures in 1 year
Failures Number Component's Number
Component
39
640
6,09 IDE Disk
2
140
1,43 SCSI Disk
10
192
5,21 Mainboard
5
300
1,67 Power Supply
4
384
1,04 CPU
4
576
0,69 RAM
10
TOTALE
% Failures
Other
74
0,15% CPU time/year lost for hardware failures
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – Assistance Calls
Failures Number
39
2
10
5
4
4
10
TOTALE
74
Component
IDE Disk
SCSI Disk
Mainboard
Power Supply
CPU
RAM
Other
% on Total
52,70
2,70
13,51
6,76
5,41
5,41
13,51
% on Total
Other; 13,51
RAM; 5,41
CPU; 5,41
Power Supply; 6,76
IDE Disk; 52,70
Mainboard; 13,51
SCSI Disk; 2,70
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – Assistance Calls
HD IDE Month Failures
9
8
7
6
5
4
3
2
1
0
08/2002
09/2002
10/2002
11/2002
12/2002
01/2003
02/2003
03/2003
04/2003
05/2003
New Machines Installed
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo
Farm BaBar – References
•BaBar Farm Home Page
http://bbrweb.pd.infn.it:5211/
•WebDB
http://bbrweb.pd.infn.it:5211/farm/farmcfg
•PerfMC
http://www.pd.infn.it/~marzolla/publications/pdp2003.pdf
•Farm System Administration
http://bbrweb.pd.infn.it:5211/farm/sysadmin
E-mail: [email protected]
Workshop CCR, Paestum Giugno 2003
Alberto Crescente, servizio calcolo