CSN1 Settembre 2005
ATLAS HLT/DAQ
Stato e prospettive
Valerio Vercesi
Outline
 Pre-series


Status in USA15/SDX1
Commissioning and exploitation
 Large Scale Test


Activities, experiences
Lessons learnt
 Activities


Monitoring, ROD Crate DAQ
Algorithms development and deployment
 Finance


Accounting
2006 requests
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
2
 S. Falciano (Roma1) Coordinatore Commissioning HLT
 A. Negri (Pavia) Coordinatore Event Filter Dataflow
 A. Nisati (Roma1) TDAQ Institute Board chair e Coordinatore
Muon Slice PESA
 F. Parodi (Genova) Coordinatore b-tagging PESA
 V. Vercesi (Pavia) Deputy HLT leader e Coordinatore PESA
(Physics and Event Selection Architecture)
 Attività italiane









Trigger di Livello-1 muoni barrel (Napoli, Roma1, Roma2)
Trigger di Livello-2 muoni (Pisa, Roma1)
Trigger di Livello-2 pixel (Genova)
Event Filter Dataflow (LNF, Pavia)
Selection software steering (Genova)
Event Filter Muoni (Lecce, Napoli, Pavia, Roma1)
DAQ (LNF, Pavia, Roma1)
Monitoring (Cosenza, Napoli, Pavia, Pisa)
Pre-series commissioning (LNF, Pavia, Roma1)
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
3
ATLAS TDAQ system
(
)
≅
TDAQ
Latency
Rates
Muon
40 MHz
Calo
LVL1
~75 kHz
~10 ms
RoI
LVL2
LVL2
farm
ROD
ROB
every million
Inner
Pipeline
Memories
~2 ms
1 selected event
ROD
Readout
ROD Drivers
ROB
Readout
ROB Buffers
~1600
LEVEL-1 TRIGGER
• Hardware-Based
• Coarse granularity from
calorimeter & muon systems
LEVEL-2 TRIGGER
• Regions-of-Interest “seeds”
• Full granularity for all
subdetector systems
• Fast Rejection “steering”
Event builder network
~2 kHz
EF farm
~1000 CPUs
~1s
EVENT FILTER
• Possibly “seeded” by Level 2
• Full event access
• Algorithms inherited by offline
EF
~200 Hz
Storage: ~ 300 MB/s
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
High-Level Trigger
4
TDAQ
 Trigger e Data Acquisition hanno da sempre in fase di
commissioning un doppio ruolo


Come “server” per il commissioning dei rivelatori
Come “client” per utilizzare le informazioni realistiche
dell’esperimento per i propri studi di funzionalità e
performance

La situazione si è già presentata durante il Combined Testbeam 2004
 Il TDAQ di ATLAS è un progetto in piena evoluzione in
cui development/commissioning/exploitation sono ancora
fasi molto miscelate



Presentazione di risultati e indicazione delle prospettive
Maggiore enfasi alle componenti con forte partecipazione
italiana
Descrizione del piano di commissioning generale
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
5
Pre-series design
“Module-0” of final system
8 racks (~10% of final dataflow)
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
6
Pre-series reality
ROS rack
LVL2 rack
6 racks SDX1
Switch rack
CSN1 Settembre 2005
EF rack
Online rack
V. Vercesi - INFN Pavia
7
Commissioning and exploitation
 Fully functional, small scale, version of the complete
HLT/DAQ

Equivalent to a detector’s ‘module 0’
 Purpose and scope of the pre-series system

Pre-commissioning phase



Commissioning phase


To validate the complete, integrated, HLT/DAQ functionality
To validate the infrastructure, needed by HLT/DAQ, at point-1
To validate a component (e.g. a ROS) or a deliverable (e.g. a Level-2
rack) prior to its installation and commissioning
TDAQ post-commissioning development system


Validate new components (e.g. their functionality when integrated into a
fully functional system)
Validate new software elements or software releases before moving
them to the experiment
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
8
Pre-Series Commissioning
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
9
Commissioning LVL2+ROS
First measurements
with full LVL2 rack
feeded by ROS data
Using separate Control
and Data networks
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
10
Commissioning EF
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
11
 Pre-series


Status in USA15/SDX1
Commissioning and exploitation
 Large Scale Test


Activities, experiences
Lessons learnt
 Activities


Monitoring, ROD Crate DAQ
Algorithms development and deployment
 Finance


Accounting
2006 requests
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
12
Large Scale Tests
 Pre-serie work will help understanding the TDAQ system in terms
of functionality

Forms the basis for future deployments/exploitations
 Complexity of ATLAS TDAQ system arises also from the size of
bulk components involved

Topology of communications, size of LVL2/EF farms, software, …
 Test scalability of HLT system using presently available large
installations

Understand issues like configuration, startup time, communication, control,
error reporting, …
 UCB/TRIUMF WestGrid Cluster (http://www.westgrid.ca)

60 racks x 14 nodes = 840 Dual-CPU nodes
(3 GHz CPUs / 2-4 GB RAM)
 CERN LXSHARE Cluster (http://batch.web.cern.ch/batch)

Up to ~700 nodes (various flavours)
 Reference page for all tests

http://atlas-tdaq-large-scale-tests.web.cern.ch
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
13
State transitions
USR_RUNNING_TIME
(default is 30 s)
RUNNING
Luke Warm Start
Luke Warm Stop
CONFIGURED
Configure
Unconfigure
INITIAL
Boot
Cold Start
Cold Stop
Shutdown
ABSENT
Setup
configure: load
configure
CSN1 Settembre 2005
start: prepareForRun
startTrigger
Close
stop: stopTrigger
unconfigure: unconfigure
stopFrontEnd
unload
stopDataCollection
stopEventFilter
stopRecording
V. Vercesi - INFN Pavia
15
LVL2 transition times
 State transistion timing quite acceptable

No significant differences between 2 and 3 tier Run Control
Time (secs)
2 Tier Run Control
80
70
60
50
40
30
20
10
0
0
5
10
15
20
25
setup
boot
conf
start
wait
stop
unconf
shutdown
close
Configuration Id (# L2PU nodes 8 -> 256)
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
16
EF results @ LST
 Effect of realistic algorithm:
TrigMoore vs HelloWorld
config. / total times, s
EF TrigMoore with Oracle/MySQL
800
config Oracle
700
config MySQL
600
Total test Oracle


Total test MySQL
500
400

300

200
100
 MySQL vs Oracle DB in TrigMoore
0
0
100
200
300
400
500
600
700

Number of processes (1EFD+2PTs)

Timings HelloWorld / TrigMoore
250
config HelloWorld
200
config TrigMoore MySQL


time, s
backend setup HelloWorld
150
backend setup TrigMoore MySQL
100

50
0
0
100
200
300
400
500
600
Number of processes (1EFD+2PTs)
CSN1 Settembre 2005
EF standalone 1EFD+2PTs / node
up to 200 nodes
MySQL as geometry DB used
significant slow down due to
access/reading geom. DB
700
EF standalone, 1EFD+2PTs / node
Oracle DB – up to 160 nodes
MySQL DB – up to 200 hosts
MySQL works faster at “small
scales”, while Oracle looks better
at higher scales - to be
investigated more
not able to on higher than 200
nodes with any of both partitions –
to be investigate further (do we
need to replicated DBs ?)
V. Vercesi - INFN Pavia
17
 Pre-series


Status in USA15/SDX1
Commissioning and exploitation
 Large Scale Test


Activities, experiences
Lessons learnt
 Activities


Monitoring, ROD Crate DAQ
Algorithms development and deployment
 Finance


Accounting
2006 requests
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
18
Cosmics Tile setup
MobiDAQ (Mobile DAQ):
read out of 8 drawers in
the pit with temporary
RODemu but real TDAQ
(tdaq-01-02-00), tests of
electronics, cosmic
muons runs
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
19
GNAM Monitoring chain
 Framework per monitoring on-line a basso livello

Core: trasporto di eventi, istogrammi e comandi
Plugin dinamici: decodifica e istogrammazione
Possibilita’ di correlazione fra diversi rivelatori

Comandi asincroni (reset, rebin, update)


 Status



Incluso in TDAQ da aprile
Utilizza i servizi disponibili
per il monitoring
Validato al CTB04; in uso in
alcuni siti di commissioning
GNAM al commissioning





Acquisizione dati nell’ambito del software TDAQ
Commissioning di MDT usa GNAM per monitoring online ed analisi dati
Sampling completo a livello di ROS [~2 KHz per noise test, ~200 Hz per pulser]
Integrazione di librerie: completato per MDT, in via di sviluppo per RPC
Stato del monitoring MDT:


Richiesta minimale di informazioni allo shifter (nomi camere)
Output: file di istogrammi e file di testo con risultati di analisi dati per ciascuna camera e per
ciascun run
 In sviluppo:



ottimizzazione dell’analisi dati on-line
installazione del presenter per la visualizzazione degli istogrammi
on-line event display
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
21
PMP Presenter
 Visualizzazione asincrona on-line di istogrammi



Interattivo (reset, rebin, zoom, fitting, ecc...)
Operazioni grafiche su istogrammi (ROOT canvas)
Grafica configurabile
 Status




Incluso in TDAQ da settembre
Utilizza i servizi disponibili
per il monitoring
Pienamente funzionale
al CTB04
Riprogettato per nuove
funzionalità e maggiore
scalabilità
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
22
GNAM&PMP: sviluppi futuri
 GNAM

Completare l’integrazione nel framework del TDAQ



Supporto per la generazione di allarmi automatici


Livelli di severità, routing
Verifica delle prestazioni e delle risorse necessarie


Configurazione di Core e librerie dal database del TDAQ
Messaggistica e gestione degli errori software
CPU, Memoria, Banda
Studio della scalabilità
 PMP

Completare la nuova versione



Minimizzazione del traffico di rete
Adattamento della IGUI alla nuova struttura
Generazione allarmi

Plugin di analisi degli istogrammi
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
23
ROD Crate DAQ
 RCD is used as interface with the RODs
for



Control
Configuration
Monitoring



Statistics
Event sampling
Data readout (through VME)
 User guide for detectors developers
available
 Validation system in Bld. 40
 DAQ Commissioning – Phase 1:


The ROD Emulator system will be
used in order to validate all common
RCD software and infrastructure
After adding and validating the
detector sw and hw specific items,
multi crate event building will be used
in the absence of the full DAQ chain
CSN1 Settembre 2005
R
C
C
M C
E O
Config & Control
M
R
O
R B
Data readout
Y O
R
O
S
ROD
Emulator
VMEbus memory +
CORBO
V. Vercesi - INFN Pavia
=
Memory +
Registers +
Interrupt
capability
R
E
B
Event Fragments
24
RCD exploitation
 Experience from Combined Testbeam extremely useful


Recall almost all detector used it in the CTB
Successful workshop to put forward new requirements
 As a consequence, several improvements during last months





Configurable interrupt handling
Simplified user interface to access ordered event fragments
Data driven event building for multicrate acquisition in the commissioning
phase
Simplified ROD emulation
Hardware trigger distribution
 All sub-detector commissioning (but LAr…) sites use RCD

MDT and RPC on the forefront

BB5 integration, Point 1 with MROD, Lab testing with ROD emulators
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
25
Algorithms: Muon slice
 LVL1 simulation is the fundamental input for the measurement of
the full muon vertical slice performance
 LVL2 and EF Muon algorithms have been extensively tested on
data simulated in ATLAS

Rome Physics Workshop: June 2005
 LVL2: Fast


Confirm the LVL1 trigger with a more precise PT estimation within a
Region of Interest (RoI)
Global pattern recognition, track fit, fast PT estimate via Look Up Table
(LUT) with no use of time consuming fit methods
 Event Filter: TrigMoore



Based on offline reconstruction algorithm Moore
Can run seeded (reconstruction starting from RoI of previous levels)
Precise PT determination
 General goal is now to achieve more realistic estimate of trigger
selections and corresponding rates

Real geometry, configuration and conditions database usage, …
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
26
LVL1 Coincidence Windows
 Athena release 10.0.4
 Low pT 6 GeV Threshold
Efficiency curve
CSN1 Settembre 2005
Low-pT Inefficiency map
V. Vercesi - INFN Pavia
27
MuFast endcap extension
 Early stages of project
 Endcap differs from Barrel

-2.16
M and O station are outside B field
Inhomogeneous B field – bending is local
 Algorithm
TGC
-2.18
eta

Event display #17
TGC SP
-2.2
MDT EM
-2.22
MDT EO
-2.24
-23000
-21000
-19000
-17000
-15000
-13000
ROI
Z

Pattern recognition and fit in TGC → position and slope in EM

Extrapolate segment into MDT EM/EO → Roads in EM/EO, find hits, fit
Forward
Endcap
PT
0
10
20
30
40
0.00E+00
-2.00E-06
Slope
-4.00E-06
-6.00E-06
-8.00E-06
-1.00E-05
-1.20E-05
-1.40E-05
 Next


pattern recognition and fit in MDT as in mFast – not done yet
Extrapolation into EI and LUT
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
29
MuFast: MDT miscalibration
 Commissioning the algorithms: realistic approach to data handling
 The plot shows the muFast resolution for two different scenarios:


the correct MDT r-t function is used, red points
a systematic shift of + 0.2 mm is added to the radius returned by the correct
r-t function, blue points
+10% degradation @ 50 GeV,
to be compared with a + 5%
expected by a naive calculation
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
30
TrigMoore: MDT miscalibration
 Single muons (with pT = 6, 10, 20, 40, 100 GeV/c, produced for the Rome Initial
Layout) have been reconstructed in two different scenarios


Using the correct MDT r-t relation function (red squares in pictures)
Applying a systematic +0.2 mm shift on the radius obtained with the correct MDT
r-t function (blue circles in pictures)
The relative degradation in (pT) is
+5% for muons with a 6 GeV/c
transverse momentum, increases to
+13% around 50 GeV/c
pT resolution (Moore)
CSN1 Settembre 2005
This MDT miscalibration leaves almost
unaffected  resolutions.
V. Vercesi - INFN Pavia
32
LVL2 tracking: SiTrack
 Preliminary results obtained on DC1 b-jet samples at initial luminosity
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
33
B-tagging @ LVL2
 Results obtained with the “standard” SiTrack algorithm on DC1 data
 b-tagging: likelihood ratio using transverse and longitudinal impact parameters
 Upgraded version to be tested soon: should improve both efficiency and track
parameters resolution and hence significantly improve the b-tagging performance
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
34
Algorithms steering
EM60Hy
EM25Hy
EM60
EM25
T2Calo
T2Calo
Cluster60
Cluster25
eg60Hy
eg25Hy
eg60
eg25
EMtrackHard
EMtrackSoft
g60Hy
iSg60
gIsol60
l2g60i
e60Hy
iSe60
eIsol60
l2e60i
g25Hy
iSg25
gIsol25
l2g25i
e25Hy
iSe25
eIsol25
l2e25i
g20Hy
iSg20
gIsol20
l2g20i
e20Hy
iSe20
eIsol20
l2e20i
g15Hy
iSg15
eIsol15
l2e15i
e15Hy
iSe15
eIsol15
l2e15i
TrackHardEM
TrackSoft25EM
EMROI
EM20Hy
EM15Hy
CSN1 Settembre 2005
EM20
EM15
T2Calo
T2Calo
Cluster20
Cluster15
eg20Hy
eg15Hy
eg20
eg15
EMtrackSoft
EMtrackSoft
TrackSoft02EM
TrackSoft15EM
V. Vercesi - INFN Pavia
35
ATLAS Commissioning Phases
 Commissioning means bringing ATLAS systems from
“just installed” to “operational”. It is broken in 4 phases




Subsystem standalone commissioning
Integrate subsystems into full detector
Cosmic rays, recording data, analyze/understand, distribute to
remote sites
Single beam, first collisions, increasing rates, etc…
 A consistent part of commissioning activities will be done
during the installation itself
 Phases will overlap since different systems may be at
different development levels


For the barrel calorimeter commissioning will start soon
Tile calorimeter is already taking data
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
36
HLT Commissioning
 Commissioning is a set of activities which spans the time interval from the
installation of the HLT racks and nodes …


A rack is the elementary unit for commissioning
OS, Dataflow and Online software are installed
 ... to the phase when the HLT is filtering physics data and recording them



HLT selection algorithms are installed and running stably
The complete trigger menu (at least for early physics) is configured
The trigger selection efficiencies and background rejection rates are understood and
can serve as input for physics measurements
 Phase-1 Commissioning definition is the most urgent

Heavily use the Pre-series to exercise the procedures for installation and
commissioning
 Important steps will cover the integration of detectors into full system


Involve operations that have a very strong coupling with the offline commissioning
activities
Development of specific algorithms looking at simple data decoding (cabling,…)
 Final commissioning phases extend far beyond the data-taking startup (interface
with run coordinator team)
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
37
Cosmic muons in ATLAS
Concrete
Air
Surface building
PX14/16 shielding
(2.5 g/cm3)
PX16
PX14
(12.6 m Inner Ø )
(18.0 m Inner Ø)
ATLAS
Rock ~ Silicon
Geant Simulation
Initial detector
600m x 600m x
200m deep
(2.33 g/cm3)
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
38
Overall plan
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
39
Outlook
 A lot of work during this year, system entering phase of complete
deployment



Purchase plan proceeding as scheduled, with some minor delays
(Wo)manpower situation not always healthy
More help and support welcome
 Three big tasks awaiting us in the next months

Commissioning the pre-series and extract a coherent and complete set of
system performance measurements


On-line trigger selections evaluation (rates, efficiencies, physics
coverage,…) evolving towards more realistic approach


Based on previous experiences and on already established partial results
Calibration, geometry “as installed”, mis-alignment, error handling, complete
trigger menus, physics analysis based on trigger objects
Prepare for cosmic run next year

“Cosmic” slices and trigger menu (Tile, LVL1, “digital” LVL2)
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
40
 Pre-series


Status in USA15/SDX1
Commissioning and exploitation
 Large Scale Test


Activities, experiences
Lessons learnt
 Activities


Monitoring, ROD Crate DAQ
Algorithms development and deployment
 Finance


Accounting
2006 requests
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
41
Accounting
 Contributo INFN alla Pre-serie



Read-Out System: 51 kCHF (ROS Racks)
Online Computing System: 40 kCHF (Monitoring, Operations)
Online Network System: 44 kCHF (Switches, FileServer)


Inviati al CERN a Dicembre 2004
VV riceve in copia tutte le fatturazioni dei singoli acquisti ed un sommario
mensile dello stato finanziario
 Contributo CORE 2005

Online Computing System: 45 kCHF (Monitoring, Operations)



Inviati al CERN a Maggio 2005
Già acquisiti due file server
Read-Out System: 275 kCHF (ROS Racks)


Questo acquisto si espleta secondo una gara e non con un semplice market
survey o price inquiry come fino ad ora avvenuto
Richiesta alla Giunta l’autorizzazione per partecipare alla gara


Grazie a Speranza che si è prodigata per espletare le pratiche necessarie
Il CERN preferisce gestire la gara su un periodo di due anni


CSN1 Settembre 2005
Omogeneità dei componenti vs miglioramento delle prestazioni
Si sommano i 275 kCHF del 2006
V. Vercesi - INFN Pavia
42
Cost Profile (kCHF)
2004
2005
2006
2007
2008
2009
Total
140
0
0
0
0
0
140
Detector R/O
0
275
275
0
0
0
550
LVL2 Proc
0
0
65
195
230
160
650
Event Builder
0
0
50
50
110
70
280
Event Filter
0
0
170
180
570
380
1300
Online
0
45
135
0
0
0
180
Infrastructure
0
0
80
80
20
20
200
INFN Total
140
320
775
505
930
630
3300
TDR Total
1048
3357
4087
4544
7522
4543
25101
INFN Percentage(%)
13.4
9.5
19.0
11.1
12.4
13.9
13.1
Pre-series
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
43
Missioni Estere
 LNF

Commissioning HLT/DAQ Pre-serie e pit: 6 m.u.

Ferrer, Kordas (+ Miscetti, Giovannella)
 Pavia



VV coordinatore PESA e duputy HLT: 1 m.u.
Negri A. responsabile Event Filter: 1 m.u.
Scannicchio D. commissioning HLT: 2 m.u.
 Roma1



Speranza responsabile commissioning HLT: 2 m.u.
Leandro chair IB, coordinatore slice mu: 1 m.u.
ROD crate DAQ e HLT/DAQ muoni : 4 m.u.

Pasqualucci, Di Mattia, …
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
44
Milestones
 30/06/2005

TDAQ - Installazione, test e uso della "Pre-serie"
(~ 10% TDAQ slice)

“ragionevolmente” raggiunta: ritardi accumulati soprattutto sugli acquisti delle
componenti
 24/12/2005

TDAQ - Installazione e test dei ROS di Pixel, LAr, Tile, Muon
(interfacciamento al ROD Crate e integrazione nel DAQ)

Parte del piano di commissioning in esecuzione: piccola dipendenza dalla data
di consegna dei ROS
 30/04/2006

Completamento dei test sulla pre-serie e definizione delle funzionalità per il
supporto al commissioning TDAQ
 31/08/2006

Commissioning delle slice di ROS dei rivelatori utilizzando le funzionalità
della pre-serie (modulo-0 del sistema finale)
 31/12/2006

Presa dati integrata dei rivelatori nel pozzo con raggi cosmici
CSN1 Settembre 2005
V. Vercesi - INFN Pavia
45