Maggio 2007
ATLAS HLT/DAQ
Progress report
V. Vercesi on behalf of the ATLAS Italia HLT/DAQ Group
A. Negri (Irvine, Pavia) Coordinatore Event Filter Dataflow
A. Nisati (Roma1) TDAQ Institute Board chair e Coordinatore Muon Slice
F. Parodi (Genova) Coordinatore b-tagging PESA
V. Vercesi Coordinatore TAPM (TriggerAlgorithmsPerformance&Menus)
Attività italiane
• Trigger di Livello-1 muoni barrel (Napoli, Roma1, Roma2)
• Trigger di Livello-2 muoni (Pisa, Roma1)
• Trigger di Livello-2 pixel (Genova)
• Event Filter Dataflow (LNF, Pavia)
• Selection software steering (Genova)
• Event Filter Muoni (Lecce, Napoli, Pavia, Roma1)
• DAQ (LNF, Pavia, Roma1)
• DCS (Napoli, Roma1, Roma2)
• Monitoring (Cosenza, Napoli, Pavia, Pisa)
• Online Data Quality Muon Slice (Lecce)
• Responsabilità integrazione online Muon Slice (Pavia, Roma1)
• HLT/DAQ system commissioning and exploitation (Everybody)
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
2
TDAQ in ATLAS site
Outputs
(SFOs)
DataFlow
Manager
Event data
pulled:
partial events
@ ≤ 100 kHz,
full events
@ ~ 3.5 kHz
Regions Of Interest
LVL2
Supervisor
RoI
Builder
~1600
Event
Filter
(EF)
~100
Event
Builder
~ 500
SubFarm
Inputs
LVL2
farm
(SFIs)
Secondlevel
trigger
SDX1
pROS
Network
switches
stores
LVL2
output
Network switches
~150
PCs
Read-Out
Subsystems
(ROSs)
Gigabit Ethernet
5
Local
Event rate
Storage
~ 200 Hz
SubFarm
Event data requests
Delete commands
Data
Storage
multi-CPU nodes
Requested event data
CERN
computer
centre
USA15
Data of events accepted
1600
by first-level trigger
ReadVME Dedicated links
Out
Links Read-
Out
Drivers
(RODs)
Timing Trigger Control (TTC)
Firstlevel
trigger
UX15
ATLAS
detector
Event data pushed @ ≤ 100 kHz,
1600 fragments of ~ 1 kByte each
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
3
Executive summary
Moltissime attività nel corso degli ultimi mesi, forte
collaborazione con i sotto-rivelatori, intensa fase
preparatoria
Dal read-out, al monitoring, ai cosmic runs, al
commissioning, all’event building, al deployment delle slice
di selezione, la comunità TDAQ è in continua accelerazione
Large Scale Tests, Technical Runs, Cosmics Data Taking e
Integration Tests gettano le basi del sistema che dovrà essere
pronto alla fine dell’anno
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
4
Hardware 2007
 Moreover



Fully equipped RoI Builder (+ 1 at bldg 32)
All network switches & cables needed in 2007
IPMI control
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
5
Hardware 2007 notes
ROD-ROS readout
links
• LAr, Tile, L1Calo, MDT : installed and ~ tested
• SCT : installed, not yet connected
• Pix, TRT, CSC, TGC, RPC : not yet installed
130 [quad-core dual- • Effort required, schedule available
cpu] boxes installed and • Need contribution from all Institutes involved
• Dynamic allocation to LVL2 or EF farms
to be commissioned
~33% ATLAS EB
capability (#SFI)
100% ATLAS data
recording capability
(#SFO)
CSN1 Maggio 2007
• Should cover all needs for partitioning during
commissioning
• Full throughput to CDR for calib/align/…
Valerio Vercesi - INFN Pavia
6
Infrastructure in SDX
Shaft PX15
Cable Tray USA15
Racks enter via this door
Landing
350daN/m2
door
Row 4
Row 2
70-90 cm
EF
EF
EF
EF
EF
AIR FLOW
EF
EF
EF
EF
17 Racks
FRONTs
EF
EF
EF
3
EF
5
6
63 cm (taking into account the cooler)
EF'
16 Racks
FRONTs
7
8
The lower limit is due to the power
distribution boxes and will affect
most of the racks.
LVL2' L2SV' Switch'
18 Racks
EF
e-box
4
100 cm
EF
BACKs
water pipes
Rack Number
EF
9
11
12
13
EFIO' Online'
80-100 cm
e-box
e-box
10
The lower distance is due to structural
beams or ventillation flaps and and
applies to ~5 racks.
door
Row 6
Ventilation duct (on the flor)
Landing
350daN/m2
doo
14
15
16
Landing
350daN/m2
SDX Level 1 Layout
17
18
19
20
Crinoline Ladder
Select which year
2005
2006
2007
2008
2009
SDX Level 2 Layout
Landing
350daN/m2
Racks enter via this door
Row 6
Row 4
Row 2
Ventilation duct (on the flor)
door
EF/L2 EF/L2
EF/L2 EF/L2 EF/L2 EF/L2 EF/L2 EF/L2 NWbb NWbb T DCS cDCS DSS
2 Racks
AIR FLOW
BACKs
EF/L2
13 Racks
PP
63 cm (taking into account the cooler)
FRONTs 18 Racks
EF/L2 EF/L2 SFO
3
PP
LVL2 LVL2 LVL2 LVL2 LVL2 LVL2 LVL2 LVL2 LVL2
water pipes
Rack Number
74-90 cm
FRONT
SFO
DC
SFI Backend
SFI
switch
DC switch
rack
DC rack
switch
Online
rack
switchOnline
rack Online Online Online
17 Racks
e-box
4
CSN1 Maggio 2007
5
6
100 cm
80-100 cm
e-box
e-box
7
8
9
10
11
12
13
Flap Trap
14
15
16
17
18
Valerio Vercesi - INFN Pavia
19
20
7
Read Out System (USA15)
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
8
SFI, DFM and friends
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
9
Installing HLT racks…
Ancient monuments were built without cranes…
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
10
HLT racks
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
11
Next steps
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
12
Event Builder
Local Disk  CASTOR
Integrated and commissioned the first 1/3 of the
ATLAS final Event Builder
Event
Builder
(SFIs)
SFI
• Assembles full events and serves Event Filter, or Local
Disk (and then pulled to CASTOR)
• In parallel, serves Events to Monitoring Tasks in spying
mode (e.g., to be used to Event Display)
DFM
DFM
• Triggers Event Building using either Internal trigger, or
L2Decision messages
Technical run
14-16 February
• 32 SFIs, writing to local disk, 10 had their data files
pulled to CASTOR (100 SFI in full ATLAS)
• 5 DFMs, exercised triggering it without a LVL2 system
• 60 ROSs, producing fake events either at the ROS PC
level, or the ROBIN level
Perfect scaling
of EB rate with
# of SFIs
• #SFIs
• 29
• 20
• 10
CSN1 Maggio 2007
EB Rate
7.3 kHz
5.0 kHz
2.5 kHz
(~250 Hz per SFI)
Valerio Vercesi - INFN Pavia
Switches
Read-Out
Subsystems
(ROSs)
13
Monitoring
DQMF (MWG):
online-offline common
project
New
implementation of
OHP
Reference
histogram handling
Data Quality Desk
CSN1 Maggio 2007
•
•
•
•
Core sw depends on ROOT only
At present, input can be any ROOT object (only)
Online I/O interface implemented
Next: configure histograms & algorithms
• Target: network & CPU load optimization
• Graphical configuration to be improved
• To come: plugin manager
• Based on COOL
• Implemented within ATHENA
• 5 millions histograms...
•
•
•
•
OHP: standard TileCal, LAr, Muon histograms
Online DQMF: prototype version with basic display
Offline monitoring: ROOT display of ROOT histo files
Event Display: Atlantis in loop (demo) mode taking events
from EF server (or file)
Valerio Vercesi - INFN Pavia
14
Counting Room Desks
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
15
Large Scale Tests
November – December 2006
LXBATCH facility, farm size up to 1000
nodes
Configure HLT infrastructure and
algorithms, both Level 2 and Event Filter
Exercise DB access for configuration and
metadata
Run HLT with a set of trigger slices and
algorithms on a realistic mix of MC events
Configure (emulated) ROD crates
Infrastructure timings, monitoring
CSN1 Maggio 2007
Data volumes
used for HLT
configuration:
Valerio Vercesi - INFN Pavia
• COOL database 38 MB
• Geometry database 23 MB (SQLite
file)
• Trigger configuration database 1 MB
• POOL files 31 MB including the
catalogues
• Magnetic field file 5 MB (packed)
16
Technical run March 2007
6000 events (mixed physics processes LVL1 pre-selected)
• They contain about:
• ~55% J0, J1, J2, J3, J4, J5, J6 jet-jet samples
• ~15% Wee
• ~13% W
• ~3% Wtauhad
• ~2% Zee
• ~7% Z
• ~5% JF17 (dijets filtered to be very electromagnetic)
Execute the full DAQ/HLT chain including LVL2 trigger, Event Builder and Event
Filter, write accepted events written to disk
Emulate LVL1 trigger; use final Region of Interest Builder (ROIB)
Test performance in terms of the various resources
• CPU usage, network usage, configuration time, algorithm execution times
Pre-series test-bed used from the ATLAS Control Room
Good participation from DAQ, HLT, TAPM developers
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
17
It’s fun…
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
18
Online trigger menu
LVL1
L2
Egamma
Jet
Muon
Tau
Bphysics
MET
EM01
—7 GeV thr—
JT15
—15 GeV thr—
MU06
—6 GeV thr—
HA08
—8 GeV thr—
JT04
—4 GeV thr—
MU06
HA05
—5 GeV thr—
jet20
signature
1. Sum Et of cells
in cone
mu6
signature
1. Find track in
muon
2. Find tracks in
ID
3. Match tracks
tau10
signature
1. Find had cluster
2. Find tracks
3. Match
e10 signature
1. Find cluster
2. Find track
3. Match
g10 signature
1. Find cluster
EF
e10 signature
1. Find cluster
2. Find track
3. Match
g10 signature
1. Find cluster
CSN1 Maggio 2007
jet20
signature
1. Find clusters
2. Sum Et of
clusters in cone
mu6
Signature
1. Find track in
muon
tau10 signature
1. Find had cluster
2. Find tracks
3. Match
Valerio Vercesi - INFN Pavia
BsDsPhiPi
Bmu6mu6
BsDsPhiPi
dummy
algorithm
MET10
19
Timing: LVL2 Calo
Photon run 2373
mean 6.6 ms/RoI
Egamma run 2309
mean 7.4 ms
Electron run 2515
mean 8.0 ms
Combined run 2368
mean 6.9 ms
20
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
Online and offline: egamma EF
AthenaPT
Online
TrigEFTrackHypo
TrigEFTrackHypo
Pt=7.0  8.25
TrigEgammaRec
Pt=6.9  8.1
TrigEgammaRec
E=43.18  43.70
CSN1 Maggio 2007
E=43.63  42.38
Valerio Vercesi - INFN Pavia
21
More plots
E/p
SiTrack pT
MuFast residuals
1/pT
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
22
Technical run next week
Morning
9am-1pm
Afternoon
1pm-5pm
Evening
5pm-8pm
Long-Run
M
T
W
Th
F
Infrastructure tests
LVL2 + EB tests
EF tests
other HLT slices
Database tests
ROIB
integration+Tests
EB tests
combined system
other HLT slices
/ cosmic
Database tests
LVL2 tests
EB+EF tests
combined system
reserved
/ cosmic
cosmic slice
run with SFI
run with SFO,
writing to disk
run with SFO,
writing to castor
BBQ
run with SFO,
writing to castor
23
‘Final’ Hardware
• ROIB (+ LVL1 emulator), 153 ROS, 30 SFI, 5 DFM, 3-4 HLT racks (up to 130 dual quad-core machines), …
Basically same software setup as March TR
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
23
Physics Trigger Menus
Detailed information available at
• https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerMenuVersions
CSC-06
• Used for production with offline release 12.0.6.1
• triggers for 1031 cm-2s-1
• triggers for 1033 cm-2s-1
• triggers for optimization studies and validation
CSC-06-900GeV
• Used exclusively for B-physics production in 12.0.6.1
• Contains lower muon threshold values and specific B-physics triggers
Physics menus
• Current emphasis mostly (but not entirely) on menu for 1031cm-2s-1
• Physics analysis studies starting to use trigger information available in Release 12 for studies of physics
menus.
• CSC production provides required data and tools to go through first iteration
• Important to develop close collaboration between trigger and physics communities
Trigger Rate Tool
• Develop toolkit to simplify trigger rate studies (collection of scripts and useful plotting tools)
• Provide “realistic” trigger rates (including overlap, independent fraction, prescales) of a full trigger menu
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
24
Trigger Aware Analysis
Trigger information available (result and features) for all the samples produced now
Many Trigger aware analyses presented at last Trigger&Physics Week based on the new CSC samples across
many groups (SM, Top, SUSY, Higgs, Exotics, B-Physics,..…..)
• Trigger efficiencies and distributions since 12.0.6 reconstructed AODs
Example from the Higgs group: VBF H→bb and bbH
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
25
CSN1 Maggio 2007
Valerio Vercesi - INFN Pavia
26