The Muon System
Electronics Upgrade
Alessandro Cardini
on behalf of
The LHCb Muon Upgrade Group
Cagliari, Ferrara, Firenze, Frascati, PNPI, Roma 1, Roma2
The Muon System Upgrade in brief
• Phase 1: get ready for the 40 MHz readout
– nODE (with nSYNC ASIC) for an efficient detector readout by TELL40
– L0muon trigger implemented on TELL40
– nPDM and nSB for an efficient chamber pulsing and control using GBT
through GBT-SCA
– Ready @ LS2
• Phase 2: new high readout granularity detectors for highluminosity regions equipped with a new highly-integrated
FEE ASIC
– Baseline planning: prototypes ready @LS2, then construction, to be
installed @ LS3
– I will not discuss this part today
13 June 2013
A. Cardini for the Muon Upgrade Group
2
Converging to a viable solution
• A lot of work has been done in the last month mainly pushed by the time schedule for the upgrade
approval within INFN
• Weekly meeting within the “Muon Electronics
Upgrade Group”
• Meeting within the “Tell40 Firmware Working Group”
- where many details were discussed
• Meetings with L0muon Marseille colleagues to
discuss the impact of the new muon system readout
architecture on the LLT
13 June 2013
A. Cardini for the Muon Upgrade Group
3
Design considerations / 1
• HIT and TDC info on same link?
– Pros
• Direct correspondence between TDC data and logical channel
– Cons
• The number of link to the TRIG40 will increase (need to send TDC
info on every link)
• The latency due to the data generation will increase: HITS are
NZS (low latency), but for TDC data a certain form of ZS is
needed, and this will require a certain elaboration time  might
not be compatible with LLT latency
 use separate links
13 June 2013
A. Cardini for the Muon Upgrade Group
4
Design considerations / 2
• Use GBT WideBus (112 bits) or not (80 bits)?
– Use of GBT WideBus seems much more efficient: +40%
information can be transmitted per frame  better use of
the bandwidth and important reduction of the number of
links
– Up to know we have been using a system with 8B/10B
encoding (no error correction capabilities and only a
minimal capabilities of error detection) and we did not see
any problem in this  we believe we could also use
WideBus without problems
– No need to re-cable M4R1 and M5R1 ODE crates
 Use WideBus
13 June 2013
A. Cardini for the Muon Upgrade Group
5
Design considerations / 3
• Modularity
–
We want to use a single nODE type
–
Taking into account the number of active inputs on our ODEs (120, 168, 192), the number of
Trigger Unit (TU) per ODEs and the number of logical channels per TU (10, 14, 16, 24, 28), the
usable modularity to be used to group input channels are:
• No re-cabling
 96 o 192 channels
• By re-cabling M4/M5 R1
 64, 96 o 192 channels
• By re-cabling M4/M5 R1 and with new TB
 32, 64, 96 o 192 channels
–
By using standard (80 bits) mode GBT modularity 96 (or 192) is not usable, this would imply to
choose a modularity of 64 or 32 than NEEDS re-cabling  WideBus preferred
Station
M2 or M3
M4 or M5
13 June 2013
Region
Logical
Channels
per TU
SYNCs per
TU
Active
channels
per SYNC
R1
R2
R3
R4
R1
R2
R3
R4
28
16
28
28
24
14
10
10
4
2
4
4
3
2
2
2
7
8
7
7
8
7
5
5
TU per ODE
Active
ODE per
(Active
ODE
quadrant
Optical links) Channels per station
A. Cardini for the Muon Upgrade Group
6
12
6
6
8*
12
12
12
168
192
168`
168
192
168
120
120
2
2
2
2
2*
1
1
1
6
We keep HITS and TDC info separate
(on different links)
• 2 GBT for HITs
• 2 GBT for TDC info
–
We use a 32 channel (nSYNC) chip on
nODE
• Max. 32 bits/chip for TDC information
(implication on max. occupancy)
We remove Intermediate Boards (IB) in
M5R4
–
Use High granularity pad detectors in
M2R1 (384 pads/chamber) and M2R2
(192 pads/chamber)
13 June 2013
GBTx #2
TDC
VTRx
GBTx
ECS
A. Cardini for the Muon Upgrade Group
New
SYNC
New
SYNC
New
SYNC
New
SYNC
New
SYNC
New
SYNC
96 Input ch
GBTx #2
trig
VTTx
VTTx
–
GBTx #1
TDC
TDC out
–
Use GBT WideBus (112 bits) 96 bits for
data (hits or TDC info) + 16 bits header
(BX counter, ZS information, other
info…)
Trig out
–
GBTx #1
trig
96 Input ch
Our working hypothesis (Paolo’s
hypothesis “E”)
TDC out
•
Trig out
System readout architecture
GBT
SCA
7
•
•
Substitute present ODE with nODE
–
2 GBTx per nODE in wide frame mode format (112 data bit)
•
•
96 bit for trigger hits
16 bit for header (format to be defined)
Remove IB in M5R4
Use high granularity chambers in M2R1 and M2R2
Station
M2
M3
M4
M5
Region
Active
Channels
per nODE
Logical
Channels
per TU
TU per
nODE
TU per
GBT
nSYNC per
nODE
GBT per
nODE
R1
R2
R3
R4
R1
R2
R3
R4
R1
R2
R3
R4
R1
R2
R3
R4
192
192
168
168
168
192
168
168
96 /192
168
120
120
96 /192
168
120
144
96
48
28
28
28
16
28
28
24
14
10
10
24
14
10
24
2
4
6
6
6
12
6
6
4/8
12
12
12
4/8
12
12
6
1
2
3
3
3
6
3
3
4
6
6
6
4
6
6
3
6
6
6
6
6
6
6
6
3/6
6
6
6
3/6
6
6
6
2
2
2
2
2
2
2
2
1 /2
2
2
2
1 /2
2
2
2
nODE per
O.L per
quadrant per quadrant per
station
station
6
6
2
2
2
2
2
2
2
1
1
1
2
1
1
2
nODE required: 35/quadrant  140 in total
Preliminary
12
12
4
4
4
4
4
4
3
2
2
2
3
2
2
4
Paolo Ciambrone INFN- LNF
•
Muon mapping
Hit Link summary
8
•
The number of O.L. for each region of a quadrant (4 stations) is compatible with 1 AMC40
–
If the resources of each AMC40 and/or ATCA40 were enough, 1 TELL 40 could elaborate the data of a
quadrant
ATCA40
Muon mapping
TRIG40 arrangement
12 (M2)
4 (M3)
3
3
(M4)
(M5)
Qi
R1
AMC40
O.L per quadrant
M2
M3
M4
M5
tot
R1
12
4
3
3
22
R2
12
4
2
2
20
R3
4
4
2
2
12
R4
4
4
2
4
14
12 (M2)
4 (M3)
2
2
(M4)
(M5)
AMC40
4
4
(M2)
(M3)
2
2
(M4)
(M5)
4 TELL40 required to readout HIT information
n
n
Qi
R2
AMC40
4
4
(M2)
(M3)
2
4
(M4)
(M5)
Custom – 10 Gbps
GBTx
Preliminary
Qi
R3
Qi
R4
AMC40
Paolo Ciambrone INFN- LNF
Region
9
Region
M2
M3
M4 or M5
M4 or M5
Active
Active
Max.
nSYNC per
Channels
Channels
Occupancy
nODE
per nODE
per nSync
174
192
168
168
168
192
168
168
96 o 192
168
120
120
96 o 192
168
120
144
R1
R2
R3
R4
R1
R2
R3
R4
R1
R2
R3
R4
R1
R2
R3
R4
6
6
6
6
6
6
6
6
3o6
6
6
6
3o6
6
6
6
32
32
28
28
28
32
28
28
32
28
20
20
32
28
20
24
~25%
~25%
~28%
~28%
~28%
~25%
~28%
~28%
~25%
~28%
~40%
~40%
~25%
~28%
~40%
~33%
GBT per
nODE
2
2
2
2
2
2
2
2
1o2
2
2
2
1o2
2
2
2
nODE per
O.L per quadrant
quadrant
per station
per station
6
6
2
2
2
2
2
2
2
1
1
1
2
1
1
2
12
12
4
4
4
4
4
4
3
2
2
2
3
2
2
4
These seem reasonable numbers – to be checked
O.L per quadrant
Region
M2
M3
M4
M5
tot
Amc40 per
quadrant
R1
12
4
3
3
22
1
R2
12
4
2
2
20
1
R3
4
4
2
2
12
0,5
R4
4
4
2
4
14
1
4 TELL40 required to readout TDC information
Preliminary
Muon mapping
Station
Paolo Ciambrone INFN- LNF
TDC Links info
10
Muon Detector Control and Pulsing
8 x Service Board Crate
For station M2,M3,M4,M5
Service Board
SB
ELMB
ELMB
Custom
Backplane
S
B
S
B
S
B
S
B
S
B
S
B
P S
D B
M
S
B
S
B
S
B
S
B
ELMB
S
B
ELMB
CANbus
16
Can
nodes
17
Can
nodes
16
Can
nodes
Pulse Distributor Module
PDM
ELMB
CANbus
Valerio Bocci
Complex system with more than 600 microcontrollers and 150 flash-based FPGA,
controlled by 6 computers, to set front-end parameters, pulse and monitor more
than 120k physical channels
13 June 2013
A. Cardini for the Muon Upgrade Group
11
Muon Detector Control Upgrade
•
Up to now we were only considering to replace 8 PDM boards in M2-M5:
– Use new clock distribution via GBT
– Use GBT-SCA to communicate with old Service Boards (SB), although through a
complicated protocol translation (GBT-SCA <-> I2C <-> CANbus <-> I2C)
•
Experts however considered that:
–
–
–
–
•
Current SB are already 10 years old
Based on even older microcontroller boards (ELMB)
Protocol translation would be extremely complicated and absolutely inefficient
Bottleneck is CANbus: we will not improve the current situation where we need 5’ to
configure the detectors and more than 20’ to read all scalers
We reached the conclusion that it is necessary to built 120 new SB:
– All control electronics will be renewed and will be ready to operate up to ~2030
– We will be able to use the high-speed communication via GBT for a fast
configuration/reconfiguration on-the-fly of the muon system (that we know is
necessary) and for an accurate real-time monitoring of the front-end boards (that is
something that we would really take advantage for)
13 June 2013
A. Cardini for the Muon Upgrade Group
12
nPDM block diagram
Do we need
more than 1
bidirectional
GBT link? NO
GBT has 16 I/O
lines, to
connect up to
16 GBT-SCA in
parallel, each
one directly
driving 16 I2C
lines
New backplane
implementation?
 we could
drive up to 256
I2C lines per
crate  OK
with current
setup
YES: details still
to be understood
Definition of all details is currently in progress
13 June 2013
A. Cardini for the Muon Upgrade Group
13
nSB block diagram
Solution1:
New Service Board Module
Depending on
new backplane
implementation
the design of
these boards
could be slightly
different
16 x I2C
(multiple GBT new Backplane)
I2C
FE
Flash
FPGA
I2C
Matrix
Flash
FPGA
CLK40
BC Pulse
test
pulse
logic
Flash
FPGA
1
2
Test/pulse
3
I2C
FE
1
Test/pulse
I2C
FE
Test/pulse
SCL
SDA_IN
SDA_OUT
Test/Pulse
RESET
2
3
3x
LVDS I2c
each ELMB
1
2
3
I2C
FE
1
Test/pulse
3
2
ttl/lvds
converter
Valerio Bocci 2013 May
13 June 2013
A. Cardini for the Muon Upgrade Group
14
Conclusions
•
The definition of all details of the muon system upgrade is progressing fast
•
Waiting for a feedback from Marseille colleagues on the new readout
architecture, but preliminary discussions suggest that is it reasonably ok
•
Muon system upgrade was presented last week to the INFN CSN1
(Commissione Scientifica Nazionale 1) where it was well received and we
have been asked to proceed  we are preparing a document for the INFN
CTS (Comitato Tecnico Scientifico), to be presented on July 12
•
Next steps:
–
–
–
–
July 2013: new architecture defined
October 2013: all details defined
November 2013: new electronics review
January 2014: Muon System Upgrade TDR
13 June 2013
A. Cardini for the Muon Upgrade Group
15
Spare slides
New alta
high-granularity
M2R1
granularità
M2R1 detectors
Keeping the l’attuale
currento layout
logic glayout
detector
will have avere
384 348 • • Mantenendo
lgi coevery
oni camera
dovrebbe
pads
in 4 TU
87da
logical
pads
pad
dagrouped
raggruppare
in 4ofTU
87 pad
logiche
nODEper
/ detector
quadrant
– – 2 2nODE
camera 66nODE
nODE/ per
quadrante
Every
nODEhawill
have
174 active
inputs
(2 TU
logical
pads)
– – Ogni
nODE
174
ingressi
relativi
a 2 TU
da of
8787
pad
logiche
ciascuna
– Need 12 links/ quadrant  +16 nODE
– Occorrono 12 link a quadrante
Out 0-7
Out 0-7
Out 0-7
Out 0-7
Out 0-7
Out 0-7
CARDIAC CARDIAC CARDIAC CARDIAC CARDIAC CARDIAC
•
PAD physical ch. = X-Logical ch.
–
–
0
0
7
7
0
0
7
7
0
0
7
0
7
0
0
13 June 2013
Out 0-7
Out 0-7
Out 0-7
Out 0-7
CARDIAC CARDIAC CARDIAC CARDIAC
Out 0-7
Out 0-7
Out 0-7
Out 0-7
CARDIAC CARDIAC CARDIAC CARDIAC
7
Wire physical ch. = Y-Logical ch.
–
–
•
–
–
•
6.3x31.3 mm2
348 Ch./chamber
Trigger sector
–
–
–
•
6.3x 253 mm2
48 ch./chamber
Logical pad
16 X-Logical ch.
12 Y-Logical ch.
4 TS/chamber
14 CARDIAC for readout
–
7
WIRE NUMBER
•
37.5x31.3 mm2
64 ch./chamber
2 x 112 channels
47
A. Cardini for the Muon Upgrade Group
17
M2R2high-granularity
alta granularità M2R2 detectors
New
Mantenendo
l’attuale
o layout
g lgievery
co odetector
ni camera
• •Keeping
the current
logic
layout
will dovrebbe
have 192 avere 192
padgrouped
da raggruppare
TU da pads
48 pad logiche
pads
in 4 TU ofin484 logical
–
–
–
–1 nODE
1 nODE
per camera
6 nODE
per quadrante
/ detector
 6 nODE
/ quadrant
nODE
willha
have
active relativi
inputs (4
of da
48 logical
–Every
Ogni
nODE
192192
ingressi
a TU
4 TU
48 padpads)
logiche ciascuna
12 links/12
quadrant
 +16 nODE
–Need
Occorrono
link a quadrante
Out 0Out 0Out 0Out 0Out 0Out 0CARDIAC
CARDIAC
CARDIAC
CARDIAC
CARDIAC
CARDIAC
7
7
7
7
7
7
0
7
7
0
0
7
7
0
0
7
0
7
0
0
13 June 2013
0
Out 0-7
Out 0-7
Out 0-7
Out 0-7
CARDIAC CARDIAC CARDIAC CARDIAC
Out 0-7
Out 0-7
Out 0-7
Out 0-7
CARDIAC CARDIAC CARDIAC CARDIAC
7
•
PAD physical ch.
•
X-logical channel
•
Wire physical ch. = Y-Logical ch.
47
A. Cardini for the Muon Upgrade Group
–
–
–
–
–
•
•
OR of 4 PADs
150x62.6 mm2
16 ch./chamber
12.5 x 253 mm2
48 ch./chamber
12.5x62.6 mm2
192 Ch./chamber
Trigger sector
–
–
–
•
75x31.3 mm2
64 ch./chamber
Logical pad
–
–
7
WIRE NUMBER
–
–
4 X-Logical ch.
12 Y-Logical ch.
4 TS/chamber
14 CARDIAC for readout
–
2 x 112 channels
18
Senza ricablaggio (sono mostrati solo i primi 96 canali, i secondi 96 sono uguali)
0
7
15
M2/M3 R1:28 Logical channels per TU
(Le pad catodiche passano per le IB, le pad dei fili
vanno direttamente alle ODE)
TU 1
TU 1
55
63
71
TU 4
TU 2
TU 3
TU 3
87
95
TU 5
TU 6
TU 3
TU 3
TU 2
TU 2
79
TU 3
TU 2
TU 1
TU 1
47
TU 3
TU 1
M4/M5 R1:24 Logical channels per TU
M4/M5 R3/R4:10 Logical channels per TU
39
TU 2
TU 2
M2/M3 R3/R4:28 Logical Channels per TU
M4/M5 R2:14 Logical channels per TU
31
TU 1
(2 tipi di Tb a causa dell’orientamento dei canali
logici)
M2/M3 R2:16 Logical channels per TU
23
TU 4
TU 4
TU 4
TU 5
TU 5
Canali logici attivi sul connettore di ingresso
Canali non “attivi” (floating) sul connettore di ingresso
Canali logici appartenenti alla stessa TU
TU 6
TU 6
Paolo Ciambrone INFN- LNF
•
ODE Upgrade
ODE Input Connectors
19
Ricablando M4/M5 R1 (sono mostrati solo i primi 96 canali, i secondi 96 sono uguali, ad
eccezione di M4/M5 R1 che deve avere un cablaggio speculare rispetto ai primi 96)
0
7
15
M2/M3 R1:28 Logical channels per TU
(Le pad catodiche passano per le IB, le pad dei
fili vanno direttamente alle ODE)
31
39
TU 1
(2 tipi di Tb a causa dell’orientamento dei canali
logici)
M2/M3 R2:16 Logical channels per TU
23
TU 1
M4/M5 R1:24 Logical channels per
TU (nei secondi 96 canali il cablaggio è
55
63
71
TU 2
TU 3
TU 4
87
TU 5
TU 6
TU 2
TU 3
speculare)
M4/M5 R2:14 Logical channels per TU
M4/M5 R3/R4:10 Logical channels per TU
TU 1
TU 1
TU 2
TU 2
TU 3
TU 3
95
TU 3
TU 2
TU 1
79
TU 3
TU 2
TU 1
M2/M3 R3/R4:28 Logical Channels per TU
47
TU 4
TU 4
TU 5
TU 5
Canali logici attivi sul connettore di ingresso
Canali non “attivi” (floating) sul connettore di ingresso
Canali logici appartenenti alla stessa TU
TU 6
TU 6
Paolo Ciambrone INFN- LNF
•
ODE Upgrade
ODE Input Connectors
20
Ricablando M4/M5 R1 e cambiando la TB (sono mostrati solo i primi 96 canali, i secondi
96 sono uguali)
0
7
15
M2/M3 R1:28 Logical channels per TU
(Le pad catodiche passano per le IB, le pad dei fili
vanno direttamente alle ODE)
TU 1
M4/M5 R1:24 Logical channels per TU
M4/M5 R3/R4:10 Logical channels per TU
39
TU 2
TU 1
55
TU 3
63
71
TU 4
TU 2
TU 3
TU 3
87
TU 5
95
TU 6
TU 3
TU 3
TU 2
TU 2
79
TU 3
TU 2
TU 1
TU 1
47
TU 2
TU 1
M2/M3 R3/R4:28 Logical Channels per TU
M4/M5 R2:14 Logical channels per TU
31
TU 1
(2 tipi di Tb a causa dell’orientamento dei canali
logici)
M2/M3 R2:16 Logical channels per TU
23
TU 4
TU 4
TU 5
TU 5
Canali logici attivi sul connettore di ingresso
Canali non “attivi” (floating) sul connettore di ingresso
Canali logici appartenenti alla stessa TU
TU 6
TU 6
Paolo Ciambrone INFN- LNF
•
ODE Upgrade
ODE Input Connectors
21
Bunch cross synchr.
Bunch cross synchr.
Bunch
crosssynchr.
synchr.
Bunch cross
Bunch cross synchr.
Bunch cross synchr.
Sync
Sync
Sync
Sync
Sync
Sync
LVDS Receivers
LVDS Receivers
LVDS
Receivers
Translators
LVDS
Receivers
Translators
LVDS
Receivers
Translators
LVDS
Receivers
Translators
Translators
Translators
192 Input
channels
nSYNC
nSYNC
nSYNC
nSYNC
nSYNC
Hit
GBT
nSYNC
Hit
GBT
Hit
GBT
format
interf.
Hit
GBT
format
interf.
Hit
GBT
format
interf.
Hit
GBT
format
interf.
format
interf.
Hist.
format
interf.
Hist.
Hist.
Hist.
Hist.
TDC
TDCHist.
TDC
TDC
Zero
GBT
TDC
Zero
TDC GBT
Zero
GBT
Supp.
interf.
Zero
GBT
Supp.
interf.
Zero
GBT
Supp.
Zero interf.
GBT
Supp.
interf.
Supp.
interf.
Supp.
interf.
ECS/TFC
To/from TFC
1 link
duplex
DATA Path
GBTx
GBTx
Hit
HITdata
data
serializer
serializer
GBTx
GBTx
Hit
data
TDC
data
serializer
serializer
To Trigger
2 links
simplex
VTTx
Optical
transmitter
(Hits)
VTTx
To TELL40
2 links
simplex
Optical
transmitter
(TDC)
Clock management
GBTx
GBT-SCA
Data
Ser/Des
Control
Monitor
Clock recovery
Power up
sequencer
Voltage
regulators
VTRx
Optical
transceiver
Phase
adjust
Clock
driver
Power section
DC/DC
converter
13 June 2013
6 optical
links
nODE
A. Cardini
for therequired
Muon Upgrade/Group
nODE
22
FROM FEE/IB
TDC
+
Hist
TDC
+
Hist
TDC
+
Hist
Hit + TDC
BX synchronization
Bxid + Hit + TDC
TFC
IF
TDC’s
Hits
Header
TO Trig. GBT
Frame Header Gen.
I2 C
TDC ZS
TDC’s
Header
Prog.
Buffer
TO TDC GBT
13 June 2013
A. Cardini for the Muon Upgrade Group
nSYNC
block
diagram
23
System readout architecture / 2
• Design considerations and first decisions taken:
– Need to readout both HITs and their time (TDC): time information
is required to time align the detector and to monitor it
– HIT and TDC info on same/different link?  two different links
– Modularity: only one nODE type and we will not change the
cabling on the back of the crates  96 bits for data (naturally
compatible only with WideBus) + 16 bits for additional
information (Bxid, …)
– GBT WideBus (112 bits) or not (80 bits)?  WideBus
– A new ASIC for nODE  a 32 channels nSYNC
13 June 2013
A. Cardini for the Muon Upgrade Group
24
System readout architecture / 3
• Current system layout…
–  104 nODE, 4 TELL40 for HIT and L0muon readout, 3 TELL40 for muon TDC
info
– Very compact system, no need to exchange data between AMC40 outside
ATCA40
• + Remove IB from M5R4…
– M5R4 is readout by 2 IB/quadrant
–  108 nODE, 4 TELL40 for HIT and L0muon readout, 3 TELL40 for muon TDC
info
• + New high-granularity detectors in M2R12 (see next 2 slides)
– Need to add 32 nODE (or the equivalent of it) to readout 12 M2R1 detectors
with 384 pads and 24 M2R2 detectors with 192 pads
–  140 (equiv.) nODE, 4 TELL40 for HIT and L0muon readout, 4 TELL40 for
muon TDC info
• Verification of all details with Marseille colleagues in progress
13 June 2013
A. Cardini for the Muon Upgrade Group
25