3D Scene Calibration for Infrared Image Analysis
i r f m
cadarache
V. Martin, V. Gervaise, V. Moncada, M.H. Aumeunier, M. Firdaouss, J.M. Travere (CEA)
S. Devaux (IPP), G. Arnoux (CCFE) and JET-EFDA contributors
Workshop on Fusion
Data
Processing
Validation
and Analysis,
ENEA Frascati, 26-28
March 2012
V. Martin
et al.
1 (19)
WFDPVA,
ENEA Frascati
28/03/12
3D IR Scene Calibration
JET #81313 KL7
• Issue: a complex thermal scene
(images in DL)
1. Wide angle views with high geometrical effects:
depth of field and curvature
2. Many metallic materials (Be, W) with different
and changing optical (reflectance) and thermal
(emissivity) properties
• Objective: Match each pixel with the 3D
scene model of in-vessel components for:
W
coated
CFC
Bulk
Be
1. getting the real geometry of the viewed objects
2. reliable linking between viewed objects and their
related properties
Bulk
Be
Bulk
Be
Be
coated
linconel
• Applications
1. Image processing (e.g. event characterization)
W coated
CFC
2. IR data calibration: Tsurf = f(material emissivity)
Bulk
W
V. Martin et al. 2 (19)
WFDPVA, ENEA Frascati
28/03/12
Methodology
• Calibration chain
Camera
NUC
Dead pixel Map
Reference image
Image
Correction
Image
Stabilization
V. Martin et al. 3 (19)
2D/3D scene
models
Knowledge base
of the thermal
scene
2D/3D Scene
Model
Mapping
WFDPVA, ENEA Frascati
Image
Processing
Registered &
Geo-calibrated
Image
28/03/12
Illustration of Motion in Images
• Camera vibrations lead to misalignments of ROIs (PFC RT
protection) = false alarms or worth missed alarms
• Image stabilization is a mandatory step for heat flux deposit
analysis based on Tsurf(t)-Tsurf(t-1) estimations
V. Martin et al. 4 (19)
WFDPVA, ENEA Frascati
28/03/12
Image Stabilization
• Important factors for method selection
•
Deformation type: planar (homothety), non-planar
•
Target application: real-time processing, off-line analysis
•
Data quality and variability: noise level, pixel intensity changes, image entropy
•
Required precision level: pixel, sub-pixel
• Applications in tokamaks (non-exhaustive list)
Motion amplitude
Target application
Precision Difficulty
required
JET KL7
wide-angle
5-10 pixels
(camera vibrations)
Hot spot detection PFC
protection
pixel
low image entropy
JET KL7
windowed
up to 15 pixels
(disruptions)
Physics analysis (e.g.
heat load during
disruptions…)
pixel
pixel intensity
changes
JET KL9
divertor tiles
<1 pixel
(sensor affected by
magnetic fields)
Physics analysis (power
deposit influx)
sub-pixel
low resolution, slow
motion, aliasing
V. Martin et al. 5 (19)
WFDPVA, ENEA Frascati
28/03/12
Image Stabilization
• Classical Methodology
1. Feature Detection
•
Local descriptors: Harris corners, MSER, codebooks, Gabor wavelets (see Craciunescu
talk), SIFT, SURF, FAST…
•
Global descriptors: Tsallis entropy (see Murari talk), edge detectors…
•
Fourier analysis: spectral magnitude & phase, pixel gradients, log-polar mapping…
2. Feature Matching
•
Spatial cross-correlation techniques: normalized cross-correlation, Hausdorff distance…
•
Fourier domain: normalized cross-spectrum and its extensions
3. Transform Model Estimation
•
Shape preserving mapping (rotation, translation and scaling only)
•
Elastic mapping: warping techniques…
4. Image transformation
•
2D Interpolation: nearest neighboor, bilinear, bicubic…
See Zitova’s survey, Image and Vision Computing, vol. 21(2003), pp. 977-1000
V. Martin et al. 6 (19)
WFDPVA, ENEA Frascati
28/03/12
Proposed Algorithm
1. Masked FFT-based image registration [1]
 Deterministic computing time
 Accelerating hardware compatible algorithm (e.g. FFT on GPU) → real time
applications
 Local analysis with dynamic intensity-based pixel masking (e.g. mask the
divertor bright region)
2. with sub-pixel precision [2]
 Slow drift compensation
3. and dynamic update of the reference image
 Robust to image intensity and structural changes
 Evaluation of the registration quality over time
[1] D. Padfield, IEEE CVPR’10, pp. 2918-2925, 2010
[2] M. Guizar-Sicairos et al., Opt. Lett., vol. 33, no. 2, pp. 156-158, 2008
V. Martin et al. 7 (19)
WFDPVA, ENEA Frascati
28/03/12
Principle of Fourier-based Correlation
• Let Iref a reference image, It an image at time t and DFT the Discrete 2D Fourier transform
such as It ( x , y ) = Iref ( x-x0 , y-y0 )
F1  DFT ( I ref )
F2  DFT ( I t )
FNCC (.,.) 
F1 (.,.)  F2 (.,.)
F1 (.,.)  F2 (.,.)
Iref
NCC  DFT -1 ( FNCC )
• NCC is the Normalized Cross Correlation figure (image)
It
NCC(Iref, It)
and the position of the peak gives the coordinates of the
translation ( x0 , y0 )
max (NCC(Iref, It))
x0 , y0   arg max NCC
x, y
V. Martin et al. 8 (19)
WFDPVA, ENEA Frascati
28/03/12
Sub-pixel Precision
• Up-sample k times the DFT of NCC (trigonometric interpolation):
 NCC ( u k , v k ) if . k is an integer
NCC (u , v)  
otherwise
0
UP
UP
FNCC
 DFT ( NCCUP )
UP
UP
FNCC
(low frequencie s)  FNCC
UP
FNCC
(high frequencie s)  0 (anti - aliasing)
UP
NCCUP  DFT -1 ( FNCC
)
• The peak coordinates ( x0 , y0 ) give F the translation with 1/k pixel of precision:
x0 , y0   1 arg max NCC UP
k
x, y
V. Martin et al. 9 (19)
WFDPVA, ENEA Frascati
28/03/12
Reference Image Updating
• Goal: maintaining a good reliability of the motion estimator (NCC
peak value) while image appearance changes during the pulse.
V. Martin et al. 10 (19)
WFDPVA, ENEA Frascati
28/03/12
Reference Image Updating
• Solution: use the NCC peak value
to trigger the update of Iref such as:
update
Iref
update
Iref
if
Tmin  max( NCC(t ))  Tmax
then I ref  I t
update
Iref
update
Iref
NCC peak too low,
no Iref update
V. Martin et al. 11 (19)
WFDPVA, ENEA Frascati
28/03/12
Results
• JET #81313 (MARFE, disruption), KL7, 480x512 pixels, 50 Hz,
251 frames
k=1/4 pixel
V. Martin et al. 12 (19)
WFDPVA, ENEA Frascati
28/03/12
Results
• JET #80827 (disruption), KL7, 128x256 pixels, 540 Hz,
13425 frames
k=1/2 pixel
V. Martin et al. 13 (19)
WFDPVA, ENEA Frascati
28/03/12
Results
• JET #82278, KL9B (slow drift), 32x96 pixels, 6 kHz, 4828 frames
96 pixels
Tstab  Tunstab  25,10
32 pixels
V. Martin et al. 14 (19)
WFDPVA, ENEA Frascati
28/03/12
Computational Performance
• High frame rate performance using GPU
256x256, k=1/4
→ 700 fps
V. Martin et al. 15 (19)
WFDPVA, ENEA Frascati
28/03/12
From 2D to 3D
• Challenge
– transform pixel coordinates into machine coordinates: (x, y)  (r, θ, φ)
• Method
– Ray-tracing method from 3D/simplified CAD files
V. Martin et al. 16 (19)
WFDPVA, ENEA Frascati
28/03/12
3D Scene Model for Image Processing
S. Palazzo, A. Murari et al., RSI 81, 083505, 2010
Z Map (depth)
1
mm
2
2m
1
Blobs 1 & 2
must not be
merged!
7m
2
1
2
V. Martin et al.
V. Martin et al. 17 (19)
WFDPVA, ENEA Frascati
28/03/12
Integrated Framework
• An integrated software for IR data stabilization & analysis
NUC
Dead pixel Map
Reference image
2D/3D scene
models
Knowledge base
of the thermal
scene
Load/save
translations
Used
for event
triggering
Camera
Image
Correction
Image
Stabilization
2D/3D Scene
Model
Mapping
Registered &
Calibrated
Image
temperature
Image
Processing
Set
mask
Used for
evaluation
Plasma ImagiNg data Understanding Platform
Set(PINUP)
sub-pixel precision
factorfor PFC protection
Used
V. Martin et al. 18 (19)
WFDPVA, ENEA Frascati
28/03/12
Conclusion
• Summary
– Complex IR scenes require a new approach for reliable data analysis including
image stabilization and 3D mapping.
– A robust and fast image stabilization algorithm with sub-pixel precision has
been proposed.
– A first demonstration of 3D model for IR data analysis has been successfully
carried out at JET on the wide-angle ITER-like viewing system (KL7).
– An integrated software (PINUP) implementing these features is available for
users upon request.
• Outlook
– Test of the stabilization algorithm on visible imaging data (JET KL8) with
rotation compensation
– Full integration of 3D scene models into PINUP
– Improvement of image processing algorithms (e.g. hot spot detection) with 3D
information
V. Martin et al. 19 (19)
WFDPVA, ENEA Frascati
28/03/12