European Statistical Training
Programme: “Evaluation"
Intro-Beginners and Advanced
Courses On Seasonal Adjustment 2008-2012
Prepared by Adriano di Guglielmo
Unit B1 Quality Methodology and Research, Eurostat.
European Statistical Training Programme:

The purpose of the European Statistical Training Programme (ESTP) is to provide
European statisticians the opportunity to participate in international training courses,
workshops and seminars at postgraduate level.

The programme is coordinated by Eurostat. Courses are delivered either at Eurostat
premises in Luxembourg or in national training sites of EU and European Free Trade
Association (EFTA) countries.

It is tailored to meet the specific needs of the European Statistical System (ESS) by
taking into account the different levels of statistical knowledge and working experiences.
It comprises courses in Official Statistics, IT applications, Research and Development
and Statistical Management.

More info here
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
2
Analysis of Customer Satisfaction:

Studying the questionnaires on
evaluation, proposed at the end of
each course, we are going to verify
our attendants’ satisfaction. With this
descriptive analysis will be possible
to understand where to take action to
improve the service offered.
"ESTP - Customer Satisfaction" by Adriano di Guglielmo


On the other hand, our data are
not collected in such a way as to
make a more in depth analysis
(such as MCA or PLS PM).
Therefore it would be useful to
know all the sequence answers of
each user and also a series of
illustrative variables of them (the
users) useful to describe the
differents interviewees profiles,
whitout invading any privacy.
3
Questionnaires Structure:
Topics Expectations
New Concepts/
Methods/
Techniques
Deepening
methods/techniques
Programme
Course
content
Course Work
Overall
evaluation
of the
course
Final
Evaluation
Theoretical –
practical
Training
methods
&
training
support
SA/Demetra+
Combination
of methods
Materials
Quality
Actives
Trainers
Presentations
Length
Pace
Logistics
Course
administration
Course
organisation
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
Support
provided
4
BEGINNERS COURSES
 Number of attendants, number of applicants
 Who are the course leaders, who are the trainers,
who are the attendants?
 Questionnaires outcomes
 Time Series evaluation scores
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
Number of Attendants 2006 – 2012
Beginners
Intro - Beginners
35
30
7
25
20
18
0
0
7
0
14
N of Not Accepted Invitation
N of Accepted Invitation
15
21
20
22
22
17
15
14
10
5
0
2006
2007
2008
2009
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
2010
2011
2012
6
Beginners Courses 2009-2012
2009
2010
2011
2012
Course Leader
G. L. Mazzi; C.Calizzani
G. L. Mazzi; C.Calizzani
D. Buono
D. Buono
Trainers
D. Buono; A. Maravall;
B. Monsell; D. Ladiray
D. Ladiray
E. Infante
D. Buono
E. Infante
D. Buono
22/29
17/24
14/28
15/33
Duration – Days
4
4
3
3
Duration – Hours
-
27
15
15
Duration – Breaks
1h
1h
2h
2h
-
6h 45min
5h
5h
Attendants/
applicants
Daily Average
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
7
To be noticed that:
• Countries with largest number of presences:
Germany (7), Czech Republic (6), Greece (6),
Sweden (6), Italy (6), Romania (6), Slovenia
(6), Slovakia (6).
• Countries that are never present:
Macedonia, France, Liechtenstein,
Switzerland.
• Last presence more than 4 years ago:
Estonia, Denmark, Latvia, Luxembourg,
Hungary.
• General balance between students from old* and new** EU countries
*Old EU countries = Join EU before 2004
**New EU countries = Join EU after 2004 + EFTA countries
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
8
N
D
ew
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
C
n
Fi
n
rt
p
vid
n
ed
ua
tio
ro
n
s
st
ic
at
io
is
al
Ev
al
Su
pp
o
or
ga
n
Lo
gi
ns
s
lity
ne
r
Q
ua
Tr
ai
al
ds
en
ta
tio
Pr
es
es
Ac
tiv
s
et
ho
m
ia
l
of
th
ce
ct
ic
Pa
Le
ng
et
ra
+
em
al
/p
ra
tic
at
er
M
na
tio
or
e
SA
/D
es
ue
s
iq
u
or
k
m
e
m
hn
iq
hn
s/
te
c
ou
rs
e
bi
Th
e
et
ho
d
m
og
ra
/w
ta
tio
ns
ec
ou
rs
e
Pr
C
/e
xp
s/
te
c
ics
et
ho
d
m
om
ng
C
en
i
ep
ts
/
nc
ee
p
co
To
p
Analysis of all variables - Beginners 2009
Satisfaction 2009
16
Agree
S.
14
12
Agree
10
Neutral
8
6
Disagree
4
2
Disagree
S.
0
No response
9
N
D
C
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
C
n
of
rt
ua
tio
n
d
n
id
e
ov
pr
va
l
al
E
pp
o
st
ic
s
is
at
io
Lo
gi
ga
n
or
Pr
Fi
n
Su
Tr
a
lity
ds
in
er
s
es
en
ta
tio
ns
Ac
tiv
es
se
et
ho
ia
ls
Q
ua
m
ce
ct
ic
al
Pa
th
et
ra
+
s
Le
ng
em
tic
al
/p
ra
at
er
M
na
tio
or
e
ou
r
bi
Th
e
SA
/D
es
m
e
or
k
qu
e
iq
u
hn
i
te
c
hn
s/
te
c
ds
/
et
ho
d
se
/w
ou
r
ec
ta
tio
ns
Pr
og
ra
m
C
/e
xp
ics
et
ho
m
om
in
g
ce
pt
s/
m
en
d
co
n
ee
p
ew
To
p
Analysis of all variables - Beginners 2010
Satisfaction 2010
12
Agree
S.
10
Agree
8
Neutral
6
4
Disagree
2
Disagree
S.
0
No response
10
N
co
nc
ep
C
ou
rs
e/
w
ta
ti o
ns
Pa
ce
et
i
c
C
a
om
l /p
ra
bi
ct
na
ic
ti o
al
n
of
m
et
ho
M
ds
at
er
ia
ls
Q
ua
Ac
lit
tiv
y
es
Tr
ai
ne
rs
Pr
es
en
ta
ti o
ns
Lo
C
gi
ou
st
ic
rs
s
e
or
ga
ni
sa
Su
tio
pp
n
or
tp
ro
vi
de
Fi
d
na
lE
va
lu
at
io
n
Th
eo
r
Le
ng
th
or
k
P
ro
ts
/m
gr
D
am
et
ee
ho
m
pe
ds
e
nd
/ te
in
ch
g
ni
m
qu
et
ho
es
ds
/te
ch
ni
qu
es
SA
/D
em
et
ra
+
ew
To
pi
cs
/e
xp
ec
Analysis of all variables - Beginners 2011
Beginners Satisfaction 2011
12
10
Agree
S.
8
Agree
6
4
Neutral
2
Disagree
0
The duration of the course decreases (3 day – 5h per day)
The evaluation scores increase
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
11
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
n
rt
sti
d
n
ide
ua
tio
ov
pr
n
cs
isa
t io
va
l
al
E
pp
o
ga
n
or
Fi
n
Su
se
Lo
gi
ns
rs
lity
ine
en
ta
tio
sT
ra
es
Pr
Ac
tiv
e
sQ
ua
ds
l
ce
th
ca
et
ho
cti
Pa
Le
ng
l/ p
ra
m
s
et
ra
+
m
qu
e
es
m
e
m
iqu
hn
i
ial
of
tic
a
at
er
M
na
tio
te
c
hn
SA
/D
e
ds
/
or
e
Co
ur
bi
Th
e
et
ho
m
s
/w
or
k
og
ra
s/t
ec
Pr
se
ta
tio
n
ec
Co
ur
/e
xp
ics
m
et
ho
d
Co
m
ing
ep
ts/
nc
en
d
co
De
ep
Ne
w
To
p
Analysis of all variables - Beginners 2012
Satisfaction 2012
16
14
12
Agree
S.
10
Agree
8
6
Neutral
4
2
Disagree
0
The duration of the course decreases (3 day – 5h per day)
The evaluation scores increase
12
Final Evaluation – Beginners
Agree Vs Disagree - Beginners
100%
0%
90%
23%
0%
0%
22%
80%
70%
22%
60%
Disagree
100%
50%
40%
100%
Agree
77%
30%
Neutral
56%
20%
10%
0%
2009
2010
2011
2012
12 old EU
10 new EU
9 old EU
8 new EU
6 old EU
7 new EU
9 old EU
4 new EU
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
17
ADVANCED COURSES
 Number of attendants, number of applicants
 Who are the course leaders, who are the trainers,
who are the attendants?
 Questionnaires outcomes
 Time Series evaluation scores
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
Number of Attendants 2008 – 2012
Advanced Courses
Advanced
35
30
25
8
8
19
20
15
N of Not Accepted Invitation
20
21
0
0
N of Accepted Invitation
16
10
14
15
2011
2012
5
0
2008
2009
2010
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
19
Advanced Courses 2008-2011
2008
2009
2010
2011
Course Leader
G. L. Mazzi, C.Calizzani
C.Calizzani
G.L. Mazzi,
J.M. Museux
D. Buono
Trainers
D. Findley; R. Kirchner A.
Maravall; B. Monsell
R. Kirchner;
A. Maravall;
T. Mc Elroy
D. Findley
D. Buono
A. Maravall
R. Kirchner; D. Buono;
K. Attal-Toubert;
A. Kocak; J. Palate
20/28
21/29
16/16
14/33
Duration – Days
4
5
4
3
Duration – Hours
24h 30min
32h
26h
15
Duration – Breaks
1h
1h
1h
2h
More than 6h
Less then 6h 30min
6h 30min
5h
Attendants/
applicants
Daily Average
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
20
To be noticed that:
 Countries with largest number of
presences (4):
Turkey, Portugal, Slovenia, Romania,
United Kingdom.
 Countries that are never present:
Cyprus, Czech republic, Macedonia,
France, Liechtenstein.
• In the next advanced course of 2012
there will be also a Mexican attendant.
• It is strange that attendants from
Czech Republic are very interested
only in the beginners courses.
• General balance between students from old and new EU countries
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
21
N
D
C
or
e
tic
s
em
et
ra
+
Le
ng
th
qu
e
es
m
e
m
iq
u
hn
i
te
c
SA
/D
ds
/
hn
s/
te
c
or
k
Pa
ce
al
/p
na
r
ac
tio
tic
n
al
of
m
e
th
M
od
at
er
s
ia
ls
Q
Ac
ua
tiv
lit
es
y
Tr
ai
ne
Pr
rs
es
en
ta
tio
ns
C
Lo
ou
gi
rs
st
e
ic
or
s
ga
ni
Su
sa
t io
pp
n
or
tp
r
o
Fi
vid
na
ed
lE
va
lu
at
io
n
Th
e
et
ho
m
og
ra
/w
ta
tio
ns
ec
ou
rs
e
Pr
C
/e
xp
ics
et
ho
d
m
bi
om
in
g
ep
ts
/
nc
en
d
co
ee
p
ew
To
p
Analysis of all variables - Advanced Course - 2008
Satisfaction 2008
16
Agree
S.
14
12
Agree
10
Neutral
8
6
Disagree
4
2
Disagree
S.
0
No response
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
22
N
co
nc
ep
C
ou
rs
e/
w
ta
ti o
ns
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
C
ou
rs
e
Lo
gi
st
ic
s
or
ga
ni
sa
Su
tio
pp
n
or
tp
ro
vi
de
Fi
d
na
lE
va
lu
at
io
n
Pa
ce
et
ic
C
al
om
/p
ra
bi
ct
na
ic
ti o
al
n
of
m
et
ho
M
ds
at
er
ia
ls
Q
ua
Ac
lit
tiv
y
es
Tr
ai
ne
rs
Pr
es
en
ta
ti o
ns
Th
eo
r
Le
ng
th
or
k
P
r
ts
o
/
gr
m
D
am
et
ee
ho
m
pe
d
e
nd
s/
te
in
ch
g
ni
m
qu
et
ho
es
ds
/te
ch
ni
qu
es
SA
/D
em
et
ra
+
ew
To
pi
cs
/e
xp
ec
Analysis of all variables - Advanced Course - 2009
Satisfaction 2009
16
14
Agree
S.
12
10
Agree
8
Neutral
6
4
Disagree
2
Disagree
S.
0
No response
23
ta
ti o
ns
Pa
ce
et
i
ca
Co
l /p
m
ra
bi
cti
na
ca
ti o
l
n
of
m
et
ho
M
ds
at
er
ia
ls Q
ua
Ac
lit y
tiv
es
Tr
ai
ne
rs
Pr
es
en
ta
ti o
ns
Lo
Co
gi
sti
ur
cs
se
or
ga
nis
at
Su
ion
pp
or
tp
ro
vi d
Fi
ed
na
lE
va
lua
tio
n
Th
eo
r
Le
ng
th
Co
ur
se
/w
co
or
nc
k
ep
Pr
ts/
o
g
m
De
ra
et
m
ep
ho
m
en
ds
e
/ te
din
ch
g
ni
m
qu
et
ho
es
ds
/te
ch
niq
ue
SA
s
/D
em
et
ra
+
Ne
w
To
pic
s/
ex
pe
c
Analysis of all variables - Advanced Course - 2010
Satisfaction 2010
14
12
Agree
S.
10
Agree
8
Neutral
6
Disagree
4
2
Disagree
S.
0
No response
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
24
co
nc
ep
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
Lo
gi
sti
cs
or
ga
nis
at
ion
Su
pp
or
tp
ro
vi d
ed
Fi
na
lE
va
lua
tio
n
Co
ur
se
Th
eo
r
Pa
ce
et
ic a
l /p
Co
ra
cti
m
bi
ca
na
l
ti o
n
of
m
et
ho
ds
M
at
er
ia
ls
Qu
al
it y
Ac
tiv
es
Tr
ai
ne
rs
Pr
es
en
ta
ti o
ns
Le
ng
th
Pr
og
ts/
ra
m
m
m
et
e
ho
De
ds
ep
/
t
en
ec
hn
din
iq
g
ue
m
s
et
ho
ds
/te
ch
niq
ue
Q
s
ua
li ty
co
nc
ep
ts
Ne
w
ta
ti o
ns
Co
ur
se
/w
or
k
To
pic
s/
ex
pe
c
Analysis of all variables - Advanced Course - 2011
Satisfaction 2011
12
10
Agree
S.
8
Agree
Neutral
6
Disagree
4
Disagree
S.
2
No response
0
The duration of the course decreases (3 day – 5h per day)
The evaluation scores increase
25
Final Evaluation - Advanced
Agree Vs Disagree - Advanced
100%
10%
90%
7%
7%
16%
80%
70%
60%
Disagree
50%
40%
Neutral
90%
84%
93%
93%
2008
2009
2010
2011
10 old EU
10 new EU
13 old EU
7 new EU
5 old EU
10 new EU
7 old EU
6 new EU
Agree
30%
20%
10%
0%
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
30
Findings (Beginners)

As shown in the previous slides, we find extremely positive results for most of beginners
courses (years 2009/2011/2012).

On the other hand, the course of 2010 wasn’t able to fully satisfy the course participants:
it seems to be crucial for the debacle of this course, the poor satisfaction shown for the
“training methods and training support” where more of the half of interviewees (51%)
didn’t show satisfaction (Neutral, Disagree, Disagree Strongly). More critical aspects
involve the questions: “Theoretical and practical training was combined in an appropriate
way” and “The combination of methods (lectures, presentations, group work, discussions,
exercises, etc.) was appropriate”.

The course of 2012 (73% of Agree Strongly) results the best ever with a peak of 93% of
Agree Strongly in the Course Administration category.

No neutral or negative scores in the final evaluation of the years 2011/2012.

Balance between students from old and new EU countries.
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
38
Findings (advanced)

The analysis performed on the advanced courses shows satisfactory evaluation scores.
As for the beginners courses, 2010 is the year in which the satisfaction of the advanced
course reaches its minimum point (47%) in terms of “Agree Strongly”, even if it doesn’t hit
the negative peak of the corresponding beginners course (47% vs. 11% respectively).

To be noticed that the advanced course of 2009 reaches the 63% of Agree Strongly and
that the question: “The course was relevant to my job and will help me in my current
and/or future work” collected 74% of Agree Strongly.

On the other hand, when comparing the scores (Agree vs. Disagree), it can be noticed
that the general satisfaction lies around a common threshold for the four years under
consideration, with peaks in the years 2010 and 2011.

No negative scores in the final evaluation of all the years considered.

Balance between students from old and new EU countries.
"ESTP - Customer Satisfaction" by Adriano di Guglielmo
39