A review of the state of the art,
present norms and future
trends in the field of
measurement uncertainty
Walter Bich
INRIM – Istituto nazionale di ricerca metrologica
Torino (Italia)
JCGM-WG1
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
1
Uncertainty of measurement
measurement uncertainty
uncertainty of measurement
uncertainty
parameter characterizing the dispersion of the values being
attributed to a quantity, based on the information used
standard measurement uncertainty
standard uncertainty
standard deviation of the random variable describing the state of
knowledge about a quantity
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
2
Some dates
• 1977-79 BIPM questionnaire on uncertainties
• 1980 Recommendation INC-1
• 1981 Establishment of WG3 on uncertainties under ISO TAG4: BIPM,
IEC, IFCC, ISO, IUPAC, IUPAP, OIML
• 1981 Recommendation CI-1981
• 1986 Recommendation CI-1986
• 1993 Guide to the expression of uncertainty in measurement (GUM)
• 1995 Reprint with minor corrections
•1997 Establishment of the Joint Committee for Guides in Metrology,
JCGM. ILAC joins in 1998
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
3
Joint Committee for Guides in
Metrology
Present Chair: the BIPM Director
The JCGM has two working groups (WGs)
WG 1 “Expression of uncertainty in measurement”, has the task “to promote
the use of the GUM and to prepare Supplements for its broad application”
WG 2 “on International vocabulary of basic and general terms in
metrology”, has the task “to revise and promote the use of the VIM”
In the following, emphasis will be on documents from WG1
See also http://www.bipm.org/en/committees/jc/jcgm/
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
4
Published documents
Common banner: Evaluation of measurement data
Guide to the expression of uncertainty in measurement. JCGM 100:2008
(GUM 1995 with minor modifications)
Supplement 1 to the “Guide to the expression of uncertainty in
measurement” — Propagation of distributions using a Monte Carlo
method. JCGM 101:2008
Supplement 2 to the “Guide to the expression of uncertainty in
measurement” — Extension to any number of output quantities. JCGM
102:2011
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
5
Published documents II
An introduction to the “Guide to the expression of uncertainty in
measurement” and related documents. JCGM 104:2009
Role of measurement uncertainty in conformity assessment.
JCGM 106:2012
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
6
Documents in preparation
Guide to uncertainty in measurement. JCGM 100:201X
Examples of uncertainty evaluation. JCGM 110:201X
Supplement 3 to the “Guide to the expression of uncertainty in
measurement” — Modelling. JCGM 103:20XX
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
7
Documents planned
Applications of the least-squares method
Concepts and basic principles of uncertainty evaluation
Bayesian methods
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
8
JCGM 104:2009
As the title says, it is a «soft» introduction to the GUM and related
documents, but also to the topic of uncertainty. Formulae are kept to a
minimum.
It should also help the reader to decide which of the JCGM documents
is appropriate for his specific application.
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
9
JCGM 106:2012
This is not a guidance document on evaluating uncertainty, but on using
uncertainty in a specific application, conformity assessment based on
measurement result.
It deals with items having a single scalar property with a requirement
given by one or two tolerance limits, and a binary outcome in which there
are only two possible states of the item, conforming or non-conforming,
and two possible corresponding decisions, accept or reject.
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
10
JCGM 106:2012
This document addresses the technical problem of calculating the
conformance probability and the probabilities of the two types of incorrect
decisions, given a probability density function (PDF) for the measurand,
the tolerance limits and the limits of the acceptance interval.
It considers topics such as decision rules, and the different types of
risk.
It adopts a Bayesian view of measurement uncertainty.
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
11
GUM – JCGM 100:2008 and
100:201X
• First publication in 1993
• Reprint in 1995 with some corrections
• JCGM 100:2008 (free of charge) GUM 1995 with minor modifications
• Until now, a large number of documents based on the GUM has
been written. The GUM has been translated into many languages
• In addition, the GUM has been adopted as a standard, in some
cases as a law, in many countries
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
12
Courtesy BIPM
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
13
http://www.cen.eu/cen/pages/defaul
t.aspx
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
14
Caution
The same uppercase symbol is used with two meanings:
A (physical, chemical…) quantity
The associated random variable used to describe the experimental
outcome
Estimates (i.e., measured values), are denoted by the corresponding
lowercase symbol
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
15
The GUM method
A measurement model is requested, relating the measurand
Y to the input quantities Xi
𝑌 = 𝑓 𝑋1 , 𝑋2 , … , 𝑋𝑁
One seeks the following items of information (𝑖 = 1, … 𝑁):
𝑁 estimates 𝑥𝑖 for the input quantities 𝑋𝑖
𝑁 standard uncertainties 𝑢(𝑥𝑖 )
𝑁(𝑁 − 1)/2 covariances u 𝑥𝑖 , 𝑥𝑗 between pairs of input estimates
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
16
The GUM method II
The estimate 𝑦 of the measurand Y is obtained by «propagating»
the input estimates 𝑥𝑖 through the measurement model
𝑦 = 𝑓 𝑥1 , 𝑥2 , … , 𝑥𝑁
The standard uncertainty about the measurand 𝑢(𝑦) is obtained
by propagating the input uncertainties 𝑢(𝑥𝑖 ) and covariances u
𝑥𝑖 , 𝑥𝑗 through the Law of propagation of uncertainties
𝑁
2 2
𝑐
𝑖=1 𝑖 𝑢
𝑢2 (𝑦) =
where 𝑐𝑖 =
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
𝜕𝑓
𝜕𝑥𝑖
𝑥𝑖 + 2
𝑁−1
𝑖=1
𝑁
𝑗=𝑖+1 𝑐𝑖 𝑐𝑗 u
𝑥𝑖 , 𝑥𝑗 ,
are the sensitivity coefficients.
1st PACMAN Workshop
2-4 February 2015, CERN Genève
17
The GUM method III
The method provides reliable standard uncertainties in the vast
majority, not in the totality of cases
From estimate and standard uncertainty:
conservative coverage intervals for 𝑌 can always be
constructed,
realistic coverage intervals for 𝑌 can be constructed in a
very limited number of cases.
Expanded uncertainty 𝑈(𝑦) = 𝑘𝑢(𝑦) gives no added value
to standard uncertainty, unless the corresponding coverage
probability 𝑝 is known.
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
18
Drawbacks of the GUM method
• No guidance on the (frequent) case of many measurands
• Poor guidance on the construction of a coverage interval (emphasis is
on standard uncertainty), limited to a situation optimistically considered
as frequently occurring
• Other (comparatively minor) weak sides, such as poor consideration to
– non-symmetric distributions
– non-linear measurement models
• The cases above are difficult, probably they had not been considered in
the first edition on purpose
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
19
Are the cases not covered in the
GUM of practical importance?
•
Any calibration of a set of artefacts, be they weights, capacitors,
gauge blocks or similar, is a multivariate case
• The CIPM MRA asks for CMCs at the 95 % coverage probability, i.e,
CMCs are coverage intervals
• Not a few quantities of practical importance are such that the current
practice 𝑈 = 𝑘𝑢 (with typically 𝑘 = 2) is inappropriate
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
20
Remedies
• Difficult problems typically imply difficult solutions
• Coverage interval (and more): JCGM 101:2008 (Supplement 1)
• Multivariate case: JCGM 102:2011 (Supplement 2)
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
21
Supplement 1
The GUM is based on propagation of estimates and uncertainties,
i.e., first and second moments of the random variables 𝑋𝑖, to provide
the corresponding moments of the random variable 𝑌.
This procedure is distribution free, whereas realistic coverage intervals
depend on the probability density function (PDF) 𝑔𝑌(𝜂).
In Supplement 1, 𝑔𝑌(𝜂) is obtained by propagating the PDFs 𝑔𝑿(𝝃)
through the measurement model.
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
22
Supplement 1
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
23
Supplement 1
The (numerical) mechanism is the Monte Carlo method:
From each input PDF draw at random a value 𝑥𝑖 for the random
variable 𝑋𝑖.
Use the resulting vector 𝒙𝑟 (𝑟 = 1, … 𝑀) to evaluate the model, thus
obtaining a corresponding value 𝑦𝑟 . The latter is a possible value for
the measurand 𝑌.
Iterate 𝑀 times the preceding two steps, to obtain 𝑀 values 𝑦𝑟 for 𝑌 .
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
24
Supplement 1
Extensive guidance given on:
assigning PDFs based on available knowledge, and
sampling at random from these PDFs
obtaining a coverage interval from the PDF for 𝑌.
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
25
Example
(from Supplement 1)
Red dotted: GUM
Solid blue: S1
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
26
Supplement 1
Asymmetric PDFs, as well as coverage intervals asymmetric about
the mean, arise naturally
No longer purities 𝑝 > 1,
or fractions 𝑐 < 0!!!
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
27
Supplement 2
Generalizes to the multivariate case:
The law of propagation of uncertainties, and
The construction of a (joint) PDF 𝑔𝒀(𝜼) for the vector measurand 𝒀,
from which
a coverage region for 𝒀 can be obtained.
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
28
Side effects of remedies
• The GUM and its Supplements are now
inconsistent
Why didn’t we write Supplements consistent with the GUM?
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
29
The GUM is ambiguous
The definition of uncertainty in the GUM is
parameter characterizing the dispersion of the values being
attributed to a quantity, based on the information used
This is an intrinsically Bayesian view of uncertainty – uncertainty
concerns the measurand
The definition contrasts with the way in which uncertainty is obtained,
essentially frequentist – uncertainty concerns the measurand estimate,
and is itself uncertain
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
30
Supplements are unambiguous
• In Supplement 1, PDFs (probability density functions) are used to
describe the state of knowledge about each input quantity
• Accordingly, the state of knowledge about the measurand is described
by a PDF obtained from those of the input quantities through the
measurement model (in a way that is not relevant here)
• This is an intrinsically Bayesian attitude, and is consistently adopted
throughout the Supplements
No alternative was possible!
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
31
JCGM 100:201X
• Main purpose: to make the GUM consistent with its Supplements
• Secondary purposes:
• to make it consistent as much as possible with VIM3
• to broaden its applicability to “new” needs
• to minimize notational and terminological ambiguities
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
32
Alignment with Supplements
• Uncertainties (and estimates) are:
– estimates of moments of frequency distributions, in the current GUM (they have
degrees of freedom)
– exact moments of state-of-knowledge distributions, in the Supplements (no
degrees of freedom)
• In the revised GUM JCGM 100:201X, uncertainties (and estimates)
will be exact moments of state-of-knowledge distributions, as in the
Supplements
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
33
Practical impact on standard
uncertainty
• With respect to the current GUM, input standard uncertainties obtained
from a sample of n > 3 repeated indications will be larger by a factor
𝑛 − 1 𝑛 − 3 . When 𝑛 ≤ 3, use of prior knowledge is needed
•
As a consequence, the output standard uncertainty, ceteris paribus, will
change, being anyway consistent with the (uncertain) uncertainty
provided by the current GUM
• Classification into Type A and Type B evaluations loses its scientific
basis – will be kept (de-emphasized) due to non-scientific considerations
• No longer effective degrees of freedom attached to the output
uncertainty - Welch-Satterthwaite formula no longer needed
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
34
Practical impact on coverage
intervals
• In the revised GUM there will be mostly generic guidance on the
construction of coverage intervals, this task being given to
Supplement 1
• Distribution-free coverage intervals, based on Chebyshev or Gauss
inequalities, will be given
• Expanded uncertainty de-emphasized
• Greater consideration to non-symmetric coverage intervals
• Possible impact on KCDB, Appendix C
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
35
Cosmetic changes
• Suffix «c» in the combined standard uncertainty uc dropped (as in
JCGM 101, JCGM 102 and JCGM 106)
• New notation ux allowed as an alternative to u(x)
• Introduction of the hatted symbol 𝑇, say, for the estimate of a
temperature 𝑇 (when appropriate)
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
36
Further notable features
• Increased guidance on the evaluation of input uncertainties
• Guidance on the evaluation of input covariances and on their effect
on the output uncertainty
• Increased guidance on reporting and recording a measurement result
• Clarification of the meaning of loose expressions such as
«uncertainty of…» or «covariance between quantities»
• Enhanced examples. Examples concerning the GUM and its
Supplements will be collected in a separate document, JCGM 110.
Help from users needed!
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
37
If your experiment needs statistics, you
ought to perform a better experiment
(Lord Rutherford)
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
38
Thank you for your attention
ISTITUTO
NAZIONALE
DI RICERCA
METROLOGICA
1st PACMAN Workshop
2-4 February 2015, CERN Genève
39