Evaluation Units Open days, Tito Bianchi, 3/07/2006
Evaluation of large Scale Industrial
projects
Evaluation Units’ Open Days
Rome, 3/06/2006
Tito Bianchi
Public Investment Evaluation Unit (UVAL)
Department of Development Policies
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
INDEX OF THE PRESENTATION
1. Evaluation mandate
2. Features of the policy to be evaluated
3. Evaluation Method
4. First results
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
1. Evaluation mandate I: Debate
• Mounting criticism of subsidies to private
firm investment
• Declining resources for industrial
subsidies
• Negotiated/contractual approach to large
scale industrial promotion disputes
resources to “objective-automatic”
subsidies
• Drafting of new 2007-2013 CSF
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
Evaluation mandate II: institutional
request for Evaluation
In 2003 CIPE - Ministerial Committee for
economic planning - debates over the
effectiveness of subsidies to firms…
… asks UVAL and the network of evaluation
Units to conduct an evaluation of incentives for
industrial promotion “to base future decisionmaking” (Decree N.16/03)
…UVAL chooses to evaluate Contratti di
Programma (Negotiated contracts w/large firms)
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
2. Features of the policy to be evaluated
Contratto di Programma (CdP):
Bi-lateral agreement
Central Govt 
Large industrial firm
Subsidizes Firm’s
Commits to:
Investment Plan
Employment levels
Research activities
Destination of equipment
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
2. Policy Features cont’d: “Contratti di
programma” (CdPs)
 Funded by domestic fund for lagging
regions
 Started 1987, continuing to present
 Originally funded only large scale
industrial firms..
 .. Recently extended to agri-food and
tourism
 ....Extended to consortia of smaller firms;
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
Policy Features: Number, size and sector
CdPs approved at december 2005
N.
Public resources
awarded
(000 Euro)
Average
public
subsidy
%
Expendi
ture
First generation (1987-1996)
16
1.722.510
107.657
83,1
Second generation (1996-98)
14
1.158.501
82.750
51,8
Present generation (1998-present)
85
9.704.499
114.171
12,8
In present generation, Food-processing chains
and tourism consortia are the majority. 40 such
contracts are only approved by CIPE, not yet
signed.
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
Policy features: selection process
unclear
• Lack of a transparency of rules along the
pipeline of ex-ante evaluation, approval
• Timing uncertain, proposed CdPs are
never rejected
• Selection criteria specified allow large
discretion
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
What should be the future of CdPs
“negotiated-discretionary” industrial
subsidies?
• Regions have started to fund similar
projects, claim this policy tool is in their
responsibility
• Attraction of foreign firms remains a priority
in the new draft CSF
• Eastern EU outcompetes Italy in terms of
labor costs and higher State Aid Ceilings
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
3. Evaluation Method: Technical
Steering Committee
A 8-member steering committee appointed
itself including representatives of UVAL, the
Network of evaluation Units, the Ev. Unit of
the Ministry of Industry, 1 expert of industrial
policy.
…it was put in charge of designing evaluation
activities, conducting the evaluation, reporting
to CIPE
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
3. Evaluation Method: Specifying
research questions
Impact Evaluation was specified as:
1. Additionality; would the private investment
have been done w/out subsidies? Of the
same size, in the same place?
2. Was there a sustained economic benefit for
the area receiveing the investment?
3. What was the national level impact (on the
industrial sector)?
4. Relative impact of CdPs involving consortia
vs. Individual large firms.
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
3. Evaluation Method: choice of an
Intermediate option
Number of Cases
0
.. 5 ...
10
... hundreds... thousands
In depth
CaseStudy
Quantitative study
Comparative
case-based
evaluation
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
3. Evaluation Method: components of
the evaluation project
1. Main study: comparative study of 9 cases
of CdPs, based on data, documents and
interviews
2. Additional case-study (FIAT) – outlier
largest and most studied – based only on
secondary sources
3. Data analisys of recent CdPs funding
consortia, testing internal coherence of
projects
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
3. Evaluation Method: choice of cases for
the main study, pragmatic segmentation
1. We picked among those (~40) that had
finished the investment plan (first 2 genr.)
2. Included two Consortia, the first to be
financed
3. Covering 7 regions of the South
4. Eight diferent Industrial sectors:
Electronics (2), Textiles, Shoes, Autoparts, Food-processing, Oil refining,
Jewelry, Chemicals
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
3. Evaluation Method: contracting
evaluation experts
hired 9 experts in the evaluation of industrial
policies through a competitive process:
1. Publication on DPS website of call for bids
2. Experts fees set at € 20.000 for each study
3. Candidate experts were evaluated based on CV
and on a two-page proposal
4. Each selected to evaluate specific identified CdP
5. Common meetings have discussed preliminary
evaluation findings, shared methodology
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
3. Evaluation Method: Comparative
Analisys
• We did not impose paragraph structure but
asked to answer the selected questions
• Asked to calculate minimum set of
indicators: firm, contract, and territory.
• Encouraged to pursue non-considered
research hypotheses, calculate additional
indicators
• Currently steering group is writing
comparative part
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
4. Preliminary Findings
 CdPs fare better than the reputation they have
 Research activies of firms often discontinued,
they were unrealistic or non serious
 Manufacturing plants can take root in the area,
but never had strong generation effects
 Employment goals, especially the purpose to
mitigate localized employment crises, are
weakening factors
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
4. Lessons about evaluation
 Skills for the evaluation of industrial
policies are not abundant
 Windows of opportunity to evaluate
policies arise, have to be timely
exploited
 Managing a comparative multiplestudy evaluation of this kind has
advantages and disadvantages
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp
Evaluation Units Open days, Tito Bianchi, 3/07/2006
4. Lessons for Evaluation: ++ and -- of
comparative case-study method
+
 Involves larger number
of independent
intelligences
 The process itself, not
only the final product,
contributes to a shared
view
 Virtuous competition
b/ween experts
_
 Risk to Lose control of
timing
 Pursuit of consensus
b/ween different views
may reduce the
sharpness of the
method, of the
message
Dipartimento per le Politiche di Sviluppo del Ministero dell’Economia e delle Finanze
s
dp