Nuove frontiere e prospettive di finanziamento
per la ricerca europea nel periodo 2007-2013
Valutazione delle proposte e
probabilità di successo
FP6 IST
Giancarlo Abbate
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
1
Sommario
La valutazione delle proposte Area Tematica del 6 PQ:
Tecnologie per una società dell’informazione
 Le procedure di valutazione
 I criteri ed i sottocriteri di valutazione per i
differenti “strumenti” di finanziamento
 Il processo di selezione delle proposte
 Gli esperti indipendenti. Lettura individuale
Consensus meeting
Lettura incrociata. Panel meeting. Classifica.
Audizione
 Commenti sul processo di selezione
 Paragone con le procedure di valutazione di alcuni
altri programmi.
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
2
Le procedure di valutazione

Procedura a due stadi: esempio FET-Open STREP
ftp://ftp.cordis.lu/pub/ist/docs/fet/int-o-1.pdf
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
3
Le procedure di valutazione

Procedura a due stadi: esempio FET-Open STREP
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
4
Le procedure di valutazione

Procedura a due stadi: esempio FET-Open STREP
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
5
Le procedure di valutazione

Procedura a due stadi: esempio FET-Open STREP
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
6

Procedura a singolo stadio: Esempio FP6-IST2002-2.3.2.2 Optical, opto-electronic, &
photonic functional components
http://www.europa.eu.int/comm/dgs/information_society/e
valuation/pdf/5_y_a/ist_5ya_final_140105.pdf
The scientific and technical evaluation of proposals is carried out by
external experts from the public and private sector. Full proposals are
evaluated, ranked and selected. The IST theme under FP5 and FP6
generally has a high oversubscription of the budget – e.g. the first call for
the IST Priority Theme in FP6 attracted 1400 proposals requesting €
6.2 B, whereas the available budget was only € 1.07 B.
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
7
I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento
Common evaluation criteria for evaluating proposals
A number of evaluation criteria are common to all the
programmes of the Sixth Framework Programme and
are set out in the European Parliament and the Council
Regulations on the Rules for Participation (Article 10).
These are:
a) Scientific and technological excellence and the degree of
innovation;
b) Ability to carry out the indirect action successfully and
to ensure its efficient management, assessed in terms
of resources and competences and including the
organisational modalities foreseen by the participants;
c) Relevance to the objectives of the specific programme;
d) European added value, critical mass of resources
mobilised and contribution to Community policies;
e) Quality of the plan for using and disseminating the
knowledge, potential for promoting innovation, and
clear plans for the management of intellectual property.
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
8
I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento
Furthermore, in applying paragraph (d) above, the
following criteria are also to be taken into account:
a) For networks of excellence, the scope and degree of
the effort to achieve integration and the network’s
capacity to promote excellence beyond its
membership, as well as the prospects of the durable
integration of their research capabilities and
resources after the end of the period covered by the
Community’s financial contribution;
b) For integrated projects, the scale of the ambition of
the objectives and the capacity of the resources to
make a significant contribution to reinforcing
competitiveness or solving societal problems;
c) For integrated initiatives relating to infrastructure,
the prospects of the initiative’s continuing long term
after the end of the period covered by the
Community’s financial contribution.
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
9
I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento






Horizontal issues
Are there gender issues associated with the subject of the
proposal? If so, have they been adequately taken into
account?
Have the applicants identified the potential ethical and/or
safety aspects of the proposed research regarding its
objectives, the methodology and the possible implications of
the results? If so, have they been adequately taken into
account in the preparation of the proposal?
To what extent does the proposal demonstrate a readiness to
engage with actors beyond the research community and the
public as a whole, to help spread awareness and knowledge
and to explore the wider societal implications of the
proposed work?
Have the synergies with education at all levels been clearly
set out?
If third country participation is envisaged in the proposal,
is it well justified and the participation well integrated in the
activities?
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
10
I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento

1. Relevance (threshold score 3 out of 5)

The extent to which the proposed project addresses the objectives
of the work programme.
2. Potential impact (threshold score 3 out of 5)
The extent to which:




the proposed project is suitably ambitious in terms of its strategic
impact on reinforcing competitiveness (including that of SMEs)
or on solving societal problems.
the innovation-related activities and exploitation and/or dissemination
plans are adequate to ensure optimal use of the project results.
the proposal demonstrates a clear added value in carrying out the
work at European level and takes account of research activities at
national level and under European initiatives (e.g. Eureka).
3. S&T excellence (threshold score 4 out of 5)
The extent to which:




13/05/2005
the project has clearly defined objectives.
the objectives represent clear progress beyond the current
state-of-the-art.
•the proposed S&T approach is likely to enable the project to
achieve its objectives in research and innovation.
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
11
I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento
4. Quality of the consortium (threshold score 3 out of 5)
The extent to which:






the participants collectively constitute a consortium of high quality.
the participants are well-suited and committed to the tasks assigned to
them.
there is good complementarity between participants.
the profiles of the participants, including those to be included later, have been
clearly described.
the real involvement of SMEs has been adequately addressed.
5. Quality of the management (threshold score 3 out of 5)
The extent to which:




the organisational structure is well matched to the complexity of the project
and to the degree of integration required.
the project management is demonstrably of high quality.
there is a satisfactory plan for the management of knowledge, of intellectual
property and of other innovation-related activities.
6. Mobilisation of resources (threshold score 3 out of 5)
The extent to which:





•the project mobilises the minimum critical mass of resources (personnel,
equipment, finance…) necessary for success.
the resources are convincingly integrated to form a coherent project.
the overall financial plan for the project is adequate.
Overall threshold score 24 out of 30.
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
12
I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento
1. Relevance (threshold score 3 out of 5)
The extent to which the proposed project addresses the objectives of the work
programme.

2. Potential impact (threshold score 3 out of 5)
The extent to which:






Europe has a strategic need to strengthen S&T excellence on the topic by means
of a restructuring of the existing research capacities and the way research is carried
out.
the goals of the network are, in that connection, suitably ambitious particularly, in
terms of achieving European leadership and acting as a world force on this topic.
the proposal demonstrates a clear added value in carrying out the work at European
level and takes account of research activities at national level and under European
initiatives (e.g. Eureka).
there is an effective plan for spreading excellence, exploiting results and
disseminating knowledge, including to SMEs and to those outside the network.
the proposed approach is likely to have a durable structuring impact on European
research.
3. Excellence of the participants (threshold score 3 out of 5)
The extent to which:




13/05/2005
the participants are currently conducting excellent research relevant to the topic of
the network or are capable of important contributions to the joint programme of
activities.
the participants are well suited to the tasks assigned to them.
they collectively have the necessary critical mass of expertise and resources to
carry out the joint programme of activities successfully.
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
13
I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento
4. Degree of integration and the joint programme of activities
(threshold score 4 out of 5)
The extent to which:




the expected degree of integration justifies supporting the proposal as a
network of excellence.
the joint programme of activities is sufficiently well designed to achieve
the expected degree of integration.
the participating organisations have made a convincing commitment
towards a deep and durable integration continuing beyond the period of
Community support.
5. Organisation and management (threshold score 3 out of 5)
The extent to which:





the organisational structure of the network provides a secure framework
for any necessary structural decisions to be taken
the management of the network is demonstrably of high quality.
there is a well-considered plan for promoting gender equality in the
network.
Overall threshold score 20 out of 25.
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
14
I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento
1. Relevance (threshold score 3 out of 5)
The extent to which the proposed project addresses the
objectives of the work programme.
 2. S&T excellence (threshold score 4 out of 5)
The extent to which:




the project has clearly defined and well focused objectives.
the objectives represent clear progress beyond the current stateof-the-art.
the proposed S&T approach is likely to enable the project to
achieve its objectives in research and innovation
3. Potential impact (threshold score 3 out of 5)
The extent to which:




13/05/2005
the proposed project is likely to have an impact on reinforcing
competitiveness or on solving societal problems.
the proposal demonstrates a clear added value in carrying out the
work at European level and takes account of research activities at
national level and under European initiatives (e.g. Eureka).
exploitation and/or dissemination plans are adequate to ensure
optimal use of the project results.
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
15
I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento
4. Quality of the consortium (threshold score 3 out of 5)
The extent to which:





the participants collectively constitute a consortium of high quality.
the participants are well-suited and committed to the tasks
assigned to them.
there is good complementarity between participants.
the opportunity of involving SMEs has been adequately addressed.
5. Quality of the management (threshold score 3 out of 5)
The extent to which:



the project management is demonstrably of high quality.
there is a satisfactory plan for the management of knowledge, of
intellectual property and of other innovation-related activities.
6. Mobilisation of resources (threshold score 3 out of 5)
The extent to which:





the project foresees the resources (personnel, equipment,
financial…) necessary for success.
the resources are convincingly integrated to form a coherent
project.
the overall financial plan for the project is adequate.
Overall threshold score 21 out of 30.
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
16
Il processo di selezione delle proposte



Quality. Projects selected for funding must
demonstrate a high scientific, technical and
managerial quality in the context of the objectives of
the RTD programme in question.
Transparency. In order to provide a clear
framework for researchers preparing proposals for
funding and for evaluators evaluating proposals the
process of reaching those funding decisions must be
clearly described and available to any interested
party. In addition, adequate feedback must be
provided to proposers on the outcome of the
evaluation of their proposals.
Equality of treatment. A fundamental principle of
EU RTD support is that all proposals should be
treated alike, irrespective of where they originate or
the identity of the proposers.
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
17
Il processo di selezione delle proposte



Impartiality. All proposals are treated
impartially on their merits.
Efficiency and speed. The procedures
have been designed to be as rapid as
possible, commensurate with maintaining
the quality of the evaluation, appropriate
use of public money and respecting the legal
framework within which the specific
programme is managed.
Ethical considerations. Any proposal
which contravenes fundamental ethical
principles may be excluded from being
evaluated or selected at any time.
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
18
Il processo di selezione delle proposte
Appointment of independent experts



Calls for applications from individuals published in the Official
Journal of the European Communities; or
Calls addressed to research institutions with a view to
establishing lists of suitable candidates.
The lists of individuals from which panels of experts
may be chosen, are drawn up by the Commission using
the following selection criteria:





an appropriate range of competencies;
an appropriate balance between academic and industrial
expertise and users;
a reasonable gender balance;
a reasonable distribution of geographical origins of
independent experts;
regular rotation of independent experts.
I valutatori sono retribuiti dalla CE !
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
19
Il processo di selezione delle proposte
Impegni sottoscritti dai valutatori
 Conflict of interest
 Confidentiality

(Independent observers)
Eventualmente ammessi al processo di selezione
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
20
Il processo di selezione delle proposte


Step 1: Briefing of the independent experts
Step 2: Individual evaluation of proposals
Each proposal is evaluated against the applicable criteria
independently by several experts who fill in individual
evaluation forms giving marks and providing comments.
Proposal marking
0 - the proposal fails to address the issue under examination or
can not be judged against the criterion due to missing or
incomplete information
1 - poor
2 - fair
3 - good
4 - very good
5 – excellent
Where appropriate, half marks may be given.
The outcome of this step is the Individual Assessment
Report
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
21
Il processo di selezione delle proposte

Step 3: Consensus meeting
For each proposal a consensus report is prepared. The
report faithfully reflects the views of the independent
experts referred to in Step 2.
Nella (quasi) totalità dei casi il Consensus meeting termina con il
consenso unanime dei valutatori. Il meeting è moderato da uno
“scientific officer” della CE.
The experts attempt to agree on a consensus mark for each
of the blocks of criteria. They justify their marks with
comments suitable for feedback to the proposal
coordinator and agree on an overall consensus report,
which is signed by them.
The outcome of the consensus step is the consensus report
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
22
Il processo di selezione delle proposte

Step 4: Panel evaluation
A panel discussion may be convened, if necessary, to
examine and compare the consensus reports and marks in a
given area, to review the proposals with respect to each other
and, in specific cases (e.g. equal scores) to make
recommendations on a priority order and/or on possible
clustering or combination of proposals. The panel discussion
may include hearings with the proposers.
A Commission official acts as moderator of the Panel



Cross readings
Comparison of consensus reports
grouping of proposals
The outcome of the Panel meeting is
An evaluation summary report for each proposal;
A list of proposals passing thresholds, if any, along with a final
mark for each proposal passing the thresholds and the panel
recommendations for priority order.
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
23
Il processo di selezione delle proposte

Hearings (Audizione)
Per i nuovi strumenti del 6° PQ è prevista un’audizione
individuale per ogni proposta che ha superato la soglia.
Il panel di cui al punto precedente si riunisce
(eventualmente in una composizione ridotta) ancora
una volta (dopo un tempo opportuno, cioè circa un
mese) e, al termine delle audizioni, propone la
graduatoria di merito finale.

Final steps
Reporting on the evaluation process
Feedback to the proposers
Negotiation
Selection of proposals
These final steps do not involve the independent experts
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
24
Commenti sul processo di selezione


Il mio personale commento è che il processo di
selezione raggiunge tutti gli scopi che si prefigge ed è il
più efficiente ed obiettivo tra i vari processi a cui ho
partecipato come soggetto attivo e/o passivo.
Alcuni feedback da valutatori IST:
It was fascinating to see how such a complex problem, namely
how to rank such a large number of disparate proposals,
could be solved, and it was my pleasure to participate in the
process.
I'm glad that finally we made it! It was hard working, as you
say, but I really enjoyed it: I enjoyed working with clever and
skilled people as evaluators and very professionals people in
the EU commission. I enjoyed the international involvement
and the possibility of knowing interesting people in the field
from all over Europe. I enjoyed the possibility of having an
overview of top level research people is now working on.
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
25
Commenti sul processo di selezione
I learned a lot of very important things about science, present
and future trends, and mainly how a good proposal should be
prepared and how it should not
It was great experience to take part in EC Project evaluations for
me. I have learned how to work hard, precisely and
effectively and without any external pressures
I wish to say that I was very glad to participate in evaluations of
IST proposals. Not only because of the scientific topic which is
extremely interesting for me but also because of perfect
organisation, competence of your team and friendly
atmosphere during evaluation. Thank you very much
I have been already in Growth panels in the past and had the
opportunity to realize that the commission do everything is
possible to be fair and objective when evaluating proposals
and distributing public money. I do believe that IST staff is
the paradigm in that manner
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
26
Paragone con le procedure di valutazione di alcuni altri programmi

INTAS
The International Association for the Promotion of Co-operation
with Scientists from the New Independent States (NIS) of the
Former Soviet Union

COST
Founded in 1971, COST is an intergovernmental framework for
European CO-operation in the field of Scientific and Technical
Research, allowing the co-ordination of nationally funded
research on a European level. COST Actions cover basic and
pre-competitive research as well as activities of public utility
Running Actions ~ 200

Total Budget (2005) € 13,500,000
MIUR - PRIN
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
27
13/05/2005
G. Abbate Dip. di Sc. Fisiche
Università di Napoli Federico II
28