SIPER provides three types of publicly
accessible data for each of the coded evaluation reports:
- a basic classification of the policy measure / instrument including its
target, modality and objectives
- basic
information on each evaluation report such as title, author(s), language
or country
- a factual characterisation of elements that can be
inferred from the reports themselves, e.g. methods, timing and topics.
The evaluations and
associated documents available via SIPER have been identified and accessed using
a semi-structured search process involving targeting websites and approaching the
relevant individuals in regional and national ministries and agencies,
international and supranational organisations and leading organisations in STI
policy evaluation. The OECD STIP Compass now also includes a
question about whether an evaluation report is available for a collected
measure.
Several criteria are applied to ensure that the
identified evaluations are suitable for applying the coding process. To be
included, an evaluation should:
- contain
a systematic determination of the subject's merit, worth and significance
- relate
directly to a science and innovation policy instrument or measure, i.e.
all public interventions that support science and innovation activities
(not limited to dedicated science and innovation ministries/agencies)
- evaluate
a specific policy instrument or group of instruments
- have
a distinguishable methodology and
- provide evidence regarding its performance.
As shown in the chart below, constructing SIPER
and keeping it up to date requires a multi-stage procedure:
- Collecting
science and innovation policy evaluations from a wide range of sources,
e.g. websites of ministries, government or other agencies, national and
supranational organisations as well as those of leading evaluation
practitioners.
- Performing
the factual characterisation for each report so that each evaluation
report is searchable by these characteristics (e.g. timing and methods of
the evaluation, modality of support and objectives of the policy measure)
via the web interface.
- Uploading
the characterisation and the relevant document(s) to the database.
To ensure data quality, we apply the standardised
coding template and coder manual developed during RISIS 1. Furthermore, to
ensure inter-coder reliability, we apply a double coding process by different
team members to a random selection of reports.