Data & Methods

SIPER provides three types of publicly accessible data for each of the coded evaluation reports:

• a basic classification of the policy measure / instrument including its target, modality and objectives
• basic information on each evaluation report such as title, author(s), language or country
• a factual characterisation of elements that can be inferred from the reports themselves, e.g. methods, timing and topics.

The evaluations and associated documents available via SIPER have been identified and accessed using a semi-structured search process involving targeting websites and approaching the relevant individuals in regional and national ministries and agencies, international and supranational organisations and leading organisations in STI policy evaluation. The OECD STIP Compass now also includes a question about whether an evaluation report is available for a collected measure.

Several criteria are applied to ensure that the identified evaluations are suitable for applying the coding process. To be included, an evaluation should:

• contain a systematic determination of the subject's merit, worth and significance
• relate directly to a science and innovation policy instrument or measure, i.e. all public interventions that support science and innovation activities (not limited to dedicated science and innovation ministries/agencies)
• evaluate a specific policy instrument or group of instruments
• have a distinguishable methodology and
• provide evidence regarding its performance.

As shown in the chart below, constructing SIPER and keeping it up to date requires a multi-stage procedure:

1. Collecting science and innovation policy evaluations from a wide range of sources, e.g. websites of ministries, government or other agencies, national and supranational organisations as well as those of leading evaluation practitioners.
2. Performing the factual characterisation for each report so that each evaluation report is searchable by these characteristics (e.g. timing and methods of the evaluation, modality of support and objectives of the policy measure) via the web interface.
3. Uploading the characterisation and the relevant document(s) to the database.

To ensure data quality, we apply the standardised coding template and coder manual developed during RISIS 1. Furthermore, to ensure inter-coder reliability, we apply a double coding process by different team members to a random selection of reports.