Data and methods
The SIPER database of characterised evaluations has three types of publicly accessible data:
- Policy measure characterisation: a basic three layer classification of the related policy measures (using a novel typology developed by the SIPER team — see below).
- Basic information: evaluation title, author, language, country, related files, etc.
- Factual Characterisation: characteristics that can be inferred from evaluation reports themselves (methods, timing, topics, etc.).
The evaluations covered
The evaluations and associated documents available through SIPER have been identified and accessed through a semi-structured search process involving targeting websites and approaching relevant individuals in regional and national ministries and agencies, international and supranational organisations and leading organisations in STI policy evaluation.
A number of criteria have been applied to characterise the identified evaluations in order to ensure that they are appropriate for the application of our characterisation process. However, we wish to emphasise that inclusion or non-inclusion of an evaluation in the SIPER database is in no way associated with any judgement of the 'quality' (howsoever that may be defined) of the evaluation or measure to which it refers. To be included, an evaluation should:
- Contain a systematic determination of a subject's merit, worth and significance, (generally using criteria governed by a set of standards)
- Relate directly to a science and innovation policy instrument or measure (i.e. all public intervention that supports science and innovation activities, not just confined to dedicated science and innovation ministries/agencies)
- Evaluate a specific policy instrument or group of instruments
- Have a distinguishable methodology
- Provide some sort of evidence of performance
How do we characterise related policy measures?
We characterise a science and innovation policy measure or instrument by locating it in a virtual three dimensional space according to the following three main elements or attributes:
- The target of the measure (i.e. recipient or primary beneficiary of the support) — 10 categories
- The modality (i.e. how the support is provided) — 7 categories
- The measure's policy objectives (i.e. the principal rationale for the support) — 16 categories
How is SIPER constructed?
Step 1 Collecting science and innovation policy evaluations from a wide range of sources across the world;
Step 2 Uploading evaluation documents into the SIPER repository;
Step 3 Characterising the factual information (timing, topics, methods, designs, etc.) of each evaluation and its related policy measure (target group, modality, objectives, country, etc.) so that evaluations are searchable by these characteristics, by any user through a simple web interface.
Step 4 Assessing the evaluations by policymakers: in a later stage of the process, SIPER will add a second layer of characterisation, asking policy makers who have commissioned evaluations in the last 3 years to assess these evaluations. This will be done through interviews with the core research team. This characterisation will not be made public, and is purely for research purposes. Any academic publications based on this research will make sure that no identification of specific evaluations and assessments will be possible, all will be anonymized.
How do we ensure the characterisation is correct?
All characterisations are done by evaluation experts or researchers trained by experienced evaluators following strict explicit guidelines and quality controls. We have developed detailed internal manuals for coders, and thorough protocols for recruiting external partners. We ensure inter-coder reliability through triangulation between coders.