OTKA Case Study

Supporting a public research funding body with an independent evaluation of its strategy, processes and impact involving the survey of a representative sample of over 800 funded researchers and a review by an international high-level Evaluation Committee

 

Introduction

The Hungarian Scientific Research Fund (OTKA) was established in 1986 with a mission to provide support to outstanding discovery-orientated (basic) research in all fields of science carried out in Hungarian institutes.  The main institutional recipients of OTKA funding include universities and colleges, as well as the research institutes of the Hungarian Academy of Sciences. It now operates under the auspices of The National Office for Research, Development and Innovation.

The Challenge

As a Member Organisation of the ESF, OTKA commissioned ESF to coordinate its first international review and to examine its activities over a five-year period: 2009 – 2013. 

The evaluation had the overall goal of identifying strengths and recommendations to further enhance OTKA’s governance and management structures.

The scope of the evaluation included assessment of OTKA’s governance and management structures and its funding portfolio, to review its coherence with OTKA’s strategic goals, its internationalisation, and the efficiency and transparency of procedures around OTKA’s funding schemes.

The solution

Carrying out a review of this nature required adopting a rigorous and comprehensive approach, including careful scoping of the requirement and detailed process planning.

The evaluation design featured a mixed-methods approach including discovery meetings, expert peer review, desk research and an outcomes survey of OTKA’s grant applicants (both successful and unsuccessful). 

The survey measured satisfaction levels of OTKA’s applicants with grant administration and procedures. It also assessed the impact of OTKA research funding and the internationalisation of OTKA funded research. A response rate of 68% was achieved from a randomised sample. This permitted a high level of confidence in the findings and their overall statistical reliability across the entire group of applicants.

In parallel, an international expert panel was established by ESF to review OTKA’s funding portfolio, governance, strategy and procedures. The panel visited OTKA and held meetings with OTKA governance representatives, staff and clients, as well as other important external stakeholder organisations. The panel also analysed OTKA’s self-evaluation report, as well as reference/background documents provided by OTKA and ESF.  A final report was then produced.

The results

The evaluation demonstrated that OTKA was a well-managed funding body for basic research and a valuable asset in the Hungarian research system that could be more intensively utilised. 

A series of detailed findings and recommendations were provided to OTKA to facilitate its future development. The report identified that, while OTKA has a high degree of political autonomy, it is somewhat distanced from the policy system and would benefit from potentially greater integration and an enhanced level of communication among key stakeholders and R&D agencies. This would also enable OTKA to develop multi-annual strategies of long term benefit to Hungarian research. 

Further recommendations were made around OTKA’s funding portfolio, the grants awarding system, the development of doctoral students and early stage researchers, researcher mobility, grant outcomes and applicant perceptions of the process.  These involved enhancing organisational and operational systems which were deemed overall to be of existing high quality.

The conclusions from the independent Evaluation Committee review and the survey were very positive in terms of OTKA’s governance, administrative procedures and its potential for an expanded role in the Hungarian research system. 

The findings enabled OTKA to concentrate further on areas where it performs very strongly and areas where further enhancement would be of long term benefit.

This was a very important process which has supported OTKA in its recent evolution, where it now operates under the auspices of The National Office for Research, Development and Innovation.

Testimonial

“The Hungarian Scientific Research Fund (OTKA) is pleased to acknowledge the excellent service provided by the European Science Foundation (ESF) in conducting its international evaluation within the general context of Hungarian and European public research funding. ESF established an evaluation protocol including appropriate evaluation criteria and methodology, as well as a detailed work plan in consultation with OTKA. The implementation of an independent evaluation study by a committee of experienced R&D executives and researchers, along with the expert assistance from dedicated ESF staff, led to the publication of the Organisational Evaluation of the Hungarian Scientific Research Fund– a report that will serve as a point of reference for a long time to come.”

László P. Kollár, President, Hungarian Scientific Research Fund (OTKA)

DOWNLOAD