1. INTRODUCTION
Doctors of the World is an international and independent solidarity association that promotes human
development through the fundamental right to health and dignified life for all people, especially for
vulnerable populations, excluded or victims of natural disasters, famines, diseases, armed conflicts, or
political violence.
MDM has been present in Ukraine since 2015 responding to the needs of the affected population by
the conflict in the eastern part of the country. After the military offensive of the Russian Federation
and the escalation of the war, MdM scaled up its operations and presence in the country.
The objective of this document is to establish the Terms of Reference of the final evaluation of the
project " To ensure access to comprehensive healthcare services for conflict-affected population,
including most vulnerable groups. (elderly, people with disabilities, GBV survivors) both host and
internal displaced population in highly affected areas" implemented in Ukraine and financed by ECHO.
The project is implemented in consortium within 3 chapters of the MdM International Network: MdM
Spain (leading partner), MdM Germany and MdM France.
2. DESCRIPTION OF THE INTERVENTION
The intervention ensures access to quality primary health care services (including SRH and MHPSS) in
the eastern and southern parts of Ukraine, in collaboration with the local and regional health
authorities, the program includes direct provision of PHC (including SRH) and MHPSS services and
support to health facilities (including donations, financial aid, trainings, rehabilitation, etc.). The project
included a CM (Crisis Modifier) as Result 3 that was not activated.
Objectives and Results of the Project
The Project has the following outcomes and Results:
Project Outcome: To ensure access to comprehensive healthcare services for conflict-affected
population, including most vulnerable groups (elderly, people with disabilities, GBV survivors), both
host and internally displaced population in highly affected areas
Result 1: Healthcare services are functional, accessible, and reachable for conflict-affected
populations.
Result 2: Affected population accessing to MHPSS through the direct provision of services by adapted
modalities.
Result 3: Crisis Modifier: Emergency assistance to ensure access to healthcare in areas at risk in newly
accessible areas and IDP hosting areas.
TDR Evaluation " To ensure access to comprehensive healthcare services for conflict-affected population, including most vulnerable groups.
(elderly, people with disabilities, GBV survivors) both host and internal displaced population in highly affected areas”
3. PURPOSE AND OBJECTIVES OF THE EVALUATION
The purpose of the external evaluation is on one hand to determine the extent to which the project
targets are met at output and outcome levels, to identify unexpected or unplanned issues that may
have hindered or facilitated the success of the project, and on the other hand, to provide MdM and
ECHO with detailed and in-depth information about the relevance, effectiveness and efficiency of the
project, to document lessons learned and to provide practical recommendations for follow-up
questions.
The specific objectives of the Evaluation include:
Assessing whether the key outputs and outcomes were well identified and designed to
positively affect targeted population
Assessing project achievements for affected populations/communities and health facilities.
The evaluation framework is built upon the Core Humanitarian standards and ALNAP evaluating
humanitarian action using the following OECD-DAC criteria: Relevance and appropriateness;
Connectedness; Coherence; Coverage; Efficiency and Effectiveness; and Impact.
The evaluation is expected to employ a “mixed methods approach” that combines quantitative and
qualitative techniques.
The main requirement against the methodology of the current evaluation is that it should be
representative, which means that consultations with all stakeholders should be undertaken
considering such criteria as gender, age, disability, and other consultations with all stakeholders should
be undertaken considering such criteria as gender, age, disability, and other vulnerability. The
evaluation is expected to involve men/women, partners and stakeholders, field visits, and review of
program documents and program data. Data collection techniques may include desk reviews, key
informant interviews, focus group discussions, satisfaction surveys and observations.
The objective of the evaluation is to obtain an independent assessment of the intervention, thus
assessing relevance and coherence concerning the context and needs of rightsholders, as well as how
to analyze and extract lessons and learning elements about the implementation model and its
effectiveness and efficiency, so as the degree of incorporation of cross-cutting approaches.
The evaluation will also provide information on good practices identified during the implementation
of the Project, recommendations to guide, among other issues, future strategic programming and
lessons learned that will allow, through their socialization, to strengthen the MdM teams and the
partner entities of the Project.
TDR Evaluation " To ensure access to comprehensive healthcare services for conflict-affected population, including most vulnerable groups.
(elderly, people with disabilities, GBV survivors) both host and internal displaced population in highly affected areas”
The outputs of the evaluation will be used as inputs that feed the processes of reflection and decisionmaking
on the planning, management and implementation of projects and their link with the MdM
Strategy in Ukraine, and they will also be shared with partners and donors.
4. SCOPE OF THE EVALUATION
Type of evaluation
This is an external ex-post evaluation, which will assess the degree of compliance with the results and
objectives, as well as the evaluation criteria of the DAC (OECD Development Assistance Committee).
Strategic and programmatic dimension
All the components of the Project will be considered both at the level of its Objectives, Results and
Activities, as well as from its methodologies and work approaches.
Geographical dimension
The project has been developed in the southern and eastern parts of Ukraine. Specifically in Mykolaiv,
Dnipro, Zaporizhzhia and Kharkiv Oblast.
Temporal dimension
The time horizon of the evaluation corresponds to the period between 01.03.2023 and 30.04.2024.
This period covers the entire implementation of the project from its start-up to its completion.
Actors involved.
The main actors involved in the evaluation process are:
- MdM Spain, MdM Germany and MdM France both field and HQ teams
- Sub-implementing local partners: Charitable organization 100% Life Dnipro and Mykolaiv Regional
organization of the Ukrainian Red Cross Society.
- Public health structures collaborating with MdM.
5. EVALUATION CRITERIA AND QUESTIONS
The evaluation process will have two levels of analysis: a global level, with a strategic vision of the
Project as a whole, and an operational level.
At both levels, the evaluation will incorporate a comprehensive and systemic analysis, which assesses
the key aspects of the context, design, structure, processes, and results, in a way that makes visible
and explains the relationships that exist between the different dimensions and facilitates an integrated
analysis. It is not a question of approaching all dimensions with the same depth, but of answering the
TDR Evaluation " To ensure access to comprehensive healthcare services for conflict-affected population, including most vulnerable groups.
(elderly, people with disabilities, GBV survivors) both host and internal displaced population in highly affected areas”
objectives and questions of the evaluation by analyzing how the different dimensions relate and
influence each other.
The evaluation must answer the following questions organized according to the criteria established for
the evaluation of the Project. However, the advisory team may specify additional questions as deemed
appropriate. The evaluation questions shall structure the analysis and the evaluation report, which
shall provide a clear answer to each of them:
RELEVANCE, APPROPIATENESS, AND COHERENCE: It will assess the extent to which the objectives and
design of the intervention respond to the needs, policies and priorities of the beneficiary population,
partners/institutions, and the country, as well as global priorities, and continue to do so even as
circumstances change. Specifically, the following questions will be answered:
- Is the intervention proposed consistent with the priorities established in response to the
conflict aligned and coordinated with other programs or projects launched by other
organizations?
- To what extend the project targeted the most affected population and geographical areas of
the country?
- Is the intervention relevant to existing health authorities’ mechanisms at local and regional
levels?
- Are the objectives and design of the project formulated according to the needs, demands and
priorities of the community, healthcare professionals and patients?
- Does the project have a coherent intervention logic between the objectives of the project, and
the instruments proposed to achieve it?
CONNECTEDNESS: The coordination among the implementing partners, consortium partners and
external stakeholders will be assessed, specifically the following questions will be answered:
- Has the project been implemented with appropriate and effective inter-agency and
partnership strategies? What has been the nature and added value of these partnerships?
- Has the project built on/matched the individual and local capacities/needs?
EFFICIENCY: The results obtained about the processes carried out, the activities carried out and the
resources mobilized (material, financial and human resources) will be valued. Specifically, the
following questions will be answered:
- Have the budget and schedule initially established in the document been respected?
- Have the resources (funds, HR, time, etc.) been efficiently used to achieve the relevant
outputs?
TDR Evaluation " To ensure access to comprehensive healthcare services for conflict-affected population, including most vulnerable groups.
(elderly, people with disabilities, GBV survivors) both host and internal displaced population in highly affected areas”
EFFECTIVENESS: The degree of achievement of the outcome and the expected results of the Project
through the activities carried out will be assessed. Specifically, the following questions will be
answered:
- What factors have influenced what will or will not achieve the expected results?
- What are the main challenges MdM faces in achieving the goals?
IMPACT: The effects of the project will be assessed, specifically the following questions will be
answered:
- What was the general effect (both positive and negative) of the project on its direct and
indirect beneficiaries?
CROSS-CUTTING APPROACHES. GENDER ANDINCLUSIVE APPROACH:
To what extent was the contribution of the project towards cross-cutting approaches, specifically the
following questions will be answered:
- To what extent does the project contribute to gender equality and non-discrimination?
- To what extent does the project contribute to the inclusion of people with disabilities?
To what extent the project's environmental effects were considered? Specifically: medical-waste
management policies/tools implemented
6. METHODOLOGY
The assessment shall be based on an established methodology with a rigorous design. The prosecution
of the object of this evaluation will be articulated in a systematized and logical manner. The
methodological approach that will guide the evaluation should be indicated and should be reflected in
the proposal.
Preferably the fieldwork will be carried out in the places of execution of the project, there is the
possibility of considering the realization of the fieldwork remotely, in exceptional cases only, provided
that an adequate methodology is presented and adapted to the situation and respect the required
quality standards.
7. EVALUATION MANAGEMENT
The evaluation will be carried out by ensuring the participation of the main actors concerned in the
object of the evaluation.
TDR Evaluation " To ensure access to comprehensive healthcare services for conflict-affected population, including most vulnerable groups.
(elderly, people with disabilities, GBV survivors) both host and internal displaced population in highly affected areas”
For the management of the evaluation, a Monitoring Committee will be created composed of
representatives of MdM Spain from the Headquarters and the Field Team and by the MEAL Unit of
MdM Spain with the following functions:
- Design and approve terms of reference.
- Select the evaluation team.
- Advise and supervise the evaluation methodologically.
- Provide the evaluation team with access to all relevant information.
- Maintain a permanent dialogue with the evaluation team.
- Carry out quality control and ensure compliance with deadlines.
- Validate the different products of the evaluation,
- Facilitate the dissemination of evaluation results.
8. WORK PLAN AND SCHEDULE
Deadlines
The work will be divided into cabinet phase, fieldwork and return of results:
Cabinet phase: approximate duration of two weeks. May 2024
Fieldwork: approximate duration of four weeks. May/June2024
Delivery of the draft of the Final Report. June 2024
Delivery of the Final Report final version with the comments of the Monitoring Committee:
June 2024
Work plan
The planned schedule for the evaluation process has been estimated for an approximate period of 8
weeks from the formalization of the contract and will follow the following distribution of phases, tasks,
and products:
PHASES TASKS AND PRODUCTS
Phase I: Preparatory activities
and cabinet study
- Initial meeting of the evaluation team with the Monitoring
Committee, first collection of information and adjustment of
7 days
the order.
- Identification and analysis of the documentation related to
the Project
- Selection of stakeholders to be interviewed, from those
selected at least 50% must be local stakeholders
TDR Evaluation " To ensure access to comprehensive healthcare services for conflict-affected population, including most vulnerable groups.
(elderly, people with disabilities, GBV survivors) both host and internal displaced population in highly affected areas”
(subimplementing local partners and beneficiaries of the
action).
Drafting of the Inception Report.
Phase II. Fieldwork
1 month
- Fieldwork in Ukraine: application of methodological
techniques and tools designed for the collection of
information.
Considering the geographical scope of the Project and the
difficulty of mobility in the different places of action, it is
proposed that the fieldwork be carried out in Dnipro and
including at least 2 more geographical areas of intervention
(Mikolaivka and/or Zaporizhzhia and/or Kharkiv) that would
be agreed in within the evaluator and MdM according ot the
security situation. :
- team and ToD interviews
- interviews with partners and other key actors
Phase III. Analysis and
interpretation of the
- In-depth analysis and interpretation of the information
collected.
information, preparation, and
presentation of the draft of the
final report
- Writing a first draft report in English
- Conducting a validation session of the findings and
conclusions
21 days
- Integration of observations and comments to the report,
channelled by the Monitoring Committee.
- Preparation of the final report in English
- Preparation of a presentation (PPT) summary of the
evaluation in English.
- Presentation of the final report via Teams to MdM and other
stakeholders, in a 1-hour session, in English.
The indicative planning can be modified at any time always with the approval of the Monitoring
Committee depending on the evolution of the context and in particular of the security.
TDR Evaluation " To ensure access to comprehensive healthcare services for conflict-affected population, including most vulnerable groups.
(elderly, people with disabilities, GBV survivors) both host and internal displaced population in highly affected areas”
Expected products
1.- Draft of the Final Report: it must contain the key aspects of the evaluation carried out and a first
assessment of the questions and criteria raised. Doctors of the world will make the appropriate
comments to this draft to reach a consensus. This process can generate more than one version of the
draft report.
2.- Final Report. The final document must respond to all the questions raised following the format
indicated in this ToR.
3.- Presentations of the Final Report: a presentation of the results will be made on dates to be agreed
upon.
The evaluation will lead to the completion of a report, a version in English, of a maximum length of 60
pages, with annexes not included. This shall be accompanied by an executive summary of a maximum
of 5 pages which shall appear at the beginning of the Report. Finally, a summary of the evaluation must
be submitted following the format established by the OECD DAC for the inventory of evaluations of
this institution.
All evaluation products will be delivered to the Monitoring Committee in digital format in Word (or
other editable programs).
Report Format
The format of the report referred to below will therefore be strictly respected:
1. Cover page:
- Title of the evaluation report: (operation, counterpart, country, cooperation sector(s))
- Evaluation period.
- Name of the evaluator.
- Express mention that the report has been prepared at the request of MdM within the
framework of a project funded by ECHO and that the comments found reflect only
the opinion of the evaluator.
2. Index of subjects.
3. Executive Summary:
- Name of partner organizations.
- Objective of the evaluated operation.
- Title of the operation.
- Number of the Operation Contract.
- Country(s) of the operation.
TDR Evaluation " To ensure access to comprehensive healthcare services for conflict-affected population, including most vulnerable groups.
(elderly, people with disabilities, GBV survivors) both host and internal displaced population in highly affected areas”
- Duration of the operation.
- Purposes and duration of the evaluation.
- Method used: analyzed documents, talk show, etc.;
- Main conclusions and recommendations in order of priority
- (Maximum 5 pages)
4. The main body of the report shall be structured by the specific objectives set out in point 5.
4.1 Executive summary
4.2 Introduction
o Background and Objective of the evaluation.
o Main questions and value criteria: definition.
4.3 Summary description of the intervention evaluated.
4.4 Methodology used in the evaluation
o Methodology and techniques applied.
o Conditions and limits of the study carried out.
5. Analysis of the information collected and evidence around the previously established issues.
Interpretation of the evidence about the evaluation questions stated.
6. Conclusions of the evaluation about the established evaluation criteria.
7. Lessons learned that emerge from general conclusions indicating good practices and that can be
extrapolated and feedback on the actions of the intervention in progress or for future
interventions, as appropriate.
8. The recommendations derived from the evaluation are classified according to the criteria chosen
by the evaluation team
9. Annexes containing:
Evaluation ToRs
The work plan, composition, and description of the mission.
Proposed methodology, techniques and sources used to collect information
Documentary review: list of secondary sources used.
Interviews: list of informants, interview guide, transcripts, and notes.
Surveys: models, raw data collected and statistical analysis.
TDR Evaluation " To ensure access to comprehensive healthcare services for conflict-affected population, including most vulnerable groups.
(elderly, people with disabilities, GBV survivors) both host and internal displaced population in highly affected areas”
Allegations and comments from different actors to the draft report if it is considered
relevant, especially if there are disagreements and have not been reflected in the body of
the report. (See premise of responsibility).
Summary of the evaluation
After its presentation in electronic format and once approved, the team will deliver the final version
of the Final Report, with the document in electronic format.
The Final Report must be validated by the Monitoring Committee (Technician responsible for the IPR
department project and the DPE Quality and Evaluation Unit).
9. PREMISES OF THE EVALUATION
The evaluation will be guided by the OECD/DAC Quality Standards for Development Assessment and
the Physicians of the World Monitoring and Evaluation Framework. Throughout the evaluation
process, the evaluation team must observe the following premises:
INDEPENDENCE: The members of the evaluation teams must be able to work freely and without
interference and will not be subject to pressure to modify the content of their reports or evaluations.
To ensure the independence of the evaluation, the members of the external evaluation team may
not have maintained an employment relationship with the grant beneficiary entity (MdM) or its
counterpart in the framework of this intervention and the mission in Ukraine, at least during the two
years before the recruitment proposal for the performance of the evaluation, nor have been linked
to the design, management or execution of the intervention to be evaluated.
IMPARTIALITY AND ETHICS: To ensure impartiality, it must be ensured that evaluation teams are
selected with due concurrence, have sufficient knowledge and experience to perform the tasks
entrusted and are free of bias. In turn, the members of the evaluation units and teams must previously
communicate possible conflicts of interest, provide respectful and non-discriminatory treatment to all
individuals and groups involved in the evaluation processes, and perform their tasks with integrity and
honesty.
TRANSPARENCY: Clear communication regarding the purpose, key questions, and intended uses of
evaluation results will maximize the transparency of evaluations. To ensure the transparency of the
evaluation function, it is required that external evaluation teams be hired based on clear and preestablished
criteria and procedures and that the main products of the evaluations be disseminated to
all stakeholders through appropriate means.
TDR Evaluation " To ensure access to comprehensive healthcare services for conflict-affected population, including most vulnerable groups.
(elderly, people with disabilities, GBV survivors) both host and internal displaced population in highly affected areas”
CONFIDENTIALITY: During the development of the evaluations, respect for privacy and the proper
treatment of personal data must be guaranteed. Especially in contexts of conflict or risk to personal
integrity, extreme measures will be taken to avoid possible damages derived from the identification of
the people participating in the evaluations.
RESULTS ORIENTATION: The monitoring and evaluation system should analyze the extent to which
MdM’s work contributes to the objectives set out in policies, strategies, and interventions (projects
and programs).
CREDIBILITY: For the products of the evaluations to be credible and acceptable to the different actors
involved, the evaluations must respond to the information needs, they must be carried out at the
appropriate times, and they must be carried out in a systematic and methodologically robust way so
that the findings and recommendations are founded. The evaluation teams shall receive appropriate
cooperation and access shall be ensured to all available information necessary for the performance of
their work. Doctors of the World is committed to ensuring that monitoring and evaluation are carried
out with high quality based on accepted international standards and based on reliable data and
observations.
10. PROFILE OF THE EVALUATION TEAM
An evaluation person or team that fits the proposed maximum budget, with demonstrable capacity,
experience and professional qualification is required to evaluate this nature and with knowledge of
the geographical and political context of the region of the themes of Mental Health, Public Health,
Gender Age and Diversity (GAD) approaches.
The evaluator or evaluation team must accredit:
- Experience of at least 3 years in evaluation or social research.
- Experience in the intervention area.
- University degree
- Oral and written fluency in English and Ukrainian (at least 50% of the evaluation team must be
Ukrainian speakers).
- Knowledge of different methodologies of evaluation and triangulation of information
- Experience in participatory/qualitative information collection methodologies
- Experience with evaluations of complex strategies or projects in international development
cooperation, especially in health projects
TDR Evaluation " To ensure access to comprehensive healthcare services for conflict-affected population, including most vulnerable groups.
(elderly, people with disabilities, GBV survivors) both host and internal displaced population in highly affected areas”
- Not have maintained an employment relationship with the beneficiary entity of the subsidy (MdM)
or with its counterpart in the framework of this intervention and the mission in Ukraine, at least
during the two years before the hiring proposal for the realization of the evaluation, nor have been
linked to the design, management, or execution of the intervention to be evaluated.
The person responsible for the evaluation team will always act as an interlocutor and representative
before the Monitoring Committee. It will be necessary to include in the technical proposal the times
of dedication to the evaluation of each of the members of the evaluation team, as well as the formal
commitment to belong to the evaluation team during the duration of the contract. Any change in the
composition of the evaluation team must be previously agreed with the Monitoring Committee.
11. PRESENTATION OF THE OFFER AND EVALUATION CRITERIA
The Technical Proposal should include:
o The constitution of the team, the distribution of responsibilities among its members, the
proposed CVs and the availability of the members.
o Work plan.
o Detailed methodological proposal including an evaluation matrix.
o Experience in similar jobs.
o Estimated budget for the realization of the evaluation in which all the expenses derived from
the realization of the evaluation and proposed form of payment are included, in the format of
a proforma invoice.
The criteria for the evaluation of the proposals submitted will be the following:
1. Profile of the evaluation team: maximum 50 points
2. Quality of the technical proposal: maximum 50 points
The maximum budget for evaluation shall be 10.000 Euros, taxes included, which must include all
expenses such as accommodation and national and international travel.
Key facts and dates:
- Receipt of offers by Doctors of the World: until 25/03/2024, at 23:59h local time by email to
evaluacion@medicosdelmundo.org.
- Communication of the selected proposal in April.
- Expected start of evaluation work: May 2024
How to apply
Receipt of offers by Doctors of the World: until 25/03/2024, at 23:59h local time by email to
evaluacion@medicosdelmundo.org.
- Communication of the selected proposal in April.
- Expected start of evaluation work: May 2024.
