Terms of Reference
and Request for Proposals

October 2001

THIS DOCUMENT IS DISPLAYED FOR INFORMATION PURPOSES ONLY. THE EVALUATION IS ALREADY UNDERWAY.

1. Background

The Sphere Project was launched in mid-1997 by a consortium of humanitarian agencies out of concern that a massively increased demand for humanitarian relief world-wide was in danger of outstripping the response capacity of the humanitarian system, thereby leading to inconsistent quality in relief efforts. A fundamental principle of the Sphere Project is that those affected by disasters have a right to assistance and that the international community has an obligation to respond in an accountable way. The Sphere Project's goals were to improve the quality of humanitarian assistance and the accountability of humanitarian agencies to their beneficiaries, their members and their sponsors.

The importance of working consistently to standards that respect human dignity during humanitarian responses has been demonstrated time and time again over the last few decades. Where agencies are unprepared or incompetent, those affected by disasters suffer. For example, while much good work was carried out by NGOS in 1994, tens of thousands of people lost their lives in Goma, Eastern Zaire, to cholera and dysentery, as agencies struggled to provide an adequate level of service. Indeed, it was this experience that led to the call for a system-wide initiative to improve the quality of disaster interventions, and to the creation of the Sphere Project by concerned NGOs and the IFRC.

The Sphere Project has evolved into an extensive international, interagency effort. It has employed a co-operative, collaborative process to develop a Humanitarian Charter for persons affected by disaster and an associated set of Minimum Standards in essential areas of humanitarian response.

The Humanitarian Charter sets out the rights of those affected by disasters based on international humanitarian law, refugee law and human rights conventions. The Minimum Standards and key indicators respectively describe what people need and measurements to help determine if these needs are being met. The standards and indicators reflect concerns such as gender equity, respect of local culture, reducing the risk of violence to populations affected by disaster, and the minimization of environmental impact.

The Sphere Project has been a joint effort of non-governmental organisations (led by InterAction and the Steering Committee for Humanitarian Response) and supported by VOICE and ICVA. The International Red Cross and Red Crescent movement has also been an active participant.

2. Evaluation Overview

It is vital that Sphere be evaluated, both to serve as a means of accountability to all the many people and organizations that have been involved in the project, and as a means of learning lessons to take forward the work of improving the standards of humanitarian performance and the accountability of aid organizations to those affected by humanitarian disasters.

To this end, an independent assessment will be commissioned to evaluate four areas:

The evaluators will not be asked to assess the appropriateness of the individual targets that make up the Sphere Minimum Standards. However, where strong statements are made on the appropriateness of individual standards, these should be recorded by the team and submitted to the Sphere committee responsible for overseeing the evaluation, in particular where the achievement of Sphere Standards resulted in the realisation that these were inadequate in some way: e.g., enough water points to meet Sphere standards but continued diarrhea, etc

3. Specific objectives
The four main goals of the evaluation have various sub and intermediate objectives as outlined below.

1. Whether or not Sphere has made a difference to the quality of humanitarian aid, and ultimately to the lives of those affected by disasters
   • Is there any evidence of difficulties between resident and disaster-affected populations, especially in the case of the standards and indicators being higher than those of the resident population?
   • Whether the Humanitarian Charter has led to a greater involvement in humanitarian program design and implementation by affected populations.
   • Have they led to a greater involvement in humanitarian programme design and implementation by affected people? Have they led to changed attitudes amongst staff of humanitarian agencies and others towards beneficiaries? Are beneficiaries consulted more frequently as part of the project cycle?
   • In Angola, some aspects of the Standards/indicators have been enshrined into Minimum Operational Standards (MINOPS). Has this government adoption affected the application of Sphere in any way?
   • Increased involvement of and/or satisfaction of beneficiaries
   • Any disadvantages of using Sphere will also be assessed. For example, what happens where Sphere standards are not in line with host government regulations? Have the Minimum Standards had any positive or adverse protection implications? Have governments provided sufficient funds to NGOs to perform to Sphere standards? Have governments used inability to perform to standards as an excuse for refusing to fund interventions inconsistent with other foreign policy objectives? Has utilization of the standards stifled NGO creativity or initiative?

2. If and how Sphere has provided a common framework and supported a cooperative effort in the field and determine the impact of a common language and if Sphere has led to an increase in the professionalisation of aid workers
   • Has the use of the Humanitarian Charter and the Minimum Standards led to improved co-ordination?
   • That standards are not met may have no reflection on the impact of Sphere, nor on the appropriateness of the standards, but on many other factors, such as lack of funding and difficult operating environments. However, in such cases, Sphere should, arguably, lead to greater pressure to meet such standards, through the lobbying of donors, governments in crisis-affected countries and other key institutions and individuals.
   • Have the existence of the Minimum Standards been useful as a way of generating funding for particular programmes? Has this made funding proposals and funding less arbitrary than before?
   • Whether the introduction of Sphere into the working practice of NGOs has led to increased professionalism, a heightened understanding of, or commitment to, the importance of meeting standards.
3. The processes by which agencies have incorporated Sphere with emphasis on creative tools and approaches.
   • While attainment of standards could be considered the gold standard for judging the success of Sphere, there are other intermediate indicators. For example, this will include assessing whether NGOs are using the handbook, and in which ways. Have agencies incorporated the Humanitarian Charter into their overall goals, whether explicitly into mission statements, or implicitly? Are agencies seeking to meet the Minimum Standards? Has the Humanitarian Charter been the driving force towards achieving the Minimum Standards - or have they been taken as separate entities?
   • Is Sphere considered important by the agencies? Is training given to staff? Is reference to Sphere made in project reports, and are Minimum Standards incorporated into monitoring and evaluation systems? Are agencies discussing Sphere standards and reporting back to the Sphere Project with comments?
   • What was the experience of implementation? Did it help to focus programme administrators? Did it create difficulties?
   • Were all members of an agency or a partnership between northern and southern agencies equally committed to the Minimum Standards, or did this vary? Were local government and/or beneficiary representatives aware of the use of the standards and what were their opinions? Were there clashes/synergies with other standards?
   • Has the introduction of Sphere led to any changes in the nature of programmes attempted, and in the coverage of such programmes? Do agencies look at global targets and seek to meet these in the best manner possible, or are some agencies taking a narrower view and trying to meet 'lower' more specialised Sphere standards instead? For example, in theory, agencies could aim for wider geographical distribution of water points in an affected area, or aim to ensure that in one particular camp, standards are met even if they are much lower in camps in another area.
   • What lessons has the piloting of the incorporation of Minimum Standards in the agencies' policy and practice revealed? How far were agencies able to establish monitoring systems in relation to the Standards and indicators? What were the constraints to monitoring? Are there any examples of NGOs including Sphere in project proposals but not in practice? whether NGOs and IOs are making reference to Sphere in their everyday work and how that reference is translated into action.
   • Has the drive to meet Sphere standards led to increased or decreased costs of operations? Has this had any impact on funding available for other humanitarian or development programmes? Has this had any impact on funding available for national or international NGOs?
   • What happens where Sphere standards are not in line with national government regulations?
4. Review the way in which the Sphere project was created, implemented and carried out with a view to learning lessons for future such initiatives.
   The evaluators will be expected to write a short review of the history of the Sphere Project, and comment on the manner in which the process of developing the standards has been undertaken. This should include consideration of:
   • While knowledge of Sphere is important, acceptance of Sphere as an initiative that needs support by those involved in humanitarian activities, whether as providers or beneficiaries is key. What are attitudes to the Sphere project? And can any link be drawn between the number and location of training sessions/seminars carried-out and knowledge of and acceptance of the need for standards? The Sphere Project has involved a lengthy consultation process, leading to the production of a Sphere Handbook, a number of leaflets and brochures and the development of a dedicated website. Following the production of the first draft of the Handbook, the Sphere Project has included a component of dissemination and training. Numerous presentations, training sessions and seminars have been conducted around the world. A pilot agency implementation program was undertaken. Arguably, these processes have been as important as the physical outputs themselves - offering the opportunity for a period of debate on the need for standards, on the form that such standards should assume, and on the appropriate application of such standards.
   • Do southern agencies feel that Sphere is appropriate for them? Do they feel their voice has been heard in the debate about the application of Sphere?
   • Were agencies assisted in the institutionalization of the Sphere Standards? Could the Sphere Project have done more in this regard?
   • Whether there has been an increased awareness around the world (amongst NGOs, intergovernmental bodies, governments and affected populations) of the existence of and need for Minimum Standards and a Humanitarian Charter.
   • What was the history of the management of the project? How much support did the agencies behind the initiative give to the goals of the project? What were the organisational positions within the humanitarian community towards the project, and were they a function of organisational interests or project goals? What are the tensions in the creation of standards in a diverse civil movement?
   • Who was involved in the preparation of the standards and the handbook? Was this an inclusive process? Were (potential) humanitarian beneficiaries and their representatives adequately consulted? Were field level personnel included as well as headquarters staff consulted? Have any language or other barriers prevented the inclusion of certain groups from becoming involved?
   • Where have workshops taken place? How have these been assessed? What has been the basis for selecting locations for workshops and training sessions? Who has received the handbook? How many hits have been received on the web-site? From where?
   • Do the standards and the charter tie-in with international humanitarian legal instruments?
   • What resources were consumed in the development of the handbook and the website? Could this have been more efficiently produced, without compromising the inclusive nature of the process?
   • Has the management structure supervising the Sphere Project been appropriate? Are the handbook, brochures and website appropriate for the target audience? How else could the Sphere message have been disseminated? Is there synergy in the various dissemination channels?

4. Coverage of objectives

For each of the objectives detailed above the following questions of variation should be considered:

Does knowledge, experience or impact of implementation of the Sphere Project vary geographically? Do some parts of the world know more about the Project than others?

Does knowledge or experience of implementation of the Sphere Project vary according to the operational sector involved? For example, is there greater knowledge and use of Sphere standards amongst those involved in delivery of water and sanitation services than amongst those involved in food distributions?

Does knowledge or experience of implementation of the Sphere Project vary within an organisation/government? Do, for example, NGO headquarters staff know more about Sphere than fieldworkers?

Are agencies attempting to incorporate Sphere values in their work and have they been alone in such attempts in a disaster area, or have all agencies shared similar goals? Can any explanation be found for why some agencies adopt Sphere and others do not? Does the operating environment in which agencies are working (natural disaster or complex emergency, Africa or Europe) make a difference to ease of implementation?

5. Evaluation Process and Methodology

The evaluation will commence in January 2002, and take place over a 20 month period. The first step will involve the selection of an independent team of evaluators to carry out the work. Their first task will be the collection and analysis of baseline data on pre-Sphere humanitarian performance, to allow for a comparison of humanitarian programs implemented pre- and post-Sphere (it is clear that this is a very difficult task and will need careful consideration). The evaluation team will need to make suggestions as to how such data will be collected, and by whom, and how this will be used to draw conclusions on impact. Data needs need to be developed for each of the sectors of disaster response - water and sanitation; food aid; nutrition; shelter and site selection; and health. The evaluation will be greatly facilitated through the support of a group of nineteen NGOs based in Africa, Asia, Europe, Central America and the USA who have, as pilot agencies, been systematically incorporating Sphere into policies and practice in disaster response over the last year. The Sphere Project is supporting this process and learning lessons from their experiences.

Ideally, it should be possible to compare programmes carried-out with reference to Sphere standards and the Humanitarian Charter to programmes that made no reference to Sphere, in response to the same emergency.

As well as collecting data from a range of emergency operations, the team will also design and implement two in-depth case-studies of humanitarian interventions - one in an emergency scenario as soon as possible after the evaluation is launched, the other at a later stage in the analysis (assuming such emergencies arise). This will allow for an unfolding assessment of the impact of Sphere as it is disseminated over a range throughout the humanitarian system. These case studies will include contemporaneous comparisons between those agencies adopting Sphere standards and those, for whatever reason, choosing not to adopt them.

In their tender submission, the evaluation team will need to show it will seek to answer the questions laid out above. It is assumed that any approach will include the interviewing, and reviewing documentation of, a sample of NGOs (northern and southern), international organizations, intergovernmental bodies, national and local governments and (potential) disaster-affected populations. While the focus of enquiry should be on knowledge and acceptance of the Sphere Project, knowledge and acceptance of other standards and accountability tools should also be recorded. In this way, the 'penetration' of Sphere can be assessed in the context of the wider debate on standards and accountability.

For such comparisons to be possible, selected humanitarian programmes will need to be selected as case studies. The tender document should detail how the evaluators would select case study programmes. Such a selection should take into account the different natures of humanitarian programmes, for example long-term chronic emergencies, natural disasters, etc.

The evaluation team will be expected to make a few field trips as part of this work. The tender document should detail how the selection of locations will be made.

After each field trip, the evaluation team will be expected to report back preliminary findings to a short workshop consisting of NGOs (international and national) and IOs working in the country as well as appropriate governmental officials. Follow-up reports should be submitted to the Sphere Management Committee, the Advisory group and the pilot agencies.

Mid-way through the evaluation, the evaluation team should meet up with the Advisory Group to discuss progress and preliminary findings. There should also be a further meeting to discuss the draft report of the evaluation, once this is ready. The report should be circulated two weeks prior to the meeting to allow for preliminary review by agencies and their partners, and followed by a one-month formal agency comment period.

6. Application

Those wishing to tender for the evaluation are invited to submit a document of no more than 5 pages that outlines:

• the name of the firm/organisation tendering and their previous experience in carrying-out such evaluations;
• the approach to be adopted, with reference to the terms of reference;
• a proposed timeline for the evaluation (within the 20 month window);
• details of the team who would carry-out the work - names, sectors of expertise and brief biographies, with full CVs in an annex;
• an indicative budget, outlining proposed expenditure on fees, per diem, travel and other costs

Competencies:

- knowledge and experience of the humanitarian sector, including direct experience of complex emergencies
- experience of research design with operationally practical objectives
- excellent oral and writing skills
- a demonstrated understanding of the contents of the Sphere handbook
- an understanding of humanitarian principles and the legal instruments of the Humanitarian Charter
- excellent management skills including strong track record in the management of research/evaluation
- Need to show cross-fertilisation between various parts of the evaluation.
- agencies tendering would need to show how they would manage recruitment and appointment process for sub-contracts

Professionalism of the bid, team experience (professional and analytical), degree of parity with the terms of reference, likelihood of achieving the evaluation timetable, and realism not just competitiveness in the cost submission are essential.

Tenders will be accepted from "freelance" teams as well as from company, PVO or academic teams. Tenders are welcome from regional teams.

SUBMISSION DEADLINE: MONDAY 12 NOVEMBER BY ELECTRONIC MAIL TO buzard@ifrc.org and jbishop@interaction.org

It is anticipated the selection process will be complete by December 2001 and that the Evaluation team will meet with the Sphere Management Committee in Geneva in 25 January 2002.

7. Evaluation team and timeframe

It is anticipated there will be a core team of at least two people, with others drawn in as necessary. The Team Leader should have a relevant skill and a proven background in emergency evaluations. The appropriate balance of professional and analytical skills amongst the remaining team members should be determined following a preliminary examination of the task to be undertaken. It is likely, however, that sector expertise in areas such as water and sanitation, public health and shelter will be required. At least one person from the region should be included in the team that makes the field visits.

All team members should be gender aware, and a reasonable gender balance within field teams is desirable.

Consultants or independent evaluation teams short-listed in the tendering process should seek the Sphere Project's approval for any proposed changes to the composition of the team originally submitted.

The evaluation timeframe should allow for the circulation of a first draft by May 2003 followed by presentation of the draft by the evaluation consultant(s) to agencies and other interested parties a week later. A formal comment period, of at least one month, for participating agencies and their partners will then follow. The completion date for the Final Evaluation Report will be August 2003 the consultants having addressed agencies' comments as appropriate.

8. The Reports

The evaluation reports should consist of:

- two-three page interim reports every quarter
- executive summary and recommendations (not more than 10 pages)
- main text, to include index, context in which the evaluation took place, evaluation methodology, commentary and analysis addressing the above questions and a section on lesson-learning, conclusions and recommendations (not more than forty pages)
- appendices, to include evaluation terms of reference, maps, sample framework, summary of agency activities, sub-team report(s), end notes (where appropriate) and bibliography. (All material collected in the undertaking of the evaluation process should be lodged with the Sphere Secretariat prior to termination of the contract)

9. Management of the Evaluation

Sphere Management Committee member, InterAction, will assume both fiduciary responsibility for the funds and the obligation to provide whatever substantive and financial accounting the donor would require. In collaboration with other members of the Management Committee, and the project Manger, InterAction will select the organisation(s), firm(s) or individual(s) contracted to perform the evaluation. In partnership with its Management Committee partners, InterAction, will be responsible for cancellation of the evaluation contract and the return of unexpended funds to the Mellon Foundation, if the contractor's performance should be determined to be failing to meet contract terms.

In order to ensure a highly credible and independent evaluation, an independent Advisory Group will be identified who will assess progress reports and drafts of the evaluation findings. James K. Bishop, InterAction's Director for Humanitarian Response, will be the key individual charged with liaising with the donor. Mr. Bishop will oversee the work of the Sphere Project Manager, who will have day-to-day responsibility for the evaluation.