1NSP 2^nd Follow up Impact Evaluation:

1.1Background of NSP Evaluation;

The National Solidarity Program (NSP) is the flagship program of the Government of Afghanistan. Since its inception in 2003, NSP has established 22,500 Community Development Councils across 361 districts in all of Afghanistan’s 34 provinces and has financed over 50,000 projects involving water and sanitation, rural roads, electrification, irrigation, and human capital development.

In recognition of the importance of understanding how this path-breaking program affects Afghan communities, in 2007 the Ministry of Rural Rehabilitation and Development (MRRD) and the World Bank commissioned researchers from Harvard University to undertake an independent impact evaluation of NSP using state-of-the-art methodologies. Working in cooperation with the Vulnerability Analysis Unit, the evaluation team recently completed the first follow-up survey for the evaluation, covering communities in the provinces of Balkh, Baghlan, Daikundy, Ghur, Herat, and Nangarhar that were previously surveyed in the baseline survey between August and September 2007 and a first follow-up survey between May and October 2009.

Based on the Memorandum of Understanding between the Impact Evaluation Team and the Vulnerability Analysis Unit (VAU) in the Ministry of Rural Rehabilitation and Development (MRRD) which has clarified the role of each party and the reporting chain, and thus to facilitate the efficient and accurate collection and dissemination of data during and after the second follow-up survey. The VAU is responsible to support the Evaluation Team in conduction of SNP Impact Evaluations in all aspects of evaluation essentially, to manage the data collection and data processing activities.

The Prospective Evaluation of Phase-II of the National Solidarity Programme (NSP) seeks to rigorously measure the economic, political, and social impacts of the NSP program on participating communities such as; local governance structures, access to services and infrastructure, economic activity, female participation in community activities, political opinion, and social cohesion. A report summarizing the evaluation results will be delivered to the Government of Afghanistan, donor agencies, and implementing NGOs after completion of second follow-up survey which is going to be done at the early year 2012.

1.2Survey Scope:

Based on the survey operation plan 5,000 households, 500 communities, 10 districts (Adreskan, Gulran, Farsi, Chesht-e-sharif, Balkh, Khost-wa-freng, Sang-e-takht, Dulina, Hesarak and Sherzad) and Six provinces (Herat, Balkh, Baghlan, Dykundi, Ghur, Nangarhar)  identified for launching of the survey.

1.3Design of the Survey Instrument:

The survey instruments were drafted by The Evaluation Team in English and were arranged for translation in Dari. Six types of questionnaires such as; male household, female household, male focus group, female focus group, village map and radio form were designed and were printed out for data collection aspects based on the survey sample plan. Likewise, other instruments like; survey guideline, survey books, historical cards, literacy cards, occupation cards and etc. were prepared and printed out to facilitate the data collection activities. The pilot test of the draft questionnaires has launched in 3 districts of Kabul province (Estalif, Shakardara and Mirbacha kot).

1.4Hiring and Training of Survey Supervisors, Enumerators, and other Survey Staff:

Around 140 candidate surveyors from various provinces were called for survey training in, Kabul. The training was launched for 17 days in three stages and among 120 candidates 65 were selected by evaluation team, as field supervisors and surveyors to administrate the data collection of the survey.  The training of enumerators, supervisors, and survey staff was conducted in accordance with guidelines established by the Evaluation Team.

1.5Administration of data collection:

The second follow-up survey data collection launched followed the completion of training, selection of survey personnel and finalization of survey instruments and implementation guidelines. Sixty-five selected surveyors and supervisors divided to 4 teams, each consist of one Supervisor, 3 Editors and 12 Enumerators (in couple bases) in accordance with residential area and familiarity of surveyors to assigned locations. Subsequently, each team repeatedly subgrouped to 2 couples (4 Enumerators with one Team Leader) and one Editor in each. The data collection process was started on May 2011 and was completed almost end of October 2011. The data collection activities opened in Adreskan district of Herat province and followed by Balkh, Khost-o-Freng, Gulran, Daikundy, Hesarak, Farsi, Dulina, Chesht-e-Sharif and Shirzad districts in a sequential order. Financial and budgetary arrangements pertaining to the second follow-up survey were well managed and finalized in consultation with the NSP Finance Department, the World Bank, and the Evaluation Team. The budget and expenditures were arranged in accordance with the survey monthly action plan and were documented based on the NSP financial policy and regulations. (For information about project budget and expenditures see Annex1)

 

NSP 2nd Follow up Survey, Total Household and Focus Groups Questionnaire
Locations					HH Questionnaire			FG Questionnaire			Actual Form Exist
No	Province Name	District	No of District	No of Villages	Male	Female	Total HH	Male	Female	Total HH
1	Herat	Adreskan	1	50	468	458	926	49	48	97	1,023
2	Baghlan	Khost-o-firing	1	50	500	500	1000	50	50	100	1,100
4	Balkh	Balkh	1	50	500	499	999	50	50	100	1,099
5	Heart	Gulran	1	30	277	277	554	30	30	60	614
3	Nangarhar	Hesarak	1	42	420	300	720	42	30	72	792
6	Daikundy	Sang-e-takht	1	52	515	515	1030	41	43	84	1,114
7	Herat	Farsi	1	50	487	487	974	40	47	87	1,061
8	Ghur	Dulina	1	37	359	359	718	35	35	70	788
9	Nangarhar	Shirzad	1	36	360	0	360	36	0	36	396
10	Heart	Chesht Sharif	1	50	468	448	916	42	41	83	999
Grand Total		10	10	447	4,354	3,843	8,197	415	374	789	8,986

1.6Monitoring and Supervision:

As monitoring and supervision activities are the most important and credible question for reliability and confidentiality of the survey, particularly, during data collection, therefore, the VAU is seriously considering on this topic and rigorously conduct various methods of monitoring during and after completion of data collection.

1. Direct Monitoring: While the data collection of NSP 2^nd follow-up survey was in progress, around 12 field monitors were sent to 10 covered districts to check the quality and progress of data collection in aspects of following of “Implementation Guideline” as well as noticing of enumerators and supervisors problems were encountered in the field. All monitors conveyed a standard supervision form outlined for checking of data collection activities and manifestation of deficiencies and gaps undertaken by field staff. They implied their revisory recommendations to found problems defects. Findings were reported to office and were considered for next plans.

2. Track Points by TRACK 24: Modern technology such as; TRACK 24 and GPS devices were used as online monitoring by central office. Each couple of assigned teams conveyed a TRACK 24 which directly was connected to office PCs and was online controlled each 10 minutes with exact information about location of surveyors and data collection progress. GPSs were also used to track point each interviewed household, which will be used for map designing and household determination in future.

3. Re-monitoring and Data Quality Checking: After completion of data collection in each district a monitor was sent to the same district to check and compare the collected data with several local agencies and stakeholders to verify the accuracy of obtained data.

1.7Data Processing Management:

Based on the MoU (Memorandum of Understanding) between MRRD and Impact Evaluation Team the VAU was expected to manage all data processing arrangements of NSP 2^nd follow-up survey pursuing the data collection of the survey. Behind the pilot test and finalization of survey questionnaire, a database using the last version of MS Access, a user friendly package, was designed to facilitate the manual data entry of the survey. A team of Data Entry Clerks, supporting data entry process were hired and well trained. Database and data entry process were piloted and finalized for actual data entry occasion. All necessary technology and equipment such as, desktop computers, stabilizers, UPS batteries and etc. required for data processing response to an electronic database were procured and assembled. The completed survey forms were transferred to the data entry center in Kabul at regular base intervals. The data entry process was started after completion of data collection in first district and behind checking of filled forms in central office on 19^th June 2011. It was progressively continued up to 31^th Oct 2011. The double data entry system was designed for consistency check and quality control, and the process has started awhile after single entry was initiated. The double data entry for 2^nd follow-up survey continued without work stoppage till last returned questionnaire and was completed on 30 November 2011. The data entry process rigorously supervised by VAU management and datasets were supplied to Evaluation Team in regular basis of one-week interval. Daily discovered problems in data entry, database structure and database appending process were solved by VAU Data Processing Management Team.

Data Entry & Data Double Checking Progress till end of December 2011

No

Province

District

Single Entry

Double Entry

Checking Number (Double System)

Checking Text (Double System)

Data Checking

Database Restructure

Triple Entry

Checking Number (Double System)

Checking Text (Double System)

Questionnaire Data Checking

1

Herat

Adraskan

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

2

Baghlan

Khost-o-Firing

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

3

Herat

Gulran

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

4

Balkh

Balkh

Done

Done

Done

Not Done

Not Done

Not Done

Not Done

Not Done

Not Done

Not Done

5

Daikundy

Sang-e-takht

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

6

Nangarhar

Hesarak

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

7

Herat

Firsi

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

8

Ghur

Dulina

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

9

Nangarhar

Shirzad

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

10

Herat

Chesht-e-Sharif

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

2Village Benefit Distribution Analysis (VBDA) Survey:

2.1Background:

This pre-analysis plan for the Village Benefit Distribution Analysis (VBDA) specifies methodologies and specifications to be employed in testing hypotheses pertaining to the impact of institutional changes generated by the National Solidarity Programme (NSP) and related interventions on local governance outcomes. The plan was written prior to data analysis and pre-commits the authors to defined specifications for estimating impacts.

The plan is structured around four sections, one table, and three appendices. Section II provides an overview of the study, detailing the nature of the treatment and associated interventions, the sample, and the experiment deployed to generate data for the study. Section III lists the hypotheses for the study. Section IV describes the methodology and specifications to be employed in hypothesis testing. Table I lists the hypotheses and indicators for the study. Appendices I – II list questions included in the Village Benefit Distribution Surveys and Appendix III lists questions from the Second Follow-Up Survey that are to be used in the analysis.

2.2Description of Dependent and Explanatory Variables and Data Collection:

The VBDA is a behavioral experiment designed to identify how externally-imposed institutional changes generated by NSP impact the performance of local leaders in delivering public services. The experiment will also identify impacts of mandating the participation of different groups of local leaders in service delivery.

The experiment generates quantitative measures of leader behavior by organizing and then monitoring village-level distributions of food aid across 500 villages in Afghanistan. The distribution of food aid provided by external agencies is a common public service performed by local leaders in rural Afghanistan. Such distributions present leaders with the option to distribute aid equitably to vulnerable villagers, distribute aid equally among all villagers, or embezzle aid for personal or familial benefit. Data on how leaders distribute food aid are to be collected by surveys administered to random samples of village households and aid recipients following the distribution.

The sample for the study comprises a sub-sample (to be selected based on accessibility given local security and weather conditions) of 500 villages spread evenly across ten districts in Balkh, Baghlan, Daikundy, Ghur, Herat, and Nangarhar provinces.5 The ten districts were selected based on size, security conditions, and the constraint that no villages in the district had previously participated in NSP. In each of these ten districts, NSP Facilitating Partners were asked to select 50 villages which they deemed suitable for NSP implementation,6 which in turn provided the sample of 500 villages.

Although the 500 sample villages cannot - by virtue of their clustering in only ten of 398 districts and the non-random selection of the ten districts in which they are located - provide a random sample of villages in Afghanistan, the 500 selected villages are representative of Afghanistan’s geographic, ethnic, and economic diversity. An assessment of the demographic and economic characteristics of the 500 villages reveal few substantive differences with those of a random sample of villages surveyed by the 2007-08 National Risk and Vulnerability Assessment.

2.3Hiring and Training of Survey Supervisors & Enumerators:

Around 200 candidate surveyors from various provinces were called and trained in different training courses and different regions such as; Balkh, Kabul and Herat provinces for administration of VBDA survey. The training was launched in various sections and stages, among 200 candidates were participated in these trainings around 100 of them were selected by evaluation team, as field supervisors and surveyors to administrate the data collection of the survey.  The training of enumerators, supervisors, and survey staff was conducted in accordance with guidelines established by the Evaluation Team.

2.4Structure of Village Benefited Distribution and Administration of Data Collection:

The food aid distribution and the associated data collection generated quantitative data on the behavior of village leaders in the performance of a key public service, which served as the dependent variable for the study. Distribution and data collection activities were implemented during three visits to each village made by distribution agents and enumerators over three, non-consecutive days. The data collection activities was started on 15^th July and completed on 31^st Oct 2011.

Day 1: A distribution agent was hired and trained by the evaluation team, traveled to the village and convened a short meeting of the village leaders. The agent informed the leaders that the village has been selected to receive a fixed quantity of wheat (approximately one 50 kilogram bag for one sixth of the households in the village) from the World Food Programme (WFP) and that, in three days, WFP will be delivering the wheat to the village. Leaders were informed that the wheat was intended for distribution among vulnerable members of the village community, such as widow-headed households, and were directed to determine which households in the village are to receive the wheat. In accordance with WFP monitoring requirements, leaders were asked to prepare a list of households in the village that will receive wheat, complete with the amount that each household will receive, and were informed that the list will be collected by a WFP representative on the day of delivery.

Day 2: Three days later, a different distribution agent returned to the village with the quantity of wheat was allocated for distribution in that village, which was transported either by a WFP vehicle or the vehicle of an agency contracted by WFP to conduct the delivery. The agent collected the list of wheat recipients were prepared by the village leaders and, after verifying the legibility of the list, authorized the transfer of the wheat to the village leaders.

Day 3: Ten days after the distribution of wheat, a team of 12 male and 12 female enumerators returned to the villages to administer household surveys to three groups of villagers:

First Survey: Random sample of households in the village;

Second Survey: Random sample of households that were listed by village leaders to receive wheat;

Third Survey: Random sample of households that First Survey respondents indicated received wheat, but which were not included in the list provided by the village leaders.

Survey questionnaires are administered conterminously to a male and a female respondent in the same household. Questionnaires were administered to 12 First Survey households and 15 total Second and Third Survey households.18 First, Second, and Third Survey questionnaires were identical and were collected basic demographic and socioeconomic data on the characteristics of the respondent’s household, as well as information on the distribution and characteristics of recipient households. (For information about project budget and expenditures see Annex1)
 

VBDA Survey Coverage Details
Location				# of Questionnaires and Households Interviewed
ID	Province	District	Total Villages assessed	1st Survey-Male	1st Survey-Female	2nd Survey-Male	2nd Survey-Female	3rd Survey Male	3rd Survey Female	Total-Male Questionnaire	Total-Female Questionnaire	Total HHs Questionnaire
1	Herat	Adreskan	43	667	685	329	364	158	169	1,154	1,218	2,372
2	Balkh	Balkh	50	592	593	424	427	246	246	1,262	1,266	2,528
3	Baghlan	Khost-o-Firing	47	669	669	531	531	120	120	1,320	1,320	2,640
4	Herat	Farsi	45	632	643	440	445	113	118	1,185	1,206	2,391
5	Nangarhar	Hesarak	38	570	570	378	378	192	192	1,140	1,140	2,280
6	Herat	Gulran	28	425	424	236	253	94	102	755	779	1,534
7	Daikundy	Sang-e-takht	50	706	710	381	388	73	73	1,160	1,171	2,331
8	Ghur	Dulina	26	369	294	298	235	45	35	712	564	1,276
9	Nangarhar	Shirzad	36	541	0	480	0	54	0	1,075	0	1,075
10	Herat	Chesht-e-Sharif	41	544	536	439	437	96	101	1,079	1,074	2,153
Grand Total			363	5,171	4,588	3,497	3,021	1,095	1,055	9,763	8,664	18,427

2.5Monitoring and Supervision:

The same procedure and process as NSP 2^nd follow-up survey have been used for monitoring and supervision of VBDA survey data collection to maintain the data quality and credibility.

2.6Data Processing Management:

As per agreement of Evaluation Team the data entry process of VBDA survey contracted with a private company, while designing of database, double checking system and data entry supervision have been done by VAU data processing team. The data entry and data cleaning of VBDA survey is expected to completed at end of April 2012.

3Foundation of Monitoring and Evaluation Training Program:

The 5 core staff of VAU participated in Foundation of M&E training course conducted by Arab Administrative Development Organization (ARADO) was held on 12^th – 17^th March 2011, at Sharjeh, U. A. E.

The workshop provided an overview and an introduction to results-based monitoring and evaluation (M&E) with an emphasis on practical applications of tools and methods. The training was intended for participants with prior experience with monitoring and evaluation.

The participants have gained an improved understanding of how M&E can improve the quality of their projects while promoting learning and accountability. Similarly, they learned how to be better managers and consumers of evaluations conducted by others. Additional sources of web-based information on M&E were provided at the end of the training materials.

The objectives of the course were as follows:

1) To gain confidence to apply the acquired skills and knowledge to M&E work.

2) To be able to share and learn from other country experiences to strength their M&E systems.

The training mainly focused on the concepts of results-based monitoring and evaluation, results chains and frameworks indicators, evaluation designs, data collection, sampling, data analysis, reporting and managing results-based evaluations, based on feedback from participants the course was really productive and useful for further support of a VAU team within AIRD/MRRD. The training was delivered through lecture, presentation, focus group, cause study and other usual methods.

4Foundation of M&E and an Explanation on MDG and ANDS Training Workshop:

Coincide with returning of VAU staff from abroad training, another training program, focus on M&E tools and methods with extra point on MDG and ANDS designed for the rest of VAU and AIRD staff (almost 25 participants). The training launched for 5 days on April 2011.

The workshop provided an overview on Millennium Development Goals (MDG), its objectives and indicators, similarly, an introduction on Afghanistan National Development Strategy (ANDS) pillars, sectors and indicators. Moreover, the program was focused on M&E tools and methods.

The participants have gained an improved understanding of how M&E can improve the quality of their projects while promoting learning and accountability. Additional sources of web-based information on M&E were provided at the end, along with training materials.

The objectives of the course were as follows:

1. To gain some information on MDG and ANDS objectives and indicators.

2. To gain confidence to apply the acquired skills and knowledge to M&E work.

. The training was delivered through presentation, group exercises and other usual methods.

5Training Workshop on MS Access 2010 and Database Design:

Following to the last correlated training course titled “Foundation of MS Access 2007” carried out on 2009, on this occasion an advance level of training in the field of database and MS Access titled “MS Access 2010 and Database Design” has launched on 20th April to 15th May 2011, with attendance of around 30 participants consisting VAU, AIRD and MRRD programs employees. The key concepts of the training were; updates and utilities in last version of MS Access package, how to use MS Access Objects (table, query, form and etc.), making relationship, making different queries, designing of database and etc. Based on feedback from the participants the course topics were really productive and useful for all participants particularly, those who are typically employed in the field of database management, MIS, research, survey, M&E and so on. The training was delivered through presentation, project base practical work, group and individual assignments and other usual methods. At the end and after a final test a benchmark certificate was distributed to those with full attendance.

6 Databases Designed for Different Units within AIRD:

• Gender in development study Database for AIRD Research Unit.

• Failed Water Points and Poor Off take of Sanitation database for research unit.

• DDA Rapid Survey Database for AIRD Research Unit.

• Resource Inventory Survey database (RTP Unit/ AIRD)

• Library Database for electronic recording of all existing books within AIRD Library.

• Updating of Library Database.

7 Future Plans:

• Conduction of NERAP (National Emergency Rural Access Program) 1^st Follow-up Survey (VAU extension and the WB agreement are required).

• Conduction of FSMS round 9^th survey for World Food Program (WFP) in 2012.

•  VAU impart in launching of 2^nd round “Afghanistan Rural Development Report” via conduction of a nationwide research.

• VAU inclusion in implementation of “Support Change Management” baseline survey for MRRD.

• Conduction of evaluations and other comprehensive surveys for MRRD  Program and other organizations ( Agreement of MRRD Management is required)

• Statistical and Impact Evaluation capacity building of VAU staff via conduction of training programs inside and outside of the country.

•  Technical capacity building of VAU, symphonic with the world standards for conduction of researches/surveys.

•  Consolidation, presentation and dissemination of existing data within VAU/AIRD in AIRD/VAU website by designing of an OLAMP (Online Micro-analytical data processing System) database to provide a user-friendly means of dissemination.

8 Challenges:

• Deficiency in financial sustainability of VAU in capacity building and survey implementations perspectives.

• Minor shortage of technical capacity in update and standardized technologies and tools required for implementation of impact Evaluations and other comprehensive surveys and advance level of Impact Evaluation skills.

• Security restrictions in some part of the country against implementation of impact evaluations and other surveys.

• Lack of administrative facilities/support versus VAU activities and survey implementation such as; pick and drop transportation, telecommunication, local transportations, internet connection and electricity for utilization of new technologies in some regions and etc.

 

9Suggestions:

 

• Financial support of MRRD Management for further sustainability of VAU

• Further financial/technical support and collaboration of the WB with VAU in future and after closing of IDF project.

• Identification and clarification of VAU position within MRRD and AIRD.

•  Specification of an allocated amount for impact evaluation of MRRD development programs by Programs Managements (has not considered by some programs yet).

• Preservation of VAU/AIRD independency in implementation of Impact of Evaluations for governmental and nongovernmental organizations.

• Agreement of MRRD Management for inception and implementation of surveys/researches for other organizations.

• Further support of the WB and MRRD Management in conduction of planned training programs for capacity building of VAU/AIRD