VAU Progress Activities - Ministry of Rural Rehabilitation and Development

VAU Progress Activities

Projects Progress and Achievements in the year 2011

1NSP 2nd Follow up Impact Evaluation:

1.1Background of NSP Evaluation;

The National Solidarity Program (NSP) is the flagship program of the Government of Afghanistan. Since its inception in 2003, NSP has established 22,500 Community Development Councils across 361 districts in all of Afghanistan’s 34 provinces and has financed over 50,000 projects involving water and sanitation, rural roads, electrification, irrigation, and human capital development.

In recognition of the importance of understanding how this path-breaking program affects Afghan communities, in 2007 the Ministry of Rural Rehabilitation and Development (MRRD) and the World Bank commissioned researchers from Harvard University to undertake an independent impact evaluation of NSP using state-of-the-art methodologies. Working in cooperation with the Vulnerability Analysis Unit, the evaluation team recently completed the first follow-up survey for the evaluation, covering communities in the provinces of Balkh, Baghlan, Daikundy, Ghur, Herat, and Nangarhar that were previously surveyed in the baseline survey between August and September 2007 and a first follow-up survey between May and October 2009.

Based on the Memorandum of Understanding between the Impact Evaluation Team and the Vulnerability Analysis Unit (VAU) in the Ministry of Rural Rehabilitation and Development (MRRD) which has clarified the role of each party and the reporting chain, and thus to facilitate the efficient and accurate collection and dissemination of data during and after the second follow-up survey. The VAU is responsible to support the Evaluation Team in conduction of SNP Impact Evaluations in all aspects of evaluation essentially, to manage the data collection and data processing activities.

The Prospective Evaluation of Phase-II of the National Solidarity Programme (NSP) seeks to rigorously measure the economic, political, and social impacts of the NSP program on participating communities such as; local governance structures, access to services and infrastructure, economic activity, female participation in community activities, political opinion, and social cohesion. A report summarizing the evaluation results will be delivered to the Government of Afghanistan, donor agencies, and implementing NGOs after completion of second follow-up survey which is going to be done at the early year 2012.

1.2Survey Scope:

Based on the survey operation plan 5,000 households, 500 communities, 10 districts (Adreskan, Gulran, Farsi, Chesht-e-sharif, Balkh, Khost-wa-freng, Sang-e-takht, Dulina, Hesarak and Sherzad) and Six provinces (Herat, Balkh, Baghlan, Dykundi, Ghur, Nangarhar)  identified for launching of the survey.

1.3Design of the Survey Instrument:

The survey instruments were drafted by The Evaluation Team in English and were arranged for translation in Dari. Six types of questionnaires such as; male household, female household, male focus group, female focus group, village map and radio form were designed and were printed out for data collection aspects based on the survey sample plan. Likewise, other instruments like; survey guideline, survey books, historical cards, literacy cards, occupation cards and etc. were prepared and printed out to facilitate the data collection activities. The pilot test of the draft questionnaires has launched in 3 districts of Kabul province (Estalif, Shakardara and Mirbacha kot).

1.4Hiring and Training of Survey Supervisors, Enumerators, and other Survey Staff:

Around 140 candidate surveyors from various provinces were called for survey training in, Kabul. The training was launched for 17 days in three stages and among 120 candidates 65 were selected by evaluation team, as field supervisors and surveyors to administrate the data collection of the survey.  The training of enumerators, supervisors, and survey staff was conducted in accordance with guidelines established by the Evaluation Team.

1.5Administration of data collection:

The second follow-up survey data collection launched followed the completion of training, selection of survey personnel and finalization of survey instruments and implementation guidelines. Sixty-five selected surveyors and supervisors divided to 4 teams, each consist of one Supervisor, 3 Editors and 12 Enumerators (in couple bases) in accordance with residential area and familiarity of surveyors to assigned locations. Subsequently, each team repeatedly subgrouped to 2 couples (4 Enumerators with one Team Leader) and one Editor in each. The data collection process was started on May 2011 and was completed almost end of October 2011. The data collection activities opened in Adreskan district of Herat province and followed by Balkh, Khost-o-Freng, Gulran, Daikundy, Hesarak, Farsi, Dulina, Chesht-e-Sharif and Shirzad districts in a sequential order. Financial and budgetary arrangements pertaining to the second follow-up survey were well managed and finalized in consultation with the NSP Finance Department, the World Bank, and the Evaluation Team. The budget and expenditures were arranged in accordance with the survey monthly action plan and were documented based on the NSP financial policy and regulations. (For information about project budget and expenditures see Annex1)

 

NSP 2nd Follow up Survey, Total Household and Focus Groups Questionnaire

Locations

HH Questionnaire

FG Questionnaire

Actual Form Exist

No

Province Name

District

No of District

No of Villages

Male

Female

Total HH

Male

Female

Total HH

1

Herat

Adreskan

1

50

468

458

926

49

48

97

1,023

2

Baghlan

Khost-o-firing

1

50

500

500

1000

50

50

100

1,100

4

Balkh

Balkh

1

50

500

499

999

50

50

100

1,099

5

Heart

Gulran

1

30

277

277

554

30

30

60

614

3

Nangarhar

Hesarak

1

42

420

300

720

42

30

72

792

6

Daikundy

Sang-e-takht

1

52

515

515

1030

41

43

84

1,114

7

Herat

Farsi

1

50

487

487

974

40

47

87

1,061

8

Ghur

Dulina

1

37

359

359

718

35

35

70

788

9

Nangarhar

Shirzad

1

36

360

0

360

36

0

36

396

10

Heart

Chesht Sharif

1

50

468

448

916

42

41

83

999

Grand Total

10

10

447

4,354

3,843

8,197

415

374

789

8,986

1.6Monitoring and Supervision:

As monitoring and supervision activities are the most important and credible question for reliability and confidentiality of the survey, particularly, during data collection, therefore, the VAU is seriously considering on this topic and rigorously conduct various methods of monitoring during and after completion of data collection.

  1. Direct Monitoring: While the data collection of NSP 2nd follow-up survey was in progress, around 12 field monitors were sent to 10 covered districts to check the quality and progress of data collection in aspects of following of “Implementation Guideline” as well as noticing of enumerators and supervisors problems were encountered in the field. All monitors conveyed a standard supervision form outlined for checking of data collection activities and manifestation of deficiencies and gaps undertaken by field staff. They implied their revisory recommendations to found problems defects. Findings were reported to office and were considered for next plans.

  1. Track Points by TRACK 24: Modern technology such as; TRACK 24 and GPS devices were used as online monitoring by central office. Each couple of assigned teams conveyed a TRACK 24 which directly was connected to office PCs and was online controlled each 10 minutes with exact information about location of surveyors and data collection progress. GPSs were also used to track point each interviewed household, which will be used for map designing and household determination in future.

  1. Re-monitoring and Data Quality Checking: After completion of data collection in each district a monitor was sent to the same district to check and compare the collected data with several local agencies and stakeholders to verify the accuracy of obtained data.

1.7Data Processing Management:

Based on the MoU (Memorandum of Understanding) between MRRD and Impact Evaluation Team the VAU was expected to manage all data processing arrangements of NSP 2nd follow-up survey pursuing the data collection of the survey. Behind the pilot test and finalization of survey questionnaire, a database using the last version of MS Access, a user friendly package, was designed to facilitate the manual data entry of the survey. A team of Data Entry Clerks, supporting data entry process were hired and well trained. Database and data entry process were piloted and finalized for actual data entry occasion. All necessary technology and equipment such as, desktop computers, stabilizers, UPS batteries and etc. required for data processing response to an electronic database were procured and assembled. The completed survey forms were transferred to the data entry center in Kabul at regular base intervals. The data entry process was started after completion of data collection in first district and behind checking of filled forms in central office on 19th June 2011. It was progressively continued up to 31th Oct 2011. The double data entry system was designed for consistency check and quality control, and the process has started awhile after single entry was initiated. The double data entry for 2nd follow-up survey continued without work stoppage till last returned questionnaire and was completed on 30 November 2011. The data entry process rigorously supervised by VAU management and datasets were supplied to Evaluation Team in regular basis of one-week interval. Daily discovered problems in data entry, database structure and database appending process were solved by VAU Data Processing Management Team.

Data Entry & Data Double Checking Progress till end of December 2011

No

Province

District

Single Entry

Double Entry

Checking Number (Double System)

Checking Text
(Double System)

Data Checking

Database Restructure

Triple Entry

Checking Number (Double System)

Checking Text
(Double System)

Questionnaire
Data Checking

1

Herat

Adraskan

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

2

Baghlan

Khost-o-Firing

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

3

Herat

Gulran

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

4

Balkh

Balkh

Done

Done

Done

Not Done

Not Done

Not Done

Not Done

Not Done

Not Done

Not Done

5

Daikundy

Sang-e-takht

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

6

Nangarhar

Hesarak

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

7

Herat

Firsi

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

8

Ghur

Dulina

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

9

Nangarhar

Shirzad

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

10

Herat

Chesht-e-Sharif

Done

Done

Done

Ongoing

Ongoing

Done

Not Done

Not Done

Not Done

Not Done

 

2Village Benefit Distribution Analysis (VBDA) Survey:

2.1Background:

This pre-analysis plan for the Village Benefit Distribution Analysis (VBDA) specifies methodologies and specifications to be employed in testing hypotheses pertaining to the impact of institutional changes generated by the National Solidarity Programme (NSP) and related interventions on local governance outcomes. The plan was written prior to data analysis and pre-commits the authors to defined specifications for estimating impacts.

The plan is structured around four sections, one table, and three appendices. Section II provides an overview of the study, detailing the nature of the treatment and associated interventions, the sample, and the experiment deployed to generate data for the study. Section III lists the hypotheses for the study. Section IV describes the methodology and specifications to be employed in hypothesis testing. Table I lists the hypotheses and indicators for the study. Appendices I – II list questions included in the Village Benefit Distribution Surveys and Appendix III lists questions from the Second Follow-Up Survey that are to be used in the analysis.

2.2Description of Dependent and Explanatory Variables and Data Collection:

The VBDA is a behavioral experiment designed to identify how externally-imposed institutional changes generated by NSP impact the performance of local leaders in delivering public services. The experiment will also identify impacts of mandating the participation of different groups of local leaders in service delivery.

The experiment generates quantitative measures of leader behavior by organizing and then monitoring village-level distributions of food aid across 500 villages in Afghanistan. The distribution of food aid provided by external agencies is a common public service performed by local leaders in rural Afghanistan. Such distributions present leaders with the option to distribute aid equitably to vulnerable villagers, distribute aid equally among all villagers, or embezzle aid for personal or familial benefit. Data on how leaders distribute food aid are to be collected by surveys administered to random samples of village households and aid recipients following the distribution.

The sample for the study comprises a sub-sample (to be selected based on accessibility given local security and weather conditions) of 500 villages spread evenly across ten districts in Balkh, Baghlan, Daikundy, Ghur, Herat, and Nangarhar provinces.5 The ten districts were selected based on size, security conditions, and the constraint that no villages in the district had previously participated in NSP. In each of these ten districts, NSP Facilitating Partners were asked to select 50 villages which they deemed suitable for NSP implementation,6 which in turn provided the sample of 500 villages.

Although the 500 sample villages cannot - by virtue of their clustering in only ten of 398 districts and the non-random selection of the ten districts in which they are located - provide a random sample of villages in Afghanistan, the 500 selected villages are representative of Afghanistan’s geographic, ethnic, and economic diversity. An assessment of the demographic and economic characteristics of the 500 villages reveal few substantive differences with those of a random sample of villages surveyed by the 2007-08 National Risk and Vulnerability Assessment.

2.3Hiring and Training of Survey Supervisors & Enumerators:

Around 200 candidate surveyors from various provinces were called and trained in different training courses and different regions such as; Balkh, Kabul and Herat provinces for administration of VBDA survey. The training was launched in various sections and stages, among 200 candidates were participated in these trainings around 100 of them were selected by evaluation team, as field supervisors and surveyors to administrate the data collection of the survey.  The training of enumerators, supervisors, and survey staff was conducted in accordance with guidelines established by the Evaluation Team.

2.4Structure of Village Benefited Distribution and Administration of Data Collection:

The food aid distribution and the associated data collection generated quantitative data on the behavior of village leaders in the performance of a key public service, which served as the dependent variable for the study. Distribution and data collection activities were implemented during three visits to each village made by distribution agents and enumerators over three, non-consecutive days. The data collection activities was started on 15th July and completed on 31st Oct 2011.

Day 1: A distribution agent was hired and trained by the evaluation team, traveled to the village and convened a short meeting of the village leaders. The agent informed the leaders that the village has been selected to receive a fixed quantity of wheat (approximately one 50 kilogram bag for one sixth of the households in the village) from the World Food Programme (WFP) and that, in three days, WFP will be delivering the wheat to the village. Leaders were informed that the wheat was intended for distribution among vulnerable members of the village community, such as widow-headed households, and were directed to determine which households in the village are to receive the wheat. In accordance with WFP monitoring requirements, leaders were asked to prepare a list of households in the village that will receive wheat, complete with the amount that each household will receive, and were informed that the list will be collected by a WFP representative on the day of delivery.

Day 2: Three days later, a different distribution agent returned to the village with the quantity of wheat was allocated for distribution in that village, which was transported either by a WFP vehicle or the vehicle of an agency contracted by WFP to conduct the delivery. The agent collected the list of wheat recipients were prepared by the village leaders and, after verifying the legibility of the list, authorized the transfer of the wheat to the village leaders.

Day 3: Ten days after the distribution of wheat, a team of 12 male and 12 female enumerators returned to the villages to administer household surveys to three groups of villagers:

First Survey: Random sample of households in the village;

Second Survey: Random sample of households that were listed by village leaders to receive wheat;

Third Survey: Random sample of households that First Survey respondents indicated received wheat, but which were not included in the list provided by the village leaders.

Survey questionnaires are administered conterminously to a male and a female respondent in the same household. Questionnaires were administered to 12 First Survey households and 15 total Second and Third Survey households.18 First, Second, and Third Survey questionnaires were identical and were collected basic demographic and socioeconomic data on the characteristics of the respondent’s household, as well as information on the distribution and characteristics of recipient households. (For information about project budget and expenditures see Annex1)
 

VBDA Survey Coverage Details

 

Location

# of Questionnaires and Households Interviewed

ID

Province

District

Total Villages assessed

1st Survey-Male

1st Survey-Female

2nd Survey-Male

2nd Survey-Female

3rd Survey Male

3rd Survey Female

Total-Male Questionnaire

Total-Female Questionnaire

Total HHs Questionnaire

1

Herat

Adreskan

43

667

685

329

364

158

169

1,154

1,218

2,372

2

Balkh

Balkh

50

592

593

424

427

246

246

1,262

1,266

2,528

3

Baghlan

Khost-o-Firing

47

669

669

531

531

120

120

1,320

1,320

2,640

4

Herat

Farsi

45

632

643

440

445

113

118

1,185

1,206

2,391

5

Nangarhar

Hesarak

38

570

570

378

378

192

192

1,140

1,140

2,280

6

Herat

Gulran

28

425

424

236

253

94

102

755

779

1,534

7

Daikundy

Sang-e-takht

50

706

710

381

388

73

73

1,160

1,171

2,331

8

Ghur

Dulina

26

369

294

298

235

45

35

712

564

1,276

9

Nangarhar

Shirzad

36

541

0

480

0

54

0

1,075

0

1,075

10

Herat

Chesht-e-Sharif

41

544

536

439

437

96

101

1,079

1,074

2,153

Grand Total

363

5,171

4,588

3,497

3,021

1,095

1,055

9,763

8,664

18,427

2.5Monitoring and Supervision:

The same procedure and process as NSP 2nd follow-up survey have been used for monitoring and supervision of VBDA survey data collection to maintain the data quality and credibility.

2.6Data Processing Management:

As per agreement of Evaluation Team the data entry process of VBDA survey contracted with a private company, while designing of database, double checking system and data entry supervision have been done by VAU data processing team. The data entry and data cleaning of VBDA survey is expected to completed at end of April 2012.

3Foundation of Monitoring and Evaluation Training Program:

The 5 core staff of VAU participated in Foundation of M&E training course conducted by Arab Administrative Development Organization (ARADO) was held on 12th – 17th March 2011, at Sharjeh, U. A. E.

The workshop provided an overview and an introduction to results-based monitoring and evaluation (M&E) with an emphasis on practical applications of tools and methods. The training was intended for participants with prior experience with monitoring and evaluation.

The participants have gained an improved understanding of how M&E can improve the quality of their projects while promoting learning and accountability. Similarly, they learned how to be better managers and consumers of evaluations conducted by others. Additional sources of web-based information on M&E were provided at the end of the training materials.

The objectives of the course were as follows:

1) To gain confidence to apply the acquired skills and knowledge to M&E work.

2) To be able to share and learn from other country experiences to strength their M&E systems.

The training mainly focused on the concepts of results-based monitoring and evaluation, results chains and frameworks indicators, evaluation designs, data collection, sampling, data analysis, reporting and managing results-based evaluations, based on feedback from participants the course was really productive and useful for further support of a VAU team within AIRD/MRRD. The training was delivered through lecture, presentation, focus group, cause study and other usual methods.

4Foundation of M&E and an Explanation on MDG and ANDS Training Workshop:

Coincide with returning of VAU staff from abroad training, another training program, focus on M&E tools and methods with extra point on MDG and ANDS designed for the rest of VAU and AIRD staff (almost 25 participants). The training launched for 5 days on April 2011.

The workshop provided an overview on Millennium Development Goals (MDG), its objectives and indicators, similarly, an introduction on Afghanistan National Development Strategy (ANDS) pillars, sectors and indicators. Moreover, the program was focused on M&E tools and methods.

The participants have gained an improved understanding of how M&E can improve the quality of their projects while promoting learning and accountability. Additional sources of web-based information on M&E were provided at the end, along with training materials.

The objectives of the course were as follows:

  1. To gain some information on MDG and ANDS objectives and indicators.

  2. To gain confidence to apply the acquired skills and knowledge to M&E work.

. The training was delivered through presentation, group exercises and other usual methods.

5Training Workshop on MS Access 2010 and Database Design:

Following to the last correlated training course titled “Foundation of MS Access 2007” carried out on 2009, on this occasion an advance level of training in the field of database and MS Access titled “MS Access 2010 and Database Design” has launched on 20th April to 15th May 2011, with attendance of around 30 participants consisting VAU, AIRD and MRRD programs employees. The key concepts of the training were; updates and utilities in last version of MS Access package, how to use MS Access Objects (table, query, form and etc.), making relationship, making different queries, designing of database and etc. Based on feedback from the participants the course topics were really productive and useful for all participants particularly, those who are typically employed in the field of database management, MIS, research, survey, M&E and so on. The training was delivered through presentation, project base practical work, group and individual assignments and other usual methods. At the end and after a final test a benchmark certificate was distributed to those with full attendance.

6 Databases Designed for Different Units within AIRD:

  • Gender in development study Database for AIRD Research Unit.

  • Failed Water Points and Poor Off take of Sanitation database for research unit.

  • DDA Rapid Survey Database for AIRD Research Unit.

  • Resource Inventory Survey database (RTP Unit/ AIRD)

  • Library Database for electronic recording of all existing books within AIRD Library.

  • Updating of Library Database.

7 Future Plans:

  • Conduction of NERAP (National Emergency Rural Access Program) 1st Follow-up Survey (VAU extension and the WB agreement are required).

  • Conduction of FSMS round 9th survey for World Food Program (WFP) in 2012.

  •  VAU impart in launching of 2nd round “Afghanistan Rural Development Report” via conduction of a nationwide research.

  • VAU inclusion in implementation of “Support Change Management” baseline survey for MRRD.

  • Conduction of evaluations and other comprehensive surveys for MRRD  Program and other organizations ( Agreement of MRRD Management is required)

  • Statistical and Impact Evaluation capacity building of VAU staff via conduction of training programs inside and outside of the country.

  •  Technical capacity building of VAU, symphonic with the world standards for conduction of researches/surveys.

  •  Consolidation, presentation and dissemination of existing data within VAU/AIRD in AIRD/VAU website by designing of an OLAMP (Online Micro-analytical data processing System) database to provide a user-friendly means of dissemination.

8 Challenges:

  • Deficiency in financial sustainability of VAU in capacity building and survey implementations perspectives.

  • Minor shortage of technical capacity in update and standardized technologies and tools required for implementation of impact Evaluations and other comprehensive surveys and advance level of Impact Evaluation skills.

  • Security restrictions in some part of the country against implementation of impact evaluations and other surveys.

  • Lack of administrative facilities/support versus VAU activities and survey implementation such as; pick and drop transportation, telecommunication, local transportations, internet connection and electricity for utilization of new technologies in some regions and etc.

 

9Suggestions:

 

  • Financial support of MRRD Management for further sustainability of VAU

  • Further financial/technical support and collaboration of the WB with VAU in future and after closing of IDF project.

  • Identification and clarification of VAU position within MRRD and AIRD.

  •  Specification of an allocated amount for impact evaluation of MRRD development programs by Programs Managements (has not considered by some programs yet).

  • Preservation of VAU/AIRD independency in implementation of Impact of Evaluations for governmental and nongovernmental organizations.

  • Agreement of MRRD Management for inception and implementation of surveys/researches for other organizations.

  • Further support of the WB and MRRD Management in conduction of planned training programs for capacity building of VAU/AIRD

 

Annex-1: Project wise Budget and Expenditures of year 2011

No

Project Name

Donor Agency

Grand Total Amount (2009-2011)

Total Expenditures 2009 & 2010

Total Expenditures   up to end December  2011

Balance in USD

% of Expenses

US$

AFS

1

Impact Evaluation Capacity Building (IDF)

The WB

383,000

0

200,157

324,733

58,267

85%

2

NSP 2nd Follow-up Impact Evaluation

The WB (NSP)

402,000

0

0

301,961

100,039

75%

3

“Village Benefited  Distribution Analysis” Survey

The WB (NSP)

184,000

0

0

103,235

80,765

56%

Total

969,000

0

200,157

729,929

239,071

 

 

Annex: 2. List of VAU Contract Staff 2011

#

Name

Position

Email Address

Project

Qualification

Work Experience

1

Dr. Mohammad Sediq Baraky

Survey Manager

sediq.baraky@mrrd.gov.af

IDF/WB

MD

7 YEARS

2

Ahmad Tariq Wardak

Database Manager

tariqwardak@yahoo.com

IDF/WB

BSc.

11 YEARS

3

Mohammad Pervez Jalili

Admin/Finance Manager

pervez.jalili@mrrd.gov.af

IDF/WB

BBA

7 YEARS

4

Abdul Hakim Zarin

Sr. Data Analyst

hakim.zarin@mrrd.gov.af

IDF/WB

High School

8 YEARS

5

Saifudeen Ghfoori

Data Analyst

saiff.jalaly@mrrd.gov.af

IDF/WB

BSc.  ongoing

8 YEARS

6

Ahmad Reshad Nabi

Data Analyst

reshad143m@gmail.com

IDF/WB

BSc. ongoing

5 YEARS

7

Mohammad Rafi Azimi

Survey Assistant

rahimi_2222@gmail.com

IDF/WB

14th Class

4 YEARS

8

Abdul Ghafar Rahimi

Survey Assistant

abdulghafar_rahimi@yahoo.com

IDF/WB

High School

7 YEARS

9

M. Zekria Fetrat

Survey Assistant

zekria2009@gmail.com

IDF/WB

DBA

7 YEARS

10

Abdul Qader Bahadurzai

Survey Assistant

qadir_khan121@yahoo.com

IDF/WB

BCS

5 YEARS

 

 

 

 

List of VAU Civil Servant Employees -2011

No

Name

Position

Email Address

Grade

Qualification

Work Experience

1

Hezbollah Shahidzoi

Evaluation Manager

Hezbullah.shahidzoi@gmail.com

4

Bachelor

1 Year

2

Masooma Bahar

Database Officer

 

5

14th Class

1 Year

3

Tamana

Data Collection Asso.

 

6

BBA ongoing

1 Year