Assessing the Impact of Integrated Natural Resource Management: Challenges and Experiences

María Verónica Gottret¹ and Douglas White²

• Abstract
• Introduction
• Impact Assessment Challenges of Integrated Natural Resource Management
  • Finding feasible spatial and temporal boundaries
  • Stakeholder participation in the impact assessment process
  • Linking research products to development impact
  • Selecting the criteria of success
• An Analytical Framework for INRM Impact Assessment
• Assessing the Impact of INRM: a Proposed Methodology
  • The "paths to development impact" methodology
  • A path to development impact methodology for CIAT INRM research
  • Implementing the path to development impact with local stakeholders
• Emerging Lessons and Future Research Questions
• Responses to this Article
• Acknowledgments
• Literature Cited
• Appendix 1

Assessing the impact of integrated natural resource management (INRM) research poses a challenge to scientists. The complexity of INRM interventions requires a more holistic approach to impact assessment, beyond the plot and farm levels and beyond traditional analysis of economic returns. Impact assessment for INRM combines the traditional "what" and "where" factors of economic and environmental priorities with newer "who" and "how" aspects of social actors and institutions. This paper presents an analytical framework and methodology for assessing the impact of INRM. A key feature of the proposed methodology is that it starts with a detailed planning process that develops a well-defined, shared, and holistic strategy to achieve development impact. This methodology, which is known as the "paths of development impact" methodology, includes the mapping of research outputs, intermediate outcomes, and development impacts. A central challenge is to find a balance between the use of generalizable measures that facilitate cross-site comparison and slower participatory process methods that empower local stakeholders. Sufficient funding for impact assessment and distinct stakeholder interests are also challenges. Two hillside sites in Central America and one forest margin site in Peru serve as case studies.

KEY WORDS: agriculture, impact assessment, integrated natural resource management, monitoring and evaluation, natural resources, sustainable rural livelihoods.

Published: December 20, 2001

Research conducted during the past 30 years by international agricultural research organizations has been successful in boosting productivity and providing enough food to feed the world. However, problems of food security, poverty, and natural resource degradation have persisted (Consultative Group on International Agricultural Research 2000). As a result, a new management and research approach called integrated natural resource management (INRM) is emerging. This is an evolving research and development (R&D) approach that combines the interconnected goals of poverty reduction, food security, and environmental sustainability. It is defined as " ... the conscious process of incorporating multiple aspects of natural resource use into a system of sustainable management to meet explicit production goals of farmers and other uses (e.g., profitability, risk reduction) as well as goals of the wider community (sustainability) ... " (Consultative Group on International Agricultural Research 2000). To increase its potential impact, research on INRM is normally developed and planned by multiple organizational partners. As a result, INRM is not only conventionally defined research, but also "action research" that supports development initiatives led by or collaboratively managed with partners.

International R&D has also entered a new phase in which these multifaceted efforts must be accomplished with the efficient use of financial resources. During the last decade, slower growth in international research funding has resulted in calls to demonstrate research impact. For example, the real growth rate of funding to the Consultative Group on International Agricultural Research (CGIAR) declined from 4.0% per year in the 1980s to 0.5% per year since 1990 (Alston et al. 2000). To meet the triple challenge of (1) demonstrating more diverse impact with (2) fewer financial resources and (3) showing how this can be achieved, INRM requires a more holistic impact assessment (IA) approach that evaluates not only economic changes but also changes in the environment, people, and their organizations.

Impact assessment can also be a powerful tool when it come to quantifying benefits and obtaining a better understanding of the processes by which research translates into impacts. The quantification of the benefits that accrue from research interventions and outputs responds to the need for both more transparent accountability and good public relations. Impact assessment also facilitates learning by providing feedback to improve the efficiency and effectiveness of research. It should be noted that IA is not a retrospective exercise to be conducted at the end of a research project, but rather requires systematic and continuous monitoring (D. Pachico et al., 1998, unpublished manuscript). By identifying potential positive and negative research outcomes, it is possible to provide feedback to processes for planning and assigning priorities. Thus, information produced by IA permits scientists and other stakeholders to make more informed decisions.

Because of the larger-scope issues of INRM, there is still a lot to do before IA can be used to evaluate the final effects of more complex R&D projects. Moreover, a balance must be found between the use of generalizable measures that facilitate cross-site comparison and slower participatory methods that empower local stakeholders.

In this paper we present the advances and accomplishments of scientists from the Centro Internacional de Agricultura Tropical (CIAT) who developed and implemented an IA framework for three INRM sites in Latin America that represent two distinct agroecosystems: hillsides and forest margins. The objectives of this paper are to propose an analytical framework and methodology for impact assessment and to discuss the challenges encountered during its initial application at the reference sites.

International agricultural research and the methodologies used to assess its impact have long reflected contemporary rural development thought. For decades, scientists have measured the impact of research, producing a vast number of studies (see Anderson et al. 1998). Themes related to the adoption of agricultural technologies and the economic returns on related investments dominate this literature. Reviews can be found in Lee et al. (1991), Horton et al. (1993), and Collinson and Tollens (1994). These studies continue to be synthesized and extended via new, robust quantitative methodologies that use economic surplus methods and estimate the net present value of benefits, internal rates of return, and cost-benefit ratios (Alston et al. 1995). Other impact assessment approaches include quantitative, qualitative, and often noneconomic measures that capture a variety of R&D effects (see Horton et al. 1993). Even though these measures do not use a common metric such as economic returns, they provide important information for giving priority to certain types of research. The potentially broad impacts of research on integrated natural resource management (INRM) require more comprehensive assessment measures than those of economic returns and the adoption of agricultural technologies.

Impact assessment is related to other concepts that require definition. For example, "monitoring" refers to the systematic and continuous process of assessing the progress and changes caused by the implementation of an activity, usually by means of predetermined indicators or recurrent questions. "Evaluation" identifies the broader positive and negative outcomes of an activity or process, draws conclusions about its overall value, and decides whether its objectives have been met (Guijt 1998). Monitoring and evaluation (M&E) are often considered to be a linked process.

No commonly accepted protocol exists for assessing the impact of INRM. The following four major themes represent different dimensions of the challenges involved in assessing its impact.

Finding feasible spatial and temporal boundaries

The impacts of INRM can be hard to evaluate because of its multifaceted nature. At different spatial and temporal scales, INRM themes change and can be assessed in terms of multiple objectives, e.g., poverty alleviation, ecological resilience, natural resource conservation, economic growth, and human and social development, that reflect the needs and expectations of different stakeholders (Izac and Sanchez 1998). Plot- and farm-level analyses that are relatively manageable expand to the more unwieldy scales of the community, the watershed, and even larger areas. Hierarchies and a wider scope introduce numerous complicating ecological, social, cultural, institutional, economic, and political factors.

The measurement of impact across spatial scales is a key issue in the impact assessment of INRM research (D. Pachico et al., 1998, unpublished manuscript). For example, measures to assess the impact of erosion on the natural resource base differ from plot to slope face to watershed to river basin. Unlike agricultural research, impact assessment (IA) for INRM research concerns a number of ecosystem services that are distinct at each spatial scale, e.g., farming system, village, watershed, landscape, regional, and global (Izac 1998), and therefore depend on the spatial pattern of adoption.

With respect to temporal scales, Scherr (2000) argues that longer time periods are required to assess sustainability, despite the fact that these longer periods increase the likelihood of major socioeconomic or biophysical shifts taking place. Impact assessment faces the so-called "attribution problem" that confounds the impact measurement of R&D innovations with changes in other structural variables over time, such as changes in macroeconomic policies or environmental disasters. Therefore, IA for INRM must be designed to consider long-term change and, at the same time, analyze shorter-term successes and/or failures with regard to the achievement of the desired impact with M&E. To assess long-term changes, different instruments can be used, such as time series of remotely sensed variables, panel data sets, historical reconstruction of plot- or community-level trends with local people, and participatory historical mapping (Scherr 2000). At the same time, the issue of realizing results after a protracted time is not a new challenge to INRM alone. Research programs for agricultural crop improvement took at least a decade from their outset to deliver the first outputs, and in many cases even longer. INRM research may not require much longer than that to produce results (D. Pachico et al., 1998, unpublished manuscript).

Stakeholder participation in the impact assessment process

As a result of scale hierarchies, many perspectives, e.g., those of farmers, rural inhabitants, community organizations, support organizations, private entrepreneurs, scientists, donors, and policy makers, can lead to conflicting objectives, interests, and opinions. Therefore, a central concern of INRM is to both identify and assess the trade-offs among the various stakeholders (Izac 1998). Communication breaches and distinct stakeholder perspectives pose special challenges to measuring the impact of INRM. Differences in the ways stakeholder groups communicate with each other can sabotage dialogue, learning, and cooperation. These communication difficulties may, in fact, be symptoms of deeper, more fundamental differences in the ways that these groups learn and view the world (van Dusseldorp and Box 1993).

Understanding and properly addressing both private and public incentives are the key difference between previous R&D efforts and those of INRM. To make optimal decisions, it is necessary not only to analyze trade-offs to determine who will benefit and who will not, but also to establish a forum for careful negotiation and collective action.

Participatory monitoring and evaluation (PM&E) approaches have been proposed for efficient, effective, and socially inclusive assessment (Abbot and Guijt 1998, Guijt 1998). The PM&E approach has four identified benefits. First, because the application of INRM research produces a broad range of impacts in communities, including many of a social nature, these changes can be difficult for outsiders to observe (e.g., forest or water access). Participatory M&E improves the accuracy and relevance of the collected data, thereby reducing some of the problems inherent in externally defined IA studies, which risk misinterpreting the findings that result when variables are omitted (Scherr 2000). Second, participation in self-evaluation can be a powerful contribution to local institution building (Scherr 2000). Guijt (1998) emphasizes that PM&E enhances local capacity to record and analyze change and improve community-based initiatives. Third, PM&E is cost-effective because it employs local capabilities rather than expensive external experts (Guijt 1998). Fourth, including stakeholders in the entire process increases the probability that the analyzed data and results will be used.

Although PM&E has its advantages, it also has drawbacks. Data collection on a voluntary, unremunerated basis, as often happens in participatory processes, is unlikely to be sustained unless the information has some direct relevance or value for community members, community organizations, and local and national organizations. In addition, PM&E can be expensive for local people because it involves a significant amount of time that could otherwise be spent on income-earning activities. Before embarking on a PM&E process, a basic understanding is needed of the limitations of what is possible to avoid poor-quality work and disillusionment.

Different assessment needs and interests may require parallel IA processes conducted by external and local groups; results and analysis can then be integrated. Therefore, a central challenge is to find a balance between faster generalizable measures that facilitate comparison across sites, and slower participatory processes that empower local stakeholders and validate results and analyses.

Linking research products to development impact

Demonstrating a clear link between research products and changed livelihood strategies that lead to development is a major difficulty in assessing the impact of INRM. Even though connections are plausible, they are often treated as a "black box" that cannot be analyzed or explained in detail (Goldsmith 1993). An impact cannot be attributed to an intervention unless it can be logically explained and justified. The final impact may be great, but the causality is usually too subtle to measure exactly. Complications are particularly acute with the diffusion of agricultural research and technology, because a long chain of events separates innovations made in the scientist's laboratory from their deployment in the farmer's field. Furthermore, it remains unclear how far scientists should work across the research-development continuum to increase the probability of making an impact.

Therefore, understanding the linkage between research products and development impact is a critical issue for both IA and M&E. To assess the impact of research, we must determine if and how people and organizations react, and whether they change their actions as a result (Sander, 1998, unpublished manuscript). These measures, called project "reach," refer to both the number and type, e.g., local or regional, of organizations affected. The methodology described below traces the multiple, and often nonlinear, chains of events that link researchers and their findings with other actors essential to the change process.

Selecting the criteria of success

Indicators are central to IA and M&E approaches because they help to communicate information about complex processes, events, or trends to a wider audience (Guijt 1998). However, selecting indicators is a difficult step in establishing these processes. These problems include the fact that:

• a range of different indicators is required for each "output," "outcome," and "impact";
• chosen indicators are likely to change over time as the external environment changes and as the project objectives are adjusted;
• stakeholders select indicators based on different cultural values, priorities, information needs, and expectations;
• indicators are only proxy measures of a more complex reality that are required for empirical analysis; and
• indicators, even when they are relevant and accurate, are influenced by practicality and cost concerns related to data collection and analysis.

In summary, the challenges of impact assessment reflect the complexity of the interrelated goals and actors of INRM.

An impact assessment (IA) methodology for integrated natural resource management (INRM) should help clarify how an intervention affects a society's economic, financial, natural, social, human, physical, and other resources. The "sustainable rural livelihoods" framework described by Scoones (1998) has scope for broad application to IA methods including INRM. Drawing on Chambers and Conway (1992), Scoones (1998) defines "sustainable rural livelihoods" as:

... the capabilities, assets (including both material and social resources), and activities required for a means of living. A livelihood is sustainable when it can cope with and recover from stresses and shocks, and maintain or enhance its capabilities and assets, while not undermining the natural resource base.

This definition can be divided into two subcomponents that reflect the themes of INRM. The first focuses on well-being or livelihoods and includes aspects of employment, income, and poverty reduction. The second is the sustainability dimension, which includes the resilience of livelihoods and the natural resource base on which they depend.

The IA analytical framework is based on the framework of sustainable rural livelihoods (see Fig. 1). This IA framework contains four key components that reflect the state of development (including the context), the process of development (livelihood strategies), institutions and organizations, and R&D interventions. An initial assessment or baseline study describes the current state of livelihood resources or the "capital" base from which different production processes are derived for each reference site. This capital base has five dimensions:

1. Economic/financial capital: the capital assets (cash, credit/debt, and savings) that are essential for the pursuit of any livelihood strategy.
2. Physical capital: the household assets and farm infrastructure, including production equipment, technologies, and plantations.
3. Natural capital: the stock of natural resources (soil, forests, water, air, genetic resources, etc.) and environmental services (hydrological cycle, carbon sequestration, etc.) from which both resource flows and useful services for livelihoods are derived.
4. Human capital: the capacities, skills, knowledge, ability to work, good health, and physical capability important for the successful pursuit of different livelihood strategies. Human capital can be developed consciously through formal education and training and unconsciously through experience.
5. Social capital: the social resources (networks, social relations, affiliations, associations, norms, trust, and disposition to work for the common good) upon which people draw when pursuing different livelihood strategies requiring coordinated and collective action.

Fig. 1. Analytical framework for integrated natural resource management impact assessment on sustainable rural livelihoods (adapted from Scoones 1998).

The baseline also provides a description and a contextual analysis of conditions, trends, and policy setting in the community. These components include the exogenous characteristics (structural variables) of a site such as its history, politics, macroeconomic conditions, terms of trade, climate, agroecological conditions, demography, and social differentiation.

Households and communities use three broad clusters of livelihood strategies: (1) agricultural intensification/extensification, (2) livelihood diversification within agricultural activities or nonfarm activities, and (3) migration. Livelihood strategies are part of the development processes that enable individuals, households, and communities to reach a modified development state and move from an initial development state toward a subsequent one. If people change their livelihood strategies, then their livelihood outcomes will also change. Because this is a dynamic process, livelihood outcomes are not static.

Institutions and organizations are at the center, as befits their role in binding together the elements of the framework. According to Scoones (1998), understanding institutional processes is a prerequisite to identifying restrictions/barriers and opportunities with regard to sustainable rural livelihoods. Because formal and informal institutions mediate access to livelihood resources, an understanding of institutions and organizations is critical for designing R&D interventions.

For INRM research interventions to have an impact on rural livelihoods, it is not enough to merely produce research outputs (the "what") that permit a better understanding of system dynamics and processes in a variety of sites (the "where"). It is essential to identify "who" is going to implement and adopt changes, and "how" to best improve livelihoods. Organizations, which are the vehicles of change, are thus the target for R&D interventions and the collective-action platform for planning, implementing, and evaluating them. Institutions provide the rules and norms by which individuals and their organizations operate, and therefore provide structures that can either hinder or foster the development processes. Adopting a sustainable livelihoods approach forces the R&D process to recognize the potentially " ... enormous level of organizational and/or institutional failures that exists and (therefore affects) the impact of agricultural research ... " (Shaxson, 1999, unpublished manuscript).

The natural resource management (NRM) projects of the Centro Internacional de Agricultura Tropical (CIAT) work in three different agroecosystems throughout Latin America, including hillsides (Yorito, Honduras, and San Dionisio, Nicaragua), savannas (Puerto Lopez, Colombia), and forest margins (Pucallpa, Peru). At these sites, CIAT integrates germ plasm, technology, NRM, and institutional innovations into pilot activities that, together with development partners, are introduced to achieve CIAT's mission of reducing poverty while preserving the natural resource base. The partners lead or play a major role in development planning and implementation. The main objective of the reference site approach is not the development of the reference site per se, but rather the refinement of social, economic, biophysical, and institutional innovations that can be extrapolated beyond the reference site.

The "paths to development impact" methodology

To assess the impact of INRM R&D efforts, the "paths to development impact" methodology is a helpful tool in establishing a monitoring and evaluation (M&E) system. These paths make it easier for all stakeholders to understand the ways in which research-based interventions are interrelated within complex systems; they also make it possible to gather credible evidence and analyze it in a convincing way. At the planning stage, three different points along the path can be distinguished: (1) outputs, or the immediate products of a project after using the given inputs, (2) outcomes or consequences of the outputs, and (3) impacts, i.e., the broader and longer-term goals.

As one moves along the pathway from a research intervention to a development impact, control over specific activities declines. Although performance or output indicators rely exclusively on the project, outcomes depend on other factors besides the intervention itself. Final development impacts go beyond the sphere of influence of an intervention and depend on a range of structural factors. These include the context conditions and trends, the given livelihood resources, and the institutional processes and organizational structures (Smutylo 1998).

Paths to development impact identify the direct users and indirect beneficiaries of research and also represent a set of possible routes by which research may produce an impact. They serve not only to set up an appropriate M&E process and effective IA, but also to provide a basis for designing delivery strategies for research outputs. This is because some paths will be more likely than others to achieve the desired impacts.

A path to development impact methodology for CIAT INRM research

A generic path to development impact was constructed for CIAT's INRM strategy (Fig. 2). Because INRM interventions move beyond the farm to the landscape, they require collective action that involves local institutional planning, implementation, and evaluation activities. The rounded boxes (five at the top and three at the bottom of Fig. 2) are the CIAT R&D interventions and correspond to those targeted at the institutions and organizations presented in the above analytical framework. The center row of four double-bordered boxes represents the process steps. The unshaded boxes are the expected outcomes of the interventions.

Fig. 2. Paths to development impact of CIAT's integrated natural resource management strategy. DST stands for "decision support tools."

The first intervention, organizational models, promotes collective action among community associations and support organizations for INRM. A second intervention includes two types of decision support tools (DSTs). The DSTs for local planning provide essential information for the design of a strategic plan for INRM, including a diversified portfolio of market chains to be promoted, current and potential land uses, and different alternatives for promoting sustainable land use management. The strategic plan also identifies the demands for technological land-use management options and provides feedback to NRM scientists.

Once a given portfolio of products is selected, DSTs for market chains and food security provide a better understanding of agricultural and land use constraints and opportunities for the development of market chains and food security. These action plans identify demands for germ plasm and integrated crop management technologies and serve as a feedback channel to scientists.

The third intervention, technological options and information, includes two groups of technologies (bottom of Fig. 2): land-use management options and integrated production system technologies. The first group permits a scaling-out of technological options from the farm to the landscape. The second combines germ plasm with integrated soil and pest management technologies at the farm scale for a given commodity.

The fourth area of intervention, participatory research methodologies, improves the effectiveness of the technology development, adaptation, validation, and diffusion process of both integrated production system technologies and land-use management options.

The strategic and action plans, along with the adoption of technologies, improve land use management, strengthen competitiveness, promote integrated production systems, and foster community empowerment. These changes in turn affect broader sustainable development goals by producing changes in the economy, the environment, the people, and their organizations.

Once the paths to development impact have been mapped, indicators for each critical outcome along the pathway need to be selected. Therefore, when objectives and users are clear and when outputs, outcomes, and impacts are mapped, the process of selecting and developing indicators becomes easier, and it is possible to avoid the trap of collecting a great deal of irrelevant information. Only indicators that are critical and feasible to monitor, analyze, and disseminate with the resources available should be chosen. In the case of CIAT, once the generic path to development was constructed, a list of proposed intermediate and development indicators that reflected the different capitals was prepared with the scientists involved (Appendix 1).

Implementing the path to development impact with local stakeholders

Initial efforts to develop a methodology to assess the impact of CIAT's NRM projects at three reference sites focused on creating the paths to development impact and developing and/or selecting indicators for the collection of baseline data. To implement the methodology with CIAT stakeholders, paths to development impact and their indicators were constructed in workshops conducted at the reference sites.

In April 1999, a workshop was organized with the dual objectives of establishing a structure to measure the impact of R&D efforts and to improve institutional coordination in Pucallpa, Peru. The effort followed a training seminar on participatory planning by objectives and logical frameworks held 2 years before. Representatives from more than 20 organizations attended a two-day Indicator Framework and Impact Assessment Workshop. Nongovernmental organizations, universities, and national and international institutions from diverse sectors of agricultural research and extension, health, and NRM attended the workshop.

Small groups of participants constructed paths to development impact by first identifying the goals of sustainable development. Goals developed from the previous workshop were used to avoid repeating the exercise. A series of intermediate steps representing the activities required to achieve these goals connected the research outputs with development impacts. Although some of the paths were relatively easy to construct, those related to institutional cooperation and policy formulation were the most challenging. The themes of technology development, management and conservation of natural resources, human and environmental health, and agribusiness development required less effort. However, some groups were dominated by the input of a few participants.

The participants agreed that an IA framework would facilitate the transparent reporting of results within their organizations and to donors. After the meeting, they set up working groups whose tasks were to refine indicators and collect data, but the participants met only a few times. Factors that contributed to the disintegration of these groups included the problems of overworked contributors, difficulties in finding convenient times to meet, and different institutional objectives. Had financial resources been specifically dedicated to this effort, the outcomes might have been more positive. Although these discussions could have led to an agreement on the paths to development impact and the indicators to be used, it was not possible to establish a participatory monitoring and evaluation (PM&E) process in this case. With decision-making power concentrated in the capital, there were irresistible incentives to obfuscate and even embellish reports in the highly politicized environment.

In July 2000, workshops and interviews were conducted at the Nicaragua and Honduras hillside reference sites to identify different development perspectives and goals of stakeholders ranging from community members and organization leaders to personnel from local organizations. Development objectives were discussed using a participatory tool developed by the CIAT team. First, a visioning exercise of the desired "development state" was conducted. The participants reflected on how they would like the watershed and its communities to be after 10 years. Based on this desired development state, current conditions were described. Finally, intermediate outcomes to be reached in the next 3 to 5 years were also proposed and discussed.

An analysis of the results shows that all of the groups were interested in human, social, economic, and environmental objectives. Scientists identified a broad range of indicators, whereas the inhabitants of poor rural communities had simpler, but specific and short-term, expectations. Rural people wanted to have a community where people have enough to eat, as well as a more diversified diet, and where they can buy clothes and shoes for their families. They also wanted access to the basic infrastructure and to appropriate education and health services. In contrast, the expectations of community organizations were focused on economic goals and tended to relate improved income and production as a means of meeting basic needs. Whereas scientists tended to see indicators and capitals as separate objectives, community organizations had a more complex and integrated perception of the different capitals.

It is clear that development agencies, donors and scientists consider central values such as gender, equity, and the environment to be significant. However, although community organizations are beginning to talk about gender as an important issue, they seem merely to be repeating the development discourse promoted by external organizations. Economic equity is rarely mentioned by community organizations. This could be related to the perception that "we are all poor," or the fact that the poorer community members often do not participate or express their opinions in community meetings.

With respect to the environment, whereas scientists talked about preserving or improving the natural resource base, community members placed more emphasis on the appropriate use and management of their natural resources. As a result of land degradation, recent natural disasters, water scarcity, and promotional campaigns by external organizations, an awareness of NRM appears to be growing. Rural dwellers give priority to the themes of improving land productivity and water availability, with an emphasis on having a community with more trees, especially around the watershed, and using farming practices that maintain soil fertility.

Often caught in the middle between donors and recipients, local governmental and nongovernmental support organizations have another perspective. Their central objective is building the capacity of local communities to analyze, reflect, and find collective solutions or alternatives to their problems. Communities, on the other hand, prefer fewer organizations, but with solid and clear goals that respond to community needs, and more external support. The emphasis placed by communities on the satisfaction of their basic needs provides a potentially effective entry point for intervention.

The preliminary results of impact assessment research for integrated natural resource management (INRM) carried out by the Centro Internacional de Agricultura Tropical (CIAT) are mixed, reflecting the multifaceted demands of INRM research. A major constraint to advancing the impact assessment (IA) effort was a lack of well-described pathways tracing research outputs to development impact. The principal reason for this situation was that, although it is useful for monitoring project performance and reporting, the logical framework does not provide a means of carefully tracking how research outputs translate into development impacts. Furthermore, the logical framework does not consider who would adopt project outputs and why, or how the adoption of these outputs will then contribute to the project purpose and goals (Sartorios 1996, Technical Advisory Committee of the Consultative Group for International Agricultural Research 1999). The development and integration of the paths took longer than expected, and a scientist was assigned to facilitate the task. By developing the paths with organizational partners, gaps and overlaps appeared, thus clearly illustrating both unmet needs and opportunities for collaboration.

These advances will enable cross-site comparison of the reference sites and could raise interesting new research hypotheses. From the CIAT research perspective, this effort to standardize IA has improved coordination across sites and among scientists. A synthesis of the development indicators (Appendix 1) reveals that economic and agricultural production indicators are similar across sites, whereas environmental indicators reflect local issues. Baseline indicators for the reference sites range in scale from plot level to farm, landscape, watershed, region, and continent. Interestingly, scientists found it difficult to agree on definitions of scale beyond the farm level. Although physical boundaries such as watersheds are helpful when associating biophysical measures, policy decision making generally follows political boundaries such as municipalities. In the end, there was a consensus that the data should be analyzed using human-defined scales.

At the heart of INRM impact assessment lie important questions about how practical it is when financial resources are scarce. The CIAT impact assessment advances show a dependence on both the organization's research priorities and the willingness of or buy-in from national partners. As part of an action research approach, the challenge will be to define where research stops and development begins. In other words, how far along the path to development must/should research organizations dedicate financial and scientific resources? The answer will depend on the strength of local organizations and on donor requirements. In Peru, stakeholder interest was so ephemeral that the project was placed on hold. In Central America, with its diverse stakeholder objectives, CIAT research efforts have a higher priority. These outcomes reflect a scarcity of scientific and financial resources within CIAT.

Preliminary results of the application of this analytical framework and methodology in Central America show a promising opportunity and have made it possible for the processes of monitoring, evaluation, and impact assessment to move forward at the reference sites. However, fieldwork has only just begun and must be completed as planned for 2001 and 2002. Two parallel processes are being started at the reference sites in Central America. The first is a participatory monitoring and evaluation (PM&E) process for three specific projects that were selected because they were already working with partners and had been fostering local collective action. The second process is a longer-term IA process based on the analytical framework and the methodology described in this paper. When these processes are completed, their results will need to be compared and integrated.

This research is just the start of a broader effort to develop an appropriate methodology for the assessment of the impact of more complex R&D interventions such as INRM. Further research is required to develop a model (or a methodology) to assess and validate the pathways that link INRM research outputs with intermediate outcomes and final development impact. This will permit an analysis of the influence relationships among them, and the contributions of research outputs to development outcomes. Research should also be conducted to provide an ex ante analysis of alternative INRM R&D interventions and pathways that affect poverty and sustainable development. Given that the INRM process often takes years, ex ante modeling efforts are needed to allow researchers and stakeholders to analyze likely development scenarios. Coordination between both modeling and impact assessment efforts, which require measures of many variables, can permit validation of these models and prevent a duplication of data gathering.

Despite the arguments in favor of such approaches, INRM will be hard to justify if its impact is subject to the sort of assessment used for crop improvement research. Impact assessment estimates of agricultural research using annual economic returns are often sophisticated and elegant. Moreover, they simplify decision making to relatively simple numbers and, even if these analyses are becoming suspect (see Alston et al. 2000), these estimates remain an easy point of comparison and represent nearly 40 years of cutting-edge research efforts. The wider and less measurable goals of INRM will continue to challenge attempts to estimate impact.

Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a comment, follow this link. To read comments already accepted, follow this link.

The authors wish to emphasize the CIAT-wide effort to advance the theme of NRM Impact Assessment and acknowledge input from Jacqueline Ashby, Nathalie Beaulieu, Alejandro Imbach, Douglas Pachico, Bruno Barbier, Jaime Jaramillo, Olaf Westermann, Sue Allan, Dean Holland, Mark Lundy, Tamsyn Murray, Manuel Winograd, Annie Jones, and Sam Fujisaka. Anonymous reviewers also provided valuable comments and feedback that substantially improve the quality of the paper. The authors are responsible for any errors or omissions.

List of proposed development indicators.

Food security

Total and gender-differentiated available food in the household
Seasonal available food in the household

Income

Farm income (U.S.$/ha)
Household income (U.S.$/household, U.S.$/capita)
Household income stability (standard deviation)
Income sources (on/off farm, agricultural/nonagricultural)
New products or processes introduced in the agricultural sector
Participation on final consumer price (community gate price/final consumer price of most important selected commodities)
Value added to primary production (selling price/primary product farm-gate price)
Distribution of income sources

Equity

Farm and household income comparisons by well-being group and by gender
Distribution of economic surplus or value added
Land tenure situation (percentage of farmers who own land, mean farm size)

Employment generation

Number of jobs (total, gender-differentiated, and seasonal)
Percentage of active population employed (total, employees, self-employment, and gender-differentiated)

Production system

Land use intensification

Number of cropping cycles per year
Fallow area as percentage of total area
Fallow time as percentage of cropping cycle
Fallow species diversity (native and enriched with introduced species)
Legume diversity
Land with improved pasture

Land use diversification

Number of species/crops per farm and per landscape
Number/type of cropping systems within the farm
Number of weeks per year that the soil is exposed to erosive effects (wind and water)
Number of perennial crops in the cropping systems
Spatial distribution of crops
Temporal distribution of crops
New resource management practices introduced in the agricultural sector

Land productivity

Kilograms per hectare per year for selected crops, livestock, and forest products
Total factor productivity
Product/energy ratio (biophysical measure)

Forest

Area of forest cover (ha/farm, ha/community, ha/municipality)
Map of forest cover (spatial and temporal distribution)
Secondary forest regeneration
Species diversity in secondary forest (native successions and enrichment with introduced species)

Soils

Land productivity

Kilograms per hectare per year for selected crops (already an indicator of production systems)

Water balance

Area and spatial and temporal distribution of degraded land and pasture

Water

Water pollution

Contamination points

Water quality

Sedimentation
Turbidity
Levels of nitrates and phosphates
Conductivity

Water availability

Volume of water by season
Volume of water by levels of rain

Other indicators

Agro-chemical use (kg/farm, number of applications)
Burning of agricultural residues (Yes/No)
Air pollution (Yes/No, CO, CO₂)

Community empowerment and equity in decision making

Projects designed and mobilized by the community and/or local and/or external organizations (number and description)

Participation in local policy decision making

Existing channels and mechanism for participation in democratic governance
Community access to political decision making
Community capacity to influence political decisions that affect the community

Human capital

Education and experience

Individual capacity (number of years formal/nonformal education, number of years experience)

Individual capacities

Capacity to participate in key decisions that affect NRM and primary production and agro-enterprises
Capacity to innovate in agriculture, agro-enterprises, and NRM
Knowledge of key causes and effects of NRM problems
Knowledge of key causes and effects of agricultural production, processing, and marketing relationships
Capacity to manage agro-enterprises, local agricultural research committees, community organizations, consortia, and farms

Access to opportunities

Percentage of children with access to primary education
Percentage of youths with access to secondary, technical, and university education
Technology
Market
Infrastructure
Health
Information


Social capital

Structural social capital

Organizational density
Networks
Support organizations
Procedures and norms
Precedents

Cognitive social capital

Conflict resolution
Collective action
Solidarity
Trust
Reciprocity
Cooperation

Nutritional levels
Access to health services and medicines
Access to consumer goods (clothes, shoes, etc.)
Migration
Local well-being indicators

Abbot, J., and I. Guijt. 1998. Changing views of change: participatory approaches to monitoring the environment. SARL Discussion Paper Number 2. International Institute for Environment and Development, London, UK.

Alston, J. M., G. W. Norton, and P. G. Pardey. 1995. Science under scarcity: principles and practice for agricultural research evaluation and priority setting. Cornell University Press, Ithaca, New York, USA.

Alston, J. M., C. Chan-Kang, M. C. Marra, P. G. Pardy, and T. J. Wyatt. 2000. A meta-analysis of rates of return to agricultural R&D: ex pede herculem? International Food Policy Research Institute Research Report Number 113. IFPRI, Washington, D.C., USA.

Anderson, J., R. W. Herdt, and G. M. Scobie. 1998. Science and food: the CGIAR and its partners. World Bank, Washington, D.C., USA.

Chambers, R., and G. Conway. 1992. Sustainable rural livelihoods: practical concepts for the 21st century. IDS Discussion Paper Number 296. Institute for Development Studies, Brighton, UK.

Collinson, M. P., and E. Tollens. 1994. The impact of the International Agricultural Centers: measurement, quantification, and interpretation. Issues in Agriculture, Number 6. Consultative Group on International Agricultural Research, Washington, D.C., USA.

Consultative Group on International Agricultural Research. 2000. Integrated natural resources management research in the CGIAR: a brief report on the INRM workshop held in Penang, Malaysia, 21-25 August 2000. Consultative Group on International Agricultural Research, Washington, D.C., USA.

Goldsmith, A. A. 1993. Institutional development in national agricultural research: issues for impact assessment. Public Administration and Development 13:195-204.

Guijt, I. 1998. Participatory monitoring and impact assessment of sustainable agricultural initiatives: an introduction to key elements. SARL Discussion Paper Number 1. International Institute for Environment and Development, London, UK.

Horton, D., P. Ballantyne, W. Peterson, B. Uribe, D. Gapasin, and K. Sheridan, editors. 1993. Monitoring and evaluating agricultural research. CAB International, Wallingford, UK.

Izac, A. M. 1998. Assessing the impact of research in natural resources management: synthesis of an international workshop, 27-29 April, International Centre for Research in Agroforestry (ICRAF), Nairobi, Kenya. International Centre for Research in Agroforestry, Nairobi, Kenya.

Izac, A. M., and P. A. Sanchez. 1998. Towards a natural resource management paradigm for international agriculture: example of agroforestry research. International Centre for Research in Agroforestry, Nairobi, Kenya.

Lee, D. R., K. Steven, and N. Uphoff, editors. 1991. Assessing the impact of international agricultural research for sustainable development; proceedings from the CIIFAD Symposium at Cornell University, Ithaca, N.Y. Cornell University Press, Ithaca, New York, USA.

Sartorius, R. 1996. The third generation logical framework approach: dynamic management for agricultural research projects. European Journal of Agricultural Education and Extension 2:49-62.

Scherr, S. J. 2000. Hillsides research in the CGIAR: towards an impact assessment. Consultative Group for International Agricultural Research, Washington, D.C., USA.

Scoones, I. 1998. Sustainable rural livelihoods: a framework for analysis. Institute for Development Studies, Brighton, UK.

Smutylo, T. 1998. Rethinking impact measurement: IDRC's experience. International Development Research Centre, Ottawa, Ontario, Canada.

Technical Advisory Committee of the Consultative Group for International Agricultural Research. 1999. Manual on logframes within the CGIAR system. Technical Advisory Committee of the Consultative Group for International Agricultural Research, Rome, Italy.

van Dusseldorp, D., and L. Box. 1993. Local and scientific knowledge: developing a dialogue. Pages 20-27 in W. de Boef, K. Amanor, K. Welland, and A. Bebbington, editors. Cultivating knowledge: genetic diversity, farmer experimentation and crop research. Intermediate Technology, London, UK.