Ecology and Society
Home | Archives | About | Login | Submissions | Subscribe | Contact | Search
 E&S Home > Vol. 15, No. 3 > Art. 13

Copyright © 2010 by the author(s). Published here under license by The Resilience Alliance.
Go to the pdf version of this article

The following is the established format for referencing this article:
Vreugdenhil, H., J. Slinger, W. Thissen, and P. Ker Rault. 2010. Pilot projects in water management . Ecology and Society 15(3): 13. [online] URL:


Pilot Projects in Water Management

Heleen Vreugdenhil 1, Jill Slinger 1, Wil Thissen 1 and Philippe Ker Rault 1

1Delft University of Technology, Faculty of Technology, Policy and Management, Policy Analysis Section


Pilot projects appear in many forms in policy making and management. In an effort to understand the nature and use of pilot projects and improve their effectiveness, we undertake a practice-based and theoretical study of the pilot project phenomenon. First, we examine the roles assigned to pilot projects in the policy development literature and explore their use in a Dutch water innovation platform. Second, we determine characteristics of pilot projects to deepen insights into the nature of the pilot project phenomenon and the dimensions useful in the design of pilot projects. Third, we identify three pilot types and nine ways to use a pilot project and we develop a Pilot Project Nonagon that can be used to assess pilot projects’ uses and to compare stakeholders’ perspectives on these uses. Fourth, we identify hurdles to diffusion of the knowledge developed from pilot projects and suggest strategies to overcome these. Lastly, we formulate a research agenda aimed at addressing the identified knowledge gaps.

Key words: Multi-Actor Context; Pilot Project Nonagon; Pilot Project Characteristics; Pilot Projects; Pilot Project Uses; Policy Development; Water Management; WINN


Water managers regularly use pilot projects as instruments for testing innovations and implementing policies on a restricted scale. Pilot projects are the means of applying new approaches in a confined field setting to learn about the innovation–context interaction and to use these lessons for improving the innovation or adjusting management practices and policies. According to Greenberg and Shroder (2004), the use of pilot projects is growing. Indeed, it is not uncommon for studies on complex societal problems to recommend pilot projects before policy development proceeds further, nor for large research projects to include pilot projects in their initial proposals, such as with the EU-Interreg IIIb Freude am Fluss project. Historically, the formal use of pilot projects as a policy instrument was stimulated by the demand of the US congress in the 1960s to develop evidence for legislation (Cabinet Office 2003) and to justify policies (Martin and Sanderson 1999). Today, they are widely considered to provide a means of dealing with the complexity of social–ecological systems and their associated uncertainties, the dynamics and new challenges deriving from climate change and ongoing and increasing pressure on natural resources (Cosgrove and Rijsberman 2000, Olsson et al. 2004, Dehnhardt and Petschow 2008, Frantzeskaki et al. 2008).

Despite the positive attributes and the high expectations that pilot projects will contribute to learning and policy change, evaluations of pilot projects have been more critical. According to De Groen et al. (2004), outcomes are no more than learning from failure and pilot projects are therefore considered as ineffective instruments for policy innovation. Sanderson (2002) is even more critical when finding that no policy learning took place in the pilot projects he studied. Instead, the pilot projects were undertaken as a diversion or served as demonstration projects. Sanderson (2002) thus concludes that the pilot project label can be abused and that its legitimacy is questionable.

The differences between the expectations and outcomes of pilot projects show that pilot projects and their functioning are complex in nature, subject to uncertainty and influenced by actor behavior. Additionally, pilot projects are used for different purposes (Raven et al. 2008, Huitema et al. 2009) and so the meaning of the term can be contested. What one actor deems core to pilot projects, others might consider less important, for example, feedback to policy versus implementation of the pilot.

Despite their prevalence and the lack of coherence and even criticism of their functioning and use, pilot projects themselves have not been the subject of much study, particularly in the natural resources field. In this paper, we address this inadequacy by characterizing the pilot project phenomenon, its uses and contributions to policy development. We take as a starting point for the analysis that there is no agreed common definition or conventional meaning of the term pilot project. Instead, we analyze projects that function in the spirit of experimentation in a field setting, whether they are termed pilot project or not. It is our intention that through the development of a conceptual framework, practitioners, scientists and policy makers who initiate, study, or are involved in pilot projects can analyze the nature and use of pilot projects. They can then understand the extent to which they are probing the future and develop strategies to enhance the influence of the knowledge derived from pilot projects on management practice.

In the paper we perform an exploratory analysis of the pilot project concept drawing upon both theoretical and empirical material. The literature consulted deals with policy change and decision-making, while the empirical material includes interviews, web-published pilot project descriptions of the WaterInnovatiebron (WINN, and observations from WINN workshop attendance. WINN is a platform of the Dutch Ministry of Transport, Public Works and Water Management that conducts pilot projects as part of a broad water management innovation strategy. First, on the basis of a literature review, we identify three views on the roles of pilot projects in policy development. Second, we characterize pilot projects and contrast them with laboratory experiments and routine water management projects. Third, we identify three pilot types and nine different uses of pilot projects for which we develop a Pilot Project Nonagon. The Pilot Project Nonagon functions as a tool to assess perspectives on pilot projects and the developments therein over time. Fourth, we discuss hurdles to the diffusion of pilot projects and strategies to enhance the effectiveness of pilot projects in water management. We conclude the paper by proposing a research agenda aimed at further deepening understanding of the pilot project phenomenon and its relationship to policy development.


The data used in this study derive from four sources, namely: an extensive review of socio-political and social–ecological literature, ten semi-structured interviews with experts in water management and pilot project applications in the Netherlands and North-West Europe, an analysis of pilot project descriptions available on the WINN website in October 2008[†], and attendance at a WINN review workshop held in December 2008.

The public administration, governance and policy change literature reviewed focused on the views and expectations of pilot projects and their role in policy development, decision making and problem solving (e.g. Lee 1993, Pawson and Tilley 1997, Hoogerwerf 1998, Bovens et al. 2001, Rotmans et al. 2001). The literature on quasi-experiments, innovation, pilot projects and evaluation included Campbell (1975), Rogers (1995), Raven et al. (2008), Hoogma et al. (2002) and Sanderson (2002). Additionally, literature on water management practice was consulted extensively.

Interviewees were selected on the basis of their long experience in water management and pilot projects in the Netherlands and North-West Europe. The interviews with past and present representatives of various departments of the Dutch Ministry of Transport, Public Works and Water Management, the Ministry of Agriculture, Nature and Food Quality, the World Wildlife Fund, Wageningen University and the International Commission for the Protection of the Rhine were conducted by the first author in 2007. The interviews were used to reflect upon developments in water management and policy in the Netherlands and North-West Europe over the past decades and the role of pilot projects therein. As such, the contextual aspect of pilot projects, including the relationship between pilots and policy development and specific pilot mechanisms, such as learning and the role of key individuals in policy change, could be better understood.

The WINN website contained a presentation of a total of 27 pilot projects conducted within the innovation platform in the period 2002 to 2007. The authors selected 16 of these as input for this study. The projects with minimal information and the laboratory experiments, that is, projects in non-field settings aimed at learning about the innovation itself, were excluded. The types of projects range from coastal eco-engineering (bio-bouwers) to dealing with the re-establishment of estuarine dynamics (van zoutbestrijden naar zoutbegeleiden) and the use of digital tools in multi-stakeholder design processes (digitale ontwerptafel). The quality of the data is highly variable; some projects include extensive descriptions of activities and measures, illustrations, methods and results, whereas others refer to related project pages or are limited to a one paragraph project goal description.

Observations from the WINN review workshop held in December 2008 at which approximately 50 WINN participants were present, provided additional data. In this workshop, the three projects INSIDE, Rijke Dijk/Biobouwers, and Zandmotor were discussed extensively.

In exploring the pilot project phenomenon and developing a conceptual framework, we carry out four steps, namely: (1) analyzing the role of pilot projects in policy development, (2) characterizing pilot projects, (3) identifying the uses of pilot projects and (4) determining hurdles to the effectiveness of pilot projects and some strategies to overcome these. For each of these steps, a careful analytical process in which theoretical and empirical data sources were combined to generate grounded insights on pilot projects in water management was followed, as described and schematized in detail in Appendix 1. The analysis of the qualitative data is based on the logic of structuring and categorizing as explained in Miles and Huberman (1994) and Braun and Clarke (2006). This analysis enabled an in-depth assessment of the WINN pilot projects and the generation of a database as part of this study of the nature of pilot projects and the uses to which pilot projects are put within Dutch water management. We develop a Pilot Project Nonagon in which, instead of merely presence or absence, the extent to which a pilot project adheres to a certain use can be visualized. We illustrate an initial application of the Pilot Project Nonagon using two projects within WINN.

Finally, we reflect upon the study and identify knowledge gaps, issues that were not addressed and new questions that arise from the analysis and propose a research agenda on pilot projects.


Within public administration in general and water management in particular, different perspectives exist about policy development processes and decision making. There are many models describing these processes and thereby implicitly or explicitly assigning a role to pilot projects. In this section, we distinguish three categories of policy development models in public administration on the basis of the view they hold of policy development and discuss the functions and expectations associated with pilot projects. First, we follow established practice in differentiating between Analytical Models and Political Models (Bovens et al. 2001). Next, we identify another category of models that we define as Holistic Models. An overview of the three categories and the accompanying models is provided in Figure 1. The different categories of models are not necessarily exclusive, but can be used alongside and in addition to each other.

The Analytical Models prescribe distinguishable, often rational, activities to resolve problems that are undertaken one after another or contemporaneously (e.g. Simon 1977, Miser and Quade 1985, Hoogerwerf 1998). The underlying paradigm is that of informed decision making. Decisions are made based on analyses and evaluated in terms of effectiveness. Examples of Analytical Models are the Phase Model (Simon 1977) and Parallel Model (Geldof 2005).

The Political Models describe policy making as a complex and seemingly chaotic and incremental process, providing discussion rounds, for instance, in which agendas are formulated and windows of opportunity can arise (e.g. Wildavsky 1979, Kingdon 1984/1995, Lindblom 1993, Teisman 1995). The underlying paradigm in the Political Models is that of opportunism and capricious chance. The political discussion continues during implementation of policies and results are politicized. The focus of the models is on the political and strategic levels where the debates take place. The biophysical system itself is less directly of interest. Policy development is considered a process in which many actors are interrelated and systems are interconnected. The models stress the differences, conflicts and competition among actors, but also the emergence of opportunities to cooperate (De Bruijn and ten Heuvelhof 2008). Examples of the Political Models are the Rounds Models (Teisman 1995) and Streams Models (Kingdon 1995).

The Holistic Models view the policy development process within a broader societal context of interacting institutional, societal and biophysical systems and recognize systemic uncertainties. Their view is holistic when describing large societal changes, but becomes pragmatic when seeking the means of inducing change at the micro-level through pilot projects. The Holistic Models attempt to deal with criticism of the limited impact of policy evaluation on policy making (Bennett and Howlett 1992, Argyris and Schön 1996, Birckmayer and Weiss 2000, Schwandt 2003) and the increased recognition of the complexity of social–ecological systems (Carpenter et al. 2001, Scheffer et al. 2001) by broadening and further institutionalizing the use of early evaluation throughout the entire policy development process and so rationalizing the policy making process (Sanderson 2002). Examples of the Holistic Models are Adaptive Management (Holling 1978, Lee 1993, Gunderson 1999, Pahl-Wostl 2008), Adaptive Co-management (Olsson et al. 2004), Transition Management (Rotmans et al. 2001, Van de Poel 2003) and Integrated Water Management (Mitchell 1990).

Contrasting the policy development models

In addition to general differences in their perspectives on the policy process, the three categories of policy models differ in the explicit presence and roles they ascribe to pilot projects. In this section, we contrast the three model categories on the basis of their views on pilot projects, the inclusion of the complexity of the social–ecological systems, and the governance style (the process of exercising authority) they consider appropriate, whereby governance styles range from authoritative, that is, a closed process, to consultative and cooperative and finally to facilitative (Ker Rault 2008). An overview of this discussion is provided in the last two columns of Figure 1.

The primary purposes of pilot projects in the Analytical Models are to test and apply technological innovations in particular contexts and to mitigate well-defined issues in the biophysical system. Pilot projects are used to streamline and to collect resources, such as money, knowledge and the commitment of other actors. The knowledge developed in the evaluation stage ought to flow back into the problem solving and decision making process. Learning, gaining experience and dealing with biophysical risks are considered major contributions of pilot projects to the problem solving process, but the focus is on testing a particular innovation, often an artefact or technology. The users of such pilot projects are often single actors, such as experts (Van den Bosch and Rotmans 2008). An authoritative governance style is common because interdependencies of systems, interactions of actors and uncertainties are not recognized or are excluded from the process. Indeed, the competent authority defines the problem and makes decisions in relative isolation.

In the Political Models, pilot projects are used by policy entrepreneurs to influence actor-networks. Here, actor-networks are explicitly present and are considered to be at the heart of policy development (Bovens et al. 2001). For instance, actors try to steer the policy development process by attracting or distracting attention. Pilot projects are recognized as a strategy to deal with risks and uncertainties, mainly those deriving from the societal system, such as actor behavior. Another rationale for conducting pilot projects in this view is to test policies on a controlled scale to prevent larger financial and personal failures (Cabinet Office 2003). Adversely, pilots can be used or abused to postpone policy decisions or as demonstration projects(Sanderson 2002). These pilots lack learning and feedback to policy. Sanderson views this as a loss of the meaning of pilot projects, but we consider this a strategic use/abuse of pilot projects. The contributions of pilot projects are reflected in enhanced actor-interactions, changed actor perspectives, increased involvement of actors and policy adjustments. The governance style is usually cooperative, meaning that problem definitions and solutions are defined among the competent authority and other stakeholders, but power over the formal decision is not distributed. More authoritative styles can also be exercised, but given the existing interdependencies these can lead to conflict.

In contrast to the Political and Analytical Models in which the use of pilot projects remains implicit, the Holistic Models explicitly assign a role to pilot projects. Pilot projects are primarily meant for learning-by-doing and reflection on practice or policy. Based on the pilot projects, practices and policies can be adjusted to changing circumstances. As such, pilot projects are considered a means of dealing with the uncertainties inherent to complex social-ecological systems. The pilots enable the incorporation of research findings, societal actors’ interests and policy practitioners’ ideas in the early stages of the policy cycle (Dietz et al. 2003, Olsson et al. 2004, Carlsson and Berkes 2005, Pahl-Wostl 2007). Furthermore, pilot projects are used for exploring options within local contemporary contexts, evaluating hypotheses, and as instruments to induce long-term societal changes, particularly when the effects of multiple pilots accumulate (Gunderson et al. 1995, Pahl-Wostl 2006, Raven 2007, Van Sandick and Weterings 2008). In their most far-reaching form, management and policy making can themselves be considered as ongoing social–ecological experiments (Walters 1986, Campbell and Russo 1999). In these models, a broad range of actors deriving from different policy- and societal levels is involved in the pilot projects and a multitude of mutual relationships and interdependencies arises. Consequently, the direct influence of the competent authority is limited and its role becomes cooperative and facilitative (Vreugdenhil and Ker Rault 2009). The competent authority sets the boundaries within which actors share decision making and implementation responsibilities.

In summary, three categories of policy development models have been distinguished on the basis of their views on the policy development process and their expectations of pilot projects. This has led to the identification of a broad range of uses. Before continuing to determine the uses and how they contribute to policy making in practice, we must first develop a deeper understanding of the nature of pilot projects and their characteristics.


In this section we characterize pilot projects in general and provide a framework for assessing the characteristics of specific pilot projects. We identify diverse project descriptors and determine the range in which pilot projects fall according to these descriptors. As such, we identify the characteristics of the pilot project concept and characterize the concept as a whole. In doing so, we develop a framework that supports the identification of the nature of a specific pilot. The primary input for this analysis derives from the literature survey on pilot projects and their social–ecological contexts. Insights are further validated and refined using insights from: (1) ten interviews with water managers and (2) sixteen WINN pilot projects. To clarify the pilot project phenomenon further, we contrast their characteristics with those of archetypical laboratory experiments and routine water management projects on the basis of the project descriptors. By laboratory experiment, we mean experiments undertaken in non-field settings that are intended to test hypotheses. By routine, we mean conventional projects and operational or daily management.

The six project descriptors to identify pilot characteristics are: (1) Relationship to policy and local context, (2) Scale, (3) Innovation, (4) Knowledge Orientation, (5) Special Status, and (6) Actor Network. In Table 1, we indicate how pilot projects, laboratory projects and routine water management projects differ on the basis of these descriptors. It is not our intention to claim that these descriptors provide a comprehensive description of all pilot projects in water management. Instead, they represent a grounded characterization of the pilot projects we encountered in our research.

Relationship to policy and local context

Projects are embedded within a particular biophysical, societal and institutional context. This embedding in policy and local context can be described in terms of connection to policy, local contextual dependence and incidence of occurrence.

First, projects are not independent entities conducted in policy isolation, but are connected to existing policies and projects in some way. This connection can be in the policy periphery or at the core of policies. Pilot projects are shaped by, and exert influence on, water management processes, yet can be conducted as part of a policy program or replace a planned management project and so be at the core of policy, or can be conducted in the policy periphery. In the last situation more freedom exists than in the other situations, because the direct links to policies are weaker. Van Eeten and Roe (2002) refer to this as de-coupling. The WINN projects, because they are conducted under the WINN umbrella and are selected as potentially valuable, have a specific connection to the Dutch Ministry of Transport, Public Works and Water Management, and yet can operate with more freedom from existing policies. Laboratory experiments are in the policy periphery where there is more freedom to experiment and do not necessarily have to account for policy implications. In contrast, in routine projects the link is strong; they are conducted as a result of existing policy and do not deviate from this policy.

Second, in a pilot project, the interaction with the local context and so the local contextual dependency is an instrinsic part of the pilot process. Biophysical and societal contextual factors interact with the pilot project; design and development are influenced by involved actors who have local knowledge and who in turn may be influenced by the pilot process, that is, they may learn. More strongly, contextual factors can only be controlled to a limited extent (Lee 1999). This interaction provides the basis for the development of context dependent knowledge (Flyvbjerg 2006). In contrast, a core characteristic of laboratory experiments is that contextual factors are controlled (Lee 1999). Accordingly, they are repeatable. Routine projects are moderately intertwined with the context. They are undertaken in a biophysical and societal context, but are often subject to standardization in order to eliminate contextual uncertainty.

Third, the incidence of occurrence of pilot projects signifies that they can be part of a series or implemented as a single event. Since the same applies to laboratory experiments and routine water management projects, pilot projects cannot be distinguished in this respect.


The descriptor of scale limitations enables projects to be characterized on the basis of whether they are confined in scale or applied at full scale. Scale dimensions include time and space (Douthwaite et al. 2003, Sendzimir et al. 2007, Karstens 2009) and problem scope. Pilot projects are confined in at least one of these dimensions. Indicators of confined scales include budgetary constraints; limited timelines, for example 0.5 to 2 years; local implementation, for example, a single dike or a floodplain; and a limited number of issues and consequently actors involved. Confining the scale of a pilot project acts to prevent large flaws and is a means of dealing with risk and uncertainty (Cabinet Office 2003). Laboratory experiments are confined in all dimensions, while routine projects are applied at full scale.

A second descriptor of scale is reversibility. Reversibility describes the ability to return to the reference situation following the implementation of the project. This is an effect-oriented descriptor rather than a design descriptor. Although scale and reversibility do not have a one-to-one correspondence, that is to say that the type of innovation is also of importance, the degree of reversibility is primarily influenced by the scale of application. Therefore we include it as a descriptor of scale. For instance, a short project duration provides a means of discontinuing the project after that period. Reversibility of pilot projects, however, is limited to biophysical aspects and formal institutions, but is not achievable for softer aspects such as relations among involved actors and acquired experience. In the softer aspects, a pilot project does not differ from any other project. However, routine projects are meant to be permanent, while laboratory experiments are not applied in real-world settings and so the reversibility to the reference situation is not relevant.


Testing of an innovation or stimulating innovation in general are the reasons often given for conducting pilot projects in the first place. However, innovation is a relative notion, depending on what is known to actors in particular areas at a certain period of time. For instance, within WINN, a number of projects explicitly state that the technology had already been implemented in another context, such as for example, the pilot “ecobeach” in Denmark, but that it is innovative for the circumstances that exist along the Dutch coast.

The types of innovation applied in a water management pilot project can include technological, conceptual, and institutional or process innovations, such as different public participation practices. In laboratory experiments, the type of innovation is generally restricted to technological innovations.

Innovation development, and thus pilot projects, can be demand-driven or supply-driven. Demand-driven pilot projects aim to address operational concerns in a more effective and efficient way, while supply-driven pilot projects aim to address the long-term, potential envisaged needs. Laboratory experiments are supply-driven.

The level of innovation of a pilot project can vary, ranging from radical to incremental (Henderson and Clark 1990). Sendzimir et al. (2007) indicate that compromise building for ecological, economic and social sectors limits the level of innovation of pilots. In routine projects, the level of innovation is low since proven approaches are used; types and drivers of innovation are therefore not relevant. In laboratory experiments, in contrast, the level of innovation is high since the constraints of contextual dependency are absent.

Knowledge orientation

Knowledge orientation refers to the way in which a project is designed for knowledge creation and learning. In general, two stances on knowledge design can be identified. These include: (1) an expert-driven model and (2) a communicative model.

In the expert-driven model, experts or their clients define the problem, provide knowledge and prepare solutions. Monitoring focuses primarily on biophysical or technological impacts. The types of knowledge created are of a hard, substantive and relatively general nature (Dosi 1988, Flyvbjerg 2006) such as knowledge of the innovation itself, routine procedures and measuring methodologies. The type of learning taking place is of a single-loop nature, meaning that the actors modify their actions according to the difference between expected and obtained outcomes (Argyris and Schon 1996, Raven et al. 2008). In the communicative model, stakeholder needs and learning from each other hold a central position. Local stakeholder knowledge is of importance, while expert knowledge is merely supportive. Social learning is used as a central mechanism to encourage participation and foster learning. A social learning process can lead to the development of a common understanding of the system or problem at hand, agreement and collective actions through communication and the interaction of different actors in a participatory setting (Muro and Jeffrey 2008). The types of knowledge that are created are of a process, soft and contextual nature (Dosi 1988, Flyvbjerg 2006), such as interactions and dependencies among actors and the interaction between actors and the innovation. The type of learning can be characterized as double-loop learning. In double-loop learning, actors question and modify the values, assumptions and policies that led to the actions (Argyris and Schön 1996, Raven et al. 2008). It is therefore learning about single-loop learning. Additionally, a broad range of actors gains experience (Kolb 1984, Dreyfus and Dreyfus 1988, Flyvbjerg 2006). Accordingly, the range of actor learning and the extent of learning are broader. Initiators can facilitate learning in both models; for example, in WINN “master classes” are offered to participants. However, how much is eventually learnt depends on the individuals involved and their personalities.

For laboratory experiments, the first knowledge model is most common, whereas in pilot projects both models can exist. The communicative model with learning through social processes is, however, increasingly advocated from an Adaptive Management and Transition Management point of view (e.g. Pahl-Wostl 2006, Raven et al. 2008). For routine projects, the expert model is most common in the sense that experts shape the project. However, a focus on learning is lacking, because well-known methods and technologies are used.

The extent to which knowledge creation is considered important is reflected in the monitoring intensity and the type of knowledge and subsequent reporting and communication of knowledge. In laboratory experiments, controlled observation in a controlled context is exercised (Lee 1999). In routine projects, little or no monitoring takes place since there is little potential for new knowledge. The monitoring intensity in pilot projects is diverse, despite the claimed focus on knowledge creation and learning (e.g. Pawson and Tilley 1997, Raven 2005, Pahl-Wostl 2006). The type of monitoring is systemic, aimed at detecting surprises (Lee 1999) and is generally of a before-after nature. The contextual dependency in water management makes every project unique to some extent. Randomized control trials, a technique commonly applied in medical studies, explicitly ignore contextual dependency and are therefore not suitable.

Special status

The special status descriptor is a strong distinguishing characteristic of a pilot project. The special status is reflected in attitudes towards the project, its flexibility and the level of resource allocation.

First, the attitude towards pilot projects differs from the attitude towards routine projects, because people have different expectations of pilot projects (Geels and Raven 2006). Accordingly, behavior changes. Pilot projects are associated with innovation and one can identify a learning attitude and a tolerance toward what under non-pilot conditions would be considered failure. Instead, all pilot outcomes provide input for learning. This is manifested as a spirit of experimentation (Weiss 1977). As a result of these expectations, outsiders are attracted and commitment is enhanced. For example, the risk for a person in a high position, such as a minister, is perceived as smaller and they can therefore risk committing themselves. As a result, new, previously non-existent, cooperation between actors takes place. Additionally, there is space for creativity, because there is more policy freedom, in contrast to routine projects where a conformist attitude is exercised. It should be noted that for some people the status of pilot project means that the project does not have to be taken seriously and that it is a way to not have to meet policy requirements. In WINN however, the view of pilot projects was positive. They were associated with the chance to leave existing paths, collect resources, experience personal development and reduce personal responsibilities, among others. Like pilot projects, laboratory experiments enjoy a special status although this is of a different nature. The allowance for failure is high, but the level of actor participation is limited. Interaction with the context is not desirable in a laboratory experiment. The goal is knowledge about the innovation itself, not about the context–innovation interaction.

Flexibility is a second aspect of the special status. Flexibility means the freedom not to have to follow standard procedures. In routine projects, both creativity and flexibility are constrained by having to follow strict rules and guidelines and the need to meet policy goals. In pilot projects, there is more autonomy and it is possible to tailor applications to the biophysical and societal context and so to accommodate dynamics. Adjustments can be made in the execution of the project during the process as well as in the analysis methods, the objects analyzed and the interpretation of the findings from the pilot project. The use of this freedom can itself be a cause of conflict because of differences in the expectations and interpretations of different actors (Van Lente 1993, Geels and Raven 2006). In laboratory experiments, flexibility is high in the design phase but not during the experiment itself, because answers to specific research questions are then sought.

Lastly, the special status is reflected in the level of resource allocation. Enhanced allocation of resources occurs because actors wish to be at the forefront of innovation, make societal contributions or try to influence the course of the pilot and related policies. By participating, actors bring their usual resources to the project, but might also be prepared to invest additional resources. Accordingly, resources become available that under routine circumstances would not be available. Within WINN, the pilot projects received support from the ministry in the form of people, access to information and knowledge, and through the provision of study sites. The resource allocation to laboratory experiments is variable, because it depends on the extent to which the experiment is understood to be potentially valuable. In routine projects, the resource allocation, and particularly the sources thereof, are fixed.

Actor network

The initiators, participants and governance styles employed are of interest in characterizing projects by their actor-network. Initiators of laboratory experiments and routine projects are generally restricted to a specific single actor, such as researchers and government agencies. In pilot projects, both the type and number involved in the initiation can be more diverse. Initiators can be governmental agencies (Hoogma et al 2002), stakeholder groups, including citizens, research institutes, companies, or temporary alliances among these (Brown and Vergragt 2008). In addition to the initiators, other actors may participate in pilots dealing with integrated water management, leading to a multi-actor alliance. These participants can include the same type of actors as the initiators, but in a different role, for example as users instead of developers.

In terms of governance styles, pilots have the ability to foster more facilitative, open styles because responsibilities are less pressing and “failure” is more tolerated. Moreover, the exploration of more open styles of governance could be a goal in itself, so as to enhance creativity, democracy and social learning. For instance, this took place in the WINN pilot, “Combi-kering”, where urban development and coastal protection were combined through co-production with citizens. However, this does not imply that a more open governance style is exercised in every pilot, nor that it should be. A consequence of such a style is a strong increase in the involved network of actors, in turn requiring appropriate process management skills. This is in contrast to routine projects in which specialized staff members from competent authorities generally work on the issues and the actor network is often more limited. In laboratory experiments, governance styles are not applicable, because of the single-actor setting and the distance from operational practice.


Now that we understand how pilot projects can be characterized, we can move on to examine how pilot projects are used in practice. In our comparative analysis of policy development models, we established that although pilot projects are nominally used for testing innovations in a real-world setting, their actual uses are highly diverse. In this section, we seek to develop a coherent overview of pilot project uses within the policy development process.

We follow Huitema et al. (2009) in first distinguishing two major types of pilot projects, the Research Pilot and the Management Pilot. Additionally, based on the political view of policy making (Figure 1), we identify a third type, the Political-Entrepreneurial Pilot. We subsequently divide the three types into nine uses (Table 2). These uses derive from an extensive literature review, supported by interviews with water managers. To depict the different uses we develop the “Pilot Project Nonagon” and then illustrate its application using examples from WINN.

1. Research Pilot projects focus on knowledge development. They aim to fill knowledge gaps identified using other research techniques (Walters 1997). Knowledge development can occur both through exploration and evaluation. The Research Pilot projects are associated with the Analytical Models and Holistic Models of policy development. Pilot projects are considered to supply rational decision making with knowledge (Simon 1977, Miser and Quade 1985) and provide learning platforms to develop ongoing insights and so deal with social–ecological dynamics (Pahl-Wostl 2006).

Research Pilot projects can be used for explorative and evaluative purposes. Explorative pilot projects are used to test and refine innovations in their context and to gain experience. Evaluative pilot projects are used to evaluate policies that are already in development at an early stage. This means that policies are first implemented at a confined scale and are evaluated before the full policy is rolled out (Cabinet Office 2003). Results are used to inform policy making and refine the policy (Weiss 1975). In contrast, the explorative pilot project is usually undertaken at an earlier stage of the innovation development trajectory and lies in the policy periphery rather than having a strong link with policy. This implies that in the explorative pilot, the level of innovation can be higher and that researchers are the initiators instead of policy makers.

2. Management Pilot projects are used for communication, problem mitigation, policy implementation and as insurance. A communication pilot is used to initiate communication among actors on the specific topic addressed by the pilot. As such, social learning processes are initiated and new channels of communication can open up. Additionally, the pilot is used as an open conduit for spreading existing knowledge. By this type of communication, we do not mean convincing or converting others to one’s own point of view. This will be referred to as advocative.

Pilots used for problem mitigation are applied to mitigate an existing practical, often biophysical problem, such as for example, excessive vegetation along a Dutch river leading to an increased risk of flooding. Standard tools to resolve the issue are lacking. In such a pilot, innovative solutions can be sought and found (Huitema et al. 2009). The driving motivation for this use of a pilot project lies in the problem at the particular site itself, rather than with an innovation that requires an application site.

Pilots can be used for policy implementation. Operational managers are the people who ultimately put policies into practice. However, the translation from policy to implementation is often considered troublesome. The use of a pilot project can increase acceptance, by creating favorable conditions and delaying sensitive trade-offs, and thus provide impetus for the implementation and enforcement of an existing policy. This differs from an evaluative use of a pilot project in that knowledge development is not central, rather, the pilot project is used pragmatically to put fully developed policy into practice.

Lastly, pilot projects are used as insurance against failure, as they enable risk minimization and facilitate dealing with uncertainties. Since pilot projects are undertaken on a confined scale, their impacts are small. As a result, perceptions of the associated risks change. Failure is now permitted, which enables actors holding responsible positions to participate. In addition, by confining the scale in any dimension of time, space or problem scope, boundaries are set for the studied system. The chance of unexpected biophysical and social developments are reduced.

3. Political-Entrepreneurial pilot projects are used to influence a policy process for personal or strategic reasons. This aspect of pilot project use is recognized and embraced in the political view of policy development. Uses include playing a political game, providing an incentive and carrying out advocacy.

Pilot projects are used as a political game when the real intentions of the initiators are disguised and they are trying to serve their own interests. The pilot is then used as: a diversionary tactic or to set the agenda, that is, to guide policy attention; to delay decision-making; save political face; as a symbolic gesture to give the impression that something has been done or to gain commitment from actors who would otherwise not be forthcoming. Additionally, the pilot itself can be considered an opportune way to commercially implement a project. Owing to extensive use of policy freedom there is no commitment to policy requirements while the pilot project label helps to gain support for the innovation.

Pilots can provide incentives to individuals, organizations or larger groups (Schneider and Ingram 1990). Conditions are created that allow individuals to be creative, profile themselves and learn. Their organizations can also learn and build experience and their preferences can carry the day, be implemented and their problems solved. From a commercial perspective, pilot projects offer an organization the possibility of developing their business by implementing prototype technologies in a practice setting. For larger groups, the use of pilot projects can provide a setting that stimulates innovation. The WINN program as a whole is an example of this, in that it created conditions within which pilot projects could be developed and in so doing stimulated innovation in general.

Finally, pilots are used for advocacy purposes. This means that the pilot project is specifically used to convince other actors of one’s own point of view or to lobby for specific solutions to envisaged future problems. The pilot project itself is used as an example to convince others of the excellence of the innovation and the conditions that are necessary for it to work well. An example is the zandmotor project (sand motor). Sand nourishment is undertaken at a distance from the coast for coastal flood defence purposes. The sand piles form islands that are eroded by the water over time and subsequently deposited on the beach, the ultimate purpose. One aspect of the pilot is to give these islands a societal function and to find financial resources. As a result, project developers are invited to participate. What is advocated to them in this project is not the core of the pilot, namely the concept of the sand motor, but the idea that properties can be developed that take the ephemeral nature of the islands into account. The act of convincing is considered necessary because project developers would otherwise not be interested in such small-scale, high risk, innovative projects when there are enough other places where conventional business can be done.

An analysis of the WINN database revealed that most projects were used for multiple purposes, instead of for a single use (Table 2). Not all potential uses were explicitly mentioned, such as the use of pilots as a political game or as insurance, and some were only alluded to, such as evaluative or policy implementation uses. The reasons for this include: (1) the implicit nature of the goal, which is difficult to explain, (2) it is assumed to be a general characteristic of a pilot project, such as insurance, and was therefore not considered worth mentioning, (3) it is at a meta-level, such as an incentive, and (4) it is a hidden intention, such as a political game. We have included these uses in our conceptual framework since they were mentioned during the interviews and can be found implicitly in the literature (e.g. Schneider and Ingram 1990, de Bruijn and ten Heuvelhof 2008). Since the use of a pilot is subject to interpretation, we expect that when asking different actors, different sets of uses will appear for one pilot. Actors have their own perspectives on the pilot, depending on their role, background and reasons to participate in the pilot. These reasons vary between actors, including initiators (Raven et al. 2008). Moreover, some uses will dominate over others. Therefore, it is more appropriate to speak of the degree to which a certain use is considered to be present in a pilot. Additionally, we note that all project descriptions officially claimed that the pilots were exploratory tools contributing to knowledge development. However, the extent and nature of this knowledge development was not always provided. It seems that claiming an exploratory use provides sufficient justification for naming a project a pilot project.

We developed the Pilot Project Nonagon (see Figures 2 and 3) to visualize the extent to which a pilot project is used for different purposes. The Nonagon can be applied in three ways. First, Nonagons can be applied in the analysis of pilot projects. By developing multiple Nonagons, the identification of differences in actors’ perspectives within one pilot and the changes in perspectives over time can be determined. Second, Nonagons can be used in communicating differences in perspectives and the associated expectations to pilot participants. Insights regarding differences in perspectives and the accompanying expectations held by pilot participants might prevent disappointments or avoid conflicts at later stages (Van Lente 1993). Third, from a research perspective, the Nonagon enables a standardized comparison of pilot projects. Two examples of the application of the Nonagon to projects within WINN are described below and depicted in Figures 2 and 3. The Pilot Project Nonagons depicted reflect the interpretation of the authors of the pilot project data available through the WINN project descriptions and at the WINN workshop.

Example 1: INSIDE (see Figure 2)
In the pilot project INSIDE, alternative techniques for strengthening dikes were tested. Traditionally, dike strengthening requires space, which is often not available in densely populated areas or conservation areas in the Netherlands. Three techniques that strengthen the dike from the inside were developed. One of these, entitled “mixed-in-place”, is already widely applied in other countries, such as Japan, while the other two are new. After initial technology testing in the period 2004-2007, pilot projects were started in the Netherlands. From the pilot projects, technological knowledge became available to the commercial developers. Despite promising technological results, diffusion to policy developers and implementers is still in the initial stages. We view the pilot project as highly explorative, as providing an incentive for commercial developers and as having a high insurance function. The insurance use is achieved as a result of the confined scale and the careful selection of non-vulnerable sites. The link with policies and operational problems is limited, but the technologies might contribute to national safety policies in the future and even strengthen weak dikes at present. Communication was mentioned as a function, but could be stronger, particularly with potential implementers. For the developers, the pilot has an advocative function, but for the pilot facilitators this is not an issue. The Pilot Project Nonagon in Figure 2 represents the viewpoint of the authors of this paper on the functional use of the INSIDE pilot project.

Example 2: “Combi-kering Den Helder”(combined water defence Den Helder) (see Figure 3)
The sea dike protecting the city of Den Helder from coastal flooding is not considered safe enough in the long term in view of anticipated climate change. Strengthening the dike requires space that is needed for multiple functions such as habitation, recreation and economic activities. Within WINN, possibilities for combining flood defence with a pleasant living environment were sought. Three technically and legally feasible designs were co-developed by citizens and engineers. The pilot project was initiated with the primary aim of enhancing communication between citizens, engineers and policy makers, rather than undertaking interventions per se. They jointly developed the problem formulation, learned about each others’ interests, and about how different functions influence each other. The Nonagon in Figure 3 reflects the viewpoint of the authors regarding the uses of the pilot project “Combi-kering”. The pilot project was triggered by a problem in the physical environment, but was used primarily in enhancing actor-network interactions, communication and as a platform for exploring and co-developing alternatives to dike strengthening. The pilot was used moderately to mitigate the perceived problem and as an incentive for the region to adapt to climate change. The pilot was developed in line with the general policy of adapting to climate change and so also moderately reflects policy implementation. The pilot was used minimally as a political game, as insurance and for advocacy purposes and was not used for evaluation.


Now that we have developed a conceptual framework to analyze the characteristics of pilot projects and how they are used, we move on to address their effectiveness in influencing the policy process. It should be noted that effectiveness is essentially a measure of the goals of a pilot and can therefore take many forms. Accordingly, we focus our discussion of effectiveness on the diffusion of new practice-based knowledge from pilots into standardized policy and management. In diffusion, the feedback of lessons learned from the pilot project to policy and management is central. The lessons include negative lessons that the innovation is not working, is too risky or too expensive. Based on the lessons from the pilot project, policy or management is adjusted and new projects and policies are initiated. Diffusion thereby entails both scaling up and dissemination processes (Vreugdenhil et al. 2009). By scaling up we mean that at least one of the scale dimensions is enlarged when institutionalizing or expanding the pilot project. By dissemination, we mean repetition of the pilot on a comparable scale.

Many authors (e.g. Bennett and Howlett 1992, Argyris and Schön 1996, Birckmayer and Weiss 2000, Sanderson 2002 and Schwandt 2003) and WINN project managers associate pilot projects with diffusion. This association is reflected in pilot project definitions such as: “something done or produced as an experiment or test before wider introduction” (Compact Oxford Dictionary 2008), a “seed for societal change” (Van Mierlo 2002) and a “stepping stone for societal change” (Van Sandinck and Weterings 2008). At the same time both within WINN and in the aforementioned literature, the impact of pilot projects on policy development is often considered limited. In this section, the hurdles to diffusion are presented and discussed, as found in the literature and as indicated by project managers in the WINN pilots. Additionally, we suggest some strategies to overcome or prevent hurdles from arising.

We identify five categories of hurdles. These are: (1) Limited Representativeness, (2) Limited Learning, (3) Lack of Institutionalization, (4) Poor Timing and (5) a Wait-and-See attitude.

Limited Representativeness implies that the design, conditions and results of pilot projects are of only limited applicability to new projects and so the usefulness of the pilot projects in new situations is subject to doubt (Martin and Sanderson 1999, Hoogma et al. 2002). In addition to the general issue of the contextual dependency of knowledge (Flyvbjerg 2006), reasons for the limited representativeness lie in the specific conditions and design of the pilot project. By using confined scales that reduce uncertainties and risks or by enhancing the availability of resources, the representativeness of pilot projects for standard projects is reduced. Scaling up from the pilot project conditions implies that the complexity of the problem increases (e.g. Mitleton-Kelly 2003, Collins and Ison 2006). Within WINN, the exposure to financial risks associated with moving from pilot projects to new projects was considered problematic by the project managers. Without a structure to address the question of responsibility for potential financial losses, risks were considered too large for future users, such as regional water boards, to be willing to take them. Strategies to reduce or cope with this hurdle include the provision of explanations about the contextual dependency of the knowledge. The particulars of the context help to determine what is transferable and what is not. Additionally, the inclusion of future users through open governance styles and co-financing arrangements increases their commitment. At the same time, the innovation can be tailor-made for further implementation, because users are involved in the design process.

Limited Learning implies that the different types of knowledge developed within the pilot are not transferred to future users. Reasons include the low quality and short duration of monitoring; problems with attributing change to specific factors (Martin and Sanderson 1999); a lack of comprehensive knowledge, that is, no single actor has an overview of all the options, mechanisms and impacts; and the limited impact of information on decision making in contrast to ideology and interest (Weiss 1980). Knowledge of a soft or tacit nature is particularly difficult to transfer. Pilot initiators indicated that they developed enthusiasm during the pilot project, but were not able to influence users positively afterwards. Social learning and its associated open governance style can function as a means to reduce or prevent such a hurdle from arising in the actor network (Muro and Jeffrey 2008). By stimulating social learning in the pilot project, more actors will understand the pilot and their dependencies upon other actors and the level to which they feel ownership over the pilot will increase. These effects in turn lead to more deliberate trade-offs in future decision-making, thereby using knowledge from the pilot. Additionally, knowledge spread and validation could be enhanced. WINN project participants were of the opinion that they could have benefited more from being within a program such as WINN by creating cross-pilot learning events. Validation through scientific and professional publications and by means of multiple pilots was considered to contribute to enhancing the status of the innovation, the familiarity among actors and thus, the willingness to adopt the innovation (see also Pawson and Tilley 1997, Raven 2007).

Lack of institutionalization implies that the innovation has not been internalized as an option on the list of options of a future user, such as a water board. Many users, particularly governmental bodies, are restricted to choosing from an approved list of options. If the innovation has not yet received an official stamp of approval, it may not represent a feasible option for such a user. Strategies to reduce this hurdle include the early identification of potential future users or application sites and of the nature of what ought to be diffused, such as an artefact or soft knowledge. These users can then become familiar with the pilot project and can indicate their requirements for institutionalization. Additionally, the development of multiple successful pilots means that the successful innovative practices have become common practice. For instance, the implementation of multiple pilot projects in the Netherlands that focused on combining flood defence and nature development contributed to the present standard practice of combining these previously isolated issues. Moreover, despite the idea of tolerating failure so as to encourage learning, successful implementation of an innovation is needed for diffusion. Failures are difficult to sell.

Poor Timing implies that by the time the pilot is finished, the policy climate no longer supports the adoption of the innovation, because the policy climate has changed during the course of the pilot (Liebowitz and Margolis 1995, Morris and Chiu 2001, Cabinet Office 2003). More generally, the market is lacking. This is also reflected in a lack of urgency to change existing practices. An example of poor timing is the WINN project, Artificial Reefs, that was meant for coastal protection. While the project was in development, an external event meant that the innovation was not picked up by the intended end-users, coastal cities. The Delta Committee, a national advisory body on coastal flood defence and water management in the Netherlands, published an advice in which artificial reefs were not recommended. Accordingly, the coastal cities opted for approved technologies. So, even though the pilot and the Delta Committee were independent, the advice of the Delta Committee influenced decision making regarding the artificial reefs. A second example is the WINN sand motor project in which a sense of urgency regarding the necessity for change was lacking. Project developers were not interested in developing adaptable constructions for ephemeral islands because there is sufficient traditional work elsewhere. A potential strategy to deal with this hurdle is to maintain flexibility in the pilot so that it can be adjusted to developments that may arise.

Wait-and-see attitude: In many of the pilot projects, a wait-and-see attitude to diffusion could be identified. Either diffusion was expected to occur by itself or strategies to enhance the diffusion of knowledge and learning were put in place after the pilot ended. Reasons included: (1) it is common practice to consider knowledge dissemination only after a project is concluded, (2) diffusion goals are not explicitly included in the pilot and so no budget is made available for achieving them, and (3) pilot projects are approached as routine projects that can be closed after pilot implementation and monitoring ends and participants return to their daily work. Accordingly, the overarching strategy related to this hurdle is a meta-strategy with two components, namely: (1) to include diffusion strategies within the pilot and (2) to put them in place at an early stage, because many of the strategies need time to become effective.


Using a mixture of theory and empirical material, we have developed a characterization of pilot projects in water management, identified different ways pilot projects are used and developed a framework to assess the perceived use of a pilot project at a particular time. However, many questions regarding the contribution of pilot projects to policy processes remain unanswered and new questions have emerged from this research. Consequently, we propose a research agenda for pilot projects (Table 3).

The first item on the agenda relates to the applicability of the findings of this paper. Although our empirical material derives from the water management field, our theoretical material has a broader base in the innovation, public administration, governance and natural resource management fields. We therefore anticipate that our findings on the characteristics and uses of pilot projects will have broad applicability within natural resource management. The tensions between policy continuity and innovation and the roles of pilot projects therein are not limited to water management. Testing and evaluating the applicability of the conceptual framework to other fields of natural resource management, such as forestry or marine management, could yield valuable insights in this regard.

Second, in this paper we did not explicitly relate pilot project characteristics to their use functions and diffusion. Future research, particularly through in-depth, empirical case studies, could focus on establishing such relationships. Relevant questions include: “Do pilot projects with particular characteristics result in particular types of diffusion?”, but also: “Do pilot projects with similar uses show similarities in their characteristics?” Pilot project initiators could benefit from understanding such relationships and could then design their pilots in such a way that the goals are more likely to be achieved. We expect, for instance, that pilots strongly embedded in policy programs are more likely to be implemented at full scale, but the freedom to experiment will probably be reduced and the learning may be limited. Extensive testing and application of the Pilot Project Nonagon in empirical studies also belongs on the research agenda as a means of establishing the validity of the conceptual framework.

Third, diffusion is adopted in this paper as a measure of the effectiveness of pilot projects. However, the effects of pilot projects are wider and include interventions in biophysical and institutional systems as well as learning. Huitema et al. (2009) recognized that a generic overview or evidence of the different contributions of pilot projects to public administration and management is lacking. We concur with this view and place a comprehensive study of the effects of pilot projects and particularly the diffusion of pilot projects into standardised policy on the research agenda for pilot projects.

Fourth, in this paper we focus on pilot projects that have been implemented. However, many pilot project initiatives do not achieve implementation, because no pilot site can be found or no partners can be identified who are interested in the pilot project, for instance. We found some evidence of this in our empirical material. Similarly, little attention has been given to the termination of pilot projects or the dying out of ideas. Future research on the dynamics of the process of pilot project development from conception to termination could throw light on these mechanisms and their underlying causes.


Pilot projects are used increasingly in policy making and management processes, certainly in the water domain. This is reflected both in theory, such as in the Holistic Policy development models, and in practice, as evidenced by their application within the Dutch innovation platform WINN. Despite their increased use, there is no common interpretation of the pilot project term. In this paper, we develop a conceptual framework for pilot projects. We characterize pilot projects in terms of six dimensions, namely: (1) Relationship to policy and local context, (2) Scale, (3) Innovation, (4) Knowledge Orientation, (5) Special Status, and (6) Actor Network. We also establish that although pilot projects are nominally used for testing innovations in a real-world setting, their actual uses are highly diverse. We distinguish between pilots used for: (1) research, (2) management and (3) political-entrepreneurial purposes. The three types of pilot projects are subsequently divided into the nine uses depicted in the Pilot Project Nonagon. The application of the Pilot Project Nonagon is further illustrated using examples drawn from the WINN pilot project database. We discover that the very lack of clarity associated with the pilot project phenomenon facilitates its wide use, but limits its effectiveness in diffusing new practice-based knowledge from pilot projects into standardized policy. We identify some hurdles to diffusion and propose a number of strategies to overcome these. The hurdles and accompanying strategies are by no means a complete set. Instead, they are intended to encourage discussion and research on the design and use of pilot projects. We propose a research agenda focusing on the links between the characteristics and uses of pilot projects and their effects on policy development and implementation.

We expect the findings of this paper to be of interest to those involved in natural resource management. In any field related to natural resources, a tension exists between continuity and change. Pilot projects address future issues and the questions and insights that derive from them can be used to revise ongoing policies. Understanding of this process will deepen understanding of the role of pilot projects in policy development.


Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a response, follow this link. To read responses already accepted, follow this link.


The authors wish to acknowledge the interviewees Tjalle de Haan, Jos Karssemeijer, Dick de Bruin, Henk Saeijs, Ben van de Wetering, Paul Berends, Jannemarie de Jonge, Wim Silva, Kirsten Schuyt and Pieter Huisman. Dave Huitema, Menno Bruggeling, Miriam Cuppen, the people from WINN, particularly Sonja Karstens, and three anonymous reviewers greatly contributed to this paper with their critical and constructive comments. The research was conducted within the Multi-Actor Systems research and Water Research Centre Delft of the Delft University of Technology.


Argyris, C., and D. Schön. 1996. Organizational Learning II: Theory, Method and Practice. Addison-Wesley, Reading, Massachusetts, USA.

Bennett, C. J., and M. Howlett. 1992. The lesson of learning: Reconciling theories of policy learning and policy change. Policy Sciences 25:275-294.

Birckmayer J. D., and C. Weiss. 2000. Theory-based evaluation in practice: What Do We Learn? Evaluation Review 24(4):407-431.

Bovens M., P. 't Hart, M. van Twist, and U. Rosenthal. 2001. Openbaar Bestuur: Beleid, Organisatie en Politiek. 6th. ed. Kluwer, Alphen aan den Rijn, The Netherlands.

Braun, V., and V. Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3:77-101.

Brown H. S., and P. Vergragt. 2008. Bounded socio-technical experiments as agents of systemic change: The case of a zero-energy residential building. Technological Forecasting & Social Change 75:107-130.

Cabinet Office. 2003. Trying It Out, The Role of ‘Pilots’ in Policy-Making: Report of a Review of Government Pilots. City University, London, UK.

Campbell, D. T. 1975. Reforms as Experiments. Pages 71-100 in E. L. Struening and M. Guttentag, editors. Handbook of Evaluation Research. Sage, Thousand Oaks, California, USA.

Campbell, D. T., and M. J. Russo. 1999. Social experimentation. Sage Classics 1. Sage, Thousand Oaks, California, USA.

Carlsson, L., and F. Berkes. 2005. Co-management: concepts and methodological implications. Journal of Environmental Management. 75:65-76.

Carpenter S., B. Walker, J. M. Anderies, and N. Abel. 2001. From Metaphor to Measurement: Resilience of What to What. Ecosystems 4:765-781.

Collins, K. and R. L. Ison. 2006. Dare we jump off Arnstein’s ladder? Social learning as a new policy paradigm. Proceedings PATH (Participatory Approaches in Science & Technology)Conference 4-7 June 2006, Edinburgh, Scotland.

Compact Oxford English Dictionary. 2008. 3rd revised edition. Oxford University Press, Oxford, UK.

Cosgrove, W. and R. Rijsberman. 2000. World Water Vision: Making Water Everybody's Business. World Water Council and Earthscan Publications, London, UK.

De Bruijn, H., and E. ten Heuvelhof. 2008. Management in Networks: On multi-actor decision-making. Routledge, London, UK.

De Groen J., A. Potze, B. de Jonge, and J. Rutjens. 2004. Innovatie van de overheid: Een Buitenkans. Positionering van de afdeling Strategie en Innovatie. NSOB, Den Haag, The Netherlands.

Dehnhardt A. and U. Petschow. 2008. Governance in River Basins - Introduction and Overview. In A. P. Dehnhardt and U. Petschow, editors. Sustainability in River Basins. A Question of Governance. Oekom-Verlag, Munich, Germany.

Dietz, T., E. Ostrom, and P. C. Stern. 2003. The struggle to govern the commons. Science 302(12):1907-1911

Dosi G. 1988. Technical Change and Economic Theory. Burns and Oates, London, UK.

Douthwaite, T. K., E van de Fliert, and S. Schulz. 2003. Impact pathway evaluation: an approach for achieving and attributing impact in complex systems. Agricultural Systems 78:243-265.

Dreyfus, H., and S. Dreyfus S. 1988. Mind over Machine: the Power of Human Intuition and Expertise in the Era of the Computer. Free Press, New York, USA.

Flyvbjerg, B. 2006. Five Misunderstandings About Case Study Research. Qualitative Inquiry 12(2):219-245.

Frantzeskaki, N., H. S. I. Vreugdenhil, J. H. Slinger, and E. van Daalen. 2008. Do we need a new river basin management paradigm? The missing link for socio-ecological system health. Proceedings of the Freude am Fluss Closing Conference, 22-24 November 2008 Nijmegen. The Netherlands. [online] URL: do we need.doc

Geels, F. and R. Raven. 2006. Non-linearity and Expectations in Niche-Development Trajectories: Ups and Downs in Dutch Biogas Development (1973-200). Technology Analysis and Strategic Management 18(3/4):375-392.

Geldof, G. 2005. Interactive Implementation. Proceedings of the 10th International Conference on Urban Drainage, 21-26 August 2005, Copenhagen, Denmark.

Greenberg, D. and M. Shroder. 2004. The Digest of Social Experiments. 3rd ed., Urban Institute Press, Washington D.C., USA.

Gunderson, L. H., C. S. Holling, and S. S. Light, editors. 1995. Barriers and bridges to the renewal of ecosystems and institutions. Columbia University Press, New York, USA.

Gunderson, L. 1999. Resilience, flexibility and adaptive management - - antidotes for spurious certitude? Conservation Ecology 3(1):7 [online] URL:

Henderson, R., and K. Clark. 1990. Architectural Innovation: The Reconfiguration of Existing Product Technologies and the Failure of Established Firms. Administrative Science Quarterly 35(1):9-30.

Holling, C. S. 1978. Adaptive Environmental Assessment and Management. John Wiley and Sons, London, UK.

Hoogerwerf, A., editor. 1998. Het ontwerpen van beleid. Een handleiding voor de praktijk en resultaten van onderzoek. Samson Tjeenk Willink, Alphen aan den Rijn, The Netherlands.

Hoogma, R., R. Kemp, J. Schot, and B. Truffer. 2002. Experimenting for Sustainable Transport. The approach of Strategic Niche Management. Spon Press, London, UK.

Huitema, D., E. Mostert, W. Egas, S. Moellenkamp, C. Pahl-Wostl, and R. Yalcin. 2009. Adaptive water governance; Assessing adaptive management from an institutional perspective. Ecology and Society 14(26). [online] URL:

Karstens, S. 2009. Bridging boundaries: making scale choices in policy analysis studies on water management. PhD thesis, Delft University of Technology, Delft, The Netherlands.

Ker Rault, P. A. 2008. Public Participation in Integrated Water Management – a Wicked Concept for a Complex Societal Problem. Dissertation, Cranfield University, UK

Kingdon, J. W. 1984/1995. Agenda's, Alternative and Public Policy. Little, Brown and Company, Boston, Massachusetts, USA.

Kolb, D. A. 1984. Experiental Learning: experience as the source of learning and development. Prentice Hall, Upper Saddle River, New Jersey, USA.

Koppenjan, J. 1993. Management van de beleidsvorming. Een studie naar de totstandkoming van beleid op het terrein van het binnenlands bestuur. VUGA BV, Den Haag, The Netherlands.

Lee, K. 1993. Compass and Gyroscope: Integrating Science and Politics for the Environment. Island Press, Washington D.C., USA.

Lee, K. 1999. Appraising Adaptive Management. Conservation Ecology 3(2):3. [online] URL:

Liebowitz, S., and S. Margolis S. 1995. Path dependecy, Lock-in and History. The journal of law, economics and organization 11(1).

Lindblom, C. E. 1993. The policy making process. 3rd ed. Prentice Hall, London, UK.

Martin, S., and I. Sanderson. 1999. Evaluating public policy experiments: Measuring outcomes, monitoring processes or managing pilots. Evaluation 5:245.

Miles, M., and A. M. Huberman. 1994. An expanded sourcebook. Qualitative data analysis, second edition. Sage Publication, Thousand Oaks, California, USA.

Miser, H., and E. S. Quade, editors. 1985. Handbook of Systems Analysis, overview, uses, procedures, applications and practice. Wiley, New York, USA

Mitchell, B. 1990. Integrated Water Management. Pages 1-21 in B. Mitchell, editor. Integrated Water Management: International experiences and perspectives. Belhaven Press, London, UK.

Mitleton-Kelly, E. 2003. Ten Principles of Complexity & Enabling Infrastructures. Pages 23-54 in E. Mitleton-Kelly, editor. Complex Systems & Evolutionary Perspectives of Organisations: The Application of Complexity Theory to Organisations. Elsevier, Amsterdam, The Netherlands.

Morris, C., and C. Chiu. 2001. Experimental Design: Large-scale Social Experimentation. Pages 5086-5090 in N. J. Smelser and P. B. Baltes, editors. International Encyclopedia of the Social & Behavioral Sciences. Elsevier, Amsterdam, The Netherlands.

Muro, M., and P. Jeffrey. 2008. A critical review of the theory and application of social learning in participatory natural resource management processes. Journal of Environmental Planning and Management 51(3): 325.

Olsson, P., C. Folke, and T. Hahn. 2004. Social-ecological transformation fro ecosystem management: the development of adaptive co-management of a wetland landscape in Southern Sweden. Ecology and Society 9(4):2. [online] URL:

Pahl-Wostl, C. 2006. The importance of social learning in restoring the multifunctionality of rivers and floodplains. Ecology and Society 11(1): 10. [online] URL:

Pahl-Wostl, C. 2007. Transitions towards adaptive management of water facing climate and global change. Water Resources Management 21:49-62.

Pahl-Wostl, C. 2008. Requirements for Adaptive Water Management. Pages 1-22 in C. Pahl-Wostl, P. Kabat, and J. Moeltgen, editors, Adaptive and Integrated Water Management: Coping with Complexity and Uncertainty. Springer-Verlag, Berlin, Germany.

Pauli, D. A. S. 2001. Ambiguïteit in het spel. De casus PolyVinylChloride. Dissertation. Eburon, Delft, The Netherlands.

Pawson, R., and N. Tilley. 1997. Realistic Evaluation. Sage Publications, London, UK.

Raven R. P. J. M. 2005. Strategic niche management for biomass. Dissertation, Eindhoven University of Technology, Eindhoven, The Netherlands.

Raven, R. 2007. Niche accumulation and hybridisation strategies in transition processes towards a sustainable energy system: An assessment of differences and pitfalls. Energy Policy 35:2390-2400.

Raven R., S. van den Bosch, G. Fonk, J. Andringa, and R. Weterings. 2008. Competentiekit Transitie-experimenten. Utrecht, the Netherlands.

Rogers, E. M. 1995. Diffusion of innovations, fourth edition. The Free Press, New York, New York, USA.

Rotmans, J., R. Kemp, M. van Asselt. 2001. More evolution than revolution. Transition management in public policy. Foresight 3(17).

Sanderson, I. 2002. Evaluation, Policy Learning and Evidence-based policy making. Public Administration 80(1):1-22.

Scheffer M., S. Carpenter, J. A. Foley, C. Folke, and B. Walker. 2001. Catastrophic shifts in ecosystems. Nature 413.

Schneider, A., and H. Ingram. 1990. Behavioral assumptions of policy tools. Journal of Politics 52(2).

Schwandt, T. 2003. ‘Back to the Rough Ground!’ Beyond Theory to Practice in Evaluation. Evaluation 9(3): 353-364.

Sendzimir, J., P. Magnuszewski, Z. Flachner, P. Balogh, G. Molnar, A. Sarvari, and Z. Nagy. 2007. Assessing the Resilience of a River Management Regime: Informal Learning in a Shadow Network in the Tisza River Basin. Ecology and Society 13(1):11. [online] URL:

Simon, H. 1977. Models of discovery and other topics in the methods of science. Dordrecht, Netherlands, Reidel.

Teisman, G. 1995. Complexe Besluitvorming. Een pluricentrischs perspectief op besluitvorming over ruimtelijke beslissingen. VUGA BV, Den Haag, The Netherlands.

Van den Bosch, S., and J. Rotmans. 2008. Deepening, Broadening and Scaling up: towards a Conceptual Framework for Transition Experiments. Rotterdam, The Netherlands.

Van de Poel, I. 2003. The transformation of technological regimes. Research Policy 32:49-68.

Van Eeten M. J. G., and E. M. Roe. 2002 Ecology, Engineering and Management: Reconciling Ecosystem Rehabilitation and Service Reliability. Oxford Univerisity Press, Oxford, UK.

Van Lente, H. 1993. Promising Technology: The Dynamics of Expectations in Technological Developments. Dissertation. University of Twente, Enschede, The Netherlands.

Van Mierlo, B. 2002. Kiem van Maatschappelijke Verandering. Verspreiding van zonnecelsystemen in de woningbouw met behulp van pilotprojecten. Dissertation, University of Amsterdam, Amsterdam, The Netherlands.

Van Sandinck, E., and R. Weterings. 2008. Maatschappelijke Innovatie Experimenten: Samenwerken in baanbrekende initiatieven. Koninklijke van Gorcum, Assen, The Netherlands.

Vreugdenhil, H. S. I., and P. A. Ker Rault. 2009. Pilot Projects as a Platform for exploring Participative Accountable and Transparent Governance in Water Management: Some Insights on the Diffusion of Experience. Proceedings of the 7th International Science Conference on the Human Dimensions of Global Environmental Change, 26-30 april 2009. Bonn, Germany. [online] URL:

Walters, C. J. 1986. Adaptive management of renewable resources. McGraw Hill, New York, USA.

Walters, C. 1997. Challenges in adaptive management of riparian and coastal ecosystems. Conservation Ecology 1(2):1. [online] URL:

Weiss, C. 1975. Evaluation Research in the Political Context. Pages 13-26 in E. L. Struening and M. Guttentag (editors). Handbook of Evaluation Research. Sage Publications, Thousand Oaks, California, USA.

Weiss, C. H. 1977. Research for policy’s sake: The enlightenment function of social research. Policy Analysis 3:531–545.

Weiss, C. H. 1980. Knowledge creep and decision accretion. Knowledge 1:381–404.

Wildavsky, A. 1979. Speaking Truth to Power: The Art and Craft of Policy Analysis. Little, Brown and Company, Boston, Massachusetts, USA.

[†] The webpage has changed since then, and the majority of project descriptions have since been removed. The original texts are, however, available upon request.

Address of Correspondent:
Heleen Vreugdenhil
Jaffalaan 5, 2628 BX Delft


Home | Archives | About | Login | Submissions | Subscribe | Contact | Search