Ecology and SocietyEcology and Society
 E&S Home > Vol. 24, No. 1 > Art. 30
The following is the established format for referencing this article:
McFadgen, B. K. 2019. Connecting policy change, experimentation, and entrepreneurs: advancing conceptual and empirical insights. Ecology and Society 24(1):30.

Connecting policy change, experimentation, and entrepreneurs: advancing conceptual and empirical insights

1Institute for Environmental Studies, Vrije University, Amsterdam, The Netherlands


With global environmental problems worsening, policy makers and nonstate actors are looking for viable solutions through policy innovation, entrepreneurship, and experimentation. Research into the use of experiments to innovate is increasing, but the role of experimentation in policy change has yet to be specifically addressed in the context of climate governance. My aim is to improve understanding by examining how entrepreneurs, key agents of change, might use experiments to advance their climate innovations. Policy entrepreneurs can benefit in several ways from using experiments, including assessing public response to new ideas and learning. I address the question: What role can experiments play in an entrepreneur’s change strategies? To answer this, a set of 18 policy experiments from Dutch water management was analyzed to understand how the policy experiments functioned as 4 different policy change strategies. The results revealed that organizers use experiments to evaluate their preformed ideas, to soften local communities to the idea of experimentation, to build broad but centrally controlled coalitions, and to link with influential political actors and national programs to maintain visibility and relevance. These insights formed a list of suggestions that the experiment organizers identified as key to the change strategies. Based on this, a number of recommendations about design choices were made for entrepreneurs who want to experiment. Analyzing experiments as change strategies contributes a novel perspective on how policy experiments function as venues for invention and provides useful suggestions on how experiments can be designed to improve their influence over policy-making processes.
Key words: bricolage; climate adaptation; Dutch water management; policy change strategies; policy entrepreneur; policy experimentation


There is a very strong spotlight currently shining on the use of experiments in environmental governance. As society grapples with critical environmental issues borne out of resource use and exacerbated by climate change, academics in a variety of scientific fields are advising policy makers and societal groups to develop and test new ideas by experimenting. Experimentation has been described as a “protean” concept because of its multiple uses in disciplines addressing environmental policy issues, including understanding human behavior, evaluating policy interventions, and solving sustainability and climate change issues (Ansell and Bartenberger 2016).

One field in which experiments are gaining traction is climate governance. Experiments are said to more competently address the “multi-dimensional and complex nature” of climate change than other, more traditional governance modes (Kivimaa et al. 2017:17). Recent publications have attempted to categorize the broad and diffuse uses of climate governance experimentation, facilitate consistent meanings, and critically examine their uses and outcomes (Ansell and Bartenberger 2016, Laakso et al. 2017, McFadgen and Huitema 2017, 2018, Weiland et al. 2017).

Establishing how experiments can be effective in facilitating change is also gaining attention. The field of sustainability studies has recently identified success factors (van der Heijden 2014, Antikainen et al. 2017), explained how an experiment can perform as a “pathway for change” (Bai et al. 2010, Farrelly and Brown 2011), and shown how experiments are embedded in societal and political fabric by deepening, broadening, and scaling up (Laakso et al. 2017). In the policy sciences, experiments primarily affect change by providing evidence of an innovation’s effects (Greenberg et al. 2003, Ettelt et al. 2015, McFadgen and Huitema 2017). Studies have also shown how experiments can be used to manipulate the policy process by delaying decision making (Greenberg et al. 2003), bring a broad range of actors into the policy process by creating “shadow networks” (Olsson et al. 2006), and build acceptance among a local community (Millo and Lezaun 2006). Still, Kivimaa et al. (2017) observe that how experiments bring about change is underexplored, particularly in the climate literature.

In studies on experiments for climate governance, the government’s role is peripheral, partly because experiments are expected to upset the status quo by working outside common decision-making processes (Hoffman 2011, Broto and Bulkeley 2013). However, there is value in the view that governments still have a key role to play in climate innovation, particularly their use of or participation in policy experiments. Drawing from this, policy experiments are defined as “temporary, controlled field-trials of policy-relevant innovations that produce evidence for subsequent policy decisions” (McFadgen and Huitema 2017:1768). The primary intention of policy experimentation is therefore to encourage learning of the effects of new proposals (Peters 1998, Sanderson 2002, Ansell and Bartenberger 2016).

The focus in the climate literature on experiments being initiatives developed outside the ambit of government has also led to any political dynamics emerging from experimentation being ignored or underexplored (Hoffmann 2011, Huitema et al. 2018). Too often, an experiment is generally seen as a neutral tool that evaluates how governance “can be done” as opposed to a mode of governance itself that is intrinsically political in nature (Brodkin and Kaufman 2000, Simons and Voß 2018). An experiment has political dynamics surrounding it, as well as within it, with the choice to experiment, the interpretation of results, and whom to involve all being politically driven choices (Huitema et al. 2018). It is for this reason that a policy science perspective is taken to better understand the relationship between change and experiments.

Along with policy experimentation, the second key concept I have used is the notion of the policy entrepreneur. The idea of a policy entrepreneur was first developed by Kingdon (1984) and is an agent-based explanatory theory that has been increasingly used to explain policy change (e.g., Cohen 1988, Young 1991, Roberts 1992, Mintrom 2000, Huitema and Meijerink 2010). Roberts (1992:56) defined public entrepreneurship as the “generation of a novel or innovative idea and the design and implementation of the innovative idea into public sector practice.” She goes on to label those individuals who “generate, design, and implement” those innovative ideas “public entrepreneurs” (Roberts 1992:56). Recently, the focus has moved to policy making more specifically, and policy entrepreneurs are defined as “energetic actors who work with others in and around policymaking venues to promote significant policy change” (Mintrom and Luetjens 2017:1363). Policy change theories have been used to build understanding of the policy entrepreneur’s role in managing change, and specific strategies that policy entrepreneurs might use have been isolated (Huitema and Meijerink 2010, Brouwer and Huitema 2018).

The policy literature is limited in its discussion on the relationship between policy experiments and policy entrepreneurs, despite their shared association with policy innovation and change. The few studies that exist include Roberts (1992), who claims entrepreneurs use experiments to test the survivability of their innovations, and, recently, John (2017), who identified those policy makers that use experimental methods as entrepreneurs. Meijerink and Huitema (2010) conclude that entrepreneurs miss out on learning opportunities because they use pilots to sell an idea to the public as opposed to testing it. Taking a different tack, Bernstein and Hoffmann (2018) suggest that entrepreneurs can use experiments to generate normative changes, whereas Martí and Mair (2009) describe how experiments are used by operators in niches to resist dominant governance approaches. These studies suggest that entrepreneurs have various uses for experimentation, including testing an idea, operationalizing radical interventions, and producing evidence, and I hope to build on these empirical observations.

To explore the dynamics between experimentation and entrepreneurship, how experiments emulate the strategies that entrepreneurs use is examined. My main research question is: What role can experiments play in an entrepreneur’s change strategies? To answer this question, the literatures on policy experiments and policy entrepreneurship are reviewed. It is argued that entrepreneurs have a range of strategies at their disposal, and how experimentation could play a role in each of those is explored. This inductive strategy leads to several hypotheses about the way experiments might fit in the toolbox of the entrepreneur (see the next section). To see how and to what extent experiments can perform as change strategies in empirical settings, 18 experimental cases in Dutch water management were analyzed. The need for climate adaptation is acute in the vulnerable delta of the Netherlands, especially in relation to increased flooding. The Methods section describes each case and sets out the methods of data collection and analysis used in the study. The section Exploring the role of experiments in the four change strategies provides a list of best practices for how entrepreneurs can design experiments to improve their chances to influence change. The Discussion and conclusion section suggests that entrepreneurs can heighten the relevance of experiments by testing ideas that solve multiple problems at once, mimicking the act of bricolage. The findings provide practical and theoretical insight into how entrepreneurs can use experiments to influence the policy process.


The role of experimentation in policy change has yet to be specifically addressed in the context of climate governance, and I aim to improve understanding by examining how entrepreneurs, key agents of change, might use experiments to advance climate innovations. To do this, the policy continuity and change theories that dominate the policy science literature are relied on. These theories include the multiple streams framework (Kingdon 1984), the advocacy coalition framework (Sabatier 1988), and the punctuated equilibrium framework (Baumgartner and Jones 1991). They consider policy development a random and rare process (Kingdon 1984, Baumgartner and Jones 1991), which is seldom brought about by an individual or group of people. However, taking the same view as Olsson et al. (2006), Huitema and Meijerink (2010), and Brouwer and Huitema (2018), it is argued that policy change may be deliberately affected and “navigated” by people in absence of crisis events (Olsson et al. 2006).

As explained previously, links are increasingly being made between entrepreneurship and experimentation. Following John (2017), it is claimed that those who experiment are characteristically entrepreneurial. Experiment organizers can originate from inside or outside government (McFadgen and Huitema 2018) and might use one or a selection of strategies to increase the visibility and impact of their experiments. Starting from the list of five entrepreneurial strategies found in Huitema and Meijerink (2010), two are identified as directly relevant to experimentation, i.e., “exploiting policy windows” and “developing new ideas,” and one of their strategies, “building coalitions and sell ideas,” is divided into two: “demonstration” and “building coalitions.” The other strategies of Huitema and Meijerink, “recognizing multiple venues in modern society” and “orchestrating networks,” did not resonate with the utilized understanding of experimentation and so were left out of the analysis. Subsequently, the four relevant strategies and how they are related to theories of experimentation are set out.

The first of the four strategies examined is that of “developing new ideas.” Entrepreneurs trigger policy change by inventing a new idea that is often an innovative solution to a pressing problem, which they can do alone or in concert with others (Huitema and Meijerink 2010). These new ideas are also understood as new policy frames (Schön and Rein 1994), policy images (Baumgartner and Jones 2002), or alternative system configurations (Olsson et al. 2006) that emerge as “a germ of an idea” and suggest new ways the situation might change (Meijerink and Huitema 2009:31).

To better understand the extent to which policy experiments are used to develop new ideas, Ansell and Bartenberger’s (2016) classification of controlled and generative experiments is considered. Controlled experiments are field experiments that infer their findings deductively, that is, by exerting control to establish causality and prove or disprove the hypothesis being tested (Fischer 1995, Ansell and Bartenberger 2016). The ideas being tested in controlled experiments stem from academic sources or, in some cases, from public managers that deliver public services (John 2017). In contrast, Ansell and Bartenberger (2016:69) also identify “generative” experiments, which “iteratively refine innovations … in order to abductively generate novel solutions.” Control over the process is minimal or nonexistent, and the idea “develops” as the experiment adapts to new findings and changing circumstances. These categories suggest that experiments can be used to either develop a new idea throughout the process of implementation or strictly evaluate an idea using processes and objectives locked in at the beginning.

“Demonstrating an idea” to gain attention and “sell” it is another key policy change strategy (Mintrom 2000, Huitema and Meijerink 2010, Brouwer and Huitema 2018). Brouwer and Huitema (2018) found that entrepreneurs believed the demonstration strategy was useful for backing up their claims on the nature of a problem while also convincing their audience that the solution was the best means to solve that problem. Demonstrating an idea shifts it from the abstract to the concrete and shows off its technical feasibility, i.e., how well the idea works, and builds social feasibility, i.e., acceptance by the people affected. This second aspect of feasibility relates to the concept of “softening up,” whereby demonstrating the solution helps get people used to the new idea and helps the entrepreneur build acceptance around it (Kingdon 1984, Beem 2007).

Entrepreneurs use pilot projects to put the demonstration strategy into action, to sell and build acceptance for their ideas (Mintrom 2000, Taylor et al. 2011). Experiments can function in a similar way, although they must also evaluate outcomes (Huitema and Meijerink 2010). Evaluation signifies that the innovation’s effects are unknown, the difference between selling an idea as a “real and permanent” solution as opposed to a “potential” solution (Peters 1998:128). In this way, experiments allow organizers to claim their innovation is “more precise and evidence based” (John 2017:489) but opens them up to accusations of risk taking.

Policy change is seldom produced by an individual working alone (Imperial 2005), and coalition building is a strategy that involves linking actors and building support around a new idea (Huitema and Meijerink 2010). Coalitions take a variety of forms; for example, the advocacy coalition framework focuses on the interactions between groups of actors who share a set of policy beliefs and belong to the same policy subsystem (Sabatier 1988), whereas “shadow networks” are coalitions made up of informal networks of actors outside regular policy processes (Olsson et al. 2006). Brouwer and Huitema (2018) highlight the strategic and substantive benefits of coalition building for entrepreneurs in Dutch water management, including increased collaboration, legitimacy, support, and resources.

For some scholars, experiments provide an innovative and participatory approach to climate governance where nonstate actors contribute knowledge, capabilities, and resources (Berkhout et al. 2010). Because experiments approach a problem in a new way, they form new actor combinations by bringing together actors from inside and outside the “inner circle” (Broto and Bulkeley 2013, van Popering-Verkerk and van Buuren 2017:226). Diverse experiments have the potential to increase collaboration, legitimacy, and learning (Smith 2007, van der Heijden 2014, Ansell and Bartenberger 2016, McFadgen and Huitema 2018). However, managing a diverse coalition of actors with different interests can lead to unmanageable complexity and delays (Brouwer and Huitema 2018), and in contrast, maintaining a simple collaboration between experts and policy actors can increase knowledge acquisition (McFadgen and Huitema 2018) and reduce the chances of an experiment being hijacked by outside interests (Ansell and Bartenberger 2016). Academics and bureaucrats dominate the controlled experiments favored by the United Kingdom’s public health bureaucracy and the government’s behavioral unit (Haynes et al. 2012, Ettelt et al. 2015), as well as U.S. economic policy experiments (Brodkin and Kaufman 2000, Greenberg et al. 2003). Policy makers use academic expertise for their decision making, whereas the experts need the government “to test out [their] claims” (John 2017:484). In these experiments, societal actors are excluded and instead play the role of spectator or experimental subject (Weiland et al. 2017).

The fourth change strategy involves identifying and realizing an opportunity to get an issue on the policy agenda, known metaphorically as exploiting a “window of opportunity.” This concept forms the basis of Kingdon’s (1984) multiple streams model of policy change, the convergence of the policy, problem, and political streams. Policy entrepreneurs are particularly astute at recognizing and exploiting opportunities to influence the policy agenda (Kingdon 1984, Olsson et al. 2006, Meijerink and Huitema 2009). If an ecological, economic, or political crisis emerges, entrepreneurs will use this problem to advocate their idea as a solution (Beem 2007). They might also assemble a new solution out of a mix of existing solutions to address multiple problems in an act of bricolage (Martí and Mair 2009, Olsson et al. 2017). Concurrently, they will try to build political support for their idea by gaining the attention of political actors or taking advantage of a change in government or other political upset that swings support in favor of the solution (Huitema and Meijerink 2010).

Experiments are useful tools for combining the streams and creating a policy window. They can exert influence on the policy agenda by providing evidence of effects of a new policy solution, thereby linking the policy and political streams (Greenberg et al. 2003). Experiments can also link the problem and policy streams by drawing wider attention to an issue and claiming to potentially solve it (Caniglia et al. 2017). However, experiments are vulnerable to attention shifting in the problem and political streams. During an experiment’s implementation, a problem might be solved or replaced on the agenda by a more urgent problem, and the favorable circumstances that enabled an experiment to be commissioned in the first place can change (Brodkin and Kaufman 2000). Similarly, if support and attention for an issue decline in the political stream, it might jeopardize the impact any results may have (Sanderson 2002, Greenberg et al. 2003).

Table 1 summarizes the main points made about the ways experiments relate to the policy strategies. These elements are investigated empirically in Exploring the role of experiments in the four change strategies.


The policy experiment cases

To test tentative assumptions about the role of experiments in policy change strategies, a set of 18 water policy experiments conducted in the Netherlands between 1997 and 2016 was identified, out of a list of 147 potential experimental interventions, because they met several criteria used to identify policy experiments. The criteria were developed by operationalizing the “policy experiment” definition used in this article (McFadgen and Huitema 2017) and included testing for real-world effects, being innovative, and having a clear relevance to policy (see Appendix 1 for a comprehensive list of criteria and associated indicators). The list of 147 potential experimental interventions is available on request.

The climate change threat is particularly acute to the Netherlands, and many policy changes are needed to address current and future risks (Brouwer and Huitema 2018). The experiments aimed to solve water problems, and they are considered relevant to climate governance because adaptation in the Netherlands is largely framed as a water policy issue (Massey et al. 2014). They included five coastal management experiments, five water storage experiments, three freshwater experiments, three water variability experiments, and two dike management experiments (see Appendix 2 for information on each of the experimental cases, including description, duration, location, and list of actors). The majority of experiments were trial manifestations of already well-established concepts, like the “water plaza,” which stores water in the city after heavy rain; the “sand engine,” which naturally disperses sand along the coastline; or a “climate buffer,” which protects ecosystems from climate change using natural processes.

Multifunctional solutions were the most commonly tested (10 experiments), and these tended to try to improve on existing practices, for example, sand supplementation using natural processes to restore wetlands and provide coastal defense. Four experiments sought to meet existing policy objectives by investigating more flexible forms of water management, for example, techniques that could provide farmers with more flexible water levels on their farms. The other four experiments looked at the transfer of responsibilities for water from public to private actors, for example, testing a process in which farmers stored water on land and emptied storage basins when the water board predicted inundation.

Data collection and methods of analysis

Empirical data were primarily collected from a half-day workshop held for experiment organizers in February 2015. Out of the 18 organizers invited to attend the half-day workshop, 11 were willing to attend and discuss their experiences in conducting an experiment. The workshop participants represented a range of backgrounds: 3 were policy actors from either a local, provincial, or national state institution; 4 worked at water boards; 2 were academics; 1 represented an NGO; and 1 was a business actor. Appendix 3 lists the names and affiliations of the workshop participants.

The workshop first asked participants to discuss with one another the meaning of experimentation and why they chose to experiment. The group was then divided into two groups; the members of one group discussed how they made specific choices when designing their projects, and the members of the other group discussed how they used their experiments to influence their policy network (translated transcripts are available on request). Appendix 4 sets out the workshop questions, which have been translated from Dutch. The questions are open ended, and the intention was for organizers to discuss the questions between them and share experiences and opinions, with minimum input from the moderators. This reduced any moderator bias, although some bias was expected as a result of the participants being experiment organizers who agreed to talk openly about their experiences, and therefore less likely to discuss negative experiences from their experiments, compared with a group of participants chosen at random. Moreover, these were organizers that were willing to attend the workshop and discuss their experiences in the first place, which also affects, to some extent, the validity of the findings.

The workshop data supplement data from 23 semistructured interviews with experiment organizers conducted between 2014 and 2015, on the phone and in person, and an extensive survey conducted from June to September 2014. The survey collected data on institutional design variables from participants (n = 170) of the 18 experiments and was analyzed using SPSS 21. Documents relating to each experiment were also analyzed for background information, including experiment feasibility studies, evaluation reports, reports, and peer-reviewed articles.

Using the policy strategies as codes, a thematic analysis of the workshop and interview transcripts was conducted to identify comments and conversations about the change strategies and evidence of best practices. Relevant sentences, arguments, and conversations were coded into specific strategies and analyzed, with participant confidentiality recognized. Claims particularly pertinent to how the strategies were used were noted, and similar comments were grouped together. Strong claims were cross-checked with data from the survey, i.e., the extent to which people could request participation in an experiment.


The following section sets out the empirical analysis, which critically reviews how and to what extent the experimental cases are used as change strategies in Dutch climate adaptation and water management. The disparities found in the literature regarding how the experiments might be used are focused on the following questions: Are experiments used to develop ideas abductively or to test ideas deductively? Are they being used to sell and build support for an idea? What sorts of coalitions are built in experiments, if any, and what issues arise? How can experiments maintain visibility as problem solvers and relevance to political decision makers? These questions are addressed, and the main findings are summarized.

Germinate or evaluate a new idea?

Developing a new idea is a logical first step toward policy change, but the literature is unclear about whether experiments serve this purpose. To gauge whether Dutch policy experiments implement and evaluate a fully constructed idea or whether they cultivate the germ of an idea as they progress, three elements are considered. The first element is the extent of control used to isolate causality and test hypotheses, and the extent of control in individual cases should be evident by the type of evaluation process used. The second element is the type of actors involved in designing the experiment; for example, controlled experiments involve scientists as experimenters, whereas generative experiments involve societal and political actors in design and implementation (Weiland et al. 2017). Finally, the significance of failure features here. For deductive experiments, failing to meet preset objectives is a realistic outcome, whereas generative experiments are “success oriented” and failure might be ignored (Weiland et al. 2017:36). Organizers’ attitudes toward failure may indicate whether experiments are being used to develop or evaluate ideas.

How an experiment tested a working hypothesis indicated the extent of control. Because evaluation was a criterion for identifying policy experiments, all the experimental cases contained formal monitoring and evaluation processes to gauge whether the solutions were effective in addressing the problems. They tested technologies or processes and assessed the expected effects of the technical, ecological, or social elements. However, for these experiments, isolating causality was not the intention. Although they had predefined hypotheses, most cases did not use experimental controls to deductively disprove them, with only three experiments including control sites; for example, the freshwater storage experiment on the Dutch island of Texel replicated itself at a control site on the mainland that did not suffer the same drought effects as farmers on the island. At the other end of the spectrum, only one experiment clearly started as an open forum to work abductively and find solutions, i.e., the “Oranjepolder pilot,” and during the experiment, a monitoring framework was developed to evaluate the innovation.

Although predetermined expectations were the norm for the cases, an element of flexibility was noted in the experimental approach when unexpected consequences emerged. For example, the Waalblok experiment, which tested the feasibility of water storage beneath greenhouses, had to radically restructure its goals when the economy took a downturn and farmers were not willing to make the required investments. Also, a dynamic coastal management experiment was relocated and scaled back after complaints from local communities, and the water plaza had to be significantly redesigned after residents voiced concerns that the water storage would be dangerous for neighborhood children. These examples suggest that experiments can be flexible and open to making changes, like abductive experiments, but still hold form and test preformed assumptions, like deductive experiments.

Second, the extent of actor diversity was assessed, because deductive experiments only involve experts. However, despite being deductive in the sense that they test predetermined hypotheses, the projects averaged three to four actor types per experiment, departing from the categorizations in Ansell and Bartenberger (2016; see Appendix 2 for a list of actors per experiment). However, the most common actor types, i.e., policy actors and experts, were mostly responsible for experimental design, and the other types were offered little opportunity to craft the experiment and impress the policy agenda, which is expected in abductive experiments.

Third, whether experiments are designed to fail is indicative of whether they are abductive or deductive. Unlike success-driven experiments, the experimental cases’ investigative nature meant they were fallible. One organizer reflected that “my experiment … the director said ‘[…] is allowed to fail’ and I liked that. I thought, ‘Well, that’s nice, it may fail’” (workshop participant 2). However, the uncertainty of outcomes can make experiments unpopular, as an organization’s auditors and compliance teams were labeled by one organizer as “the greatest enemy of experiments,” because of their aversion to the risks taken during an experiment (workshop participant 9). It was noted that, for policy development generally, “there is actually little tolerance for failure, or free space to try new things” (workshop participant 4).

In sum, based on these observations, it is tentatively suggested that the policy experiments were used to operationalize novel but preexisting policy concepts using monitoring and evaluation processes akin to deductive experiments, but that they were in themselves about innovation per se. At the same time, they maintained flexibility to change their direction if complexities emerged and engaged more actor types than just experts. The workshop discussants noted that having the opportunity to fail was novel but unpopular, and this could possibly explain why there are so few experiments.

Building support for potential solutions

The second change strategy assessed is the extent to which an experiment is used to demonstrate a solution to sell it and build support. The first element assessed is the way organizers manage the inherent uncertainty of outcomes, which comes with experimentation. The second element is the extent of transparency of the experimental process. Experiments that are designed to be transparent and open to participants and the local communities in which they are embedded can help dispel fears that an experiment is a risk to its surroundings.

Regarding uncertainty, during the workshop discussions and interviews several organizers mentioned the difficulties they faced in siting and implementing their experiments and how they had to “sell” the idea of experimenting itself. As one organizer noted, “Because everyone likes the status quo as it is … if something is changing, people want to be involved and take care their interests are not going to be damaged” (workshop participant 1). To build agreement, organizers emphasized the temporary and small-scale characteristics of the projects and engaged locals in site visits and workshops to explain the potential yet uncertain implications. Two organizers noted that offering complete compensation to affected parties helped create support among locals and reassured them they would not lose out by having the experiment implemented in their neighborhood.

Another tactic to minimize negative reactions involved carefully choosing the term used to label the projects. Out of the 18 experimental cases, only 5 explicitly referred to themselves as “experiments,” with the remainder termed “pilots,” “tests,” or “trials.” This was sometimes deliberate, as one organizer shared: “The field trial runs throughout the Netherlands and we also have to convince the public and [that is why] we very consciously did not select the word ‘experiment,’ but the word ‘test’” (workshop participant 5). However, another organizer noted that in his experience, the label “experiment” signified to locals that the project would be temporary and small scale, which convinced them to give their support (workshop participant 2).

Next, I reflect on the importance of maintaining transparency through the experimental process. Not all experiments were implemented in a public space, and half of the organizers sited their experiments on private land. Six experiments were sited on horticultural farmland, two on NGO-owned land, and one on a “test farm” research station. This circumnavigates the need to sell the idea of experimenting to local communities and undermines transparency. However, performing the experiments on farms still required the support of private actors, so whether on a test farm or in public, organizers worked hard to convince participants the experiments were beneficial.

Enlisting enthusiastic participants as ambassadors for the experiments also helped. One organizer explained that his experiment in climate adaptive drainage ran into trouble because it was weather dependent, and his initial data set was disappointing, but one of the farmers he was working with agreed to become an ambassador and explain to other farmers why the technique was beneficial. Talking to one of their “peers” helped others understand the experiment’s purpose. Finally, to improve transparency, organizers noted that the technical aspects of experiment reports could be difficult for citizens and lay audiences to understand, with one sharing: “We work in science but we work for government too and its public money so they [citizens] have a right to as much information as possible, translated and made easier, it is important to me” (workshop participant 1).

Overall, involving locals in experiments via site visits, offering compensation, communicating the intentions and results of the project in a manner they can understand, and listening to their opinions can help build support and alleviate anxiety about the experiment. Maintaining transparency by translating results into lay language and being open about the experiment also aid in building support or at least substantive dialogue with the relevant and affected communities.

Coalition building: advantages and limits to drawing in new actors

Next, whether the experiments were used to build coalitions is observed by considering the extent of actor diversity in the experimental cases and the formation of new groups. Appendix 2 lists the wide range of actor types that were involved in the experiments, but despite the diversity, policy actors (50% of the total participants) and experts (23%) dominated the projects (see Fig. 1). They were joined by few NGO representatives (9%), business actors (9%), and individual citizens (8%).

The extent to which these participants formed novel coalitions also varied. In 6 experiments, most participants (more than 75%) knew fewer than half of the other participants, indicating that these experiments were used by organizers to bring together constellations of new actors. In contrast, 4 experiments involved participants that almost all previously knew each other. The remaining 10 involved participants who knew about half of the group they joined. This indicates that most experiments worked to bring together some actors that had not collaborated before, but the groups were rarely entirely new.

The experiments were used to build broad coalitions, some with new actor constellations, but to what effect? Coalitions built with different actor groups present opportunities to share different perspectives, on which experiments may rely. In one instance, an organizer recalled he was asked to join the experiment to bring his outside understanding of how locks between salt water and freshwater could be managed differently. His role was to act as a “crowbar” to help the project team think differently about the problem and potential solution and increase understanding between parties (workshop participant 11).

Although exposure to different perspectives is a key advantage of coalition building, it seems that including everyone who wants a role in an experiment was not viewed as particularly desirable. Most participants in the experimental cases were granted entrance to the experiments via invitation from the organizer, and only six experiments allowed people to “self-nominate”/volunteer themselves. An organizer argued that he “wouldn’t say that you want to involve everyone. That’s per definition almost impossible in an experiment. You also have to take into account a certain degree of friction” (workshop participant 2).

Organizers also noted the value of having participants contribute financial and other resources to the project. One experiment involved entrepreneurial farmers adopting significant responsibilities for water management, and they shared a commitment of resources with the water board, where the state paid for equipment and the entrepreneur paid for maintenance and electricity services. The organizer explained that the farmer needs “to put some effort in so it has value … that principle … something which is totally for free, it becomes … it won’t work” (workshop participant 8). Another organizer noted that some participants in his experiment had fewer means but invested significant time and effort, applying for permits and other administrative tasks, which demonstrated commitment.

Still another way of gaining participants’ commitment was including participants in writing the conclusions of the research. This helps maintain trust and ensures the findings match reality, so they will not “be surprised by a report of which they say ‘well, what’s in there is not right at all’” (workshop participant 7). However, the commitment required did not always translate into authority in the experiments. Only a third of participants in the experimental cases had any decision-making powers, and they were mostly policy actors and experts. Rarely did business actors or private citizens have a say on project design or evaluation of the results, and most took an advisory role in their experiment.

In sum, experiments in Dutch water management and climate adaptation are being used to construct partially new groups of participants who contribute financial support and other resources to the project. Control is maintained, and complexity reduced, by limiting access to the projects and designating decision making to a select few participants. This suggests that many of the experiments are used to build coalitions of similarly minded actors advocating for a specific course of action rather than coalitions of actors brought together to negotiate a common goal.

Coupling streams as the window of opportunity opens

The final policy change strategy examined is the window of opportunity strategy, in which entrepreneurs exploit chances to couple the policy, political, and problem streams to influence the policy agenda. How experiments, as potential solutions in the policy stream, interact with the problem stream is considered first, followed by an analysis of the potential to capitalize on policy windows in the political stream.

Being attached to groups bigger than themselves can make experiments robust to changes in the problem stream because they provide an outlet for experimental results, reducing the chance of sliding into irrelevance. For example, the 10-year South-West Ameland experiment tests whether salt marshes can provide adequate defense against coastal erosion. It is a very localized experiment, i.e., a small site off the coast of an island, but because it is one of the nature restoration projects included in the Climate Buffers program (now the Climate Buffers coalition) it has an outlet to record its progress and maintain visibility. Eleven experiments in total participated in national or international innovation programs that aimed to combat climate change: for example, the Delta Programme and Climate Buffers[1] and Knowledge for Climate[2] programs. Through these programs, experiments exchanged information, received funding, boosted their profile, and kept their progress visible.

Another observation relevant to the relationship between the policy and problem streams is that in many cases, one experiment was used to take care of multiple problems at once. By linking problems, innovative solutions could be imagined, such as water storage and urban development or coastal protection and nature restoration. A workshop participant explained: “We have managed to put a positive spin on the experiments and the pilots that we do … multifunctional solutions make the city stronger. When I say to politicians: for that one euro, we can solve three problems – they are still eager to do those experiments” (workshop participant 9). Connecting experiments to more than one problem makes them more attractive to decision makers and reduces the chances that the experiment becomes redundant if a problem is solved.

These tactics strengthen the policy-problem connection and allow an experiment to capitalize on a window of opportunity, but what about the relationship between the policy and politics streams? Organizers emphasized that the involvement of influential political actors was crucial to an experiment’s survivability. In response to a question about the importance of political support, it was clear how vital it is:

Well he [a political representative] has the power. He has the power, and you can tell a nice story, but that’s not going to do it. (Workshop participant 4)

You will not get the means. (Workshop participant 7)

Or the other way around. Look, if they’re just against you, then it’s “finito.” Then … just quit, it ends here. (Workshop participant 9)

It was noted in the workshop that “an experiment can be in perfect harmony with those in charge at the beginning, but not in the end” (workshop participant 4), which happened to the De Kerf experiment when it had to stop prematurely because the political will to continue monitoring disappeared (phone interview, August 2013). Working to build connections with influential actors helps an experiment navigate through the political changes that may occur during its implementation and also helps it avoid becoming irrelevant.

Building awareness with administrators and councilors about experiments was thought to increase support. One organizer noted that it was important to “keep your administrator involved, also in the front. That when something happens, he really knows what you are doing, why you do certain things, what the message is up to that point” (workshop participant 3). A strong tactic is aligning an experiment with the goals of political representatives to gain political attention. An organizer explained how “there was … the innovation programme of Balkenende [the prime minister of the Netherlands from 2002 to 2010], who had to have something to show at the end to his party … the innovation programme had no icon to be shown, the sand engine was doable, funds were there, it was decided ‘let’s do it’” (workshop participant 4). With this support, the sand engine project was initiated.

Reflection on how experiments are used as policy change strategies

Several salient points can be made based on the findings set out previously. First, a fear of failure might dissuade entrepreneurs from using predefined hypotheses in their experiment, and they may prefer a more abductive approach that evolves as it is implemented and cannot fail. However, this would reduce the capacity of experiments to critically assess new innovations, which in turn reduces the usefulness of their findings for policy development. This relates to the second observation that organizers are more likely to operationalize an existing idea as opposed to develop an idea from scratch, which implies that potential changes are immediately implemented instead of ideal solutions slowly evolving in place. This suggests that effective experiments will be more replicable because they are not so highly contextual. Therefore, organizers essentially choose between evolving, eventually successful, but highly contextual experiments and possibly failing, preformed experiments with potentially generalizable results.

Second, entrepreneurs should anticipate that communities may be skeptical about the risks associated with experimenting, and they should be ready to sell the idea of experimentation itself. Demonstration backed up with evidence shows participants how a new idea really benefits or disadvantages them and the ecological system.

Third, the extra knowledge, understanding, and resources produced by coalition building can be very useful when trying something for the first time for which there is significant uncertainty over the effects. However, the cases highlighted some control mechanisms that organizers use to reduce complexity and possibly dissent, letting themselves alone decide what knowledge is important and whose voice is relevant. By excluding new actors or not letting them contribute, an experiment may really miss out on vital learning opportunities.

Finally, the analysis shows that organizers were acutely aware of the need to connect their experiment to the problem and politics streams. Organizers highlighted the problem they intended to solve by connecting to broader, influential programs and hedging their bets by attempting to solve several problems at once through novel combinations of existing solutions. These tactics extended the visibility of the experiments beyond the original crisis or issue that paved the way for new ideas to gain traction, but they were still vulnerable to the whims of political decision makers. Not all experiments will have the good fortune to be the solution to a prime minister’s political goals, but maintaining visibility among influential people will keep the opportunity open.

Governance choices: experimental design recommendations for entrepreneurs

From this review of experimental cases, I set out some design recommendations on how to maximize an experiment’s ability to influence policy change (Table 2 provides a summary of suggestions). Some findings of the analysis resonate with recommendations in the literature on how to design for “successful” experiments that are upscaled, replicated, or adopted. Studies that recommend the use of evaluation components (finding 1) include van der Heijden (2014) and Nair and Howlett (2015), although the benefits of rigorous evaluation are questioned by Ettelt et al. (2015) and John (2017), who observe that decision makers do not feel compelled to rely on experimental evidence when making policy decisions. Antikainen et al. (2017) noted the strength of performance reporting and broad communication strategies in producing successful experiments (findings 4, 5, 11, 12, and 13).

Findings 2, 8, and 9 relate to the complex balancing act of who to include in an experiment. Dryzek (1987) emphasizes that the intended and unintended effects of experimentation mean any persons potentially affected must be able to voice their concerns. Letting participants self-nominate and empowering them align with a pragmatic, deliberative approach to democracy (Pieraccini and Cardwell 2016); in both areas, most of the experiments fell short. However, frustrations develop when actors with different interests and contributions are included and some actors have less knowledge and competence in understanding the issues than others, termed “cognitive distance” by van Buuren and Loorbach (2009).

Commonly, the literature stresses the need for strong political support to carry the experiments (finding 13). Hildén et al. (2017) conclude that constant interaction between influential high-level bureaucrats and the experiment is crucial, and Antikainen et al. (2017) and Nair and Howlett (2015) emphasize the need for connections and support of political players representing the existing regime. John (2017) warns that it is difficult to obtain support from people in authority when they fear the findings will be critical of them, which reminds us that no matter how apolitical an experiment aims to be, there are always winners and losers in politics.

Other insights that were derived more inductively from the findings include the careful use of the term “experiment” (finding 3), developing enthusiastic participants as ambassadors (finding 7), offering compensation for potential effects (finding 6), and getting involved in broader programs (finding 12).


Policy experiments produce evidence about the effects of innovative technologies and processes on the natural or social world (Ansell and Bartenberger 2016, McFadgen and Huitema 2017). They have been overlooked as an instrument in a policy entrepreneur’s tool kit, which is disappointing because, as I demonstrated, they have a lot of potential to strengthen entrepreneurial activities. A set of 18 experiments was examined to understand how an experiment can play a role in an entrepreneur’s change strategies. Analyzing experiments as entrepreneurial strategies broadens their uses and highlights ways in which experiments can be used as political tools to garner political acceptance and authority (Simons and Voß 2018).

The analysis revealed that experiments function somewhat “strategy lite.” The abovementioned findings lead us to the conclusion that an experiment can provide an intermediate step of monitoring and evaluation, between forming a new idea and selling it as a solution. Mintrom (2000) argues that entrepreneurs should demonstrate that an innovation has no adverse effects, but for truly innovative projects, how can they actually know? By admitting they do not know the outcomes of their projects, entrepreneurs can be more honest with themselves and with the communities with which they work.

Organizers felt that the risks taken with exploring the uncertainties of new innovations meant experiments were uncommon, which may also explain why half the experiments sought to improve on the status quo rather than radically challenge it. However, this might be the key to the relationship between entrepreneurs and climate policy experiments. The concept of “bricolage” has recently come up in discussions both regarding entrepreneurs and managing the social-ecological system. Bricolage “captures the experimental nature” of the work done by entrepreneurs (Martí and Mair 2009:102) and involves the “recombination of pre-existing and new ideas, concepts, or technologies to form something novel” (Olsson et al. 2017). It is suggested that single variable interventions that make radical changes cannot address root causes, and initiatives that resemble bricolage would be more effective in addressing integrated social-ecological issues and breaking path dependence (Olsson et al. 2017). Although the experiments were not so valiant in scope as those referenced by Olsson et al. (2017), the multifunctional experiments do try to solve issues with social and ecological elements. Focusing on testing these sorts of solutions will help sell the idea of experimenting to the political arena and also possibly develop more effective solutions than large radical changes, while also improving the chance an experiment will stay relevant to the problem stream.

This was an exploratory multicase analysis that drew on several data sources to identify broad patterns in the relationship between design and change strategies, and its limitations need addressing. The findings should be viewed as tentative and could be the basis of in-depth single case study analyses of the strategic uses of experiments. Moreover, the findings should be limited to experiments in water management and not extrapolated to experiments in policy making generally. I looked at broad patterns over a set of 18 cases, which meant variation between cases was lost. Studies on the different learning outcomes produced by differently designed experiments have shown design makes a difference.

Future research could explore how different types of experiments function as change strategies. For example, does an experiment conducted mostly by experts, which produces solely technical-analytical knowledge, perform as a strategy differently from one that involves a broad actor group and produces technical and reflexive knowledges? This would be the next step in a deeper analysis of how the governance of experiment affects outcomes.

The increasing focus on experiments in the environmental governance literature stems from the general push for policy innovation and change needed to solve the urgent issues exacerbated by climate change. Better understanding the relationship between experimentation and entrepreneurship arguably strengthens both concepts as tools for policy change. If the use of experiments is sufficiently broadened to reflect the strategies that entrepreneurs use, then entrepreneurs may be encouraged to open themselves up to indeterminacy and use experiments as a venue for innovation. Likewise, if experiments are understood as a practice for agenda setting and coalition building, as well as knowledge production and learning, more reasons could be cited for increasing their use and value.

[1] Information on the Climate Buffers program is available at In Dutch: Klimaatbuffers.
[2] Information on the Knowledge for Climate program is available at In Dutch: Kennis voor Klimaat. Other programs included Top Sector water ( and Innovation with Water (


Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a response, follow this link. To read responses already accepted, follow this link.


Thank you to the Dutch Knowledge for Climate program, which funded this research.


Ansell, C. K., and M. Bartenberger. 2016. Varieties of experimentalism. Ecological Economics 130:64-73.

Antikainen, R., K. Alhola, and T. Jääskeläinen. 2017. Experiments as a means towards sustainable societies – lessons learnt and future outlooks from a Finnish perspective. Journal of Cleaner Production 169:216-224.

Bai, X., B. Roberts, and J. Chen. 2010. Urban sustainability experiments in Asia: patterns and pathways. Environmental Science and Policy 13(4):312-325.

Baumgartner, F. R., and B. D. Jones. 1991. Agenda dynamics and policy subsystems. Journal of Politics 53(4):1044-1074.

Baumgartner, F. R., and B. D. Jones, editors. 2002. Policy dynamics. University of Chicago Press, Chicago, Illinois, USA.

Beem, B. 2007. Co-management from the top? The roles of policy entrepreneurs and distributive conflict in developing co-management arrangements. Marine Policy 31(4):540-549.

Berkhout, F., G. Verbong, A. J. Wieczorek, R. Raven, L. Lebel, and X. Bai. 2010. Sustainability experiments in Asia: innovations shaping alternative development pathways? Environmental Science and Policy 13(4):261-271.

Bernstein, S., and M. Hoffmann. 2018. The politics of decarbonization and the catalytic impact of subnational climate experiments. Policy Sciences 51(2):189-211.

Brodkin, E. Z., and A. Kaufman. 2000. Policy experiments and poverty politics. Social Service Review 74(4):507-532.

Broto, V. C., and H. Bulkeley. 2013. A survey of urban climate change experiments in 100 cities. Global Environmental Change 23(1):92-102.

Brouwer, S., and D. Huitema. 2018. Policy entrepreneurs and strategies for change. Regional Environmental Change 18(5):1259-1272.

Caniglia, G., N. Schäpke, D. J. Lang, D. J. Abson, C. Luederitz, A. Wiek, M. D. Laubichler, F. Gralla, and H. von Wehrden. 2017. Experiments and evidence in sustainability science: a typology. Journal of Cleaner Production 169:39-47.

Cohen, S. 1988. The effective public manager. Jossey-Bass, San Francisco, California, USA.

Dryzek, J. S. 1987. Rational ecology: environment and political economy. Basil Blackwell, New York, New York, USA.

Ettelt, S., N. Mays, and P. Allen. 2015. Policy experiments: investigating effectiveness or confirming direction? Evaluation 21(3):292-307.

Farrelly, M., and R. Brown. 2011. Rethinking urban water management: experimentation as a way forward? Global Environmental Change 21(2):721-732.

Fischer, F. 1995. Evaluating public policy. Nelson Hall, Chicago, Illinois, USA.

Greenberg, D., D. Linksz, and M. Mandell. 2003. Social experimentation and public policy making. Urban Institute Press, Washington, D.C., USA.

Haynes, L., O. Service, B. Goldacre, and D. Torgerson. 2012. Test, learn, adapt: developing public policy with randomised controlled trials. Cabinet Office, London, UK.

Hildén, M., A. Jordan, and D. Huitema. 2017. Special issue on experimentation for climate change solutions editorial: the search for climate change and sustainability solutions – the promise and the pitfalls of experimentation. Journal of Cleaner Production 169:1-7.

Hoffmann, M. J. 2011. Climate governance at the crossroads: experimenting with a global response after Kyoto. Oxford University Press, New York, New York, USA.

Huitema, D., A. Jordan, S. Munaretto, and M. Hildén. 2018. Policy experimentation: core concepts, political dynamics, governance and impacts. Policy Sciences 51(2):143-159.

Huitema, D., and S. Meijerink. 2010. Realizing water transitions: the role of policy entrepreneurs in water policy change. Ecology and Society 15(2):26.

Imperial, M. T. 2005. Using collaboration as a governance strategy: lessons from six watershed management programs. Administration and Society 37(3):281-320.

John, P. 2017. Changing how government works: the transformative potential of an experimental public management. Pages 476-494 in O. James, S. R. Jilke, and G. G. Van Ryzin, editors. Experiments in public management research: challenges and contributions. Cambridge University Press, Cambridge, UK.

Kingdon, J. 1984. Agendas, alternatives and public choices. Little Brown, Boston, Massachusetts, USA.

Kivimaa, P., M. Hildén, D. Huitema, A. Jordan, and J. Newig. 2017. Experiments in climate governance – a systematic review of research on energy and built environment transitions. Journal of Cleaner Production 169:17-29.

Laakso, S., A. Berg, and M. Annala. 2017. Dynamics of experimental governance: a meta-study of functions and uses of climate governance experiments. Journal of Cleaner Production 169:8-16.

Martí, I., and J. Mair. 2009. Bringing change into the lives of the poor: entrepreneurship outside traditional boundaries. Pages 92-119 in T. B. Lawrence, R. Suddaby, and B. Leca, editors. Institutional work: actors and agency in institutional studies of organizations. Cambridge University Press, Cambridge, UK.

Massey, E., D. Huitema, H. Garrelts, K. Grecksch, H. Mees, T. Rayner, S. Storbjörk, C. Termeer, and M. Winges. 2014. Handling adaptation policy choices in Sweden, Germany, the UK and the Netherlands. Journal of Water and Climate Change 6(1):9-24.

McFadgen, B., and D. Huitema. 2017. Are all experiments created equal? A framework for analysis of the learning potential of policy experiments in environmental governance. Journal of Environmental Planning and Management 60(10):1765-1784.

McFadgen, B., and D. Huitema. 2018. Experimentation at the interface of science and policy: a multi-case analysis of how policy experiments influence political decision-makers. Policy Sciences 51(2):161-187.

Meijerink S., and D. Huitema. 2009. Understanding and managing water policy transitions: a policy science perspective. Pages 23-36 in D. Huitema and S. Meijerink, editors. Water policy entrepreneurs: a research companion to water transitions around the globe. Edward Elgar, Cheltenham, UK.

Meijerink, S., and D. Huitema. 2010. Policy entrepreneurs and change strategies: lessons from sixteen case studies of water transitions around the globe. Ecology and Society 15(2):21.

Millo, Y., and J. Lezaun. 2006. Regulatory experiments: genetically modified crops and financial derivatives on trial. Science and Public Policy 33(3):179-190.

Mintrom, M. 2000. Policy entrepreneurs and school choice. Georgetown University Press, Washington, D.C., USA.

Mintrom, M., and J. Luetjens. 2017. Policy entrepreneurs and problem framing: the case of climate change. Environment and Planning C: Politics and Space 35(8):1362-1377.

Nair, S., and M. Howlett. 2015. Scaling up of policy experiments and pilots: a qualitative comparative analysis and lessons for the water sector. Water Resources Management 29(14):4945-4961.

Olsson, P., L. H. Gunderson, S. R. Carpenter, P. Ryan, L. Lebel, C. Folke, and C. S. Holling. 2006. Shooting the rapids: navigating transitions to adaptive governance of social-ecological systems. Ecology and Society 11(1):18.

Olsson, P., M.-L. Moore, F. R. Westley, and D. D. P. McCarthy. 2017. The concept of the Anthropocene as a game-changer: a new context for social innovation and transformations to sustainability. Ecology and Society 22(2):31.

Peters, B. G. 1998. The experimenting society and policy design. Pages 125-139 in W. N. Dunn, editor. The experimenting society: essays in honour of Donald T. Campbell. Transaction, New Brunswick, New Jersey, USA.

Pieraccini, M., and E. Cardwell. 2016. Towards deliberative and pragmatic co-management: a comparison between inshore fisheries authorities in England and Scotland. Environmental Politics 25(4):729-748.

Roberts, N. C. 1992. Public entrepreneurship and innovation. Review of Policy Research 11(1):55-74.

Sabatier, P. A. 1988. An advocacy coalition framework of policy change and the role of policy-oriented learning therein. Policy Sciences 21:129-168.

Sanderson, I. 2002. Evaluation, policy learning and evidence‐based policy making. Public Administration 80(1):1-22.

Schön, D. A., and M. Rein. 1994. Frame reflection: resolving intractable policy issues. Basic Books, New York, New York, USA.

Simons, A., and J.-P. Voß. 2018. The concept of instrument constituencies: accounting for dynamics and practices of knowing governance. Policy and Society 37(1):14-35.

Smith, A. 2007. Translating sustainabilities between green niches and socio-technical regimes. Technology Analysis and Strategic Management 19(4):427-450.

Taylor, A., C. Cocklin, R. Brown, and E. Wilson-Evered. 2011. An investigation of champion-driven leadership processes. Leadership Quarterly 22:412-433.

van Buuren, A., and D. Loorbach. 2009. Policy innovation in isolation? Conditions for policy renewal by transition arenas and pilot projects. Public Management Review 11(3):375-392.

van der Heijden, J. 2014. Experimentation in policy design: insights from the building sector. Policy Sciences 47(3):249-266.

van Popering-Verkerk, J., and A. van Buuren. 2017. Developing collaborative capacity in pilot projects: lessons from three Dutch flood risk management experiments. Journal of Cleaner Production 169:225-233.

Weiland, S., A. Bleicher, C. Polzin, F. Rauschmayer, and J. Rode. 2017. The nature of experiments for sustainability transformations: a search for common ground. Journal of Cleaner Production 169:30-38.

Young, O. R. 1991. Political leadership and regime formation: on the development of institutions in international society. International Organisation 45(3):281-308.

Address of Correspondent:
Belinda K. McFadgen
De Boelelaan 1087
1081 HV Amsterdam
The Netherlands
Jump to top
Table1  | Table2  | Figure1  | Appendix1  | Appendix2  | Appendix3  | Appendix4