Methods and procedures of transdisciplinary knowledge integration: empirical insights from four thematic synthesis processes

• Abstract
• Introduction
  • Transdisciplinary knowledge integration
  • Analytical framework for studying methods and procedures of knowledge integration
• National Research Programme 61 on Sustainable Water Management
• Methods
  • Data collection
  • Data analysis
• Results and Discussion
  • Key aspects of thematic synthesis 3
  • Similarities and differences among the four thematic syntheses
  • The leader’s role
  • Reflection on and adaptation of knowledge integration
  • Reflection on the analytical framework
• Conclusion
• Responses to this Article
• Acknowledgments
• Literature Cited

INTRODUCTION

Knowledge integration is considered a major challenge of inter- and transdisciplinary research (Bammer 2005, Bergmann et al. 2005, Truffer 2007, Klein 2008, McDonald et al. 2009, Jahn et al. 2012). It is a core feature of Swiss National Research Programmes (NRPs), the policy-oriented funding instrument of the Swiss National Science Foundation (SNSF 2012). In the context of NRPs, knowledge integration is defined as the combination of scientific results from individual projects with knowledge from different academic and nonacademic fields to generate practice-oriented solutions to current problems of national importance (SNSF 2015b). Such knowledge integration is used to generate synthesis reports published at the end of the programs, which typically run for 5 years. The content and format of the reports depend on the goals and the characteristics of the specific NRP (SNSF 2015b).

Though knowledge integration often determines the success or failure of such programs (Defila et al. 2006), there are no clearly specified methods and procedures that researchers can follow to generate syntheses. Here, we present an empirical study of the methods and procedures of knowledge integration that were applied within four thematic syntheses of the NRP 61 on Sustainable Water Management in Switzerland. We developed an analytical framework to map different methods and procedures and applied it to explore knowledge integration within the four thematic syntheses. On this basis, we discuss the variety of methods and procedures used at different stages of the four syntheses, the influence of the leader’s (or core team’s) role in the design of knowledge integration, and the need for reflection on and adaption of knowledge integration during the course of the process. Finally, we draw some conclusions for future synthesis processes.

Transdisciplinary knowledge integration

We use the term transdisciplinarity to refer to research that (1) tackles real life problems, (2) addresses the complexity of these problems by involving a variety of actors from science and practice and accounting for the diversity of their perspectives, and (3) creates knowledge that is solution-oriented, socially robust, and transferable to both scientific and societal practice (Pohl and Hirsch Hadorn 2007, Lang et al. 2012). We consider transdisciplinary research as a comprehensive, multiperspective, problem- and solution-oriented approach that transcends disciplinary boundaries and bridges science and practice (Pohl 2011).

Transdisciplinary knowledge coproduction refers to the entire process of joint knowledge production between experts from different disciplines, sectors, and decision levels, including joint problem formulation, knowledge generation, application in both scientific and societal practice, and mutual quality control of scientific rigor, social robustness, and practical relevance (Polk 2015). Transdisciplinary knowledge coproduction encompasses synthesis through transdisciplinary knowledge integration. In the context of large research programs, the latter refers to the process of taking stock of individual project results and generating new knowledge by establishing novel (i.e., previously unrecognized) connections between them (Jahn et al. 2012, Specht et al. 2015).

Transdisciplinary knowledge integration is necessarily influenced by power differentials within and between different disciplines as well as between science and practice as comprehensively discussed by MacMynowski (2007). Although we recognize the importance of an analysis of power issues for understanding the complexity of transdisciplinary knowledge integration, our prior agreement with the key actors in the four synthesis processes examined here precluded such analysis.

Analytical framework for studying methods and procedures of knowledge integration

Our analytical framework combines two aspects of integration that had previously been analyzed and structured in the context of inter- and transdisciplinary research. These aspects are the procedures by which integration is organized within a team, and the methods the team uses for knowledge integration.

Procedures of knowledge integration

An empirical study of how interdisciplinary teams integrate disparate bodies of knowledge differentiated four ideal types of procedures (originally termed “socio-cognitive frameworks”): common group learning, modeling, negotiation among experts, and integration by a leader (Rossini and Porter 1979). Here, we consider modeling as one of the methods of integration. We then distinguish three types of procedures for reaching synthesis (Fig. 1). In common group learning, integration of scientific results takes place within the group as a whole. Subquestions are distributed among the members of a larger group on the basis of their specific expertise; they are addressed by the corresponding members, discussed by the entire group, and then related to the overall question. The process repeats iteratively until the group decides that the overall question has been answered adequately. The recursive nature of this process enables mutual learning among the group members and gradual shifting of responsibility from individual experts to the entire group (Pohl and Hirsch Hadorn 2007). In negotiation among experts, the scientific results from individual research projects are combined through bilateral discussion among experts. As in common group learning, subquestions are divided among the members of a larger group according to their specific expertise. Subquestions are then addressed individually, followed by bilateral exchange among experts, mutual adaptation of subanswers, and final negotiation among experts. Contrary to common group learning, the responsibility for the subanswers and their adequate representation in the synthesis remains with each individual expert (Pohl and Hirsch Hadorn 2007). In integration by a leader, scientific results from individual projects are delivered to a leader, who may be an individual or a subgroup. The leader is responsible for linking and relating the results and functions as the sole integrator, interacting individually with each expert to understand and assimilate the various results. In contrast to common group learning and negotiation among experts, experts do not interact with each other (Rossini and Porter 1979).

Methods of knowledge integration

Six categories of integration methods have been differentiated according to their function in the process of knowledge integration (Bergmann et al. 2012): (1) definition of concepts and theoretical frameworks, in which a shared understanding is created across disciplinary boundaries; (2) formulation of research questions and hypotheses, in which societal problems are translated into research objects; (3) development of integrative methods, in which existing inter-, trans-, and intradisciplinary methods are reviewed and adapted; (4) design of assessment procedures, in which multiple criteria from different perspectives are merged into new procedures; (5) model development and application, in which theoretical and empirical descriptions of a particular part of reality are linked; and (6) creation of boundary objects, in which mutual understanding across cognitive and normative boundaries is enhanced.

Analytical framework for studying methods and procedures

Combining the three types of integration procedures with the six categories of integration methods yields a 3 × 6 matrix. This provides a structured basis for the analysis of methods and procedures used at different stages in each of the four thematic synthesis processes. Each stage can be characterized by a particular combination of integration methods and procedures. The procedures of common group learning, negotiation among experts, and integration by a leader can be combined in sequence to integrate different bodies of knowledge at a particular stage of the process.

NATIONAL RESEARCH PROGRAMME 61 ON SUSTAINABLE WATER MANAGEMENT

NRP 61 on Sustainable Water Management was intended to develop scientific foundations, methods, strategies, and solutions to meet future challenges in the field of sustainable water management in Switzerland (SNSF 2010). A transdisciplinary research approach was applied to analyze societal problems and generate socially robust solutions.

In 2008, a steering committee for NRP 61 was formed, with six experts from different fields (aquatic ecology, geophysics, hydrology, meteorology, environmental economics, and environmental engineering) and countries (Switzerland, Germany, and Austria). The steering committee reviewed the submitted pre- and full proposals for NRP 61 research projects and selected the projects to be recommended to the Swiss National Science Foundation (SNSF) National Research Council (Division IV, Presidial Board) for funding (SNSF 2008) on the basis of external and internal reviews. The steering committee was supported by an advisory board (constituted in 2010) with 10 national actors from different sectors (water resources, water supply, wastewater treatment, hydropower, agriculture, land-use planning) and decision levels (cantonal, national). Overall program management was provided by the SNSF (Table 1).

Sixteen NRP 61 research projects were initiated in 2010 with initial funding for 3 years (SNSF 2010). Subsequently, additional funding was acquired (partly through the NRP 61 program but also through other sources) to extend some projects for a fourth year. More than 88 researchers and 38 doctoral candidates from different disciplines in the natural and social sciences and engineering were involved in the projects. Projects were led mainly by researchers from universities and research institutes; some projects included coinvestigators from nongovernmental organizations (NGOs) and consultancy firms.

Program synthesis was, from the outset, conceived as a joint endeavor of the 16 NRP 61 research projects. Preliminary ideas for the synthesis were presented by the steering committee in the first progress report meeting with NRP 61 researchers in March 2011. Based on feedback from the researchers, the steering committee organized a synthesis workshop to identify synthesis topics, key questions, and target audiences of potential syntheses with researchers and members of the advisory board. Building on the results of this workshop, the steering committee decided in 2011 to produce four thematic syntheses (TSs) mainly for a technical audience (i.e., relevant local, cantonal, and federal authorities; trade associations; NGOs; and private companies) and one overall synthesis (OS) for the general public.

Following this decision, the steering committee recruited NRP 61 researchers interested in leading transdisciplinary knowledge integration. Subsequent discussions between the committee and the interested researchers were conducted to refine the four TS themes and form interdisciplinary core teams composed of three to four researchers who would be responsible for conceptualizing and implementing knowledge integration. The steering committee invited the core teams to develop proposals for the TSs and submit them to the SNSF National Research Council for approval. Concurrently, the steering committee developed the concept for the overall synthesis in collaboration with external consultants.

The proposal for the third TS (TS-3) originally included a plan for concurrent analysis of and reflection on the synthesis process. At the request of the SNSF, this component was split out from the TS-3 proposal and submitted as a separate proposal in which the analysis was extended to include all four thematic syntheses. This proposal was submitted in November 2012 and approved in January 2013 by the SNSF National Research Council.

METHODS

Data collection

We used qualitative methods of empirical social research (Denzin and Lincoln 2005) to study methods and procedures of knowledge integration. Relevant written documents were analyzed, including proposals, reports, articles, and presentations (see also Appendix 1). Semistructured interviews incorporating questions on knowledge integration defined by Bammer (2008; see also Appendix 2) were conducted by the lead author of this paper (and of TS-3): Three interviews were conducted at the beginning of the synthesis process with lead authors of TS-1, TS-2, and TS-4, and seven interviews were conducted at the end of the process with lead authors of TS-1, TS-2, and TS-4 (three interviews) and second authors of all four TSs (four interviews). In addition, participant observation in NRP 61 program and coordination meetings was used by all of the authors of this paper to study the various roles that leaders assumed in the four syntheses processes.

Data analysis

Methods and procedures of knowledge integration were analyzed using the analytical framework described previously. The methods and procedures used at different stages within the four TSs were mapped, and the societal and scientific actors involved at each stage were identified. By consistently mapping methods, procedures, and actors for each TS, we were able to compare knowledge integration across the four TSs and to examine similarities and differences in the application of methods and procedures and in the involvement of societal and scientific actors. To validate our qualitative analysis empirically, we discussed the mapped integration strategies with the lead authors and coauthors of the four TSs and refined our interpretations on the basis of this discussion.

RESULTS AND DISCUSSION

Given our direct involvement in TS-3, we use that synthesis process to illustrate the application of our analytical framework for the mapping of methods, procedures, and actors involved at different stages in each TS. Each stage in the synthesis process can be characterized in terms of the corresponding category of integration methods and the type(s) of integration procedures (Fig. 2). In TS-3, integration by leader was ubiquitous, occurring at all synthesis stages. The core team served as the leader and included three researchers from geoecology, environmental engineering, and hydrogeology. At various stages in the integration process, the core team interacted bilaterally with different actors, including the steering committee, advisory board, and scientific and practice experts. The scientific experts included seven scientists with backgrounds in environmental sciences, decision analysis, hydrogeology, hydrology, and limnology who were directly engaged in NRP 61 projects. The practice experts comprised 16 practitioners from different sectors (water supply, wastewater treatment) and decision levels (local, cantonal, and national levels). At some stages, common group learning was applied, which mainly involved the leader and scientific experts, but also at one stage, the leader, steering committee, and advisory board.

Of the six categories of integration methods, four were used: formulation of research questions and hypotheses, development of integrative methods, design of assessment procedures, and model development and application. Definition of concepts and theoretical frameworks and creation of boundary objects were not used.

Key aspects of thematic synthesis 3

In the TS-3 synthesis process, crucial integration occurred during preparation of the proposal by the core team. Together with scientific experts, the core team formulated three key research questions (Table 2) and identified the need for an integrative method to address these questions. Using common group learning and integration by a leader, and following Giupponi (2007), the core team combined key elements of the driver, pressure, state, impact, response (DPSIR) model for systems analysis (Smeets and Weterings 1999) and multicriteria decision analysis (MCDA; Eisenführ et al. 2010). Development of an integrative method provided a basis for later tasks of tackling, structuring, and systematizing the very heterogeneous results from the individual research projects and relating them to each other in a coherent and consistent way.

Following submission and approval of the TS-3 proposal, the integrative method was used both to combine results from the seven projects included in TS-3 and to address the specifc research questions (Table 2). The DPSIR component was applied to address questions 1 and 3 (Table 2) and to prioritize the current and future challenges to Swiss urban water and wastewater management. The MCDA component was used to formulate targets (i.e., addressing question 2) as well as to assess the potential of different possible courses of action to achieve those targets (i.e., addressing question 3). Target formulation was particularly important for combining results on infrastructure performance with expert knowledge on the availability and quality of surface and groundwater resources.

The final stages in this transdisciplinary integration were validation and diffusion of synthesis results. The results were validated by consultation with scientific and practice experts, the advisory board, and the steering committee, with subsequent revision by the core team. The close collaboration among the core team and the scientific and practice experts throughout the synthesis process and especially the revision of the TS-3 report by these experts were crucial for ensuring not only the reliability and the validity of the conclusions drawn in TS-3, but also their relevance for science and practice. The key findings and conclusions from TS-3 were presented by the core team at the final conference of the NRP 61. The TS-3 report (as well as the other synthesis reports) are available online through the NRP 61 website; they were summarized in practice-oriented articles that appeared in a Swiss trade journal (see Table 2).

Similarities and differences among the four thematic syntheses

Using TS-3 as a reference case, the mapping of the strategies for the other TSs (see Appendix 3) clearly illustrates the similarities and differences among the four syntheses processes. As in TS-3, integration by a leader was predominant in all TSs. Common group learning was also applied, but less frequently in the other TSs than in TS-3. None of the TSs used negotiation among experts. The other TSs also used fewer integration methods than did TS-3 (3 vs. 4, respectively). TS-1 and TS-2 used the same set of integration methods, which included creation of boundary objects (not used in TS-3). TS-4 was the only synthesis process that applied definition of concepts and theoretical frameworks as an integration method. The only method common to all four TSs was formulation of research questions and hypotheses (Table 3).

All four TSs formulated research questions, both as a means to facilitate a shared understanding and to guide further synthesis activities. All of the TSs except TS-4 addressed their research questions using model development and application. The TS-1 core team developed system models to link and relate scientific results on the impacts of climate and socioeconomic change on water resources and water uses. The TS-2 core team performed qualitative cross impact analysis (Serdar Asan and Asan 2007) to elucidate the characteristic role of different water user demands (e.g., hydropower, water protection, flood protection, agriculture, water supply, and tourism) in relation to all other demands and to identify those user demands that shape (or might shape) the development of Swiss water resources and aquatic ecosystems in the future. The method enabled the TS-2 core team to describe systematically the interrelationships between different water user demands, assess the strength of these interrelationships, and identify potential conflicts and synergies that result (or might result) from overlapping user demands.

Creation of boundary objects was applied in TS-1 and TS-2. In TS-1, two types of boundary objects were created. System models were used to visualize key elements of the system, determine the interrelationships among these elements, analyze the underlying causal relationships, and identify knowledge gaps and uncertainties in predictions on potential climate and socioeconomic change impacts. Treemaps were developed as a second type of boundary object to display data on the current state of water resources and water uses in Switzerland. Both types of boundary objects served to structure, systematize, and analyze the very fragmented data from different academic and nonacademic sources and to identify inconsistencies in the data.

Two types of boundary objects were also created in TS-2. Thematic maps (e.g., impact of hydropower on Swiss water resources, proportion of different land uses over groundwater areas) were used to visualize potential conflicts and synergies resulting from overlapping user demands. Cross-impact matrices were generated to summarize potential conflicts and synergies (see Box 1). Use of the boundary objects in TS-1 and TS-2 stimulated iterative discussion between the core team and a wide range of experts from science and practice and facilitated presentation of the synthesis results in the TS-1 and TS-2 final reports.

Design of assessment procedures was used in TS-3 as described previously and in TS-4. The TS-4 core team collaborated with scientific experts from different NRP 61 projects to develop an integrative assessment procedure for sustainable water governance in Switzerland. The assessment procedure provided a methodical framework for: integrating a variety of sustainability criteria from different NRP 61 projects by decoupling them from their particular contexts and recombining them in a new procedure; assessing cases studies on water management and water governance against these sustainability criteria, and; identifying necessary improvements to achieve sustainable water governance in Switzerland.

TS-4 was the only synthesis process in which definition of concepts and theoretical frameworks was applied. At the beginning of the synthesis process, the TS-4 core team engaged in an intensive process of clarifying key concepts such as “water management,” “water governance,” and “sustainable water governance.” The conceptual clarification was crucial for the subsequent development of a theoretical framework on multiple-loop learning (Armitage et al. 2008). With this theoretical framework, four different types of approaches (or learning loops) were differentiated through which water governance in Switzerland could be improved. Different case studies on water management and water governance provided by several NRP 61 projects were selected and systematized according to this typology.

The empirical study illustrates the diversity of approaches that have been applied in four TSs addressing various aspects of sustainable water management. This diversity is consistent with the “principle of variance,” which holds that knowledge integration is influenced by the objectives to be achieved, the research questions to be answered, and the scientific results to be integrated in a particular synthesis. It is also influenced by the participants who are involved in the process, their disciplinary and professional backgrounds, and their institutional anchoring (Klein 2012).

One common feature of the four synthesis processes was the absence of negotiation among experts. This absence can be explained by the fact that research proposals were initially designed before the four synthesis topics were defined. Effective integration using this procedure would have required that experts refine their analysis at the boundary between their research project and other projects to include substantively the findings of other experts’ analyses in their own analysis (Rossini and Porter 1979). Although negotiation at the boundary between different projects did not take place within NRP 61, it could be an important integration procedure in other settings in which synthesis is taken into account in the design of research proposals.

Another common feature was the predominance of integration by a leader. This justifies a detailed examination of the role of the leader in knowledge integration.

The leader’s role

Leaders can perceive and exercise their roles and responsibilities in a variety of ways. Drawing on Wieser et al. (2014), we differentiate four different types of roles:

1. Collaborators, who establish close collaboration with actors, particularly those they consider to be the target audiences of their research;
2. Facilitators, who support knowledge exchange between different scientific and societal actors (e.g., by organizing social spaces in which actors can meet and interact directly);
3. Scholars, who maintain a certain distance to the field under study and analyze it academically without involving and engaging with societal actors, and;
4. Advocates, who actively try to bring about a social change and tend to generate practice-relevant results that benefit those who seek to contribute to that change.

The leaders of the four TSs adopted different roles throughout the process of knowledge integration. Leaders in TS-1, TS-2, and TS-3 acted as collaborators in at least some stages in the process. For the most part, these leaders engaged in bilateral and iterative discussions with other actors. Leaders in TS-3 and TS-4 assumed the role of a facilitator at some stages, fostering exchange among scientists from different disciplines on, for example, sustainability targets for urban water and wastewater management (TS-3) or sustainability criteria for water governance (TS-4). At some stages in the process, leaders who acted as collaborators (TS-1) or facilitators (TS-4) shifted their role and acted as scholars, taking the initiative to analyze results from their own perspective and to bring in established disciplinary concepts and theories. Leaders in TS-2 acted at some stages as advocates. They sought to convince other societal or scientific actors of the need for a change in Swiss water management. This role was contested by some actors within NRP 61.

Reflection on and adaptation of knowledge integration

The dual role of the lead author of this paper (i.e., leading knowledge integration for TS-3 and, concurrently, studying methods and procedures of knowledge integration in all TSs) enabled reflection at four different levels (see also Tress et al. 2006). At the personal level, it triggered reflection on the TS-3 synthesis process, specifically how to design the synthesis process and which actors to involve at which stage, in which roles, with which responsibilities, supported by which methods and procedures. It also involved a regular reflection on what went well (or less well) in the synthesis process to improve process design (for further discussion, see Hoffmann et al. 2017) and thus allowed for a deeper understanding of the TS-3 synthesis process compared to all other TS processes.

At the TS-3 project level, it enabled reflection, particularly among the core team and the scientific experts, on the appropriate combination of methods and procedures for the integration of results from different scientific disciplines and different practical fields to answer the common questions (see Table 2). This regular reflection resulted in some adaptations of methods and procedures in the course of the TS-3 synthesis process.

Feedback to other synthesis projects constituted a third level of reflection. The overlapping time frames of the four TSs and the associated research on methods and procedures of knowledge integration enabled a process of mutual learning among the core teams of all TSs (Hoffmann 2016). The visualization of the knowledge integration strategies (which resulted from the application of our framework) facilitated a joint reflection by all core teams on the advantages and disadvantages of different combinations of methods and procedures and triggered, as in the case of TS-3, some adaptations in the ongoing processes (Hoffmann 2016).

At the program level, the dual role of the lead author stimulated reflection on transdisciplinary integration within large research programs (Hoffmann 2016, Hoffmann et al. 2017). The SNSF decided, on the basis of our empirical findings, to organize two workshops. The first workshop on transdisciplinary knowledge integration within large research programs took place in November 2014 (i.e., after the completion of our study). It allowed us to discuss our findings with steering committee presidents, program coordinators, and implementation officers of past and current NRPs, including NRP 61, as well as with other representatives of the SNSF, and reflect on challenges of transdisciplinary knowledge integration within large research programs (Hoffmann et al. 2017). The second workshop in February 2015 aimed at the ex-post self-evaluation of the four synthesis processes and their overall framework conditions. The workshop was facilitated by the authors of this paper and involved the four core teams, several scientific experts, and the NRP 61 program management. For details regarding the results of this ex-post self-evaluation see Hoffmann et al. (2017).

Reflection on the analytical framework

The analytical framework applied here highlights the potential benefit of structuring integration processes in a transparent manner from the outset. Although the framework was developed in the course of NRP 61, its relevance is not limited to this specific program. We suggest that the framework could be applied fruitfully in new integration processes: to define and sequence different synthesis stages, to assess the adequate combination of methods and procedures for each stage, and to identify the relevant actors and define their roles and responsibilities in the process. Such transparency from the outset is essential to clarify the mutual expectations of those involved in the process and thus to prevent participants’ disappointments (Barreteau et al. 2010). As a result of the ex-post self-evaluation of the four synthesis processes, the SNSF plans to encourage future leaders of NRP synthesis processes to use our analytical framework to ensure such transparency.

The framework could also be applied to review progress and track changes in the process, making formative evaluation throughout the synthesis project more feasible (Bergmann and Schramm 2008). It could also be used to assess the outcome of the synthesis project (i.e., the uptake of results by the target audiences of the four synthesis reports) as a function of the synthesis process. The uptake of results generally takes time and depends, among other factors, on the extent to which the output (i.e., the synthesis report) is made accessible to the target audiences and is promoted by all actors involved in the process (Hering et al. 2012). The SNSF intends to conduct an impact analysis of NRPs to assess whether NRPs are achieving their goal of generating knowledge tailored to the specific needs of the target audiences (SNSF 2015a), using NRP 61 as a reference case. In the context of this impact analysis, our framework could be used to link output, outcome, and impact of the four synthesis projects with the underlying process of transdisciplinary knowledge integration. In this study, however, we explicitly excluded such linkage by prior agreement with the participating core team members of the four synthesis projects.

CONCLUSION

Transdisciplinary research lacks empirical studies on knowledge integration. Thus, we developed an analytical framework for studying methods and procedures of knowledge integration and applied the framework to explore knowledge integration within four TSs. Any conscious attempt to achieve synthesis should involve a strategy for integrating different bodies of knowledge by defining the methods and procedures to be applied and by identifying the actors to be involved throughout the process. Such a strategy would allow structured integration across all synthesis stages and coordinated collaboration among all actors. Though the strategy might need to be adjusted throughout the process, it provides a common orientation for all actors and a basis for reviewing progress and reshaping subsequent synthesis stages if necessary (Lang et al. 2012). The analytical framework proposed here supports the design of such a strategy.

Any conscious attempt to achieve synthesis should also include a critical reflection on the different roles and responsibilities researchers (and practitioners) assume in the process of knowledge integration and the conflicts this might bring about (Truffer 2007, Pohl et al. 2010, Lang et al. 2012). We agree with Jahn et al. (2012) that these roles and responsibilities “are neither sufficiently discussed in the literature nor adequately reflected in research practice.” We therefore suggest exploring roles, responsibilities, tasks, and competencies of researchers (and practitioners) leading processes of transdisciplinary knowledge integration in a more systematic way by conducting further research on leadership in transdisciplinary integration endeavors.

LITERATURE CITED

Armitage, D., M. Marschke, and R. Plummer. 2008. Adaptive co-management and the paradox of learning. Global Environmental Change 18(1):86-98. http://dx.doi.org/10.1016/j.gloenvcha.2007.07.002

Bammer, G. 2005. Integration and implementation sciences: building a new spezialization. Ecology and Society 10(2):6. http://dx.doi.org/10.5751/es-01360-100206

Bammer, G. 2008. Enhancing research collaborations: three key management challenges. Research Policy 37(5):875-887. http://dx.doi.org/10.1016/j.respol.2008.03.004

Barreteau, O., P. W. G. Bots, and K. A. Daniell. 2010. A framework for clarifying “participation” in participatory research to prevent its rejection for the wrong reasons. Ecology and Society 15(2):1. [online] URL: http://www.ecologyandsociety.org/vol15/iss2/art1/

Bergmann, M., B. Brohmann, E. Hoffmann, M. C. Loibl, R. Rehaag, E. Schramm, and J.-P. Voß. 2005. Quality criteria of transdisciplinary research: a guide for the formative evaluation of research projects. Institute for Social-Ecological Research (ISOE), Frankfurt, Germany. [online] URL: http://www.isoe.de/uploads/media/st-13-isoe-2005-en.pdf

Bergmann, M., T. Jahn, T. Knobloch, W. Krohn, C. Pohl, and E. Schramm. 2012. Methods for transdisciplinary research: a primer for practice. Campus Verlag, Frankfurt, Germany.

Bergmann, M., and E. Schramm 2008. Grenzüberschreitung und Integration: die formative Evaluation transdisziplinärer Forschung und ihre Kriterien. Pages 149-175 in M. Bergmann and E. Schramm, editors. Transdisziplinäre Forschung: Integrative Forschungsprozesse verstehen und bewerten. Campus Verlag, Frankfurt, Germany.

Björnsen Gurung, A., and M. Stähli. 2014. Wasserressourcen der Schweiz: Dargebot und Nutzung - heute und morgen. NFP 61, Thematic Synthesis 1. Schweizerischen Nationalfonds zur Förderung der wissenschaftlichen Forschung, Bern, Switzerland. [online] URL: http://www.nfp61.ch/SiteCollectionDocuments/nfp61_thematische_synthese_1_d.pdf

Defila, R., A. Di Giulio, and M. Scheuermann. 2006. Forschungsverbundmanagement: Handbuch für die Gestaltung inter- und transdisziplinärer Projekte. vdf Hochschulverlag, Zurich, Switzerland.

Denzin, N. K., and Y. S. Lincoln, editors. 2005. The Sage handbook of qualitative research. Sage, Thousand Oaks, California, USA.

Eisenführ, F., M. Weber, and T. Langer. 2010. Rational decision making. Springer, Berlin, Germany.

Giupponi, C. 2007. Decision support systems for implementing the European Water Framework Directive: the MULINO approach. Environmental Modelling and Software 22(2):248-258. http://dx.doi.org/10.1016/j.envsoft.2005.07.024

Hering, J. G., S. Hoffmann, R. Meierhofer, M. Schmid, and A. J. Peter. 2012. Assessing the societal benefits of applied research and expert consulting in water science and technology. Gaia - Ecological Perspectives for Science and Society 21(2):95-101.

Hoffmann, S. 2016. Transdisciplinary knowledge integration within large research programs. Gaia - Ecological Perspectives for Science and Society 25(3):201-203.

Hoffmann, S., D. Hunkeler, and M. Maurer. 2014. Nachhaltige Wasserversorgung und Abwasserentsorgung in der Schweiz: Herausforderungen und Handlungsoptionen. NFP 61, Thematic Synthesis 3. Schweizerischen Nationalfonds zur Förderung der wissenschaftlichen Forschung, Bern, Switzerland. [online] URL: http://www.nfp61.ch/SiteCollectionDocuments/nfp61_thematische_synthese_3_d.pdf

Hoffmann, S., C. Pohl, and J. G. Hering. 2017. Exploring transdisciplinary integration within a large research program: empirical lessons from four thematic synthesis processes. Research Policy, in press. http://dx.doi.org/10.1016/j.respol.2017.01.004

Jahn, T., M. Bergmann, and F. Keil. 2012. Transdisciplinarity: between mainstreaming and marginalization. Ecological Economics 79:1-10. http://dx.doi.org/10.1016/j.ecolecon.2012.04.017

Jordi, B. 2014. Nachhaltige Siedlungswasserwirtschaft: Thematische Synthese 3: Ziele und Herausforderungen. Aqua and Gas 11:30-34.

Jörg-Hess, S., A. Björnsen Gurung, and M. Stähli. 2014. Wasserressourcen der Schweiz: Thematische Synthese 1: Dargebot und Nutzung - heute und morgen. Aqua and Gas 11:18-22.

Klein, J. T. 2008. Evaluation of interdisciplinary and transdisciplinary research: a literature review. American Journal of Preventive Medicine 35(2):S116-S123. http://dx.doi.org/10.1016/j.amepre.2008.05.010

Klein, J. T. 2012. Research integration: a comparative knowledge base. Pages 283-298 in A. F. Repko, W. H. Newell, and R. Szostak, editors. Case studies in interdisciplinary research. Sage, Thousand Oaks, California, USA. http://dx.doi.org/10.4135/9781483349541.n10

Lang, D. J., A. Wiek, M. Bergmann, M. Stauffacher, P. Martens, P. Moll, M. Swilling, and C. J. Thomas. 2012. Transdisciplinary research in sustainability science: practice, principles, and challenges. Sustainability Science 7(S1):25-43. http://dx.doi.org/10.1007/s11625-011-0149-x

Lanz, K., J. Fuhrer, W. Haeberli, and C. Stamm. 2014a. Steigender Nutzungsdruck im Wasserschloss. Thematische Synthese 2: Wasserressourcen-Bewirtschaftung unter steigendem Nutzungsdruck. Aqua and Gas 11:24-29.

Lanz, K., E. Rahn, R. Siber, and C. Stamm 2014b. Bewirtschaftung der Wasserressourcen unter steigendem Nutzungsdruck. Thematic Synthesis 2. Schweizerischen Nationalfonds zur Förderung der wissenschaftlichen Forschung, Bern, Switzerland. [online] URL: http://www.nfp61.ch/SiteCollectionDocuments/nfp61_thematische_synthese_2_d.pdf

Leitungsgruppe NFP 61. 2014. Nachhaltige Wassernutzung in der Schweiz.Gesamtsynthese: NFP 61 weist Wege in die Zukunft. Aqua and Gas 11:10-16.

Leitungsgruppe NFP 61. 2015. Nachhaltige Wassernutzung in der Schweiz: NFP 61 weist Wege in die Zukunft. Gesamtsynthese des Nationalen Forschungsprogramms NFP 61. Vdf Hochschulverlag ETH Zurich, Zurich, Switzerland. [online] URL: http://vdf.ch/index.php?route=product/product&product_id=1754

MacMynowski, D. P. 2007. Pausing at the brink of interdisciplinarity: power and knowledge at the meeting of social and biophysical science. Ecology and Society 12(1):20. http://dx.doi.org/10.5751/es-02009-120120

McDonald, D., G. Bammer, and P. Deane. 2009. Research integration using dialogue methods. ANU Press, Canberra, Australia. [online] URL: http://press.anu.edu.au?p=60381

Pohl, C. 2011. What is progress in transdisciplinary research? Futures 43:618-626. http://dx.doi.org/10.1016/j.futures.2011.03.001

Pohl, C., and G. Hirsch Hadorn. 2007. Principles for designing transdisciplinary research. Oekom Verlag, Munich, Germany.

Pohl, C., S. Rist, A. Zimmermann, P. Fry, G. S. Gurung, F. Schneider, C. I. Speranza, B. Kiteme, S. Boillat, E. Serrano, G. Hirsch Hadorn, and U. Wiesmann. 2010. Researchers’ roles in knowledge co-production: experience from sustainability research in Kenya, Switzerland, Bolivia and Nepal. Science and Public Policy 37(4):267-281. http://dx.doi.org/10.3152/030234210x496628

Polk, M. 2015. Transdisciplinary co-production: designing and testing a transdisciplinary research framework for societal problem solving. Futures 65:110-122. http://dx.doi.org/10.1016/j.futures.2014.11.001

Rossini, F. A., and A. L. Porter. 1979. Frameworks for integrating interdisciplinary research. Research Policy 8(1):70-79. http://dx.doi.org/10.1016/0048-7333(79)90030-1

Schmid, F., F. Walter, F. Schneider, and S. Rist. 2014a. Nachhaltige Wassergouvernanz: Herausforderungen und Wege in die Zukunft. NFP 61, Thematic Synthesis 4. Schweizerischen Nationalfonds zur Förderung der wissenschaftlichen Forschung, Bern, Switzerland. [online] URL: http://www.nfp61.ch/SiteCollectionDocuments/nfp61_thematische_synthese_4_d.pdf

Schmid, F., F. Walter, F. Schneider, and S. Rist. 2014b. Wege zur nachhaltigenWassergouvernanz. Thematische Synthese 4: Notwendige Anpassungsprozesse in der Schweizer Wasserwirtschaft. Aqua and Gas 11:36-40.

Serdar Asan, S., and U. Asan. 2007. Qualitative cross-impact analysis with time consideration. Technological Forecasting and Social Change 74(5):627-644. http://dx.doi.org/10.1016/j.techfore.2006.05.011

Smeets, E., and R. Weterings. 1999. Environmental indicators: typology and overview. Technical report 25/1999. European Environment Agency, Copenhagen, Denmark. [online] URL: http://www.eea.europa.eu/publications/TEC25/at_download/file

Swiss National Science Foundation (SNSF). 2008. Sustainable water management: implementation plan of the National Research Programme NRP 61. Swiss National Science Foundation, Bern, Switzerland. [online] URL: http://www.nrp61.ch/SiteCollectionDocuments/nfp61_ausfuehrungsplan_e.pdf

Swiss National Science Foundation (SNSF). 2010. Sustainable water management: portrait of the National Research Programme NRP 61. Swiss National Science Foundation, Bern, Switzerland. [online] URL: http://www.nrp61.ch/SiteCollectionDocuments/nfp61_ausfuehrungsplan_e.pdf

Swiss National Science Foundation (SNSF). 2012. Final products and programme syntheses in National Research Programmes (NRPs). Swiss National Science Foundation, Bern, Switzerland.

Swiss National Science Foundation (SNSF). 2015a. Multi-year programme 2017–2020: planning document for the Federal Authorities. Swiss National Science Foundation, Bern, Switzerland. [online] URL: http://www.snf.ch/SiteCollectionDocuments/mehrjahresprogramm_2017_2020_e.pdf

Swiss National Science Foundation (SNSF). 2015b. National research programmes (NRPs): guidelines. Swiss National Science Foundation, Bern, Switzerland.

Specht, A., I. J. Gordon, R. H. Groves, H. Lambers, and S. R. Phinn. 2015. Catalysing transdisciplinary synthesis in ecosystem science and management. Science of the Total Environment 534:1-3. http://dx.doi.org/10.1016/j.scitotenv.2015.06.044

Star, S. L., and J. R. Griesemer. 1989. Institutional ecology, ‘translations’ and boundary objects: amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907–39. Social Studies of Science 19(3):87-420.

Tress, B., G. Tress, and G. Fry. 2006. Defining concepts and the process of knowledge production in integrative research. Pages 13-26 in B. Tress, G. Tress, G. Fry, and P. Opdam, editors. From landscape research to landscape planning: aspects of integration, education and application. Springer, Dordrecht, The Netherlands.

Truffer, B. 2007. Knowledge integration in transdisciplinary research projects – the importance of reflexive interface management. Gaia - Ecological Perspectives for Science and Society 16(1):41-45.

Wieser, B., A. Brechelmacher, and G. Schendl. 2014. Identitäten und Rollen in inter- und transdisziplinärer Forschung und Lehre finden. Pages 151-166 in G. Dressel, W. Berger, K. Heimerl, and V. Winiwarter, editors. Interdisziplinär und transdisziplinär forschen: praktiken und methoden. transcript Verlag, Bielefeld, Germany. http://dx.doi.org/10.14361/transcript.9783839424841.151