Research that aims to support sustainability transformations has been challenged by the wickedness of sustainability problems in social-ecological systems (Folke et al. 2005, Termeer et al. 2013). Wickedness includes substantial amounts of complexity (Ostrom 2009), uncertainty (Pahl-Wostl et al. 2007), and normativity (Leach et al. 2010). Exclusively academic approaches to these problems often fail to significantly contribute to sustainability transformations (Pahl-Wostl 2002, Huutoniemi 2014). Hence, participatory research approaches are increasingly used in sustainability research, based on the expectation that they are socially robust and comprehensively involve scientific and societal actors in knowledge codesign and coproduction (Lang et al. 2012, Polk 2014, Moser 2016, Schneider and Buser 2018). They aim to address societally relevant questions, deliver results that are based on multiple sources of expertise, and take different actors’ perspectives and values into account. Hence, their implementation is considered to be more likely and they are meant to produce salient, credible, and legitimate results (Cash et al. 2002). However, involving societal actors in the scientific knowledge production process does not per se lead to societal impact (Reed 2008, Kaufmann-Hayoz et al. 2016).
Consequently, literature increasingly focuses on impacts of participatory research, and scholars have started to conceptually define societal impact (Godin and Dore 2005, Bornmann 2013, Miettinen et al. 2015). Since about 1990, a broad variety of frameworks have been developed to illustrate how impact can take place. Among the more popular models we find the payback model of Buxton and Hanney (1996) and the ESRC model (ESRC 2011). These models usually consist of a series of stages such as inputs, research processes, direct outputs, and further outcomes and impacts, which are connected over various feedback loops. The terminology of stages is, however, heterogeneous. Whereas some studies treat shifts in peoples’ perspectives as direct outcomes of participatory processes, others consider them as further impacts due to larger political debates. Recently, sustainability researchers have begun to apply and further develop such frameworks (Walter et al. 2007, Wiek et al. 2014, Mitchell et al. 2015, Luederitz et al. 2017). Walter et al. (2007) focus on product and process-related effects in terms of decision-making capacities. Wiek et al. (2014) emphasize research events, the quality of participation in the research process, and appropriate means for addressing actors’ needs. Mitchell et al. (2015) proceed from desired outcomes such as improving a situation, contributing to the knowledge base and distribution, or transformational learning and what activities these entail. Luederitz et al. (2017) provide an evaluation framework with thematic questions about outputs, outcomes, processes, and inputs. Other scholars, however, question the usefulness of such linear approaches and highlight the existence of multiple pathways to impact (Polk 2014, Bergmann et al. 2017, Newig et al. 2018, Muhonen et al. 2020).
So far, impact frameworks have mostly been applied to individual cases with some notable exceptions of comparative analyses of larger samples (de Jong et al. 2016, Schneider and Buser 2018, Zscheischler et al. 2018, Herrero et al. 2019, Newig et al. 2019). Although de Jong et al. (2016) find that societal actor involvement in research funding programs improves societal impact, they do not find a correlation for societal actor involvement and impact in research projects. Herrero et al. (2019) and Newig et al. (2019) point to the importance of an early involvement of societal actors for impact generation and social learning, and Schneider and Buser (2018) analyze different intensities of actor interactions in different contexts. Zscheischler et al. (2018) present a success profile of transdisciplinary research, which includes mutual learning, science-practice cooperation on equal footing, and the synthesis of results.
Other evidence for sustainability impacts of participatory research mainly relies on individual case studies in particular social-ecological contexts (e.g., Oettlé et al. 2004, Termeer and Kessener 2007, Akpo et al. 2015) but cumulative learning across cases remains limited. Identified impacts of participatory research include the spread of usable products, enhanced capacity, network effects, and structural changes in decision making, public discourses, economic benefits, or changes in institutions and behavior (e.g., Walter et al. 2007, Schneider et al. 2009, Wiek et al. 2014, Kaufmann-Hayoz et al. 2016, Luederitz et al. 2017). However, findings also suggest that socially robust knowledge does not per se contribute to further impact (Polk 2014), and that stakeholder empowerment is often insufficient (Brandt et al. 2013). Indeed, Schneider et al. (2009) emphasize the importance of an atmosphere of trust, truly acknowledging different actors’ perspectives and the possibility to create implicit and explicit knowledge. Blackstock et al. (2007) stress that adapted practices need supportive governance mechanisms, and Newig et al. (2018) identify conditioning factors such as trust, structured knowledge-integration-processes, or capacity building of participants for better outcomes in environmental decision making. Schneider and Buser (2018) find that the intensity of actor interactions need to differ with the level of contestation, actor diversity, actor interest, and existing collaborations. Moreover, they stress that the envisaged sustainability contribution highly influences the choice of participatory methods. Consequently, the question if and how participatory approaches can reach societal impact depends on several aspects including context factors and the strived for sustainability goals.
In this paper, we aim to learn from the contextualized but scattered findings published as individual case studies and to study the patterns between different intensities of actor interactions and impacts while considering different contexts and sustainability goals. We start from the premise that different intensities of actor interactions and the timing of these interactions are important for generating impact under different context conditions and sustainability goals. Our guiding research questions are the following: What are the main self-reported sustainability impacts of participatory research projects, and how do the projects generate these impacts? How and when do different intensities of actor interactions contribute to the reported impacts? We particularly focus on the role of contexts and sustainability goals in these patterns.
To address our research questions, we followed a two-step approach: first, we developed our diagnostic framework based on a literature review. Second, we conducted a meta-analysis of findings published in case study papers to identify archetypical configurations of impacts, participatory research approaches, context conditions, and sustainability goals (Rudel 2008, Oberlack and Eisenack 2018). Archetype analysis is increasingly used in sustainability research as a methodological approach to identify recurrent patterns from larger sets of case studies (Oberlack et al. 2019a). Archetypes are generally defined as context-sensitive, generalized models of recurrent patterns of factors and processes that explain the outcome of interest with case-level empirical validity (Eisenack et al. 2006, Sietz 2017). The meta-analysis approach can be seen as a systematization of reported findings from case studies in an integrative language, which allows for abstracting from individual case studies (Jensen and Rodgers 2001).
For the purpose of this study, we consider different schools of participatory research, including transdisciplinary research (Hadorn et al. 2008), codesign and coproduction (Mauser et al. 2013), collaborative and participatory action research (Selener 1997). To provide focus, we limit this study to participatory research approaches aimed at more sustainable agricultural production. These include settings with complex relations between different actors along food, feed, and energy value chains, a high degree of uncertainty because of weather and market dependency, and contradicting objectives and preferences of individual actors (Ericksen 2008, Hamm 2009). Participatory research approaches are generally seen as promising ways to address the diverse problems of different production systems, and have been applied in various settings around the globe (Binder et al. 2012, Monzote et al. 2012, Andersson 2015, Jacobi et al. 2015).
Answering our research questions required adapting existing frameworks (Walter et al. 2007, Wiek et al. 2014, Mitchell et al. 2015, Luederitz et al. 2017). Although these frameworks focus on the “how” of participatory research processes and their impacts, none of them specifically considers the context conditions in which a project develops, nor the sustainability goal as a motivation for the project. However, these points could play an important role for participatory research approaches and the chosen activities (Schneider and Buser 2018). Ostrom’s (2005) institutional analysis and development (IAD) framework provides further inspirations because it allows aligning the above mentioned frameworks with context and sustainability goals (Guimarães et al. 2018).
The developed diagnostic framework links participatory research approaches with contexts, sustainability goals, and impact (Fig. 1). It conceptually defines the engagement of different actors (participatory approach) in specific contexts in order to achieve certain sustainability goals. We presume that the participatory approaches chosen by research projects depended on the sustainability goals and their context. They represent the action situations in which new knowledge is produced, actor interactions take place, and conflicts or synergies arise.
Because the framework is based on different types of studies, it also comprises different types of results of participatory research approaches. These types include what other authors have defined as outputs, outcomes, and impacts. However, the exact differences between outputs, outcomes, and impacts do vary across studies. Therefore, for the purpose of this meta-analysis, we placed them all in the impact category, rather than distinguishing between these different types of results. Below we describe the four main components of our diagnostic framework.
The context of participatory research projects varies widely along social, economic, political, and ecological conditions (Adler et al. 2018). We included the following context variables: investigated sustainability problems, involved actors, wickedness of the situation, type of missing knowledge, and the research context. The sustainability problem, i.e., resources and environment, socioeconomic, and governance, was chosen because it potentially involves topics, problem constellations, and actor groups that might require different approaches and activities (Schneider and Buser 2018). The degree of wickedness of these problems was part of our framework because participatory research approaches are generally deemed suitable to address higher degrees of wickedness (Brown 2010). The type of missing knowledge—system, target, and transformation knowledge (Proclim/CASS 1997, Pohl and Hirsch Hadorn 2007)—and the actor context (Enengel et al. 2012) were included because they might be decisive for tailoring participatory approaches.
Sustainability goals can vary extensively between participatory research projects, e.g., some focus on the generation of more robust knowledge, others on the empowerment of involved actors and intended behavioral changes. For the purpose of this study, we used the three areas of sustainability goals suggested by Mitchell et al. (2015): knowledge products, learning, and real-world transformations. Although in many cases the ultimate goal of participatory research might be real-world transformations, i.e., substantial changes in the social-ecological system, more tangible goals concern knowledge products or learning. Knowledge products refer to decision-making tools, knowledge databases, action plans, and publications. Learning includes the development of ownership, new knowledge, networking, reflection on learning, trust building, self-confidence, capacity building, and communicating best practices. Based on Gibson et al. (2005), we categorized real-world transformations as social-ecological integrity, enhanced livelihoods, intra- and intergenerational equity, resource maintenance and efficiency, social-ecological stewardship and democratic governance, and precaution and adaptation.
Interactions between scientific and societal actors can have different intensities. Based on Arnstein (1969), we used the following categories for operationalizing intensity: information (delivering information to societal actors), consultation (inquire into societal actors’ opinions, expectations, and preferences), and coproduction of new knowledge (joint knowledge generation between researchers and societal actors during different research phases). Furthermore, we structured the research process along three main phases, namely the joint framing of problems and research questions, the coproduction of new knowledge, and the integration of this newly produced knowledge into science and societal practice (Lang et al. 2012). We also distinguished between different roles scientific and nonscientific actors play in the chosen approaches (Pielke 2014).
Impact categories correspond to the categories of sustainability goals, i.e., knowledge products, learning, and real-world transformation. We acknowledge that (a) the process of impact generation is complex and can follow different pathways, e.g., generating specific products or triggering social interactions and learning (Kaufmann-Hayoz et al. 2016). These include feedback effects influencing back on the context, the approach, and the sustainability goal (Fig. 1). Systematic descriptions of these feedback effects were not reported in the case studies and are not considered further; (b) although the investigated case studies used different concepts to categorize impact, we used the term impact in a broader sense without assuming specific sequences of outputs, outcomes, and impacts.
In our study, a case was a participatory research project and a case study was a scientific, peer-reviewed publication that presents the results of whether and how a project generated impact. For analyzing the case studies according to the introduced diagnostic framework, we adopted the study design for archetype analysis by Oberlack et al. (2016) and Oberlack and Eisenack (2018). Archetypes generalize evidence from cases into recurrent patterns, which explain how outcomes arise from particular configurations of conditions and processes (Oberlack et al. 2019a, Sietz et al. 2019). Archetypes are building blocks because a single case can involve multiple archetypes (Eisenack et al. 2019). The present study conducted a meta-analysis of case studies following the three steps of case study retrieval and selection, coding, and data analysis (Magliocca et al. 2015).
Figure 2 presents the detailed study protocol. We first retrieved case studies through a key word search in Scopus and Web of Science, yielding 389 unique references. We selected only those papers that met all inclusion criteria (Fig. 2); 26 papers were finally included in the meta-analysis (list of included case studies in Appendix 1). They cover 29 cases of participatory research projects located in 7 countries in Europe, 1 in North America, 10 in Africa, 3 in Asia Pacific, 1 in Central Asia, and 2 in Latin America. Some studies highlight their approaches and methodologies, some report about their findings and results in detail, and others reflect on the scientific and societal impacts of their projects.
We coded the case studies using MaxQDA software and the codebook, which is based on our diagnostic framework (Appendix 2). Correspondingly, we coded the reported impacts, participatory approaches, goals, and context conditions for each case study. The first author coded all 26 case studies; the second and third authors independently coded 5 papers each. The coding team kept consistent codings and resolved disagreements by discussing them and agreeing on the final code. The resulting dataset consisted of 29 cases characterized by the reported attributes per case. Using the 154 attributes of the codebook, we found 653 codes in total with some of the attributes reported frequently and others few or zero times.
Data analysis followed two steps. First, we compiled frequencies of what the studies reported on impacts, activities per research phase, and actor roles. This provides a descriptive overview of processes and impacts. Second, we applied formal concept analysis (FCA) to identify archetypical patterns in the data set (Ganter and Wille 1999) using the Concept Explorer software. FCA is a set-theoretic methodology that analyzes the dataset by compiling logical implications between attributes. It generates a concept lattice, which organizes attributes in a hierarchical structure so that higher level attributes are logical implications of connected lower level attributes, whereas lower level attributes show distinct combinations with higher level attributes (Oberlack et al. 2016, Oberlack and Eisenack 2018). We compiled the concept lattices for each of the impact categories reported to have been achieved, i.e., learning, knowledge products, and real-world transformations. For each impact category, we analyzed what participatory approaches, goals, and context conditions are recurrently associated with a particular impact, and we recorded the frequency for each association. In line with the design standards of Eisenack et al. (2019), we used a frequency threshold. We only report associations that were reported in at least 20% of all cases with a particular kind of impact in order to identify only associations that are recurrent.
The first limitation relates to potential publication biases and missing data. Both points are well-known to meta-analytical methodologies (van Vliet et al. 2016). Authors of primary studies have their own reasoning for why they report certain findings and not others. The often brief descriptions of research processes in primary studies limit our analysis because we could not find information for all our framework variables even if they were deemed important. Such examples included research conditions, the degree of wickedness, or power relations (Pohl and Hirsch Hadorn 2008, Reed 2008, Brown 2010, Avelino and Wittmayer 2016). Unreported findings can mean they were unimportant to the primary study or they were not covered; both points cannot be verified here. In addition, research “failures” are often difficult to publish despite their important learning potential for others. Furthermore, any meta-analysis faces the challenge that primary studies may have omitted variables, i.e., attributed impacts to false causes. To reduce this problem, we used methodological inclusion criteria (Fig. 2) to ensure credibility of primary case studies. Furthermore, the codebook in Appendix 2 may be a starting point to address problems of omitted variables and missing data in future studies.
Second, it is difficult to assess cause and effect because of the lack of sufficient details in many of the primary studies. It is likely that other, unmentioned events contributed to the reported impact. Based on our methodological inclusion criteria, we assumed that reported impacts were related to the applied methods and approaches to a certain degree as reported in the primary studies, which were all published in peer-reviewed, scientific journals. However, we cannot verify that reported impacts actually happened. Hence, the synthesized effects of our study need to be rigorously tested in future empirical studies.
The results of this paper should be understood as a systematization of the available framings and scientific evidence on how participatory research approaches contribute to impact. They synthesize previously scattered evidence of 29 cases into archetypical patterns, which provide insights into generalizable patterns, highlight diversity across cases, and may inspire new empirical research. They show that archetypical configurations exist in the field of agriculture and can be usefully depicted but we do not claim representativeness of the retrieved configurations for the global universe of cases.
The results show that the participatory research projects in our sample generated three main types of impacts: learning, knowledge products, and real world-transformations. Each of these impacts arose from particular intensities of interactions, contexts, and sustainability goals. To gain insight into the commonalities and diversity of the cases, we also provide descriptive statistics showing disaggregated data for activities, impacts, actor roles, and intensity of actor interactions.
Of the 29 cases, 22 reported learning as a main impact (Fig. 3) and 17 highlighted learning among others as the intended sustainability goal. Learning was reported in diverse ways, including providing new knowledge, capacity building, trust, self-confidence, or ownership (Fig. 4). Examples included new knowledge for societal actors about methods for pest and disease management (Oettlé et al. 2004), reducing soil erosion (Bagshaw and Lindsay 2009), strategy development and action-oriented knowledge for regional development (Breu et al. 2005), or technical knowledge about seedling production (Akpo et al. 2015). In some cases this knowledge was jointly produced while in others it was disseminated to actors. Other examples were insights from joint reflections of participants on what they had learned, increased trust because of the process, and improved capacity to deal with daily challenges. Several cases used activities that are not so common such as interactive site visits, field experiments, and participatory filmmaking to support educational and learning experiences. Such activities serve to better embed and empower farmers in the knowledge production process. They are also practical means for participants to engage in new topics.
A majority of cases that reported learning took place in the context of problems related to resources and environment, which concerned soil problems, water problems, or land degradation and desertification. Appendix 3 gives an overview of all reported sustainability problems in the included case studies. These occurred usually in combination with socioeconomic problems such as poverty, declining margins, and high costs, and repeatedly in combination with governance problems such as insufficient governmental services and infrastructure or mismanagement of natural resources. Many of the cases address complex, cross-sectoral problems. Table 1 shows the frequencies of each reported sustainability problem for the main variables and the second order variables. Although projects address environmental, socioeconomic, and governance problems with similar frequencies (38%, 31%, and 30% of all cases, respectively), learning is reported considerably more often as an impact if projects address problems of resources and the environment.
The pattern shows the significance of coproduction as opposed to consultation or information activities (Fig. 3). Coproduction is particularly important in the second research phase of reported learning cases (20 of 22 cases). Coproduction in the second phase involved, for example, regular meetings to discuss new insights on nutrient issues (Cabrera et al. 2008), a system analysis workshop to assess the sustainability in the milk value chain (Binder et al. 2012), or participatory scenario-building workshops for a more sustainable irrigated agriculture (Soste et al. 2015). Half of these cases are associated with coproduction in phase 1 and slightly fewer with coproduction in phase 3. Consultation is used frequently in phases 1 and 2 but always together with coproduction.
The main activities in phase 1 associated with learning are the joint definition of objectives and questions, joint problem identification and definition, and agreed-upon methodology for coproduction (Fig. 5). In phase 2, the integration of different types of knowledge from experts, farmers, and scientists were important as well as the application of integrative methods such as participatory filmmaking. For phase 3, the studies reported the integration of knowledge into societal practice and the evaluation of impact.
Ten cases reported impact in the form of knowledge products (Fig. 6). These knowledge products (Fig. 4) ranged from whole farm decision-support systems in Florida (Cabrera et al. 2008) and a sustainability assessment tool for the milk value chain in Switzerland (Binder et al. 2012), over action plans for less polluting banana production in Australia (Bagshaw and Lindsay 2009) to training material for palm seed nurseries in Benin (Akpo et al. 2015).
The archetypical configuration for knowledge products was less clear than for learning. We found a strong interlinked presence of reported coproduction in all three phases. The investigated case studies may have embedded the creation of new knowledge products well throughout the entire research process in order to increase chances for a successful implementation. The goal learning coupled with coproduction in phases 1 and 2 supported this hypothesis by emphasizing the need for a comprehensive understanding of how to design and implement such knowledge products. An example of this is a decision-support system to decrease nitrogen leaching in dairy farming (Cabrera et al. 2008). The project’s approach involved participatory workshops in phase 1 to coproduce certain model parameters, in phase 2 to jointly reflect on new scientific evidence on nutrient issues, and in phase 3 to evaluate the product and test its applicability.
Reported coproduction activities in phases 1 and 2 included jointly defining objectives and questions, joint identification and definition of problems, agreed-upon methodology for coproduction, and assign and support roles for actors (Fig. 5). Reported activities in phase 2 were the integration of different types of knowledge and the adjustment of methods when necessary, while phase 3 was mainly related to the integration of knowledge into societal practice.
Ten cases reported real world transformation impacts (Fig. 7), mainly social-ecological integrity and enhanced livelihoods. Examples included the establishment of a whole new value chain of wild rooibos production and marketing within a cooperative (Oettlé et al. 2004); or increased food security and health, better environmental performance, higher incomes, and improved communal life as a consequence of more sustainable agricultural practices (Wright 2014). Examples for enhanced livelihoods include reported improvements of daily lives of people through better production methods (Andersson 2015) and new policies tailored to customers’ needs for taxing irrigation water (Dedrick et al. 2000). Impacts on resource maintenance and efficiency, social-ecological stewardship and democratic governance, and precaution and adaptation were only reported once, while intra- and intergenerational equity was not reported (Fig. 4).
Projects aiming to achieve real-world transformations not only reported complex, interrelated contexts but also featured a combination of different sustainability goals while coproduction is pronounced in all three research phases. Reported activities mainly included agreed-upon methodology for coproduction and jointly defined objectives and questions in phase 1 (Fig. 5). These included joint proposals defining who should participate when and how in knowledge production or participatory workshops for creating visions where everyone’s objectives were considered (Oettlé et al. 2004). Phase 2 mainly contained the subcategory adjusting methods when necessary. An example here is a flexible, farmer-led but research-based approach for developing solutions for sustainable local agriculture (Mog 2006). Examples for the integration of knowledge into societal practice in phase 3 are knowledge transfers through collaborations with extension services for farmers’ concerns and a wide range of education activities to inform societal actors about research results (Mog 2006).
Figure 8 highlights the reported roles of scientific and societal actors, which vary substantially for the different impact categories. For learning, the case studies reported that societal actors were predominantly knowledge producers, followed by facilitators and experts, while scientific actors were most often facilitators and knowledge producers. The archetypes for knowledge products and real-world transformations showed similar patterns.
Reported intensity of actor interactions covered the entire spectrum from information to coproduction, but high intensities of actor interactions were most prevalent. Table 2 shows reported coproduction cases in all research phases. For example, in the learning archetype, 11 coproduction cases were reported in phase 1, which corresponds to 50% of all learning cases. In over 80% of all cases, coproduction was reported in phase 2, over 50% in phase 1, and over 30% in phase 3 (Table 2). The numbers were also high across phases 1 and 2 (45–70%) and across all three phases (23–50%).
Furthermore, we compared the reported impacts to their sustainability goals. Table 3 shows the number of cases for each reported sustainability goal and impact. Some of the variations probably stemmed from misperceptions of planned impacts, e.g., scientific publication planned but societal publication achieved. Others were difficult to assess in terms of percentages because of very small case numbers, e.g., decision-making tools or the real world transformation impacts.
Our results show that different reported impacts involve different intensities of actor interactions throughout the three research phases. Case studies with the learning archetype reported the majority of coproduction activities in phase 2, while impacts in knowledge products and real-world transformations were reported with coproduction in all three research phases, which might be because of the complexity of such problems demanding a broad range of involved actors throughout the research process. This is in line with the findings of Schneider and Buser (2018). The most pronounced use of coproduction throughout the whole research process can be observed in real world transformation impacts. This seems to be only partly in line with previous findings by Herrero et al. (2019) who highlight the importance of coconstructing research questions and problem framings to achieve social learning. However, it has to be noted that the analyses are not directly comparable because Herrero et al. do not work with the three research phases by Lang et al. (2012) and focus more, in our terminology, on the first phase of participatory research projects. Newig et al. (2019) also find a positive effect of early involvement of societal actors in the formulation of problems and the definition of research questions for societal impact but they do not find a general positive impact of societal actor involvement over the whole length of the research process. Zscheischler et al. (2018) find that the perception of success in transdisciplinary research is strongly linked to practical solutions for societal problems. In this regard, intense actor interactions in knowledge products and real-world transformations seems reasonable because we can assume that they depend more on practical solutions.
Furthermore, we observe that the archetypes knowledge products and real-world transformations feature more interlinkages between sustainability problems and sustainability goals and have more intense actor interactions than the learning archetype. We assume that the former two archetypes involve more complexity and contestation because the actual finding of solutions for complex societal problems is more challenging than learning, especially in view of our definition of learning, which apart from social learning (Herrero et al. 2019) also includes individual learning. Following this line of argument, impacts of knowledge products and real world transformation should be better addressed through more intense actor interactions (Schneider and Buser 2018). Our results show that case studies with the learning archetype more often report that their sustainability goals were achieved. Although learning may be achieved more easily, higher complexity and contestation in knowledge products and real-world transformations may make it more difficult to reach sustainability goals.
The included case studies describe the roles of societal actors as knowledge producers rather than mere knowledge receivers, whereas scientific actors frequently take up the role of facilitators because of the need to integrate societal actors in the knowledge production process and to address issues of credibility and legitimacy during the process (Pohl et al. 2010). These new roles also demand a different type of training for researchers engaging in such projects in order to address power imbalances and create trust among the participants, which are seen as key factors for impact in social learning (Herrero et al. 2019) and decision making (Newig et al. 2018).
Our findings show that the initial sustainability goals at the start of a project can diverge from the reported impacts (Table 3). Although some of the case studies reported achievements of their stated goals, others did not or ended with unenvisioned impacts. Other authors have discussed these in terms of impact or success of participatory research, which is generally difficult to trace (Walter et al. 2007, Polk 2015, Klenk and Meehan 2017, Luederitz et al. 2017, Zscheischler et al. 2018). Relating project impacts to previously defined goals highlights the necessity to reflect on and engage with mechanisms that evolve during a project, for instance through a theory of change approach (Oberlack et al. 2019b).
The archetype approach provided the methodological basis for this study (Eisenack et al. 2006, Sietz et al. 2019). The understanding of archetypes as building blocks (Eisenack et al. 2019, Oberlack et al. 2019a) allowed us to synthesize the explanations of diverse impacts reported in the case studies into a typology, which we find parsimonious and detailed at the same time. Earlier archetype studies found that their cases differed strongly with regard to the processes and causal mechanisms that generate the impacts of concern (e.g., Sietz et al. 2011, Oberlack et al. 2016, Sietz 2017, Levers et al. 2018) or with regard to social-ecological conditions (e.g., Václavík et al. 2016). In our paper, we found the largest variety in the reported impacts, whereas reported processes were dominated by coproduction activities. This shows that, within a given dataset, analysts can search for archetypes using different entry points including the search for archetypical impacts, processes, or conditions.
The question about sustainability impacts in participatory research approaches is one of the research frontiers in this field, as evidenced by numerous conferences and conference sessions in 2019, the compilation of a Special Issue (Schäfer et al. 2020), a number of ongoing research projects on this theme, as well as ongoing debates in Future Earth. This paper contributes to this topical debate by introducing a diagnostic framework, which relates participatory research approaches to their project contexts, sustainability goals, and reported impacts. We used this framework to synthesize evidence from 29 systematically selected, coded, and analyzed case studies in the field of sustainable agriculture to identify archetypical configurations of impacts in participatory research. We did not engage in primary data collection on impacts but we coded the impacts as they were reported in the selected, peer-reviewed primary studies. Thus, our results should be understood as a synthesis of knowledge in the field instead of a comparative analysis of primary data. Hence, this paper goes beyond the analyses of one or few case studies in most publications on this theme and compares different participatory research approaches in an archetype analysis.
We show that there are distinct archetypes with different participatory research approaches that are associated with impact and in which coproduction rather than mere information or consultation plays an important role. Coproduction activities were more prevalent than information or consultation activities in cases of complex sustainability problems. Although information was not a recurrent pattern, consultation processes occurred often but mostly in combination with coproduction. Complex situations with a higher diversity of sustainability problems and goals are associated with more intense actor interactions throughout the entire research process. Although learning is mainly linked to coproduction in phases 1 and 2, the archetypes knowledge products and real-world transformations involve coproduction in all three research phases. It has to be considered that the typical time frame of research projects spans around three years. This is rather short for achieving real-world transformation impacts, even more so when they concern intra- and intergenerational justice (not reported in any of the case studies). Longer time frames are needed for evaluation activities. Impacts in learning or knowledge products seem to be more feasible within shorter project time frames.
Finally, with this paper we contribute to methodologies of archetypes analysis. It provides important methodological lessons on the potentials and pitfalls of FCA for archetype analysis. FCA has enabled us to disentangle the archetypical configurations of impacts, participatory research approaches, contexts, and sustainability goals, as reported by the 29 case studies. A challenge in our analysis was insufficient information for some of our framework variables. Limited reporting in primary studies often constrains the scope of meta-analyses (van Vliet et al. 2016). Hence, a coherent way of reporting on participatory research, including methodology, lessons learned, and failures of different approaches is strongly needed.
We believe that the diagnostic framework and the archetypical patterns found in this paper contribute to more coherent understanding of how participatory research projects from different traditions can and do trigger sustainability impacts. The diagnostic framework and the associated attributes provided in the codebook (Appendix 2) can be considered in future research for deciding what factors might be important in a participatory research project.
Data Availability Statement
Data/code openly available in a public repository that issues datasets with DOIs.
We thank the anonymous reviewers for their helpful comments.
Adler, C., G. Hirsch Hadorn, T. Breu, U. Wiesmann, and C. Pohl. 2018. Conceptualizing the transfer of knowledge across cases in transdisciplinary research. Sustainability Science 13:179-190. https://doi.org/10.1007/s11625-017-0444-2
Akpo, E., T. A. Crane, P. V. Vissoh, and R. C. Tossou. 2015. Co-production of knowledge in multi-stakeholder processes: analyzing joint experimentation as social learning. Journal of Agricultural Education & Extension 21:369-388. https://doi.org/10.1080/1389224X.2014.939201
Andersson, E. 2015. Turning waste into value: using human urine to enrich soils for sustainable food production in Uganda. Journal of Cleaner Production 96:290-298. https://doi.org/10.1016/j.jclepro.2014.01.070
Arnstein, S. R. 1969. A ladder of citizen participation. Journal of the American Planning Association 35:216-224.
Avelino, F., and J. M. Wittmayer. 2016. Shifting power relations in sustainability transitions: a multi-actor perspective. Journal of Environmental Policy & Planning 18:628-649. https://doi.org/10.1080/1523908X.2015.1112259
Bagshaw, J., and S. Lindsay. 2009. Developing sustainable banana production systems: a case study from tropical Australia. Acta Horticulturae 831:23-30. https://doi.org/10.17660/ActaHortic.2009.831.2
Bergmann, M., M. Schäfer, and T. Jahn. 2017. Wirkungen verstehen und feststellen. Arbeitspapier aus dem BMBF-Verbundprojekt TransImpact, Frankfurt am Main/Berlin, Germany.
Binder, C. R., A. Schmid, and J. K. Steinberger. 2012. Sustainability solution space of the Swiss milk value added chain. Ecological Economics 83:210-220. https://doi.org/10.1016/j.ecolecon.2012.06.022
Blackstock, K. L., G. J. Kelly, and B. L. Horsey. 2007. Developing and applying a framework to evaluate participatory research for sustainability. Ecological Economics 60:726-742. https://doi.org/10.1016/j.ecolecon.2006.05.014
Bornmann, L. 2013. What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology 64:217-233. https://doi.org/10.1002/asi.22803
Brandt, P., A. Ernst, F. Gralla, C. Luederitz, D. J. Lang, J. Newig, F. Reinert, D. J. Abson, and H. von Wehrden. 2013. A review of transdisciplinary research in sustainability science. Ecological Economics 92:1-15. https://doi.org/10.1016/j.ecolecon.2013.04.008
Breu, T., D. Maselli, and H. Hurni. 2005. Knowledge for sustainable development in the Tajik Pamir Mountains. Mountain Research and Development 25:139-146. https://doi.org/10.1659/0276-4741(2005)025[0139:KFSDIT]2.0.CO;2
Brown, V. A. 2010. Tackling wicked problems through the transdisciplinary imagination. Earthscan, London, UK.
Buxton, M., and S. Hanney. 1996. How can payback from health research be assessed? Journal of Health Services Research and Policy 1:35-43. https://doi.org/10.1177/135581969600100107
Cabrera, V. E., N. E. Breuer, and P. E. Hildebrand. 2008. Participatory modeling in dairy farm systems: a method for building consensual environmental sustainability using seasonal climate forecasts. Climatic Change 89:395-409. https://doi.org/10.1007/s10584-007-9371-z
Cash, D., W. C. Clark, F. Alcock, N. M. Dickson, N. Eckley, and J. Jäger. 2002. Salience, credibility, legitimacy and boundaries: linking research, assessment and decision making. KSG Working Papers Series RWP02-046. Kennedy School of Government, Harvard, Cambridge, Massachusetts, USA. https://doi.org/10.2139/ssrn.372280
de Jong, S. P. L., T. Wardenaar, and E. Horlings. 2016. Exploring the promises of transdisciplinary research: a quantitative study of two climate research programmes. Research Policy 45(7):1397-1409. https://doi.org/10.1016/j.respol.2016.04.008
Dedrick, A. R., E. Bautista, W. Clyma, D. B. Levine, and S. A. Rish. 2000. The management improvement program (MIP): a process for improving the performance of irrigated agriculture. Irrigation and Drainage Systems 14:5-39. https://doi.org/10.1023/A:1006325314944
Economic and Social Research Council (ESRC). 2011. Branching out. New directions in impact evaluation from the ESRC’s evaluation committee. ESRC, Swindon, UK. [online] URL: https://esrc.ukri.org/files/research/research-and-impact-evaluation/branching-out-new-directions-in-impact-evaluation-from-the-esrc-s-evaluation-committee/
Eisenack, K., M. Lüdeke, and J. Kropp. 2006. Construction of archetypes as a formal method to analyze social-ecological systems. In IDGEC Synthesis Conference of the Institutional Dimensions of Global Environmental Change. Bali, Indonesia, 6–9 December. International Human Dimensions Programme of Global Environmental Change, Bonn, Germany. [online] URL: https://uol.de/fileadmin/user_upload/wire/fachgebiete/envdev/download/arch-eisenack3.pdf
Eisenack, K., S. , S. Villamayor-Tomas, G. Epstein, C. Kimmich, N. Magliocca, D. Manuel-Navarrete, C. Oberlack, M. Roggero, and D. Sietz. 2019. Design and quality criteria for archetype analysis. Ecology and Society 24(3):26. https://doi.org/10.5751/ES-10855-240306
Enengel, B., A. Muhar, M. Penker, B. Freyer, S. Drlik, and F. Ritter. 2012. Co-production of knowledge in transdisciplinary doctoral theses on landscape development-an analysis of actor roles and knowledge types in different research phases. Landscape and Urban Planning 105:106-117. https://doi.org/10.1016/j.landurbplan.2011.12.004
Ericksen, P. J. 2008. Conceptualizing food systems for global environmental change research. Global Environmental Change 18:234-245. https://doi.org/10.1016/j.gloenvcha.2007.09.002
Folke, C., T. Hahn, P. Olsson, and J. Norberg. 2005. Adaptive governance of social-ecological systems. Annual Review of Environment and Resources 30:441-473. https://doi.org/10.1146/annurev.energy.30.050504.144511
Ganter, B., and R. Wille. 1999. Formal concept analysis: mathematical concept analysis. Springer, Berlin, Germany.
Gibson, R. B., S. Hassan, S. Holtz, J. Tansey, and G. Whitelaw. 2005. Sustainability assessment: criteria and processes. Earthscan, London, UK. https://doi.org/10.4324/9781849772716
Godin, B., and J. Dore. 2005. Measuring the impacts of science: beyond the economic dimension. Helsinki Institute for Science and Technology Studies, HIST Lecture, Helsinki, Finland.
Guimarães, M. H., N. Guiomar, D. Surová, S. Godinho, T. Pinto Correia, A. Sandberg, F. Ravera, and M. Varanda. 2018. Structuring wicked problems in transdisciplinary research using the social-ecological systems framework: an application to the montado system, Alentejo, Portugal. Journal of Cleaner Production 191:417-428. https://doi.org/10.1016/j.jclepro.2018.04.200
Hadorn, G. H., S. Biber-Klemm, W. Grossenbacher-Mansuy, H. Hoffmann-Riem, D. Joye, C. Pohl, U. Wiesmann, and E. Zemp. 2008. The emergence of transdisciplinarity as a form of research. Pages 19-39 in G. Hirsch Hadorn, H. Hoffmann-Riem, S. Biber-Klemm, W. Grossenbacher-Mansuy, D. Joye, C. Pohl, U. Wiesmann, and E. Zemp, editors. Handbook of transdisciplinary research. Springer, Dordrecht, The Netherlands. https://doi.org/10.1007/978-1-4020-6699-3
Hamm, M. W. 2009. Principles for framing a healthy food system. Journal of Hunger & Environmental Nutrition 4(3-4):241-250. https://doi.org/10.1080/19320240903321219
Herrero, P., T. Dedeurwaerdere, and A. Osinski. 2019. Design features for social learning in transformative transdisciplinary research. Sustainability Science 14:751-769. https://doi.org/10.1007/s11625-018-0641-7
Huutoniemi, K. 2014. Introduction: sustainability, transdisciplinarity and the complexity of knowing. Chapter 1 in K. Huutoniemi and P. Tapio, editors. Transdisciplinary sustainability studies: a heuristic approach. Routledge, London, UK. https://doi.org/10.4324/9780203734834
Jacobi, J., M. Schneider, P. Bottazzi, M. Pillco, P. Calizaya, and S. Rist. 2015. Agroecosystem resilience and farmers’ perceptions of climate change impacts on cocoa farms in Alto Beni, Bolivia. Renewable Agriculture and Food Systems 30(2):170-183. https://doi.org/10.1017/S174217051300029X
Jensen, J. L., and R. Rodgers. 2001. Cumulating the intellectual gold of case study research. Public Administration Review 61:235-246. https://doi.org/10.1111/0033-3352.00025
Kaufmann-Hayoz, R., R. Defila, A. Di Giulio, and M. Winkelmann. 2016. Was man sich erhoffen darf - Zur gesellschaftlichen Wirkung transdisziplinärer Forschung. Pages 289-327 in R. Defila and A. Di Giulio, editors. Transdisziplinär forschen - zwischen Ideal und gelebter Praxis. Hotspots, Geschichten, Wirkungen. Campus Verlag, Frankfurt am Main, Germany.
Klenk, N. L., and K. Meehan. 2017. Transdisciplinary sustainability research beyond engagement models: toward adventures in relevance. Environmental Science & Policy 78:27-35. https://doi.org/10.1016/j.envsci.2017.09.006
Lang, D. J., A. Wiek, M. Bergmann, M. Stauffacher, P. Martens, P. Moll, M. Swilling, and C. Thomas. 2012. Transdisciplinary research in sustainability science: practice, principles, and challenges. Sustainability Science 7:25-43. https://doi.org/10.1007/s11625-011-0149-x
Leach, M., I. Scoones, and A. Stirling. 2010. Dynamic sustainabilities: technology, environment, social justice. Earthscan, London, UK. https://doi.org/10.4324/9781849775069
Levers, C., D. Müller, K. Erb, H. Haberl, M. R. Jepsen, M. J. Metzger, P. Meyfroidt, T. Plieninger, C. Plutzar, J. Stürck, P. H. Verburg, P. J. Verkerk, and T. Kuemmerle. 2018. Archetypical patterns and trajectories of land systems in Europe. Regional Environmental Change 18:715-732. https://doi.org/10.1007/s10113-015-0907-x
Luederitz, C., N. Schäpke, A. Wiek, D. J. Lang, M. Bergmann, J. J. Bos, S. Burch, A. Davies, J. Evans, A. König, M. A. Farrelly, N. Forrest, N. Frantzeskaki, R. B. Gibson, B. Kay, D. Loorbach, K. McCormick, O. Parodi, F. Rauschmayer, U. Schneidewind, M. Stauffacher, F. Stelzer, G. Trencher, J. Venjakob, P. J. Vergragt, H. von Wehrden, and F. R. Westley. 2017. Learning through evaluation - a tentative evaluative scheme for sustainability transition experiments. Journal of Cleaner Production 169:61-76. https://doi.org/10.1016/j.jclepro.2016.09.005
Magliocca, N., T. K. Rudel, P. H. Verburg, W. J. McConnell, O. Mertz, K. Gerstner, A. Heinimann, and E. C. Ellis. 2015. Synthesis in land change science: methodological patterns, challenges, and guidelines. Regional Environmental Change 15:211-226. https://doi.org/10.1007/s10113-014-0626-8
Mauser, W., G. Klepper, M. Rice, B. S. Schmalzbauer, H. Hackmann, R. Leemans, and H. Moore. 2013. Transdisciplinary global change research: the co-creation of knowledge for sustainability. Current Opinion in Environmental Sustainability 5:420-431. https://doi.org/10.1016/j.cosust.2013.07.001
Miettinen, R., J. Tuunainen, and T. Esko. 2015. Epistemological, artefactual and interactional-institutional foundations of social impact of academic research. Minerva 53:257-277. https://doi.org/10.1007/s11024-015-9278-1
Mitchell, C., D. Cordell, and D. Fam. 2015. Beginning at the end: the outcome spaces framework to guide purposive transdisciplinary research. Futures 65:86-96. https://doi.org/10.1016/j.futures.2014.10.007
Mog, J. M. 2006. Managing development programs for sustainability: integrating development and research through adaptive management. Society and Natural Resources 19:531-546. https://doi.org/10.1080/08941920600663961
Monzote, F. R. F., R. Bello, A. Alvarez, A. Hernández, E. A. Lantinga, and H. van Keulen. 2012. Identifying agroecological mixed farming strategies for local conditions in San Antonio de Los Baños, Cuba. International Journal of Agricultural Sustainability 10:208-229. https://doi.org/10.1080/14735903.2012.692955
Moser, S. C. 2016. Can science on transformation transform science? Lessons from co-design. Current Opinion in Environmental Sustainability 20:106-115. https://doi.org/10.1016/j.cosust.2016.10.007
Muhonen, R., P. Benneworth, and J. Olmos-Peñuela. 2020. From productive interactions to impact pathways: understanding the key dimensions in developing SSH research societal impact. Research Evaluation 29(1):34-47. https://doi.org/10.1093/reseval/rvz003
Newig, J., E. Challies, N. W. Jager, E. Kochskaemper, and A. Adzersen. 2018. The environmental performance of participatory and collaborative governance: a framework of causal mechanisms. Policy Studies Journal 46:269-297. https://doi.org/10.1111/psj.12209
Newig, J., S. Jahn, D. J. Lang, J. Kahle, and M. Bergmann. 2019. Linking modes of research to their scientific and societal outcomes. Evidence from 81 sustainability-oriented research projects. Environmental Science & Policy 101:147-155. https://doi.org/10.1016/j.envsci.2019.08.008
Oberlack, C., T. Breu, M. Giger, N. Harari, K. Herweg, S. L. Mathez-Stiefel, P. Messerli, S. Moser, C. Ott, I. Providoli, T. Tribaldos, A. Zimmerman, and F. Schneider. 2019b. Theories of change in sustainability science: understanding how change happens. GAIA-Ecological Perspectives for Science and Society 28(2):106-111. https://doi.org/10.14512/gaia.28.2.8
Oberlack, C., and K. Eisenack. 2018. Archetypical barriers to adapting water governance in river basins to climate change. Journal of Institutional Economics 14:527-555. https://doi.org/10.1017/S1744137417000509
Oberlack, C., D. Sietz, E. Bürgi Bonanomi, A. de Bremond, J. Dell'Angelo, K. Eisenack, E. Ellis, G. Epstein, M. Giger, A. Heinimann, C. Kimmich, M. T. J. Kok, D. Manuel-Navarrete, P. Messerli, P. Meyfroidt, T. Václavík, and S. Villamayor-Tomas. 2019a. Archetype analysis in sustainability research: meanings, motivations, and evidence-based policy making. Ecology and Society 24(2):26. https://doi.org/10.5751/ES-10747-240226
Oberlack, C., L. Tejada, P. Messerli, S. Rist, and M. Giger. 2016. Sustainable livelihoods in the global land rush? Archetypes of livelihood vulnerability and sustainability potentials. Global Environmental Change 41:153-171. https://doi.org/10.1016/j.gloenvcha.2016.10.001
Oettlé, N., A. Arendse, B. Koelle, and A. Van Der Poll. 2004. Community exchange and training in the Suid Bokkeveld: a UNCCD pilot project to enhance livelihoods and natural resource management. Environmental Monitoring and Assessment 99:115-125. https://doi.org/10.1007/s10661-004-4011-9
Ostrom, E. 2005. Understanding institutional diversity. Princeton University Press, New Jersey, Princeton, USA.
Ostrom, E. 2009. A general framework for analyzing sustainability of social-ecological systems. Science 325(5939):419-422. https://doi.org/10.1126/science.1172133
Pahl-Wostl, C. 2002. Participative and stakeholder-based policy design, evaluation and modeling processes. Integrated Assessment 3:3-14. https://doi.org/10.1076/iaij.18.104.22.16809
Pahl-Wostl, C., J. Sendzimir, P. Jeffrey, J. Aerts, G. Berkamp, and K. Cross. 2007. Managing change toward adaptive water management through social learning. Ecology and Society 12(2):30. https://doi.org/10.5751/ES-02147-120230
Pielke, R. A. 2014. The honest broker: making sense of science in policy and politics. 10th edition. Cambridge University Press, Cambridge, UK.
Pohl, C., and G. Hirsch Hadorn. 2007. Principles for designing transdisciplinary research. Swiss Academies of Arts and Sciences, Bern, Switzerland.
Pohl, C., and G. Hirsch Hadorn. 2008. Methodological challenges of transdisciplinary research. Natures Sciences Sociétés 16:111-121. https://doi.org/10.1051/nss:2008035
Pohl, C., S. Rist, A. Zimmermann, P. Fry, G. S. Gurung, F. Schneider, C. Ifejika Speranza, B. Kiteme, S. Boillat, E. Serrano, G. Hirsch Hadorn, and U. Wiesmann. 2010. Researchers’ roles in knowledge co-production: experience from sustainability research in Kenya, Switzerland, Bolivia and Nepal. Science and Public Policy 37:267-281. https://doi.org/10.3152/030234210X496628
Polk, M. 2014. Achieving the promise of transdisciplinarity: a critical exploration of the relationship between transdisciplinary research and societal problem solving. Sustainability Science 9(4):439-451. https://doi.org/10.1007/s11625-014-0247-7
Polk, M. 2015. Transdisciplinary co-production: designing and testing a transdisciplinary research framework for societal problem solving. Futures 65:110-122. https://doi.org/10.1016/j.futures.2014.11.001
Proclim/CASS Forum for Climate and Global Change/Conference of the Swiss Scientific Academies. 1997. Visions by Swiss researchers: research on sustainability and global change-visions in science policy by Swiss researchers. ProClim, Bern, Switzerland.
Reed, M. S. 2008. Stakeholder participation for environmental management: a literature review. Biological Conservation 141:2417-2431. https://doi.org/10.1016/j.biocon.2008.07.014
Rudel, T. K. 2008. Meta-analyses of case studies: a method for studying regional and global environmental change. Global Environmental Change 18:18-25. https://doi.org/10.1016/j.gloenvcha.2007.06.001
Schäfer, M., A. Lux, and M. Bergmann. 2020. Editorial to the special issue “Transdisciplinary Sustainability Research—Linking research processes and outputs to societal effects.” Environmental Science and Policy 107:206-210. https://doi.org/10.1016/j.envsci.2020.02.018
Schneider, F., and T. Buser. 2018. Promising degrees of stakeholder interaction in research for sustainable development. Sustainability Science 13:129-142. https://doi.org/10.1007/s11625-017-0507-4
Schneider, F., P. Fry, T. Ledermann, and S. Rist. 2009. Social learning processes in Swiss soil protection — the ‘from farmer - to farmer’ project. Human Ecology 37:475-489. https://doi.org/10.1007/s10745-009-9262-1
Selener, D. 1997. Participatory action research and social change. Second edition. Cornell University, Ithaca, New York, USA.
Sietz, D. 2017. Nested archetypes of vulnerability in African drylands. Environmental Research Letters 12(9).
Sietz, D., U. Frey, M. Roggero, Y. Gong, N. Magliocca, R. Tan, P. Janssen, and T. Václavík. 2019. Archetype analysis in sustainability research: methodological portfolio and analytical frontiers. Ecology and Society 24(3):34. https://doi.org/10.5751/es-11103-240334
Sietz, D., M. K. B. Lüdeke, and C. Walther. 2011. Categorisation of typical vulnerability patterns in global drylands. Global Environmental Change 21:431-440. https://doi.org/10.1016/j.gloenvcha.2010.11.005
Soste, L., Q. J. Wang, D. Robertson, R. Chaffe, S. Handley, and Y. Wei. 2015. Engendering stakeholder ownership in scenario planning. Technological Forecasting and Social Change 91:250-263. https://doi.org/10.1016/j.techfore.2014.03.002
Termeer, C. J. A. M., A. Dewulf, G. Breeman, and S. J. Stiller. 2013. Governance capabilities for dealing wisely with wicked problems. Administration & Society 47:680-710. https://doi.org/10.1177/0095399712469195
Termeer, C. J. A. M., and B. Kessener. 2007. Revitalizing stagnated policy processes: using the configuration approach for research and interventions. Journal of Applied Behavioral Science 43:256-272. https://doi.org/10.1177/0021886306294902
Václavík, T., F. Langerwisch, M. Cotter, J. Fick, I. Häuser, S. Hotes, J. Kamp, J. Settele, J. H. Spangenberg, and R. Seppelt. 2016. Investigating potential transferability of place-based research in land system science. Environmental Research Letters 11:9. https://doi.org/10.1088/1748-9326/11/9/095002
van Vliet, J., N. R. Magliocca, B. Büchner, E. Cook, J. M. Rey Benayas, E. C. Ellis, A. Heinimann, E. Keys, T. M. Lee, J. Liu, O. Mertz, P. Meyfroidt, M. Moritz, C. Poeplau, B. E. Robinson, R. Seppelt, K. C. Seto, and P. H. Verburg. 2016. Meta-studies in land use science: current coverage and prospects. Ambio 45:15-28. https://doi.org/10.1007/s13280-015-0699-8
Walter, A. I., S. Helgenberger, A. Wiek, and R. W. Scholz. 2007. Measuring societal effects of transdisciplinary research projects: design and application of an evaluation method. Evaluation and Program Planning 30(4):325-338. https://doi.org/10.1016/j.evalprogplan.2007.08.002
Wiek, A., S. Talwar, M. O'Shea, and J. Robinson. 2014. Toward a methodological scheme for capturing societal effects of participatory sustainability research. Research Evaluation 23:117-132. https://doi.org/10.1093/reseval/rvt031
Wright, S. 2014. Quantitative research performing other worlds: lessons from sustainable agriculture in the Philippines. Australian Geographer 45:1-18. https://doi.org/10.1080/00049182.2014.869293
Zscheischler, J., S. Rogga, and A. Lange. 2018. The success of transdisciplinary research for sustainable land use: individual perceptions and assessments. Sustainability Science 13:1061-1074. https://doi.org/10.1007/s11625-018-0556-3