Research

INTRODUCTION

Science and management of natural resources

Defensible and adaptive management of natural resources relies on the integration of scientific information into decision making. Although scientific information is not the only important type of information in natural resource decision making, science and scientists play an important role in helping natural resource managers (managers) evaluate the range of options available to them and envision the likely consequences of alternative management actions (Mills and Clark 2001). Scientific information can affect management in multiple ways, including shaping how managers perceive management issues, how management actions are implemented, and how we evaluate policy alternatives (Hunter et al. 2020). Use of the Best Available Scientific Information is also legally mandated in some contexts, as in the case of forest planning and endangered species listing decisions in the United States (Ryan et al. 2018).

How managers find and use scientific information has been the subject of considerable research. When making decisions, managers draw on multiple sources of information, including academic journals, government reports, and personal experience, among other sources (Pullin et al. 2004, Cook et al. 2010, Hunter et al. 2020, Barrett and Rodriguez 2021, Piczak et al. 2022). Science is generally valued, with the belief that greater access and ability to assess scientific information improves or would improve decision quality (e.g., Kadykalo et al. 2021). Although managers do adapt their actions based on scientific information (Walsh et al. 2015), the decision space of managers is more complex than simply receiving science and incorporating it directly into management. Scientists and managers occupy two communities of practice with different norms, incentives, and cultures (Roux et al. 2006). In particular, managers and scientists may disagree on what makes scientific information credible, legitimate, and salient (Cook et al. 2013) and hold different values that affect how they evaluate scientific information (Heeren et al. 2017, Karns et al. 2018). For example, experimental design plays a critical role in the precision and accuracy of ecological studies (Christie et al. 2019) and many scientists advocate that the highest quality research comes from randomized before-after control-intervention (R-BACI) designs (Christie et al. 2020) though others argue against privileging certain study designs over others (Bruskotter et al. 2017). In comparison, managers have additional considerations when evaluating the quality of a scientific study for use in natural resource management, such as relevance to their problem area and timeliness (Cook et al. 2013, Fischer et al. 2014, Heeren et al. 2017). Consequently, managers have at times received criticism for the lack of scientific justification for their decisions and actions (Pullin and Knight 2001, 2003, Pullin et al. 2004, Artelle et al. 2018).

Barriers to the flow of information between scientists and managers are numerous (Walsh et al. 2019). Insufficient capacity to find and evaluate information can hinder use. Because of journal paywalls and the file-drawer problem, managers may have difficulty accessing potentially relevant scientific information in the first place (Piczak et al. 2022). Simultaneously, the ever-increasing amounts of information may challenge managers’ ability to parse what is useful (White et al. 2019). Best Available Scientific Information is context-specific and must be both scientifically rigorous and relevant to the management context (Ryan et al. 2018). In a world of limited time and resources to evaluate all possible sources of relevant information, managers must make decisions about how to filter and select what scientific information they will attend to and use. To date, how managers evaluate the scientific information they have has received significantly less attention than how managers find scientific information. Thus, we explore how managers evaluate the usefulness of science.

We examine two factors managers may use to evaluate the quality and usefulness of a piece of scientific information. The first factor is whether the piece of information has a generally accepted characteristic of quality science. The second factor is the extent to which the information conforms to pre-existing beliefs and knowledge, through the lens of cognitive and motivational biases. We consider each in turn.

Long- vs. short-term science as an indicator of research quality

There are many characteristics that influence research quality, such as experimental design and sample size. In ecology, length of time of the study is also an important characteristic. Long-term ecological research is widely recognized for the critical role it plays in understanding natural processes (Callahan 1984, Lindenmayer et al. 2012, Jones and Driscoll 2022). The scientific community recognizes its importance; compared to short-term ecological research, long-term ecological research is cited more frequently and is disproportionately present in higher-impact journals (Hughes et al. 2017). There is strong support among ecologists and evolutionary scientists for long-term ecological studies and agreement among the community on the impact long-term experiments have had on ecological understanding (Kuebbing et al. 2018). As well, long-term studies are more likely to show up in policy documents than short-term studies (Hughes et al. 2017). Long-term ecological research can be important for managers, providing context-based information at a scale relevant to managers (Lindenmayer et al. 2010, Jones and Driscoll 2022), monitoring data with high statistical power (White 2019), and a better opportunity to detect and understand ecological “surprises” (Doak et al. 2008, Anderson et al. 2017). Indeed, managers have expressed the need for long-term, decision-specific scientific information to address management issues such as climate adaptation (Littell et al. 2012). Thus, a reasonable heuristic or rule of thumb managers may use to filter and evaluate information is to value longer-term studies more highly than shorter-term studies, all else equal.

Cognitive and motivational biases

The mere presence or provision of information does not guarantee its use. For information to impact any individual’s judgment or decision, it must be accessed, attended to, and integrated into that individual’s general understanding of the phenomena of interest (Nguyen et al. 2017). Consequently, scientific communication that relies on an information-deficit model is unlikely to be effective in substantially influencing behavior (Toomey 2023). When people receive information, scientific or otherwise, they do not evaluate it in a vacuum, but rather, in light of their pre-existing values, beliefs, and prior knowledge (Newell et al. 2014, Heeren et al. 2017). One factor that influences the evaluation of evidence is the desire for cognitive consistency or the avoidance of cognitive dissonance (Festinger 1957, Harmon-Jones 2019). The desire for cognitive consistency can take many forms, such as the desire to see one’s in-group in a positive light and the desire to protect existing beliefs (especially those that are strongly held) from challenge. Other research, conducted through the lens of “motivated reasoning” indicates individuals may protect existing beliefs from challenge; that is, when evaluating information or engaging in reasoning, individuals may select or rely on cognitive processes that are more likely to lead them to support their pre-existing beliefs (Kunda 1990). However, maintaining existing beliefs can conflict with the goal to make accurate decisions, and as a result, negatively impact decision quality. For example, Kang and Kim (2022) found when experts felt their identity as an expert was called into question by negative performance feedback, experts exhibited increased overconfidence in their predictive abilities. The general desire to maintain existing belief structures leads to what psychologists refer to as confirmation bias, that is, a bias in favor of information that confirms existing beliefs and against information that challenges such beliefs (Sherman and Cohen 2006).

Importantly, this can happen even when people are motivated to be accurate and process information deliberatively. One of the mechanisms for the biased processing of information to protect existing beliefs is the disconfirmation bias in the evaluation of arguments (Edwards and Smith 1996). Evaluating the soundness of an argument is an exercise in whether the premises of an argument are true and support the arguments’ conclusion, and not whether one agrees or disagrees with the conclusion. However, soundness is evaluated in light of prior beliefs and people struggle to separate evaluations of weak and strong arguments from their agreement or disagreement with the conclusion; arguments that are incongruent with prior beliefs are on average rated as weaker and generate more refutations than congruent arguments (Edwards and Smith 1996). Consequently, exposure to counter-attitudinal information may trigger a “backfire effect” in which people become more confident in their prior beliefs after being exposed to contrary evidence (Taber and Lodge 2006). However, confirmation bias has its limits as people can reach a tipping point of incongruent information (Redlawsk et al. 2010). For natural resource managers, research they consider high quality may be more difficult to counter-argue and thus information from high-quality research may be less prone to disconfirmation bias in argument evaluation.

The present study

Natural resource managers are not immune to cognitive and motivational biases (Wilson et al. 2011, Heeren et al. 2017, Karns et al. 2018). Managers also value scientific information for multiple reasons, though it is unclear how they weigh different characteristics about scientific information when evaluating its usefulness. The goal of this study is to examine two possible characteristics managers may value in scientific information: the longevity of the study and whether it confirms existing beliefs. We are interested in looking at the direct effects and the interaction of these two characteristics. As a “gold standard” of science, is long-term information harder to disregard when it is incongruent with prior beliefs? Is one of the potential uses of long-term ecological research overcoming biased information processing through high quality science? We are guided by the following research questions:

RQ1: How do public land managers perceive and evaluate long-term and short-term ecological data?

RQ2: How do public managers perceive and evaluate confirming and disconfirming ecological data?

RQ3: How do characteristics of scientific studies (time frame) interact with manager’s pre-existing beliefs to influence evidence evaluation?

We focus our study on three management issues relevant to public land managers in the Pacific Northwest (Oregon and Washington), USA. We selected the Pacific Northwest as our area of study for two reasons. Our goal was to balance sample size and relevance. We chose the Pacific Northwest because it is a region with (1) a sufficiently large pool of potential respondents for statistical power, and (2) enough social-ecological similarity across forested landscapes in the region that we could develop a set of management issues that our pool would either be familiar or directly interface with. We selected our three management issues in collaboration with biophysical researchers and agency personnel working in the Pacific Northwest (PNW), with the goal of selecting three issues that ranged in how stable and strong managers’ attitudes would be. We examine salvage logging as a method to mitigate future fire behavior as our management issue where managers have strong prior beliefs: managers tend to agree with each other and have less variation between each other. We examine variable density thinning of mature growth stands as our management issue where managers have medium-strength prior beliefs. We examine translocation of plant species from hotter and drier seed zones as an adaptation strategy for climate change as our management issue where managers have the weakest prior beliefs: managers may not have strong opinions and higher variation between each other. We focus on two forms of evidence evaluation: usefulness of evidence for one’s job and soundness of arguments that use the evidence for management prescriptions. We measure usefulness of evidence under the assumption that this is more realistic to how managers consume and evaluate information in their day-to-day jobs, while we measure soundness of arguments to more closely replicate previous methodologies for studies on disconfirmation bias in the evaluation of arguments (Edwards and Smith 1996, Taber and Lodge 2006). We hypothesize the following:

H1: Respondents will prefer long-term studies to short-term studies.
 H1A: Respondents will evaluate long-term evidence as more useful than short-term evidence.
 H1B: Respondents will rate arguments using long-term evidence as more sound than arguments using short-term evidence.

H2: Managers will prefer confirming evidence to disconfirming evidence.
 H2A: Respondents will evaluate confirming evidence as more useful than disconfirming evidence.
 H2B: Respondents will rate arguments using confirming evidence as more sound than arguments using disconfirming evidence.

H3: Time frame will impact the strength of confirmation bias on information preferences.
 H3A: The time frame of evidence will moderate the effect of confirmation bias on usefulness.
 H3B: The time frame of evidence will moderate the effect of confirmation bias on soundness.

METHODS

Subjects

We collected data from a web-based survey sent to public land managers working in Oregon and Washington, USA. For the purpose of this study, “manager” does not refer to a specific job title, rather anyone who identifies all or a significant portion of their job entails planning or implementing management actions on a landscape. Managers in this context do not include positions such as administrative staff (Human Resources, Information Technology, etc.), field technicians, or research scientists. We targeted state and federal managers working for the Oregon Department of Forestry (ODF), Washington Department of Natural Resources (WDNR), U.S. National Park Service (NPS), U.S. Fish and Wildlife Service (FWS), U.S. Bureau of Land Management (BLM), and U.S. Forest Service (USFS). We filed state-level public records requests and Freedom of Information Act requests for contact information for all employees working for these agencies in Oregon and Washington. We received information from the Department of Interior (DOI) agencies (NPS, FWS, BLM), WDNR, and ODF.

For each contact list we received, we removed individuals in Human Resources, Information Technology, Field Technician, and Research Scientist positions. When we were unsure what a position entailed, we left the individuals in the sample pool. Our final sample pool for Department of the Interior agencies and state agencies was 2273 potential respondents. Potential respondents were emailed by the research team and invited to participate in the study. Potential respondents received one initial invitation and up to two reminders to complete the survey.

For the USFS, we were not able to gain direct access to the sample population. Instead, our survey was sent on our behalf to approximately 450 potential respondents via internal USFS listservs that included managers in Oregon and Washington. Potential respondents in the USFS were contacted once and did not receive reminders to complete the survey.

Study design

Our study was approved by the Oregon State University Institutional Review Board, Protocol HE-2023-183, HE-2023-348, and HE-2023-399. We conducted a web-based survey using the Qualtrics survey platform (for a complete list of questions used in this study, see Appendix 1). Respondents were invited to participate in a survey about long-term ecological data in the PNW. Respondents were told the study would assess their attitudes about a variety of management issues and asked them to assess how useful hypothetical examples of scientific studies were for their job. The survey included descriptive measures and a 2x2 experimental design. Respondents were not told the survey included an experiment, that there were multiple conditions, or that the purpose of the study was to test the effect of time frame and confirmation bias on evidence evaluation. First, respondents answered a filter question designed to remove non-managers. We then measured respondents’ beliefs about how useful, necessary, and effective salvage logging, thinning, and translocation were. Belief questions were measured on a 5-point bi-polar scale (-2 to 2) from “Strongly Disagree” to “Strongly Agree.”

Next, each respondent was randomly assigned to the long-term or short-term condition for the entire survey. To reduce cognitive load, we varied time frame across respondents but not across management issues within respondents. Respondents then saw each management issue in a randomized order. For each issue, respondents were randomly assigned to either the positive or negative condition. In the positive condition, respondents received evidence from scientific studies that suggested the management issue had positive effects. In the negative condition, respondents received evidence from scientific studies that suggested the management issue had a harmful impact or did not work as intended (e.g., no positive effect). Respondents saw results from three scientific studies for each management issue and saw all three management issues. Evidence statements covered the same topic and mirrored each other across the positive and negative condition. For example, in the positive condition, respondents would read that a study suggested translocation would assist native pollinators, while in the negative condition respondents would read that translocation would not assist native pollinators. To illustrate the full experimental design, we provide an example of a hypothetical respondent. The respondent would start the survey and be randomly sorted into the long-term condition. They would be randomly assigned to the positive condition for variable density thinning, and see three positive, long-term evidence statements for variable density thinning. Then they would be randomly assigned to the negative condition for translocation and see three negative, long-term evidence statements about translocation. Finally, they would be randomly assigned into the negative condition for salvage logging and see three negative, long-term evidence statements about translocation.

Respondents rated how useful each evidence statement was for their job with a 5-point Likert scale from “Not at all useful” to “Extremely useful.” In order to reduce cognitive load, respondents evaluated one randomly selected argument for each management issue. Respondents rated the soundness of the argument prescribing a management action based on the hypothetical survey results with a 5-point Likert scale from “Not at all sound” to “Extremely sound” and were asked to explain their answer in an open-ended response. Example evidence statements and arguments are provided in Table 1. Finally, respondents answered demographics questions, including gender, ethnicity, highest level of education completed, years worked in natural resource management, which agency they worked for and which ecoregion they worked in, and the natural resource management topic areas most relevant to their job (expertise). We used agency employment and ecoregion to describe the sample. We included gender, ethnicity, education, and expertise in models as statistical controls.

Variable transformation

We used the belief statements to categorize whether respondents had received confirming or disconfirming evidence and arguments post-hoc. For each management issue, we categorized respondents as either pro or anti based on the average of their belief statements. Beliefs about each management issue were calculated by averaging respondent’s beliefs about how (1) good, (2) effective, and (3) necessary each management action is (Strong Disagree to Strongly Agree, -2 to 2). These items had sufficient internal reliability as measured through Cronbach’s alpha for each management issue to warrant combining them into a single index for each management issue (salvage logging: α = 0.88, variable density thinning: α = 0.77, translocation: α = 0.91). Respondents were then categorized into pro or anti for each management issue based on their beliefs such that (x̄ > 0 = pro, x̄ ≤ 0 = anti). Respondents were coded in the confirmation condition if they were pro for a management issue and received positive evidence, or were anti for a management issue and received negative evidence. Respondents were coded in the disconfirmation condition if they were pro for a management issue and received negative evidence, or were anti for a management issue and received positive evidence. Thus, for each management issue, respondents were in one of the following conditions (Table 2):

1. Long-term, disconfirming evidence.
2. Long-term, confirming evidence.
3. Short-term, disconfirming evidence.
4. Short-term, confirming evidence.

For each management issue, we averaged the three evidence examples into one measure of usefulness. We transformed gender into a binary numeric variable (0 = male, 1 = female). Because we combined “Other/Prefer not to say” in our survey, we removed this group from our sample when analyzing gender. We transformed ethnicity into a binary numeric variable (0 = white, 1 = non-white). In this case, “white” respondents are those who identified as white and no other ethnicity. Two important limitations of this approach are (1) we have categorized all people of color into one group, which unrealistically characterizes them as a monolith, and (2) we are not able to distinguish biracial and multiracial individuals in our sample who may have a significantly different lived experience than other people of color. Ninety-five percent of our sample reported having either a bachelor’s or a graduate degree. Consequently, we recoded education into a binary variable (0 = bachelor’s degree or less, 1 = graduate degree). For expertise, we presented respondents with a variety of topics in natural resource management (e.g., forestry and silviculture, fire management, hydrology, recreation, etc.). Respondents could choose up to 3 options from the 12 we presented, including a fill-in-the-blank option. Because our hypothetical studies focus on salvage logging, thinning, and translocation of plant species, we recoded expertise into a binary numeric variable. If respondents checked at least one box for the topics “Forestry and silviculture,” “Fire management,” and “Plant biology,” they were coded 1 for expertise. If respondents did not check any of those boxes they were coded 0 for expertise.

Analysis

We describe sample characteristics, the average beliefs about each management issue, and the average usefulness of the evidence statements and soundness of the arguments across all conditions. ChatGPT was used to assist in writing code for data cleaning and preparation and statistical analysis. Data were analyzed in R ver. 4.1.1. Open-ended responses were inductively thematically coded in NVivo ver. 12 to summarize the rationale managers provided for why the argument was sound or unsound. We developed a codebook through a three-step process of open coding, preliminary refinement, and final refinement. We summarize the most commonly occurring codes.

To test our hypotheses, we used linear regression with robust standard error. We treated our dependent variables as continuous. Our independent variables were either binary or continuous. To test for a moderating effect of time frame on confirmation, we included an interaction term. We used an independent link function, which assumes our independent and dependent variables had a linear relationship. We used the lm() function in R to test our models. For each of our models, the Shapiro-Wilks test of normality of residuals indicated our residuals were heteroscedastic (p < 0.05). To address this issue, we used robust standard errors. We used the “sandwich” and “lmtest” packages in R to compute robust standard errors for our regression coefficients. Results were similar with normal and robust standard errors. We report the variable coefficients from the robust standard error models (Table 3).

RESULTS

The data collected are available in the Environmental Data Initiative Repository (see Data Statement).

Sample characteristics

For the DOI and State agencies, our initial pool of potential respondents was 2273. Five hundred sixty-eight people clicked on the survey (response rate 25%), and 461 made it past the initial filter question (20% adjusted response rate). Because we did not confine managers to a subset of job titles, our sample frame included both managers and non-managers, which may have impacted the study response rate.

For USFS, we were not able to calculate an exact response rate. The research team did not distribute the survey, and it is unknown how many names were redundant across the USFS internal listservs. Thirty-eight people clicked on the survey and 33 made it past the initial filter question. We combined these two subsamples in subsequent analyses (n = 494).

Most of our respondents identified as male (66% male) and white only (92% white only), with a bachelor’s degree or less (63%) and expertise in fire management, forestry, and/or plant biology (86%). The median respondent had worked in natural resource management for 19 years. Because relatively few respondents worked for NPS and FWS, we combined them into one category, USFWS/NPS. Over half of the respondents worked for state agencies and in the Western Cascades or Coast Range (Fig. 1).

Beliefs about salvage logging, variable density thinning, and translocation of plant species

Respondents were generally positive toward salvage logging and variable density thinning, and more divided toward translocation. Seventy-one percent of respondents were sorted into the pro category for salvage logging, 79% were sorted into the pro category for variable density thinning, and 59% were sorted into the pro category for translocation. The similar beliefs toward salvage logging and variable density thinning suggested respondents may hold equally strong beliefs toward those management issues, rather than having strong beliefs toward salvage logging and moderate beliefs toward variable density thinning as originally intended. However, the divided responses toward translocation suggested it is an emerging issue that managers have not reached a consensus about yet.

Usefulness of scientific evidence and soundness of arguments

On average, respondents found the evidence statements slightly useful for all three management issues. In comparison, respondents had more negative evaluations of the soundness of arguments across all three management issues (Fig. 2). This was reflected in the open-ended responses describing respondents’ evaluations of soundness (Table 4). Of the 840 open-ended responses, 79% included at least one challenge to the argument while 24% included at least one affirmation (responses could both affirm and challenge the argument). Across all responses, the most common challenges were that other factors influence decision making (27%), the argument is missing important contextual considerations (16%), and the study time frame was not long enough (12%). In comparison, the most common affirmation was to offer support but with caveats (12%).

Regression analysis

Results of our regression indicated that gender and years in natural resource management did not impact (p > 0.05) respondents’ evaluation of information. Expertise and ethnicity had minimal and inconsistent effects. Education had a significant impact on the usefulness of evidence about salvage logging (t = 2.93, p = 0.004), variable density thinning (t = 3.14, p = 0.002), and translocation (t = 2.16, p = 0.032). Across all three management issues, respondents with graduate degrees were more likely than respondents with bachelor degrees or less to rate the provided information as useful. We found no significant effect of education on soundness (p > 0.05).

Hypothesis 1: Respondents will prefer long-term studies to short-term studies

For our three models estimating the usefulness of scientific information, we found no significant effect of time frame on usefulness. Thus, we found no support for Hypothesis 1A: respondents will evaluate long-term evidence as more useful than short-term evidence.

For our three models estimating the soundness of an argument using scientific information, we found mixed results for time frame. We did not find a significant effect of time frame on soundness for salvage logging or translocation. However, arguments for variable density thinning that used long-term information were rated more sound than those that used short-term data (t = 3.32, p = 0.001). Thus, we found mixed support for Hypothesis 1B: respondents will rate arguments using long-term evidence as more sound than arguments using short-term evidence.

Qualitatively, time frame seemed to have affected respondent evaluations of soundness. Many of the same themes were present in the open-ended responses for participants in the long-term and short-term conditions. Regardless of time frame, many more respondents mentioned challenges to the argument than affirmations, and the most common challenge was to highlight that other factors affect decision making. However, 5% of responses in the long-term condition described the time frame of the study was too short, compared to 22% of responses in the short-term condition.

Hypothesis 2: Managers will prefer confirming evidence to disconfirming evidence

For our three models estimating the usefulness of scientific information, we found mixed results for confirmation. We did not find a significant effect of confirmation on the uselessness of information about translocation. However, confirmation had a significant effect for the usefulness of salvage logging (t = 3.69, p < 0.001) and variable density thinning (t = 3.24, p = 0.001) information. Translocation is an emerging issue and managers may not yet have strong beliefs about it that they would seek to confirm and protect. We thus found partial support for Hypothesis 2A: respondents will evaluate confirming evidence as more useful than disconfirming evidence.

We found a significant effect of confirmation on the soundness of an argument for salvage logging (t = 3.28, p = 0.001), variable density thinning (t = 6.51, p < 0.001), and translocation (t = 3.65, p < 0.001). Thus, we found moderate support for Hypothesis 2B: respondents will rate arguments using confirming evidence as more sound than arguments using disconfirming evidence.

While open-ended responses were more likely to mention challenges than affirmations for both conditions, more responses included affirmations in the confirming condition (35%) than the disconfirming condition (16%). Similarly, more responses discussed challenges in the disconfirming condition (86%) than the confirming condition (70%).

Hypothesis 3: Time frame will impact the strength of confirmation bias on information preferences

We found no significant effect (p < 0.05) for the interaction between confirmation and time frame on the evaluation of the usefulness of scientific information. We found no significant effect for the interaction between confirmation and time frame on the soundness of arguments about salvage logging and translocation. However, we found a significant interaction between time frame and confirmation for variable density thinning (t = -3.00, p = .003).Thus, we found minimal support for Hypothesis 3.

DISCUSSION

Confirmation bias in evidence evaluation

Managers recognize the importance of science and scientific evidence for rigorous decision making (Walsh et al. 2015, Kadykalo et al. 2021). Our results provide further support; across management issues and experimental conditions, respondents rated evidence statements on average neutrally or positively useful. However, scientific information is not rated equally useful. We found respondents with advanced degrees tended to rate the scientific information as more useful than other respondents. Our results suggest attributes of managers influence how they use and evaluate science; this warrants further examination. Further, characteristics of the information affect how managers evaluate scientific evidence. Namely, for issues where managers have stronger pre-existing beliefs, scientific evidence that confirms those beliefs is rated as more useful than evidence that challenges them. Put simply, managers in this study tended to engage in confirmation bias when evaluating scientific evidence for certain management actions. This result is in line with previous studies on natural resource managers, which have found other cognitive (Wilson et al. 2011) and motivational (Heeren et al. 2017, Karns et al. 2018) biases influencing natural resource managers. Our results extend this work, shedding light on one way cognitive biases influence manager decision making by shaping how managers interpret new scientific information.

Disconfirmation in evidence evaluation is not unique to forest management in the PNW or to natural resource management more broadly. It is a phenomenon of human cognition. However, we also found the effect of confirmation on evidence evaluation was not consistent across all our experimental conditions. Thus, it is important to acknowledge that context, namely, the degree to which a natural resource management issue is entrenched, controversial, or novel may influence the extent to which cognitive biases distort evidence-based decision making. Understanding where and how biases shape natural resource management decision making is important for debiasing efforts and making governance transparent and defensible. Debiasing strategies are most effective when they align with the decision maker, the context, and the bias in question (Soll et al. 2015). Possible strategies to address confirmation bias are numerous (see Fischhoff 1982, Soll et al. 2015 for reviews). Although training to reduce bias has had a mixed history, recent studies of non-managers show promise in reducing confirmation bias by teaching evidence evaluation strategies (Morewedge et al. 2015, Sellier et al. 2019). Our results do not point to a specific debiasing strategy that will be most effective, however, it will be important to ground best practices of debiasing with the search strategies managers use to find information. To that end, future work may focus on designing interventions that leverage or nudge pre-existing search strategies used by managers when finding and evaluating scientific information.

Long-term ecological research

We found limited effect of time frame on the evaluations of evidence usefulness. Although this may seem inconsistent with previous results that suggest managers value long-term data (e.g., Littell et al. 2012), we hesitate to suggest that long-term data does not necessarily have any additional utility for managers compared to short-term data on the same subject. Rather, our results suggest some critical reflection of what makes long-term data useful is warranted. In our study we manipulated the length of time a study was conducted while holding all else constant. However, many of the professed values of long-term data are not merely the length of time the data has been collected, but the implications or consequences of that length, for example, the ability to provide deep understanding of a particular site/context at management-relevant scales (Jones and Driscoll 2022) and a platform for collaborative and multidisciplinary research (Lindenmayer et al. 2012). Indeed, these values are reflected in the open-ended responses to how managers evaluated argument soundness. One of the most mentioned themes was context; managers needed to know if or how the study applied to their context before they would consider the action prescription sound. Thus, although managers equally valued long- and short-term studies when evaluating usefulness, this is not to suggest long-term research does not have particular importance to managers. Rather, length of time in and of itself may not be persuasive, and science communicators may want to emphasize the way their study addresses manager evaluative criteria.

Although the scientific community recognizes the importance of long-term ecological research, financial support is declining (Vucetich et al. 2020). Co-production of long-term research may be an important avenue to ensure manager evaluative criteria are considered by scientists when conducting studies. Further, coproduction may provide a fruitful avenue to address the multiple challenges of declining support of long-term funding by traditional funding organizations and the simultaneous challenge managers face of scientific information overload and insufficient scientific evidence for their particular challenges.

Soundness of action prescriptions

We found managers tended to rate the soundness of action prescriptions lower than the usefulness of scientific evidence. In open-ended responses, managers generated many more refutations to the argument than affirmations. The most mentioned themes in open-ended responses were the need to know more about how the study aligned with their particular context, and that factors other than the results of the example studies also affect their decision making. These are reasonable and expected refutations; managers are often expected to manage landscapes for a diversity of values and goals. Further, scientific information by itself is not sufficient to determine the proper course of action on a landscape; management must be guided by science and social values, while following existing policy. Though managers do show flexibility and will adapt their behavior in light of new evidence (Walsh et al. 2015), managers of various natural resources acknowledge science is not the only factor in their decision making (see Kadykalo et al. 2021 and Rapp et al. 2020 for examples from fisheries and wildfire respectively). Thus, the effect of scientific evidence on manager decision making may not be readily apparent if one only examines the final decisions.

To better understand how science informs decision making requires a stronger understanding of not only what sources of information managers use and how they find them, but also at what steps in the decision-making process scientific information is used, and how it affects those steps. Science can inform and shape decisions at multiple points along the decision-making process, from shaping the scope of the decision to informing the construction of alternatives to guiding selection between them (Mills and Clark 2001, Hunter et al. 2020). At each step, different science may be necessary and used in different ways. This work begins to untangle not only how managers find scientific information, but how they begin to evaluate it and use it in their decision making. Future research should shed further light on the ways managers make decisions and apply scientific information along the way, including the way cognitive and motivational biases may impede decision making.

Additionally, many factors contribute to the quality of a research study, from the insightfulness of the research question, the quality of the research design, and the rigor of the analysis, and several tools exist to help managers evaluate scientific evidence (Mupepele et al. 2016, Christie et al. 2023). Our study examined only two factors managers may consider, and only one that could be indicative of research quality. Although managers can and should be involved in the design and conduct of research projects, inevitably managers will need to evaluate information from existing studies. Thus, it is useful to understand what factors are important for them, and in particular, how they weigh characteristics like recency, experimental design, and proximity to their problem context against each other when considering mixed results. We recommend further research to identify what characteristics managers value, and how that compares to the norms of the scientific community and prescriptive models of evidence evaluation.

CONCLUSION

Natural resource managers use and value scientific information when making decisions about how to best manage their landscapes. However, there are barriers to the use of scientific evidence external to managers (paywalls, insufficient capacity, information overload, etc.) and, as our results show, internal to managers. In our survey, managers generally found scientific evidence useful, but preferred information that confirmed their existing beliefs to information that challenged, highlighting one of the ways confirmation bias can shape land management. Our results also shed light on the way managers may value long-term scientific information. We found that ceteris paribus, longer studies are not valued more highly by managers than short-term studies. This is not to suggest that long-term ecological science is not important or does not have additional management implications than short-term work, but instead suggests there are attributes that are correlated but not inherently a part of long-term research that makes it especially valuable for management. We encourage scientists to consider the way these valuable characteristics (place-based, co-produced, management-relevant scales) can be brought into short-term studies for more actionable science. Science communicators and managers should consider how pre-existing values and beliefs shape the process of using information in decision making. Cognitive and motivational biases are very common in human decision making and not the result of moral or professional failing. However, their presence still undermines decision quality. Vigilance and humility about their effects from managers, scientists, and science communicators alike will be important for transparent and defensible decision making.

LITERATURE CITED

Anderson, S. C., T. A. Branch, A. B. Cooper, and N. K. Dulvy. 2017. Black-swan events in animal populations. Proceedings of the National Academy of Sciences 114(12):3252-3257. https://doi.org/10.1073/pnas.1611525114

Artelle, K. A., J. D. Reynolds, A. Treves, J. C. Walsh, P. C. Paquet, and C. T. Darimont. 2018. Hallmarks of science missing from North American wildlife management. Science Advances 4:eaao0167. https://doi.org/10.1126/sciadv.aao0167

Barrett, K., and S. L. Rodriguez. 2021. What sources are natural resource managers using to make decisions? Journal of Wildlife Management 85(8):1543-1553. https://doi.org/10.1002/jwmg.22112

Bruskotter, J. T., J. A. Vucetich, D. W. Smith, M. P. Nelson, G. R. Karns, and R. O. Peterson. 2017. The role of science in understanding (and saving) large carnivores: a response to Allen and colleagues. Food Webs 13:46-48. https://doi.org/10.1016/j.fooweb.2017.05.004

Callahan, J. T. 1984. Long-term ecological research. BioScience 34(6):363-367. https://doi.org/10.2307/1309727

Christie, A. P., D. Abecasis, M. Adjeroud, J. C. Alonso, T. Amano, A. Anton, B. P. Baldigo, R. Barrientos, J. E. Bicknell, D. A. Buhl, J. Cebrian, R. S. Ceia, L. Cibils-Martina, S. Clarke, J. Claudet, M. D. Craig, D. Davoult, A. De Backer, M. K. Donovan, T. D. Eddy, F. M. França, J. P. A. Gardner, B. P. Harris, A. Huusko, I. L. Jones, B. P. Kelaher, J. S. Kotiaho, A. López-Baucells, H. L. Major, A. Mäki-Petäys, B. Martín, C. A. Martín, P. A. Martin, D. Mateos-Molina, R. A. McConnaughey, M. Meroni, C. F. J. Meyer, K. Mills, M. Montefalcone, N. Noreika, C. Palacín, A. Pande, C. R. Pitcher, C. Ponce, M. Rinella, R. Rocha, M. C. Ruiz-Delgado, J. J. Schmitter-Soto, J. A. Shaffer, S. Sharma, A. A. Sher, D. Stagnol, T. R. Stanley, K. D. E. Stokesbury, A. Torres, O. Tully, T. Vehanen, C. Watts, Q. Zhao, and W. J. Sutherland. 2020. Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences. Nature Communications 11(1):6377. https://doi.org/10.1038/s41467-020-20142-y

Christie, A. P., T. Amano, P. A. Martin, G. E. Shackelford, B. I. Simmons, and W. J. Sutherland. 2019. Simple study designs in ecology produce inaccurate estimates of biodiversity responses. Journal of Applied Ecology 56(12):2742-2754. https://doi.org/10.1111/1365-2664.13499

Christie, A. P., W. H. Morgan, N. Salafsky, T. B. White, R. Irvine, N. Boenisch, R. M. Chiaravalloti, K. Kincaid, A. M. Rezaie, H. Yamashita, and W. J. Sutherland. 2023. Assessing diverse evidence to improve conservation decision-making. Conservation Science and Practice 5(10):e13024. https://doi.org/10.1111/csp2.13024

Cook, C. N., M. Hockings, and R. W. Carter. 2010. Conservation in the dark? The information used to support management decisions. Frontiers in Ecology and the Environment 8(4):181-188. https://doi.org/10.1890/090020

Cook, C. N., M. B. Mascia, M. W. Schwartz, H. P. Possingham, and R. A. Fuller. 2013. Achieving conservation science that bridges the knowledge-action boundary. Conservation Biology 27(4):669-678. https://doi.org/10.1111/cobi.12050

Doak, D. F., J. A. Estes, B. S. Halpern, U. Jacob, D. R. Lindberg, J. Lovvorn, D. H. Monson, M. T. Tinker, T. M. Williams, J. T. Wootton, I. Carroll, M. Emmerson, F. Micheli, and M. Novak. 2008. Understanding and predicting ecological dynamics: Are major surprises inevitable? Ecology 98(4):952-961. https://doi.org/10.1890/07-0965.1

Edwards, K., and E. Smith. 1996. A disconfirmation bias in the evaluation of arguments. Attitudes and Social Cognition 7(1):5-24. https://doi.org/10.1037/0022-3514.71.1.5

Festinger, L. 1957. A theory of cognitive dissonance. Stanford University Press, Redwood City, California, USA. https://doi.org/10.1515/9781503620766

Fischer, A. P., K. Vance-Borland, K. M. Burnett, S. Hummel, J. H. Creighton, S. L. Johnson, and L. Jasny. 2014. Does the social capital in networks of “fish and fire” scientists and managers suggest learning? Society and Natural Resources 27(7):671-688. https://doi.org/10.1080/08941920.2014.901463

Fischhoff, B. 1982. Debiasing. Pages 422-444 in D. Kahneman, P. Slovic, and A. Tversky, editors. Judgment under uncertainty: heuristics and biases. Cambridge University Press, Cambridge, UK. https://doi.org/10.1017/CBO9780511809477.032

Harmon-Jones, E., editor. 2019. Cognitive dissonance: reexamining a pivotal theory in psychology. Second edition. American Psychological Association, Washington, D.C., USA. https://doi.org/10.1037/0000135-000

Heeren, A., G. Karns, J. Bruskotter, E. Toman, R. S. Wilson, and H. Szarek. 2017. Expert judgment and uncertainty regarding the protection of imperiled species. Conservation Biology 31(3):657-665. https://doi.org/10.1111/cobi.12838

Hughes, B. B., R. Beas-Luna, A. K. Barner, K. Brewitt, D. R. Brumbaugh, E. B. Cerny-Chipman, S. L. Close, K. E. Coblentz, K. L. De Nesnera, S. T. Drobnitch, J. D. Figurski, B. Focht, M. Friedman, J. Freiwald, K. K. Heady, W. N. Heady, A. Hettinger, A. Johnson, K. A. Karr, B. Mahoney, M. M. Moritsch, A. M. K. Osterback, J. Reimer, J. Robinson, T. Rohrer, J. M. Rose, M. Sabal, L. M. Segui, C. Shen, J. Sullivan, R. Zuercher, P. T. Raimondi, B. A. Menge, K. Grorud-Colvert, M. Novak, and M. H. Carr. 2017. Long-term studies contribute disproportionately to ecology and policy. BioScience 67(3):271–281. https://doi.org/10.1093/biosci/biw185

Hunter, M. E., M. M. Colavito, and V. Wright. 2020. The use of science in wildland fire management: a review of barriers and facilitators. Current Forestry Reports 6(4):354-367. https://doi.org/10.1007/s40725-020-00127-2

Jones, J. A., and C. T. Driscoll. 2022. Long-term ecological research on ecosystem responses to climate change. BioScience 72(9):814–826. https://doi.org/10.1093/biosci/biac021

Kadykalo, A. N., S. J. Cooke, and N. Young. 2021. The role of western-based scientific, Indigenous and local knowledge in wildlife management and conservation. People and Nature 3(3):610-626. https://doi.org/10.1002/pan3.10194

Kang, S., and J. W. Kim. 2022. The fragility of experts: a moderated-mediation model of expertise, expert identity threat, and overprecision. Academy of Management Journal 65(2):577-605. https://doi.org/10.5465/amj.2019.0899

Karns, G. R., A. Heeren, E. Toman, R. S. Wilson, H. K. Szarek, and J. T. Bruskotter. 2018. Should grizzly bears be hunted or protected? Social and organizational affiliations influence scientific judgments. Canadian Wildlife Biology & Management 7(1):18-30.

Kuebbing, S. E., A. P. Reimer, S. A. Rosenthal, G. Feinberg, A. Leiserowitz, J. A. Lau, and M. A. Bradford. 2018. Long-term research in ecology and evolution: a survey of challenges and opportunities. Ecological Monographs 88(2):245-258. https://doi.org/10.1002/ecm.1289

Kunda, Z. 1990. The case for motivated reasoning. Psychological Bulletin 108(3):480-498. https://doi.org/10.1037/0033-2909.108.3.480

Lindenmayer, D. B., G. E. Likens, A. Andersen, D. Bowman, C. M. Bull, E. Burns, C. R. Dickman, A. A. Hoffmann, D. A. Keith, M. J. Liddell, A. J. Lowe, D. J. Metcalfe, S. R. Phinn, J. Russell-Smith, N. Thurgate, and G. M. Wardle. 2012. Value of long-term ecological studies. Austral Ecology 37(7):745-757. https://doi.org/10.1111/j.1442-9993.2011.02351.x

Lindenmayer, D. B., G. E. Likens, C. J. Krebs, and R. J. Hobbs. 2010. Improved probability of detection of ecological “surprises.” Proceedings of the National Academy of Sciences 107(51):21957-21962. https://doi.org/10.1073/pnas.1015696107

Littell, J. S., D. L. Peterson, C. I. Millar, and K. A. O’Halloran. 2012. U.S. National forests adapt to climate change through science-management partnerships. Climatic Change 110(1-2):269-296. https://doi.org/10.1007/s10584-011-0066-0

Mills, T. J., and R. N. Clark. 2001. Roles of research scientists in natural resource decision-making. Forest Ecology and Management 153:189-198. https://doi.org/10.1016/S0378-1127(01)00461-3

Morewedge, C. K., H. Yoon, I. Scopelliti, C. W. Symborski, J. H. Korris, and K. S. Kassam. 2015. Debiasing decisions: improved decision making with a single training intervention. Policy Insights from the Behavioral and Brain Sciences 2(1):129-140. https://doi.org/10.1177/2372732215600886

Mupepele, A. C., J. C. Walsh, W. J. Sutherland, and C. F. Dormann. 2016. An evidence assessment tool for ecosystem services and conservation studies. Ecological Applications 26(5):1295-1301. https://doi.org/10.1890/15-0595

Newell, B. R., R. I. McDonald, M. Brewer, and B. K. Hayes. 2014. The psychology of environmental decisions. Annual Review of Environmental Resources 39:443-467. https://doi.org/10.1146/annurev-environ-010713-094623

Nguyen, V. M., N. Young, and S. J. Cooke. 2017. A roadmap for knowledge exchange and mobilization research in conservation and natural resource management. Conservation Biology 31(4):789-798. https://doi.org/10.1111/cobi.12857

Piczak, M. L., A. N. Kadykalo, S. J. Cooke, and N. Young. 2022. Natural resource managers use and value Western-based science, but barriers to access persist. Environmental Management 69(1):17-30. https://doi.org/10.1007/s00267-021-01558-8

Pullin, A. S., and T. M. Knight. 2001. Effectiveness in conservation practice: pointers from medicine and public health. Conservation Biology 15(1):50-54. https://doi.org/10.1111/j.1523-1739.2001.99499.x

Pullin, A. S., and T. M. Knight. 2003. Support for decision making in conservation practice: an evidence-based approach. Journal for Nature Conservation 11(2):83-90. https://doi.org/10.1078/1617-1381-00040

Pullin, A. S., T. M. Knight, D. A. Stone, and K. Charman. 2004. Do conservation managers use scientific evidence to support their decision-making? Biological Conservation 119(2):245-252. https://doi.org/10.1016/j.biocon.2003.11.007

Rapp, C., E. Rabung, R. S. Wilson, and E. Toman. 2020. Wildfire decision support tools: an exploratory study of use in the United States. International Journal of Wildland Fire 29(7):581-594. https://doi.org/10.1071/WF19131

Redlawsk, D. P., A. J. W. Civettini, and K. M. Emmerson. 2010. The affective tipping point: do motivated reasoners ever “get it”? Political Psychology 31(4):563-593. https://doi.org/10.1111/j.1467-9221.2010.00772.x

Roux, D. J., K. H. Rogers, H. C. Biggs, P. J. Ashton and A. Sergeant. 2006. Bridging the science–management divide: moving from unidirectional knowledge transfer to knowledge interfacing and sharing. Ecology and Society 11(1):4. https://doi.org/10.5751/ES-01643-110104

Ryan, C. M., L. K. Cerveny, T. L. Robinson, and D. J. Blahna. 2018. Implementing the 2012 forest planning rule: best available scientific information in forest planning assessments. Forest Science 64(2):159-169. https://doi.org/10.1093/forsci/fxx004

Sellier, A.-L., I. Scopelliti, and C. K. Morewedge. 2019. Debiasing training improves decision making in the field. Psychological Science 30(9):1371-1379. https://doi.org/10.1177/0956797619861429

Sherman, D. K., and G. L. Cohen. 2006. The psychology of self-defense: self-affirmation theory. Advances in Experimental Social Psychology 38:183-242. https://doi.org/10.1016/S0065-2601(06)38004-5

Soll, J. B., K. L. Milkman, and J. W. Payne. 2015. A user’s guide to debiasing. Pages 924-951 in G. Keren and G. Wu, editors. The Wiley Blackwell handbook of judgment and decision making II. First edition. John Wiley and Sons. https://doi.org/10.1002/9781118468333.ch33

Taber, C. S., and M. Lodge. 2006. Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science 50(3):755-769. https://doi.org/10.1111/j.1540-5907.2006.00214.x

Toomey, A. H. 2023. Why facts don’t change minds: insights from cognitive science for the improved communication of conservation research. Biological Conservation 278:109886. https://doi.org/10.1016/j.biocon.2022.109886

Vucetich, J. A., M. P. Nelson, and J. T. Bruskotter. 2020. What drives declining support for long-term ecological research? BioScience 70(2):168-173. https://doi.org/10.1093/biosci/biz151

Walsh, J. C., L. V. Dicks, C. M. Raymond, and W. J. Sutherland. 2019. A typology of barriers and enablers of scientific evidence use in conservation practice. Journal of Environmental Management 250:109481. https://doi.org/10.1016/j.jenvman.2019.109481

Walsh, J. C., L. V. Dicks, and W. J. Sutherland. 2015. The effect of scientific evidence on conservation practitioners’ management decisions. Conservation Biology 29(1):88-98. https://doi.org/10.1111/cobi.12370

White, E. M., K. Lindberg, E. J. Davis, and T. A. Spies. 2019. Use of science and modeling by practitioners in landscape-scale management decisions. Journal of Forestry 117(3):267-279. https://doi.org/10.1093/jofore/fvz007

White, E. R. 2019. Minimum time required to detect population trends: the need for long-term monitoring programs. BioScience 69(1):40-46. https://doi.org/10.1093/biosci/biy144

Wilson, R. S., P. L. Winter, L. A. Maguire, and T. Ascher. 2011. Managing wildfire events: risk-based decision making among a group of federal fire managers. Risk Analysis 31(5):805-818. https://doi.org/10.1111/j.1539-6924.2010.01534.x