A novel methodology to assess fuel treatment effectiveness: application to California’s forests

Introduction

The anthropogenic modification of wildland vegetation has a long history in the fire-prone western United States (US). Native Americans modified landscapes to maintain and create open space, encourage wildlife habitat and forage, and enhance the growth of medicinal and culturally important plants (Van Wagtendonk et al. 2018; Roos et al. 2021; Knight et al. 2024). European settlement led to elimination of indigenous burning practices, vastly altering the biophysical settings, especially in areas of California and the greater Southwest (Minnich 2008). Extensive livestock grazing in mid- and low-elevation areas led to a depletion of grass, thus disrupting short-period burn intervals. A series of disastrous wildfires in that same period helped lead to a national policy of full wildfire suppression, although this was not universally seen as the most appropriate course of action (Stephens and Ruth 2005; Pyne 2015) and more recently has been demonstrated to have caused pattern shifts in fire regimes and ecosystem functionality (Parks and Abatzoglou 2020; Hagmann et al. 2021). These shifts have coincided with increases in critical fire weather metrics (e.g. temperature) (Westerling et al. 2006; Littell et al. 2009) and increases in anthropogenic ignitions (Balch et al. 2017), resulting in more severe and more destructive wildfires (Parks and Abatzoglou 2020; Modaresi Rad et al. 2023).

Wildland fuels (i.e. vegetation), weather and topography are the principal drivers of fire behavior. However, only fuels can be reasonably modified through landscape management actions. Since around the turn of the current century, there has been a marked increase by local, state and federal government entities to increase the impact and spatial scale of fuel management treatments to combat the effects of an ecologically misaligned fire suppression doctrine. These efforts tend to focus on both reducing wildfire risk to human interests and promoting resilient forests and rangelands (Forest Management Task Force 2021; USDA Forest Service and US Department of the Interior 2001; US Forest Service 2022). The four principles of fuel reduction within forests include (1) reduce surface fuels, (2) increase height to live crown, (3) decrease crown density, and (4) preserve large specimens of fire-resistant species (Agee and Skinner 2005). Treatment options for achieving fuel reduction include planned and unplanned wildland fire (i.e. prescribed fire and natural ignitions/wildfire) as well as mechanized equipment (e.g. hand tools or large machinery) (Prichard et al. 2021). The use of biological (e.g. grazing) and chemical applications is less prevalent (Van Wagtendonk et al. 2018). Fuel treatments are often referenced as ‘fuel reduction’; however, in many cases, they include treatments that rearrange existing fuels without a removal component, thus, ‘reduction’ may be misleading.

Given the considerable cost of implementing hazardous fuel reduction treatments, there is substantial interest in examining their effectiveness. The range of fuel treatment options encompasses diverse goals, so assessment of their effectiveness must consider their original intended purpose (Vaillant and Reinhardt 2017; Lydersen et al. 2019). For example, ecological effectiveness evaluates immediate post-fire severity effects within the impacted treatment and will often include a focus on the recovery/resilience of the treated stand years after any wildfire interaction (Waltz et al. 2014; Stephens et al. 2020). Landscape-level effectiveness scales up stand-level ecological evaluations, more recently including the so-called ‘shadow effects’ of modification of fire behavior outside a treated stand (Finney et al. 2007). The effectiveness of fuel treatments that serve to provide societal risk reduction should be evaluated based on the reduction of real losses specifically within the built environment (Scott et al. 2016) as well as whether the treatment was tactically leveraged during the suppression response and management (Stephens et al. 2023). Many treatments have both ecological and risk reduction goals, and often more than one treatment is applied to a tract of land, adding to the complexity of the outcome analyses.

Fuel treatment effectiveness has largely been researched through individual case studies (e.g. Safford et al. 2009, 2024; Kennedy et al. 2019; Brodie et al. 2024) and systematic reviews (e.g. Cochrane et al. 2013; McKinney et al. 2022; Ott et al. 2023), generally demonstrating fuel treatments are effective at meeting their intent (Cochrane et al. 2013; Stevens-Rumann et al. 2013; Prichard et al. 2021; Davis et al. 2024). Individual case studies often benefit from a deep exploration into the detailed treatment combinations that were burned by wildfire, but as studies grow into larger geographies, the treatment history becomes obfuscated. This research contributes a novel method to retain the granularity of treatment combinations (hereafter referred to as ‘treatment’ regardless of the number of activities that occurred at the location) while evaluating the effectiveness of fuels treatments across a large geography – in this case, all National Forest System lands in California, with the potential for application in other US National Forest System lands. This bridges the scientific gap on how the sequence/combinations of multiple hazardous fuel treatments, rather than just the last treatment before wildfire, contribute to the treatment efficacy across large geographies. We did this by fusing the Forest Service Activity Tracking System (FACTS) data, which documents all land management activities (i.e. treatments), with the data in the interagency Fuel Treatment Effectiveness Monitoring system (FTEM) which flags the overlap of the most recent fuel treatment activity with a wildfire that intersects it. While FACTS includes all management activities, it also includes naturally caused wildfires that have been identified as ‘meeting objectives’: typically characterized as fires whose effects align with historical fire regimes (natural range of variability and/or historical range of variability). This fusion provided a regional data set of site-specific treatment sequences (also referred to as combinations throughout this paper) of fuel treatments for which effectiveness was determined. It is important to note that FTEM provides a binary (yes/no) assessment of fuel treatment efficacy based on the monitors subjective evaluation of whether the treatment altered fire behavior. Despite this limitation, the dataset supports large-scale analyses of fuel treatment effectiveness across the US. We used this data to test the following hypotheses:

Effectiveness of fuel treatments varies depending on vegetation type.

This research combines site-specific treatment monitoring history with large landscape-scale reporting, and to the best of our knowledge, is the first research that leverages a fused FACTS and FTEM databases to determine efficacy rates. The combination and sequencing of fuel treatments proposed herein closely matches site-specific research, supporting the utility of this binning methodology to allow for robust efficacy evaluations considering the critical need for governmental agencies to report fuel treatment effectiveness at the regional scale. While we focused our methodology on case studies from California, given the abundance of fuel treatment effectiveness studies in the state, the approach is applicable to other geographies.

Methodology

This research uses data from the FTEM application, which is an interagency module housed within the larger Interagency Fuel Treatment Decision Support System (iftdss.firenet.gov). FTEM data documents the perceived effectiveness of hazardous fuel treatments on wildfire behavior when the treatment is intersected by wildfire. As part of the FTEM data collection protocol, trained field observers record responses to multiple questions, including ‘Did the fire behavior change as a result of the treatment?’, which was used as our measure of effectiveness. To answer this question, field observers evaluate evidence of changed fire behavior (e.g. percent canopy scorch, canopy consumption, bole char height) from outside and within the treatment perimeter. This response is recorded as either ‘yes’ or ‘no’ providing a large dataset for non-parametric statistical analysis, though not offering any information on the magnitude of change in fire behavior.

We obtained FTEM monitoring results for treatments that were impacted by a wildfire between 2017 and 2022 on National Forest System Lands within California, US. Location of the samples used in this study are included in Supplementary Fig. S1. FTEM is a subset of the larger FACTS database in that it captures (1) only the most recent hazardous fuel treatment (i.e. a treatment which primary or secondary objective was to modify hazardous fuels) and (2) only treatments that have been intersected by wildfire. To gain the historical sequence of treatments, FTEM interactions were merged with the national FACTS database using the spatial unit identification (SUID) field attached to all hazardous fuel treatments occurring on a site from 1950 to 2022. FTEM is designed to only flag treatments for monitoring of potential effects if the treatment was completed within 10 years of intersection with wildfire. Thus, all treatments analyzed in this dataset had at least one hazardous fuel treatment within a decade of a wildfire intersecting with it. All treatments that were impacted by fires less than 0.01 acre (40 m²) were eliminated from the analysis because they can be as insignificant as a campfire that has crept out of a fire ring. This analysis returned a total of 1621 treatments identified as hazardous fuel reduction that were also impacted by wildfires.

A total of 44 distinct fuel treatment sequences with confirmed wildfire interactions were identified within the data set (Fig. 1) and were initially merged into four categories that broadly explain how they impacted vegetation, thus cross-walking treatment sequences to generalized fuel arrangements:

Fig. 1.

Fuel treatment sequence and categorization. In a two-stem process, large number of specific treatments were categorized and combined based on their sequence, and then further combined based on the dominant characteristic of the modification of fuels.

Treatments often include multiple entries in a logical sequence, such as felling trees, piling or removing whole or parts of the downed trees, and/or burning. Where the FACTS data identified multiple treatments had occurred, we created 14 distinct treatment classifications (‘pre-terminal sequence’ in the middle column of Fig. 1) to account for both singular and combined treatments. We assumed the treatments were completed in a logical order – e.g. piling of fuels (rearrangement) would happen before the piles were burned (surface reduction). We then merged these further based on the dominant characteristic of the modification of fuels into the final six ‘terminal sequence’ categories (right column in Fig. 1). For example, ‘Fire + Rearrangement + Surface Reduction + Removal’ was distilled to ‘Removal + Fire’ as the impacts of rearrangement and surface reduction would be eliminated by the application of fire through the system, but the impacts of removal would remain. This effectively crosswalks treatment(s) into a coarse surface fuel model.

The FTEM database also provides the date that wildfire entered the treatment unit. A variety of meso-scale weather and fire danger indices from the location and date of first intersection between wildfire and treatment, including vapor pressure deficit (VPD), energy release component (ERC) and burning index (BI) (Bradshaw et al. 1984), were acquired from the gridMET dataset (Abatzoglou 2013). These data were appended to the merged FTEM and FACTS database to analyze the impact of weather-related wildfire covariates on fuel treatment effectiveness.

A proportions z-test was used to determine whether treatments had a statistically significant impact on modifying wildfire behavior. An independent T-test was used to determine if the size of wildfire and fuel treatments significantly impacted the treatment effectiveness outcome. A Mann–Whitney U test was used to assess whether the distributions of weather and fire danger indices associated with fuel treatments that modified wildfire behavior were statistically different than those that did not. Finally, the Chi-squared test was used to assess whether the outcome (i.e. effectiveness ratio) of fuel treatment sequences was similar. In all tests, a P-value of 0.05 is considered the threshold for statistical significance. Expanded explanations regarding the statistical tests used in the study are included in Supplementary Section S1.

Vegetation type data was amassed by using the fuel treatment locations from FTEM intersected with the LandFire Biophysical Settings (BpS) layer (v1.4.0, 2014). Only treatments that intersected with BpS vegetation groupings annotated as conifer, hardwood or shrubland were included and analyzed using a Chi-squared test.

Results

Effective modification of fire behavior is a function of treatment size and wildfire size

We hypothesized that larger treatments were significantly more likely to modify fire behavior than smaller treatments. We expected the success of treatments to modify fire behavior would be greater when interacting with smaller wildfires relative to larger ones.

Treatment completion acres

Statistical analysis confirmed our hypothesis that the size of treatments would differentially modify fire behavior across treatment sequences. Using a Mann–Whitney U test (P < 0.05), we found significant differences in the distribution of treatment sizes that did and did not modify fire behavior in the case of ‘Rearrangement’, ‘Removal + Fire’ and ‘Surface Reduction’ terminal sequences (Fig. 2, Table 3). As hypothesized, larger-sized treatments afforded more fire modification than smaller treatments (Table 3). Interestingly, while ‘Rearrangement’ and ‘Surface Reduction’ had not been found to significantly modify fire behavior as evaluated in hypothesis 1, evaluation of hypothesis 2 demonstrates that larger ‘Rearrangement’ sizes can confer effectiveness. A distribution of the frequency of size of treatments by terminal sequence is provided (Supplementary Fig. S2).

Fig. 2.

Distribution of treatment sizes associated with effective (green) and ineffective (red) cases of terminal treatment sequences. Horizontal lines show median and interquartile range on each side of the violin plot. Asterisks denote significant differences in distribution. Y-axis is presented in log-scale.

Table 3.Mean and median treatment sizes for each terminal sequence based on treatment effectiveness.

Terminal sequence	Did treatment modify fire behavior?
	Yes		No
	Mean treated acres	Median treated acres	Mean treated acres	Median treated acres
Rearrangement	97.8	22.0	47.0	14.8
Removal + Fire	48.7	19.0	11.1	11.0
Surface Reduction	53.9	25.0	29.5	15.3
Fire	654.8	59.0	1851.3	55.0
Removal	63.8	36.5	105.8	53.6
Removal + Surface Reduction	53.4	28.0	56.4	28.5

Terminal sequences with significantly different distributions of acres treated are shown in bold font.

Wildfire size

Mann–Whitney U-test showed that the distribution of wildfire sizes for cases that did and did not modify fire behavior was significantly different for ‘Removal’, ‘Removal + Fire’, ‘Surface Reduction’ and ‘Removal + Surface Reduction’ terminal sequences (Fig. 3). Overall, the results did not confirm our overarching hypothesis that treatments would be more effective with smaller fires (Table 4). We acknowledge that multiple treatments might occur within one megafire, and hence, samples may not be spatially independent. Additionally, wildfire size indicates only the final outcome, and wildfire behavior (e.g. growth, intensity) at the day of interaction with fuel treatment was not assessed in this analysis. We also note that the lack of impact of wildfire sizes on fuel treatment effectiveness is not entirely unexpected given the large variation in the size of fires, with significant outliers during 2017–2021, that could have impacted the overall analysis (see Cova et al. 2023).

Fig. 3.

Distribution of wildland fire sizes associated with effective and ineffective cases of terminal treatment sequences. Horizontal lines show median and interquartile range on each side of the violin plot. Asterisks denote significant differences in distribution. Y-axis is presented in log-scale.

Table 4.Mean and median wildfire sizes for each terminal sequence based on treatment effectiveness.

Terminal sequence	Did treatment modify fire behavior?
	Yes		No
	Mean wildfire size (acre)	Median wildfire size (acre)	Mean wildfire size (acre)	Median wildfire size (acre)
Removal	71,086	36,450	44,437	366
Removal + Fire	136,210	145,632	310,247	410,203
Surface Reduction	174,160	52,498	225,827	26,7824
Removal + Surface Reduction	204,501	145,632	67,230	52,498
Fire	172,630	145,632	140,907	113,246
Rearrangement	164,403	96,901	158,464	52,498

Terminal sequences with significantly different distributions of associated wildfires are shown in bold.

Lack of modification of fire behavior within treatments is associated with higher fire danger days

In general, this hypothesis was not supported as results were generally mixed and did not demonstrate the expected correlation between high fire danger and lack of fuel treatment effectiveness, especially for BI and ERC (see Supplementary Section S3, Supplementary Figs S3 and S4, and Supplementary Tables S1 and S2). Distributions of VPD for cases that did (green) and did not (red) modify fire behavior were significantly different for ‘Rearrangement’, ‘Removal + Fire’, ‘Removal + Surface Reduction’ and ‘Surface Reduction’ (Fig. 4; Mann–Whitney U test P < 0.05). Importantly, the direction of association between fuel treatment effectiveness and extreme weather was consistent among all terminal sequences, with all ineffective cases being associated with higher mean and median VPD than effective cases (Table 5). Generally, the results of VPD were more conclusive than traditional fire danger indices of ERC and BI, implying that extreme weather might have had a fingerprint on treatments’ failure to modify fire behavior, although not conclusively. Additionally, this analysis indicated that the effectiveness of ‘Fire’ and ‘Removal’ terminal sequences in modifying fire behavior was not affected by weather, and they are decisively impactful treatment choices.

Fig. 4.

Distribution of Vapor Pressure Deficit (VPD) associated with cases that did and did not modify wildfire behavior for various terminal treatment sequences. Horizontal lines show median and interquartile range on each side of the violin plot. Asterisks denote significant differences in distribution.

Table 5.Mean and median Vapor Pressure Deficit (VPD) associated with wildfires that impacted fuel treatments, categorized based on effectiveness of terminal treatment sequences.

Terminal sequence	Did treatment modify fire behavior?
	Yes		No
	Mean wildfire size	Median wildfire size	Mean wildfire size	Median wildfire size
Rearrangement	2.2	2.2	2.6	2.7
Removal + Fire	1.69	1.7	2.3	2.5
Removal + Surface Reduction	2.1	2.2	2.4	2.2
Surface Reduction	2.2	2.2	2.5	2.7
Fire	2.03	2.1	2.2	2.2
Removal	1.9	2.0	2.0	2.0

Terminal sequences with significantly different distributions of VPD (kPa) between effective and ineffective treatments are shown in bold font.

Effectiveness of fuel treatments varies depending on vegetation type

Percent effectiveness was plotted as a function of general vegetation types and the terminal sequence categories (Fig. 5).

Fig. 5.

Effectiveness of various terminal sequences of fuel treatment in different vegetation classes. Numbers in each cell indicate the percent ‘Yes’ responses to the question of whether treatment modified fire behavior with n = number of samples. Hatch denote samples where n < 30 and removed from statistical analyses.

There is a significant relationship between the effectiveness of the treatment type as it relates to the vegetation type (P < 0.05). Sample sizes (i.e. monitored interactions) that fall below a threshold for statistical analysis (n < 30) have been omitted. In conifer vegetation, where all terminal sequences are well represented, rearrangement and surface reduction treatments yield the lowest level of effectiveness of the six terminal sequences. Though lower in their representation of treatments, all terminal sequences in shrublands did effectively connote a modification of fire behavior, with ‘Removal + Surface Reduction’ showing the lowest effectiveness. However, results from ‘Removal + Surface Reduction’ and ‘Removal’ are influenced by a smaller sample size. While results for the hardwood vegetation show that treatments are not yielding effective modification of fire behavior, they are likely affected by small sample sizes. In line with the trends of the previous analysis, ‘Rearrangement’ and ‘Surface Reduction’ continue to demonstrate lower effectiveness rates, except in shrublands.

Discussion

Fuel treatments are designed to modify undesired fire behavior primarily to reduce risk to values and enhance fire management operational effectiveness (Moghaddas and Craggs 2007; Parks et al. 2015); this effort often also coincides with enhancement of ecological resilience (Finney 2001; North et al. 2021; Baijnath-Rodino et al. 2023; Wright et al. 2023). While previous studies on the effectiveness of fuel treatments have relied heavily on individual case studies that looked at the single most recent fire treatment or computational simulations, our approach summarizes all instances of monitored interactions across over 20 million acres of National Forest System lands in California. Conclusions were derived using a count of local observations of treatment success, then organized based on treatment sequence as opposed to the last, or nonspecifically referenced, treatment before wildfire. This methodology is transferable and can provide insight into fuel treatment success in regions or other large geographies that do not have as extensive a set of case studies as California.

Limitations

The FTEM data, which is a subset of the US Forest Service authoritative data set (FACTS) for treatments, is a qualitative evaluation of effectiveness given that trained observers are limited to the binary selection of ‘yes’ the fire behavior was modified, or ‘no’ it was not. Classifying a continuous response variable in this binary manner has limitations. However, our construction of questions and appropriate statistical analyses were designed to work within the limited inferences that can be made with such data. We are encouraged that the qualitatively derived results and crosswalk of fuel treatment sequences to coarse fuel model types align with quantitative studies, thus opening the door to linking quantified data sets (e.g. light detection and ranging (LiDAR) or remotely sensed data) with the qualitative data outputs.

We also acknowledge the limitations of our day-of-start derived weather indices knowing that larger fires can span multiple weeks before spread is contained. Ideally, perimeter growth maps would be used to refine this analysis. However, those maps are generally only available for large fires that have aerial support, which would be a small fraction of the fires included in this data set. Given that a substantial proportion of monitored wildfires are between 10 and 100 acres (Li and Banerjee 2021), wildfire interactions with the fuel treatments would be dominated by fires that were effectively suppressed during the first few operational shifts. Hence, weather conditions during the time of the interaction with treatment are accurately captured by the fire weather indices adopted in this study. Further, large fires are typically the product of suppression failure within the first few operational shifts, with significant growth during their initial establishment; thus, we offer that the adopted fire weather indices sufficiently represent the weather under which the monitored interactions between wildfire and fuel treatments likely occurred. While this assumption is expected to impose additional uncertainty on our analyses, the agreement of our results with site-specific case studies gives us confidence that our assumption was reasonable. Lastly, due to the limitations of FTEM data collection methodology, these results are specific to project-level analysis and are inappropriate for evaluating ecological effectiveness or risk reduction. However, the alignment of the fuels categorization schema (Fig. 1) may afford a path forward to evaluate metrics of effectiveness that were out of the scope of the current study.

It is important to consider that the treatments which comprise the ‘Rearrangement’ and ‘Surface Reduction’ (e.g. piling and burning, chipping/mastication, precommercial thinning and thinning for hazardous fuels reduction) are common treatments around WUI, and therefore may have benefitted through the addition of suppression resources that leveraged the favorable fuel conditions during initial attack. These results underscore the confounding variable of management/suppression action on the outcome and evaluation of fuel treatment effectiveness. The analysis and evaluation of the effectiveness of fuel treatments based on size is not necessarily a straightforward question based on the data used in this analysis alone. Small treatments may be entered within FTEM, as effective in smaller fires if they are aggressively and successfully suppressed; however, these same smaller treatments will likely not modify fire behavior where intersected by larger fires. Research relating treatment size to outcome should be vigilant regarding the contribution of firing operations or reinforcement by aerial support. Additionally, our research did not assess daily fire behavior impacts on fuel treatment effectiveness, as information regarding the day-to-day firing operations and locations is scant and difficult to collect from Incident Action Plans (e.g. Incident 209 forms; see Gannon et al. 2023). Furthermore, our analysis is blind to how active suppression tactics and incident management decisions were implemented, putting an increased burden on the treatments to be effective as stand alone. It is well established that treatments that are leveraged during suppression operations for firing or holding are more successful than treatments that are not (Green 1977; Reinhardt et al. 2008). Lastly, due to the overwhelming amount of small fires as compared to larger incidents, future research may benefit through the lumping of fire size categories in analysis.

Conclusion

This research offers a novel approach by blending historical fuels treatment data (FACTS) into a logical treatment categorization schema to evaluate fuel treatment effectiveness at the regional scale using monitoring information found within the FTEM database. Our methodology presents a novel and statistically substantiated crosswalk between myriad fuel treatment methods and coarse fuel models that can be used for determining treatment effectiveness. By successfully testing our methodology in California, which has a rich case study history, we showed that this methodology may apply to large geographic regions that do not have the benefit of multiple robust case studies. Our findings indicated that most fuel treatment sequences were effective in modifying fire behavior, although mere rearrangement or surface reduction treatments were less effective. Further analysis indicated that although the footprint of treatment size and wildfire size on the effectiveness of fuel treatments was minimal, larger applications of even less effective treatments can enhance treatment efficacy. Additionally, the analysis of the fire weather relationship with fuel treatment effectiveness returned inconclusive results but generally indicated a lack of impact of fire weather on fuel treatment effectiveness. These results cast optimism that regardless of wildfire size or intensification of fire weather, fuel treatments are an important management tool for increasing the resilience of landscapes in the face of an anticipated increase in wildfire activity. Future research can further refine our analyses through leveraging wildfire progression maps (see Brodie et al. 2024) to acquire weather and wildfire behavior data coincident with the dates that wildfires intersected with treatments. Future research may also investigate whether treatments restore historical normal fuel loads at the treated site, and also the age of treatment when intersected by wildfires, to better capture the nuances of fuel treatment effectiveness. We also suggest evaluation of other fire regime metrics that may better describe fuel treatment effectiveness, most notably high severity patch size. Finally, we quality-controlled the raw data from FACTS, specifically for coordinates of fuel treatments, but note that future quality control of this invaluable data source is warranted.

The management implications of this research are far-reaching and highlight the need for removing residual activity fuel (most effectively using fire) and focus on treatment completion through the burning of rearrangement treatments (i.e. burning piles instead of turning them into jackpots of fuel). Continued research on evaluating the vulnerabilities of rearrangement treatments, which are often the interim stage of a multi-treatment plan, would provide value to management. Further study of the dissection of data by vegetation types could also provide additional valuable information for managers to consider during the planning and implementation of fuel treatments. The analytical results presented here should be considered as one facet of the wildfire management decision space. This contribution of which fuel treatments and under what conditions, demonstrates the highest effectiveness in modifying fire behavior, is part of many considerations, including social acceptance and the cost of implementation.

Supplementary material

Supplementary material is available online.