Towards benchmarking the dynamically downscaled CMIP6 CORDEX-Australasia ensemble over Australia

1.1. CORDEX downscaling

The Coordinated Regional Downscaling Experiment (CORDEX) is a powerful framework that coordinates the creation and comparison of ensembles of RCM simulations (Gutowski et al. 2016). Ensemble methods are vital in the production of RCM projections owing to the considerable uncertainties associated with the regional manifestation of anthropogenic climate change, particularly with regard to rainfall change and the water cycle. This is especially the case in the Australian region, where natural climate variability is already extremely high (e.g. Nicholls et al. 1997). CORDEX is the regional counterpart of the Coupled Model Intercomparison Project (CMIP), currently in its sixth phase (Eyring et al. 2016).

Through the first phase of CORDEX, downscaled climate simulations were produced for 14 continental-scale regions. In the Australasian region, five modelling centres participated in dynamically downscaling CMIP5 projections over Australia, New Zealand and the western Pacific (Evans et al. 2021). A total of 24 GCM–RCM dynamically downscaled pairs were produced (Isphording et al. 2024a). The CORDEX-Australasian ensemble (CORDEX-CORE) was also used in the Intergovernmental Panel on Climate Change Sixth Assessment Report (IPCC AR6) in their assessment of regional changes through the AR6 Atlas (Giorgi et al. 2021; Gutiérrez et al. 2021). Statistical downscaled projections were produced for 22 GCMs by the Bureau of Meteorology in the National Hydrological Projections (NHP) project (Wilson et al. 2022); this ensemble was used to underpin the Bureau of Meteorology’s Australian Water Outlook service. These experiments have formed the basis of state and federal climate assessments such as the 2019 Victorian Climate Projections (Clarke et al. 2019) and the Energy Sector Climate Information Project (CSIRO et al. 2021).

More recently, Australian modelling centres have commenced the downscaling of CMIP6 models to the CORDEX-Australasian domain, and this will form a core resource for new national projections (Grose et al. 2023). This effort involves four modelling centres, three independent RCMs and five RCM configurations. Two RCMs were run through the Australian Climate Service (ACS) and two by the state governments of New South Wales and Queensland. For clarity, in this paper we distinguish between the terminology ‘RCM’ to describe each model configuration or downscaling methodology, ‘RCM simulation’ to describe a single downscaling of a GCM by a RCM and ‘RCM ensemble’ to refer to the full set of RCM simulations.

1.2. RCM evaluation

To assist users in effectively using dynamically downscaled data to study the different impacts of climate change, it is informative to assess the capacity of RCMs to produce credible or physically plausible simulations of future climate. This traditionally relies on a rule of thumb, where the ability to sufficiently represent the observed climate is used as a prerequisite for being considered in projections and part of the evidence used to assign confidence in projections. To achieve this evaluation, downscaled simulations of CMIP Tier-1 Historical experiments, which use contemporary levels of radiative forcing, are assessed against corresponding contemporary climate observations in order to assess the skill of RCMs at simulating recent climate.

Cross-ensemble evaluation studies enhance the congruency of RCM data by enabling users to assess the performance of multiple potential data sources at once. To this effect, Vautard et al. (2021) carried out a comprehensive cross-ensemble evaluation of the EURO-CORDEX ensemble, assessing mean-state biases and the representation of extremes in a set of climate and impact-based variables. Particular focus was given to surface temperatures, rainfall and the water cycle. Evans et al. (2021) assessed monthly and annual biases, root mean square errors (RMSEs) and spatial correlations of temperatures and precipitation in CORDEX-Australasia Phase 1, downscaling CMIP5. They identified systematic biases common to most models in the ensemble, including underestimated diurnal temperature ranges and southern Australian rainfall. Meyer et al. (2021) assessed precipitation within the North American CORDEX ensemble, with a particular focus on seasonality. An extension into process-based evaluation, such as the evaluation of atmospheric processes and pressure patterns associated with synoptic systems (e.g. Pinto et al. 2018) provides further insight into RCM simulation performance, particularly in data-sparse regions such as southern Africa.

1.3. From evaluation to benchmarking

The Australian CORDEX-CMIP6 RCMs have been evaluated individually using different evaluation approaches. Chapman et al. (2023) found that downscaling CMIP6 models with the Conformal Cubic Atmospheric Model (CCAM) led to improvements mainly in coastal and mountainous regions and for extremes when compared with the host GCM. Howard et al. (2024) presented the Bureau of Meteorology Regional Projections for Australia (BARPA-ACS), a RCM that is contributing downscaled GCMs to CORDEX-Australasia for the first time under CORDEX-CMIP6. BARPA-ACS is based on the same modelling components used by the Bureau of Meteorology for weather and seasonal predictions and by the UK Met Office for global and regional projections. Schroeter et al. (2024) assessed the ERA-5 downscaling of CCAM-ACS. Di Virgilio et al. (2025a) evaluated ERA5-driven NARCliM2.0 (NSW and Australian Regional Climate Modelling) simulations for mean climate, attributing improvements in the simulation of precipitation principally to the driving ERA5 and improvements in maximum temperature principally to RCM design choices. Ji et al. (2024) evaluated ERA5-driven NARCliM2.0 simulations for precipitation extremes, reporting that RCMs captured climatology and the coefficient of variation of precipitation extremes well but struggled with temporal correlation and trends. Di Virgilio et al. (2025b) further assessed CMIP6 GCM-driven NARCliM2.0 simulations, finding much smaller biases in maximum temperature and reduced wet biases compared with NARCliM1.0 and 1.5, though with minimal improvement in minimum temperature. Further details of the participating CORDEX-Australasia RCMs are provided in Section 2 below.

There are two major caveats to traditional evaluation methodologies when considering CORDEX simulations. The first is that RCM simulations are reliant on a host model, so evaluation of both the host model simulation and the RCM simulation is needed to get a complete picture, with RCM simulation evaluation giving insights only into the stage in the process that it plays. The second is the lack of objective thresholds for establishing reasonable performance or for informing confidence in projections. Historically, a set of various evaluation metrics have been produced (often not comprehensive or objectively selected), and then the fitness for purpose of a model estimated using expert judgment. However, there are advantages in moving to a more transparent, quantitative and systematic framework, hence the call for attempting a ‘benchmarking’ approach rather than ad hoc or subjective evaluation. Although benchmarking aims to improve the objectivity of model evaluation, some degree of subjectivity cannot be entirely eliminated and remains in the choice of benchmarked metrics and thresholds.

1.4. Benchmarking

Within the Earth Sciences, the land-surface modelling community has led the adoption of benchmarking model evaluation frameworks (e.g. Abramowitz et al. 2005, 2012). The development of the ‘ILAMB’ (International Land Model Benchmarking) model evaluation tool (Collier et al. 2018) is an example of this, providing a standard computational framework to compare model data. ILAMB provides a flexible framework to assess any two-dimensional model against a corresponding observational data set, comparing biases, seasonal peaks, RMSEs and spatial correlations. Other tools, such as ESM-Val-Tool (Eyring et al. 2020), have a wide uptake in the CMIP global modelling community. Benchmarks can be used to measure generational model improvements (Alexander and Arblaster 2009, 2017; Flato et al. 2013; Sillmann et al. 2013; Fiedler et al. 2020), with increased spatial resolution (Bador et al. 2020; Nishant et al. 2022) and to assess the benefits and degradations associated with particular parameterisations, experimental designs, or GCM–RCM combinations (Ji et al. 2014; Liu et al. 2024). The value of moving from evaluation to benchmarking has been recognised for selecting CMIP models fit for downscaling (Nguyen et al. 2024) and in Extreme Event Attribution applications (Grose et al. 2023).

A particular focus on benchmarking precipitation has emerged in recent years (Ahn et al. 2022, 2023; Martinez-Villalobos et al. 2022). United States Department of Energy (2020) provided a seminal guidance document, which Isphording et al. (2024a) have adapted into a two-tier set of objectively defined benchmarks. The first tier of benchmarks seeks to establish a ‘minimum standard’ in model performance, defining metrics to assess rainfall bias, spatial distribution, seasonal cycle and long-term trends. The second tier of versatility metrics can provide a deeper understanding of temporal variability, distributions, extremes and drought.

1.5. Objectives of this study

This paper presents the first study applying a standardised evaluation and benchmarking approach across the CORDEX-Australasia CMIP6 ensemble. It combines the ILAMB and Isphording et al. (2024a) methodologies to benchmark precipitation and near-surface temperatures for the CORDEX-Australasia downscaling of CMIP6. We assess aggregated bias, spatial patterns, the seasonal cycle, temporal distributions, long-term trends and extreme indices using benchmarks drawn from Isphording et al. (2024a) where possible, and the CMIP6 ensemble mean in other cases. The specific thresholds from Isphording et al. (2024a) are used for inter-comparability, except some modifications and extra discussion are added, particularly for trends. These assessed quantities have been informed by the ILAMB project, and an ILAMB dashboard is provided as a supplementary resource. In order to ensure compatibility with the ILAMB framework, some minor adjustments have been made to the minimum standard metrics defined in Isphording et al. (2024a). Namely, spatial mean absolute percentage errors have been replaced with a centred root mean squared error (cRMSE) in order to facilitate display on a Taylor diagram and aggregated bias metrics have been included.

This study provides an exploratory view on the use of systematic benchmarking ahead of more subjective evaluation in a comparative analysis of a multi-GCM, multi-RCM climate downscaling ensemble. Rather than simply presenting a binary perspective, care has been taken to ensure that the underlying quantitative performance is provided for each metric. Our interpretation of benchmarks is as an outside reference point of good model performance, rather than as a hard binary to exclude models. Model climatology assessments can sometimes present bias maps as a routine component of evaluation; however, the choice of colour limit bounds can easily distort reader perception of bias magnitude, and a reference benchmark can aid in avoiding this. Some benchmarks, such as the extreme index benchmarks, are defined to be ambitious. Performance against a single benchmark should not be taken as a reason to reject a model out of hand. Rather, the benchmarks should be considered in concert with each other, and interpreted through the lens of each user application. This approach highlights the strengths and weaknesses of different models. The results from the benchmarking can also inform the needs and strategies for further processing of the modelling data to generate application-specific data. For example, benchmarking may inform the selection of models for developing contrasting storyline-based analysis, or the degree of bias correction deemed necessary before RCM simulation data can be used for impact-modelling applications.

This paper proceeds as follows. Section 2 describes the three different RCM designs that are benchmarked in this work for their simulated surface air temperature and precipitation over four Australian climate zones. The benchmarking methodologies in terms of regions, aggregation strategies and definitions of benchmarks, are also described. Section 3 presents an analysis of the observational uncertainty present in the benchmark values. Section 4 describes the benchmarks and benchmarking performance in the mean state, focussing on mean state characteristics of the RCM simulations, including biases, spatial patterns, seasonal cycles, temporal distributions and long-term trends. This is followed by Section 5, which benchmarks the RCMs with climate indicators that focus on wet precipitation and hot temperature extremes. Section 6 discusses the overall results and concludes the paper.

2.1. Models

This paper presents one approach to benchmark-based assessment of the RCM ensemble generated by four Australian regional climate modelling groups for CORDEX-CMIP6. These groups have participated in dynamical downscaling of CMIP6 projections for the Australasian region using three independent RCMs and five distinct model configurations. GCM selection followed a co-ordinated ‘sparse matrix’ approach to facilitate overlap in the selected GCMs and test the uncertainty in climate projections to RCM configuration. The model selection process is documented by Chapman et al. (2023), Grose et al. (2023) and Di Virgilio et al. (2022). Table 2 presents the sparse matrix of selected GCM–RCM simulation combinations.

The CCAM is a variable resolution global climate model (VR-GCM), possessing more freedom in its use as a downscaling model compared with the more traditional limited area models (LAMs). CCAM has been used in two separate configurations by two different modelling groups: the Queensland Future Climate Science Program (QFCSP; collaboration between the University of Queensland and the Queensland Government), producing QldFCP-2 (Queensland Future Climate Projections 2), and the Australian CSIRO in partnership with the Australian Climate Service (ACS) producing CCAM-ACS. CCAM uses the Community Atmosphere Biosphere Land Exchange (CABLE).

The CSIRO-ACS configuration of CCAM uses a C384 grid with a high-resolution region of 12.5 km. The model employs 54 vertical levels between 20 m and 40 km in the atmosphere and 40 levels in the ocean (to a depth of 5 km). To constrain the model when downscaling an ensemble of GCMs, CCAM uses a spectral nudging approach (Thatcher and McGregor 2009). Winds, air temperature and surface pressure are nudged approximately a length scale of 3000 km and above 850 hPa. Water vapour is not nudged, but allowed to evolve by the atmosphere model’s dynamics and physics. This is to avoid accidently preventing or duplicating precipitation events if the timing of rainfall differs between the regional model and the host GCM. CCAM has a coupled configuration for improving sea surface temperatures (SSTs) in the vicinity of coastlines (Thatcher et al. 2015), but with spectral nudging to ensure SSTs agree with the host GCM at a length scale of 1000 km. This approach to downscaling ensures that CCAM is reasonably constrained to follow the host GCM, but it will also inherit large scale biases and errors from the GCM.

The QldFCP-2 configuration of CCAM is a C288 stretched grid with a high-resolution region of 10 km. The model deploys 35 vertical levels and 30 levels in the ocean. QldFCP-2 uses bias and variance corrected SSTs and sea ice, as documented by Hoffman et al. (2016) and Chapman et al. (2023) and Atmospheric Model Intercomparison Project (AMIP)-style integrations (Gates 1992; Haarsma et al. 2016). In addition the CMIP6 radiative forcings, which consist of time-varying solar forcing, greenhouse gases (CO₂, N₂O, CH₄, chlorofluorocarbons, CFCs), ozone change, aerosols (sulfate, organic, black carbon, dust, volcanic, dimethylsulfide) and transient land cover change were used. Five simulations were also run in ocean-coupled mode, with bias-corrected SSTs with spectral nudging to ensure SSTs agreed with the host GCM at a length scale of 1000 km (see Table 1). However, in order to restrict realisation to one per GCM–RCM pair and to compare like with like, the ocean-coupled QldFCP-2 simulations are not directly assessed by this study. Using this experimental set-up allowed downscaling additional CMIP6 models that did not provide high frequency data for downscaling with a traditional RCM (Thatcher et al. 2015). Both the CCAM-ACS and CCAM-QldFCP-2 have similar configurations for atmosphere, ocean, land surface and aerosol parameterisation. Hence, the differences in the projections arise mostly from the downscaling experiment design (i.e. nudged v. bias-corrected) and some differences in the horizontal and vertical resolution and stretching.

BARPA-ACS refers to the Bureau of Meteorology moderate-scale climate downscaling model based on uses the UK Met Office Unified Model for modelling the atmosphere and Joint UK Land Environment Simulator (JULES) for the land surface. These are the same modelling components used by the Bureau for weather and seasonal predictions and reanalysis, and by the UK Met Office for global and regional projections. Set up with a horizontal regular latitude–longitude grid spacing of 0.1545°, it is configured with the global atmosphere and land (GAL) physics configuration HadREM3-GA7-05 (Walters et al. 2019; Tucker et al. 2022) with additional changes to improve land surface characterisation, grid point storms and convection (Su et al. 2022). The model top is either 41 km across 64 vertical levels, or 32 km across 61 vertical levels, depending on the height of available global model data. The former vertical level set is used with all but CESM2, CMCC-ESM2 and NorESM2-MM experiments. The model is forced at the lateral boundary and SST data from global models. It is also dynamically nudged towards global fields of temperature and winds between 11 and 37 km above the surface.

NARCliM provides high-resolution climate projections for CORDEX-Australasia and south-eastern Australia at different resolutions tailored to the specific needs of the regions. To date, NARCliM has completed three major phases: the first generation of NARCliM(1.0) was delivered in 2014 (Evans et al. 2014), the second generation of NARCliM(1.5) was delivered in 2020 (Nishant et al. 2021) and the third generation of NARCliM(2.0) was delivered in 2024 (Di Virgilio et al. 2025b). These NARCliM projects downscaled GCM outputs from the CMIP Phases 3, 5 and 6 (CMIP3, CMIP5 and CMIP6). NARCliM2.0 comprises of two Weather Research and Forecasting (WRF) version 4.1.2 (Skamarock et al. 2019) RCMs downscaling five CMIP6 global climate models contributing to CORDEX-Australasia at 20-km resolution, and south-east Australia at 4-km convection-permitting resolution (Di Virgilio et al. 2025b). The two RCMs (R3 and R5) were selected from 78 combinations of physics parameterisations in WRF based on their performance in simulating the recent Australian climate and statistical independence. The five GCMs were shortlisted from the CMIP6 ensemble considering their performance, statistical independence and possible future changes (Di Virgilio et al. 2022). Evaluations and comparison of reanalysis and GCM-driven simulations across three generations of NARCliM indicate that NARCliM2.0 significantly reduced biases in maximum temperature and precipitation, particularly in south-east Australia, making it more reliable for climate-impact assessments and future planning (Ji et al. 2024; Di Virgilio et al. 2025b). These improvements are primarily due to advancements in the RCMs (Di Virgilio et al. 2025b). The model configurations for BARPA-ACS, CCAM-ACS, QldFCP-2 and NARCliM2.0 are summarised in Table 2.

2.2. Data

The core climatological period considered for this work is 1985–2014, the last 30 years of the CORDEX historical time period. For long-term trend analysis (Section 4.5), this time period is extended to 1960–2014, which is the maximum overlapping time period available from all RCM simulations considered.

The Australian Gridded Climate Data (AGCD) Version 1 provides a gridded daily 0.05 × 0.05° analysis of station daily maximum and minimum 2 m (screen-level) temperature data, and daily precipitation accumulation (Jones et al. 2009). For brevity, these temperatures are referred to as tasmax and tasmin, and precipitation as pr. This study used CSIRO’s commercially licensed version of the Bureau of Meteorology’s AGCD data set (see https://doi.org/10.25914/6009600304b02), which is accessible through the National Computing Infrastructure. For more information on this version, please refer to https://github.com/AusClimateService/agcd-csiro. The AGCD grids are generated using an optimised Barnes successive-correction method that applies weighted averaging to the station data. Topographical information is included by using anomalies from long-term (monthly) averages in the analysis process. The AGCD analysis errors for tasmax are larger near the coast around north-west Australia and around the Nullarbor Plain owing to strong temperature gradients between the coast and inland deserts and a sparse observational network (Jones et al. 2009). The coast of Western Australia and parts of the Northern Territory are likely to share this analysis issue. The analysis errors are larger for tasmin, especially over Western Australia and the Nullarbor Plain. For precipitation, the rain gauge analysis of daily accumulation over Australia was produced using the Barnes method in which the ratio of observed rainfall to monthly average is used in the analysis process (Jones et al. 2009). There is a north–south gradient in the AGCD analysis errors, with larger analysis errors in the northern tropical regions, where the length scales of rainfall events are shorter and more convective. A spatial mask is applied to all precipitation fields to exclude regions where station influence is low; the excluded region is shown in grey in Fig. 1.

Fig. 1.

Cluster masks for NRM superclusters and subclusters used in this study, following Clarke et al. (2015). The precipitation data mask is shown in grey.

Spatial aggregation is performed using four Australian climate zones, known as the National Resource Management (NRM) superclusters (Clarke et al. 2015). These supercluster regions are shown with filled colours in Fig. 1. As well as following the boundaries of Australia’s NRM regions, these superclusters are climatologically distinct and represent the four major climate zones of Australia. The four NRM superclusters were favoured over the 8 NRM clusters and the 15 NRM subclusters owing to the large quantity of models being analysed in this paper. Superclusters were judged to reduce the quantity of results to an acceptable level while still separating these four most distinct climates in Australia. The four clusters approximately correspond to tropical (northern Australia), subtropical (eastern Australia), temperate (southern Australia) and arid climates (rangelands). The accompanying ILAMB dashboard includes a further breakdown of the assessment into eight clusters for increased granularity, and also includes spatial maps. However, as observed rainfall trends considered in Section 4.5 are localised, three NRM subclusters with well-studied trends have been selected as case studies. These regions are indicated with outlines in Fig. 1.

Climate indices are calculated metrics used to represent the state of the climate system and its changes. In this study, the open-source Python library ICCLIM (Index Calculation for CLIMate) Version 6.5.0 (Pagé et al. 2022, Aoun et al. 2024) is used to calculate the following monthly indices: maximum 1-day total precipitation (RX1day), average precipitation during wet days (SDII), number of wet days when precipitation is greater than or equal to 1 mm (RR1), maximum consecutive wet days when precipitation is greater than or equal to 1 mm (CWD), maximum consecutive dry days when precipitation is less than 1 mm (CDD), maximum daily maximum temperature (TXx) and maximum daily minimum temperature (TNx). These indices were chosen to capture heavy precipitation and hot temperature extremes over Australia and are calculated using daily precipitation, tasmax and tasmin. Indices computed using ICCLIM definitions provided by the European Climate Assessment & Dataset with detailed descriptions and the equations of individual indices can be found in the Algorithm Theoretical Basis Document (ATDB; Royal Netherlands Meteorological Institute 2021).

2.3. Methodology

This benchmarking assessment focuses on two atmospheric variables: near-surface temperature and precipitation, divided into three core and six additional indices. The reference observational data set used throughout this study is the AGCD (Section 2.2). As such, the present study is an attempt at benchmarking the statistics of the diagnosed variables of interest, not of processes underpinning the simulation of these variables (although the concept has value for this too; see Discussion).

The observed and modelled core variables (tasmax, tasmin and daily precipitation) are aggregated to monthly means and assessed in Section 4. The assessment for the ICCLIM indices is presented in Section 5. For each core variable, benchmarks are applied to assess model biases, spatial distributions, temporal distributions, seasonal cycles and long-term trends. The ICCLIM indices are benchmarked by their seasonal mean absolute errors (MAEs). A summary of each benchmark is provided in Table 3, and the flow chart shown in Fig. 2 illustrates the computational method for each benchmark. The description of each benchmark in Sections 4 and 5 is split into a ‘benchmark description’ subsection that describes the rationale and computation method for each benchmark, and a ‘benchmark performance’ subsection that discusses the performance of the RCM ensemble at meeting the benchmark.

Fig. 2.

Flow chart of computation methodologies for applied benchmarks. Coloured boxes indicate time means (green), spatial means (pink), benchmark threshold definitions (blue), benchmarked metrics (purple) and benchmark definitions (orange).

We use a weighted mean approach to aggregate the core variables from their local grids to NRM superclusters, where weights derive from the fractional overlap between each model grid cell and each NRM supercluster. Benchmarking of spatial patterns is performed on each model’s native grid by upscaling observational data to the model grid.

In order to define meaningful measures of performance, the bias and temporal distribution analyses both benchmarks against the CMIP6 historical experiment. These benchmarks are defined taking an ensemble mean of CMIP6 GCM performance across all models listed in Table 1 for which the required data were available. Column 6 of Table 1 indicates where GCMs have been excluded from the ensemble mean owing to data availability issues. Each GCM is given equal weighting in the ensemble mean, regardless of how many times it was downscaled. This ensures that the same benchmark values are applied to each RCM simulation, independently of the performance of its individual host model. To generate benchmark thresholds for ICCLIM indices, daily GCM and AGCD data are both conservatively regridded to a common 1.5° grid before computation of the indices. Thresholds for benchmarking were developed from principles of regional downscaling or taken from Isphording et al. (2024a).

4.1. Bias benchmark

4.1.1. Benchmark definition

The bias benchmark compares the magnitude of the large-scale land-based mean-state biases of the RCM ensemble across the four super-NCM clusters with the CMIP6 GCM ensemble that is downscaled. This benchmark is based on the expectation that RCMs should not substantially degrade the representation of the climatic mean state at length scales that are well resolved in GCMs. Figure 3 presents the bias benchmarks for the summer and winter seasons and for precipitation, tasmax and tasmin. The benchmark value is taken as the ensemble average of the absolute NRM-aggregated bias for each CMIP6 historical experiment that has been downscaled in CORDEX-Australasia. This benchmark is then applied to the absolute value of the aggregated bias in each RCM simulation. For informative purposes, the sign of the RCM simulation bias is shown in Fig. 3 for reference. However, our analysis focuses on the magnitude of these biases when comparing with benchmarks. Supplementary Fig. S1 provides the breakdown of the benchmark value across the CMIP6 ensemble subset.

Fig. 3.

Aggregated bias benchmark for each core variable, season and GCM–RCM pair. Colours and numbers indicate the aggregated bias. The benchmark value is given in the top row. Biases larger than the benchmark are marked by purple squares. Lighter colours represent better performance, whereas darker colours indicate worse performance: (a) precipitation, (b) tasmax, (c) tasmin. Precipitation biases are presented as a percentage of the observed seasonal precipitation, whereas temperature biases are presented in degrees Celsius.

This approach extracts large-scale bias by averaging out smaller-scale spatial inhomogeneity. However, because the ensemble mean is taken on absolute values, negative and positive biases across the different GCMs do not cancel each other. This approach has been deemed appropriate because the GCMs and RCM simulations possess very different levels of spatial inhomogeneity owing to their differing resolutions. Additionally, the approach applied here differs from an added value approach, where RCMs are directly compared with their driving models. Here, we attempt to quantify a single base-line level of bias that we would like the RCMs to exceed. This approach does not quantify the reduction in bias associated with the downscaling process, as would be achieved by an added value study.

4.2. Spatial pattern benchmark

4.2.1. Benchmark definition

The second benchmark assesses the ability of each RCM simulation to reproduce the spatial pattern of the mean state of the three core variables. Following Taylor et al. (2001), a Taylor diagram format has been used to display the assessment. This assessment was performed by first remapping the AGCD observational data to the respective model grid, and secondly computing correlations and standard deviation ratios on each native grid. This benchmark follows Isphording et al. (2024a) in prescribing fixed thresholds for spatial pattern benchmarks. Isphording et al. (2024a) determined these thresholds through consultation with data users on their expectations of model performance. However, in order to implement the Taylor diagram presentation, this approach does diverge from Isphording et al. (2024a) by using cNRMSE instead of MAPE. We note that the cNRMSE benchmark is a stronger condition than the correlation benchmark. Thresholds of 0.7 and 0.65 were used for spatial correlation and cNRMSE, with the former matching Isphording et al. (2024a).

Benchmarks are presented in Fig. 4, which shows Taylor diagrams of the climatological mean core variables, with benchmarks indicated by magenta lines. RCM simulations are indicated by blue, orange, red and green markers with colours indicating the RCM. Where possible, driving GCMs are labelled with an index according to Table 1; however, this label has been omitted where markers overlap.

Fig. 4.

Taylor diagrams indicating spatial pattern benchmark for core variables: (a) precipitation; (b) tasmax; (c) tasmin. The radial axis indicates the spatial standard deviation of each climatological mean normalised by that of observations. The angle axis indicates the cosine of the spatial correlation between each modelled climatological mean and observations. Curved grey contours show contours of cNRMSE. The black star indicates a hypothetical perfect match between a model and observations. The convex hull of the CMIP6 ensemble is shown by grey shading. Between colours and shapes indicate RCM, numbers indicate GCM. Numbers are only shown when clustering allows. Magenta lines indicate correlation benchmark (dashed) and cNRMSE benchmark (solid). All models pass these benchmarks.

4.3. Seasonal cycle benchmark

4.3.1. Benchmark definition

The seasonal cycle benchmark for each NRM region follows the recommendation from Isphording et al. (2024a) for a unimodal seasonal cycle. This requires that the 3-month observed high and low peaks of the seasonal cycle occur within the modelled highest and lowest 6 months of the year respectively. The seasonal cycle benchmark was applied to all three core variables. However, the benchmark was unanimously passed for both temperature variables, and therefore is only presented for precipitation. Figures 5 and 6 indicate the months that meet the seasonal cycle benchmark for precipitation for each super-NRM region, with colours and numbers indicating the ranking of the months from driest (1) to wettest (12). The observed seasonal cycle derived from AGCD is shown in the top row. Following the recommendations of Isphording et al. (2024a), the magnitude of seasonal mean rainfall for each GCM–RCM pair is indicated in Fig. 7. For brevity, this figure does not distinguish between the downscaled GCMs.

Fig. 5.

Seasonal cycle benchmark following Isphording et al. (2024a) for a unimodal seasonal cycle in (a) northern and (b) eastern Australia. Format follows Fig. 3, where values highlighted in purple indicate where a simulation does not meet our benchmarking standards. Colours and numbers indicate the ranking of monthly mean precipitation, with 1 being the driest and 12 being the wettest.

Fig. 6.

As per Fig. 5 but for (a) rangelands and (b) southern Australia.

Fig. 7.

Annual cycle magnitude and ranges for each RCM for (a) southern Australia, (b) eastern Australia, (c) rangelands and (d) northern Australia. Thick solid lines represent the ensemble mean for each RCM, whereas thinner dotted lines represent the ensemble maximum and ensemble minimum, with colours as per the legend. The observational comparison is shown in black.

4.4. Temporal distribution benchmark

4.4.2. Benchmark performance

Heatmaps of the temporal distribution benchmark are provided in Fig. 8. Despite the removal of the mean, the temporal distribution scores for precipitation are correlated with the bias benchmark presented in Fig. 3, but the temperature-based temporal distribution scores are more independent. The benchmark and RCM values of both temperature temporal distribution scores are all quite high, suggesting that CMIP6 and CORDEX have similar performance at simulating the temporal variability of large-scale temperatures – as may be expected. Tasmax are low in QldFCP-2 in southern Australia for both seasons despite small mean-state biases. Tasmin temporal distribution scores are low in NARCliM2.0 during DJF. Both the benchmark and RCM simulation precipitation temporal distribution scores are low for eastern Australia between July and August, suggesting that the rainfall distribution for this region and season is difficult to simulate.

Fig. 8.

Temporal distribution score benchmarks for unbiased distributions of (a) daily precipitation; (b) daily maximum temperatures; and (c) minimum temperatures. Presentation is as per Fig. 3. Blue colours indicate better performance, whereas red colours indicate worse performance.

4.5. Long-term trend benchmark

4.5.1. Benchmark definition

Although the previous benchmarks reviewed in this study consider model performance at simulating a climate assumed to be approximately stationary, the ability to simulate realistic trends is crucial for climate change applications. Isphording et al. (2024a) presented trends in Australian mean rainfall as one of the core minimum standard benchmarks. The appropriate design of a long-term trend benchmark for downscaled climate models is subtle as it must take into account multiple factors. Firstly, interannual and decadal variability are decoupled from reality in the driving GCMs, and so a model should not be penalised for having independent variability. Secondly, there exists a delicate balance between GCM-based and RCM-based trends, and the degree to which a RCM simulation should follow its host model. On one hand, GCMs are often selected for downscaling on the basis of their projected trends (e.g. Grose et al. 2023) and therefore it may be desirable that the RCM simulations follow their host models. On the other hand, if a RCM deviates from its driving model for a legitimate reason, this may be a sign of added value (e.g. Di Virgilio et al. 2020).

For the reasons listed above, we have opted to benchmark against observed trends that have been considered by previous observational studies. To reduce the amount of decadal variability captured in the trends, we have increased the study period for the long-term trend analysis from 30 to 55 years for this benchmark only. In order to account for the influence of the driving GCM trends on the RCM simulation trends, the benchmarks are shown on a scatter-plot where the corresponding GCM trend is shown on the x-axis and the RCM simulation trend on the y-axis (Fig. 9 and 10). The long-term trend benchmark assesses the ability of RCM simulations to capture AGCD-based observed trend signals in the full historical period, from 1960 to 2014. Trends in NRM-aggregated seasonal mean core variables are computed using a Theil–Sen slope estimator (Theil 1950; Sen 1968). Confidence intervals are computed with a significance value of P < 0.05, and the benchmark is deemed to be satisfied if these confidence intervals overlap. Confidence intervals are marked as error bars on the figures for the AGCD-based value and for all simulations for which the RCM-based confidence intervals do not overlap with observations.

Fig. 9.

Large-scale trend benchmark plots for precipitation: (a) July–August Murray Basin; (b) July–August SSW Flatlands; (c) December–February Monsoonal North (West). The x-axis shows the long-term trend of the GCM and the y-axis shows the long-term trend of the RCM simulation. Total agreement of the GCM and RCM simulation is indicated by the black line. The AGCD-based trend is indicated by the black cross on the black line. Markers are as per Fig. 4. Confidence intervals at the P < 0.05 level are marked for observations and all models that do not meet the benchmark. Models that do not meet the benchmark are indicated in bold.

Fig. 10.

Large-scale trend benchmark plots for annual mean tasmin (a–d) and tasmax (e–h) as per Fig. 9.

All super-NRM warming trends in AGCD are statistically significant at the P < 0.05 level and are likely to be driven by climate change. As observed rainfall trends are highly seasonal and do not follow NRM supercluster boundaries, rainfall trend benchmarking is only considered for three NRM subclusters and seasons for which observed rainfall trends have previously been studied, namely the Murray Basin (McKay et al. 2023) and SSW-Flatlands (West) during JJA (Raut et al. 2014), and the Monsoonal North West during DJF (Borowiak et al. 2023; Fahrenbach et al. 2024). These NRM subclusters are indicated by solid lines on Fig. 1.

Figures 9 and 10 indicate the large-scale trend plot for the core variables. The figures plot GCM trends on the x-axis against RCM simulation trends on the y-axis. Each RCM simulation ensemble member is indicated by a labelled point, with colour indicating the RCM used and the label indicating the index of the GCM. The black line indicates parity, and the black cross indicates the AGCD-based observed trend.

5.1. Benchmark definition

Unlike the majority of the former benchmarks that dealt with spatially aggregated means, the climate index benchmarks have been designed to capture statistics at the RCM’s native grid resolution. This makes a GCM-based benchmark difficult to define, as GCMs are fundamentally unable to represent these statistics. Because of this, the defined benchmark requires that the RCM simulation be able to represent climate indices at the fine scales with at least the same fidelity that the GCMs are able to represent the indices at coarse scales. This is an ambitious and high benchmark, as finer spatial scales are typically harder to reproduce than coarser scales. However, it has been selected so that the benchmarked indices are those with practical application to impact-based studies.

In order to benchmark climate indices across scales, the selected ICCLIM indices (RR1, RX1day, SDII, CWD, CDD, TXx and TNx) were computed from observations both at the GCM length scale (a standard 1.5° grid, onto which the GCMs were also regridded) and at the native RCM resolution. The benchmarks were computed by comparing the GCM-based indices with the former, and applied to the RCM simulation by comparing the RCM simulation based indices to the latter. The benchmark metric was MAPE for precipitation-based indices, and MAE in units of degrees Celsius for temperature-based indices.

5.2. Benchmark performance

The representation of the different rainfall-based climate indices is shown in Fig. 11. Additionally, aggregated biases of these indices are provided in Supplementary Fig. S7. These figures give insight into the way that convection is represented in the respective RCMs, with GCM forcing providing relatively less explanatory power of model performance for CCAM-ACS, BARPA-ACS and QldFCP-2. Wet day frequency is well represented by BARPA-ACS, potentially owing to the convective-memory enhancement to its convection scheme (Howard et al. 2024), whereas CCAM-ACS shows too frequent wet days across most GCMs. The exception here is CCAM-ACS-CESM2, which has improved wet day frequencies but was shown in Fig. 3 to have a strong summer dry bias in northern Australia and rangelands. Conversely, the annual maximum daily rain rate (RX1Day) is very well represented by QldFCP-2, but is overestimated by both BARPA-ACS and CCAM-ACS. However, the representation of rainfall intensity (SDII) is flipped when compared with RX1Day, with many models that perform well at simulating RX1Day performing poorly at SDII. Both wet and dry spell length (CWD and CDD) are well represented, with CDD being particularly well represented by QldFCP-2.

Fig. 11.

ICCLIM rainfall index benchmarks for (a) RR1, (b) RX1day, (c) SDII, (d) CWD and (e) CDD. Figure structure follows Fig. 3. Lighter colours represent better performance, whereas darker colours indicate worse performance.

The NARCliM2.0 simulations show the greatest degree of inter-RCM spread across all the RCMs. N2.0-R5 shows pronounced underestimations in the number of wet days and overestimations of dry spell lengths when downscaling MPI-ESM1-2-HR and EC-Earth3-Veg, but very good performance at these metrics when downscaling the remaining three GCMs (Fig. 11 and Supplementary Fig. S7). By contrast, N2.0-R3 shows improvements to the simulation of these metrics when downscaling MPI-ESM1-2-HR and EC-Earth3-Veg. This result is consistent with the improvement of the NARCliM2.0 dry bias between N2.0-R3 and N2.0-R5.

Finally, benchmarks for annual maximum daytime and overnight temperatures are given Fig. 12, with aggregated biases presented in Supplementary Fig. S8. Consistent with Fig. 3, BARPA-ACS overestimates maximum temperatures in northern Australia, whereas NorESM2-MM performs poorly when downscaled by QldFCP-2. TNx benchmark performance is closely linked to the DJF biases in tasmin discussed in Section 4.1, with CNRM-ESM2-1 and seven NARCliM2.0 RCMs highlighted in Fig. 12 owing to the former’s cold bias and the latter’s northern Australian warm bias. Most GCMs and RCMs consistently show relatively high TNx errors in southern Australia compared with the other regions.

Fig. 12.

ICCLIM rainfall index benchmarks for (a) TXx and (b) TNx. Figure structure follows Fig. 3. Lighter colours represent better performance, whereas darker colours indicate worse performance.