[v2.15.0] `recipe_aod_aeronet_assess.yml` fails due to missing data

[flicj191]

Recipe fails due to missing data AERONET, version 20250726, version updates every month with updated data so would depend on when the download and cmoriser is run.

ESMValTool/esmvaltool/recipes/recipe_aod_aeronet_assess.yml

Line 47 in aedc108

- {dataset: AERONET, project: OBS6, mip: AERmon, tier: 3, type: ground, version: 20250726}

For example, file: OBS6_AERONET_ground_20250823_AERmon_od440aer_199305-202507.nc the version 20250823 is a month after the data range date.

Could either wildcard the version in recipe? Or change recorded version in CMORiser though existing files would need to be updated.

Details

Out file:

2026-07-13 00:59:39,176 UTC [3801353] INFO    
______________________________________________________________________
          _____ ____  __  ____     __    _ _____           _
         | ____/ ___||  \/  \ \   / /_ _| |_   _|__   ___ | |
         |  _| \___ \| |\/| |\ \ / / _` | | | |/ _ \ / _ \| |
         | |___ ___) | |  | | \ V / (_| | | | | (_) | (_) | |
         |_____|____/|_|  |_|  \_/ \__,_|_| |_|\___/ \___/|_|
______________________________________________________________________

Earth System Model Evaluation Tool

A community tool for the evaluation of Earth system models.

https://esmvaltool.org

The Earth System Model Evaluation Tool (ESMValTool) is a community
diagnostics and performance metrics tool for the evaluation of Earth
System Models (ESMs) that allows for routine comparison of single or
multiple models, either against predecessor versions or against
observations.

Tutorial: https://tutorial.esmvaltool.org
Documentation: https://docs.esmvaltool.org
Contact: esmvaltool-dev@listserv.dfn.de

If you find this software useful for your research, please cite it using
https://doi.org/10.5281/zenodo.3387139 for ESMValCore or
https://doi.org/10.5281/zenodo.3401363 for ESMValTool or
any of the reference papers listed at https://esmvaltool.org/references/.

Have fun!

2026-07-13 00:59:39,178 UTC [3801353] INFO    Package versions
2026-07-13 00:59:39,178 UTC [3801353] INFO    ----------------
2026-07-13 00:59:39,178 UTC [3801353] INFO    ESMValCore: 2.15.0rc4
2026-07-13 00:59:39,178 UTC [3801353] INFO    ESMValTool: 2.15.0.dev71+g2df7c303e
2026-07-13 00:59:39,178 UTC [3801353] INFO    ----------------
2026-07-13 00:59:39,178 UTC [3801353] INFO    Reading configuration files from:
/g/data/tm70/fc6164/pixi_cache/envs/ESMValTool-5809249647550530018/envs/esmvaltool/lib/python3.12/site-packages/esmvalcore/config/configurations/defaults (defaults)
/home/189/fc6164/.config/esmvaltool (default user configuration directory)
2026-07-13 00:59:39,179 UTC [3801353] INFO    Writing program log files to:
/scratch/tm70/fc6164/esmvaltool_output/recipe_aod_aeronet_assess_20260713_005939/run/main_log.txt
/scratch/tm70/fc6164/esmvaltool_output/recipe_aod_aeronet_assess_20260713_005939/run/main_log_debug.txt
/scratch/tm70/fc6164/esmvaltool_output/recipe_aod_aeronet_assess_20260713_005939/run/cmor_log.txt
2026-07-13 01:00:01,306 UTC [3801353] INFO    Starting the Earth System Model Evaluation Tool at time: 2026-07-13 01:00:01 UTC
2026-07-13 01:00:01,307 UTC [3801353] INFO    ----------------------------------------------------------------------
2026-07-13 01:00:01,307 UTC [3801353] INFO    RECIPE   = /home/189/fc6164/esmValTool/repos/ESMVal/ESMValTool/esmvaltool/recipes/recipe_aod_aeronet_assess.yml
2026-07-13 01:00:01,308 UTC [3801353] INFO    RUNDIR     = /scratch/tm70/fc6164/esmvaltool_output/recipe_aod_aeronet_assess_20260713_005939/run
2026-07-13 01:00:01,308 UTC [3801353] INFO    WORKDIR    = /scratch/tm70/fc6164/esmvaltool_output/recipe_aod_aeronet_assess_20260713_005939/work
2026-07-13 01:00:01,308 UTC [3801353] INFO    PREPROCDIR = /scratch/tm70/fc6164/esmvaltool_output/recipe_aod_aeronet_assess_20260713_005939/preproc
2026-07-13 01:00:01,308 UTC [3801353] INFO    PLOTDIR    = /scratch/tm70/fc6164/esmvaltool_output/recipe_aod_aeronet_assess_20260713_005939/plots
2026-07-13 01:00:01,308 UTC [3801353] INFO    ----------------------------------------------------------------------
2026-07-13 01:00:01,473 UTC [3801353] INFO    Running tasks using at most 8 processes
2026-07-13 01:00:01,473 UTC [3801353] INFO    If your system hangs during execution, it may not have enough memory for keeping this number of tasks in memory.
2026-07-13 01:00:01,474 UTC [3801353] INFO    If you experience memory problems, try reducing 'max_parallel_tasks' in your configuration.
2026-07-13 01:00:01,559 UTC [3801353] INFO    Creating tasks from recipe
2026-07-13 01:00:01,559 UTC [3801353] INFO    Creating tasks for diagnostic od440aer_climatologies
2026-07-13 01:00:01,559 UTC [3801353] INFO    Creating diagnostic task od440aer_climatologies/aeronet
2026-07-13 01:00:01,600 UTC [3801353] INFO    Creating preprocessor task od440aer_climatologies/od440aer
2026-07-13 01:00:01,600 UTC [3801353] INFO    Creating preprocessor 'default' task for variable 'od440aer'
2026-07-13 01:00:01,609 UTC [3801353] ERROR   No files were found for Dataset:
{'diagnostic': 'od440aer_climatologies',
 'variable_group': 'od440aer',
 'dataset': 'AERONET',
 'project': 'OBS6',
 'mip': 'AERmon',
 'short_name': 'od440aer',
 'alias': 'OBS6',
 'frequency': 'mon',
 'long_name': 'Ambient Aerosol Optical Thickness at 440nm',
 'modeling_realm': ['aerosol'],
 'original_short_name': 'od440aer',
 'recipe_dataset_index': 0,
 'standard_name': 'atmosphere_optical_thickness_due_to_ambient_aerosol_particles',
 'tier': 3,
 'timerange': '1994/2014',
 'type': 'ground',
 'units': '1',
 'version': 20250726}
session: 'recipe_aod_aeronet_assess_20260713_005939',
using data sources:
- data source: LocalDataSource(name='local', project='OBS6', priority=1, rootpath=PosixPath('/g/data/ct11/access-nri/replicas/esmvaltool/obsdata-v2'), dirname_template='Tier{tier}/{dataset}', filename_template='{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc', ignore_warnings=[])
  message: No files found matching glob pattern /g/data/ct11/access-nri/replicas/esmvaltool/obsdata-v2/Tier3/AERONET/OBS6_AERONET_ground_20250726_AERmon_od440aer[_.]*nc
- data source: LocalDataSource(name='era5-derived', project='OBS6', priority=1, rootpath=PosixPath('/g/data/ct11/access-nri/era5-derived'), dirname_template='Tier{tier}/{dataset}', filename_template='{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc', ignore_warnings=[])
  message: No files found matching glob pattern /g/data/ct11/access-nri/era5-derived/Tier3/AERONET/OBS6_AERONET_ground_20250726_AERmon_od440aer[_.]*nc
- data source: LocalDataSource(name='kj13', project='OBS6', priority=1, rootpath=PosixPath('/g/data/kj13/datasets/esmvaltool/obsdata-v2'), dirname_template='Tier{tier}/{dataset}', filename_template='{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc', ignore_warnings=[])
  message: No files found matching glob pattern /g/data/kj13/datasets/esmvaltool/obsdata-v2/Tier3/AERONET/OBS6_AERONET_ground_20250726_AERmon_od440aer[_.]*nc
- data source: LocalDataSource(name='av17', project='OBS6', priority=1, rootpath=PosixPath('/g/data/av17/access-nri/esmvaltool/obsdata-v2'), dirname_template='Tier{tier}/{dataset}', filename_template='{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc', ignore_warnings=[])
  message: No files found matching glob pattern /g/data/av17/access-nri/esmvaltool/obsdata-v2/Tier3/AERONET/OBS6_AERONET_ground_20250726_AERmon_od440aer[_.]*nc
2026-07-13 01:00:01,609 UTC [3801353] ERROR   Set 'log_level' to 'debug' to get more information
2026-07-13 01:00:01,610 UTC [3801353] INFO    Creating preprocessor task od440aer_climatologies/od440aer_season
2026-07-13 01:00:01,610 UTC [3801353] INFO    Creating preprocessor 'ma_season_mean' task for variable 'od440aer_season'
2026-07-13 01:00:01,644 UTC [3801353] INFO    Found input files for Dataset: od440aer, AERmon, CMIP6, UKESM1-0-LL, historical, r1i1p1f2, mon, gn, v20190406
2026-07-13 01:00:01,661 UTC [3801353] INFO    Found input files for Dataset: od440aer, AERmon, CMIP6, HadGEM3-GC31-LL, historical, r1i1p1f3, mon, gn, v20190624
2026-07-13 01:00:01,763 UTC [3801353] INFO    Found input files for Dataset: od440aer, AERmon, CMIP6, EC-Earth3-AerChem, historical, r1i1p1f1, mon, gn, latest
2026-07-13 01:00:01,776 UTC [3801353] INFO    Found input files for Dataset: od440aer, AERmon, CMIP6, GFDL-ESM4, historical, r1i1p1f1, mon, gr1, v20190726
2026-07-13 01:00:01,792 UTC [3801353] INFO    Found input files for Dataset: od440aer, AERmon, CMIP6, MPI-ESM-1-2-HAM, historical, r1i1p1f1, mon, gn, v20190627
2026-07-13 01:00:01,795 UTC [3801353] INFO    PreprocessingTask od440aer_climatologies/od440aer_season created.
2026-07-13 01:00:01,795 UTC [3801353] ERROR   Could not create all tasks
2026-07-13 01:00:01,795 UTC [3801353] ERROR   Missing data for preprocessor od440aer_climatologies/od440aer:
- Missing data for Dataset: od440aer, AERmon, OBS6, AERONET, mon, 20250726
2026-07-13 01:00:01,795 UTC [3801353] ERROR   Not all input files required to run the recipe could be found.
2026-07-13 01:00:01,795 UTC [3801353] ERROR   If the files are available, please check the debug log and the configuration of your data sources.
2026-07-13 01:00:01,795 UTC [3801353] ERROR   To see your current configuration, run the command `esmvaltool config show` and check the "data" entries.
2026-07-13 01:00:01,795 UTC [3801353] ERROR   To see the available example configurations, run the command: `esmvaltool config list`.
2026-07-13 01:00:02,385 UTC [3801353] INFO    Maximum memory used (estimate): 0.3 GB
2026-07-13 01:00:02,386 UTC [3801353] INFO    Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2026-07-13 01:00:02,387 UTC [3801353] ERROR   Could not create all tasks

[jlenh]

We had a discussion during the latest releases regarding this, maybe we should find a way to easily handle this? (see #4171 for v2.13.0 for example)