cfradial1 to cfradial2 failed

[chiaweh2]

Hi all,

Thanks again for solving the issue #317. I have a different file from the same field experiment that got stuck on a similar issue but different error message. (other files seems to work fine with the new version). The sample code to generate the error is below (the data file is shared through url this time for easy testing)

import xarray as xr

ds = xr.open_dataset('https://data.gdex.ucar.edu/special_projects/ard_radar/kerchunk/cfrad.20220525_021216.343_to_20220525_021248.129_SPOL_PrecipRhi1_RHI.nc', engine='h5netcdf',decode_timedelta=False,chunks={})

ds.xradar.to_cf2()

The error message is

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[7], [line 1](vscode-notebook-cell:?execution_count=7&line=1)
----> [1](vscode-notebook-cell:?execution_count=7&line=1) ds.xradar.to_cf2()

File ~/test_xradar/.venv/lib/python3.11/site-packages/xradar/accessors.py:180, in XradarDataSetAccessor.to_cf2(self)
    178 def to_cf2(self):
    179     """Alias for CfRadial1 to CfRadial2."""
--> [180](https://vscode-remote+ssh-002dremote-002bcasper-002ehpc-002eucar-002eedu.vscode-resource.vscode-cdn.net/glade/u/home/chiaweih/test_xradar/test/~/test_xradar/.venv/lib/python3.11/site-packages/xradar/accessors.py:180)     return self.to_cfradial2_datatree()

File ~/test_xradar/.venv/lib/python3.11/site-packages/xradar/accessors.py:172, in XradarDataSetAccessor.to_cfradial2_datatree(self)
    170 def to_cfradial2_datatree(self):
    171     """Convert a CfRadial1 Dataset to CfRadial2 DataTree."""
--> [172](https://vscode-remote+ssh-002dremote-002bcasper-002ehpc-002eucar-002eedu.vscode-resource.vscode-cdn.net/glade/u/home/chiaweih/test_xradar/test/~/test_xradar/.venv/lib/python3.11/site-packages/xradar/accessors.py:172)     return to_cfradial2(self.xarray_obj)

File ~/test_xradar/.venv/lib/python3.11/site-packages/xradar/transform/cfradial.py:157, in to_cfradial2(ds, **kwargs)
    151     dtree["georeferencing_correction"] = DataTree(
    152         georeferencing, name="georeferencing_correction"
    153     )
    155 # Attach sweep child nodes
    156 sweep_groups = list(
--> [157](https://vscode-remote+ssh-002dremote-002bcasper-002ehpc-002eucar-002eedu.vscode-resource.vscode-cdn.net/glade/u/home/chiaweih/test_xradar/test/~/test_xradar/.venv/lib/python3.11/site-packages/xradar/transform/cfradial.py:157)     _get_sweep_groups(
    158         ds,
    159         sweep=sweep,
    160         first_dim=first_dim,
    161         optional=optional,
    162         site_coords=True,
    163     ).values()
    164 )
    165 dtree = _attach_sweep_groups(dtree, sweep_groups)
    167 return dtree

File ~/test_xradar/.venv/lib/python3.11/site-packages/xradar/io/backends/cfradial1.py:198, in _get_sweep_groups(obj, sweep, first_dim, optional, site_coords)
    195 ds_vars = ds_vars.reset_coords(coords_to_drop, drop=True)
    197 # stack/unstack to extract variables into 2 dimensions
--> [198](https://vscode-remote+ssh-002dremote-002bcasper-002ehpc-002eucar-002eedu.vscode-resource.vscode-cdn.net/glade/u/home/chiaweih/test_xradar/test/~/test_xradar/.venv/lib/python3.11/site-packages/xradar/io/backends/cfradial1.py:198) ds_vars = ds_vars.stack(n_points=["time", "range"])
    199 ds_vars = ds_vars.unstack("n_points")
    201 ds = ds.drop_vars(ds_vars.data_vars)

File ~/test_xradar/.venv/lib/python3.11/site-packages/xarray/util/deprecation_helpers.py:144, in deprecate_dims.<locals>.wrapper(*args, **kwargs)
    136     emit_user_level_warning(
    137         f"The `{old_name}` argument has been renamed to `dim`, and will be removed "
    138         "in the future. This renaming is taking place throughout xarray over the "
   (...)    141         PendingDeprecationWarning,
    142     )
    143     kwargs["dim"] = kwargs.pop(old_name)
--> [144](https://vscode-remote+ssh-002dremote-002bcasper-002ehpc-002eucar-002eedu.vscode-resource.vscode-cdn.net/glade/u/home/chiaweih/test_xradar/test/~/test_xradar/.venv/lib/python3.11/site-packages/xarray/util/deprecation_helpers.py:144) return func(*args, **kwargs)

File ~/test_xradar/.venv/lib/python3.11/site-packages/xarray/core/dataset.py:5289, in Dataset.stack(self, dim, create_index, index_cls, **dim_kwargs)
   5287 result = self
   5288 for new_dim, dims in dim.items():
-> [5289](https://vscode-remote+ssh-002dremote-002bcasper-002ehpc-002eucar-002eedu.vscode-resource.vscode-cdn.net/glade/u/home/chiaweih/test_xradar/test/~/test_xradar/.venv/lib/python3.11/site-packages/xarray/core/dataset.py:5289)     result = result._stack_once(dims, new_dim, index_cls, create_index)
   5290 return result

File ~/test_xradar/.venv/lib/python3.11/site-packages/xarray/core/dataset.py:5237, in Dataset._stack_once(self, dims, new_dim, index_cls, create_index)
   5234 indexes = {k: v for k, v in self._indexes.items() if k not in drop_indexes}
   5235 indexes.update(new_indexes)
-> [5237](https://vscode-remote+ssh-002dremote-002bcasper-002ehpc-002eucar-002eedu.vscode-resource.vscode-cdn.net/glade/u/home/chiaweih/test_xradar/test/~/test_xradar/.venv/lib/python3.11/site-packages/xarray/core/dataset.py:5237) return self._replace_with_new_dims(
   5238     new_variables, coord_names=new_coord_names, indexes=indexes
   5239 )

File ~/test_xradar/.venv/lib/python3.11/site-packages/xarray/core/dataset.py:925, in Dataset._replace_with_new_dims(self, variables, coord_names, attrs, indexes, inplace)
    916 def _replace_with_new_dims(
    917     self,
    918     variables: dict[Hashable, Variable],
   (...)    922     inplace: bool = False,
    923 ) -> Self:
    924     """Replace variables with recalculated dimensions."""
--> [925](https://vscode-remote+ssh-002dremote-002bcasper-002ehpc-002eucar-002eedu.vscode-resource.vscode-cdn.net/glade/u/home/chiaweih/test_xradar/test/~/test_xradar/.venv/lib/python3.11/site-packages/xarray/core/dataset.py:925)     dims = calculate_dimensions(variables)
    926     return self._replace(
    927         variables, coord_names, dims, attrs, indexes, inplace=inplace
    928     )

File ~/test_xradar/.venv/lib/python3.11/site-packages/xarray/core/variable.py:3143, in calculate_dimensions(variables)
   3141             last_used[dim] = k
   3142         elif dims[dim] != size:
-> [3143](https://vscode-remote+ssh-002dremote-002bcasper-002ehpc-002eucar-002eedu.vscode-resource.vscode-cdn.net/glade/u/home/chiaweih/test_xradar/test/~/test_xradar/.venv/lib/python3.11/site-packages/xarray/core/variable.py:3143)             raise ValueError(
   3144                 f"conflicting sizes for dimension {dim!r}: "
   3145                 f"length {size} on {k!r} and length {dims[dim]} on {last_used!r}"
   3146             )
   3147 return dims

ValueError: conflicting sizes for dimension 'n_points': length 615670 on 'n_points' and length 771210 on {'n_points': 'DBMHC'}

And input related to the error? It seems to get n_points of 615670 but the n_points in the netcdf file is 867162 which confused me a bit on which n_points it is referring to.

[kmuehlbauer]

@chiaweh2 Sorry for the delay. This fell through the cracks.

AFAICT, this is due to time = 434, range = 1999 needing n_points of 434*1999 = 867566. But n_points dimension is only 867162 (404 less).

What might be a bit confusing is, that the algorithm tries to extract the different sweeps from the source array. This finally leads to smaller n_points.

We might try to make the algorithm more robust trying to overcome issues like that by extracting a fitting and consistent subset. As this is not priority one for me, any volunteer is welcome.

[chiaweh2]

No worries! Thank you for looking into this and for the explanation.