feat(zarr-metadata): partial metadata types

[d-v-b]

Adds partial forms of ArrayMetadataV2, ArrayMetadataV3, GroupMetadataV2, GroupMetadataV3 typeddicts, i.e. typeddicts where all keys are optional. Today, these types can be consumed by zarr-python test code (in a follow-up PR). In the future, if we every want to expose type-safe APIs for updating array / group metadata, then these types would be useful there.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.34%. Comparing base (8bfcc66) to head (acebb29).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3982   +/-   ##
=======================================
  Coverage   93.34%   93.34%           
=======================================
  Files          88       88           
  Lines       11819    11819           
=======================================
  Hits        11033    11033           
  Misses        786      786           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[d-v-b]

thank you max!

[chuckwondo]

@d-v-b, I'm still not convinced these are necessary. Using zarr-rs as a reference point, offhand I don't see any such equivalents. Am I missing something?

[d-v-b]

@d-v-b, I'm still not convinced these are necessary. Using zarr-rs as a reference point, offhand I don't see any such equivalents. Am I missing something?

I agree that these are not strictly necessary! they are just a convenience that zarr-python will only use in tests, to ensure that we can properly annotate fragments of metadata documents. but I anticipate that other libraries might find these partial types useful in other applications.

[chuckwondo]

@d-v-b, I'm still not convinced these are necessary. Using zarr-rs as a reference point, offhand I don't see any such equivalents. Am I missing something?

I agree that these are not strictly necessary! they are just a convenience that zarr-python will only use in tests, to ensure that we can properly annotate fragments of metadata documents. but I anticipate that other libraries might find these partial types useful in other applications.

That's part of my point though. I don't think these are necessary for tests either. Rather, I'm suggesting that the tests should be written differently. I'm failing to see what you feel cannot be properly tested without these types? Why do you think they're needed here, when they are not needed in zarr-rs (as far as I can tell, so please correct me if I missed something).

In my view, tests should generally be reflective of real usage, and these types should not be available to users, so I'm still not seeing their necessity. Again, perhaps I'm missing something. I feel that these additional types may simply add undue clutter/complexity/maintenance overhead and potential for confusion/cognitive overload should users discover them.

[d-v-b]

I'm failing to see what you feel cannot be properly tested without these types?

There's nothing that can't be properly tested without these types. These types are helpful for any test that consumes a fragment of a metadata document, like this one.

But these types are required for making this API type safe:

@dataclass
class ArrayMetadata:
    shape: tuple[int, ...]
    # remaining array metadata keys as data class fields
    ...
    def update(self, **changes: Unpack[ArrayMetadataV3Partial]) -> Self: ...

The above API is a very sane API we should support. A single update is more ergonomic than with_shape, with_dtype, etc, because zarr metadata couples different fields together, and so you sometimes have to update the shape and the chunk grid together, for example.

[chuckwondo]

I'm still not convinced, but we can conclude the conversation for now, as I suspect I still have significant gaps in my understanding that I need to fill before I can see things more clearly. My lack of being convinced at this point is based on my general development experience, not my specific (lack of) understanding of the zarr ecosystem, so gaining more knowledge of it may very well change my perspective here.

Thanks for your patience with my push backs 😅

[d-v-b]

no worries! I could also be wrong, and maybe these types are useless (or python gets a Partial utility type :) ). Worst case scenario nobody uses these types and that's an OK outcome

[maxrjones]

fwiw I held some of the same skepticism as @chuckwondo, but think we should value experimentation in the metadata library to try different approaches and iterate faster towards a stable v1 API. IMO we do not need to have the same standard as in the core library that everything should be undeniably useful. That tolerance for experimentation is part value of using this monorepo approach.