Chunk batch feed - #16
Merged
Merged
Conversation
…16) The original chunk-batch-feed branch (PR #16, commits d5c8a66/5e84268/ 7451b09) was branched before both the multi-frame decode redesign and the expand-based dtype dispatch, and before the c2py23 dependency was dropped from the build -- rebasing it directly produced conflicts against code that no longer exists in the same shape, so this re-implements its two genuinely new capabilities from scratch against the current architecture rather than resolving those conflicts hunk by hunk: - note_chunk(): an extern "C" function whose only job is writing a chunk's address+length into caller-owned pointers/lengths arrays, so c2py23's own buffer acquisition (on note_chunk's "chunk" parameter) extracts the address -- not numpy's .ctypes.data. _gather_chunks() in __init__.py wraps a Python loop over this. compressed_ptrs/ compressed_lengths still come back as int64/int32 numpy arrays rather than raw bytearrays: bslz4_multi_*/bslz4_csc_multi_* infer nframes from compressed_ptrs.itemsize (== 8 for int64), which a byte-oriented bytearray (itemsize 1) can't satisfy -- and numpy is already a hard dependency of this package elsewhere (masks, outputs), so keeping the scratch arrays as numpy costs nothing while still dropping ctypes. chunk2sparse/chunk2sparseCSC/chunk2sparseMulti/chunk2sparseCSCmulti/ bslz4_to_sparse() all build their frame-pointer arrays via _gather_chunks() now instead of npbuf()+.ctypes.data; buffers can be bytes/bytearray/memoryview/mmap slices directly, no numpy conversion needed first (note_chunk's C parameter is read-only, so nothing needs a writability workaround either). _blocksize_bytes() now reads via len()/indexing rather than .size, so it also works on those directly. - bslz4_csc_multi_base_<suffix>: like bslz4_csc_multi_<suffix>, but for chunks that all share one base buffer (e.g. an mmap'ed or fully-read HDF5 file) addressed by byte offset rather than one buffer object per frame -- avoids a read_direct_chunk() call per frame for callers who already have the whole file. harvest_chunk_offsets() enumerates a (1,ni,nj)-chunked dataset's {frame: (byte_offset, byte_size)} via h5py's chunk_iter/get_chunk_info; pack_offsets_lengths() packs a chosen frame list into the int64/int32 arrays the C function expects. Re-targeted at the current bslz4_csc_decode_multi (dense/sparse routed) and generated via expand like the other three families (10 dtypes x 3 backends, not the original's 8 x 2) rather than the pre-expand hand-duplicated style; the old bslz4_core.hpp px>0->!=0 fix and TODO the original branch carried are moot here, already present in the rewritten bslz4_csc_decode_multi (see bug_variant.md-adjacent history on the multichunk-csc-redesign branch). No higher-level class wraps bslz4_csc_multi_base_* yet, matching the original branch's own scope -- it's exposed as a building block. test_dot.py: harvest_chunk_offsets/pack_offsets_lengths/ bslz4_csc_multi_base_* have no other coverage (note_chunk/_gather_chunks are exercised implicitly by every existing test now, since the base classes route through them), so test_csc_multi_base_matches_multi reads sparsetest.h5's own bytes as the shared base buffer and checks the result against chunk2sparseCSCmulti (already pyFAI-validated above) reading the same chunks. All 8 tests pass; also verified building with zero c2py23 installed still works (c2py_runtime/, wrapper regenerated from the same v0.5.4 tag the vendored runtime matches). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jonwright
force-pushed
the
chunk-batch-feed
branch
from
September 15, 2026 08:03
7451b09 to
3c4e928
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.