Repository navigation
Conversation
Tests for reading around a member much slower than the rest of its stripe: a read of its data is reconstructed from the stripe's other blocks when its latency estimate says that's faster. Five or six 1G members, 3+1 stripes unless noted, and a dm-delay member at 500 ms. - ec_read_slow_device: reads of the slow member's data don't wait for it. - ec_read_healthy, ec_read_around_disabled: no read-arounds on a healthy pool, or with ec_read_around_penalty=0. - ec_read_around_pq: 4+2 stripes with two slow members. - ec_read_around_fallback, _fallback_reconstruct: a read-around that reads a bad block falls back to the member, then to the full reconstruct. - ec_read_around_small_extents, ec_read_around_nocsum: read amplification with 4k extents, and unchecksummed data. - ec_repair_reads_around_slow_device, ec_reconcile_reads_around_slow_device: evacuation and reconcile rebuild a slow member's blocks from the rest of the stripe. - ec_reconcile_learns_failing_device, ec_evacuate_learns_failing_device: from a cold estimate, with a member at 7 s per read, reconcile's and evacuation's own reads bring its estimate up. - ec_reconstruct_skips_failed_block: a reconstruct after a read error doesn't read the failed block again. Tests with a delayed member warm its estimate first with random reads, read-around off. Needs the read-around series. Replaces koverstreet#130, whose tests asserted that an evacuating device is never read.
With a promote target, a read-around promotes as a direct read does, so a second read of a slow member's data comes from the promote target. Five members in group hdd, two behind dm-delay, and a promote target in group ssd, set only after the setup's reads and the prewarm. The first pass reads around; the second must not read around at all, and must read most of the file from the promote target. ec_ra_setup takes per-device labels from ec_ra_labels. Needs the read-around series with "ec: promote read-around results".
Tests for the stripe scrub a read queues when it finds a bad block. ec_read_around_fallback_scrubs_stripe and ec_read_around_nocsum_scrubs_stripe corrupt the parity block of three stripes with data on a slow member, read the file, wait for the stripes to be rebuilt, and read it again. With that patch, the reads in ec_read_around_fallback_reconstruct queue a scrub too. Its repair reads the bad blocks, counted as stripe_read_csum_err, and re-keys extents the reads are self healing; a self heal that loses that race gives up, and four of them are past the slowpath counter check's slack. Both are expected there now. ktest_expect_counters, shaped like ktest_expect_sb_errors, lets a test declare the slowpath counters its design makes unavoidable.
On ec_read_around the stripe repair reads the bad parity block, which counts stripe_read_csum_err as scrub's own repair does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tests for bcachefs-tools#1100: a read that finds a bad stripe block queues a scrub of the stripe, which repairs it. The first two commits are #136; they're here until it merges.
ec_read_around_fallback_scrubs_stripe and ec_read_around_nocsum_scrubs_stripe corrupt the parity block of three stripes with data on a slow member, read the file, wait up to 300 s for the stripes to be rebuilt, and read it again. Without the patch, all three bad blocks are still there after 300 s and the second read fails the same way.
The patch also makes ec_read_around_fallback_reconstruct's reads queue a scrub, whose repair races the reads' own self heals; ktest_expect_counters, a new knob in bcachefs-test-libs.sh shaped like ktest_expect_sb_errors, lets that test declare the data_update_key_fail it can't avoid. On testing 8e6613cec with the patch, all 16 read-around tests and replication.ktest's three EC scrub tests pass.