Repository navigation
fix(ci): retry databend testcontainers bring-up past the image's metasrv race - #44729
Conversation
…srv race The `testcontainers (databend, 10)` job fails intermittently on master -- most recently in run 36296832911 (2026-09-27, master @ eb2ffa2), and before that in run 34816535396 (2026-09-14) -- with: TimeoutError: Container did not emit logs containing 'listened at 0.0.0.0:8000' within 120.000 seconds. Container status: running The cause is in the image's own entrypoint, not in Superset. In databendlabs/databend `docker/bootstrap.sh` (identical at the failing build `v1.2.881-ca29960f5c-simd`, at tag v1.2.881 and on main), the entire metasrv readiness gate is a bare `sleep 1`: databend-meta ... --single &>/tmp/std-meta.log & PROCESSES+=($!) # wait for meta to be ready sleep 1 ... databend-query -c "$QUERY_CONFIG_FILE" &>/tmp/std-query.log & PROCESSES+=($!) ... wait "${PROCESSES[@]}" Before metasrv binds its gRPC port it must finish an on-disk data upgrade (V003 -> V004, including sled-tree removal) and single-node leader election. When that overruns the one second -- occasionally, on a loaded runner -- databend-query's single connection attempt is refused (`Connection refused (os error 111)`), query exits, and nothing restarts it. The closing `wait` keeps waiting on the surviving metasrv and log-tail PIDs, so the container reports `running` forever while the banner line the wait strategy looks for can never be emitted. That rules out widening the wait timeout the way test_db2.py legitimately needs to: no timeout can make a dead one-shot process re-emit a line. Only a fresh container can re-run the race, so bring-up is retried up to three times, stopping each container it gives up on and re-raising the original TimeoutError once the attempts are exhausted. The durable fix belongs upstream (poll the gRPC port instead of sleeping, and supervise databend-query rather than letting `wait` mask its death); this keeps master green in the meantime. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #44729 +/- ##
==========================================
- Coverage 81.13% 81.13% -0.01%
==========================================
Files 2956 2956
Lines 178453 178453
Branches 41345 41345
==========================================
- Hits 144784 144782 -2
- Misses 30965 30967 +2
Partials 2704 2704
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![bito-code-review[bot]](https://avatars.githubusercontent.com/in/1061978?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review Agent Run #860174
Actionable Suggestions - 1
-
tests/testcontainers/db_engine_specs/test_databend.py - 1
- Container leak on start failure · Line 84-90
Review Details
-
Files reviewed - 1 · Commit Range:
5907a32..5907a32- tests/testcontainers/db_engine_specs/test_databend.py
-
Files skipped - 0
-
Tools
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
-
/review- Manually triggers an incremental AI Review. -
/review full- Manually triggers a full AI Review. -
/pause- Pauses automatic reviews on this pull request. -
/resume- Resumes automatic reviews. -
/resolve- Marks all Bito-posted review comments as resolved. -
/abort- Cancels all in-progress reviews.
Refer to the documentation for additional commands.
Configuration
This repository uses Superset You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.
Documentation & Help
AI Code Review powered by
| try: | ||
| container.start() | ||
| break | ||
| except TimeoutError: | ||
| container.stop() | ||
| if attempt == START_ATTEMPTS: | ||
| raise |
There was a problem hiding this comment.
The try/finally at lines 92-100 only guards the post-loop path, so a non-TimeoutError exception from container.start() (docker daemon error, image pull failure, docker.errors.APIError) propagates out of the loop without container.stop(), leaking the container. Add a broad except Exception: container.stop(); raise alongside the TimeoutError handler.
Code Review Run #860174
Should Bito avoid suggestions like this for future reviews? (Manage Rules)
- Yes, avoid them
container.start() failures other than the metasrv-race TimeoutError (docker daemon error, image pull failure) previously propagated without calling container.stop(), leaking the container. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Code Review Agent Run #6a3b9eActionable Suggestions - 0Additional Suggestions - 1
Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
…srv race (#44729) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
SUMMARY
The
testcontainers (databend, 10)job fails intermittently on master — run36296832911 (2026-09-27, master @ eb2ffa2) and run 34816535396 (2026-09-14),
13 consecutive green nightly runs apart — with:
No specific introducing Superset change identified:
tests/testcontainers/db_engine_specs/test_databend.pyhas one commit in its whole history (3e13b89, #43502) and is unmodified since.
The root cause is in the image's own entrypoint. In databendlabs/databend
docker/bootstrap.sh— identical at the failing buildv1.2.881-ca29960f5c-simd,at tag
v1.2.881, and onmain— the entire metasrv readiness gate is a baresleep 1:Before metasrv binds its gRPC port it must complete an on-disk data upgrade
(V003 → V004, including sled-tree removal) and single-node leader election.
When that overruns the one second — occasionally, on a loaded runner —
databend-query's single connection attempt is refused (
Connection refused (os error 111)), query exits, and nothing restarts it. The closingwaitkeeps waiting on the surviving metasrv and log-tail PIDs, so the container
reports
runningforever while the banner line the wait strategy looks forcan never be emitted.
That rules out widening the wait timeout the way
test_db2.pylegitimatelyneeds to: no timeout can make a dead one-shot process re-emit a line. Only a
fresh container can re-run the race, so bring-up is retried up to three times,
stopping each container it gives up on and re-raising the original
TimeoutErroronce the attempts are exhausted. The comment onSTART_ATTEMPTSrecords the mechanism so this isn't later "fixed" by raising the timeout.
The durable fix belongs upstream — poll the gRPC port instead of sleeping, and
supervise
databend-queryrather than lettingwaitmask its death. This keepsmaster green in the meantime.
TESTING INSTRUCTIONS
Only the databend testcontainers fixture changes, so:
SUPERSET_TESTCONTAINERS_STRICT=true pytest -v -m testcontainers ./tests/testcontainers/db_engine_specs/test_databend.pyBoth tests should pass, and on a clean first start the log should show a
single
datafuselabs/databendcontainer (the retry adds no work to thehappy path).
by hand with a pre-populated V003 meta dir, or throttle its I/O
(
docker run --device-read-bps ...) so metasrv's upgrade overrunsbootstrap.sh'ssleep 1.docker ps -ashould show an abandonedcontainer removed and a second one started, with the suite still passing.
workflow_dispatch— and confirm thedatabendleg is green.ADDITIONAL INFORMATION
🤖 Generated with Claude Code