Repository navigation
Keep a stuck graphics driver from holding the window at start - #206
Merged
Merged
Conversation
Startup's GPU detection runs under the stall guard in a new gpu_detect lane (1 s warning, 4 s limit), as in #166. The same lane now also holds every other graphics call before the window's first paint: the banner's and the window's screen lists, the renderer's DXGI adapter list, and a wgpu probe that lists the window's adapters on another thread before the window's thread makes its first graphics call. After one call hangs, the others are refused at once. While the probe does not return, or an earlier call is still stuck, a message box that Windows draws without the graphics driver says why the window is not there yet, and the window opens once the driver answers. The marker names the probe's thread while it lists adapters, so a driver crash there still counts against the window's backend. A detection given up on keeps what it finds when it returns. The next pipeline, as after Restart, uses that graphics card. Until then the processor encodes, and the Stream page shows the software-encoding notice, saying the driver did not respond at startup.
…x/gpu-detect-timebox-035 # Conflicts: # host/src/supervisor.rs
The candidate's log-capturing tests call prepare_log_capture first, so a stall line another test's thread logged before any subscriber still reaches the capture. The detection test asserts that line too.
…x/gpu-detect-timebox-035 # Conflicts: # host/src/supervisor.rs
Review of the first version found five gaps; this closes them. - Crash avoidance read an empty GPU list while detection was stuck: its DXGI list was refused in the startup lane, so a known-bad backend was picked again and a crash recorded under an empty key. The window now waits for the startup lane to free first, then configures the renderer, then runs the wgpu probe, in a lane of its own (stall=gpu_probe). - The window's thread still creates its device and swap chain unwatched. A watcher now runs from run_native until the first frame is presented; past the 4 s limit it shows the same notice, closed by the first frame. Cancel there runs the app's quit path, now one closure main.rs and the watcher share, and ends the process. - After OK the notice never came back, so quitting was impossible while the driver stayed stuck. OK now hides it for 15 s. A Cancel that lands as the driver answers still quits. - After the first frame, the renderer's DXGI list (as a renderer rebuilt after a lost device makes it) uses the gpu_facts lane, not startup's. While it lists, the crash marker names its thread, so a driver crash there is still the window's. - The tests no longer depend on timing: the notice, probe and watcher tests order events on channels and a hand-moved clock, and the in-time detection test has a 1 s warning threshold.
…x/gpu-detect-timebox-035 # Conflicts: # host/src/main.rs
…x/gpu-detect-timebox-035 # Conflicts: # host/src/encoder/mod.rs # host/src/gui/mod.rs # host/src/gui/page_stream.rs # host/src/gui/tests.rs # host/src/stats.rs
…that returned Second review of #206 found five more gaps; this closes them. - The first-paint watcher stopped at the second UI pass, which egui counts whether or not a frame was presented: egui-wgpu returns from a paint without presenting after a failed acquire, and recreates the surface at the start of the next one. The watch now ends only when the renderer confirms the window drawing (frames for a while with no surface error, visible). The notice goes up when the window's thread has begun no UI pass for 4 s; the window repaints at least every 200 ms, and a minimized one is left alone. - Configure could still commit an empty GPU key when its own DXGI list timed out. The list now runs before configure, in the startup lane, until one returns in time; configure gets that list, and the window keys its adapters by it until it is confirmed drawing. - A Cancel from the watcher left the renderer's marker, so the next launch counted a deliberate quit as a failed start. Cancel now clears it, and the marker and that quit share one lock, so a worker still in the driver cannot save it again. - The end of the wait, the user's answer and raising the notice now change one state under one lock: a Cancel that comes as the window draws still quits, and no notice goes up after the wait ended. - The first DXGI list had no marker. It now leaves one before it starts and names its thread once it runs, until a later step replaces it. A launch that dies there makes the next ones key the GPUs by wgpu's driver text instead, until clean launches retry the list.
…x/gpu-detect-timebox-035
… a slow driver Third review of #206 found five more gaps; this closes them. - The watcher counted a minimized window as stuck: eframe runs no UI pass for one, so the renderer never saw it minimized, and a healthy host got the driver notice 4 s after Win+D. The watcher now checks the window live (IsIconic, or the move/size and menu loops Windows runs while the title bar is held), and counts the 4 s from the latest of its start, the last UI pass and the last time it saw the window held. - Confirming the window could still pass with nothing presented: after a failed surface recreation egui-wgpu's paints return early with no surface error. The renderer now asks for a screenshot once the window has drawn long enough, and confirms only when it arrives; egui-wgpu takes one only from a frame it acquired, drew and submitted. - A driver that always answers after 4 s never opened the window: each try was given up on and started again. The list and the probe are now one call each, waited for as long as they run; the notice goes up at their limit, and a late answer opens the window. - Tests whose steps answer use 60 s limits, and the watcher's panics reach the test. - The notice box closes on a thread of its own, so the window's thread never waits on it.
…x/gpu-detect-timebox-035 # Conflicts: # host/src/gui/renderer.rs
…x/gpu-detect-timebox-035 # Conflicts: # host/src/gui/mod.rs # host/src/main.rs # host/src/stall.rs
Fourth review of #206 found three more gaps; this closes them. - A window whose paints return without a surface keeps running UI passes, so the watcher's 4 s without a pass never came, and its screenshots were asked for again and again with no notice. The watcher now also raises the notice when the window has run UI passes for 12 s without being confirmed drawing (a working one is at about 3.5 s). Both bounds count from the latest of the start and the last time the window was held, so a minimized or dragged window suspends them. - Any screenshot marked as the proof was accepted, also a late one from before a surface error or a new device. Each ask now carries its span (a new one after each surface error, skipped frame or device) and a request id, and only an answer to the current span's asks confirms. - The watcher's decisions are a step function the tests call with each state in turn, instead of a thread they hoped had looked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #166, which was held after 0.3.4 for two review findings. Both are fixed here, along with the findings of two reviews of this PR.
What changed
A graphics driver deadlock on an AMD laptop held the host's startup for 62 s inside
GpuInfo::detect, and the window showed nothing for that minute. #166 time-boxed detection. Review found that the window still made its own graphics calls before the first paint:AnalyzerApp::newlists the screens, and the renderer and wgpu create their own DXGI factories. It also found that a timeout left the whole session on the processor (libx264, or libx265 with HEVC) with no notice.Startup lane.
GpuInfo::detectruns under the stall guard in a newgpu_detectlane: warning at 1 s, limit at 4 s (from #166). The same lane holds the screen lists for the startup banner (main.rs) and the window (AnalyzerApp::new), and the window's DXGI adapter list. After one hangs, the others are refused at once instead of each waiting out its own limit. A refused or timed-out screen list is empty: the banner saysunknown, and the screen picker fills in when Settings opens.Before the window (
gui/driver_wait.rs,before_window). The window's thread lists adapters, creates its device and draws through the driver, and Windows can't stop a thread stuck there. So beforerun_native:gpu_probe, same limits): an instance with this launch's backends lists its adapters, and DX12 creates a device on each to describe it. The window opens once it returns.The list and the probe are one call each, waited for as long as they run: past the 4 s limit the notice goes up, and a driver that answers late still opens the window (the late result is used, not thrown away and tried again).
Until the window draws (
watch_first_paint,Watch::look). The window's own device and swap chain, and a surface it recreates after a failed acquire, can't be moved off its thread. A watcher runs until the renderer confirms the window drawing, and raises the notice when either:Both are counted from the latest of the watch's start and the last time the window was held: minimized (
IsIconic), or its title bar or system menu held (GetGUIThreadInfo:GUI_INMOVESIZE,GUI_INMENUMODE). eframe runs no UI pass for a minimized window, so the watcher checks the window itself, live.Confirmed drawing. The renderer confirms the window only after frames for a while with no surface error, no skipped frame and no new device, visible, and then a screenshot: egui-wgpu (also #201's vendored painter) answers one only from a frame it acquired, drew and submitted. Each ask carries its span (a new one after each surface error, skipped frame or device) and a request id, and only an answer to the current span's asks confirms; a late answer from before a change proves nothing.
The notice. Whenever the window waits on the driver, a message box says the graphics driver is not responding and that the window will open by itself. Windows draws it without the graphics driver.
main.rsand the watcher share, run once by whichever comes first. A Cancel also clears the renderer's crash marker, so the next launch does not count a deliberate quit as a failed start.Crash markers. The DXGI list leaves a marker before it starts and names its thread once it runs; the probe names its thread while it runs. A driver crash in either is the window's, as one on the window's thread is. A launch that dies listing makes the next ones key the GPUs by wgpu's driver text instead, until clean launches retry the list. The marker and a quit share one lock, so a worker still in the driver can't save it again after a Cancel. After the window is confirmed, a renderer rebuilt later (after a lost device) lists adapters in the
gpu_factslane, so it neither waits behind nor holds the startup lane.The late result is used. A detection given up on keeps what it found when it returns (
gpu::take_late_detection). The next pipeline (Restart, or any later restart) takes it and encodes on the graphics card. Until then the processor encodes, and the Stream page shows the software-encoding notice from the start: "The graphics driver did not respond when EternalMonitor started, so the processor encodes the video for now. … Click Restart to try the graphics card again." Its button is Restart. A detection that answered and found no graphics card, or libx264 chosen in Settings, shows no notice, as before.This doesn't touch the
encoder_openstall path or the window's handling of device loss once it is up.Tests
gpu::tests: a stuck detection gets the software facts, marked as stalled, at its limit, with onestall=gpu_detectWARN. Once it returns, its facts are kept for later pipelines. A detection that returns in time keeps its facts; a failed one is not marked.gui::driver_wait::tests. The steps before the window order events on channels and use 60 s limits for steps that answer; the watcher's decisions are a step function (Watch::look) the tests call with each state in turn:renderer::tests: confirmation needs the screenshot proof, and frames that present nothing are never confirmed; a late answer from before a surface error, or from before skipped frames and a new device, proves nothing; a quit from the notice leaves no marker and no late worker writes it again; a launch that dies listing is recorded and skips the list next time; after the window is confirmed, the adapter list leaves the startup lane.supervisor::tests::a_late_detection_is_used_by_the_next_pipeline_only_after_a_stall.tests/gpu_detect_fallback.rsruns the real encode stage on the stalled facts. The processor encodes with the notice flags set; libx264 chosen in Settings and a detection that found no card set neither. With the oldset_software_fallback(false)it fails with(false, true).gui::tests::a_driver_that_did_not_answer_at_startup_is_named_with_a_restart: the stalled detection is named first, before custom settings and the driver hints for NVIDIA, AMD and Intel from Stream on libx264 when a hardware encoder open hangs, remember failed encoders, change Quick Sync's bitrate in place #202.Checks
cargo test -p eternal-host --no-fail-fast: 1225 passed, including all 13window_device_losstests;cargo clippy -p eternal-host --all-targets -- -D warningsis clean. The AMF hardware tests failed in that run: AMF did not open on the AMD iGPU of the shared test PC. In an A/B right after,amf_encoderpassed 5 of 5 at this head and failed 2 tests in 1 of 5 on the candidate (241fa1b).cargo test -p eternal-host(all 1240),cargo clippy --workspace --all-targets -- -D warnings,cargo fmt --check.Native runs (Windows 11, RTX 5080, at 8f660c2)
Test copies of the host, debug builds, synthetic capture, against the candidate. A local test-only patch held
detect_from_dxgifor 20 s, like a stuck driver, in both builds.Window renderer is drawingcame once the screenshot proof was answered.stall=gpu_detect after_ms=4019, then the notice. The list and the probe answered (The graphics driver answered call="gpu_detect" waited_ms=16019, thencall="gpu_probe"), and the window opened with the software-encoding notice.this pipeline uses its graphics card … encoder=h264_nvenc.Release notes
EternalMonitor's window no longer waits up to a minute to appear when the graphics driver stops responding at startup. It says the driver isn't responding, opens once the driver answers, and the Stream page explains why video uses the processor until Restart.