Add PCB-Merging and harden the web server for public deployment - #1
Merged
Merged
Conversation
PCB-Merging (Du et al., NeurIPS 2024): - pcb_merge() with intra-balancing (self-competition) x inter-balancing (cross-task consensus) scoring, top-ratio drop, score-weighted combine - --method pcb with --pcb-ratio/--pcb-lambda/--pcb-scope; server passthrough; frontend method + fields; preset; docs; unit tests Security hardening (all opt-in / safe-by-default; no new runtime deps): - Uploads are safetensors-only by default; pickle/code files refused (_scan_unsafe_files). Every torch.load uses weights_only=True. Closes an arbitrary-code-execution path on uploaded checkpoints. - Archive extraction gains zip-bomb defenses (entry count, uncompressed total enforced during copy, per-entry ratio) on top of the existing Zip-Slip guard. - Merge worker is sandboxed: stripped environment, job-private cwd/HOME/TMPDIR, rlimits (CPU/file size), hard wall-clock timeout with process-group kill, and no network via an unprivileged namespace (unshare -rn) when available. - Opt-in auth (MERGESE_REQUIRE_AUTH): hashed API keys + short-lived anonymous tokens (Turnstile-gated), per-caller daily/active-job quotas, and per-job ownership. Offline admin CLI (server/manage_keys.py). Off by default, so a trusted single-tenant install is unchanged and writes no auth artifacts. Also: point the web UI GitHub link at srlabUsask/MergeSE; add flask to the dev extra so the server tests run under CI; docs/SECURITY.md deployment runbook.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PCB-Merging (Du et al., NeurIPS 2024):
Security hardening (all opt-in / safe-by-default; no new runtime deps):
Also: point the web UI GitHub link at srlabUsask/MergeSE; add flask to the dev extra so the server tests run under CI; docs/SECURITY.md deployment runbook.