Feature/qapp reticulum networking

.gitignore

@@ -10,6 +10,9 @@ lerna-debug.log*
 node_modules
 .emsdk
 native/libopus-fec-wasm/build
+electron/resources/private-transport/*/qortal-private-transport
+electron/resources/private-transport/*/qortal-private-transport.exe
+electron/tools/masque-relay/
 dist
 dist.zip
 dist-ssr
@@ -70,3 +73,6 @@ electron/resources/reticulum-runtime/*
 
 # PyInstaller / venv build scratch (build-rnsd-frozen.py)
 electron/.build/
+
+# Dependency notices generated by the wallet engine build
+/public/wallet-dependency-licenses.txt

docs/development.md

@@ -2,7 +2,7 @@
 
 Requirements for the development of Qortal-Hub from sources:
 
-- installation of nodejs from the [official site](https://nodejs.org/en/download)
+- installation of Node.js 22.12 or newer from the [official site](https://nodejs.org/en/download)
 - Python 3.9+ and Git on PATH for Reticulum networking during Electron development
 - an IDE like vscode or Intellij, or similar tools
 - some knowledge of React and its ecosystem

docs/masque-relay-access.md

@@ -0,0 +1,110 @@
+# MASQUE relay admission
+
+Hub's relay admission is generic transport infrastructure. QApps and application
+backends never receive relay tickets and do not participate in authorization.
+
+## Flow
+
+1. Discover signed availability over Reticulum. Public relays remain v1/v2;
+   restricted relays must use v3. Do not fall back to the old account proof.
+2. Prefer reusable connections, then public/matching-group relays, then unknown
+   membership. Coalesce selection and hedge after 300 ms, at most two concurrent
+   candidates and six attempts within a 60-second cold-path budget.
+   Fresh membership information excludes nonmatching restricted relays, even
+   if none remain. That case returns `RELAY_NO_ELIGIBLE_RELAY` without requesting
+   tickets or opening QUIC. Unknown or stale membership still permits trying
+   a relay; public relays remain eligible.
+3. For restricted relays, fetch the current public ticket descriptor through an
+   encrypted Reticulum Link. Check its relay pin, group-policy hash and key
+   commitment against the signed advertisement. Native Go additionally validates
+   the RSA key, derived key ID and shared hourly expiry.
+4. Generate three independent random tickets and blind them with CIRCL v1.6.5
+   (RFC 9474 RSABSSA-SHA384-PSS-Deterministic). Sign a one-minute issuance challenge
+   in the trusted wallet context. The challenge binds the blinded batch.
+5. Send the account proof **only through Reticulum**. The relay checks account
+   ownership and group membership, then blind-signs the batch. Unblind and
+   verify inside native Go. Store resulting tickets only in Hub main memory.
+6. Prepare pinned QUIC and redeem one anonymous ticket. No account address,
+   public key or wallet signature goes over QUIC. Only then attach a backend.
+7. Use spare single-use tickets for reconnects; refresh batches when exhausted
+   or near expiry. Reuse the authorized outer connection across QApps. Renew
+   five minutes before expiry without replacing the connection.
+8. Logout/account switch clears credentials, pending results, timers and
+   connections. There is no direct-to-backend fallback.
+
+Expiry is issuance-hour start plus 12 hours (11–12 hours of actual validity).
+Redemption never extends it. The relay commits a spent marker durably before
+forwarding, including across crashes/restarts. Existing grants and spare tickets
+do not require Core availability. A member removed from a group can retain
+access for the remaining window, subject also to Core freshness.
+
+## Protocol and packaging
+
+Sidecar version: **0.7.0**; IPC framing remains version 2. Main-only operations:
+`prepareRelayTickets`, `finalizeRelayTickets`, `prepareRelay`, `authorizeRelay`,
+`closeRelay`, `clearRelays`. Native ticket blinding state is bounded and expires
+after a minute. Prepared connection handles remain process-local.
+
+The Reticulum service uses the persistent discovery identity and destination
+`qortal-hub-v3.community-masque-relay.v1`. Request paths are `/catalog`,
+`/challenge`, `/issue`; serialized JSON requests are transported as encrypted
+RNS Link requests/resources, not announcements. The Hub Python handler uses a
+dedicated four-worker bound and tears down each temporary link after response.
+
+A v3 advertisement has a three-byte header (version; IPv6 bit plus name length
+minus one; group count), IP, uint16 port, uint32 lease expiry, TLS pin, name,
+canonical LEB128 group IDs, full 64-byte Reticulum public identity, 32-byte
+ticket-key commitment, and 64-byte Ed25519 signature. Signature domain:
+`Qortal-MASQUE-Discovery-v3\\0`. Total is at most 316 bytes, including 16
+maximum-size IDs with IPv6 and the default relay name. The shared codec ships
+beside the bridge and in frozen/packaged builds.
+
+Policy digest: SHA-256 of compact JSON
+`{"Mode":"groups","Groups":[sorted IDs]}`.
+
+Ticket key ID: SHA-256 of
+`Qortal-MASQUE-Ticket-v1\\0 || policyHex || relayPinHex || decimalEpoch || PKCS1PublicDER`.
+Epoch is floor(Unix seconds / 3600). Ticket signed message:
+`Qortal-MASQUE-Ticket-v1\\0 || keyIdHex || random32`.
+Redemption JSON contains only epoch, keyId, base64 message and base64 signature.
+
+Issuance proof type: `masque-ticket-issue-v1`, with relayPin, policy, binding,
+nonce, expiresAt. Binding is SHA-256 of compact JSON
+`{"key":keyId,"blinded":[base64 blinded messages]}`.
+The wallet adds authorAddress/authorPublicKey and Ed25519-signs the sorted JSON,
+excluding signature. Public key and signature are Base58. This domain is only
+available to Hub's own wallet context, not an iframe.
+
+QUIC control still uses `Qortal-Relay-Control: authorize`. A restricted,
+unauthorized connection returns 401 and
+`Qortal-Relay-Challenge: {"type":"masque-ticket-required-v1"}`.
+Main passes a ticket to the sidecar, which transmits `Qortal-Relay-Ticket`.
+An authorized connection returns 204 and `Qortal-Relay-Expires`.
+Renewal sets `Qortal-Relay-Renew: 1`. Account-shaped proofs are rejected
+independently by native Go and the relay.
+
+## Boundaries
+
+This assumes the Reticulum authorization path hides the user's IP. Blind signing
+prevents direct transcript-to-ticket matching, not timing correlation, small-group
+inference, malicious issuer equivocation, or deliberate ticket sharing. A signed
+key commitment is not a global key transparency system. Do not claim full anonymity
+or non-transferable membership enforcement.
+
+The standalone relay's ACCESS.md describes operator configuration, persistence,
+capacity, Core failover and deployment. Public mode remains the default; enabling
+groups requires an explicit operator choice. Update/restart Hub and the relay,
+not the Call backend/QApp. Never restore old spent-ticket state alongside retained
+signing keys.
+
+## Verification
+
+- `npm run test:private-transport` with `QORTAL_GO_BINARY` when necessary.
+- `npx vitest run electron/src/relay-access-coordinator.test.ts electron/src/relay-ticket-wallet.test.ts src/background/reticulum-signing-policy.test.ts`.
+- `python3 -m unittest presence_bridge_test` from electron/resources.
+- `go test -race ./...` from electron/native/private-transport.
+- Relay's cross-project Python test exercises the actual Hub Python handler,
+  isolated Reticulum instances, real sidecar/relay binaries and a mock Core,
+  including denial, forwarding, spare tickets and replay after restart.
+
+These checks do not replace a real macOS wallet/cross-network smoke test.

docs/private-upload-investigation.md

@@ -0,0 +1,135 @@
+# Private upload throughput investigation — 2026-09-12
+
+## Confirmed in the local transport fixture
+
+quic-go's HTTP/3 `stateTrackingStream.enqueueDatagram` silently drops received
+datagrams once its per-stream queue contains 32 packets. The outer QUIC layer
+can already have acknowledged these packets, so outer packet-loss statistics
+do not expose this discard. Inner QUIC must recover the missing tunnel packets.
+
+A test-only Go workspace and overlay instrumented that exact discard branch.
+The module cache, running Hub, and VPS binaries were not modified.
+
+The opt-in `QORTAL_BULK_BENCH=1` test in
+`electron/src/private-channel-step3.test.ts` sends 64 opaque 524,886-byte messages
+over one keyed reliable stream with two concurrent native submissions. The
+fixture returns small acknowledgements. A local UDP proxy adds 60 ms in each
+direction. It does not intentionally drop packets; its explicit drop counter
+does not measure OS socket drops or packet reordering.
+
+| HTTP/3 receive queue | Seconds for 33,592,704 bytes | MB/s (decimal) | Instrumented relay queue drops | Inner reported loss | Outer reported loss |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| 32, run 1 | 6.297 | 5.33 | 25 | 25 | 0 |
+| 32, run 2 | 9.803 | 3.43 | 194 | 194 | 0 |
+| 1024, run 1 | 2.416 | 13.91 | 0 | 191 | 13 |
+| 1024, run 2 | 3.664 | 9.17 | 0 | 972 | 458 |
+
+The larger queue is an experiment, not a proposed production default. It removes
+this discard point but does not remove all loss or reproduce a real network
+perfectly. The exact match between the instrumented discard count and inner loss
+in both stock-queue runs establishes a concrete loss mechanism in this fixture.
+
+## Production applicability and limits
+
+The VPS relay checkout at inspection was `8cbdaf3`, declaring quic-go v0.60.0
+and masque-go v0.4.0. v0.60.0 has the same 32-packet discard branch as the Hub
+fixture's v0.62.0. This identifies a relevant production risk, not a direct
+measurement of the running VPS relay's discard count.
+
+Earlier real-upload diagnostics showed inner loss with no outer loss and little
+time spent in encryption or backend disk synchronization. Those observations are
+consistent with this mechanism, but do not prove it accounts for all slow upload
+time. The fixture uses a Go receiver, not the production Python backend, and
+does not include QApp encryption, browser work, or storage.
+
+## Follow-up
+
+- Add bounded, observable tunnel receive buffering, with per-tunnel and aggregate
+  memory limits; do not replace the cap with unbounded queues.
+- Check receive draining, burst handling and loss at the other queue boundaries.
+- Measure throughput and media latency together before choosing production bounds.
+- Validate on the actual relay/backend path before claiming a production speed fix.
+
+## Implemented validation
+
+The final implementation uses bounded rings at both DATAGRAM receive layers,
+not the experimental constant-only change. Large limits are enabled after
+admission, shared accounting is capped at 32 MiB, and packets older than 50 ms
+are discarded when processed. Upstream raw packet safety limits remain intact.
+
+With small datagrams every 20 ms on the same inner connection:
+
+| Transfer | Simulated RTT | Elapsed | MB/s | Small datagrams received | Small datagram p95 RTT |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| 33,592,704 bytes | 120 ms | 2.174 s | 15.45 | 104 / 104 | 181 ms |
+| 134,370,816 bytes | 120 ms | 6.128 s | 21.93 | 294 / 294 | 170 ms |
+
+Both runs reported zero full/budget/expiry drops at the new receive queues.
+The longer run's new raw-packet counter reported 58 drops at QUIC's existing
+256-packet unprocessed-packet cap. Inner loss was 57 packets and outer loss 18;
+these numbers measure different points and must not be subtracted to infer
+another exact loss count. The raw cap remains unchanged for flood protection.
+This local fixture is not evidence that the VPS has those exact raw drops.
+
+Validation passed: queue bounds/order/ownership/expiry/close/global-budget tests
+under the race detector; upstream QUIC and HTTP/3 race tests; Hub native race
+tests; 42 Electron transport tests; relay race tests; TypeScript checking; the
+relay Docker build stage; real Hub-sidecar/relay membership, renewal, replay and
+logout integration using isolated loopback Reticulum and generated test accounts.
+
+The VPS services were not modified by this implementation. Real deployment and
+a live-path measurement are still required to establish the user-visible speed.
+
+## Production Python receiver reproduction
+
+After deploying the relay and restarting Hub 0.10.1, a real upload still delivered
+about 84 MB in 40 s. Encryption consumed only 0.227 s of measured cumulative time;
+file reads 3.334 s. Native sends averaged 458 ms and acknowledgements 593 ms per
+roughly 512 KiB batch. These are overlapping request times, not additive durations.
+No receive-pressure reports appeared from the updated relay. The backend socket
+had `rb212992` and 3869 cumulative kernel drops (not attributable to one upload).
+
+The Step 4 opt-in benchmark now exercises the real Python QUIC server, parser,
+signed session authentication, bootstrap and attachment through the patched relay
+and sidecar. It sends the same 64 x 524886-byte reliable messages with two native
+sends in flight and 120 ms simulated RTT. A benchmark-only binary handler returns
+a small acknowledgement; storage, file encryption and browser work are excluded.
+
+| Receiver variant | Effective Linux socket receive buffer | Seconds | MB/s | Kernel drops / inner loss |
+| --- | ---: | ---: | ---: | ---: |
+| Production asyncio, default socket | 212992 bytes | 12.591 | 2.67 | 772 / 772 |
+| Production asyncio, 4 MiB SO_RCVBUF request | 8388608 bytes | 5.993 | 5.61 | 0 / 0 |
+| Test-only uvloop, same buffer | 8388608 bytes | 4.162 | 8.07 | 0 / 0 |
+| Test-only uvloop + coalesced transmit scheduling | 8388608 bytes | 3.392 | 9.90 | 238 / 238 |
+
+Linux reports doubled SO_RCVBUF accounting. Changing this setting is not itself
+a complete fix: inner RTT rose to 336 ms in the larger-buffer asyncio run and
+235 ms with uvloop, versus roughly 120 ms at the unchanged outer connection.
+The coalescing experiment still overflowed. These are diagnostic experiments,
+not approved production settings or a claim that buffering alone resolves this.
+
+Without profiling, the larger-buffer asyncio receiver consumed 4.98 CPU seconds
+for its 5.99-second transfer. A separate cProfile run (not a throughput result)
+recorded about 28900 UDP callbacks and transmit checks. Major costs were QUIC
+packet receive/send preparation, repeated header parsing, event-loop dispatch,
+and framing. AEAD decryption was about 0.16 s self-time in that profiled run.
+
+This reproduces the low-throughput symptom with the actual Python transport
+without any storage involvement. The socket drop count exactly matching inner
+loss establishes a second loss point. It also establishes substantial per-packet
+processing cost even when socket drops are eliminated. The remaining work is
+the backend receive path, including a bounded buffer and efficient processing /
+scheduling, with mixed-traffic latency validation—not another relay-only tweak.
+
+Reproduce from Hub's root with Go and uv available:
+
+```
+QORTAL_PYTHON_BENCH=1 npx vitest run electron/src/private-channel-step4.test.ts -t 'benchmarks real Python'
+```
+
+Diagnostic-only environment switches: `QORTAL_STEP4_PROFILE=1`,
+`QORTAL_STEP4_UVLOOP=1` (uses `uv run --with uvloop`, no production dependency
+change), and `QORTAL_STEP4_COALESCE=1`. These affect only the isolated fixture.
+
+Upstream implementation:
+https://github.com/quic-go/quic-go/blob/v0.60.0/http3/state_tracking_stream.go