Compatinilité with Bionic / LM Studio Client

[imedox]

👋 Hi,
it will be help full if it compatible with thos 2 Client Bionic & LM Studio for local model use.

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Hi @imedox — good news: no code change is needed, because billion-context already supports "any client you can point at a URL" through one generic mechanism. Concrete setup for each:

LM Studio (as your local model provider) — tested ✅

Any client can route through bili into a local OpenAI-compatible server like LM Studio:

1. Start the LM Studio local server (default address: http://localhost:1234/v1).
2. Start bili: bili start (it listens on port 8787 and keeps running in the background).
3. In your client's model settings, set the base URL to:
   http://127.0.0.1:8787/bili/http://127.0.0.1:1234/v1
   (pattern: bili's address + /bili/ + your LM Studio address).
   Fill the API key field with anything — LM Studio ignores it.
4. Use the model name exactly as LM Studio lists it.

I just ran this exact topology end-to-end (client → bili → local OpenAI-compatible server): streaming responses flow through cleanly and bili injects its compression pipeline into the session automatically.

Three things worth knowing for local models:

• Tell bili your model's real context size. Add this to ~/.config/billion-context/billion-context.json:

  { "models": { "your-model-name": { "context": 32768 } } }

  Without it, bili assumes 200,000 tokens for unknown models — reasonable for big cloud models, but wrong for a 32K local model (it would compress far too late).
• Tool calling helps. bili asks the model itself when to compress (it injects small compress/decompress tools). A local model that supports function calling gives the best results; one that doesn't still gets a safety net that folds context right before the window overflows.
• Sessions are tracked automatically. Clients that don't send a conversation ID (like LM Studio's own chat UI) get auto-created sessions that reattach after restarts — nothing to configure.

Bionic — quick question first

There are several projects named "Bionic". Is yours bionic-gpt — the open-source Rust agentic harness that runs on-premise?

If so, and if it lets you set a custom model endpoint (OpenAI-compatible base URL), the exact same trick above should work: point Bionic's model URL at http://127.0.0.1:8787/bili/<your-model-server>. Could you confirm:

1. Which product exactly (link / version)?
2. Does it have a configurable model endpoint, and does it speak the OpenAI-compatible API?

Once you confirm and give it a try, I'll add both LM Studio and Bionic to the list of verified setups in our docs.

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Quick correction to the config snippet above — I tested both shapes and the top-level models one is silently ignored by the proxy. The context size must be declared under the provider entry for your LM Studio address. Correct version for ~/.config/billion-context/billion-context.json:

{
  "providers": {
    "http://127.0.0.1:1234": {
      "models": {
        "your-model-name": { "context": 32768 }
      }
    }
  }
}

• Replace 1234 with your LM Studio server's actual port, and 32768 with your model's real context window (in tokens).
• The key must be the full address including http:// — that's how bili matches traffic going through /bili/.
• Verified on v0.1.196: with this shape the proxy log shows [window] ... source=configured base=8192 (my declared value wins); with the wrong top-level shape it logs ignoring unknown top-level config key(s): models and falls back to the 200K default.

Everything else in my previous comment stands — the /bili/ prefix recipe itself is unchanged.

[imedox]

thanks, it work now need testing if this repo do what suposed to do :D

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Glad it works! Quick checklist for testing that it actually does its job:

1. Window declaration took effect — in bili's log (~/.local/state/billion-context/bili.log) each model gets a [window] line. You want source=configured base=<your value>. If you still see source=table-or-registry ... fallback=true, the context size isn't picked up — check the key is the exact address incl. http://.
2. Compression actually fires — keep ONE conversation long in your client. As it grows past ~12K tokens the proxy nudges the model to call compress; you'll see the compress tool call and the generated summary in the log. Note: if your local model doesn't support function calling, model-driven compression won't trigger — then only the hard preflight backstop at your declared window protects you.
3. Live stats — curl -s http://localhost:8787/__bili/stats shows active sessions and how many requests/turns went through.

If something looks off, run bili --debug start for verbose logs and paste the relevant lines here.

One question for the records: which client frontend did you point at bili — LM Studio's own chat UI, Bionic, or something else? Once you confirm, I'll add this setup to the docs' verified list.

[imedox]

i will test this and give you full report.

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Sounds good — take your time. Paste whatever you see (log lines, stats output, or the client behavior) and I'll dig into anything that looks off.

[imedox]

Client Frontend: Bionic (along with LM Studio / Antigravity IDE via MCP)

Setup Details:
-Local Inference Server: Bionic model server exposing the OpenAI-compatible API at http://localhost:1235/v1 (with LM Studio running on 1234).
-Loaded Model: qwen3.8-27b-uncensored-hauhaucs-aggressive-mtp
-Routing: Routed requests via http://127.0.0.1:8787/bili/http://127.0.0.1:1235/v1/chat/completions.
-MCP Integration: Configured the billion-context MCP server (dist/mcp.js) connected to bili across Bionic, LM Studio, and Antigravity IDE.

Verification Results:
-Window Declaration: source=configured base=98304 native=98304 configured=98304 fallback=false logged in bili.log.
-ACP Kernel Interception: Verified ACP tag injection (renderTags=text-only), growth tracking (usage=0% (20/98304), growth=0/22500), and --usage logging ([acp-usage] input=4330).
-Live Stats (/__bili/stats): Session pfa-8e3c931674bc81b6 cleanly registered upstream http://127.0.0.1:1235 with contextWindow: 98304.

[imedox]

bili.log
Here are the log lines and stats output from the test run:

1. bili.log Lines (Window Declaration & Turn Processing)
   `[info] [window] model=qwen3.8-27b-uncensored-hauhaucs-aggressive-mtp source=configured native=98304 base=98304 launcher=none configured=98304 peek=none fallback=false upstream=host:fce1c3d2 route=host:fce1c3d2

[info] [prefix-affinity] new session pfa-8e3c931674bc81b6 at depth 1 (tail=09d3e960; ua="-" plugin=- first="Say hello in 5 words")

[info] [sess=pfa-8e3c931674bc81b6] [pfa-8e3c931674bc81b6] processTurn: 1 msgs, renderTags=text-only, 1 text tagged, 0 tool tagged (should be 0 with text-only)

[info] [sess=pfa-8e3c931674bc81b6] [pfa-8e3c931674bc81b6] nudge idle: usage=0% (20/98304), growth=0/22500 (ref=20, interval=50000), pendingT1=0/50000 model=qwen3.8-27b-uncensored-hauhaucs-aggressive-mtp, reason="max compressible 0 < threshold 50000; growth 0 < floor 22500"

[info] [sess=pfa-8e3c931674bc81b6] [pfa-8e3c931674bc81b6] request: 1 msgs, inbound=120B, outbound=17.2KiB, local=91ms

[info] [sess=pfa-8e3c931674bc81b6] forward POST -> http://:1235/v1/chat/completions

[info] [sess=pfa-8e3c931674bc81b6] [pfa-8e3c931674bc81b6] [acp-usage] input=4330 (no cache report)
2. Live Stats Output (curl http://localhost:8787/__bili/stats){
"sessions": [
{
"id": "pfa-8e3c931674bc81b6",
"protocol": "openai",
"upstream": "http://127.0.0.1:1235",
"label": "prefix-affinity 09d3e960",
"title": "Say hello in 5 words",
"requests": 1,
"contextTokens": 4330,
"contextTokensSource": "usage",
"inputTokens": 4330,
"cachedTokens": 0,
"outputTokens": 205,
"lastModel": "qwen3.8-27b-uncensored-hauhaucs-aggressive-mtp",
"contextWindow": 98304,
"lastSeen": "2026-10-14T22:19:35.332Z",
"restored": false
}
],
"blindTunnels": {
"total": 0,
"hosts": {}
},
"unrecognizedPaths": {
"total": 0,
"paths": {}
},
"conflicts": {
"sessions": 0,
"events": 0,
"active": 0,
"historical": 0
}
}
`

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Great report — I checked every line against the code and your setup is behaving exactly as designed. Line by line:

• [window] ... source=configured base=98304 fallback=false route=<host> — your per-provider context declaration won over everything else (registry + built-in table), and route= shows the traffic matched your provider key. This is the happy path.
• [prefix-affinity] new session pfa-... at depth 1 — Bionic doesn't send a stable conversation id, so bili auto-tracked your chat as an anonymous session (titled from your first message). It will re-attach to the same session across restarts as long as the history prefix matches.
• processTurn: ... renderTags=text-only, 1 text tagged, 0 tool tagged — ACP tags were injected into the outgoing request, text-only mode working as intended.
• nudge idle: usage=0% (20/98304), growth=0/22500 (ref=20, interval=50000) — this is the kernel's default cadence: a flat 50K growth step (interval=50000) with a 45% display floor (max(20000, 0.45×50000) = 22500). Nothing misconfigured.
• inbound=120B, outbound=17.2KiB — the size jump is the injected compression system prompt + tool definitions going upstream; expected overhead for proxy mode.
• [acp-usage] input=4330 (no cache report) — usage tracked fine; "no cache report" is normal for local servers (they usually don't report cached tokens).
• Stats: zero blindTunnels, zero unrecognizedPaths, zero conflicts → 100% of your traffic went through the compression path cleanly.

What you haven't seen yet: actual compression firing. With a 98K window and only ~4K tokens used, the nudge correctly stays idle — keep one conversation going until it has grown well past ~50K new tokens since the reference point; then you'll see the model get nudged to call compress and a summary appear in the log. That's the moment bili earns its keep on a small local model.

Since you confirmed it works end-to-end, I've recorded your setup in the docs' verified list (Bionic row + a note that local OpenAI-compatible servers like LM Studio work as loopback upstreams): #3216 — docs-only change, no behavior touched. Thanks for the thorough testing!

[imedox]

Thank you so much for the detailed breakdown! It's fantastic to see every piece of the pipeline working exactly as intended. Seeing our setup verified and added to the official documentation in PR #3216 means a lot.

I'm definitely going to push this conversation past the 50K mark to watch compress do its magic in real time!

I also really want to commend you on this project. Running local LLMs (like Qwen 27B) at home is always a battle against VRAM limits and context degradation. What you’ve built with billion-context is a huge game changer for the local AI community — it makes running month-long, continuous sessions on consumer hardware actually viable.

If you're ever looking for future ideas or enhancements from a local user's perspective, a few things that could make it even sweeter:

• Configurable compression thresholds for smaller local setups: An easy flag/config to adjust the compression cadence (e.g. trigger nudges earlier around 12K–20K for users running smaller 16K/32K models or tight VRAM constraints).
• Built-in background service manager: A simple command like bili service install (for systemd / Windows Scheduled Task or Service) so regular desktop users don't have to keep terminal windows open.
• Simple visual indicator or tray icon: A lightweight desktop widget or status indicator showing active session compression level, cache stats, and saved tokens in real-time.
  Thank you again for your hard work, responsive support, and dedication to this project. Keep up the amazing work! 🚀

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Thank you — that means a lot! And great timing on your suggestions: I checked all three against the codebase and what's already shipping, and two of them are closer to reality than they look.

1. Configurable compression cadence — already available today. There's a documented knob for exactly this: compress.nudgeGrowthTokens in ~/.config/billion-context/billion-context.json sets the growth step (default 50000, flat by design at every window size). For a tight-VRAM 16K/32K setup, something like:

{
  "providers": { "http://127.0.0.1:1235": { "models": { "your-model": { "context": 32768 } } } },
  "compress": { "nudgeGrowthTokens": 15000 }
}

Two honest notes: there's a built-in floor on the growth gate (max(20000, 0.45 × step)), so steps below ~20K floor out around 20K; and on small windows the window-percentage pressure bands (45% / 75% / 95% of your declared context) plus the preflight hard backstop are the primary protectors anyway — declare the real window and those scale automatically. Per-tier fine-tuning also exists if you ever want it (compress.tierNudgeTokens).

2. Background service manager — literally in the pipeline right now. An external contribution (PR #3176) adds exactly a bili service install|status|restart|enable|disable|remove command family (Windows Task Scheduler + Linux systemd user unit); it's tracked in issue #3178, the direction is approved, CI is green, and it's awaiting final verification + merge. I noted your use case in that thread as an independent demand signal.

3. Visual indicator — most of it already exists. bili ships a built-in web dashboard: open http://localhost:8787/__bili/ in your browser — the overview page shows tokens used/saved, per-session detail (context window, usage, compression state), plus a config editor and live log viewer. For scripting, curl -s http://localhost:8787/__bili/stats gives the same numbers as JSON. The one piece missing from your wishlist is an always-on tray icon / menu-bar widget — I've filed that narrow delta as a tracking issue: #3217 (marked no-dispatch; whether a first-party native widget makes sense is the maintainer's call).

So: nothing to wait for on #1, #2 lands with the next merge, and #3's data is one URL away. Now go push that conversation past 50K and watch compress do its magic! 🚀

[imedox]

I wanted to share a project I've been working on called Local-Model-Booster. It is optimized to transform local 27B-class models into highly disciplined, evidence-driven autonomous coding agents using Bionic and LM Studio.
Key Features:
• Native billion-context Integration: Coordinates seamlessly between the 128K native context window and the local compression proxy daemon to handle massive codebases without token bloat.
• 1-Click Automated Installer for quick Windows setup and configuration.
• 6 Production-Tuned Profiles ranging from deep high-precision coding to maximum inference speed.
• 35 Windows CLI Helper Tools for repository mapping, symbol search, and VRAM monitoring.
If you are experimenting with local AI agents, I think you might find this useful! You can explore the code and documentation here:
https://github.com/imedox/Local-Model-Booster
I would love to hear your thoughts or feedback.

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Great to see a real-world setup built on this — thanks for sharing! I read through the README; here are a few thoughts from the billion-context side, focused on where the two projects meet:

1. Declare the sustainable window, not the theoretical one. Your profile table already does this implicitly (98,304 instead of 131,072 for the coding profile — good instinct). Make it explicit in bili's config: providers["http://127.0.0.1:<port>"].models["<model-name>"].context = the biggest context you can actually hold in VRAM with weights + q8_0 KV cache together. Everything in bili scales off that number — the 45% / 75% / 95% pressure bands and the hard preflight backstop — so setting it to your true sustainable ceiling means compression starts protecting you before the inference server starts straining on KV memory, not after. If one model serves several profiles with different context sizes, declare the smallest one you actually use.

2. Your hand-rolled BiliProxyService scheduled task will soon have a first-class replacement. A PR currently in review ([PR] Added a one-click install feature for my plugin on the Codex side | [PR] 添加了codex端一键安装我插件功能 #3176, tracked in Codex One-Click Install Feature (PR #3176): "Persistent Auto-Start Proxy" Direction Pending Decision + PR Cleanup | codex 一键安装功能(PR #3176):「常驻自启代理」方向待拍板 + PR 清理 #3178) adds bili service install|status|restart|enable|disable|remove — Windows Task Scheduler + Linux systemd, with proper uninstall. Once it merges, installer\install.bat could call bili service install instead of registering its own task: fewer moving parts, and users get a supported way to manage/remove the service.

3. Your wiring pattern is exactly the intended one — base URL routed through the /bili/ prefix for compression, plus the bili MCP server for tool access — and your log report from earlier showed it running clean (window configured, tags injected, zero blind tunnels / unrecognized paths).

One boundary note: I'm the agent for the billion-context repo, so I can't audit or edit your project directly (any cross-repo work needs maintainer sign-off) — but if you hit anything odd on the bili side (compression behaving strangely, a session failing to re-attach, window numbers looking wrong), paste the log lines here and I'll dig in. Good luck pushing past 50K! 🚀

[imedox]

Thanks a lot for the review and the thoughtful insights! It's fantastic to hear directly from the billion-context side, and we're super excited about how well the integration works.

Here is what we've aligned based on your feedback:

1. Sustainable Window in bili config:
   That makes complete sense regarding the 45% / 75% / 95% pressure bands. We are updating the default billion-context.json template in the repo/installer from 131,072 to 98,304. That way, compression kicks in proactively around ~44K tokens, well before the RTX 3090's 24GB VRAM begins sweating under the q8_0 KV cache + 27B weights load.

2. Native bili service (PR [PR] Added a one-click install feature for my plugin on the Codex side | [PR] 添加了codex端一键安装我插件功能 #3176 / Codex One-Click Install Feature (PR #3176): "Persistent Auto-Start Proxy" Direction Pending Decision + PR Cleanup | codex 一键安装功能(PR #3176):「常驻自启代理」方向待拍板 + PR 清理 #3178):
   This is great news! As soon as PR [PR] Added a one-click install feature for my plugin on the Codex side | [PR] 添加了codex端一键安装我插件功能 #3176 merges and rolls into an npm release, we will happily update our installer\install.ps1 to replace the custom Scheduled Task with bili service install. Having first-class service lifecycle commands (status, restart, remove) will be much cleaner for end users.

3. Architecture confirmation:
   Glad to confirm the /bili/ prefix proxy routing + MCP tool wiring runs clean. Our local testing shows zero dropped sessions and seamless multi-turn compression.

Thank you again for building such a great tool — it's a game-changer for consumer hardware! Looking forward to the upcoming release! 🚀

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Glad we're aligned! Two small notes for the record:

• Your ~44K number is spot-on: 45% × 98,304 ≈ 44,237 is where the soft compression pressure starts, while the hard preflight backstop stays at the full declared window (~98K). So you get a gradual ramp instead of a cliff — which is exactly what you want on VRAM-constrained hardware.
• On timing: [PR] Added a one-click install feature for my plugin on the Codex side | [PR] 添加了codex端一键安装我插件功能 #3176 ships in whatever release the maintainer cuts after merging (releases need owner approval), so watch the GitHub Releases page — once the new version lands, swapping your installer over to bili service install should be a one-line change.

This thread turned into a nice end-to-end validation of the whole local-model story (Bionic client + LM Studio upstream + MCP + service management on the way). Thanks again for the thorough testing along the way! If anything else comes up, this issue or a fresh one both work. 🚀

[ranxianglei]

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Your numbers check out — 45% × 98,304 ≈ 44,237, which is precisely where the first soft pressure band arms, so "proactive around ~44K" is an accurate description. Dropping the template default from 131,072 → 98,304 is the right call for a 24GB card: it keeps the hard preflight wall at the ceiling you can actually hold in VRAM while letting the ramp begin comfortably below it.

And yes — once #3176 merges into a release, bili service install replaces your custom Scheduled Task cleanly, with status/restart/remove giving end users proper lifecycle control. That swap will be a small diff on your install.ps1.

Glad the integration is holding up under real multi-turn load. Thanks again for the detailed testing and for building Local-Model-Booster on top of it — that's exactly the kind of downstream use this project was meant to enable. 🚀