No input-caching for abliteration.ai

[mattswest]

Description

While trying out this software using abliteration.ai as the provider, I noticed that the free $1 in credits they give you went VERY fast. Like 3-4 calls.

I looked at the usage on their website and noticed none of the input was being cached.

Using a different AI agent I was able to get input caching working in cyberstrike as verified on the abliteration.ai website. This is what the AI agent had to say:

### Root Cause Analysis

 Investigation of the cyberstrike CLI (v1.1.16) binary, session logs, and local database (~/.local/share/cyberstrike/cyberstrike.db)
 alongside abliteration.ai's Prompt Caching Documentation (https://docs.abliteration.ai/capabilities/prompt-caching) revealed four
 interacting issues preventing cached input utilization:

 1. Few-Shot Message Oscillation Breaking the Prompt Prefix:
   - In LLM.stream(), getFewShotMessages(input) injects 4 hardcoded few-shot tool-call messages ("Check if port 80 is open on 10.0.0.1",
     bash tool-call, tool-result, assistant confirmation) whenever hasToolCalls(messages) is false.
   - While major providers are excluded via:
     ```javascript
if (pid.startsWith("anthropic") || pid.startsWith("openai") || pid.startsWith("google") || pid.startsWith("amazon") || input.small) return
[];
     ```
     abliteration-ai was omitted from this check.
   - On Turn 1, CyberStrike injected the 4 few-shot messages before the initial user prompt. Once the model issued a tool call, hasToolCalls
     evaluated to true on Turn 2 and stripped the 4 messages entirely.
   - Because prompt prefix caching requires an identical token prefix from the start of the conversation, removing the few-shot messages
     immediately after the system prompt invalidated the prefix between Turn 1 and Turn 2.
 2. Missing Session Affinity / Sticky Routing (x-abliteration-session-id):
   - abliteration.ai runs behind Cloudflare edge load balancing across a distributed GPU cluster. Without session affinity, sequential
     requests in a conversation are routed to arbitrary GPU workers whose local KV caches do not contain previous turns.
   - abliteration.ai requires a conversation routing hint:
     - Header: x-abliteration-session-id: <session_id>
     - Body: prompt_cache_key: <session_id>
   - CyberStrike only attached session headers (x-cyberstrike-session) if model.providerID.startsWith("cyberstrike"). For abliteration-ai,
     requests had no session routing hint.
 3. Option Casing Mismatch (promptCacheKey vs prompt_cache_key):
   - In ProviderTransform.options(), setting cache keys assigns result["promptCacheKey"] = input.sessionID; (camelCase).
   - @ai-sdk/openai remaps promptCacheKey → prompt_cache_key, but @ai-sdk/openai-compatible (which abliteration-ai uses) passes unknown
     fields directly into the JSON body as "promptCacheKey". abliteration.ai expects snake_case "prompt_cache_key".
 4. Cache Control Exclusively Gated to Anthropic:
   - In ProviderTransform.message(), applyCaching() was gated behind:
     ```javascript
if ((model.providerID === "anthropic" || model.api.id.includes("anthropic") || model.api.id.includes("claude") || model.api.npm ===
"@ai-sdk/anthropic") && model.api.npm !== "@ai-sdk/gateway")
     ```
   - All OpenAI-compatible providers, including abliteration-ai, were skipped.

 ────────────────────────────────────────────────────────────────────────────────

 ### Changes Applied

 #### 1. Binary Patch (getFewShotMessages)

 In all installed CyberStrike platform binaries (cyberstrike-linux-x64, cyberstrike-linux-x64-baseline, cyberstrike-linux-x64-musl, and
 cyberstrike-linux-x64-baseline-musl), patched the provider exclusion check in getFewShotMessages to recognize ablit:

 ```javascript
// Before:
pid.startsWith("anthropic") || pid.startsWith("openai") || pid.startsWith("google") || pid.startsWith("amazon")

// After:
["anthropic","openai","google","amazon","ablit"].some(x=>pid.startsWith(x))
 ```

 This prevents the 4 few-shot messages from being injected on Turn 1, keeping the prompt prefix uniform across all conversation turns.

 #### 2. Native CyberStrike Plugin (abliteration-cache.js)

 Created /home/matt/.config/cyberstrike/plugins/abliteration-cache.js:

 ```javascript
export default async function(input) {
  return {
    "chat.headers": async (ctx, output) => {
      const pid = (ctx.model?.providerID || "").toLowerCase();
      if (pid === "abliteration-ai" || pid.includes("abliteration")) {
        output.headers["x-abliteration-session-id"] = ctx.sessionID;
      }
    },
    "chat.params": async (ctx, output) => {
      const pid = (ctx.model?.providerID || "").toLowerCase();
      if (pid === "abliteration-ai" || pid.includes("abliteration")) {
        output.options = output.options || {};
        output.options.prompt_cache_key = ctx.sessionID;
      }
    }
  };
}
 ```

 Registered the plugin globally in /home/matt/.config/cyberstrike/cyberstrike.json:

 ```json
{
  "$schema": "https://cyberstrike.io/config.json",
  "plugin": [
    "file:///home/matt/.config/cyberstrike/plugins/abliteration-cache.js"
  ]
}

I hope at least some of that is useful. I felt I should open the issue since this has a high cost impact.

Steps to reproduce

Use cyberstrike with abliteration.ai (or any OpenAI compatible provider?)

Agent

cyberstrike (default)

Interface

CLI / TUI

LLM Provider

Other

CyberStrike version

No response

Operating System

No response

Terminal

No response

Logs / Screenshots