Cost optimization and intelligent model routing

Spend doesn't die on the frontier. It bleeds out on the routine — the grammar checks, the formatting passes, the lookups you never think about, each one routed through your most expensive model out of pure habit.

@Mnilax put a number on the habit. Bumping a normal task from the "high" reasoning tier to "xhigh" (Opus 4.8) costs 4× more for marginal quality — quality-per-dollar collapses from 95 to 24. 🔒 authorize for free to reveal 24 words Everything else was paying quadruple for a rounding error.

The second leak is quieter, and @AiWithIqra named it: context bloat. Claude counts tokens, not messages, so a conversation past 20 messages forces the model to re-read the entire thread on every single turn. Reset the chat, carry forward a tight summary, and you can save up to 50× tokens per turn. Her exact handoff prompt:

"Summarize the key points of this conversation in 5 concise bullet points, including essential context, decisions made, and next steps, so I can continue in a new chat."

And the third trap is assuming "cheap" means "free." @0xLogicrw's macro example — Lindy's founder Flo Crivello migrating all client-side traffic off Anthropic — saved millions, but only after building internal infrastructure to absorb "up to 100x higher" workload than expected. Open-source models cost you engineering time instead of API dollars. The bill moves; it doesn't vanish.

Route by task complexity, not by reflex. The model that reasons best is almost never the model you want doing the typing. @hadesboun101 gives the cleanest split: 80% of work to cheap fast models, 15% to mid-tier, 5% to the frontier. His verdict on the alternative is blunt — "Using premium models for simple rewrites or summaries is financially insane."

The split isn't arbitrary. Most of a working day is classification, formatting, summarization, lookup — tasks that need speed and consistency, not genius. The 15% mid-tier catches the work that needs nuance but not brilliance: client emails, structured outlines, multi-step data transforms. The frontier 5% is reserved for what genuinely requires long-horizon reasoning — cross-file migrations, architecture calls, adversarial verification.

@Sprytixl built the concrete version. His routing layer sends routine tasks to Kimi K2.6 at $0.09 while complex work stays on Claude Opus at $1.41 — and the detail that makes it work is that context never drops between switches. This is the difference between a routing system and a frustrating one: if the model switch loses the thread, the user re-establishes context by hand, burning the exact tokens they were trying to save. The mechanism is a shared state file — markdown or a knowledge graph — that both models read from and write to, so when a task escalates, the expensive model picks up precisely where the cheap one left off.

The same logic runs across providers, not just price tiers. @Voxyz_ai dispatches Fable 5 for planning and front-end work where visual output justifies the price, then writes a clear spec and hands backend implementation to Codex / GPT-5.5, where his quota sits unused. @ai_appreciator gives the cautionary inverse: Fable 5 burned 1.4M tokens and 5 subagents over 90 minutes to clean up 21 Resharper warnings — a job Codex GPT-5.5 xhigh would have done in 10 minutes, under 300K tokens, no subagents. Right model, wrong lane, 5× the cost.

A practical starting taxonomy, drawn from across the corpus (cost bands are indicative, not quotes — only the $0.09/$1.41 pair above is a measured per-call price):

Where practitioners split is on how dynamic the routing should be. @Sprytixl and @Mnilax favor explicit, human-configured rules. @hadesboun101 goes further with model ensembles via OpenRouter's Fusion API — fire one query at several models in parallel, use a judge to map where they converge and diverge, then synthesize a final answer. It costs more per query and adds latency and complexity, but it buys resilience (if a model disappears or gets banned, you're not stranded) and can match frontier quality at half the cost. His re-evaluation rule is the keeper: judge on "runtime vs. IQ" — actual shipped work, not benchmark scores.

Local inference is a cost layer, not a replacement strategy. It pays off for high-volume, low-complexity work — and quietly loses money for occasional use or anything that needs frontier reasoning.

The aspirational case is @0xLogicrw's Lindy migration: all client traffic moved from Anthropic to open-source DeepSeek v4, Atlas Cloud rented for niche compute, Claude Opus kept as an automatic fallback for complex-task failures. Claimed result: millions saved, with performance improving on many core business scenarios. Creator-reported, but the architecture — cheap-by-default, frontier-on-failure — is the part to copy.

The hardware paths people actually run:

The economics only hold under conditions the loudest claims skip over. @0xKnzo says Chinese developers are making "$400,000 running local AI" and Japanese developers killed "$249/day in AWS bills" — both [claimed], no verified revenue behind them. The honest caveats:

• Upfront cost is real. $2,400–$5,000 is not trivial for an individual.
• Thermal management is critical. @0xKnzo runs external cooling fans "the size of a toaster" to stop thermal throttling on 24/7 workloads — without them, performance silently degrades.
• Capability drops. Local DeepSeek 14B or Qwen3 235B are competent, not Opus 4.8 or Fable 5. @ai_appreciator's read after testing: Opus 4.8 is "competent and I could probably get the game done with it, but the difference is noticeable" against GPT 5.5 xhigh — and local models sit below that.
• Security is extra work. @0xKnzo wraps every agent in Docker sandboxing to stop prompt injection from leaking host credentials — overhead you don't carry on a managed API.

@marfinxx found the non-obvious failure mode. After switching from OpenClaw to Hermes Agent, his flat-file Obsidian markdown memory "floods GPUs with uncompressed markdown strings," bottlenecking the hardware. The fix — 🔒 authorize for free to reveal 8 words — is the local-inference lesson in one line: the model isn't your bottleneck, your memory format is. (For the performance ceiling, his reference rig is an 8-GPU RTX 4090 server with NVFP4 quantization.)

The decision rule: go local for the 80% tier when you have the volume and the skill to maintain it. Keep the frontier in the cloud. @gippp69, who runs local daily, draws the line cleanly:

"Does not replace frontier models for the 'hardest tasks'; keep one cloud model as a backup."

Most teams optimize which model they call and ignore how it remembers. The same model, on the same task, can vary 10× in cost depending on context architecture.

The highest-ROI move in this entire note costs nothing and takes minutes: @Mnilax's "tokenmaxxing" audit of ~/.claude/settings.json (≈125 keys, only ~40 documented). The 18 settings that actually touch billing come down to three you can act on today:

1. Trim enabledPlugins. Idle plugins load their hooks and SKILL.md at startup. He had 14 enabled, used 4.
2. Set mcpServers.enabled to false instead of deleting unused servers. Deleted entries still get parsed; 9 unused servers cost ~30K tokens of schema every session.
3. Move the cache_control breakpoint to the static/dynamic boundary so the model stops re-processing unchanged system prompts and tool definitions every turn. 🔒 authorize for free to reveal 11 words

One caveat worth copying with the technique: the "deny-rule bug" — where config says block but the binary reads anyway. Don't assume a setting took effect; verify the token drop in your next session.

That cache_control move is the manual front-end of Claude's prompt caching — cache the system prompt and tool definitions once, pay discounted rates on every subsequent turn — and it's where @elsontec's line "agent projects cheaper to run than to scope" stops being a slogan. @bonsaixbt attacks the same tax from the memory side: a Graphify + Obsidian knowledge graph that maps every conversation, file, and research snippet into a connected structure, so the agent retrieves only the relevant subgraph instead of reloading whole histories. Reported token drop: ~70×.

@AiCamila_ packages the whole discipline into five tactics — route simple tasks to cheaper models, cache repeated queries, compress context, implement early stopping, set per-user budgets — under one metric that reorders everything: cost per successful task, not cost per token. Her starting advice is deliberately small: "Start with model routing + caching."

Before you build a router or buy a GPU, kill what you already pay for and don't use. It's the fastest dollar saved.

@eyishazyer's stack is the wake-up call — the "serious AI user" tax laid out: $200/mo Claude Code Max + $200/mo ChatGPT Pro + $20/mo Cursor + $20/mo Gemini = $440/month, $5,280/year. The point isn't that any one is overpriced; it's that nobody audits the sum.

@alchemyofmax ran the enterprise version: an audit of $47,000/year in AI spend that found 14 separate subscriptions with zero shared context — a team "talking to AI in 14 separate windows with zero shared knowledge," copy-pasting outputs into Google Docs and calling it a deliverable. His fix was a unified "2nd Brain" with five parts: a daily AI briefing, a living business wiki, a shared context layer, a built-in task manager, and a custom metrics dashboard. Claimed payoff: $47K consolidated, zero Monday status meetings, three senior hires onboarded with no knowledge-transfer lag. (Creator-reported — but the diagnosis, overlap with no shared layer, is the transferable part.)

Two replacement maps for what the audit surfaces:

The synthesis: audit by function, not by tool. Map what each subscription actually does, find the overlaps, and cut anything that doesn't connect to a shared intelligence layer.

A fallback is both a safety net and a cost lever: it's the thing that lets you run lean and escalate only when the cheap model actually breaks. Without one you're forced to over-provision (Opus for everything) or under-provision (DeepSeek for everything, eating the failures).

@0xLogicrw's pattern is the reference: DeepSeek v4 primary, Claude Opus as automatic fallback on complex-task failure — no human in the loop, the system detects the miss and re-routes. @archiexzzz generalizes it to config: give every agentic workflow a primary model and a list of fallback providers — fallback_model_providers: [anthropic, openai] — and route to the next option when the primary is unavailable or restricted.

Even Anthropic runs this internally. @ArtificialAnlys documents Fable 5 falling back to Opus 4.8 on 2% of GDPval-AA tasks, with fallback occurring in fewer than 5% of sessions on average — a real but bounded cost bump, and the closest thing here to a verified (not creator-reported) number.

@AiCamila_'s self-healing agents extend the pattern into a loop: detect (error logs, failed tool calls) → diagnose (compare to baseline) → remediate (retry, fallback model, reset state) → escalate (notify a human only when auto-fix fails). Tie it back to cost per successful task — otherwise a "fallback" is just a more expensive way to keep burning tokens on a task that was never going to resolve.

Which is exactly what @mardehaym's Cursor invoice was. His strategic frame is Token Capital = Human Capital × Scaffolding × Feedback Loops — and because it's a product, "doubling your model spend when scaffolding is zero still gives you zero." The agent that looped to 1.3 billion tokens had model power and no scaffolding. Daily spend limits are not optional.

Ordered from zero-cost wins to capital investment. Each phase ships something and tells you how to check it.

Phase 1 — Audit and trim (Day 1, zero cost).

1. List every AI subscription, API key, and usage bill. Categorize by function (reasoning/coding, image, video, voice, automation, search) in @eyishazyer's format: tool, monthly cost, annual cost, what it does that nothing else does. Flag every overlap.
2. Run the tokenmaxxing audit. Open ~/.claude/settings.json. Disable unused plugins, set unused MCP servers to enabled: false, move the cache_control breakpoint — then verify the token drop next session.
3. Reset any chat past 20 messages with @AiWithIqra's 5-bullet summary prompt, and start fresh.
   • Verify: next monthly bill vs last. Target a 30–50% cut from Phase 1 alone.

Phase 2 — Implement tiered routing (Week 1–3, low cost).

1. Sort every recurring task into the 80/15/5 buckets. Identify your "hot path" — the 20% of queries eating 80% of tokens (usually simple generation, code completion, repetitive analysis); those are your cheap-tier candidates.
2. Build or adopt a router — @Sprytixl's custom layer with context preservation, @hadesboun101's OpenRouter Fusion API, or LiteLLM / LangChain routing chains (per @AiCamila_). Default to cheap; escalate to mid on failure or a complexity flag; escalate to frontier only on mid-tier failure or pre-defined "hard" categories. Switch models in Claude Code via the /model command.
3. Wire the fallback chain (@0xLogicrw's DeepSeek→Opus; @archiexzzz's fallback_model_providers) and preserve context across switches via the shared state file.
   • Verify, after one week: total cost vs prior week; task success rate ≥95%; fallback trigger rate under 10%.

Phase 3 — Local inference for the cheap tier (Month 1, capital cost, if warranted).

1. Calculate monthly spend on cheap-tier tasks. 🔒 authorize for free to reveal 10 words
2. Pick hardware (Sprytixl's 128GB PC, 0xKnzo's 4 Mac Minis, adiix's Ryzen AI Max+ 395). Install Ollama, pull models matching the cheap tier (Qwen, DeepSeek, Llama), point your tools at localhost via the env var.
3. Add cooling. Sandbox agents in Docker.
   • Verify, after 30 days: cloud bill for cheap-tier tasks approaching $0; electricity under $15/month; local-vs-cloud success within 5%.

Phase 4 — Scale and keep tuning (Month 2–3, continuous).

• Scale local from one machine only after measuring real utilization (0xKnzo's 4-node cluster for cost; marfinxx's 8-GPU RTX 4090 / NVFP4 rig for performance). Always keep the cloud fallback — local-first is not local-only.
• Monitor cost per successful task. Evaluate models with cost-constrained, not step-constrained, protocols (@0xLogicrw): a cheaper model that runs 10× more iterations inside a budget can beat an expensive one that exhausts it in 3 steps.
• Compress context, cache repeated queries, set per-user budgets and alerts (@AiCamila_). Re-evaluate model selection quarterly (@Mnilax) — last quarter's "xhigh only" may be this quarter's "high."

You have a system, not a slogan, when you can prove these numbers moved.

The final check is one question: can you swap your primary model without changing your workflow? If yes, your architecture is decoupled and you've stopped paying the loyalty tax. If no, you're still locked in — and the next price hike or model ban is your problem to absorb.

The sharpest disagreement is whether local can fully replace the cloud. @0xKnzo says you can "replace Anthropic forever" with a stack of Mac Minis. @0xLogicrw demonstrates the opposite — even after migrating everything to open-source, you keep Opus as a fallback for the hard tasks — and @ai_appreciator confirms local models are "competent" but noticeably worse on complex coding. The honest synthesis: local inference can replace your cheap tier entirely, but it cannot yet replace your frontier tier. Anyone claiming otherwise is selling hardware, not sharing data.

The second fight is over migration effort. @0xLogicrw warns of "100x higher" infrastructure load than expected; @DAIEvolutionHub frames open-source swaps as simple repo deployments. Reality sits between them: the software is free, the engineering time to configure, maintain, and debug it is not. Budget 2–4 weeks for a real migration.

And the insight under all of it, the one that reframes the whole exercise: cost optimization and quality optimization are the same problem. @Mnilax's tokenmaxxing doesn't just save money — cutting schema bloat makes Claude faster. @0xLogicrw's cost-constrained evaluation doesn't just find cheaper models — it finds models that iterate more within a budget, which produce better results. @Sprytixl's routing layer doesn't just cut spend — it guarantees every task gets the right level of attention.

The teams that win on AI cost aren't the ones spending less. They're the ones spending deliberately. Every token has a job. Every model has a lane. Every fallback has a trigger. Build that once and the savings compound — while the team that pinned Opus to everything is still watching an invoice climb in real time, wondering where the money went.