FORGE
ANYTHING.
A terminal-native LLM client built on TurboQuant — the Google Research KV-cache compression algorithm behind the paper’s benchmark tables below. No cloud. No API keys. No telemetry.
curl -fsSL https://anvil-llm.github.io/anvil/install.sh | sh
Don't take our word for it. Type.
A live slice of the anvil REPL, right here in your browser. Try help, /stats, /ctx, anvil list, or just say something.
try: help · /stats · /model · /ctx · /temp 0.9 · anvil list · anvil pull · or any message
Every API call is a lease on someone else's forge. Your prompts cross the wire, your latency rides their traffic, your context window is a line item on their pricing page.
anvil runs the whole stack on hardware you own — weights loaded straight into your GPU, KV cache compressed with TurboQuant so long contexts actually fit, tokens streaming to your terminal at native speed. It works on a plane. It works offline. It works on a $200 used GPU and on a 128GB Mac.
The model never phones home, because there's no home to phone. Just a single binary, a GGUF file, and the anvil.
Numbers don't lie.
Results straight from the TurboQuant paper (Google Research, arXiv:2504.19874). TurboQuant at 3.5 bits scores identically to the full-precision cache — same LongBench average, same needle-in-a-haystack recall.
Why TurboQuant wins.
TurboQuant is a provably near-optimal vector quantizer from Google Research — within a factor of ≈2.7 of the information-theoretic Shannon lower bound, across every bit-width. Most KV quantizers just round numbers. TurboQuant proves it can't do better.
Each KV vector is hit with a random rotation (fast Walsh–Hadamard structure). Every coordinate now follows a concentrated Beta / Gaussian distribution — outliers spread out, no coordinate dominates.
Because rotated coordinates are nearly independent, optimal Lloyd-Max scalar codebooks are precomputed per bit-width. MSE-optimal, data-oblivious, online — no calibration, no training.
MSE quantizers bias attention. TurboQuant adds a 1-bit Quantized Johnson-Lindenstrauss (QJL) pass on the residual, making inner products unbiased — the reason accuracy survives at 3.5 bits.
No calibration, no training, no per-model tuning — the codebooks are precomputed and universal.
Backend kernels for CUDA, Metal (TurboFlash), Vulkan and HIP — quantization is free at decode time.
Google Research paper with proofs of near-optimality — the benchmark tables below are straight from it.
Synthetic benches only tell you so much. Run your actual prompt through anvil and watch /stats for tokens-per-second and prompt-processing time. Then iterate on KV preset and context size until the numbers stop improving.
$ anvil run qwen3.6-27b.gguf --ctx 131072 --ngl 99 --flash-attn > /stats tokens/sec: 127.4 · prompt tokens: 842 · total tokens: 4 891 · KV: turbo4/turbo3
One swing, many strikes.
anvil supports MTP and NextN speculative decoding — a draft head proposes several tokens at once, the main model verifies them in a single pass. More tokens per step, same quality.
amber = autoregressive token · dashed = drafted · green = verified in one pass
How it works
The model's extra prediction heads guess the next few tokens cheaply. anvil feeds all of them through the main model at once — if they're right, you get them all in a single decode step.
Quality is bit-identical to normal decoding. Rejected drafts are simply discarded and re-sampled.
It finds your hardware. You don't.
anvil probes your system on launch — CPU, RAM, every GPU and its VRAM — and picks a sane layer count automatically. Override with --ngl if you know better.
From keystroke to token.
Every prompt travels the same road through the forge. Here's what your hardware is actually doing, ~40 times a second.
Your first run forges the config.
No YAML to hand-edit. The first time you launch, anvil opens a full TUI setup wizard — built with FTXUI — reads your hardware, and walks you through context, KV compression, flash attention, and temperature. Every model gets its own persisted profile.
░███ ░██░██ ░██░██ ░██ ░██ ░██ ░████████ ░██ ░██ ░██░██ ░█████████ ░██ ░██ ░██ ░██ ░██░██ ░██ ░██ ░██ ░██ ░██ ░██ ░██░██ ░██ ░██ ░██ ░██ ░██░██ ░██░██ ░██ ░██ ░██ ░██ ░███ ░██░██
- Hardware-aware defaultsIt reads your GPU's VRAM and suggests a layer count and context size that will actually fit.
- KV presets with real numbersEvery compression option shows its factor and quality cost — no guessing.
- Custom modePick K and V cache types independently, or type any context size by hand.
- Per-model profilesEach model keeps its own settings in ~/.anvil/models.json, seeded from the GGUF's trained context. Change something in-chat with /temp or /ctx and it persists.
Every model, one ledger.
anvil keeps a model registry with friendly names and a persistent per-model profile. Pull models from Ollama's registry or Hugging Face, import local GGUFs, run them by nickname.
- Friendly namesRun anvil run qwen3.6-27b — no paths, no hashes.
- Pull from anywhereOllama registry and Hugging Face GGUF downloads, checksum-verified.
- Import local filesPoint at any .gguf and anvil registers it with hardware probes.
- Per-model profiles in models.jsonn_ctx, temp, flash_attn, type_k/type_v — seeded from the GGUF, editable in-chat, persisted forever.
Read the fine print.
NAME
anvil — forge-grade local LLM inference in your terminal
SYNOPSIS
anvil run model.gguf [--ctx N] [--ngl N]
[--type-k T] [--type-v T] [--mtp]
[--top-k 40] [--top-p 0.95] [--repeat-penalty 1.1]
[--grammar f.gbnf] [-p "..."]
DESCRIPTION
anvil loads any GGUF model, auto-detects your hardware, and drops you into a streaming chat REPL. First run launches a TUI wizard; after that it just forges.
FEATURES
- TurboQuant KV cache — 2/3/4-bit WHT-rotated, asymmetric K/V
- MTP + NextN speculative decoding
- Flash Attention on by default
- GBNF grammars for structured output
- Model registry — pull, import, nickname
- Streaming markdown — thinking blocks, bold, code, tables
- Hardware autodetect — CUDA · Vulkan · Metal · ROCm · CPU
- Live stats — tokens/sec, context fill, session export
- Single binary — zero telemetry, zero cloud
Commands at the anvil.
Everything you need mid-conversation, one keystroke away.
tokens generated, avg t/s, context fill, KV types
strike out the last turn and rebuild the cache
save the session to markdown in ~/.anvil/sessions
quench the context and start fresh
arch, trained context, encoder/decoder, GPU layers
retune sampling — persists to the model's profile
context usage bar, color-coded by heat
put down the hammer
anvil vs. the field.
We respect the other tools. We also think you should know what you're giving up.
| anvil | ollama | llama.cpp CLI | LM Studio | |
|---|---|---|---|---|
| TurboQuant KV cache | ✓ native | ✗ | fork only | ✗ |
| MTP / NextN spec decode | ✓ | ✗ | fork only | ✗ |
| First-run TUI wizard | ✓ | ✗ | ✗ | GUI |
| Per-model profiles | ✓ | Modelfiles | ✗ | ✗ |
| Model registry + pull | ✓ | ✓ | ✗ | GUI |
| Streaming markdown + thinking blocks | ✓ | ✗ | ✗ | partial |
| Terminal-native | ✓ | ✓ | ✓ | ✗ (app) |
| Session export / undo | ✓ | ✗ | ✗ | partial |
| Open source | ✓ MIT | ✓ | ✓ | ✗ |
| Telemetry | none | some | none | yes |
One config. Per-model profiles.
Global settings live in ~/.anvil/config.json. Every model gets its own profile in ~/.anvil/models.json — seeded from the GGUF, editable in-chat, persisted forever.
- Seeded from the GGUFn_ctx = the model's trained context, sampling params from its metadata. Filled only when unset.
- Changes persistSet /temp 0.9 or /ctx 65536 in-chat and the profile is written back to models.json.
- CLI always winsAny flag you pass overrides the file for that run.
- Zero artificial capsn_batch tracks n_ctx, and context size is bounded only by the backend's int32 position counter — not by a hardcoded limit.
One binary, many blades.
Offline copilot
Code review and generation on a plane, in a basement, behind a firewall. Your code never leaves the room.
Long-document research
TurboQuant’s KV compression means whole papers and codebases fit in context on a single GPU.
Structured output
GBNF grammars constrain the model to valid JSON, SQL, or your own format — guaranteed by the sampler.
Batch & scripts
Single-shot mode pipes cleanly into shell pipelines. Generate, grep, forge on.
Frequently struck questions.
Yes. TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate is by Amir Zandieh, Majid Daliri, Majid Hadian and Vahab Mirrokni (Google Research / Google DeepMind / NYU), arXiv 2504.19874. It proves near-optimal distortion — within ≈2.7× of the Shannon lower bound — and shows quality-neutral KV compression at 3.5 bits. The anvil fork implements the WHT-rotated family (turbo2/3/4) with backend-native kernels.
anvil ships the WHT-rotated TurboQuant family (2/3/4-bit K and V cache types, selectable per side). How much memory you save depends on your model and context — see the benchmark tables above from the Google Research paper (arXiv:2504.19874), which show TurboQuant matching the full-precision cache on LongBench and needle-in-a-haystack.
No — anvil runs on CPU with AVX2/NEON. But with a GPU it auto-offloads layers, and TurboQuant's KV compression means even modest VRAM fits long contexts. An RTX 3060 12GB is a perfectly good anvil.
Any GGUF file that llama.cpp supports — Llama, Qwen, Gemma, DeepSeek, Mistral, Phi, and friends. MTP/NextN speculative decoding additionally needs a model with prediction heads (Gemma 4, Qwen 3.6 UDT).
ollama is a great general-purpose server. anvil is a terminal-first power tool built specifically on the TurboQuant fork — you get asymmetric KV compression, speculative decoding, a setup TUI, a model registry with per-model profiles, and streaming markdown. No daemon, no server, just a binary.
Nothing. No telemetry, no update checks, no crash reports, no network calls at runtime. The only time anvil touches the network is when you install it or pull a model.
Yes. Edit ~/.anvil/config.json or a model's entry in ~/.anvil/models.json directly, or pass CLI flags (--ctx, --type-k, --ngl, …) which override for that run. In-chat /temp and /ctx persist to the profile.
What's heating up.
First light. Chat REPL, hardware detection, TUI wizard, TurboQuant KV.
Asymmetric K/V presets, MTP/NextN, /stats /undo /export, UTF-8 streaming, session persistence.
Ollama registry + local import for anvil pull, Hugging Face GGUF downloader.
Model registry with friendly names and persistent per-model profiles. Per-model settings schemas seeded from GGUF.
Streaming markdown renderer (mdtty) with live token redraw — thinking blocks, bold, code, tables. Zero artificial context caps.
anvil serve — OpenAI-compatible local server. Multimodal vision input.
Conversation branching, context extension UI, grammar-driven function calling.
Agent / tool-use loop, multi-model routing.
Changelog.
- Streaming markdown renderer (mdtty) with live token redraw
- Thinking blocks formatted in dim grey; response markdown fully styled
- n_batch tracks n_ctx; all artificial context caps removed
- Per-model profiles persisted from in-chat /temp /ctx changes
- Model registry with friendly names + persistent per-model profiles
- Per-model settings schema seeded from GGUF metadata
- anvil pull: Ollama registry + local ollama import + Hugging Face GGUF
- Windows build fixes (NOMINMAX, portable TTY check)
- Independent K/V cache type selection with presets (turbo4/turbo3 recommended)
- MTP + NextN speculative decoding support
- GBNF grammar-constrained output
- REPL commands: /stats /undo /export /model /temp /ctx
- Live tokens/sec + context usage bar
- Safe UTF-8 streaming (no more split multi-byte chars)
- Session export to markdown
- Config v2 with automatic migration
- Initial release — chat REPL + single-shot mode
- FTXUI first-run setup wizard
- Cross-platform hardware detection (macOS / Linux / Windows)
- TurboQuant KV cache compression
One line. That's the whole install.
curl -fsSL https://anvil-llm.github.io/anvil/install.sh | sh
Detects your OS + arch, grabs a pre-built binary with checksum verification, falls back to building from source. Installs to ~/.local/bin and creates ~/.anvil.
On NVIDIA Linux it grabs the CUDA build (driver-only — no CUDA toolkit) via anvil-nvidia-install.sh — universal: any distro, any NVIDIA GPU, auto-picks the right driver branch and keeps it DKMS-rebuilt across kernel updates.
git clone https://github.com/Anvil-LLM/anvil.git cd anvil git submodule update --init --recursive cmake -B build -DCMAKE_BUILD_TYPE=Release cmake --build build -j$(nproc) sudo cp build/anvil /usr/local/bin/
Linux cmake ≥ 3.18 · g++ / clang++ · git macOS xcode-select --install · cmake · git Windows VS 2022 + CMake + Git GPU backends auto-detected: CUDA · Vulkan · Metal · ROCm · SYCL · CPU
Put your model on the anvil.
One line installs it. One command runs it. Your GPU does the rest.
curl -fsSL https://anvil-llm.github.io/anvil/install.sh | sh