{"schema_version":"agidreams.edition.v1","id":412,"slug":"paged-experts-and-tiny-quants","title":"Paged Experts and Tiny Quants","date":"2026-10-26","published_at":"2026-10-26T15:10:05Z","canonical_url":"https://agidreams.us/edition/paged-experts-and-tiny-quants","markdown_url":"https://agidreams.us/edition/paged-experts-and-tiny-quants.md","json_url":"https://agidreams.us/edition/paged-experts-and-tiny-quants.json","content_format":"markdown","content":"<!-- SECTION: 🖥️ Paged Experts and Tiny Quants -->\n\nGreenBitAI's four-bit build of DeepSeek's October 17 refresh of V4.1-Flash takes local inference to its logical extreme: a 406 GiB model that runs on machines that cannot hold it. The recipe is mixed — routed experts at 4-bit, attention and the rest at 8-bit — with the experts paged in their own containers, so a machine loads what fits and streams the rest. On a 128 GB DGX Spark it pages; on a 512 GB Mac Studio it settles resident. The card declines to guess paging slowdown: the wait is \"the SSD's, not the model's.\" (more: https://huggingface.co/GreenBitAI/DeepSeek-V4.1-Flash-1017-4bit-paged)\n\ngbx-lm ships as one signed, notarized binary — no Python, no CUDA toolkit, just a driver — speaking three wire protocols on one port: OpenAI, the Responses API Codex uses, and Anthropic's Messages for Claude Code. Measured on a Mac Studio: 22.1 tok/s decode, 410-420 tok/s prefill, with the draft head's proposals verified by the model itself. v0.8.3 claims twice-as-fast long conversations; an optional DFlash 2 draft head pushes Qwen3.8-27B to 140 tok/s on code and math. (more: https://github.com/GreenBitAI/gbx-lm)\n\nThe llama.cpp crowd runs the same race from the bottom: a Reddit build runs Qwen-Flash-Next Coder — ISTA DASLab's 3.5-bits-per-weight GSQ RCO quant, half the experts pruned for an effective 1.89 bits per parameter — on 8 GB VRAM plus 24 GB RAM with the full 262K context. Selective expert offloading and n-gram speculation on disk buy about 70 tok/s prefill, 104 with a load-mode tweak, and roughly 8 tok/s generation, PCIe-bound. The critique is in the comments: that prefill rate is minutes to first token on an agent harness, and Strata users report far better numbers — running and usable are different claims. (more: https://old.reddit.com/r/LocalLLaMA/comments/1xebdzl/8_gb_gpu_262k_context_35_bpw_qwenflashnext_coder/)\n\nAbliteratedOutTheBox ports an inference-time abliteration hook — a llama.cpp fork, now on Strata — so refusal directions can be projected out of activations at runtime, without finetuning or hunting for a pre-abliterated quant. Flagged experimental, and priced by commenters: forced non-thinking mode, 23x prompt processing, 23x KV cache, no slot sharing. And on Hugging Face's trending list sits zai-org's GLM-4.7-Flash — the Flash lineage keeps appearing wherever cheap local inference is the story. (more: https://old.reddit.com/r/LocalLLaMA/comments/1xe8x9c/abliteratedoutthebox_strata_announcement/) (more: https://huggingface.co/zai-org/GLM-4.7-Flash)\n\n<!-- SECTION: ⚡ Unlocked Silicon -->\n\nThe strangest hardware story of the week sits below the driver. nlzy's turing-unlock is a Linux kernel module — an adaptation of Zagitta's gpu-unlock — that unlocks the full-speed FP16 tensor cores of NVIDIA's TU102, the RTX 2080 Ti die, with two speed-selector writes (SS0 to 0x88888888, SS1 to 0x8) and an ACR-mutex-3 release, then the normal Booter path. No BIOS flash, no modified driver. Tested on a Dell RTX 2080 Ti 22GB, both drivers. Measured cuBLAS FP16 matmul with FP32 accumulation: 50.3 to 68.3 TFLOPS at 1024-cubed — 120% of the 56.9 TFLOPS the Turing whitepaper calls peak — and up to 91.4, 160.6% of \"peak,\" at 2048-cubed. The datasheet number describes the locked part. The README warns: unlocked tensor cores draw very high power and may damage the card. (more: https://github.com/nlzy/turing-unlock)\n\nA widely shared post collects the unlock family across three GPU generations — RTX 4090 from 165.6 to 331.6 TFLOPS, RTX 3090 Ti BF16 from 87.5 to 157.4, near-doublings on 2080 Ti and 4070 Ti Super. The replies carry the reality check: a 4070 Ti Super measured 92.8 to 177.1 TFLOPS in FP16 matmul, yet FLUX.2 image passes only sped from 5.35 to 4.54 seconds, and INT8-bound video workloads did not move. Doubled TFLOPS pay off only where the workload is FP16/BF16 compute-bound — a 4090 at double TFLOPS is still one desk card. The repo's numbers are measured; the cross-generation aggregate is community-reported. (more: https://x.com/MaxForAI/status/2114402686821159108)\n\n<!-- SECTION: 🛠️ Agents That Hold Together -->\n\nMicrosoft and Hugging Face turned ThinkingBox from a benchmark into a training signal. The benchmark's premise: an agent's claim of completion is worthless — grade the database it leaves behind. ThinkingBox-Bench runs 507 business tasks with a real backend, MCP (Model Context Protocol) tools, a simulated user, and executable checks against final database state. The motivating result: Kimi-K3 solved 93.89% of tasks at least once across 20 attempts but completed only 68 of 507 every time. The binary check from terminal state is not just a grade — \"this can be a reward function.\"\n\nThe team fine-tuned Qwen3.8-27B with GRPO — reinforcement learning that scores rollouts against their group's average — on database-check rewards, 24 H100s. Task selection is the clever part: candidates probed against the base policy, success modeled as a Beta posterior, only mixed-success tasks kept — \"a task is hard for the current model only until that model learns it.\" On the held-out benchmark, pass@1 rose from 51.70% to 60.78%, and perfect 20-of-20 runs nearly tripled from 38 tasks to 107 — past Kimi-K3's 68. GPT-5.6 Sol edges it on pass@1 with only 82 dependable tasks. Pass@20 dipped slightly — \"The model did not become broader. It became more repeatable.\" On a hotel-downgrade task violating approval policy in 15 of 20 attempts before training, it preserved state in all 20. Cost per successful attempt fell to $0.139; paper: arXiv 2608.19741. (more: https://huggingface.co/blog/microsoft/thinkingbox-training)\n\nUbicloud, serving Kimi K2.6 on vLLM 0.28, saw 92% tool-call schema accuracy, and malformed calls poison the context and multiply retries. The fix is XGrammar constrained decoding: the engine masks invalid tokens at each step, forcing the call down a grammar of valid tool names and schema-conformant arguments, with masks precomputed and applied on the GPU. Schema accuracy hit 100% on Moonshot's K2 Vendor Verifier. The history: vLLM shipped it in 0.25 and scores dropped to 91%, because forced tokens derail models whose unconstrained output diverges from their own format — gpt-oss-20b, Qwen 3.6, and GLM 5.1 users reported problems — plus broken grammar translations and wrongly rejected speculative tokens. Maintainers made opt-out the default, adding a server-side flag in 0.31; Ubicloud enabled it, and DeepSeek V4 Flash straightened out. Tool-call accuracy is an inference-engine property, not only a model property. (more: https://www.ubicloud.com/blog/fixing-tool-call-accuracy-in-vllm)\n\n<!-- SECTION: 🔒 The Tailscale Reckoning -->\n\nErica Windisch's tailworm report — an independent assessment of tailscale v1.104.2 and headscale v0.29.5, every finding reproduced on unmodified, released software — is the week's biggest security story. The thesis: a single, highly privileged root process replaced OpenSSH-style privilege separation while marketing security as its design and purpose. The headline class: the root-privileged tailscaled dials, replaces, and deletes privileged sockets on an unprivileged peer's behalf, and a rename() race — 170,000 path-state flips per second — defeats the home, /tmp, and /run/user whitelist. The verdict is three words, \"not a security boundary,\" backed by lab root-execution demos: dockerd, kubelet credential delivery, authkey capture, rogue-node enrollment under a victim's identity.\n\nThe rest compounds. conn25, the experimental app-connector, validates client-declared destinations only for address family, so any entitled user can steer egress at cloud metadata services, internal LANs, or every service in a Kubernetes cluster. An unauthenticated peer can exhaust ~536 MB in seven seconds and crash-loop headscale, blocking org-wide onboarding; a 10 MB DERP response amplifies to 1.34 GB. A one-line serve command forwarding a unix: socket exposes the localapi admin socket to remote peers — node takeover. Wormability is measured, not asserted: with policy-granted SSH plus the socket findings, the report describes \"automated spread to every device the policy permits — no passwords, no stolen keys, no user interaction, under a second per hop.\" Check mode does not constrain forwarding, and headscale v0.29.5 grants it unconditionally — \"treat every headscale SSH grant as root-equivalent.\" Containment means deleting the node at the control plane: \"reimaging alone is not containment.\" (more: https://tailworm.com)\n\nTailscale's VP of Engineering, Mike Shaver, answered on Reddit: \"we're not recommending that everyone disable Tailscale SSH.\" The confirmed issues concern what happens after authorization — \"a session may be able to do more than the configured local user should be allowed to do\" — not unauthenticated access to arbitrary machines. The mitigation matches the assessment: TS_SSH_DISABLE_FORWARDING=1 addresses several of the more serious findings; check mode adds a barrier but will not fix privilege escalation. An advisory is promised; the subject is Tailscale's own SSH server, not stock sshd over the tunnel. Lab reproduction on unmodified software is the strongest evidence an assessment can offer; per-configuration impact is still being validated. (more: https://www.reddit.com/r/Tailscale/comments/1xeoj1l/recent_tailscale_vulnerability_reports/?share_id=AK6vUwXn2gqbh3uZNkelL)\n\n<!-- SECTION: 🪲 Memory Corruption, Stock and Bundled -->\n\nThe PHP exploitation arc has its stock-config answer. When the session-handling bugs in PHP core were patched last week, the open question was whether any public chain worked against an unmodified target — the first PoC wave leaned on author-planted lab setups. 9bie's php2shell-bypass-aslr-exp is the counterexample: pure HTTP, against the official PHP image with stock php.ini-production settings, official prefork with MaxRequestWorkers=150, and a target page with only an echo statement. Ten of ten runs on stock x86_64, ten on arm64, three to four seconds per RCE. (more: https://github.com/9bie/php2shell-bypass-aslr-exp)\n\nThe chain demolishes PHP's request lifecycle. Worker pinning first: 149 half-finished POST bodies occupy every Apache child but one, so all four stages hit the same worker and stale pointers survive. Stage A1 drives an out-of-memory in end-of-request cleanup with 860 nested file fields and a PHP_SESSION_UPLOAD_PROGRESS token, leaving dangling pointers. Stage B1 turns Set-Cookie into an out-of-bounds read encoding ~800 KB of adjacent heap — chunk pointers and a zval destructor pointer appearing 1,095 times — yielding the libphp base and defeating ASLR arithmetically. A final overwrite stages a fake array whose destructor is PHP's own command-execution handler; ordinary shutdown calls it with the attacker's command. The README's recipe is blunt: compute offsets per PHP build, mass-fuzz, let crashed workers restart — authorized research only, no debugger, custom ini, or target-side files.\n\nBundled C libraries have their own CVE: CVE-2026-111497, \"Node2Shell\" or Broccoli, an out-of-bounds write in the Brotli decoder — CVSS v4.0 9.2, in CISA's KEV catalog — affecting Brotli 1.0.3-1.2.0 and the copies bundled into Node.js, Python, Bun, and Envoy. The decoder \"confuses malformed input with incomplete input, keeps filling an 8-byte state buffer past its end, and corrupts adjacent state.\" A detection-engineering gist — \"reproduction and detection — not discovery\" — runs one verified 23,068,706-byte payload through four runtimes: vulnerable Node 24.22.0 emits Z_BUF_ERROR, patched 24.22.1 hard-rejects, and Python's Brotli 1.2.0 streaming decoder raises no exception on a complete stream — \"the bug.\" An ASAN harness on the raw decoder found nothing — the corruption lives in the Node and Python wrapper retry loops. Detection ships as Sigma, Suricata, ModSecurity (verified 7/7 against libmodsecurity 3.0.14) and falco rules, with the caveat that Z_BUF_ERROR fires on any truncated stream, and \"a runtime sensor cannot see the overflow\": the observable chain is worker crash, then in-place execution. Public results are precise: Imperva's red team hit RCE on the official Node 24.22.0 arm64 image \"after several attempts\"; a blind remote attempt failed under ASLR. Two inventory traps: patched Node 24.22.1 still reports Brotli 1.2.0, and Express 5's 100kb limit counts decompressed bytes, letting the compressed trigger through. (more: https://gist.github.com/Blevene/7d9d769f54826e1fe770d8c19d5ef0b2)\n\n<!-- SECTION: 🛡️ AI Security: Campaign and Countermeasures -->\n\nThe Model Context Protocol ecosystem has its supply-chain case study. Knostic found an npm package that looks like Amazon's Selling Partner API MCP server — it reuses the open-source code — but swaps the destinations: authenticated operations send the client ID, client secret, and refresh token to a hardcoded IP over plain HTTP. All seven versions landed on June 22 within two hours, IP present from the first; the README never discloses it. The exposure outlives the package: generated integration code embeds the same IP-based URLs, so apps built on its examples keep leaking. What it does not establish is plain: who operates the destination, and whether real credentials reached it or were misused. The download counter — 929 on publication day — counts scanners and CI, not victims. The lesson: \"familiar authentication code now has a different recipient. Reviews that skip the destination miss it.\" (more: https://www.knostic.ai/blog/repackaged-amazon-sp-api-mcp-sends-app-secrets-to-an-undisclosed-ip)\n\nThe defense consolidates around one idea: evidence gates. Arad-Code, an AGPL security auditor, drives an LLM through markdown skill playbooks with real tools — a Model Context Protocol code graph, ripgrep, semgrep, a read-only shell. Its core rule: \"nothing reaches the report unless it can be pointed to in the code.\" Findings cite file, line, and verbatim quote; the engine re-reads and discards anything not there; confidence caps by evidence class — grep hit 4, traced chain 10. (more: https://github.com/fedman0/Arad-Code) xss-specialist applies the same discipline to web attacks: a CONFIRMED, LIKELY, INCONCLUSIVE, NOT_VULNERABLE state machine where \"a finding is never upgraded to CONFIRMED from model confidence\" — a headless-browser oracle is the sole authority. On its frozen 102-case benchmark: 0.941 precision, 1.000 recall — against a naive scanner's 97% false positives. Its honest failure: fine-tuning never fixed near-miss sanitizer leakage — a model that learns DOMPurify.sanitize is safe \"will happily call DOMPvrify.sanitize, OMPurify.sanitize, or sanitizeHtm1 safe too\" — so exact-identity verification replaces model judgment. \"Don't trust the model's names — trust the browser's execution.\" (more: https://github.com/SecFathy/xss-specialist)\n\n\"Detect and Suppress\" (arXiv 2610.03498) targets adversarial patches — printed patterns that flip a vision-language-action model's outputs — in OpenVLA and SmolVLA on LIBERO. Instead of weak input preprocessing or an adversarial retrain, they train a sparse autoencoder on residual-stream activations from clean trajectories, find the feature that tracks patch presence, detect the patch with a linear probe, and suppress that one feature only when the probe fires. Conditional intervention recovers success rates to oracle-detection levels across four settings; continuous suppression degrades benign performance. Limitations, stated rather than buried: SmolVLA gains are modest with benign-performance cost, and under persistent attack nothing helps — \"Even with oracle detection, suppressing the selected feature does not recover SR under persistent attacks.\" Testing on unseen attacks and physical robots would settle generalization. (more: https://arxiv.org/abs/2610.03498v1)\n\n<!-- SECTION: 🧠 Small Models, Strange Feats -->\n\nWorldTUI asks a strange question with a 360M-parameter fine-tune of Qwen2.5-Coder: can a model with no shell and no filesystem pretend to be a terminal? Keystrokes go in; predicted screen updates come out. Training data: 336k episodes, two billion tokens, of real Docker sessions with randomized hostnames, prompts, terminal sizes and personas, a 1.2% typo rate, and 35% starting mid-application. The terminal is a character grid over a 333-token vocabulary — 256 byte tokens, 64 row tokens, a COPY op encoding a scrolled row in three tokens instead of ~96, and a WAIT token for time, which is how snake moves unprompted — with grammar-constrained decoding guaranteeing valid updates.\n\nCompounding drift was the failure mode; the fix is a DAgger-style loop where the model predicts, the real screen is compared, and training uses the corrections — but only when the model could infer the fix, else \"training on those examples would actually reward hallucinations.\" PresetBench rose from 24/47 to 38/47. Full-screen redraws looked hopeless at ~3,000 tokens a frame until rows were decoded in parallel, cutting large-frame p95 latency from 6.7s to 0.87s. A 32-world memory test scored 9/13 against 2/13 and 3/13 for prompted larger models, and distillation found the load-bearing layer: removing layer 7 barely moved next-token accuracy (99.3%→98.9%) while memory readbacks collapsed 64/64→0/64. Training cost ~75 GPU-hours on one RTX 4090; the browser demo runs a 208MB int4 model locally on a JavaScript Qwen decoder with WebGPU shaders. The verdict: \"Is it useful? Not really.\" (more: https://simedw.com/2026/10/21/world-model-terminal/)\n\nThe same amateur-scale energy reached voice. A self-described \"run of the mill web-dev guy\" distilled Kokoro — the 82M-parameter TTS model that keeps turning up in local voice stacks — to run 50-100x faster, training a day on a basement 4070 Ti Super with Claude Opus 5.5 hill-climbing a clear metric. The framing is the story: \"here's something I don't even have a clue how it exactly works,\" and, \"I know it's Opus slop, but still really cool.\" The numbers are self-reported, the pattern not isolated — a top comment describes Claude optimizing an sglang stack on dual 7900XTXs, decode at 100k context from 8 to 60-100 tok/s, GEMMs ~10% faster than ported ones. (more: https://old.reddit.com/r/LocalLLaMA/comments/1xcuqor/making_kokoro_tts_50100x_faster_without_knowing/)\n\n<!-- SECTION: 🤖 The Agent Workspace -->\n\nA widely shared walkthrough of Anthropic's rebuilt Claude Code desktop app argues it is \"really not only for coders\" — same tool underneath, with access to any file and to CLIs. The redesign that matters is projects, which Anthropic has framed as a move from folder to conversation: a coordinator chat spins up threads for subtasks, each iterable without overloading the coordinator's context, sharing memory, threads runnable in the cloud while the laptop is closed; nested sessions are the local variant. Around them: in-chat to-do lists that tick off, context cards with a compact button at token thresholds, side chats that can read and message the main conversation, and artifact tools like Motion, animated video purely in code — \"far cheaper than a video model.\" The slash commands carry the wins: /doctor audited 404 of the presenter's files and found 312 problems — instructions for older models wasting tokens; /skill-doctor flagged never-used skills and \"saved me about 20,000 tokens\"; /advisor lets a cheap model consult the expensive one at critical moments. (more: https://www.youtube.com/watch?v=J5EbL1UaqG8)\n\nThe self-hosted counterweight is humux: one Docker container, MIT-licensed, no telemetry, air-gappable, running a squad — lead, developer, researcher, marketer, QA — coordinating through GitHub issues, each agent its own GitHub App with name and avatar, escalating only what a human labels. Each agent gets its own model, voice, Telegram bot, and memory; the foundation is boring on purpose: Himalaya for IMAP, CalDAV for calendars, Playwright for browsing, faster-whisper and Kokoro 82M for voice, SQLite for storage — \"no external databases, no Kubernetes, no account to sign up for.\" Credentials stay out of model context behind two-tier encryption; skills are taught by writing markdown files. And for the unglamorous middle of developer work, JSON Crack keeps doing what it has since 2022: turning JSON, YAML, CSV, and XML into interactive graphs, generating TypeScript interfaces, Go structs, and Rust serde types, running jq — all locally, nothing stored on any server. (more: https://humux.dev) (more: https://github.com/AykutSarac/jsoncrack.com)\n","word_count":2998,"content_sha256":"49e7ec71bc3f1da9c2e04d292d09e139d1656a414941a913d8c4fbf1783ca1b8","truncated":false,"sources":[{"title":"GreenBitAI DeepSeek-V4.1-Flash-1017 4-bit paged","url":"https://huggingface.co/GreenBitAI/DeepSeek-V4.1-Flash-1017-4bit-paged","domain":"huggingface.co"},{"title":"GreenBitAI/gbx-lm","url":"https://github.com/GreenBitAI/gbx-lm","domain":"github.com"},{"title":"8 GB GPU, 262K context 3.5 BPW Qwen-Flash-Next Coder 70 T/s using llama.cpp","url":"https://old.reddit.com/r/LocalLLaMA/comments/1xebdzl/8_gb_gpu_262k_context_35_bpw_qwenflashnext_coder/","domain":"old.reddit.com"},{"title":"AbliteratedOutTheBox Strata Announcement","url":"https://old.reddit.com/r/LocalLLaMA/comments/1xe8x9c/abliteratedoutthebox_strata_announcement/","domain":"old.reddit.com"},{"title":"zai-org/GLM-4.7-Flash","url":"https://huggingface.co/zai-org/GLM-4.7-Flash","domain":"huggingface.co"},{"title":"nlzy/turing-unlock","url":"https://github.com/nlzy/turing-unlock","domain":"github.com"},{"title":"MaxForAI update (x.com)","url":"https://x.com/MaxForAI/status/2114402686821159108","domain":"x.com"},{"title":"Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox","url":"https://huggingface.co/blog/microsoft/thinkingbox-training","domain":"huggingface.co"},{"title":"Fixing Tool-Call Accuracy in vLLM","url":"https://www.ubicloud.com/blog/fixing-tool-call-accuracy-in-vllm","domain":"ubicloud.com"},{"title":"tailworm.com","url":"https://tailworm.com/","domain":"tailworm.com"},{"title":"Recent Tailscale vulnerability reports","url":"https://www.reddit.com/r/Tailscale/comments/1xeoj1l/recent_tailscale_vulnerability_reports/?share_id=AK6vUwXn2gqbh3uZNkelL","domain":"reddit.com"},{"title":"9bie/php2shell-bypass-aslr-exp","url":"https://github.com/9bie/php2shell-bypass-aslr-exp","domain":"github.com"},{"title":"Blevene security gist (7d9d769f)","url":"https://gist.github.com/Blevene/7d9d769f54826e1fe770d8c19d5ef0b2","domain":"gist.github.com"},{"title":"Repackaged Amazon SP-API MCP Sends App Secrets to an Undisclosed IP","url":"https://www.knostic.ai/blog/repackaged-amazon-sp-api-mcp-sends-app-secrets-to-an-undisclosed-ip","domain":"knostic.ai"},{"title":"fedman0/Arad-Code","url":"https://github.com/fedman0/Arad-Code","domain":"github.com"},{"title":"SecFathy/xss-specialist","url":"https://github.com/SecFathy/xss-specialist","domain":"github.com"},{"title":"Detect and Suppress: A Mechanistic Defense against Adversarial Patches in VLA Models","url":"https://arxiv.org/abs/2610.03498v1","domain":"arxiv.org"},{"title":"WorldTUI: I trained a 360M model to pretend to be a terminal","url":"https://simedw.com/2026/10/21/world-model-terminal/","domain":"simedw.com"},{"title":"Making Kokoro TTS 50-100x faster without knowing how","url":"https://old.reddit.com/r/LocalLLaMA/comments/1xcuqor/making_kokoro_tts_50100x_faster_without_knowing/","domain":"old.reddit.com"},{"title":"Featured video (YouTube J5EbL1UaqG8)","url":"https://www.youtube.com/watch?v=J5EbL1UaqG8","domain":"youtube.com"},{"title":"humux.dev","url":"https://humux.dev/","domain":"humux.dev"},{"title":"AykutSarac/jsoncrack.com","url":"https://github.com/AykutSarac/jsoncrack.com","domain":"github.com"}],"topics":["AI Agents","AI Hardware","AI Policy","AI Security","Agentic Coding","Model Architecture","Open-Weight Models","Prompt Engineering","Quantization & Efficiency","RAG & Retrieval"],"audio_urls":["https://agidreams.us/static/audio/report-1793027405.mp3"]}