Paged Experts and Tiny Quants

Paged Experts and Tiny Quants

GreenBitAI's four-bit build of DeepSeek's October 17 refresh of V4.1-Flash takes local inference to its logical extreme: a 406 GiB model that runs on machines that cannot hold it. The recipe is mixed — routed experts at 4-bit, attention and the rest at 8-bit — with the experts paged in their own containers, so a machine loads what fits and streams the rest. On a 128 GB DGX Spark it pages; on a 512 GB Mac Studio it settles resident. The card declines to guess paging slowdown: the wait is "the SSD's, not the model's." (more: https://huggingface.co/GreenBitAI/DeepSeek-V4.1-Flash-1017-4bit-paged)

gbx-lm ships as one signed, notarized binary — no Python, no CUDA toolkit, just a driver — speaking three wire protocols on one port: OpenAI, the Responses API Codex uses, and Anthropic's Messages for Claude Code. Measured on a Mac Studio: 22.1 tok/s decode, 410-420 tok/s prefill, with the draft head's proposals verified by the model itself. v0.8.3 claims twice-as-fast long conversations; an optional DFlash 2 draft head pushes Qwen3.8-27B to 140 tok/s on code and math. (more: https://github.com/GreenBitAI/gbx-lm)

The llama.cpp crowd runs the same race from the bottom: a Reddit build runs Qwen-Flash-Next Coder — ISTA DASLab's 3.5-bits-per-weight GSQ RCO quant, half the experts pruned for an effective 1.89 bits per parameter — on 8 GB VRAM plus 24 GB RAM with the full 262K context. Selective expert offloading and n-gram speculation on disk buy about 70 tok/s prefill, 104 with a load-mode tweak, and roughly 8 tok/s generation, PCIe-bound. The critique is in the comments: that prefill rate is minutes to first token on an agent harness, and Strata users report far better numbers — running and usable are different claims. (more: https://old.reddit.com/r/LocalLLaMA/comments/1xebdzl/8_gb_gpu_262k_context_35_bpw_qwenflashnext_coder/)

AbliteratedOutTheBox ports an inference-time abliteration hook — a llama.cpp fork, now on Strata — so refusal directions can be projected out of activations at runtime, without finetuning or hunting for a pre-abliterated quant. Flagged experimental, and priced by commenters: forced non-thinking mode, 23x prompt processing, 23x KV cache, no slot sharing. And on Hugging Face's trending list sits zai-org's GLM-4.7-Flash — the Flash lineage keeps appearing wherever cheap local inference is the story. (more: https://old.reddit.com/r/LocalLLaMA/comments/1xe8x9c/abliteratedoutthebox_strata_announcement/) (more: https://huggingface.co/zai-org/GLM-4.7-Flash)

Unlocked Silicon

The strangest hardware story of the week sits below the driver. nlzy's turing-unlock is a Linux kernel module — an adaptation of Zagitta's gpu-unlock — that unlocks the full-speed FP16 tensor cores of NVIDIA's TU102, the RTX 2080 Ti die, with two speed-selector writes (SS0 to 0x88888888, SS1 to 0x8) and an ACR-mutex-3 release, then the normal Booter path. No BIOS flash, no modified driver. Tested on a Dell RTX 2080 Ti 22GB, both drivers. Measured cuBLAS FP16 matmul with FP32 accumulation: 50.3 to 68.3 TFLOPS at 1024-cubed — 120% of the 56.9 TFLOPS the Turing whitepaper calls peak — and up to 91.4, 160.6% of "peak," at 2048-cubed. The datasheet number describes the locked part. The README warns: unlocked tensor cores draw very high power and may damage the card. (more: https://github.com/nlzy/turing-unlock)

A widely shared post collects the unlock family across three GPU generations — RTX 4090 from 165.6 to 331.6 TFLOPS, RTX 3090 Ti BF16 from 87.5 to 157.4, near-doublings on 2080 Ti and 4070 Ti Super. The replies carry the reality check: a 4070 Ti Super measured 92.8 to 177.1 TFLOPS in FP16 matmul, yet FLUX.2 image passes only sped from 5.35 to 4.54 seconds, and INT8-bound video workloads did not move. Doubled TFLOPS pay off only where the workload is FP16/BF16 compute-bound — a 4090 at double TFLOPS is still one desk card. The repo's numbers are measured; the cross-generation aggregate is community-reported. (more: https://x.com/MaxForAI/status/2114402686821159108)

Agents That Hold Together

Microsoft and Hugging Face turned ThinkingBox from a benchmark into a training signal. The benchmark's premise: an agent's claim of completion is worthless — grade the database it leaves behind. ThinkingBox-Bench runs 507 business tasks with a real backend, MCP (Model Context Protocol) tools, a simulated user, and executable checks against final database state. The motivating result: Kimi-K3 solved 93.89% of tasks at least once across 20 attempts but completed only 68 of 507 every time. The binary check from terminal state is not just a grade — "this can be a reward function."

The team fine-tuned Qwen3.8-27B with GRPO — reinforcement learning that scores rollouts against their group's average — on database-check rewards, 24 H100s. Task selection is the clever part: candidates probed against the base policy, success modeled as a Beta posterior, only mixed-success tasks kept — "a task is hard for the current model only until that model learns it." On the held-out benchmark, pass@1 rose from 51.70% to 60.78%, and perfect 20-of-20 runs nearly tripled from 38 tasks to 107 — past Kimi-K3's 68. GPT-5.6 Sol edges it on pass@1 with only 82 dependable tasks. Pass@20 dipped slightly — "The model did not become broader. It became more repeatable." On a hotel-downgrade task violating approval policy in 15 of 20 attempts before training, it preserved state in all 20. Cost per successful attempt fell to $0.139; paper: arXiv 2608.19741. (more: https://huggingface.co/blog/microsoft/thinkingbox-training)

Ubicloud, serving Kimi K2.6 on vLLM 0.28, saw 92% tool-call schema accuracy, and malformed calls poison the context and multiply retries. The fix is XGrammar constrained decoding: the engine masks invalid tokens at each step, forcing the call down a grammar of valid tool names and schema-conformant arguments, with masks precomputed and applied on the GPU. Schema accuracy hit 100% on Moonshot's K2 Vendor Verifier. The history: vLLM shipped it in 0.25 and scores dropped to 91%, because forced tokens derail models whose unconstrained output diverges from their own format — gpt-oss-20b, Qwen 3.6, and GLM 5.1 users reported problems — plus broken grammar translations and wrongly rejected speculative tokens. Maintainers made opt-out the default, adding a server-side flag in 0.31; Ubicloud enabled it, and DeepSeek V4 Flash straightened out. Tool-call accuracy is an inference-engine property, not only a model property. (more: https://www.ubicloud.com/blog/fixing-tool-call-accuracy-in-vllm)

The Tailscale Reckoning

Erica Windisch's tailworm report — an independent assessment of tailscale v1.104.2 and headscale v0.29.5, every finding reproduced on unmodified, released software — is the week's biggest security story. The thesis: a single, highly privileged root process replaced OpenSSH-style privilege separation while marketing security as its design and purpose. The headline class: the root-privileged tailscaled dials, replaces, and deletes privileged sockets on an unprivileged peer's behalf, and a rename() race — 170,000 path-state flips per second — defeats the home, /tmp, and /run/user whitelist. The verdict is three words, "not a security boundary," backed by lab root-execution demos: dockerd, kubelet credential delivery, authkey capture, rogue-node enrollment under a victim's identity.

The rest compounds. conn25, the experimental app-connector, validates client-declared destinations only for address family, so any entitled user can steer egress at cloud metadata services, internal LANs, or every service in a Kubernetes cluster. An unauthenticated peer can exhaust ~536 MB in seven seconds and crash-loop headscale, blocking org-wide onboarding; a 10 MB DERP response amplifies to 1.34 GB. A one-line serve command forwarding a unix: socket exposes the localapi admin socket to remote peers — node takeover. Wormability is measured, not asserted: with policy-granted SSH plus the socket findings, the report describes "automated spread to every device the policy permits — no passwords, no stolen keys, no user interaction, under a second per hop." Check mode does not constrain forwarding, and headscale v0.29.5 grants it unconditionally — "treat every headscale SSH grant as root-equivalent." Containment means deleting the node at the control plane: "reimaging alone is not containment." (more: https://tailworm.com)

Tailscale's VP of Engineering, Mike Shaver, answered on Reddit: "we're not recommending that everyone disable Tailscale SSH." The confirmed issues concern what happens after authorization — "a session may be able to do more than the configured local user should be allowed to do" — not unauthenticated access to arbitrary machines. The mitigation matches the assessment: TS_SSH_DISABLE_FORWARDING=1 addresses several of the more serious findings; check mode adds a barrier but will not fix privilege escalation. An advisory is promised; the subject is Tailscale's own SSH server, not stock sshd over the tunnel. Lab reproduction on unmodified software is the strongest evidence an assessment can offer; per-configuration impact is still being validated. (more: https://www.reddit.com/r/Tailscale/comments/1xeoj1l/recent_tailscale_vulnerability_reports/?share_id=AK6vUwXn2gqbh3uZNkelL)

Memory Corruption, Stock and Bundled

The PHP exploitation arc has its stock-config answer. When the session-handling bugs in PHP core were patched last week, the open question was whether any public chain worked against an unmodified target — the first PoC wave leaned on author-planted lab setups. 9bie's php2shell-bypass-aslr-exp is the counterexample: pure HTTP, against the official PHP image with stock php.ini-production settings, official prefork with MaxRequestWorkers=150, and a target page with only an echo statement. Ten of ten runs on stock x86_64, ten on arm64, three to four seconds per RCE. (more: https://github.com/9bie/php2shell-bypass-aslr-exp)

The chain demolishes PHP's request lifecycle. Worker pinning first: 149 half-finished POST bodies occupy every Apache child but one, so all four stages hit the same worker and stale pointers survive. Stage A1 drives an out-of-memory in end-of-request cleanup with 860 nested file fields and a PHP_SESSION_UPLOAD_PROGRESS token, leaving dangling pointers. Stage B1 turns Set-Cookie into an out-of-bounds read encoding ~800 KB of adjacent heap — chunk pointers and a zval destructor pointer appearing 1,095 times — yielding the libphp base and defeating ASLR arithmetically. A final overwrite stages a fake array whose destructor is PHP's own command-execution handler; ordinary shutdown calls it with the attacker's command. The README's recipe is blunt: compute offsets per PHP build, mass-fuzz, let crashed workers restart — authorized research only, no debugger, custom ini, or target-side files.

Bundled C libraries have their own CVE: CVE-2026-111497, "Node2Shell" or Broccoli, an out-of-bounds write in the Brotli decoder — CVSS v4.0 9.2, in CISA's KEV catalog — affecting Brotli 1.0.3-1.2.0 and the copies bundled into Node.js, Python, Bun, and Envoy. The decoder "confuses malformed input with incomplete input, keeps filling an 8-byte state buffer past its end, and corrupts adjacent state." A detection-engineering gist — "reproduction and detection — not discovery" — runs one verified 23,068,706-byte payload through four runtimes: vulnerable Node 24.22.0 emits Z_BUF_ERROR, patched 24.22.1 hard-rejects, and Python's Brotli 1.2.0 streaming decoder raises no exception on a complete stream — "the bug." An ASAN harness on the raw decoder found nothing — the corruption lives in the Node and Python wrapper retry loops. Detection ships as Sigma, Suricata, ModSecurity (verified 7/7 against libmodsecurity 3.0.14) and falco rules, with the caveat that Z_BUF_ERROR fires on any truncated stream, and "a runtime sensor cannot see the overflow": the observable chain is worker crash, then in-place execution. Public results are precise: Imperva's red team hit RCE on the official Node 24.22.0 arm64 image "after several attempts"; a blind remote attempt failed under ASLR. Two inventory traps: patched Node 24.22.1 still reports Brotli 1.2.0, and Express 5's 100kb limit counts decompressed bytes, letting the compressed trigger through. (more: https://gist.github.com/Blevene/7d9d769f54826e1fe770d8c19d5ef0b2)

AI Security: Campaign and Countermeasures

The Model Context Protocol ecosystem has its supply-chain case study. Knostic found an npm package that looks like Amazon's Selling Partner API MCP server — it reuses the open-source code — but swaps the destinations: authenticated operations send the client ID, client secret, and refresh token to a hardcoded IP over plain HTTP. All seven versions landed on June 22 within two hours, IP present from the first; the README never discloses it. The exposure outlives the package: generated integration code embeds the same IP-based URLs, so apps built on its examples keep leaking. What it does not establish is plain: who operates the destination, and whether real credentials reached it or were misused. The download counter — 929 on publication day — counts scanners and CI, not victims. The lesson: "familiar authentication code now has a different recipient. Reviews that skip the destination miss it." (more: https://www.knostic.ai/blog/repackaged-amazon-sp-api-mcp-sends-app-secrets-to-an-undisclosed-ip)

The defense consolidates around one idea: evidence gates. Arad-Code, an AGPL security auditor, drives an LLM through markdown skill playbooks with real tools — a Model Context Protocol code graph, ripgrep, semgrep, a read-only shell. Its core rule: "nothing reaches the report unless it can be pointed to in the code." Findings cite file, line, and verbatim quote; the engine re-reads and discards anything not there; confidence caps by evidence class — grep hit 4, traced chain 10. (more: https://github.com/fedman0/Arad-Code) xss-specialist applies the same discipline to web attacks: a CONFIRMED, LIKELY, INCONCLUSIVE, NOT_VULNERABLE state machine where "a finding is never upgraded to CONFIRMED from model confidence" — a headless-browser oracle is the sole authority. On its frozen 102-case benchmark: 0.941 precision, 1.000 recall — against a naive scanner's 97% false positives. Its honest failure: fine-tuning never fixed near-miss sanitizer leakage — a model that learns DOMPurify.sanitize is safe "will happily call DOMPvrify.sanitize, OMPurify.sanitize, or sanitizeHtm1 safe too" — so exact-identity verification replaces model judgment. "Don't trust the model's names — trust the browser's execution." (more: https://github.com/SecFathy/xss-specialist)

"Detect and Suppress" (arXiv 2610.03498) targets adversarial patches — printed patterns that flip a vision-language-action model's outputs — in OpenVLA and SmolVLA on LIBERO. Instead of weak input preprocessing or an adversarial retrain, they train a sparse autoencoder on residual-stream activations from clean trajectories, find the feature that tracks patch presence, detect the patch with a linear probe, and suppress that one feature only when the probe fires. Conditional intervention recovers success rates to oracle-detection levels across four settings; continuous suppression degrades benign performance. Limitations, stated rather than buried: SmolVLA gains are modest with benign-performance cost, and under persistent attack nothing helps — "Even with oracle detection, suppressing the selected feature does not recover SR under persistent attacks." Testing on unseen attacks and physical robots would settle generalization. (more: https://arxiv.org/abs/2610.03498v1)

Small Models, Strange Feats

WorldTUI asks a strange question with a 360M-parameter fine-tune of Qwen2.5-Coder: can a model with no shell and no filesystem pretend to be a terminal? Keystrokes go in; predicted screen updates come out. Training data: 336k episodes, two billion tokens, of real Docker sessions with randomized hostnames, prompts, terminal sizes and personas, a 1.2% typo rate, and 35% starting mid-application. The terminal is a character grid over a 333-token vocabulary — 256 byte tokens, 64 row tokens, a COPY op encoding a scrolled row in three tokens instead of ~96, and a WAIT token for time, which is how snake moves unprompted — with grammar-constrained decoding guaranteeing valid updates.

Compounding drift was the failure mode; the fix is a DAgger-style loop where the model predicts, the real screen is compared, and training uses the corrections — but only when the model could infer the fix, else "training on those examples would actually reward hallucinations." PresetBench rose from 24/47 to 38/47. Full-screen redraws looked hopeless at ~3,000 tokens a frame until rows were decoded in parallel, cutting large-frame p95 latency from 6.7s to 0.87s. A 32-world memory test scored 9/13 against 2/13 and 3/13 for prompted larger models, and distillation found the load-bearing layer: removing layer 7 barely moved next-token accuracy (99.3%→98.9%) while memory readbacks collapsed 64/64→0/64. Training cost ~75 GPU-hours on one RTX 4090; the browser demo runs a 208MB int4 model locally on a JavaScript Qwen decoder with WebGPU shaders. The verdict: "Is it useful? Not really." (more: https://simedw.com/2026/10/21/world-model-terminal/)

The same amateur-scale energy reached voice. A self-described "run of the mill web-dev guy" distilled Kokoro — the 82M-parameter TTS model that keeps turning up in local voice stacks — to run 50-100x faster, training a day on a basement 4070 Ti Super with Claude Opus 5.5 hill-climbing a clear metric. The framing is the story: "here's something I don't even have a clue how it exactly works," and, "I know it's Opus slop, but still really cool." The numbers are self-reported, the pattern not isolated — a top comment describes Claude optimizing an sglang stack on dual 7900XTXs, decode at 100k context from 8 to 60-100 tok/s, GEMMs ~10% faster than ported ones. (more: https://old.reddit.com/r/LocalLLaMA/comments/1xcuqor/making_kokoro_tts_50100x_faster_without_knowing/)

The Agent Workspace

A widely shared walkthrough of Anthropic's rebuilt Claude Code desktop app argues it is "really not only for coders" — same tool underneath, with access to any file and to CLIs. The redesign that matters is projects, which Anthropic has framed as a move from folder to conversation: a coordinator chat spins up threads for subtasks, each iterable without overloading the coordinator's context, sharing memory, threads runnable in the cloud while the laptop is closed; nested sessions are the local variant. Around them: in-chat to-do lists that tick off, context cards with a compact button at token thresholds, side chats that can read and message the main conversation, and artifact tools like Motion, animated video purely in code — "far cheaper than a video model." The slash commands carry the wins: /doctor audited 404 of the presenter's files and found 312 problems — instructions for older models wasting tokens; /skill-doctor flagged never-used skills and "saved me about 20,000 tokens"; /advisor lets a cheap model consult the expensive one at critical moments. (more: https://www.youtube.com/watch?v=J5EbL1UaqG8)

The self-hosted counterweight is humux: one Docker container, MIT-licensed, no telemetry, air-gappable, running a squad — lead, developer, researcher, marketer, QA — coordinating through GitHub issues, each agent its own GitHub App with name and avatar, escalating only what a human labels. Each agent gets its own model, voice, Telegram bot, and memory; the foundation is boring on purpose: Himalaya for IMAP, CalDAV for calendars, Playwright for browsing, faster-whisper and Kokoro 82M for voice, SQLite for storage — "no external databases, no Kubernetes, no account to sign up for." Credentials stay out of model context behind two-tier encryption; skills are taught by writing markdown files. And for the unglamorous middle of developer work, JSON Crack keeps doing what it has since 2022: turning JSON, YAML, CSV, and XML into interactive graphs, generating TypeScript interfaces, Go structs, and Rust serde types, running jq — all locally, nothing stored on any server. (more: https://humux.dev) (more: https://github.com/AykutSarac/jsoncrack.com)