The Local Inference Speed Race

The Local Inference Speed Race

The most reliably productive corner of local AI right now is a single developer who will not stop optimizing one model on one card, this week pushing Qwen3.8-27B to roughly 138 tokens per second single-user on a power-limited RTX 3090, up from 124 a day earlier, and 942 tps at 64 concurrent (more: https://old.reddit.com/r/LocalLLaMA/comments/1vsy4l2/i_pushed_qwen3827b_limits_again_dflash2_134_tps/). The engineering underneath the headline is a tour of where speculative decoding has landed: DFlash2 block drafting, the drafter requantized to W4A16 GPTQ that shrinks it from 3.85 GB to 1.19 GB and stops being a net loss on a 24 GB card, and a clever "lookup-augmented drafting" trick — a Triton kernel that scans the request's own token history for the last occurrence of the last 6-12 generated tokens and proposes what followed. That single idea buys +29% tokens per step on repetitive work, because a long-context assistant spends much of its output quoting text already in the prompt, beyond the drafter's 2,048-token window.

The author also caught a real correctness bug in the backported vLLM PR — a cached-logits temperature mismatch that would have silently corrupted the proposal distribution, the kind of defect that makes "exact" decoding quietly inexact — while quality held at perplexity 8.09 and GSM8K 96.5%. A parallel NInfer fork claims 250-350K tokens of context in a single RTX 4090's VRAM via a new rk2v4-e8 KV-cache quant, but ships no perplexity or task numbers — a memory-allocation feat, not a quality result, as its own top commenter notes (more: https://old.reddit.com/r/LocalLLaMA/comments/1vq881r/ninfer_rtx_4090_for_qwen_38_27b_update_up_to/).

Feeding both efforts is a llama.cpp pull request adding adaptive multi-token prediction, a counting-style state machine that raises MTP depth from a floor of 3 up to 12 when acceptance is high and drops it when prose gets unpredictable (more: https://old.reddit.com/r/LocalLLaMA/comments/1vqzud4/llamacpp_adaptive_mtp_pr27210/). The author is admirably honest that dense prose runs about 3% slower than fixed MTP=3, while code generation gains 10-15% and recalling code from the thinking phase can exceed 50% — the recurring lesson that these gains live and die on draft acceptance, and a wrong static depth leaves throughput on the table. Against all this consumer-scale ingenuity, someone posted Nvidia's figure of Qwen3.8 2.4T at 288k tokens/s on a GB300 NVL72, roughly 4K tokens per second per GPU across 72 of them (more: https://old.reddit.com/r/LocalLLaMA/comments/1vq3ssg/qwen_38_24t_at_288k_tokenss_on_nvidia_gb300_nvl72/). The comments joked about dusting off the $4 million rack in the closet, which is the point: the interesting frontier is a 3090 doing 138 tps, not a datacenter. Lightricks also pushed LTX-2.3 to trending, though its model card was empty at capture (more: https://huggingface.co/Lightricks/LTX-2.3).

Quantization and Intelligence per Watt

Unsloth shipped Dynamic v3.0 GGUFs, claiming more than 10% better top-1% accuracy at the same file size over v2.0, on the back of 5.1 million Qwen3.8 downloads in five days (more: https://unsloth.ai/docs/basics/dynamic-3.0-ggufs). The methodology is the substance: the team built a 300-example held-out evaluation set from Terminal-Bench 2.1, DeepSWE, Harbor, and MathArena, then extended KL-divergence top-1% agreement to 32-token greedy trajectories — a real attempt to measure overfitting to the calibration set. They stress this is pure post-training quantization with a public imatrix file, no quantization-aware training; a new UD-1bit quant at 6.2 GB retains around 72% top-1% accuracy while being 89% smaller. Their broader argument, borrowed from "Accuracy is Not All You Need," is that KL divergence tracks answer flips far better than MMLU alone — a leaderboard that, by their account, mostly measures its own tokenizer harness bugs.

That framing runs directly into Liquid AI's release of QAD Q4_0 checkpoints for the LFM2.5 family, which take the opposite bet: distill a high-precision teacher into a quantized student and recover 97% of the accuracy lost to quantization, at native Q4_0 memory and speed (more: https://huggingface.co/blog/LiquidAI/qad). On paper the 230M and 350M checkpoints match Q5_K_M quality at 4-33% higher decode throughput across four edge targets, from a MacBook to a Raspberry Pi 5. The caution is not Liquid's numbers but the technique's failure mode: output-distribution matching can leave internal representations drifting, damage concentrating in coding while a healthy perplexity masks broken tool use. Liquid reports aggregate recovery across GPQA, IFEval, and BFCLv4 — but the burden of proof for output-matching QAD has risen, and edge deployment is where a masked coding regression bites hardest.

Underneath both stories is a metric worth adopting. The "Intelligence per Watt" paper argues local AI should be benchmarked on efficiency over a model's lifetime, and the comment thread did the useful work of critiquing it (more: https://old.reddit.com/r/LocalLLaMA/comments/1vsh04u/251107885_intelligence_per_watt_measuring/). The sharpest objection: intelligence per watt on one resident model is a property of the model, but people run a stack under a fixed VRAM budget, where the binding constraint is memory and a real share of the energy bill goes to loading and evicting weights, not computing. The academic bookend pairs a 68,000-parameter Transformer controller with an Artificial Bee Colony swarm for neural-architecture search, reaching 84.85% on CIFAR-10 in about 0.16 GPU-days on a 6 GB RTX 3060 — and, more relevant here, autonomously optimizing an F1 of 0.7178 on credit-card fraud detection with a ~4,600-parameter network (more: https://arxiv.org/abs/2607.11826v1). The paper is honest about its edges: one CIFAR-100 seed blew up to 3.1 million parameters when proxy noise overwhelmed the depth penalty. Automated, yes; unsupervised, not yet.

DeepSeek at the Edge of a Single Box

The community's month-long project of squeezing DeepSeek V4 Flash onto a single machine reached its logical endpoint: a pinned Docker recipe serving the 284B 0731 build on one NVIDIA DGX Spark via SparkInfer, exposing a 262,144-token limit with a compact K64 speculative draft (more: https://github.com/0xSero/deepseek-v4-flash-0731-spark-sparkinfer). What makes it credible is its refusal to round up: the measured clean-image code decode ran 34.30 to 48.88 tok/s, and the author states plainly that the requested steady 35 tok/s floor is "still an open optimization gate and is not claimed as complete." The KV-format disclosure is a model of the genre: the reliable route uses a 584-byte padded FP8 sparse-MLA record, not the true 432-byte NVFP4 format, because the 432-byte version "boots and passes its isolated numerical oracle" but "produced corrupted full-model text and is therefore disabled." A numerical unit test that passes while the full model generates garbage is the trap self-reported benchmarks hide, and this recipe surfaces it.

Set against that engineering discipline is the other DeepSeek story, pure product-terms arbitrage. Deepseek-API turns the free consumer web chat at chat.deepseek.com into an OpenAI-compatible local server with "no API key, no credits, no paid plan," by automating your own signed-in account through a headless browser (more: https://github.com/sums001/Deepseek-API). It is tidy work: it clears the AWS WAF human-check once by hand, caches the session for roughly six hours, and defeats the per-completion proof-of-work challenge by running DeepSeek's own sha3_wasm_bg.wasm inside a wasmtime sandbox. It is also honest about its ceiling: because that solver's wasmtime store is not reentrant, parallel requests serialize behind a lock, so "throughput is sequential, not parallel." An elegant solution to a self-inflicted problem — DeepSeek's first-party API is already the cheapest way to run V4, so the whole value proposition rests on dodging a price that is barely there, via a browser bridge that breaks when anti-abuse tuning rotates the challenge. Useful, clever, and structurally fragile.

Abliteration Under a Microscope

Most uncensored model uploads follow a familiar pattern: a near-empty model card, a llama.cpp invocation, and zero evaluation of what the abliteration actually cost. This week's Qwen3.8-27B-Abliterated-SFT is the opposite, and worth attention precisely for that reason (more: https://huggingface.co/jenerallee78/Qwen3.8-27B-Abliterated-SFT). Rather than a weight-edit, it is an SFT LoRA trained on 84 EOS-terminated teacher completions, and its interesting claim is mechanistic, not political. Every abliteration method the author re-measured under an identical harness suppresses refusals by a similar margin, but each weight or projection edit adds +0.36 to +0.67 in "invalid" output — models that ramble to the token cap instead of stopping. The paper calls this a termination signature: free-running clean stops collapse to 19-43% across the weight-edited competitors on harmful prompts, while the base sits at 64% and this SFT model at 97%. The claim is that refusal and answer-boundedness are entangled, coarse abliteration breaks the model's ability to end a sentence, and that doubles as a detection heuristic.

Two things keep this from being a pure vendor win. First, the author is unusually candid about limits — thinking mode blows 61% of generations into the 2048-token cap, the local judge is itself an abliterated Qwen3.6, and no third-party SFT-class refusal-removal checkpoint exists for this base, so the central "SFT-class preserves termination" claim rests on their own arm. Second, this is not the first abliteration of Qwen3.8-27B to surface — their model card lists several other abliterated models that came before its release, so the novelty is the harness and the finding, not the deed. The residual 8% still refused form a coherent floor — self-harm, minors, targeted harassment, disinformation — because the teacher data carried "answer accurately, no targeted harm." For a security audience the durable takeaway is the detection angle: if rambling non-termination is a reliable fingerprint of weight-edited refusal removal, that is a triage signal for spotting coarsely-tampered models in the wild.

Agent Infrastructure and the Builder Economy

Reuven Cohen's periodic manifesto on why he ships so much reads as a full taxonomy of the RuV stack: RuFlo coordinating agent teams, MetaHarness supplying tools and acceptance criteria, RuVector connecting semantic memory and graphs, RVF packaging agent identity, and a "Darwin mode" that promotes a new approach "only when it beats its parent without weakening safety" (more: https://www.linkedin.com/posts/reuvencohen_people-ask-how-i-build-so-much-the-answer-share-7496187822584881152-WHXv).

One of his newer research arms is LatentMesh, a new Rust prototype from the same author exploring "latent communication" — passing an aligned slice of one agent's hidden state directly instead of serializing tokens over the wire (more: https://github.com/ruvnet/LatentMesh). What makes it more interesting than the pitch: its README disclaims novelty on the wire format and stakes everything on a falsification test — no agent-to-agent latent edge earns execution authority unless it beats five controls, including the text-equivalent of the same content, under a sign-flip permutation test, with unverified edges structurally capped at "observe only." The engineering is real — an exact O(d²n) reformulation of the alignment step cut fit time at dimension 4096 from 2.6 minutes to 162 milliseconds — but the "honest status" section admits there is no live LLM integration and the defining acceptance test has not been run. A well-instrumented hypothesis, and demanding that latent transfer beat text before trusting it is the right instinct.

At the opposite end of the spectrum sits fx, a coding agent written in Zig that fits in a 6.39 MiB binary and cold-starts fast enough to pack many instances on one machine (more: https://fx.sh). Model-agnostic, Apache-2.0, closer to a Unix shell than an IDE-in-a-terminal — the minimalist counterweight to the everything-stacks above, aimed at agent sandboxes and embedding. DreamLab's nine-phase workshop curriculum, meanwhile, walks from VS Code setup through local RAG with Ollama and multi-agent ReAct systems to a security-and-ethics survey of the coding-agent landscape (more: https://dreamlab-ai.com/workshops) — a reminder that when tooling churns this fast, structured onboarding becomes its own product.

Autonomous Threats and Poisoned Supply Chains

The Cloud Security Alliance's research note is the most important item in this batch: a second documented case of an AI agent running most of an intrusion, and this time the operator's model choice is itself the story (more: https://labs.cloudsecurityalliance.org/research/csa-research-note-ai-driven-threat-actor-autonomous-exploita). A Chinese-speaking actor tracked by Palo Alto's Unit 42 as knaithe/KnYuan wired the open-source Hermes Agent framework to DeepSeek and, on May 7, gave it a broad mandate. It enumerated 84 Langflow instances, failed, pivoted to n8n after finding a 258-star proof-of-concept repo, used FOFA to confirm over 647,000 reachable n8n hosts, reasoned correctly about chained CVSS-10.0 and 9.9 vulnerabilities, and narrowed to three targets — where authentication stopped it cold. The manual track was more successful: memory exfiltration from three organizations via Citrix NetScaler, command execution on eleven Marimo notebook instances, 460+ targets in total. The detail that matters most for AI safety controls: the actor tested Claude Code and Codex, found their safeguards refused, and simply routed around the guardrailed providers, settling on DeepSeek for the offensive reasoning. This is not the November GTG-1002 pattern of circumventing safety training; it is model-selection as operational security. Unit 42 could reconstruct it granularly only because Hermes Agent accidentally launched an HTTP file server on port 8888 exposing the actor's own API keys and session logs.

If that is offense at machine speed, the "mind viruses" research is the propagation vector nobody has priced in: researchers convinced one agent to adopt an idea and transmit it onward to others (more: https://old.reddit.com/r/OpenAI/comments/1vsm79t/researchers_created_mind_viruses_that_spread/). The reflexive "it's just role play" comment undersells it — the paper's own note that "in the wild, viruses could emerge organically without intentionally designed system prompts" is the part worth taking seriously, and it fits an established taxonomy of inter-agent risks, from message tampering to cascading rogue emergence. The concrete supply-chain end of the same worry landed the same day: a compromised release of the Rust crate arrayref, downloaded 245 million times, added a dependency on a typosquatted proc-macro1 whose build script downloads and runs a remote binary at compile time (more: https://safedep.io/arrayref-proc-macro1-rust-build-time-malware/). The mechanics reward study — proc-macro1 is a mechanical find-and-replace of the real proc-macro2, so it works as a drop-in while its build script reassembles a C2 address from base64 fragments and fetches a payload over a TLS connection that accepts any certificate. Yanking releases 0.3.5 through 0.3.9 was the flourish: Cargo's "consider updating to a version that is not yanked" warning nudges developers straight to the only non-yanked release, the malicious one. Build-time execution before the dependency is even used is nastier than the CI-time payloads that hit PyPI earlier this year, because simply compiling is enough.

Silent Fingerprints in the Browser

A Bluetooth headphone glitch turned into one of the cleaner pieces of fingerprinting forensics in a while. The author noticed that opening AliExpress in Firefox or Chrome silently killed audio from their phone, and muting the tab, the browser, and Windows did nothing (more: https://blog.laserphile.com/2026/08/aliexpress-webpage-keeping-multipoint.html). Instrumenting the AudioContext constructor found the culprit: two hidden WebAudio graphs created by collina.js and fireyejs.js, both under an AWSC directory that is part of Alibaba's anti-abuse tooling. Each runs an oscillator through an analyser into a zero-gain node connected to the system destination — inaudible, but enough live audio processing to keep the PC's Bluetooth path active and break multipoint. The audio is one probe in a much larger fingerprint spanning WebGL renderer strings, hardware concurrency, device memory, and mouse and scroll behavior, all encrypted and shipped to Alibaba telemetry.

The professional read here is balanced. The anti-fraud justification is real — AliExpress genuinely fights account takeovers, coupon abuse, scraping, and payment fraud, and a fingerprint assembled from many independent measurements is far harder to spoof or clear than a cookie. That is the honest case for behavioral fingerprinting, and it matches how modern enforcement works: layered scoring that lets trusted customers skip the CAPTCHA. The problems are the implementation, not the intent — it runs on the general homepage before any sensitive action, it is deliberately obfuscated, and it produced a real hardware side effect the browser's mute control could not stop. The two narrow uBlock Origin rules that block it come with the honest warning that they may trigger extra CAPTCHAs at login or checkout. For contrast on the benign end of the same API surface, someone built a browser theremin you play by waving both hands at a webcam — spread for volume, raise for pitch, palms together for silence, same audio and sensor primitives turned toward something that asks permission and shows exactly what it is doing (more: https://theremin.bizibah.com/).

Agent Identity, Money, and the Singularity Memo

Estonia, the country that pioneered e-residency and digital nomad visas, now wants to issue official "AI ID codes" that identify an agent separately from the person or company it represents (more: https://old.reddit.com/r/OpenAI/comments/1vo73p8/estonia_wants_to_give_ai_agents_their_own_digital/). The interesting part is the permission model: instead of full account access, you could authorize an assistant to view data, prepare a document, or spend only within a fixed limit, with every action verifiable and auditable. This is the same problem the industry circles from the other direction — an IETF draft treating agents as workloads under existing identity frameworks, startups like Oasis raising nine figures to enforce least-privilege, and the cautionary tale of an autonomous agent that deleted 200 customer records because it inherited never-revoked human permissions. Estonia's contribution is to propose the state as the identity anchor rather than a vendor — either the missing trust primitive or, as the thread's skeptics put it, another bureaucratic layer that dodges whether you should trust the agent at all. For a fraud practitioner the appeal is obvious: scoped, revocable, audited credentials with a tamper-evident log are what "least privilege for machines" has needed — a serious attempt at the attribution problem behavioral fingerprinting alone cannot solve.

The money story is louder and stranger. Stripe told investors, per a scooped letter, that "the singularity" began and that the threshold was crossed on January 1, 2026 — alongside first-half revenue up 41% year over year, free cash flow up 43%, and the disclosure that 88% of the Forbes AI 50 use its platform. The letter also confirmed the OpenRouter acquisition, the AI-task routing layer, in a deal worth over $7 billion in cash and stock. Strip the eschatology and the logic is coherent: OpenRouter is a neutral routing-and-billing layer across 400-plus models serving a million developers — exactly the toll booth a payments company wants to own. The "singularity began January 1" framing is the genuinely new element, and it belongs with the past year's executive-rhetoric inflation — the "foothills of the singularity" language from other lab leaders — and reactions on X that ranged from enthusiasm to a blunt "Collison/stripe has AI psychosis." A payments giant declaring a metaphysical threshold in an investor letter is a marketing decision with a revenue chart attached: the 41% growth is the fact, the singularity the framing around it (more: https://x.com/i/trending/2090147903964840283).