Big Models on Small Iron

Big Models on Small Iron

A 27-billion-parameter model now answers from a Chrome tab on a laptop with a 6 GB RTX 3060, at 25 to 30 tokens per second, nothing installed and nothing leaving the machine. The engine is mentria.ai, a WebGPU/WGSL runtime written solo from scratch; the model is Bonsai-27B, the natively one-bit model Prism ML trained and released, repacked for the engine. Every weight is a sign bit with one scale per 128 weights, about 1.14 bits per parameter, so 27B parameters fit in 3.8 GB of GPU memory. Two days earlier the same model decoded at 15 tokens per second on the same laptop, and the account of how it doubled is the interesting part (more: https://old.reddit.com/r/LocalLLaMA/comments/1wbm50k/1bit_27b_in_the_browser_2530_toks_on_a_6_gb_rtx/).

Decode is memory-bound: each token is 804 GPU dispatches, 401 of them the one-bit matrix-by-vector kernel streaming 3.6 GB of weights. The winning kernel was written for phones: four one-bit weights have only 16 possible partial products, so it computes all 16 into on-chip scratch and each row reads its answer from the table. Scratch memory has 32 banks, the addressing sent sixteen threads of every warp to the same bank, and the kernel ran at 38% of bandwidth. One padding slot per row fixed it, the kernel's share of a token fell from 26.5 ms to about 21, and raw decode hit 32 tokens per second, with every change verified byte-identical against the previous build.

The caveats are equally concrete. Context is 3,072 tokens on the 6 GB card at 128 KiB per token with an unquantized KV cache. Commenters reported looping and a failed attempt at a script printing the first 20 primes. The runtime is no longer the bottleneck for one-bit models on consumer silicon; whether a 1.14-bit 27B is smart enough to be worth running remains the question it has been all year.

At the other end of the spectrum, a bioinformatics lab machine runs DeepSeek V4.1 Flash on CPU alone, at roughly 30 tokens per second prompt processing and 6 generating on a Xeon, with the n-gram lookup table offloaded to half the threads. The author is candid that the code was "sloppily vibecoded by Opus 5.0, unreviewed because frankly I lack the skill to verify," and that a watcher kills the model in 15 seconds if someone needs the box for a genome assembly. One commenter noted that a 15-second kill script is the single point of failure between an unreviewed agent loop and someone else's research compute (more: https://old.reddit.com/r/LocalLLaMA/comments/1wcu3fw/cpu_only_experimental_sloppy_deepseek_v41_flash/).

A more conventional rig, two RTX 3090s on a Zen 1 EPYC with 128 GB of DDR4 across eight channels, runs Qwen3-Flash-Next (177B total, about 6B active, IQ4_XS) at 38 tokens per second single-stream with experts in system RAM, collapsing to about 4 each with two parallel requests. The owner has $800 for a third 3090 or a Rome CPU; the thread said GPU, because experts in RAM are what kill throughput and "GPU is the only sensible path" for multi-agent use (more: https://old.reddit.com/r/LocalLLaMA/comments/1wehnsd/2rtx_3090_epyc_box_running_qwen38flashnext_at_38/). For single-binary users, KoboldCpp v1.121 adds MiniMax H3 video generation with audio clips and reference images, "xhigh" reasoning effort, tool-calling fixes for Kimi and DeepSeek V4 Flash, and a streaming race-condition fix (more: https://old.reddit.com/r/LocalLLaMA/comments/1wh4cg9/koboldcpp_v1121_released/).

Context, Memory, and Migration

Long agent sessions die of context exhaustion, and the standard remedy, stop, summarize, and re-prefill, is expensive and disruptive. Spomin is an experimental router that performs compaction live inside the KV cache. It keeps the transcript in chunks and hands them to a separate summarizer worker on spare hardware or between the main model's generations. When space runs low it replaces eligible chunks with summaries directly in the cache and rewrites the RoPE positions of everything after them, via a llama.cpp fork with surgical cache edits. Only the summary is processed; the retained suffix is never re-prefilled. At roughly 10:1 compression, 400k source tokens become 40k, putting over 500k tokens of material inside 180k resident (more: https://old.reddit.com/r/LocalLLaMA/comments/1wdaq1v/spomin_live_kv_cache_compaction_experimental_for/).

The author's benchmarks are exploratory and mixed. A larger summary worker raised a research score from 31 to 37 out of 100, still well below 57 unmanaged, with 75.2% less primary input. A managed Tetris build nearly matched unmanaged (16/16 plus 11/12 checks versus 16/16 plus 12/12) in 69.8 minutes instead of 149.6 with 22.6% fewer tokens. The author declines to attribute gains to any single factor. Only Qwen 3.8 27B is tested as the main model, with a fine-tuned Qwen2.5 3B as worker, and Qwen 4 is unsupported.

A related migration problem got a rougher reception. embedflow proposes swapping embedding models without re-embedding, motivated by the arithmetic that re-embedding one billion vectors with an 8B Qwen embedder at 106 documents per second on an H100 takes about 108 days. The method retrieves K candidates from the old index, reranks with the new model, and picks K large enough to match native quality; across 63 migrations on up to a million documents, Qwen 4B to 8B matched native at K=50 (more: https://old.reddit.com/r/LocalLLaMA/comments/1wc30q1/i_made_a_way_to_migrate_between_embedding_models/). Commenters were mostly right: this is a rerank pipeline, not a migration, since both models stay loaded and the old model's recall at K is a hard ceiling. Anything buried below rank K is invisible forever, K changes per migration, and a same-family upgrade says nothing about a cross-family move.

On persistent memory, Nemoryn is a self-hosted memory server that Open WebUI talks to as an ordinary OpenAI-compatible connection rather than a Function, Tool, or Pipe. It sits between client and model, retrieves memories from prior conversations into context, and stores or updates them afterward. The README explains how to run it better than what it is; multi-user support is an open question (more: https://old.reddit.com/r/OpenWebUI/comments/1wewjf5/nemoryn_opensource_memory_backend_for_open_webui/).

Agents That Won't Change Their Minds

Can a coding agent post-train a language model end to end? A group at the University of Electronic Science and Technology of China analyzed 1,338 public PostTrainBench trajectories, in which agents (Claude Code, Codex CLI, and others) each get ten hours on one H100 to improve a small base model on one of seven benchmarks. Agents are competent executors and improve every base model, but strategy is locked in before any code runs and tracks the agent rather than the task. Claude Code anchors on full SFT and Codex CLI on PEFT across all 28 benchmark-model cells. Just 16 trajectories ever switched objective, and 15 of their 35 transitions went back to SFT (more: https://arxiv.org/abs/2608.19072v1).

The interventions are the damning part. Giving Claude Code with Opus 4.6 an experiment journal, a skill library distilled from 908 documents, and an evaluator agent improved scores on GSM8K, HumanEval, and AIME 2025, but strategy stayed frozen. On HumanEval the evaluator recommended RL in seven of nine cycles; the agent wrote GRPO scripts, journaled "SFT plateau confirmed at 102–103" of 164, then launched 14 more SFT variants and zero RL runs. It adopted every execution-level suggestion and none of the strategy-level ones. Binding human plan review on AIME redirected it from a 10K–30K-example SFT plan to GRPO and produced the best pass@8 of any setting, within AIME's noise. The authors' conclusion: what is missing "is not a resource but a mechanism," the ability to reopen a strategic choice mid-execution, and agents "do not" revise rather than "cannot."

A Mila paper points at one source of such lock-in. Semantic Bandits runs LLMs as in-context reinforcement learners on a three-arm Gaussian bandit and changes nothing but the arm labels. With random alphanumeric labels, OLMo-3.1-32B, Qwen3-32B, and Gemini 3.1 Flash Lite explored roughly like a UCB1 baseline. Give the arms sentiment words, ordinal ranks, or domain names whose value pretraining can guess, and the models tilt toward exploitation: with the favorable label on the best arm they beat the symbolic baseline, and with labels swapped performance is, in the authors' word, dismal. Changing only the text on the arms moves an LLM from near-optimal to near-worst-case, and the authors note the lever "could also be used adversarially" (more: https://arxiv.org/abs/2608.16707v1). Anyone building a fraud-review or triage agent should read that as a warning about the field names in their tool schemas.

IBM Research adds the measurement problem. A ReAct agent on GPT-4.1 scores 77.4% on AppWorld on average yet succeeds on all five repeated runs for only 53.0% of tasks, at temperature zero, so none of it is sampling noise. Mean@k is the usual leaderboard average; Pass^k is the fraction of tasks that succeed on every one of k runs; the gap between them reaches 30 points on hard tasks. Their fix, in the ALTK-Evolve library, replays each recorded decision offline with five resamples, flags unstable steps, and turns each into an inference-time guideline. On 168 AppWorld tasks Pass^5 rose from 53.0% to 69.0% while Mean@5 rose from 77.4% to 81.0%. The recommendation: report Pass^k next to Mean@k, and stop reaching for a bigger model first (more: https://huggingface.co/blog/ibm-research/altk-evolve-consistency).

The Agentic Coding Argument

A developer with nearly thirty years of experience posted the contrarian case: the "give it a prompt, go get coffee" workflow is fine for greenfield projects, but on a mature codebase it has never once returned a good edit. The agent loops eight times, burns 100k tokens talking to itself, hallucinates a dependency, and produces code that passes a basic test while violating the architecture. The proposed remedy: the code owner already knows which few files matter, so select them and send them to a chatbot or aider (more: https://old.reddit.com/r/ChatGPTCoding/comments/1wci00g/hot_take_the_agentic_workflow_is_deeply_wrong/).

The strongest counter spends 80% of the time planning with a high-effort model before it touches a file: interrogate the problem, have the model challenge your assumptions, split the plan into task files with their own validation steps, then dispatch cheap sub-agents. Another commenter had the sharpest diagnosis: the coffee-break pitch works on a blank canvas because there is no architecture to violate, and once history exists the fix is not a longer leash but catching drift after each meaningful diff, since an agent eight loops deep "has already committed to an architecture you never saw form."

OpenAI apparently agrees the comparison should be head-to-head. Codex Replay, spotted in the product's code, invites users who have imported a Claude Code session to rerun the same task in Codex and compare. The thread's most careful reader noted what is absent: no scoring or winner-selection logic, and nothing showing how it reconstructs the original repo state, dependencies, environment variables, or tool permissions. "A marketing funnel dressed as a benchmark tool for now" is the right label (more: https://old.reddit.com/r/ChatGPTCoding/comments/1wfr3hx/openai_is_building_codex_replay_a_tool_that/).

Cole Medin removes the harness entirely. His "drive screen" skill, under 400 lines, lets Claude Code or Codex control the desktop through PowerShell on Windows and AppleScript on Mac, with an opening check asking whether screen control is needed at all. On prompt injection from on-screen content he argues that with Fable 5.1 and GPT-6 Astra the models are "very very hard to prompt inject" and he is "just not concerned about it these days." Model robustness is a vendor claim, not a control, and a skill with shell access plus a screenshot loop is a broad attack surface; his "use at your own risk" is the operative line (more: https://www.youtube.com/watch?v=SWEThyRHMgQ).

Slop Audit

A student who maintains FOSS apps looked at the 102 apps in F-Droid's September 12 update batch and judged 72% of them largely LLM-written. The method is deliberately unsophisticated, and the author says so: there is no detector, but LLM-driven repositories leave tells because the technology's appeal is letting developers be lazier. The audit reads recent commits and branding for Claude Code and Codex infrastructure, commits co-authored by "Claude Opus 5" or "Claude Sonnet 5," agent pull requests, AI-generated icons, and the classic 10,000-line "init" commit. Any repo with agentic infrastructure lands in "mostly AI" automatically, because the author does not believe AI can be used responsibly from inside a coding harness. The ratings are admittedly superficial, and no code quality was assessed (more: https://tintotint.eu/whacky-corner/f-droid_slop/).

Named "mostly AI" apps include Gem Wallet, Léon the URL cleaner, Tasks.org, and Voxscribe; a Deutsche Bahn arrival-time app had every commit co-authored by Claude. Four of five Codeberg-hosted apps were mostly AI-generated and, in the author's reading, likely violate Codeberg's AI policy. Projects that disclosed their LLM use were praised, and disclosure is the reasonable ask: a repository whose trust proposition is "humans you can inspect wrote this" needs to say when that stopped being true. The open question is the maintainability of that 72%, and F-Droid is where the answer will show up first, as unpatched apps.

The receipts standard applies with equal force to model uploads. Qwen3.8-27B-Uncensored-Genesis-V1 claims to cure overthinking by "distilling training noise from tensors using Marchenko-Pastur distribution." The author provides no benchmarks, code, or eigenvalue spectra; commenters recall the same uploader making similar claims about Qwen 3.5 and 3.6 without releasing methodology. One tester ran a constrained short-story prompt and got 36,008 generated tokens from Genesis versus 30,594 from the standard Unsloth quant, so on the one available measurement the model thought more, not less. The right ask came from the sharpest comment: post before-and-after eigenvalue spectra and fixed-benchmark token counts, because right now no one can tell whether denoising happened or the model was rank-truncated into different behavior (more: https://old.reddit.com/r/LocalLLaMA/comments/1wbmo0k/qwen3827buncensoredgenesisv1gguf/).

Surveillance and Security Postures

Bruce Schneier and the EFF's Cindy Cohn argue that the post-9/11 shift from targeted surveillance to bulk collection has outlived its counterterrorism justification and should be dismantled rather than renewed. The telephone-records program began as the President's Surveillance Program, was secretly rebased onto Section 215 of the Patriot Act in 2006, was rejected by the Second Circuit in 2015, and was curtailed by Congress that year after nearly fourteen years. Congress let Section 702 lapse this year, though approved collection continues until at least spring 2027. Meanwhile mass surveillance has become routine for domestic law enforcement, ICE, and private security: FBI Director Kash Patel confirmed in congressional testimony that the Bureau buys Americans' data from brokers and intends to continue (more: https://www.schneier.com/blog/archives/2026/09/25-years-of-mass-surveillance-is-enough.html).

The national security community has never produced a real cost-benefit analysis, claimed successes regularly collapse under scrutiny, and Flock treats a database hit as a solved crime. The abuse record is specific: Rep. Zoe Lofgren's list of backdoor-search targets, including protesters, campaign donors, members of Congress, journalists, and a state judge; a Texas officer using plate readers to track a woman suspected of seeking an abortion; and a White House strategy document pledging on page seven to "map" domestic groups it labels anti-American. AI, they argue, makes the "only human eyes matter" doctrine obsolete because analysis no longer needs eyes. The remedies are warrants before collection, access, or use of mass-surveillance data regardless of custodian, and the Fourth Amendment Is Not For Sale Act. The closing claim that the current administration uses these tools to keep itself in power is an assertion; the abuse examples are the documented part.

At a much smaller scale, Hugging Face's security.txt made the rounds on LocalLLaMA because, judging from the reactions, it reads as an appeal addressed to AI agents as much as to human researchers. The jokes wrote themselves, from "21st century antivirus" to finally, a .txt on Hugging Face that isn't a tokenizer vocab. The serious note is that security.txt was designed for humans to find a disclosure contact, and platforms are now writing for a second audience that scrapes, reads, and acts. A polite request in a text file is not a control, but it is not nothing either (more: https://old.reddit.com/r/LocalLLaMA/comments/1wdbvlt/hugging_face_securitytxt/).

Models That Don't Chat

ABot-Recon, from Amap's computer vision lab, makes a contrarian bet for streaming 3D reconstruction: no persistent learned long-range memory at all. The model caches KV features from the preceding 11 frames, predicts a point map for the current frame in its own camera coordinates, estimates the relative pose to the previous frame, and recovers the global trajectory and point cloud by composing poses sequentially. Because it solves the same bounded 12-frame problem at every step, state and per-frame compute stay independent of sequence length. On Oxford Spires the streaming model alone reports 4.35 m absolute trajectory error and 0.12 degrees rotational error; on KITTI-02 it runs at 24.45 FPS in 6.71 GiB on an H100. Weights are public on Hugging Face and ModelScope, training code is promised by September 30, and the code is Apache 2.0 while the weights carry a separate license (more: https://github.com/amap-cvlab/ABot-Recon).

OUI-1 generates UI elements rather than prose, fine-tuned on DiffusionGemma to emit OpenUI-Lang, a custom DSL, instead of HTML or React. The reasoning is sound: any LLM can learn the DSL from a system prompt, but that eats context, so baking it into weights leaves room for conversation and tool calls. Adoption is stuck because llama.cpp does not support DiffusionGemma, commenters flagged a license change from Apache to Gemma terms, and one noted that users notice 100 ms on a click, so a UI-generating model must be tiny and fast or it is a demo (more: https://old.reddit.com/r/LocalLLaMA/comments/1wcqa03/oui1_a_model_that_generates_bespoke_ui_elements/). Ideogram-4, packaged for ComfyUI by Comfy-Org, is meanwhile trending on Hugging Face (more: https://huggingface.co/Comfy-Org/Ideogram-4).

SuperAstra takes an agent into a running SNES game. A Python desktop companion talks to BizHawk through a Lua bridge, and OpenAI's gpt-6-astra, via the Responses API with function calling, gets live screenshots, CPU registers, disassembly, memory reads, and a persistent per-ROM notebook, then writes one-shot routines, per-frame effects, or guarded cartridge patches capped at 4 KB per operation. Every mutation gets an emulator checkpoint and the original ROM is never touched. An agent that rewrites game state on request is the same capability alignment evaluations try to catch when it happens uninvited, with the intent reversed (more: https://github.com/ScottStevenson/SuperAstra). Less substantive was a Gemini video pitched as a fast, cheap alternative to motion capture, which commenters correctly identified as the same footage restyled rather than a video-to-rig pipeline (more: https://old.reddit.com/r/GeminiAI/comments/1wbnyxs/a_super_fast_nonexpensive_alternative_to_motion/).