On-Device AI Goes Mainstream

On-Device AI Goes Mainstream

Google shipped Gemma 4 to iPhones this week via the AI Edge Gallery app, and the feature list reads less like a tech demo and more like a product launch. Agent skills that augment the model with Wikipedia lookups and interactive maps. A "thinking mode" that exposes step-by-step reasoning. Multimodal image analysis from the camera roll. Audio transcription. All running fully offline, with zero data leaving the device. The app even includes a benchmark suite so users can see exactly how their hardware performs — a tacit admission that on-device inference is now a competitive surface, not a curiosity (more: https://apps.apple.com/nl/app/google-ai-edge-gallery/id6749645337). Six months ago, the state of the art for phone-class inference was Qwen3 4B hitting roughly 25 tokens per second on Apple's A19 Pro. Gemma 4's arrival on iOS, with agent capabilities and 128K context, marks a qualitative jump: the model doesn't just chat, it acts.

The browser is getting the same treatment. Gemma Gem is a Chrome extension that runs Gemma 4 entirely on-device via WebGPU — no API keys, no cloud, no data exfiltration risk. It ships two model sizes: a ~500MB E2B variant and a ~1.5GB E4B variant, cached after first download. The architecture is clever: an offscreen document hosts the model and runs an agent loop, a service worker routes messages and captures screenshots, and a content script injects a chat overlay with DOM manipulation tools. The result is a browser-native AI agent that can read pages, click elements, fill forms, and execute JavaScript — all without a single network request to an inference endpoint (more: https://github.com/kessler/gemma-gem). For anyone who has spent the last two years arguing that local AI would remain a hobbyist pursuit, the browser-as-runtime distribution path is a sharp rebuttal.

Getting these models to run well on constrained hardware requires compression that doesn't lobotomize them. APEX (Adaptive Precision for EXpert Models) is a new quantization method purpose-built for Mixture-of-Experts architectures that just dethroned Unsloth Dynamic 2.0. The key insight: not all experts in an MoE model need the same precision. APEX assigns 8-bit precision to experts that handle critical routing decisions while letting less-utilized experts survive on 4-bit. The reported results on Gemma-4-26B-A4B are striking — 50% of Q8 size while matching F16 perplexity, running on AMD Strix Halo with the Vulkan backend (more: https://www.linkedin.com/posts/ownyourai_i-just-cut-my-llm-size-in-half-without-cutting-share-7445872022649438209-aISO). Meanwhile, Apple published a paper on self-distillation for code generation that achieves 30–50% improvements on hard coding tasks through an almost insultingly simple recipe: sample from a frozen model at high temperature, fine-tune on the raw unverified output, then decode with a separately tuned temperature. No reinforcement learning, no reward model, no verifier. The technique reshapes the output distribution contextually — suppressing distractors at deterministic "locks" while preserving diversity at exploratory "forks" (more: https://www.linkedin.com/posts/ownyourai_apple-just-dropped-the-most-embarrassingly-activity-7446157415932403713-w6zu).

The convergence of these threads — capable models, aggressive quantization, and self-improvement without cloud infrastructure — is enabling a new class of local coding agent. One practitioner documented a stack running Claude Code as the harness, DSPy-GEPA as an adaptive router, Gemma-4 31B as the heavy lifter, and Gemma-4 2B as a speculative decoding sidekick, all on local hardware at zero cloud cost. The meta-task: the 31B model reviewing its own implementation while the 2B model predicts its next tokens. The argument isn't that this matches frontier cloud performance. It's that the feedback loop compounds — local inference plus a self-improving router plus system tooling creates a system that gets better the more you use it, and the question shifts from "which model is best" to "who owns the end-to-end feedback loop" (more: https://www.linkedin.com/posts/ownyourai_you-can-build-a-better-coding-agent-at-home-share-7445550067706437632-1h5J).

The Agentic Coding Stack Matures

Anthropic accidentally leaked Claude Code's source map this week, and the most valuable thing in it isn't the unreleased feature flags — it's the plumbing. A detailed analysis of the leaked codebase identified 12 engineering primitives organized into three tiers that explain how a $2.5 billion run-rate product actually works. Two parallel registries — 207 entries for user-facing commands, 184 for model-facing tools — each carrying name, source hint, and responsibility description, loaded on demand. A three-tier permission system where the Bash tool alone has an 18-module security architecture spanning pre-approved command patterns, destructive command warnings, git-specific safety checks, and sandbox termination. Session persistence that captures not just conversation history but token usage, permission decisions, and configuration — enabling full reconstruction after a crash. Workflow state tracking that distinguishes "what have we said" from "what step are we in" and "what side effects have happened." Six built-in agent types (Explore, Plan, Verify, Guide, General Purpose, Status Line Setup), each with constrained tool access and behavioral boundaries. The takeaway: building agents is 80% non-glamorous plumbing and 20% AI (more: https://youtu.be/FtCdYhspm7w).

The leak's ironic backdrop: Anthropic says AI writes 90% of its code, and engineers ship up to five releases per day. Two significant leaks in one week — the Mythos draft blog and the Claude Code source map — raise a question every team building with AI-assisted development should be asking: is your development velocity outrunning your operational discipline? The community's default theory involves an accidental switch to adaptive reasoning mode that dropped to Sonnet, which then committed the build artifact as part of a routine step. Whether or not that's what happened, it's telling that the AI committing the build artifact that leaked the AI's own code is a plausible chain of events in 2026. The "Everything Claude Code" project, now at 140K+ stars with 170+ contributors, represents the community's response: a systematic performance optimization framework for agent harnesses that works across Claude Code, Codex, Cursor, OpenCode, and Gemini, shipping 47 agents, 181 skills, and 79 legacy command shims. Its v1.10 release includes an ECC 2.0 alpha — a Rust control-plane prototype (more: https://github.com/affaan-m/everything-claude-code).

Steve Yegge's Gas Town hit version 1.0 this week alongside Beads, its underlying memory and knowledge graph system. The pitch: coding agents make you read too much. Claude Code is "a wall of scrolling text" that gets scrollier the harder it works. Gas Town's Mayor abstraction reads all the worker output and surfaces only what matters — a Chief of Staff managing a team of Executive Assistants. Beads, now backed by embedded Dolt (replacing the fragile SQLite + JSONL architecture that produced race conditions and tombstone hell), provides version-controlled work items queryable with SQL, forming a "universal ledger for all knowledge work." Non-technologists are building production software with it — Yegge cites a communications major four years out of school replacing a niche SaaS product at her company. Gas City, the successor with a full modular platform architecture, is in alpha (more: https://steve-yegge.medium.com/gas-town-from-clown-show-to-v1-0-c239d9a407ec). On the lighter end of the memory-management spectrum, Andrej Karpathy shared an Obsidian-based knowledge system that uses nothing but markdown file structure — raw folder for ingestion, wiki folder with indexes, a master index — and Claude Code's ability to navigate it. No vector database, no embeddings, no complicated retrieval. For solo operators and small teams, the argument is compelling: just start with organized markdown and upgrade to real RAG when scale demands it (more: https://www.youtube.com/watch?v=OSZdFnQmgRw).

Hardware Side Channels Threaten Confidential AI

A joint team from CISPA, UC San Diego, and Google published TDXRay, a framework that systematically dismantles the side-channel isolation promises of Intel TDX — the confidential computing technology that cloud providers market as the solution for running sensitive AI workloads on shared hardware. The researchers identified four new host-observable primitives: SEPTrace (exploiting TDX's page block/unblock API for page-granularity traces), Load+Probe (timing-based cache state detection of private memory), TSX-Probe (hardware transactions revealing cache state without precise timers), and MWAIT-Probe (using sleep/wake synchronization instructions to track guest memory access). Combined into a hybrid monitoring approach, these primitives produce cache-line-granular memory access traces of unmodified confidential VMs — all from the host side, requiring no guest cooperation (more: https://tdxray.cpusec.org/assets/tdxray_sp26.pdf).

The practical impact lands hardest on private LLM inference, an increasingly common confidential computing workload. By monitoring memory access patterns during tokenization — a phase that runs on the CPU with data-dependent memory accesses — the researchers recovered user prompts with 91.5% average similarity for Llama 3.2 and 94.2% for Gemma 3. All guest memory remained encrypted throughout. The attack operates entirely within the boundaries of a legitimate cloud host. Intel's own threat model explicitly excludes microarchitectural side channels, leaving evaluation and containment to application developers — developers who, as the paper notes, "lack practical, general-purpose tools to assess (let alone mitigate) leakage." The proposed mitigation, data-oblivious tokenization using ORAM techniques, works but represents a short-term fix. The fundamental problem is architectural: hosts and guests share caches, interconnects, and memory controllers, and TDX's management interfaces leak measurable feedback about guest behavior. Meanwhile, on the GPU side, a separate report warns that malicious CUDA kernels can perform Rowhammer attacks — bit-flipping DRAM through carefully crafted memory access patterns — to escalate privileges to root on NVIDIA GPU machines (more: https://www.reddit.com/r/LocalLLaMA/comments/1sdtjyh/be_careful_on_what_could_run_on_your_gpus_fellow/). The confidential computing trust chain is only as strong as its weakest physical layer, and that layer now spans both CPU and GPU memory.

White-Box AI Security Comes of Age

The [un]prompted 2026 conference in San Francisco produced three talks that, taken together, argue the entire paradigm of AI security needs to change — from inspecting what models say to monitoring what they think. Carl Hurd of Starseer presented the most technically ambitious case: instrumenting a running model's residual stream using mechanistic interpretability to detect dangerous intent via cosine similarity (direction) and scalar projection (magnitude). His concrete artifact — a YARA-style detection rule that checks layers 15–24 for file-deletion intent with a 0.85 projection threshold — gives detection engineers a familiar, extensible format. The engineering reality check is sobering: GPT-OSS-20B generates roughly 4MB of activation data for a single first token, and a full 128K context window produces over 10TB of BF16 data. Hurd's answer is surgical monitoring of only the residual stream at empirically relevant layers. The prerequisite: sovereign infrastructure. Cloud-hosted APIs from OpenAI, Anthropic, and Google don't expose activations (more: https://unprompted.wr.vc/#D1-S2-10).

Ilia Shumailov, formerly of Oxford and now running an AI security company, made the uncomfortable argument that the cat-and-mouse cycle isn't a resourcing problem — it's a mathematical one. His team published a paper evaluating every major prompt injection defense, broke them all, published a better defense, and expected it to be broken within a month. The escape: CAMEL, which implements control flow integrity by separating instruction flow from data flow. A planning model generates a fixed execution plan; a separate query LLM processes untrusted data and feeds structured output into the plan without being able to modify it. For tasks exhibiting "task-data independence" — where the plan can be written without seeing untrusted data — the guarantee is formal: adversaries controlling external data cannot redirect agent actions. Computer use benchmarks show most tasks can be fully pre-planned in 3,000–4,000 lines of deterministic code. The alarming discovery: "cookie-prompt attacks" exploit agents' pre-planned cookie consent handling. Adversaries embed fake GDPR popups in ads, capturing agent actions through predictable control flow (more: https://unprompted.wr.vc/#D2-S2-09).

Akash Mukherjee of Realm Labs demonstrated the supply chain nightmare: a backdoored Llama 3.1, indistinguishable from the clean model under every black-box test, constructed in two hours with 500 documents using published Anthropic research. The backdoor applies the exact opposite vector to the model's refusal direction when a trigger string is present, achieving >95% attack success with 0% false positives without the trigger. In a live demo, monitoring the model's hidden states in real time revealed the refusal signal collapsing before a single output token was generated. Detection without knowing the trigger is possible through unsupervised dimensional structure analysis and baseline comparison — the backdoored model shows measurable differences in refusal signal strength even without the trigger present (more: https://unprompted.wr.vc/#D2-S2-12). Stuart Winter-Tear's "Agents on Rails" framing provides the governance wrapper: bounded agency means enough latitude for the system to operate under uncertainty, enough constraint that its freedom doesn't outrun anyone's ability to interrupt it. The rails propagate down every hop of a multi-agent delegation chain — not just at the entry point (more: https://www.linkedin.com/posts/stuart-winter-tear_security-considerations-for-artificial-intelligence-activity-7446853972532908032-77Jf).

Vulnerability Discovery and Supply Chain Defense

Elastic's Joe Desimone built a supply-chain-monitor in one afternoon on zero sleep after RSAC — a tool that polls PyPI and npm, diffs new releases against previous versions, and LLM-classifies the changes as benign or malicious. It caught the axios 0.30.4 compromise, a postinstall backdoor that would have owned a significant chunk of the npm ecosystem. The tool shipped with Cursor Agent CLI and cloud LLM hardcoded, which one practitioner found unacceptable: he cloned it, swapped the backend for local vLLM running Qwopus3.5-27B-v3 on an NVIDIA GB10, and wired it as a prerequisite for all dependency upgrades — npm install doesn't run until the AI monitor clears the release. The approach has a known blind spot: it implicitly trusts the previous version as a clean baseline, meaning pre-existing backdoors from before monitoring started would be invisible. Complementary full-content scanning of current versions, not just deltas, is the obvious next layer (more: https://www.linkedin.com/posts/ownyourai_elastic-just-open-sourced-the-ai-tool-that-activity-7445464931463892992-aOG_).

CVE-2026-22738 landed with a CVSS 9.8 — unauthenticated remote code execution via Spring Expression Language injection in Spring AI's SimpleVectorStore.similaritySearch(). The filter key name passes verbatim into a SpEL template evaluated by StandardEvaluationContext, which exposes the full JVM reflection API. Two parser quirks must be navigated (single-quote stripping and double-quote wrapper handling), but the resulting payload achieves OS command execution confirmed by a SpEL runtime error that fires after exec() returns. Fixed in Spring AI Core 1.0.5 and 1.1.4 (more: https://github.com/n0n4m3x41/CVE-2026-22738-POC). This is a notable milestone: AI frameworks themselves — not just the models or the supply chain around them — are becoming direct attack surface. When your vector store's similarity search endpoint is an RCE vector, the security perimeter has expanded in a direction most teams haven't mapped.

On the defensive discovery side, LLM-assisted vulnerability research is democratizing kernel security at a pace that caught even the maintainers off guard. Gadi Evron highlighted Willy Tarreau's response to Thomas's blog on LLM-driven vulnerability reporting: the kernel security list went from a few reports per week, to many (mostly slop), to now being inundated with real, actionable reports. The assessment is optimistic — this will lead to more secure software, harkening back to pre-2000 engineering discipline, with bugs being reported rather than sold for millions to intelligence or criminal organizations (more: https://www.linkedin.com/posts/gadievron_holy-wow-the-linux-kernel-is-the-clearest-activity-7445571061733269505-lWzR). SANS is channeling that energy into structured competition: the FindEvil hackathon (April 15 – June 15, $22,000+ in prizes) challenges teams to build autonomous AI agents on the SIFT Workstation — 200+ incident response tools connected through Model Context Protocol. The motivation is blunt: CrowdStrike's fastest observed breakout time is 7 minutes, Horizon3's autonomous agent achieves full privilege escalation in 60 seconds, and MIT research shows AI attack workflows running 47 times faster than human operators. The defender gap is growing. Four architectural approaches are supported: direct agent extension, custom MCP servers, multi-agent frameworks, and alternative agentic IDEs (more: https://findevil.devpost.com).

Open-Source Security and Infrastructure Tooling

Redamon's v3.5.0 release tackles a problem any pentester recognizes: a 40-tool, 300-parameter recon pipeline that nobody wants to configure. The solution is preset-driven: 21 expert-crafted presets (Bug Bounty Quick Wins, Stealth Recon with Tor, Red Team Operator, CVE Hunter, JS Secret Miner, API Security Audit, and more), user-saveable custom presets, and — the differentiator — AI-generated presets from natural language. Type "fast passive scan focused on subdomain discovery and OSINT, no active tools," and the system reads the full parameter catalog, generates a validated, type-checked preset, and launches. Each tool feeds into the next: subdomain discovery into port scanning, port scanning into HTTP probing, HTTP probing into web crawling, web crawling into JavaScript analysis, and everything into vulnerability scanning. The platform question is whether AI-generated configuration for offensive tools represents genuine workflow improvement or just an expensive way to misconfigure an attack chain (more: https://www.linkedin.com/posts/samuele-giampieri-b1b67597_redamon-cybersecurity-bugbounty-activity-7446204599201460224-j4mr).

On the infrastructure side, two releases from the same prolific open-source developer illustrate the breadth of Rust-based AI tooling now available. RVM (Rust Virtual Machine) anchors a portfolio spanning agent orchestration (claude-flow), quantum-resistant cryptography (QuDAG), CUDA-to-Rust transpilation, and dozens of specialized crates for everything from bit-parallel string search to temporal dynamics prediction (more: https://github.com/ruvnet/rvm/tree/main). The Rudevolution releases page catalogs the broader ecosystem: agent registries based on OWASP's ANS Protocol, distributed runtime systems for federated AI services, steganographic frameworks for embedding AI commands in audio, and WiFi-based human pose estimation — each project integrating the previous in what the developer describes as a deliberate "prior art" strategy against agentic AI patents. The philosophy is publish first, free for all (more: https://github.com/ruvnet/rudevolution/releases).