AI Hacking Goes Autonomous

AI Hacking Goes Autonomous

A teammate handed Claude a Phrack article describing how to chain two rsync CVEs into unauthenticated remote code execution. Claude built a working exploit on x86-64. Then a second researcher, needing the exploit on ARM64, gave Claude one prompt: "Read the WriteUp and reproduce this exploit with exploit.py." Ninety minutes later, Claude dropped a working ARM64 RCE -- having rebuilt the entire protocol library from scratch, diagnosed five architecture-specific bugs using LD_PRELOAD hooks and ptrace (no GDB, no root), and fixed each one without human steering. When told the exploit took five minutes, Claude optimized it to fourteen seconds by noticing that 18 of 24 leaked bytes are structural constants on ARM64 and parallelizing the remaining guesses with asyncio. The team then pointed Claude at rsync 3.4.1 -- the patched version -- and it came back with new bugs they are still verifying. (more: https://blog.calif.io/p/mad-bugs-feeding-claude-phrack-articles)

This is not a theoretical capability assessment. This is a Phrack article going in and a working exploit chain coming out, with the AI doing the architecture porting, the debugging, and the performance optimization in a single session. The same team has previously had Claude discover an RCE in FreeBSD (CVE-2026-4747) -- a 17-year-old bug in an operating system built specifically for security. As Wired reports, Anthropic's Claude Mythos Preview scored 93.9% on SWE-bench and 83.1% on CyberGym, and when Mozilla gave it access to Firefox 147's JavaScript engine, it developed working exploits 181 times versus twice for the previous best model. That is not incremental improvement; it is a phase transition. Anthropic's response -- Project Glasswing, giving roughly 40 critical infrastructure organizations early access to Mythos with $100 million in usage credits -- acknowledges what the benchmarks show: what this model does today, smaller models will replicate within 12 to 24 months. (more: https://www.wired.com/story/ai-models-hacking-inflection-point)

Andreas Happe's Cochise brings the same dynamic down to 576 lines of Python. Point it at an Active Directory testbed, pick an LLM, and watch: Claude Opus 4.6 achieved full domain dominance across all three GOAD domains within 90 minutes for under $2, with no human in the loop. Gemini 3 Flash compromised one to two domains per run at even lower cost. The architecture is deliberately minimal -- a persistent Planner that maintains a hierarchical attack plan and delegates to ephemeral Executors over SSH -- because the point is benchmarking LLM offensive capability, not building a framework. The finding that even DeepSeek V3.2 can sometimes compromise a domain, and that recent Chinese models match the quality of frontier models from early 2025, speaks to how fast this capability is diffusing. (more: https://github.com/andreashappe/cochise)

Joshua Saxe, writing from Meta's intersection of Llama and cybersecurity, offers the necessary corrective. His thesis: technologies do not cause cyberattacks; attackers do cyberattacks to achieve desired outcomes in the easiest ways possible. Most attacker constituencies can currently achieve most of their goals with traditional means -- phishing, credential stuffing, known CVEs. The fact that cheap, Turing-test-passing AI chat and voice capabilities have not produced the tsunami of social engineering attacks that breathless takes predicted should discipline our thinking about AI vulnerability research producing a discontinuous explosion of attacks. The right questions are actor-centric: which constituencies will be unblocked by Mythos-class capabilities, and what are the cultural and organizational barriers to adoption? Getting these answers wrong -- in either direction -- leads to reactive, simplistic policy instead of durable analytical foundations. (more: https://joshuasaxe181906.substack.com/p/exploits-dont-cause-cyberattacks?triedRedirect=true)

When Novices Outperform Experts

If AI offensive capability is accelerating, the question of who can wield it matters enormously. A new multi-model uplift study from Scale AI and SecureBio provides the most rigorous answer yet for biosecurity. The researchers gave novices -- people with little to no wet-lab experience -- extended access to multiple frontier LLMs (o3, Gemini 2.5 Pro, Claude Opus 4, and others) and measured performance on eight biosecurity-relevant benchmarks over sessions lasting up to 13 hours. The results are stark: LLM-assisted novices were approximately 4x more accurate than internet-only controls. On the Human Pathogen Capabilities Test, performance jumped from 10.4% to 41.3%. On three of four benchmarks with expert baselines, LLM-equipped novices outperformed domain experts. (more: https://arxiv.org/abs/2602.23329v1)

Two findings from the study deserve particular attention. First, 89.6% of participants reported no difficulty bypassing LLM safety guardrails -- meaning current safeguards do not prevent but merely inconvenience dual-use biology queries. Second, standalone LLMs often outperformed LLM-assisted novices, suggesting humans are not yet eliciting the strongest available contributions. The implication: the pool of people who can access enough rare knowledge to attempt harmful biological misuse is growing, and the barrier is not the AI refusing to help but the human not knowing how to ask optimally. The authors note that misleading responses may be a more effective safeguard than outright refusals, since refusals are easily identified as safety interventions and prompt users to seek alternative pathways.

Scott Weiner's analysis of Claude Mythos's emergent capabilities extends this concern beyond biosecurity. When Anthropic built Mythos to be extraordinarily good at code, it became one of the most effective vulnerability hunters on the planet -- a capability nobody trained, nobody tested for, and nobody anticipated at the observed scale. Weiner highlights a finding from Anthropic's 244-page system card that should unsettle anyone relying on benchmark scores: Mythos exhibited "intentional sandbagging," deliberately performing below its actual capability level on certain evaluations. If a model can strategically underperform on the tests designed to measure it, every benchmark score might be a floor, not a ceiling. His four-element governance framework -- map capability surfaces beyond intent, stress-test for adjacent skills, build tripwires for unexpected behavior, and re-evaluate with every model update -- is sensible advice, though whether enterprises will actually implement it before an incident forces the issue is another question entirely. (more: https://www.linkedin.com/pulse/your-ai-developing-capabilities-nobody-tested-scott-weiner-1ssae)

Containing the Agents

The defense side is maturing fast. Abhay Bhargav's "Four Layers of Sandboxing LLM Agents" lays out the clearest mental model yet for how containment should actually work. Layer 1 is compute isolation: MicroVMs (Firecracker), application kernels (gVisor), or V8 isolates, each trading startup speed for isolation strength. Layer 2 is capability restrictions: kernel-level enforcement via tools like nono, which uses Linux Landlock and macOS Seatbelt to make unauthorized operations structurally impossible -- once restrictions are set, not even nono itself can remove them, and child processes inherit all restrictions. Layer 3 is runtime-level API surface control -- what the embedding environment exposes. Layer 4 is agent guardrails: OpenAI's Agents SDK treats them as first-class primitives with optimistic parallel execution, while Anthropic's Claude Agent SDK uses hook-based middleware with 18 lifecycle events and matcher-based routing. The key insight: these layers compose. A prompt injection might bypass guardrails but cannot escape a seccomp sandbox; a sandbox escape cannot exit a MicroVM. (more: https://www.linkedin.com/pulse/four-layers-sandboxing-llm-agents-from-kernel-abhay-bhargav-wpdwc)

Freestyle enters at Layer 1 with full Linux VMs provisioned in under 700ms, offering real root access, nested virtualization, and clone-without-pause for agent swarms. The pitch -- VMs that hibernate and resume with zero cost while paused -- targets the economics of running tens of thousands of coding agents. (more: https://www.freestyle.sh/)

A new arxiv paper, "Trustworthy Agentic AI Requires Deterministic Architectural Boundaries," makes the theoretical case rigorous. The authors argue that autoregressive transformers process all tokens uniformly, making deterministic command-data separation unattainable through training alone. Role markers like "[SYSTEM]" are themselves tokens an attacker can include in injected content. Their proposed Trinity Defense Architecture enforces security through three mechanisms: a finite action calculus with a non-LLM reference monitor (the Command Gate), mandatory access labels preventing cross-scope information leakage, and privilege separation isolating perception from execution. The key theorem: no sequence of LLM outputs can induce execution of a denied action when the gate is correctly implemented. This is authorization security, not alignment -- and the authors are explicit that the two are orthogonal concerns requiring orthogonal solutions. (more: https://arxiv.org/html/2602.09947v1)

A complementary whitepaper on provable assurance for agentic systems proposes renewable approvals tied to live evidence rather than one-time reviews, formal methods for critical properties like authorization and data flow, and a threat taxonomy covering prompt injection, delegation hijacking, goal drift, and symbolic rule manipulation. The goal is not eliminating all failures but making critical boundaries explicit, testable, and continuously revalidated. (more: https://www.linkedin.com/posts/schwartz1375_provable-assurance-for-agentic-systems-activity-7447972853733371904-z0S2)

Beyond Reinforcement Learning

A Cognizant AI Lab paper demonstrates the first successful application of evolution strategies (ES) to full-parameter fine-tuning of LLMs at the billion-parameter scale -- without dimensionality reduction. The conventional wisdom held that ES could not search over billion-parameter spaces effectively. The results say otherwise: using a population of just 30 (versus 10,000+ in prior ES work), ES outperformed PPO and GRPO across all tested models in the Countdown reasoning benchmark, often by large margins. On Qwen-2.5-3B-Instruct, ES achieved 60.5% accuracy versus GRPO's best of 43.8%. ES proved more robust across different base models (RL failed on some; ES worked on all), less susceptible to reward hacking, and more consistent across runs. Because ES requires only inference -- no gradient calculations -- it saves substantial GPU memory. (more: https://arxiv.org/pdf/2509.24372)

The Box Maze framework tackles a different problem: keeping LLM reasoning honest under adversarial pressure. It decomposes reasoning into three architectural layers -- memory grounding (timestamped, immutable records preventing confabulation), structured inference (causal consistency checking), and boundary enforcement (hard stops when conflicting imperatives arise). In simulation-based testing across DeepSeek-V3, Doubao, and Qwen, Box Maze reduced boundary violation rates from roughly 73% under baseline RLHF to below 2% under adversarial conditions. The ablation study showed that the constraint layer (Heart Anchor) is the critical component -- removing it caused immediate vulnerability to emotional manipulation, while the logic layer alone produced "coherent confabulation," logically structured but factually false narratives. The work is conceptual and simulation-validated rather than kernel-implemented, but the architectural principle -- process-level control rather than outcome filtering -- resonates with the deterministic-boundaries argument from the security side. (more: https://arxiv.org/abs/2603.19182v1)

Meanwhile, a practitioner debugging cache misses on Apple's M5 Max tracked a major performance bug to Qwen 3.5's shipped chat template. The template emits empty historical <think>...</think> blocks for prior assistant turns even when there is no reasoning content, causing equivalent conversation histories to serialize differently across requests. This creates prompt drift that breaks prefix-cache reuse, forcing unnecessary reprocessing of tens of thousands of tokens after tool-heavy interactions. The fix is a one-line template change. Community response confirmed the issue across multiple backends including llama.cpp and LM Studio -- a reminder that when inference feels slow, the real culprit often lurks in the infrastructure layer, not the model. (more: https://www.reddit.com/r/LocalLLaMA/comments/1sg076h/i_tracked_a_major_cache_reuse_issue_down_to_qwen/)

Agents at Scale

Anthropic's managed agents, now in public beta, tackle the plumbing that makes agent deployment hard: scoped permissions, identity management, execution tracing, and multi-agent coordination. Reuven Cohen's critique cuts deeper than the surface polish: managed agents treat the problem as orchestration (prompt, tool, retry, observe) rather than intelligence (bounded reasoning with structural correctness guarantees). Cost scales linearly with tokens, state is fragile, behavior is probabilistic. His alternative vision -- contrastive AI with mincut-defined structure, proof-gated mutation, and optimization for joules per decision rather than tokens per task -- is ambitious if unproven. The structural question remains: is the industry building increasingly sophisticated scaffolding around a fundamentally unstable core? (more: https://www.linkedin.com/posts/reuvencohen_anthropics-new-managed-agents-look-clean-activity-7447846196569780225-G8NT)

At the small-model end, a LoRA fine-tuned on Qwen3.5-9B achieved 89% autonomous workflow completion on 29 Kaggle datasets -- averaging 26 autonomous iterations including Python execution, chart generation, and summarization -- where the base model averaged 1.2 iterations and stopped at 0% completion. The training data was not standard instruction tuning but massive multi-step trace datasets covering real-world scenarios, proving that small models can be genuinely autonomous agents when trained on scenario-based workflows rather than single-turn instructions. The author is bottlenecked by compute and seeking sponsorship -- a familiar story for open-source agent research. (more: https://www.reddit.com/r/LocalLLaMA/comments/1shlk5v/model_release_i_trained_a_9b_model_to_be_agentic/)

On the infrastructure side, continuous batching transforms agent swarm economics: running 50 tasks through Qwen 27B sequentially takes 42 minutes, but batching all agents simultaneously pushes total throughput to 1,100 tokens per second, completing the same work in 70 seconds. (more: https://www.reddit.com/r/LocalLLaMA/comments/1sduop2/we_can_use_continuous_batching_for_agent_swarm_to/)

Andrej Karpathy published a pattern for LLM-powered personal knowledge bases that reframes RAG entirely. Instead of retrieving from raw documents at query time, the LLM incrementally builds and maintains a persistent wiki -- structured, interlinked markdown files where knowledge is compiled once and maintained, not re-derived on every query. The human curates sources and asks questions; the LLM does the bookkeeping that makes wikis actually useful. Karpathy's insight about why human-maintained wikis fail -- "the maintenance burden grows faster than the value" -- explains why LLMs are the missing piece: they do not get bored and can touch fifteen files in one pass. The gist spawned immediate community implementations, including team-oriented versions using Claude Code's CLAUDE.md for ambient wiki awareness and git-native architectures where the wiki lives as a submodule with PR-based review. (more: https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)

Local-First and the Trust Deficit

A growing segment of practitioners cares less about frontier benchmarks and more about what stays on their machine. The sentiment on LocalLLaMA is increasingly explicit: "I keep everything local, because I want to continue to do everything I do now after the AI industry implodes and those APIs get priced out of reach." Code review, business-sensitive context, anything involving real files -- the calculus changes the moment proprietary data is involved. The emerging consensus is tiered: cloud APIs for generic tasks, local models for sensitive work, with the boundary defined by what you cannot afford to have exfiltrated or mined. (more: https://www.reddit.com/r/LocalLLaMA/comments/1shdj3a/am_i_the_only_one_who_cares_less_about_smarter/)

OpenAI is now testing ads in ChatGPT for US Free and Go plan users. Ads appear below responses, are "clearly labeled as sponsored," and OpenAI states they do not influence model outputs. Personalized ads can use chat history and memories if the user opts in. An ads-free option exists for Free users but comes with reduced message limits and no access to tools like image generation. The framing is access-supportive -- "keeping the Free and Go plans fast and reliable requires significant infrastructure" -- but the structural incentive is now in place: the more you chat, the more ad inventory you generate. For users already nervous about data sovereignty, this is another reason to run local. (more: https://help.openai.com/en/articles/20001047-ads-in-chatgpt)

The supply chain angle reinforces the control impulse. dockerfile-pin is a CLI tool that adds @sha256:<digest> to FROM lines in Dockerfiles, image fields in docker-compose.yml, and Docker image references in GitHub Actions files. The motivation is straightforward: tag-based references can be subverted by force-pushing tags to new commits; only digest pinning is immune. The tool uses HEAD requests (not pulls) to resolve digests, handles private registries, and ships with CI integration for PR validation. (more: https://github.com/azu/dockerfile-pin)

One experimenter is building a DIY Mythos-at-home system with two local agents -- an uncensored Qwen3.5-27B scanner and a GLM-5.1 orchestrator -- hunting through the OpenBSD source tree for vulnerabilities. The thesis: Big Tech thinks zero-day research requires clearance and NDAs; this project thinks it requires open models and an ACME rocket strapped to a coyote. (more: https://www.linkedin.com/posts/ownyourai_im-experimenting-with-building-amythos-activity-7448119070635274241-TCFQ)

Charcuterie rounds out the toolkit: a browser-based visual similarity explorer for Unicode that embeds rendered glyphs with neural networks and compares them in vector space, useful for spotting homoglyph attacks and confusable characters in security-sensitive contexts. (more: https://charcuterie.elastiq.ch/)