Qwen lands in llamacpp: MoE trade-offs and pruning realities

Qwen lands in llama.cpp

Qwen3 Next is arriving in local runtimes, but expect churn. A ready-for-review branch adds Qwen3 Next support to llama.cpp, with maintainers warning it’s not final: quantized models may need to be re-downloaded later and speed isn’t optimized yet (more: https://www.reddit.com/r/LocalLLaMA/comments/1oes4ez/qwen3_next_support_in_llamacpp_ready_for_review/). Community testing highlights the usual MoE trade-offs: if you’ve got VRAM, Next-80B/32B can deliver strong throughput; if not, MoE still degrades “gracefully” under CPU/RAM offload compared to dense models, though you’ll be picking quantizations like Q2 to stay coherent on 12 GB GPUs (more: https://www.reddit.com/r/LocalLLaMA/comments/1oes4ez/qwen3_next_support_in_llamacpp_ready_for_review/).

Vision is joining the party, albeit via pre-release forks. An unofficial llama.cpp build runs Qwen3-VL-32B/4B Instruct via GGUF; another branch based on earlier work is also circulating, and contributors are pushing for an upstream PR. Early testers report heavy CPU load and high RAM use, which is unsurprising for pre-release VL support and large context buffers (more: https://www.reddit.com/r/LocalLLaMA/comments/1od59hx/qwen3vl32binstruct_gguf_with_unofficial_llamacpp/).

On hardware, AMD’s Strix Halo APUs are seeing improving Linux stories. One user reports llama.cpp at roughly 47 tokens/sec on a 120B GGUF Q4_K_M with ROCm 7.0.2 on Ubuntu 24.04; others are using Vulkan as a comparable fallback. vLLM on ROCm still takes elbow grease without prebuilt wheels, but community toolboxes can unlock 125 GB of shared memory for AI on Linux, with early forks for Ollama-on-AMD progressing as well (more: https://www.reddit.com/r/ollama/comments/1odxx3b/hows_halo_strix_now/).

MoE trade-offs and pruning realities

Community sentiment is catching up to MoE realities. Qwen3 Next sparked debates over how to compare “knowledge” and “intelligence” versus dense models. The old “sqrt(active × total params)” heuristic for MoE capability is increasingly viewed as dated; users note Qwen3-VL 30B beating 8B and approaching 32B on some internal tables, and practical impressions that GLM 4.5 Air (12B active) exceeds dense ~35B “knowledge” in some tasks—suggesting improved routing/training has shifted old rules of thumb (more: https://www.reddit.com/r/LocalLLaMA/comments/1oes4ez/qwen3_next_support_in_llamacpp_ready_for_review/).

Pruning isn’t a free lunch. Users testing GLM4.5 Air REAP (pruned experts) report tool-use brittleness that benchmarks didn’t surface: “unable to follow more than 5 tool calls at a time,” versus the full model handling long chains. Community responses point out that structured pruning often hurts sequential reasoning and multi-step tool calls more than isolated evals reflect—a reminder that agentic workloads need workflow-level tests, not just single-turn metrics (more: https://www.reddit.com/r/LocalLLaMA/comments/1oewke2/glm_air_reap_tool_call_problems/).

Practical advice emerges: if your workload depends on long tool-call chains or “agentic” reliability, choose the full model at a lower bit quant over a pruned variant. If memory-bound, MoE can still be attractive due to better offload behavior—just temper expectations on chain-of-action depth and plan to validate on your actual tool graphs, not only leaderboards (more: https://www.reddit.com/r/LocalLLaMA/comments/1oes4ez/qwen3_next_support_in_llamacpp_ready_for_review/;), (more: https://www.reddit.com/r/LocalLLaMA/comments/1oewke2/glm_air_reap_tool_call_problems/).

Agents need sandboxes, not hype

Agent stacks are maturing around well-defined boundaries. A developer wired a simple REST API to a Model Context Protocol (MCP) server and let a coding agent populate a genealogy database from instructions—then connected ChatGPT’s MCP client to a local toolchain for repo checkout, builds, tests, and PRs. The appeal: inside ChatGPT Plus, the MCP connector drives heavy tool-calling without separate per-token API billing, though limits may evolve. The take-away is less “magic,” more about MCP standardizing tool interfaces so LLMs can operate safer, repeatable workflows (more: https://www.reddit.com/r/ChatGPTCoding/comments/1oawtoj/built_my_own_mcp_server_for_my_app_and_was/).

Meta and Hugging Face are placing structure above improvisation with OpenEnv, an open hub and spec for “agentic environments.” These environments define the tools, APIs, credentials, and execution context required for a task—sandboxes that can be used both for training (e.g., RL post-training with TRL, TorchForge+Monarch, VeRL) and deployment. RFCs include MCP tool encapsulation through environment abstractions, unified action schemas across tool-calling and CodeAct, and Docker-based local runs. The project prioritizes clear semantics, isolation, and reproducibility—exactly what many ad-hoc agent demos lack (more: https://huggingface.co/blog/openenv).

Within this frame, Meta’s Code World Model (CWM) shows how far task-grounded training is going. CWM is a dense 32B LLM for code generation and “world modeling” of program state, mid-trained on execution trajectories and agentic interactions, then post-trained with multi-task RL for verifiable coding and multi-turn software engineering. It ships under a non-commercial research license, needs a specific system prompt with explicit … “reasoning” mode, and runs on an 80 GB GPU with quantization. On SweBench Verified, CWM reports 53.9% (65.8% with tts) on the full set—competitive among open models—with clear disclaimers it’s not a general chat assistant (more: https://huggingface.co/facebook/cwm).

Evaluation itself remains a soft spot. A new paper warns that design failures in LLM judge benchmarks can silently undermine validity—timely advice as communities increasingly use LLM judges to rate agent steps and outputs. If your training signal or deployment gating comes from judge scores, small design errors can distort the gradient in big ways (more: https://arxiv.org/abs/2509.20293v1).

Multimodal image models evolve fast

New image stacks are converging on two-stage pipelines and test-time controls. Representation Autoencoders (RAE) use frozen representation encoders (e.g., DINOv2, SigLIP, MAE) with trained ViT decoders, then train a Stage 2 Diffusion Transformer (DiT) on the autoencoder’s latent space. The official PyTorch repo ships decoders, DiT-XL weights, LightningDiT, configs, sampling scripts, and an ADM-based FID evaluation path. Practical touches matter: ImageNet stats, FID-50k, autoguidance/classifier-free schedules in YAML, and DDP training/sampling recipes—all geared for reproducibility and speed (more: https://github.com/bytetriper/RAE).

DreamOmni2 unifies multimodal instruction-based editing and generation in one open model and claims superior identity/pose consistency for subject-driven generation, plus strong handling of abstract attributes (materials, textures, styles)—even “surpassing commercial models” in some cases, according to the repo. Crucially, it distinguishes editing (preserve non-edited regions) from generation (retain ID/attribution, recompose the rest), offers separate web demos for each, and supports reference images to capture details language can’t describe well (more: https://github.com/dvlab-research/DreamOmni2).

For modern MMDiT/Flow-Matching T2I (e.g., FLUX, SD3.5), Stitch introduces a training-free, test-time position control method that automatically infers bounding boxes, binds early generation to regions, uses attention-driven mid-generation segmentation to extract foregrounds in the latent space, and then composites and refines without constraints. The authors also propose PosEval, a positional benchmark that extends GenEval. The promise is pragmatic: improved spatial adherence without retraining or external segmenters, and without sacrificing final image quality (more: https://arxiv.org/abs/2509.26644v1).

Also spotted: Tencent’s POINTS-Reader listed on Hugging Face. Details are in the repository for those exploring new multimodal baselines and tooling (more: https://huggingface.co/tencent/POINTS-Reader).

Local-first apps, small models, real workflows

Local-first research notebooks are getting serious. Deta Surf is an open-source AI notebook that stores your data locally in open formats via SFFS, lets you organize a library of files and web links, and brings them directly into a stream-of-thought UI with tabs, split views, and deeplinks back to sources (timestamps, pages). You can ask questions over YouTube, PDFs, or the web, invoke tools for web search, and even generate interactive applets—using the model of your choice via BYO keys—across macOS, Windows, and Linux (more: https://github.com/deta/surf).

Small, task-specific models continue to shine. A 18.7 MB webcam picture-in-picture detector on Hugging Face uses YOLO to locate webcam panes in livestream/screen-share footage, with users discussing use cases like fast frame-by-frame segmentation for video manipulation pipelines. It’s a good reminder that “tiny but focused” models can be the right tool for media workflows without hauling in a full-blown VLM (more: https://www.reddit.com/r/LocalLLaMA/comments/1oei8pl/picture_in_picture_webcam_detect_model_on/).

Hardware hacks are equally creative. Tommy turns ESP32 devices into through-wall Wi‑Fi motion sensors with local processing, one-click flashing, Docker or Home Assistant integration, and zone-based detection. It’s in beta, free to try, with caveats: pets/fans cause motion today, filtering is planned, and it detects movement, not stationary presence (expected in Q1 2026). With only two devices per zone, it can cover whole areas without line-of-sight (more: https://www.tommysense.com).

There’s also work on making agent collaboration safer and more interpretable. Story Keeper’s PACT-AX focuses on context sharing, state transfer (in development), policy alignment, and trust scoring between agents—framed as EI + AI (emotional + artificial intelligence) for distributed collaboration. It’s early, but it speaks to a need for principled context exchange and governance as multi-agent systems grow (more: https://github.com/neurobloomai/pact-ax).

Coding with AI, from demos to production

Anecdotes show that “agentic coding” is increasingly practical when the spec is clear. One developer reports building a production SaaS in a weekend with Claude Code—auth with SSO, Stripe subscriptions, document distribution, audit trails, and a responsive frontend—iterating by describing intent, testing, and having the agent refactor. The same agent even assisted with UAE licensing requirements. It’s not a silver bullet—prompt precision matters—but the workflow is viable for well-scoped products (more: https://www.reddit.com/r/ClaudeAI/comments/1odfwn4/i_built_a_production_saas_in_one_weekend_with/).

MCP is accelerating this shift by standardizing tool access for coding agents. As above, users are letting ChatGPT (or other MCP-aware clients) connect to custom MCP servers that expose repo operations, builds, tests, and CI steps—keeping data local, actions auditable, and costs predictable inside subscription products. Combined with environment specs like OpenEnv, the pattern is moving from “AI pair programmer” to “AI build-and-ship pipeline” with clearer boundaries (more: https://www.reddit.com/r/ChatGPTCoding/comments/1oawtoj/built_my_own_mcp_server_for_my_app_and_was/;), (more: https://huggingface.co/blog/openenv).

Meta’s CWM underscores why domain-grounded training matters: it mid-trains on execution traces and agentic interactions, then post-trains with RL across verifiable coding and multi-turn software engineering. The result is a code-first research model with strong SweBench numbers and explicit prompting requirements—not a general chat bot—which is a healthy constraint when the goal is reliable system changes, not small talk (more: https://huggingface.co/facebook/cwm).

Security, licensing, and the unglamorous essentials

Supply-chain safety for AI artifacts is getting real attention. Hugging Face now continuously checks files across its 2.2M+ public repos against VirusTotal by hash—no raw contents shared—and surfaces detection status and related intelligence on repo/file pages. It’s a practical boost to transparency and an easy hook for CI policies that gate downloads on clean results (more: https://huggingface.co/blog/virustotal).

Meanwhile, critical infrastructure keeps learning the same lesson: patch and segment. Foreign attackers breached the Kansas City National Security Campus (a core NNSA facility) via unpatched on-prem Microsoft SharePoint, raising fresh concerns about lateral movement from IT to OT and the need to extend zero-trust beyond office networks. The incident highlights the cost of delay at the IT/OT boundary and the long tail of on-prem software (more: https://www.csoonline.com/article/4074962/foreign-hackers-breached-a-us-nuclear-weapons-plant-via-sharepoint-flaws.html).

For developers shipping local LLMs, licensing remains a maze. Options range from hosting models on your own CDN and showing original licenses, to redirecting users to Hugging Face for acceptance. Community replies note that some non-permissive licenses allow re-distribution with license files (e.g., Mistral AI Non-Production), while others… look the other way and hope enforcement stays lax. The safest path is to prefer permissive models and surface license flows prominently—Ollama’s approach raises questions, but the burden ultimately sits on app developers to comply (more: https://www.reddit.com/r/LocalLLaMA/comments/1obvmh6/how_do_you_handle_model_licenses_when/).

Converting meshes back to solids (mostly)

Working with STLs in real CAD requires a careful pipeline, not wishful thinking. A FreeCAD workflow popularized by The Savvy Engineer shows how to turn triangle meshes into usable solids: Part workbench → create shape from mesh → convert to solid → refine shape → Part Design workbench → new Body with the refined shape as BaseFeature → export STEP. That yields an interoperable solid, but not the original parametric history—no sketches, constraints, or feature tree (more: https://hackaday.com/2025/10/21/reverse-engineering-stl-files-with-freecad/).

The caveat is important: you get geometry you can machine or measure, but you can’t tweak it like a native CAD design. Dimensions are driven by the imported surface; you’re editing what’s there, not re-driving features. Commenters note this “barely above STL” for true parametric needs, but it’s the difference between mesh hacks and legitimate solid operations in downstream tools (more: https://hackaday.com/2025/10/21/reverse-engineering-stl-files-with-freecad/).

Alternatives exist for simple edits (e.g., TinkerCAD on meshes), but for professional workflows or STEP export, the Part→Solid→Refine approach is the right path. Just budget time to reconstruct intent if you need parametric control—reverse engineering remains engineering, not a button click (more: https://hackaday.com/2025/10/21/reverse-engineering-stl-files-with-freecad/).