Custom Quantization Beats Pre-Built Models

Custom Quantization Beats Pre-Built Models

The local LLM community has long relied on pre-quantized GGUF models from popular distributors like TheBloke, but a growing faction argues this convenience comes at a significant cost. The core problem: someone else chose the quantization format, calibration data, and weight preservation strategy, optimizing for nobody in particular. A new open-source pipeline called LlamaPajamas aims to change this by automating the process of downloading full-precision models and converting them with hardware-specific and domain-specific optimizations (more: https://www.reddit.com/r/LocalLLaMA/comments/1p1dkzh/youre_using_huggingface_wrong_stop_downloading/).

The technical argument is compelling. Different model architectures require different backends: vision and speech models like Whisper and YOLO primarily use matrix multiplications and convolutions, making them suited for CoreML on Apple Silicon or TensorRT on NVIDIA GPUs. Large language models, with their attention mechanisms and KV caches, demand different treatment—MLX excels on Apple Silicon for text generation, while GGUF remains the universal CPU choice. Running vision models through GGUF or MLX produces suboptimal results because these backends weren't designed for such workloads.

The quantization ladder reveals where most users leave performance on the table. While Q4_K_M represents a 4x compression ratio, importance quantization (IQ) formats like IQ3_XS achieve 6x compression with minimal accuracy loss when paired with domain-specific calibration data. The project's benchmark results are striking: a medical model quantized with medical calibration data maintains 95%+ task accuracy at IQ3_XS (900MB), while the same model with generic calibration drops to 85%. That 10% accuracy difference comes purely from calibration data selection at identical file sizes.

The project encountered several painful lessons worth noting. The llama-imatrix tool requires at least 4,096 tokens to generate useful importance matrices—initial tool-calling datasets with only 1,650 tokens failed completely. Even more concerning, initial evaluation methodology was fundamentally flawed. A "lenient mode" that accepted any answer containing the correct letter showed 90% accuracy, but strict mode requiring exact matches revealed the true figure: approximately 50%. The project's author puts it bluntly: "The 90% number was a lie that made us feel good." Community reception was mixed, with some noting the project's apparent heavy reliance on AI-generated code and documentation.

Function Calling Pushes LLM Limits

Backend code generation represents one of the most demanding tests of LLM function-calling capabilities, and a new benchmark called AutoBE aims to quantify just how far different models can be pushed. The project generates entire backend applications through extensive function calling, including Abstract Syntax Tree (AST) structures of infinite depths for database schemas, API specifications, and test functions. The developers describe it as "the most extreme function calling benchmark ever" (more: https://www.reddit.com/r/LocalLLaMA/comments/1p2ziil/hardcore_function_calling_benchmark_in_backend/).

The results expose significant capability gaps between models. Claude Sonnet 4.5 and GPT-5.1 create 630 and 2,000 test functions respectively for identical topics, while Qwen3-next-80b-a3b produces only 360. This discrepancy points to an inherent limitation: the benchmark currently measures whether models can successfully construct extremely complex types through function calling, but lacks controlled variables for proper scoring. The developers acknowledge this openly—a model generating more test functions should receive higher scores, but the current system only tracks phase completion.

OpenRouter's "exacto" suffix emerged as a notable detail in the discussion. These precision tool-calling endpoints route requests to providers with measurably better tool-calling performance. While model weights are identical across providers, real-world inference quality differs significantly. OpenRouter's billions of monthly requests provide visibility into how models behave across different inference stacks, allowing curation of the most accurate providers for agentic workloads. For anyone using Model Context Protocol (MCP) integrations, the exacto variants are worth considering for improved accuracy.

A critical configuration issue surfaced during testing: GPT-5.1's reasoning is disabled by default, unlike previous models. Without explicitly setting reasoning effort, the model performs poorly on agentic tasks. When reasoning was enabled, GPT-5.1 went overboard in the opposite direction, generating 550 DTO schemas for a "simple todo app" request and essentially building a Jira-like task management system. Google's Gemini 3, despite strong performance on other benchmarks, underperformed significantly because its announced support for standard JSON schema features like $ref and anyOf doesn't work properly in practice.

Latency Optimization Goes Beyond Model Size

Most developers blame LLM latency on model size, but the real culprits often lurk elsewhere in the stack. Infrastructure problems—request queues, batching strategies, token schedulers, and memory pressure—frequently cause more delay than the model itself. When multiple users hit the same endpoint, requests pile up in queues causing delays even when GPU resources sit idle (more: https://www.reddit.com/r/LocalLLaMA/comments/1p71cas/hidden_causes_of_llm_latency_its_not_just_the/).

The distinction between static and continuous batching proves particularly important. Static batching groups requests together but forces everything to wait for the longest sequence in the batch, wasting GPU cycles. Continuous batching allows new requests to join ongoing batches while completed sequences free memory instantly, keeping GPU utilization high. Systems using continuous batching with paged attention—vLLM, TGI, and TensorRT-LLM—generally handle high-load scenarios better than static implementations.

For local inference, different optimization strategies apply. Long prompts benefit enormously from saving and restoring the KV cache. One practitioner reports reducing multi-minute prompt processing to under a second by loading cached states from SSD or RAM. Even trillion-parameter models like Kimi K2 have caches of only a few gigabytes, making this approach practical. The key insight: place variable content at the end of prompts so most cached prefix remains valid across sessions.

Hardware configuration matters more than many realize. Running dual-socket Xeon systems without proper NUMA settings can limit memory bandwidth to a single node. The command flag `--cpunodebind=0 --interleave=all` can unlock additional memory channels and improve token generation speeds. For CPU-bound inference, backend choice matters significantly: ik_llama.cpp reportedly delivers roughly double the token processing speed of standard llama.cpp in recent tests, with approximately 10% improvement in token generation.

NVIDIA's Jet Models Target Edge Deployment

NVIDIA's newly published Jet-Nemotron models claim significant gains in prompt processing and inference speed, and independent analysis confirms the improvements—with important caveats. The models, derived from Qwen2.5-1.5B and Qwen2.5-3B respectively, achieve 2.6x and 2.3x faster prompt processing at 64K tokens after adjusting for model size differences (more: https://www.reddit.com/r/LocalLLaMA/comments/1p558vw/in_depth_analysis_of_nvidias_jet_nemotron_models/).

The throughput numbers on consumer hardware tell a nuanced story. On an RTX 3090 with 65,536-token prompts, Jet-Nemotron-2B achieves 12,074 tokens per second for prefill compared to Qwen2.5-1.5B's 6,197. For decode at the same batch size, the 2B model runs at 117 tokens per second versus 76 for the baseline. The real advantage emerges at higher batch sizes: because Jet models use significantly less VRAM, they can run at much larger batch sizes, achieving up to 12x throughput gains. The claimed 47x improvement likely requires 80GB H100 hardware.

Edge deployment appears to be the target use case. The models offer faster prompt processing, somewhat faster inference, and substantially lower memory footprint—ideal characteristics for mobile or embedded systems. They could also serve well for multi-user server deployments. However, only base models are currently released, limiting practical utility. The authors indicate instruct-tuned versions are in development.

A significant limitation dampens enthusiasm: the models carry a non-commercial license. For hobby use on mobile devices through applications like Pocketpal, this restriction may be irrelevant. For anyone considering commercial deployment, the licensing terms effectively eliminate the option regardless of technical merits.

GPU Wars: ROCm Versus CUDA Reality Check

The eternal question of AMD versus NVIDIA for local LLM inference continues generating heated discussion, with practical experience painting a more nuanced picture than either camp typically admits. One user considering 3090s versus 7900 XTX cards sought community input on ROCm support and Vulkan performance (more: https://www.reddit.com/r/LocalLLaMA/comments/1p2whr9/rtx_3090_vs_rx_7900_with_rocm_also_vulcan/).

Home users report surprisingly smooth experiences with AMD cards. ROCm works well within LM Studio, with one 7900 XTX owner achieving 12 tokens per second on GPT-OSS 120B while seamlessly leveraging system RAM for overflow. The ability to use integrated system memory alongside VRAM represents a genuine advantage for memory-constrained setups. For straightforward inference with GGUF models, the AMD ecosystem has matured considerably.

Enterprise use cases tell a different story. Professional users working with vLLM for batch processing and tensor parallelism describe AMD support as "too bothersome"—the CUDA stack simply has more mature tooling and broader community support. The gap narrows for simpler workloads but widens significantly for production deployments requiring advanced features.

VRAM remains the decisive factor for serious local inference. Multiple commenters emphasize that 24GB represents the minimum for meaningful capability, with 32GB cards offering more headroom. The cost difference between 24GB and 32GB cards has narrowed enough that the extra VRAM typically justifies the premium, especially given current model sizes and the trend toward larger context windows.

Kubernetes Scales to 130,000 Nodes

Google Cloud has achieved what may be the largest Kubernetes cluster ever constructed: 130,000 nodes running in experimental mode on Google Kubernetes Engine. This doubles the officially supported limit and required innovations across multiple dimensions beyond raw node count, including sustained Pod throughput of 1,000 operations per second (more: https://cloud.google.com/blog/products/containers-kubernetes/how-we-built-a-130000-node-gke-cluster/).

The engineering challenges at this scale are formidable. At 130,000 nodes, read requests to the API server can overwhelm the object datastore. Two key enhancements address this: consistent reads from cache enable the API server to serve strongly consistent data directly from in-memory storage, while snapshottable API cache generates B-tree snapshots for serving LIST requests without repeatedly querying the datastore. The distributed storage backend, based on Google's Spanner, required 13,000 queries per second just to update lease objects for node health checks.

Workload scheduling becomes vastly more complex at scale. The default Kubernetes scheduler handles individual Pods, but AI/ML environments require job-level management. Kueue, a job queueing controller, brings batch system capabilities to Kubernetes, enabling all-or-nothing scheduling for entire jobs and orchestrating complex mixes of competing training, batch, and inference workloads. The system demonstrated impressive preemption performance: 39,000 Pods preempted in 93 seconds, with median Pod churn reaching 990 Pods per second.

Power constraints now eclipse chip supply as the limiting factor for massive clusters. A single NVIDIA GB200 GPU requires 2,700W. With tens of thousands of these chips, a single cluster's power footprint scales to hundreds of megawatts, ideally distributed across multiple data centers. Google reports numerous customers already operating clusters in the 20,000-65,000 node range, with anticipated demand stabilizing around 100,000 nodes. The hardening work benefits smaller clusters too—the improvements create substantial headroom for average deployments, making them more resilient and tolerant of API misuse.

Data Races Plague Go Concurrency

Go's reputation for easy concurrent programming comes with a dark side: the language provides numerous ways to create data races inadvertently. A detailed analysis documents four categories of bugs that have bitten production systems, demonstrating that even experienced developers routinely fall into these traps (more: https://gaultier.github.io/blog/a_million_ways_to_data_race_in_go.html).

The most insidious bug involves accidental variable capture in closures. The difference between correct and incorrect code is literally one character: using `err =` (assignment to outer variable) versus `err :=` (new local variable declaration). When an outer variable is implicitly captured by closures running in separate goroutines, concurrent mutations produce data races. The Go memory model explicitly warns that races on multiword data structures can lead to inconsistent values and "arbitrary memory corruption."

HTTP client documentation is misleadingly dangerous. The standard library states that `http.Client` is "safe for concurrent use by multiple goroutines," but this only applies to request execution—modifying fields like `CheckRedirect` concurrently causes data races, I/O races, and potential nil pointer dereferences. The recommended documentation should read: "Once constructed, performing an HTTP request is concurrency safe, provided that the http.Client fields are not modified concurrently."

Mutex lifetime mismatches represent a particularly subtle trap. Code can appear to use mutexes correctly while completely failing to provide synchronization. When a global map is protected by a mutex created fresh for each HTTP handler invocation, the result is one map protected by N different mutexes—none shared between concurrent units. Go's shallow copy semantics for structures containing reference types like maps and slices compound the problem, making it easy to accidentally share underlying data while thinking each handler has its own copy.

Graph Databases Meet SQL Simplicity

A new embedded graph database called GraphLite promises to bring the simplicity of SQLite to graph queries, implementing the ISO GQL (Graph Query Language) standard in a portable Rust package. The project aims to eliminate the complexity of client-server architectures for applications requiring graph database capabilities (more: https://github.com/GraphLite-AI/GraphLite).

The technical foundation includes a grammar optimized from the OpenGQL project, which provides the open-source reference grammar for the ISO GQL standard. GraphLite supports powerful MATCH clauses for graph traversal, full transaction support with isolation levels, and cost-based query optimization. The underlying storage uses Sled, a high-performance embedded database, keeping everything contained in a single binary.

Installation follows SQLite conventions. Users can add GraphLite as a Cargo crate for Rust applications, install the CLI directly from crates.io, or build from source for development work. The API deliberately mirrors SQLite patterns: open a database connection, execute GQL queries, and retrieve results without external dependencies or server processes.

For the LLM and AI tooling ecosystem, lightweight embedded graph databases offer interesting possibilities. Knowledge graphs, entity relationships, and structured memory systems benefit from graph semantics without requiring separate infrastructure. Whether GraphLite achieves production readiness remains to be seen, but the design philosophy—bringing standards-compliant graph queries to embedded scenarios—addresses a genuine gap in available tooling.

LLM Agents Need Aligned Communication

When LLM agents possess asymmetric information, how well can they collaborate? New research extends Einstein Puzzles into a collaborative tabletop game where two agents must reason, communicate, and act to satisfy spatial constraints despite having partial, different knowledge. The findings have significant implications for human-AI interaction design (more: https://arxiv.org/abs/2510.25595v1).

The experimental setup divides constraints between two players, ensuring neither can complete the task independently while together possessing all necessary information. Agents take turns placing objects into destination bins, sharing information, or requesting information from partners. The researchers tested four communication configurations: share only, ask only, both share and ask, and no communication.

Critical alignment emerged as the key finding. Agents with both information-seeking and information-providing capabilities collaborate most effectively. Mismatched abilities—one agent that can only share paired with one that can only ask—produces significant performance degradation. The research confirms that aligned interaction protocols matter as much as raw capability.

A counterintuitive result deserves attention: agents without communication still achieved high task performance. However, deeper analysis revealed these agents lacked genuine rule understanding and scored poorly on trust from human evaluators. Their success came from pattern matching rather than comprehension. Human participants consistently favored agents that proactively shared information, even when such agents performed less efficiently on pure task completion metrics. The gap between efficiency and human preference suggests that proactive information sharing should be prioritized in human-AI systems, even at some cost to raw performance.

FLUX.2 Brings Multi-Image Generation

Black Forest Labs has released FLUX.2, a significant departure from FLUX.1 rather than an incremental update. The new model features a completely new architecture and pre-training from scratch, supporting both text-to-image and image-to-image generation with the ability to take multiple reference images as inputs (more: https://huggingface.co/blog/flux-2).

Architectural changes are substantial. FLUX.2 uses a single text encoder (Mistral3 Small) instead of FLUX.1's dual-encoder setup, simplifying prompt embedding computation while stacking outputs from intermediate layers for richer representations. The DiT architecture shares modulation parameters across transformer blocks rather than maintaining individual parameters per block, removes all bias parameters, and shifts the balance dramatically toward single-stream blocks—73% of parameters versus 46% in FLUX.1.

Hardware requirements are daunting. Inference without offloading requires over 80GB of VRAM. Practical deployment options include CPU offloading (dropping to ~62GB on H100), BitsAndBytes 4-bit quantization (enabling ~20GB operation on consumer cards), remote text encoder deployment via inference endpoints (~18GB VRAM), or group offloading (enabling 8GB GPUs with 32GB RAM). The multi-image input capability supports up to 10 reference images, though each additional image increases memory requirements.

Advanced prompting capabilities include structured JSON formatting with precise control over scene descriptions, object positioning, color palettes with specific hex codes, and lighting configurations. LoRA fine-tuning presents challenges due to memory constraints, but training scripts support memory-saving techniques including remote text encoding, pre-encoding images, and FP8 or BitsAndBytes quantization during training.

Binary Quantization Gets Dynamic Grouping

Binary quantization of LLMs has long represented an aspirational goal with disappointing results—previous state-of-the-art achieved perplexity of 35.04 compared to 5.68 for full precision on LLaMA 7B. A new approach using dynamic grouping dramatically closes this gap (more: https://arxiv.org/abs/2509.03054v1).

The core innovation identifies optimal groupings of unstructured sub-matrices without relying on computationally expensive Hessian calculations. Previous methods segmented weight matrices into uniform grids, then subdivided based on Hessian and magnitude distributions. The new approach systematically searches for groupings that minimize quantization loss while penalizing inefficient partitioning.

Results demonstrate transformative improvement. On LLaMA 3.2 3B, the method achieves perplexity of 8.23 at an average bit length of 1.007—essentially true binary—compared to the original 7.81 at full precision. Previous binary PTQ (BiLLM) achieved perplexity of 123.90 on the same model, making this a dramatic advance. The approach proves competitive with 4-bit methods like GPTQ in both performance and efficiency.

Practical efficiency makes the approach viable: quantizing the full LLaMA 3.2 3B model takes only 14 seconds on a single CPU core, with the entire process completing in under 100 minutes. The algorithms exhibit embarrassingly parallel properties, enabling straightforward scaling. Three algorithm variants balance speed and accuracy, with Windowed Greedy Merging offering the most practical trade-off for contemporary architectures.

Kimi K2 Thinking Sets New Benchmarks

Moonshot AI's Kimi K2 Thinking establishes new state-of-the-art performance on several demanding benchmarks while introducing capabilities that extend well beyond typical reasoning models. The trillion-parameter Mixture-of-Experts architecture activates 32 billion parameters per forward pass and supports 256K context windows (more: https://huggingface.co/moonshotai/Kimi-K2-Thinking).

The headline numbers on Humanity's Last Exam (HLE) with tools deserve attention: 44.9% accuracy exceeds GPT-5's 41.7% and Claude Sonnet 4.5's 32.0%. On the "heavy" setting with extended compute, K2 Thinking reaches 51.0%. More impressive still is the stable long-horizon agency—the model maintains coherent goal-directed behavior across 200-300 consecutive tool invocations, vastly exceeding prior models that typically degrade after 30-50 steps.

Agentic search tasks showcase the model's practical capabilities. On BrowseComp, K2 Thinking achieves 60.2% versus GPT-5's 54.9% and Claude Sonnet 4.5's 24.1%. The Frames benchmark shows similar strength at 87.0%. These results reflect end-to-end training to interleave chain-of-thought reasoning with function calls, enabling autonomous research, coding, and writing workflows that persist across hundreds of steps without drift.

Native INT4 quantization through Quantization-Aware Training during post-training delivers lossless 2x speedup in low-latency mode while reducing GPU memory requirements. The model's performance on AIME25 reaches 99.1% with Python tools and 100% in heavy mode—matching or exceeding the best closed-source alternatives while remaining fully open-weight.

RAG Context Requires Careful Threading

A common pitfall in RAG implementations surfaces when follow-up queries ignore previous context entirely. One developer building a FastAPI system with Qwen2.5-7B-Instruct found that initial queries worked correctly but follow-ups retrieved completely unrelated information despite chat history being available (more: https://www.reddit.com/r/LocalLLaMA/comments/1p52fb0/rag_followups_not_working_qwen25_ignores_previous/).

The diagnosis is typically straightforward: if the model loses context, the context isn't being sent. The payload structure—including system prompt, retrieved context documents, chat history, and current query—must be correctly assembled into the LLM's conversation format. Many frameworks handle this automatically, but custom implementations require explicit attention to message threading.

Community responses pointed to several common issues. First, upgrading to Qwen3 provides better instruction-following out of the box. Second, the formatting of how FastAPI passes the request into the LLM context often contains bugs—developers should verify the exact prompt structure reaching the model. Third, retrieved context needs proper integration with chat history, not just concatenation of separate message types.

The broader lesson applies across RAG implementations: debugging requires visibility into the exact prompt reaching the model. Logging the complete formatted input before inference often reveals obvious errors that remain invisible when examining individual components separately.

Developer Tools Expand LLM Integration

The open-source ecosystem continues producing tools that extend LLM capabilities into development workflows. ThinkReview, a browser extension powered by Ollama, performs code reviews on GitLab and Azure DevOps pull requests, summarizing changes, identifying security issues, evaluating best practices, and providing scoring (more: https://www.reddit.com/r/ollama/comments/1p1rv91/browser_extension_powered_by_ollama_for_code/).

Mimir, a shared memory system for LLM agents, received a significant security update adding OAuth authentication, RBAC, GDPR/FISMA/HIPAA compliance, and automated OWASP security testing. Crucially, the update implements proper locking for memory operations—previously, two agents could collide on updates to the same node, corrupting the graph. The new lock manager includes conflict detection and retries for multi-agent setups (more: https://www.reddit.com/r/ChatGPTCoding/comments/1p4epih/mimir_oauth_and_gdpr_compliance_vscode_plugin/).

GoTOON provides a Go implementation of Token-Oriented Object Notation, a compact format designed for passing structured data to LLMs with reduced token usage. The format achieves 30-60% fewer tokens than JSON by removing redundant punctuation, replacing braces with whitespace, and declaring keys once for tabular data. Benchmarks using GPT-5's tokenizer show reductions ranging from 35-59% depending on data structure (more: https://github.com/alpkeskin/gotoon).

For LangChain users, a new Python script enables stress-testing agents against infinite loops—a common failure mode when agents get stuck in recursive tool-calling patterns. Open Logic testing helps identify conditions that cause agents to spin indefinitely before deploying to production (more: https://www.reddit.com/r/LocalLLaMA/comments/1p5kde5/python_script_to_stresstest_langchain_agents/).

Image Relighting Gets LoRA Treatment

Creative applications of fine-tuned models continue expanding, with a new LoRA for Qwen-Edit-2509 enabling image relighting through natural language descriptions. Users can specify lighting conditions like "soft, diffuse light from curtains" and have existing images transformed accordingly (more: https://huggingface.co/dx8152/Relight).

The implementation requires combining the relighting LoRA with a base lightning LoRA for optimal results. The Chinese trigger word "重新照明" (meaning "relighting") activates the capability, followed by natural language descriptions of desired lighting effects. This represents an interesting application of language-guided image editing, where semantic descriptions control photorealistic lighting modifications.

Hardware discussions for creative workloads often intersect with inference optimization. One user exploring multi-GPU setups for running models like GLM-4.5-Air at reasonable speeds faces the common constraint of insufficient PCIe lanes on consumer platforms. Z790 motherboards paired with i9-14900K offer limited lanes, pushing users toward Threadripper or EPYC platforms for serious multi-GPU configurations (more: https://www.reddit.com/r/LocalLLaMA/comments/1p2lenc/question_about_motherboards/).

The RAM market complicates upgrade decisions. Memory prices have reportedly tripled in recent months as AI builders drive demand for high-capacity configurations. Platform choices increasingly depend on which RAM remains affordable rather than pure performance characteristics—a practical consideration that specification comparisons often overlook.

Boot Sector Pong Shows Assembly Mastery

Sometimes the most impressive engineering happens within the tightest constraints. A new implementation of Pong fits entirely within a 512-byte boot sector, complete with the required 0x55 0xAA magic bytes—leaving just 510 bytes for playable game code (more: https://hackaday.com/2025/11/20/pong-gets-the-boot/).

The implementation uses 80×25 text mode and writes directly into video memory, requiring assembly language and no operating system functions—none exist at that point in the startup sequence. Modern development makes such projects vastly more accessible than historical attempts: QEMU allows testing without rebooting physical hardware and provides debugging capabilities that would have seemed magical to 1990s developers.

This isn't the first boot sector game, and the article acknowledges prior work in the space. But the engineering challenge remains genuine. Every byte matters, and the absence of libraries means implementing game logic, input handling, and display updates from scratch. Comment threads noted that VMware Player version 9 actually includes a hidden Pong game that plays when attempting to boot from an empty disk image—an Easter egg discovered when a user tried to format a zero-byte file as a system disk.