Local models went from mostly useless to actually useful really fast. What changed?

The Rapid Evolution of Local LLMs: From Privacy Toys to Production-Ready Tools An analysis of the paradigm shift in local Large Language Models (LLMs), transitioning from basic privacy-centric experiments to high-perform…

→ View original source

It looks like Rio 3.5 397B could've simply been a semi-failed embezzling of funding

Controversy Surrounding Rio 3.5 397B: Allegations of Misrepresented Training and Misused Funding Recent community findings suggest that the Rio 3.5 397B model may have been misrepresented as a sophisticatedly trained mod…

→ View original source

GLM 5.2 API is live, weights are on HF, and ollama has it already

GLM 5.2 Released: API Access, Open Weights, and Ollama Integration The GLM 5.2 model has transitioned from a restricted subscription-based access model to a widely available release, featuring open weights under the MIT …

→ View original source

Mistral - New family of open-weight models @ July

Mistral Announces New Family of Open-Weight Models Mistral AI has unveiled a new series of open-weight models, continuing their commitment to providing high-performance, accessible AI architectures to the developer commu…

→ View original source

GLM-5.2 just dropped open weights and it already looks weirdly strong for coding

GLM-5.2 Released: Open Weights Model Demonstrates High Proficiency in Coding Tasks The release of GLM-5.2 introduces a powerful open-weights model featuring a massive 1M token context window and dual reasoning effort mod…

→ View original source

GLM-5.2 is the first open-weights model to cross 80% on Terminal-Bench and beats every other open model available

GLM-5.2 Sets New Benchmark in Terminal-Bench, Outperforming Existing Open-Weights Models The release of GLM-5.2 marks a significant milestone in open-weights AI, becoming the first model of its kind to surpass the 80% th…

→ View original source

Why might DiffusionGemma be better at tool calls than its benchmark quality suggests

Analyzing DiffusionGemma: The Potential for Superior Tool Calling via Parallel Block Generation While current benchmarks may suggest lower overall quality compared to Gemma 4, DiffusionGemma's non-autoregressive architec…

→ View original source

Be wary of Qwen/Claude distillations - they're often worse than the base model

Evaluating the Efficacy of Qwen and Claude Model Distillations A critical analysis of recent trends in model distillation, specifically focusing on the performance degradation observed in Qwen and Claude-based distilled …

→ View original source

Donate your coding sessions to an open CC-BY-4.0 dataset to help train open-weight and open source models

Trace Commons: Combating Data Oligopolies Through Open-Source Coding Agent Traces A new community-driven initiative, Trace Commons, aims to democratize the training of open-weight and open-source large language models (L…

→ View original source

Diffusion Gemma Jailbreak

Exploring Prompt Injection and Policy Overrides in Diffusion Gemma A newly reported system prompt technique demonstrates a potential "jailbreak" for Diffusion Gemma and other open-source models, leveraging a hierarchical…

→ View original source

HalBench: 29 OSS models tested on a custom built Sycophancy and Hallucination Benchmark, Qwen 3.6 and Gemma 4 scoring far above their weight! (While Meta keeps proving they forgot how to spend their money...)

HalBench: Evaluating Sycophancy and Hallucination Across 29 Open-Source LLMs HalBench, a specialized open benchmark designed to quantify model sycophancy and hallucination rates, has released new results testing 29 open-…

→ View original source

React Native ExecuTorch now runs Gemma 4 (Vulkan and MLX accelerated)

Integrating Gemma 4 into React Native via ExecuTorch for On-Device Inference Developers can now deploy Gemma 4 within React Native applications using ExecuTorch, enabling fully offline LLM execution with hardware acceler…

→ View original source

Stop using Ollama

Critical Evaluation of Ollama in Local LLM Deployments A community discussion on the r/LocalLLaMA subreddit raises concerns regarding the continued use of Ollama for local large language model orchestration, suggesting a…

→ View original source

We trained a cybersecurity-focused Mythos like LLM open weights on HuggingFace

OpenMythos: A Domain-Specific LLM Engineered for Cybersecurity Tasks Researchers have introduced OpenMythos, an open-weights Large Language Model (LLM) specifically fine-tuned for the cybersecurity domain to mitigate hal…

→ View original source

An agent that plans with a frontier model but runs most of tokens locally (built it for my own dual-3090 rig)

Hybrid AI Orchestration: Combining Frontier Model Planning with Local Execution on Consumer Hardware A developer has implemented a hybrid agentic workflow designed to balance high-level reasoning capabilities with cost-e…

→ View original source

UI/svg block rendering by ServeurpersoCom · Pull Request #24080 · ggml-org/llama.cpp

Integration of SVG Block Rendering in llama.cpp A new Pull Request (#24080) submitted by user ServeurpersoCom to the ggml-org/llama.cpp repository introduces the capability to render SVG blocks within the UI, expanding t…

→ View original source

Made a macOS app that creates highly personal macOS apps. Works with models as small as Gemma 4 E2B

Ironsmith: Enabling Prompt-Based macOS Application Generation via Small Language Models A new open-source project called Ironsmith allows users to generate personalized macOS applications using natural language prompts, …

→ View original source

Voice-to-voice chatbot update

Optimizing Local Voice-to-Voice Pipelines: Achieving Near Real-Time Latency with Qwen3.5 and SNAC A new implementation of a fully local voice-to-voice chatbot demonstrates the feasibility of near real-time, interruptible…

→ View original source

Anyone know how to turn off download images when compiling llama.cpp?

Optimizing llama.cpp Build Processes: Addressing Automatic Asset Downloads During Compilation A technical inquiry from the developer community highlights the automatic downloading of UI-related image assets (such as PWA …

→ View original source

Strange numbers of pp and tg rx7900xtx on ROCm and Vulcan with Qwen3.6-27b nonMTP and MTP

Performance Anomalies: Analyzing PP and TG Metrics for Qwen 3.6-27B on AMD RX 7900 XTX A technical report from the local LLM community highlights unexpected performance metrics regarding prompt processing (PP) and token …

→ View original source

Built a local AI assistant because I always knew this day would come, yesterday just made it feel very real

Mitigating Dependency Risks: Implementing a Local AI Assistant via Gemma 4B Following recent service disruptions at Anthropic, a developer has showcased "Bantz," a local AI assistant designed to ensure data sovereignty a…

→ View original source

it sounds like Meta is abandoning in-house LLM development and reassigning those employees

Meta Restructures AI Workforce: Shift in LLM Development Strategy Following a significant organizational overhaul, Meta is reassigning thousands of employees previously dedicated to AI model training, signaling a potenti…

→ View original source

Dual DGX Sparks- 40tk/s single 1M ; 350 tk/s agg. - Deepseek V4 Flash (vs RTX Pro 6000 vs Mac M2 Ultra 192)

High-Performance DeepSeek V4 Flash Execution on Dual DGX Sparks: A Benchmark Analysis A technical deep dive into leveraging dual DGX Sparks systems for accelerated inference of large Mixture-of-Experts (MoE) models, incl…

→ View original source

Codebase getting larger - Qwen3.6-27B starting to compound issues - how to work smartly with this model?

Scaling Challenges with Qwen3.6-27B: Managing Codebase Expansion and Regression A developer reports a decline in code quality and the emergence of "micro-bugs" when using Qwen3.6-27B for large-scale feature expansion, hi…

→ View original source

Not looking good for GLM 5.2 Air... but maybe a flash model?

Speculation on GLM 5.2 Model Roadmap: Shift Toward Full-Scale and Flash Architectures Recent leaks from unofficial discussions on the Z.ai Discord suggest a strategic shift in the GLM 5.2 development cycle, potentially d…

→ View original source

I am losing my mind with FOMO and need some sanity checking about model capabilities

Addressing "Model FOMO": Evaluating the Diminishing Returns of LLM Capability Scaling A practitioner's reflection on the psychological and technical strain caused by the rapid release cycle of Large Language Models (LLMs…

→ View original source

DeepSeek v4 Pro is too big for such a "midrange" performance, or am I missing something?

Analyzing the Parameter-to-Performance Ratio of DeepSeek v4 Pro A critical examination of the architectural efficiency of DeepSeek v4 Pro, questioning whether its 1.6 trillion parameter count yields a proportional increa…

→ View original source

Add arch support for cohere2-MoE by michaelw9999 · Pull Request #24260 · ggml-org/llama.cpp

llama.cpp Integrates Architecture Support for Cohere2-MoE: Enabling North-Mini-Code 1.0 The llama.cpp ecosystem has expanded its architectural support to include the Cohere2-MoE framework, facilitating the local deployme…

→ View original source

GLM 5.2 is deployed in GLM Coding Plan. API and MIT weights in a week. Voting and benchmarks on X.

GLM 5.2 Deployment: Enhanced Context Windows and Thinking Modes for Advanced Coding The GLM 5.2 model has been officially deployed within the GLM Coding Plan, introducing a massive 1-million token context window and spec…

→ View original source

Unsloth Kimi-K2.7-Code-GGUF

Unsloth Releases GGUF Quantization for Kimi-K2.7-Code Unsloth has begun uploading GGUF-formatted quantizations of the Kimi-K2.7-Code model to Hugging Face, enabling more efficient local deployment and inference for code-…

→ View original source

ZONOS2: real-time TTS with 8B params, 900M active, and high-fidelity voice cloning

ZONOS2: High-Fidelity Real-Time TTS Featuring 8B Parameters and Advanced Voice Cloning Zyphra introduces ZONOS2, a powerful Text-to-Speech (TTS) model utilizing an 8B parameter architecture with 900M active parameters to…

→ View original source

when fable gets banned but it's ok because you've about to download qwen3.7_67b_21a_mythos_father_fable_mother_distilled_ablated_ablitereted_uncensored_agi_sparse_attention_MTP_SuperHOT_q6_maybe_q7_AGI_FINAL.gguf from huggingface

The Rise of Hyper-Specialized GGUF Quantizations: Analyzing the "Mythos-Father-Fable" Model Variant A community discussion on r/LocalLLaMA highlights the trend of utilizing highly modified, uncensored, and distilled mode…

→ View original source

I scaled test-time compute for Qwen-3.6-27B and Gemma-4-31B to surpass Claude Mythos in code optimizations and speedups.

Scaling Test-Time Compute: Qwen-3.6-27B and Gemma-4-31B Outperform Claude Mythos in Code Optimization A new implementation leveraging an advanced compute scaffold has demonstrated that scaling test-time compute for Qwen-…

→ View original source

Do you ever see a post so bad that you ask yourself what was the prompt if this is the output and what model wrote this.

Analyzing the Phenomenon of Low-Quality AI-Generated Content and "Prompt Reverse-Engineering" A discussion within the LocalLLaMA community highlights the increasing visibility of incoherent AI-generated outputs, promptin…

→ View original source

Comparing dual-GPU inference speed between llama.cpp row/tensor split and ik_llama graph split

Comparative Analysis of Dual-GPU Inference: llama.cpp Row/Tensor Split vs. ik_llama Graph Split A technical evaluation of multi-GPU inference strategies, comparing the standard row/tensor splitting method implemented in …

→ View original source

MiniMax Sparse Attention (MSA)

MiniMax Sparse Attention (MSA): Scaling Long-Context Capabilities via Blockwise Sparsity MiniMax Sparse Attention (MSA) introduces a blockwise sparse attention mechanism built upon Grouped Query Attention (GQA) to mitiga…

→ View original source

We should heavily discourage and moderate cloud API (deepseek api, GLM api, etc.) topics and discussion. This is LOCAL first.

Community Debate: Prioritizing Local Execution Over Cloud API Discourse in LocalLLaMA A growing sentiment within the LocalLLaMA community advocates for stricter moderation of cloud-based API discussions to preserve the s…

→ View original source

Open Dungeon: local roleplay with Gemma 4 QAT + inline Uncen-FLUX images, running at full 256K context under 8GB RAM (OS)

Open Dungeon: Achieving Full 256K Context Local Roleplay via Gemma 4 QAT and FLUX Integration A new local implementation called "Open Dungeon" enables private, high-fidelity roleplay by combining a quantized Gemma 4 mode…

→ View original source

EAGLE3 has landed in llama.cpp

EAGLE3 Speculative Decoding Integration Now Available in llama.cpp The llama.cpp ecosystem has officially merged EAGLE3, a sophisticated speculative decoding framework designed to accelerate inference speeds by optimizin…

→ View original source

Open sourcing InfiniteKV: a KV cache that files old tokens as 104-byte searchable records in RAM or on disk instead of deleting them. Mistral-7B answered from token 76,747, 2.3x past its trained window. Colab demo

Open Sourcing InfiniteKV: Extending LLM Context via Searchable KV Cache Offloading InfiniteKV introduces a novel approach to KV cache management by storing aged tokens as compact, 104-byte searchable records in RAM or on…

→ View original source

MTPLX V1: The Swift App For Running & Creating MLX MTP Models (2x TPS Qwen 3.6 27B)

MTPLX V1: Accelerating MLX MTP Model Inference via Native Swift Integration MTPLX V1 introduces a dedicated macOS application built with Swift to streamline the execution and creation of Multi-Token Prediction (MTP) mode…

→ View original source

Some contrived tests comparing the accuracy of different Gemma and Qwen quantizations

Comparative Analysis of Quantization Accuracy: Gemma vs. Qwen Models A community-driven empirical evaluation exploring the impact of various quantization levels on the accuracy of Gemma and Qwen models, specifically focu…

→ View original source

xdna-top: unified NPU+iGPU terminal monitor for Strix Halo (Ryzen AI Max) — finally see the NPU work

If you're running local models on a Ryzen AI Max / Strix Halo box, you've probably noticed it's hard to see what the NPU is actuallydoing. amd-smi is still broken on gfx1151 (ROCm #

→ View original source

As we know Minimax M3 is just going to be open sourced in few days and because of that I was surfing on internet searching for its scores and I found out pretty interesting results. Is Minimax M3 really that good in agentic stuff and in coding? Is it better than older gpt models?

Has anyone personally compared the Minimax M3 model against other proprietary models to determine its relative performance tier? I am trying to understand where it currently ranks in

→ View original source

Any chances for a 12B diffusion Gemma?

Evaluating the Potential for a Diffusion-Based Gemma 12B Architecture A technical discussion emerges regarding the feasibility and performance advantages of implementing a diffusion-based generation mechanism within the …

→ View original source

Can't seem to enable reasoning in llama.cpp

Challenges in Enabling Reasoning Capabilities within llama.cpp A user report from the LocalLLaMA community highlights technical difficulties in activating "reasoning" or "thinking" modes when deploying specific models, s…

→ View original source

Refiner: Robotics library from the ex-Hugging Face pre-training team

Refiner: A New Robotics Data Refinement Library from Former Hugging Face Pre-training Experts A team of former Hugging Face pre-training specialists has introduced "Refiner," a specialized library designed to streamline …

→ View original source

How can Deepseek v4 top the coding leaderboards and still sit 8 months behind the frontier?

Analyzing the Performance Gap: DeepSeek v4's Coding Dominance vs. General Frontier Lag A critical analysis of the discrepancy between DeepSeek v4's top-tier coding benchmarks and its overall general-purpose capabilities …

→ View original source

I wired a fully offline voice loop to Ollama + LM Studio — 100% CPU, no GPU, nothing leaves your machine (Silero VAD + Parakeet STT + Supertonic TTS 3)

Building a Fully Offline Voice Loop: Local CPU-Based Integration with Ollama and LM Studio A technical implementation of a privacy-centric, 100% local voice-to-voice pipeline utilizing ONNX runtimes to enable voice inter…

→ View original source

Remove padding and multiple D2D copies for MTP by gaugarg-nv · Pull Request #24086 · ggml-org/llama.cpp

Optimizing MTP in Llama.cpp: Removing Padding and Streamlining Device-to-Device Copies A recent pull request (#24086) by gaugarg-nv proposes performance improvements for Multi-Token Prediction (MTP) in the llama.cpp repo…

→ View original source

DeepMind Just Dropped "DiffusionGemma" — Text Generation via Image-Style Diffusion Model

Another open weight model got dropped today, this one's from DeepMind, seems like a good day for the OSS geeks. Released under Apache 2.0 Instead of generating text sequentially t

→ View original source

FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention

Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving. In this report, we propose Lookahead Sparse Att

→ View original source

How long do you think it will take for the stock market to notice that Apple and Microsoft announced at the same time that they're all-in for local AI?

The Strategic Pivot Toward Local AI: Analyzing the Convergence of Apple and Microsoft A growing industry consensus suggests that both Apple and Microsoft are aggressively shifting their hardware and software strategies t…

→ View original source

Can you really replace paid models with a local model?

Evaluating the Viability of Local LLMs as Replacements for Frontier Closed-Source Models A critical examination of the current gap between local, open-weight models and proprietary frontier AI, questioning whether the pe…

→ View original source

Local LLms releases

Analyzing Trends in Local LLM Release Frequency: A Quantitative Perspective A recent community analysis of Local Large Language Model (LLM) release patterns suggests that the peak volume of model deployments occurred las…

→ View original source

Anyone gotten Gemma 4 12B (unified audio) to actually attend to speech with a large system prompt?

Addressing Attention Drift in Gemma 4 12B Unified Audio Models with Large System Prompts Developers are reporting a potential degradation in audio attention when utilizing the Gemma 4 12B unified model for single-pass au…

→ View original source

Without open llm competition, closed source LLM companies will become insatiable.

The Critical Role of Open-Source Competition in Preventing LLM Monopolies A critical perspective on the necessity of open-source Large Language Models (LLMs) as a market counterbalance to prevent closed-source providers …

→ View original source

Furiosa AI selling inference chip to consumer market will be a game changer to local llm

FuriosaAI's Renegade Chip: A Potential Paradigm Shift for Local LLM Inference South Korean startup FuriosaAI is developing a high-performance inference chip utilizing TSMC 5nm technology and HBM3 memory, posing a signifi…

→ View original source

OSCAR RotationZoo - Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

OSCAR RotationZoo: Implementing Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization A new optimization technique called OSCAR (RotationZoo) introduces offline spectral covariance-aware rotation to …

→ View original source

Text-to-Speech (TTS) Benchmark Revamped with Objective Standards and Blind Voting (46 models and counting)

Revamping Text-to-Speech (TTS) Benchmarking: Implementing Objective Standards and Blind Voting A new community-driven initiative is transforming how Text-to-Speech (TTS) models are evaluated by introducing a blind voting…

→ View original source

ggml-webgpu: Improve prefill speeds for k-quants + refactor matmul for Q4/Q5/Q8 and k-quants by yomaytk · Pull Request #24225 · ggml-org/llama.cpp

Optimizing WebGPU Prefill Performance: Matrix Multiplication Refactor for K-Quants in llama.cpp A recent pull request to the ggml-org/llama.cpp repository introduces significant performance gains for k-quantized models o…

→ View original source

2X tk/s (from 19.4 -> 38.1 tk/s on 1 x MI50) Playing with a hypothesis like speculative decoding.. but instead of an additional side model, exploiting that I can run multiple computations side-by-side AS IF I had Qwen3.6-27B loaded twice in memory - small quants don't use all the available compute.

Optimizing Throughput on AMD MI50: Achieving 2x Token Speed via Parallel Computation Hypothesis A developer has demonstrated a significant performance increase on the AMD MI50 GPU, boosting token generation from 19.4 tk/…

→ View original source

When every other post is an AI generated benchmark report, a question about the best model, or a slop-coded application or engine that pretends to be groundbreaking

The Proliferation of AI-Generated Noise: Addressing the "Slop" Crisis in Local LLM Communities A growing concern within the LocalLLaMA community highlights a decline in original technical discourse due to the saturation …

→ View original source

mtp: support for gemma-4 E2B and E4B assistants by max-krasnyansky · Pull Request #24282 · ggml-org/llama.cpp

Integration of MTP Support for Gemma-4 E2B and E4B in llama.cpp A new pull request in the ggml-org/llama.cpp repository introduces Multi-Token Prediction (MTP) support for the Gemma-4 E2B and E4B model variants, signific…

→ View original source

Me: Arguing with an AI bot who just posted something on this sub about Llama 3.1.

Addressing the Proliferation of Autonomous Bot Interactions in LLM Communities A discussion within the LocalLLaMA community highlights the growing friction between human users and AI bots posting outdated information, em…

→ View original source

Quick note on the QAT of recent

Technical Analysis: Quantization Discrepancies in Recent Google Model Releases A technical review of recent quantization implementations reveals critical misalignment and hardcoded errors in Google's quantization process…

→ View original source

Qwen3.6-35B-A3B tool calling benchmark: ByteShape vs. Unsloth GGUFs, KV cache quants & long context performance

Evaluating Qwen3.6-35B-A3B: Tool Calling Performance Across ByteShape and Unsloth GGUF Quantizations A technical exploration into the qualitative performance of the Qwen3.6-35B-A3B model, specifically comparing ByteShape…

→ View original source

Friends from the localllama community, if you love local llm, don't participate in the IPO (spaceX, OpenAI, Anthropic)

The Economic Impact of Frontier AI Labs on Local LLM Infrastructure A critical perspective from the LocalLLaMA community argues that the aggressive expansion and valuation strategies of frontier AI laboratories are drivi…

→ View original source

OpenEnv is now owned by HF, Torch, Prime Intellect, Unsloth, Modal, Mercor, and more! Use it for training agents.

OpenEnv Transitions to Multi-Organization Governance to Accelerate Open-Source Agent Training OpenEnv, a specialized framework for creating agentic execution environments, has announced a transition to a committee-led go…

→ View original source

[Benchmark] DFlash Speculative Decoding + KV Cache Compression on RTX 5090 — 3.26x Speedup

Benchmarking DFlash Speculative Decoding and KV Cache Compression on NVIDIA RTX 5090 Recent benchmarks conducted on the NVIDIA RTX 5090 demonstrate a significant performance increase in local LLM inference, achieving a 3…

→ View original source

QATs Q4_0 from Google have more precision than Q4_K_XL from Unsloth (at least some)

Comparative Analysis of Quantization Precision: Google QAT Q4_0 vs. Unsloth Q4_K_XL A preliminary comparison of Quantization-Aware Training (QAT) implementations for Gemma 4 suggests that Google's Q4_0 weights may offer …

→ View original source

Galaxy Z Fold6 as a local inference node — llama.cpp/Vulkan, homelab telemetry, SHA-256 model verification

Galaxy Z Fold6 as a Local Inference Node: Llama.cpp, Vulkan Acceleration, and On-Device GGUF Model Execution This article details the implementation of an Android application, Pocket Node, which utilizes the Galaxy Z Fol…

→ View original source

Qwen 3.6 27B KV cache quant benchmarks: 75 pairs, q8/q6/q5/q4, KVarN, Turbo/TCQ

Benchmarking KV Cache Quantization for Qwen 3.6 27B: Evaluating KVarN, TurboQuant, and TCQ A detailed technical analysis of KV cache quantization methods for the Qwen 3.6 27B model, evaluating the trade-offs between prec…

→ View original source

llama-server router: a model pinned to one GPU still grabs a CUDA context on every card, so it OOMs when my others are full. Am I missing a flag or is this just how it is?

CUDA Context Allocation Issues in llama-server Router Mode A technical report on a memory management issue within llama-server where models pinned to specific GPUs still trigger CUDA context allocations across all availa…

→ View original source

NVFP4 on llama.cpp?

Integrating NVFP4 Support in llama.cpp: Enabling Next-Gen Quantization for RTX 50-Series GPUs Recent updates to the llama.cpp ecosystem have introduced support for NVFP4, a specialized 4-bit floating-point format designe…

→ View original source

dvlt.cu: inference engine written from scratch in CUDA/C++ for NVIDIA's DVLT 3D transformer model

dvlt.cu: A High-Performance CUDA/C++ Inference Engine for NVIDIA's DVLT 3D Transformer A new lightweight, standalone inference engine, dvlt.cu , has been developed to run NVIDIA's DVLT 3D transformer model. By bypassing …

→ View original source

Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding

Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding A new framework called Domino introduces a method to decouple causal modeling from the autoregressive drafting process, achieving si…

→ View original source

Qwen 3.6 27B MTP - Adding spec-type and spec-draft-n-max is dropping tps and reducing GPU utilization

Performance Degradation in Qwen 3.6 27B MTP: Impact of Speculative Decoding Parameters on Throughput A technical report indicates that enabling specific speculative decoding configurations—namely --spec-type draft-mtp an…

→ View original source

Qwen3.6-35B-A3B-Uncensored-Claude-4.6-Genesis-APEX-GGUF

Qwen3.6-35B-A3B-Uncensored-Claude-4.6-Genesis-APEX-GGUF: A High-Performance Delta Merge A new GGUF quantization of a specialized delta merge model is now available, combining the Qwen3.6-35B architecture with Claude 4.6 …

→ View original source

DeepSeek V4 Flash is amazing! (WIP llama.cpp PR #24162)

DeepSeek V4 Flash Integration Begins in llama.cpp via Pull Request #24162 Early-stage support for the DeepSeek V4 series is currently being implemented in the llama.cpp ecosystem, enabling initial local experimentation w…

→ View original source

Gemma 4 with quantization-aware training

Google Releases Gemma 4 with Quantization-Aware Training (QAT) Google has expanded the Gemma 4 ecosystem by introducing models optimized via Quantization-Aware Training (QAT), significantly reducing deployment overhead w…

→ View original source

Unsloth just dropped MTP GGUF weights for Gemma 4!

Unsloth Releases MTP GGUF Weights for Gemma 4 Model Suite Unsloth has expanded its optimization library by providing GGUF-formatted weights for the Multi-Token Prediction (MTP) variants of Google's Gemma 4, covering mult…

→ View original source

finally

Insufficient Data for Technical Analysis: Reddit Thread "finally" A report on a community submission within the r/LocalLLaMA subreddit regarding a development labeled "finally," though the source provides no technical sp…

→ View original source

OpenLumara - A different kind of AI agent, written from scratch, not vibecoded. Extremely token-efficient, super small system prompt, made for local models. Everything is modular.

OpenLumara: A Modular, Token-Efficient Framework for Local AI Agents OpenLumara introduces a streamlined approach to AI agent architecture, prioritizing manual coding and token efficiency over "vibecoding" to enable high…

→ View original source

sycl : port multi-column MMVQ from CUDA backend (~45% speculative decoding speedup on Intel Arc) by masonmilby · Pull Request #21845 · ggml-org/llama.cpp

SYCL Integration of Multi-Column MMVQ: Accelerating Speculative Decoding on Intel Arc GPUs A significant performance optimization has been merged into the llama.cpp repository, porting the multi-column Mixed-Precision Ve…

→ View original source