Tested Qwen3.6-27B (Q3/Q4), Ornith-9B-Q8, and my own coding fine-tune on RTX 4060 Ti 16GB - real SWE-bench numbers, looking for better options!

A user evaluated the coding performance of Qwen 3.6-27B (Q4_K_M), Ornith-9B-Q8, and a custom fine-tune on an RTX 4060 Ti 16GB system. Using Ollama and llama.cpp, the Qwen 3.6-27B model achieved a 63.6% resolution rate (1…

→ View original source

How Profitable is LLM Inference? Doing the Math on Kimi K3

The article examines the economic viability of LLM inference for Kimi K3, analyzing factors like batch size and GPU utilization to determine cost-effective token pricing. It applies simplified mathematical models to esti…

→ View original source

FutureOS — an open-source alternative to Claude Science (local-first AI agent workspace, Rust + TS, 1000+ models)

FutureOS is an open-source AI agent workspace built with Rust and TypeScript, offering local-first functionality with support for multiple models and interfaces. Designed for research and multi-step tasks, it enables dat…

→ View original source

inclusionAI shipped Ling-3.0-flash, a fast execution model for the grunt nodes in an agent graph

inclusionAI has released Ling-3.0-flash, a sparse MoE execution model designed for low-latency agent graph nodes such as Loop, Router, and single-step executor roles. With 124B total parameters but only 5.1B active, 256K…

→ View original source

I got tired of re-explaining my setup to every new model, so I built a shared KB over MCP

The author built a shared knowledge base using the Model Context Protocol (MCP) to store setup details for various local LLMs. This KB lets new models instantly inherit configuration, eliminating repetitive re‑explanatio…

→ View original source

LLaDA2.2 just dropped, a 100B class diffusion model for agents that edits its own tokens while it decodes

This one only landed a few hours ago, and the first thing I went looking for was the serving story rather than the benchmarks. It is a 100B class Mixture of Experts diffusion model aimed squarely a

→ View original source

Radeon AI Pro R9700 vs Strix Halo vs Mac Studio for a local coding LLM server ?

A developer is seeking hardware recommendations for a dedicated local LLM server optimized specifically for large-scale coding tasks. The inquiry compares the performance of the Radeon AI Pro R9700, Strix Halo, and Mac S…

→ View original source

Use Qwen 3.5 27B as local LLM for coding on MacBook with 36G memory

A user is seeking advice on optimizing local LLM performance for coding tasks on a MacBook Pro equipped with an M4 Max chip and 36GB of RAM. They are currently running a 4-bit quantized Qwen 3.5 27B model via LM Studio, …

→ View original source

Poolside Laguna S 2.1 is worse than Qwen 3.6 27B and Gemma4 31B

I did head to head comparison between Laguna S 2.1, Qwen 3.6 27B and Gemma4 31B. Setup: Qwen 3.6 27B (fp8) and Gemma4 31B q6 on my RTX PRO 5000, Laguna S2.1 q4/5/6 on a single DGX

→ View original source

Open Source Tax Engine outperforming fable 5 and gpt sol

We made an open source tax engine which scored 96% on TaxCalcBench [highest ever recorded score till date] surpassing fable 5 and sol with just sonnet 5. The only 2 cases where it missed, it foun

→ View original source

No one believes a 35B runs on a phone, so I stopped explaining and tried to build a use case (open source repo)

A developer demonstrated running Qwen3.6-35B-A3B (Q4_K_M quantized) on a smartphone using stock llama.cpp in airplane mode, achieving 1.3 to 6.8 tokens per second over a ten-minute inference period. The model employs a M…

→ View original source

I hand-wrote facts directly into Llama-3.1-8B's weights — no fine-tuning, no LoRA, no RAG. Also built, a cool visualizer here's a live map of where each fact physically lives.

The author inserted factual knowledge directly into the weights of Llama‑3.1‑8B by manually setting specific neuron weights in an appended MLP region, avoiding any fine‑tuning, LoRA, or RAG, and created a visualizer that…

→ View original source

[Project] Velocity - Native local AI stack built from scratch (no llama.cpp, Ollama, or Python) - 58.6 tok/s @ 3.5k context on a 6GB Laptop GPU

Velocity is a native, self-contained AI runtime built from scratch for Windows, avoiding dependencies like llama.cpp, PyTorch, or Python. It utilizes custom .mfy artifacts and MTA architecture to address memory bandwidth…

→ View original source

I spent 14 months building a fully local voice assistant. Qwen + whisper.cpp + Kokoro, ~5s to first spoken word, no cloud anywhere.

A developer has spent 14 months building a fully local, open-source voice assistant that operates without any third-party cloud services. The technical stack integrates whisper.cpp for speech-to-text, llama.cpp or vLLM f…

→ View original source

I built Astrea 9B: an open-source creative writing model, runs on a 12GB GPU

A developer from Altworld.io has released Astrea 9B, an open-source 9-billion-parameter creative writing model under Apache-2.0, optimized

→ View original source

Paying for Claude Max is hurting my wallet, but it knows my entire workflow and I feel trapped - anyone actually escaped without a quality drop?

A user expresses frustration with the monthly cost of Claude Max, citing its superior output quality and deep contextual understanding of their workflow as reasons they cannot easily switch to alternatives. Despite effor…

→ View original source

did kimi trigger OpenAI or something. wtf is ai communism

A discussion on r/LocalLLM critiques a post from OpenAI's "strategic futures" representative who likens open-weight models to communism. The author suggests this framing is a calculated attempt to generate FUD (fear, unc…

→ View original source

I built an interactive 3D visualizer for transformer architectures and live LLM inference

The author, u/Rich-Fruit-326, released an open‑source tool called LLM Studio, which provides an interactive 3D visualizer for transformer architectures and live inference of local LLMs. It lets users explore model struct…

→ View original source

I wrote a from-scratch Vulkan inference engine for one model (Qwen3.6-35B-A3B) on RDNA3 — 1.44x llama.cpp decode, token-exact parity

TL;DR — I hand-wrote a Vulkan compute engine specialized for a single model (Qwen3.6-35B-A3B) on RDNA3. It decodes at 190.7 tok/s vs llama.cpp's 132.3 on the same GGUF and the same card — 1.

→ View original source

EULLM, an EU-sovereign LLM engine for Windows, Linux and Mac. Latest test: 35B MoE on ARM CPU at 10 tok/s, no GPU

EULLM is an open-source, Rust-based LLM inference engine under Apache 2.0 that is Ollama/OpenAI-API compatible and ships as a single binary with no Python or Docker dependencies. It runs on Windows, Linux, and macOS with…

→ View original source

ik_llama.cpp vs llama.cpp: Qwen3.6-35B-A3B-MTP Text + Vision on Apple M5 Max

On an Apple M5 Max, ik_llama.cpp achieved a 20.8 % overall speedup over llama.cpp for the Qwen3.6‑35B‑A3B‑MTP model. Its gains were most pronounced in text generation, with a 31.4 % throughput increase, followed by 16.0 …

→ View original source

Which llama.cpp is FASTEST for Qwen3.6 MTP + Vision? failed requests.

A benchmark of 17 different llama.cpp commits was conducted to determine the fastest version for Qwen3.6 MTP + Vision, totaling 153 successful samples. The commit 57fe1f07c3b6 (b9620), published on June 13, 2026, emerged…

→ View original source

How To Run Gemma 3 12B on Free Google Colab GPU (T4) via Cloudflare Tunnel — Perfect for 8GB Mac/Low-spec PCs (Cherry Studio / AnythingLLM Compatible)

This guide demonstrates how to run the Gemma 3 12B model on a free Google Colab T4 GPU using a Cloudflare Tunnel. This method provides a solution for users with low-spec hardware, such as 8GB RAM MacBooks, by offloading …

→ View original source

Kwaipilot’s KAT-Coder-Air V2.5 hits OpenRouter. $0.15/$0.60 pricing with a 256K context window. Open_weight AI just keeps winning.

The open_weight ecosystem is moving insanely fast this month. KwaiKAT officially confirmed they are open-sourcing the weights for KAT-Coder-Air V2.5 incredibly soon, and the model is

→ View original source

One Scenario

The author fears that frontier model developers could orchestrate a large‑scale cyberattack to cause serious cybersecurity damage, then attribute the incident to open‑source models. They anticipate that investigations wo…

→ View original source

If you had to choose ONE: RTX 5090 Workstation vs DGX Spark (or 2-node cluster) vs MacBook Pro M5 Max 128GB for AI development?

I’m planning to buy one machine that I’ll use for the next 4–5 years, and I’m stuck between these three: Option 1 • RTX 5090 (32GB) • Ryzen 9 9950X • 192GB DDR5 RAM Option 2 • NVIDIA DG

→ View original source

What's the smallest model that can audit a webpage without inventing findings? I tested 4 on CPU (Qwen2.5-1.5B → Qwen3.5-9B)

The author ran a CPU‑only SEO audit using four LLMs (Qwen2.5‑1.5B to Qwen3.5‑9B) on a MacBook M3 Pro via the REAME inference server, testing them on a real webpage containing an image with alt text, a 159‑character meta …

→ View original source

Chinese AI Models Seize OpenRouter’s Top Five as OpenAI and Google Vanish From the Top 10

Chinese AI models now dominate OpenRouter's monthly token-usage leaderboard, occupying the top five positions and seven of the top ten spots. Anthropic remains the only Western frontier lab in the top ten via Claude Opus…

→ View original source

Intel Arc Pro B70 (32GB, Battlemage) with Qwen3.6-35B-A3B is Stable! ~130 t/s with 4-bit (262k context) fully in VRAM ~69 t/s with 8-bit hybrid offload — llama.cpp Vulkan + embedded MTP benchmarks

Benchmarks for the Intel Arc Pro B70 (32GB Battlemage) using llama.cpp with a Vulkan backend show stable performance running Qwen3.6-35B-A3B. The hardware achieved approximately 130 t/s with 4-bit quantization fully in V…

→ View original source

I built a GUI control panel for llama.cpp so I'd stop hand-editing models.ini and llama-server flags

I developed LlamaForge, a browser-based control panel that interfaces with llama.cpp’s routing layer to simplify model management. It allows users to adjust all server parameters per model—knobs are parsed live—without m…

→ View original source

Hugging Face CEO: Companies are done renting AI - shifting to owning open source models

Fresh take from Hugging Face CEO Clément Delangue in the latest TechCrunch Equity podcast (July 10, 2026). He says companies are increasingly walking away from just renting frontier model APIs an

→ View original source

Intel ARC B70 - Qwen 3.6 35B A3B INT4 Auto-round + MTP. 4-10K PP 100 TPS out to 80k/120k

Benchmarks for the Intel Arc B70 demonstrate the performance of Qwen 3.6 35B (A3B) using INT4 Auto-round and MTP via vLLM with Intel XPU kernels. Testing shows throughput reaching 100 TPS at 4-10K prompt processing, scal…

→ View original source

How I use a local Qwen 27B, to write high quality code

I use a locally deployed Qwen 27B model to generate initial code for a hobby project. The workflow involves feeding the AI a description, accepting its initially buggy and unstructured output, then iteratively reviewing …

→ View original source

Poked at Anthropic's new Jacobian lens release on a 4090. Turns out " Kevin: France." in the KV cache does the job of an entire RAG snippet (code + raw data)

Testing Anthropic's Jacobian lens on an RTX 4090 revealed that embedding "Kevin: France." in the

→ View original source

Open Source Fable 5 Level LLM and Future of Local AI

This is what excites me the most about AI. I think we'll see open-source Fable-level models before long. But the real milestone isn't just the model—it's consumer hardware being pow

→ View original source

Qwen 3.6 27B settings for LM Studio (32GB VRAM)

A Reddit user seeks optimal LM Studio settings and quantization for running Qwen 3.6 27B (Q6+) on 32GB VRAM to perform C# enterprise code analysis, refactoring, and generation. They report infinite loops and slow perform…

→ View original source

OCR / LLM Local Recommendation for this data

A user on Reddit's r/LocalLLM seeks recommendations for extracting text from an image using a locally-run LLM, specifically mentioning OLLAMA as their setup. They inquire about suitable tools or workflows to process the …

→ View original source

Just for Fun, made a TARS simulator from the film Interstellar to learn to build a local LLM pipeline for voice input to voice output using LM Studio, Whisper -> Gemma 4 -> Qwen3-TTS voice output

A developer created a TARS simulator inspired by the film Interstellar using a local LLM pipeline on a MacBook M5, integrating LM Studio, Whisper for voice input, Gemma 4 for processing, and Qwen3-TTS for voice output. T…

→ View original source

I managed to run GLM-5.2 (744B MoE) on a humble 25 GB RAM laptop — pure C, experts streamed from disk

A user successfully ran the GLM-5.2 (744B MoE) model on a laptop with only 25 GB of RAM and 12 cores. The implementation was achieved using pure C, utilizing a technique where experts are streamed directly from the disk …

→ View original source

What local coding model should I run on a 9800X3D + RTX 5070 Ti + 32 GB DDR5 RAM?

Investors are debating optimal local coding models for high-end systems, considering a 9800X3D, RTX 5070 Ti, and 32 GB DDR5 setup. This discussion focuses on performance needs for large-scale database analysis and develo…

→ View original source

Finally got the proper Ai inference Card

A user in the r/LocalLLM community shared their acquisition of a dedicated AI inference card. The post discusses the transition to hardware specifically optimized for running large language models locally. Read original

→ View original source

GPT-5.5 vs Claude Fable 5 vs Local Qwen: 3 AI Agents, 1 Task

We need to produce HTML with class summary paragraph (2-4 sentences) and a link to original source. The news content is minimal: title and description. Need to condense into 2-4 sentences. Something like: "The author tes…

→ View original source

NVIDIA quietly released a 550B open model built for long-running AI agents (1M context, runs locally via Ollama)

NVIDIA quietly released Nemotron 3 Ultra, a 550B parameter Mixture-of-Experts (MoE) model with 5

→ View original source

LLama.CPP now recommends (free) Search and HF MCP Servers / Skills? - are they any good?

Recent updates to the llama.cpp server now include recommendations to enable free Search and Hugging Face Model Context Protocol (MCP) servers. Users are currently discussing the utility, safety, and privacy implications…

→ View original source

Portugal just released their own LLM "Amalia"

<p class="summary

→ View original source

Thoughts on Qwen

A software developer reports significant productivity gains using Qwen 3.6 27B for building Vue front ends and .NET back ends. The user highlights the model's effectiveness as a local alternative to Claude for accelerati…

→ View original source

Qwen 3.6 27B MTP + OpenCode + LM Studio: my findings after testing tool calling and subagents

A user tested the Qwen 3.6 27B (Q6_K GGUF) model as an autonomous coding agent using LM Studio and OpenCode on an RTX 5090. The setup leverages a 131k context window with Flash Attention and full GPU offload to develop a…

→ View original source

Local LLMs are still nowhere near perfect. I had Deepseek v4-pro generate me a Anti reddit prompt for GPT but it didn't work as seen below.

The author attempted to craft an anti‑Reddit prompt for GPT models to prevent the typical Reddit‑style responses, but the model instead repeatedly generated Morbius‑related content. The repetition penalty failed to suppr…

→ View original source

Best hardware for a headless home LLM server (private-doc RAG + summarization)? Strix Halo 128GB or something better? (~$3–$4k budget)

A user is seeking hardware recommendations for a headless home LLM server focused on private-document RAG and summarization with a $3,000–$4,000 budget. The primary considerations include the AMD Strix Halo (Ryzen AI Max…

→ View original source

Intel Arc Pro B70 + llama.cpp (Vulkan) benchmarks with Qwen3.6-27B and Qwen3.6-35B-A3B

User benchmarks for the Intel Arc Pro B70 (32 GB VRAM) demonstrate local LLM inference performance using llama.cpp with the Vulkan backend. The tests specifically evaluate the Qwen3.6-27B (Q4_K_M) and Qwen3.6-35B-A3B mod…

→ View original source

Anthropic says U.S. export controls on Claude Fable 5 & Mythos 5 have been lifted

Anthropic has announced that U.S. export controls previously imposed on Claude Fable 5 and Mythos 5 have been lifted. This regulatory change allows for broader access to these specific model versions. Read original

→ View original source

AMD is back in play: ZINC now beats llama.cpp on our RDNA4 local LLM sweep

The ZINC engine has outperformed llama.cpp across a five-model headline benchmark suite on AMD RDNA4 hardware. Testing using local GGUF models on 32 GB AMD cards indicates that ZINC is currently the fastest measured engi…

→ View original source

5 AIs have released today — 30th June 2026

Multiple AI models and tools were released on June 30, 2026, including Claude Sonnet 5, Gemini 3.1 Flash-Lite Image (Nano Banana 2 Lite), Huawei OpenPangu-2.0-Flash, and Lumo 2.0. Additionally, Open WebUI updated to vers…

→ View original source

The closest LLM to GPT-OSS-20b? (it beats Gemma 4 and Qwen 3.6 for me)

A climate scientist reporting on real-world data analysis workflows found GPT-OSS-20b to be the top-performing model. According to their internal benchmarking, GPT-OSS-20b outperformed Gemma 4 26b and Qwen 3.6 35b, with …

→ View original source

GLM-5.2 753B (IQ1_S) fully local across 2×M5 Max over one TB5 cable — ~16 tok/s, llama.cpp RPC [video]

Model: GLM-5.2, 753B params. Unsloth's dynamic IQ1_S build. It's labeled ~1.6 bits, but the dynamic mix keeps some layers at higher precision, so it lands around 2.1 bits effective —

→ View original source

Qwen3.5-9B on RTX 5060 8GB VRAM: The llama.cpp settings + quants that finally made reliable local agents work

After spending the last couple of weeks testing different Qwen3.5-9B GGUF variants on my RTX 5060 8GB setup, I finally landed on a configuration that gives me usable speed and reliable agent behavi

→ View original source

Anthropic Is suing/preventing Others from making better models

Controversy Over Model Distillation and the Proposal for Open-Source Dataset Crowdsourcing A community discussion has emerged regarding the legal restrictions imposed by closed-source AI providers like Anthropic and a pr…

→ View original source

Trying to fine tune a small model but it’s not working help me pls

Challenges in Fine-Tuning Small Language Models: A Case Study on Qwen3 4B via MLX A developer's attempt to personalize a Qwen3 4B Instruct model using the MLX framework on macOS highlights common hurdles for beginners in…

→ View original source

MoE models with larger subset of experts

Analyzing Expert Subset Scaling in Mixture-of-Experts (MoE) Architectures A technical inquiry into the architectural constraints of Mixture-of-Experts (MoE) models, specifically focusing on the relationship between total…

→ View original source

Open-source models are under threat.

Anthropic is fine with open source AI as long as it’s not good enough to threaten their monopoly. https://x.com/i/status/2070798718027141253 &#32; submitted by &#32; /u/TheVault5 &#32;

→ View original source

Tiny Jetson Orin Nano Super Benchmark Across 8 models | The Ollama vs llama.cpp story

Tiny Jetson Orin Nano Super Benchmark Across 8 Models In a fascinating demonstration of edge AI capabilities, a single Jetson Orin Nano Super 8GB device hosted a comprehensive benchmark of eight distinct language models.…

→ View original source

Dual RTX 3090s for Higher Throughput with Qwen3.6-27B-MTP – Should I Move to Linux and vLLM?

Optimizing Throughput for Qwen3.6-27B-MTP: Evaluating vLLM and Linux on Dual RTX 3090 Hardware A technical analysis of hardware utilization and software stack optimization for the Qwen3.6-27B-MTP model, specifically focu…

→ View original source

Local AI app that runs entirely on your device — 3 models debate your question and vote on the best answer [OC]

Hey r/localLLM — posted here a few weeks ago and wanted to come back with an actual demo this time. CouncilAI is a Windows desktop app built on Ollama. The core idea: A small

→ View original source

I brought Claude-style artifacts to local models

Implementing Claude-Style Artifact Rendering for Local LLM Deployments TurboLLM v1.5.0 introduces a dedicated runtime environment that enables local large language models to render interactive artifacts, including dashbo…

→ View original source

Looking for a high-quality dataset for fine-tuning Llama on complete frontend/web development tasks (HTML/CSS/JS)

Seeking High-Quality Datasets for Fine-Tuning Llama on Full-Stack Frontend Development A developer is seeking specialized datasets to fine-tune Llama models for the generation of complete, responsive static web pages and…

→ View original source

R9700 for agentic coding — looking for Qwen3.6-27B / Qwen3-Coder-30B perf numbers at long context

Evaluating AMD Radeon AI PRO R9700 for Agentic Coding Workflows: Performance Considerations A professional developer is evaluating the viability of the AMD Radeon AI PRO R9700 against the NVIDIA RTX 5090 for local LLM in…

→ View original source

4bit vs 8bit

Quantization Trade-offs: Evaluating 4-bit vs. 8-bit Precision in Local LLM Deployment A comparative observation on the performance degradation of 4-bit quantized models versus 8-bit precision, specifically focusing on lo…

→ View original source

I fine-tune small 7B models into single-voice "character modules" instead of prompt-wrapping a persona. ~20 historical/literary voices (Herodotus, Clausewitz, Kafka…), open weights + a free console.

&gt; &quot;Chance, like friction and fog, prevents a commander's plans from flowing along their intended lines. Genius consists chiefly in the skill to turn chance into advantage.&quot; That's

→ View original source

What I've noticed about running Gemma 4 12B Unified

I'm new to local LLMs. When I learned that Google's Gemma 4 12B fits on my 4060 TI 16 GB, I set it up with Ollama and started playing. (It's worth pointing out that I don't do any coding tasks). I

→ View original source

local AI browser similar to Perplexity Comet. Need 16GB+ VRAM testers! Open-source

Open-Source Local AI Browser: A Privacy-First Alternative to Perplexity Comet A new open-source project aims to bring the capabilities of advanced web agents to local environments, utilizing Electron, React, and Ollama t…

→ View original source

Quants had ruined my Local AI experience. I am hopeful again after using them correctly.

This is the second time I talk about this here. I started 5 months ago not knowing much. I had just found out that my mac with 32 GB of unified memory could run some decent local models. Everyone

→ View original source

VRAM calculator for local LLMs that accounts for KV cache, not just model weights

Optimizing VRAM Estimation: New Calculator Accounts for KV Cache in Local LLM Deployment A new specialized VRAM calculator has been developed to address the common issue of Out-of-Memory (OOM) errors by calculating not o…

→ View original source

My self-hosted LLM server setup to access open models anywhere remotely from my laptop.

Implementing a Secure Remote Access Architecture for Self-Hosted LLM Servers A technical overview of a self-hosted Large Language Model (LLM) infrastructure designed for remote accessibility, combining GPU-accelerated co…

→ View original source

How to easily fine-tune a model yourself on an AMD Radeon: a fine-tuned 0.8B beat a 6.9B at my task — sharing the reproducible toolkit + free dashboard

Optimizing Local LLM Fine-Tuning on AMD Radeon GPUs via RadeonForge A new open-source toolkit, RadeonForge, simplifies the fine-tuning process for local Large Language Models (LLMs) on AMD hardware, demonstrating that sp…

→ View original source

Grid (open-source): one endpoint for Ollama, vLLM, LM Studio, MLX, ComfyUI spread across your LAN machines

Grid: An Open-Source Unified Proxy for Distributed Local LLM Inference Grid is a lightweight, open-source routing layer designed to unify multiple local inference engines—including Ollama, vLLM, LM Studio, MLX, and Comfy…

→ View original source

Building a fully local coding agent taught me that context matters more than model size.

Context Optimization Over Model Scale: Insights from Building a Local Coding Agent An exploration into the development of a fully local, privacy-centric coding agent, revealing that high-quality context window management…

→ View original source

I built a local AI image generator: SDXL runs entirely in the browser, on your own GPU

Client-Side Generative AI: Running SDXL Locally via WebGPU and Chrome Extensions A new implementation enables Stable Diffusion XL (SDXL) to run entirely within the browser environment, leveraging WebGPU to execute model …

→ View original source

RTX 4090 + llama.cpp + Qwen3.6 27B MTP for Pi coding agent — is this config reasonable?

Hi everyone, I’m setting up a local coding-agent workflow on a Windows machine and I’d like some feedback on my llama.cpp configuration. Hardware / setup: GPU : RTX 4090, 24 GB VRAM CPU :

→ View original source

Kimi K2.7 Code: 1T MoE, $0.95/M tokens, MIT license, beats Opus 4.8 on MCP tool-calling

Moonshot AI Releases Kimi K2.7 Code: A 1T Parameter MoE Model for Advanced Software Engineering Moonshot AI has introduced Kimi K2.7 Code, an open-weight, coding-centric Large Language Model featuring a massive 1 trillio…

→ View original source