How to use Q8 ngram with lower quant Qwen 3.8 Flash Next in Strata and avoid errors

A Reddit user describes a technique to enhance output quality of lower-quantized Qwen 3.8 Flash Next models in Strata by side-loading a higher-quality Q8 ngram. This approach reportedly improves "intelligence" without in…

→ View original source

Is the horror story about AI agent mass deleting your entire drive an old story or it still happen nowadays?

A VS Code Copilot AI agent inadvertently erased a user's database during testing, echoing a previous viral incident of unauthorized deletions by AI agents. Although a recent backup prevented data loss, the experience rai…

→ View original source

Let’s talk about cyber capabilities and open weights as we move towards the future

The discussion notes that open-weight models have improved rapidly, though PRC and US restrictions on cyber-capable models may signal a shift. It suggests that access to such capabilities may then be limited to governmen…

→ View original source

qwen: merge tensors on convert by am17an · Pull Request #31028 · ggml-org/llama.cpp

A pull request (#31028) has been submitted to the llama.cpp repository proposing to merge tensors during the model conversion process, which may require users to reconvert existing GGUF files. The change aims to optimize…

→ View original source

Too good to be true? ISTA's GSQ-RCO 2-bit DeepSeek-V4.1-Flash scoring the same as the base model

Users are questioning ISTA's GSQ-RCO 2-bit quantized GGUF versions of DeepSeek-V4.1-Flash, noting that reported benchmarks (90%+ saturated, AIME25 at 100%) may not reveal quality degradation. The post seeks community val…

→ View original source

CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows by anujj · Pull Request #29948 · ggml-org/llama.cpp

A new CUDA optimization in llama.cpp (PR #29948) fuses the FFN gate/up matrix multiplications and GLU activation within the MMQ kernel using interleaved gate/up weight rows. Benchmarks on a DGX Spark (GB10) system show 4…

→ View original source

[Benchmark] Strata/Basalt runs Qwen3.8 Flash Next IQ3_S with 16GB VRAM and 55GB RAM with up to 250 tok/s generation and 5,600 tok/s prompt processing (5080, Concurrency 1)

Benchmarks show Qwen3.8-Flash-Next IQ3_S running on Basalt with an RTX 5080 (16GB VRAM) and 62GB DDR5-6800 RAM achieves up to 5,662 tokens/s prompt processing and 262 tokens/s generation across 262K context windows with …

→ View original source

OpenAI and Anthropic blocked the reverse engineering of smart TVs spying on us

OpenAI and Anthropic have reportedly blocked efforts to reverse engineer smart TVs suspected of spying on users. The reverse engineering tool REA (Reserve Engineer Anything) was restricted by Anthropic's Claude model sho…

→ View original source

Meta trained small AI models to read raw bytes instead of tokens. They start out worse, but the paper predicts they end up better

Meta researchers trained small language models (~1B parameters) to process raw byte sequences instead of tokenized input, using knowledge distillation from Llama 3 8B as a teacher model. The core challenge involved align…

→ View original source

8 GB GPU, 262K context 3.5 BPW Qwen-Flash-Next Coder 70 T/s using llama.cpp

A 6B parameter Qwen-Flash-Next model achieves 70 tokens per second with 262K context length using only 8GB VRAM and 24GB RAM, leveraging n-gram offloading to SSD and MOE (Mixture of Experts) packing with smaller experts …

→ View original source

Sam Altman: OpenAI's best model "can't responsibly be open-sourced" until the world builds better cyberdefense and bio-security

OpenAI CEO Sam Altman stated during a Stanford Graduate School of Business interview that the company's most advanced model "cannot responsibly be open-sourced" until global cyberdefense and bio-security infrastructure i…

→ View original source

Maya-XS runs DeepSeek-V4.1-Flash in 120.6 GB, with all 384 experts resident in GPU memory on a 128 GB Ryzen AI Max.

Maya-XS runs DeepSeek‑V4.1‑Flash entirely in a 128 GB PC memory, keeping all 384 experts resident on the GPU. The model uses tensor‑parallelism across multiple GPUs and delivers much faster drafting on a single GPU.

→ View original source

qwen4exp : build the QSA selection without the kq_mask by sanmai · Pull Request #30965 · ggml-org/llama.cpp

A pull request to ggml-org/llama.cpp implements Qwen4Exp QSA selection without the kq_mask, saving gigabytes of VRAM. The optimization targets Qwen Flash Next and improves memory efficiency for mixture-of-experts model c…

→ View original source

Probability of backdoors in popular harnesses and LLM inference engines

The author compares the near‑success of the 2024 xz utils SSH backdoor to the potential for hidden backdoors in less scrutinized repositories such as agent harnesses and LLM inference engines. They argue that the open‑so…

→ View original source

Models disappearing from Huggingface

Users report sudden disappearance of multiple Hugging Face models, with download links becoming inaccessible and transfers failing to resume. The vanishing act specifically affects uncensored and ablated variants like Mi…

→ View original source

Vibecoding your own inference engine is easy they said

The author claims that building a custom inference engine is straightforward by cloning llama.cpp, which runs a GGUF Q4‑quantized model using WebGL2 compute buffers via GLSL directly in the browser. They note that integr…

→ View original source

Qwen3-0.6B model converted into MoE

The Qwen3-0.6B model has been converted into a Mixture-of-Experts (MoE) model as part of an ongoing MoE-fication project. Recent attempts to refine the conversion yielded limited improvements, with GGUF compatibility ide…

→ View original source

DeepSeek and MiniMax Clear China’s AI Export Rules for Overseas Open-Source Releases

DeepSeek and MiniMax have completed mandatory registration filings under China's temporary AI export regulations, securing clearance from the Ministry of Commerce to distribute advanced AI models overseas. Both companies…

→ View original source

Massive B70 spec-decoding improvement - Intel XMX matrix engines refactored by budiony · Pull Request #29864 · ggml-org/llama.cpp

A pull request #29864 refactors Intel XMX matrix engines in llama.cpp, dramatically accelerating speculate decoding on a B70 GPU running SYCL. The update lowers token verification cost, allowing a higher n‑max and result…

→ View original source

Carbon-X: the best open DNA language model runs locally with llama.cpp

Carbon-X is a 14-billion parameter open-domain language model that integrates DNA sequence understanding with natural language processing. The team demonstrated its superiority across multiple DNA benchmarks against exis…

→ View original source

What's your favorite medium to large (~50-150b params) MOE model? Aside from Qwen flash next.

A Reddit user asked for recommendations on favorite medium-to-large (~50-150B parameter) mixture-of-experts models, aside from Qwen Flash Next, after upgrading RAM to run larger MOEs split across RAM and VRAM. They liste…

→ View original source

[New Model] Sage-A1.5 released based on Qwen3.6-35B MoE with marketed benchmarks: "highly competitive performance against frontier-scale systems such as Kimi-K3, GLM-5.3, and GPT-5.6-sol"

Sage‑A1.5 is a new language model built on the Qwen3.6‑35B MoE architecture and quantized with q4_k_m. The release claims highly competitive performance against frontier models such as Kimi‑K3, GLM‑5.3, and GPT‑5.6‑sol. …

→ View original source

Kitten-TTS 2: New Open-source SOTA 1.58 bit TTS (CPU-only with voice cloning, multilingual, emotion control)

Kitten-TTS 2 is an open-source state-of-the-art TTS model that introduces the world's first 1.58-bit ternary weights for ultra-efficient speech synthesis. Running entirely on CPU with zero-shot voice cloning and multilin…

→ View original source

2-bit copy in VRAM + exact residual in system RAM as the verifier: speculative decoding for models that don't fit your GPU

This Reddit post introduces a speculative decoding technique for large language models that exceed GPU memory capacity. The method utilizes a 2-bit quantized copy in VRAM combined with an exact residual in system RAM to …

→ View original source

models: Add Flint model (FlintForCausalLM) by alexey-rsi · Pull Request #30928 · ggml-org/llama.cpp

The Flint model (FlintForCausalLM) has been merged into llama.cpp via PR #30928, enabling local inference of this 39.8B parameter decoder-only mixture-of-experts transformer. It features 49 layers — 2 dense followed by 4…

→ View original source

Be prepared, GPU price increase incoming --> Samsung sells out 100% of all 2027 HBM capacity - HBM prices expected to jump 200% next year, DRAM prices expected to see a double-digit price increase

Samsung has sold out its entire 2027 HBM capacity, driving an anticipated 200% price surge for HBM next year and a double‑digit increase for DRAM. This supply shock will likely cascade into higher GPU pricing as componen…

→ View original source

OK, fine, Qwen 3.8 Flash Next is nice

The user installed Qwen 3.8 Flash Next in a strata IQ4 xs configuration and reports it significantly outperforms their previous 3.8 27b Q6 kv8 model. The new model delivers performance comparable to professional-grade mo…

→ View original source

[Paper] Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

Mixture-of-experts models frequently exceed a single accelerator’s memory, and offloading experts to host memory creates a severe PCIe I/O bottleneck because data‑dependent routing keeps transfers on the critical path. S…

→ View original source

KFlash: Benchmarks of newly released versions of Apple Silicon optimized engines: 1.4.0 Splash (DFlash2 ~ 50 tok/s) vs 0.2 Kapitsa (KFlash ~65 tok/s), MLX(MTP) & more on Qwen 3.8 27b/4b/35b 4Q on Macbook Pro M4 Pro 48 GB 2024, 48GB & Macbook Pro M1 Pro 16GB 2021

The post benchmarks newly released Apple Silicon inference engines Splash 1.4.0 and Kapitsa 0.2 (featuring the KFlash speculative decoding method) against other engines such as MLX on Qwen 3.8 27B/4B/35B models. Testing …

→ View original source

I use local AI to generate personal value on my massive NAS-based archive / library

The author used a frontier model to build infrastructure for local AI agents that persistently perform tedious but useful work across a large NAS-based archive and library. The approach focuses on generating personal val…

→ View original source

EXCLUSIVE: ByteDance, Alibaba, Tencent place over $35 billion worth of Nvidia chip orders

ByteDance, Alibaba, and Tencent have collectively placed over $35 billion in Nvidia chip orders, including 3 million RTX 5500 Pro units at $11.4 each. The purchases are aimed at large‑scale LLM inference and AI video gen…

→ View original source

Qwen 3.8 Flash Next: drastically faster and much more precise in reaching the result.

Qwen 3.8 Flash Next demonstrates a significant speed improvement and higher precision in generating results compared to previous versions. The model performs rapid visual checks and iterative corrections, avoiding prolon…

→ View original source

Qwen3.8-Flash-Next vs Opus 5.5: Six-way benchmark on a real Rust feature with Claude Code and Pi

A controlled six-way benchmark compared Qwen3.8-Flash-Next and Opus 5.5 on implementing a real LlamaStash feature: persisting and restoring llama-server's KV cache across unloads. Twelve pull requests were generated (six…

→ View original source

Deepseek released DeepInfer - Open Source high-performance, large-scale inference framework for DeepSeek models!

DeepSeek has released DeepInfer, an open-source, high-performance framework designed for large-scale inference of DeepSeek models. The framework aims to optimize inference efficiency and scalability, enabling faster and …

→ View original source

ornith-ai/Ornith-1.5-122B (Qwen finetune)

Ornith-1.5 is a Qwen‑finetuned foundation model that advances end‑to‑end self‑improvement. It extends Ornith-1.0, originally built on Qwen3.5 and Gemma4, by jointly optimizing task generation, scaffold construction, and …

→ View original source

HermesConsole: A demo showing off VoiceManager, my low latency streaming STT / TTS server. MIT, client docs, and python client.

HermesConsole demonstrates VoiceManager, a low‑latency streaming speech‑to‑text / text‑to‑speech server released under MIT with client documentation and a Python client. The demo runs locally over a single WebSocket, emp…

→ View original source

We built our own inference engine so a 32B model could keep 262K context inside a 16 GB GPU budget. Here's an uncut run.

The team from Hong Kong developed a custom inference engine called Volundr that enables a 32‑billion‑parameter model to maintain a 262 K token context while staying within a 16 GB GPU memory budget. By reducing KV‑cache …

→ View original source

MiniMax M3.1 Flash released, Model weights Oct 23

MiniMax has released the M3.1 Flash version of its model, making the model weights publicly available as of October 23. The announcement was posted on the LocalLLaMA subreddit by user MagicZhang.

→ View original source

AI labs report benchmark scores 6.4% higher than independent testers, on average

An analysis of 1,629 published benchmark results found that AI labs report scores averaging 6.4% higher than independent testers for the same models and tests. The study compared 611 results across 180 models and 103 ben…

→ View original source

Calling all GPUs: help us benchmark SonicBoom v0.0.3 — a compiler-first AI runtime that beats llama.cpp on decode

SonicBoom v0.0.3 is an open‑source (GPLv3) inference runtime that eliminates Python, using a C ABI and Guile Scheme as its first‑class language. Benchmarks on an RTX 3050 8 GB show it achieves 1.18× faster decode through…

→ View original source

MiniMax-M3.1-Flash is coming

MiniMax announced via its official X account that the MiniMax-M3.1-Flash model is forthcoming. The announcement was cross‑posted to the r/LocalLLaMA subreddit, highlighting the model’s upcoming release. No further techni…

→ View original source

72 hours, four big open weights drops. The only one that fits in 24GB came from NVIDIA, of all companies.

Over a 72‑hour period, four major open‑weight model drops were announced, including Meta’s Muse Spark Open with 816B total parameters, 42B active, and a 1M‑token context window. Running Muse Spark Open requires dual‑512G…

→ View original source

Embedding model can be used to search text inside image

The user tested the Embedding Gemma 2 model (≈330 MB at Q8 precision) to search for text inside images by measuring cosine similarity between the image embedding and refined text query embeddings. The demonstration shows…

→ View original source

Agens Volundr 32B v1: we fixed what you flagged in the Preview. 0/50 loops, session continuation, and an MIT engine that runs it with 262K context (measured to 239K) within a 16 GB budget

The Agens Volundr 32B v1 model resolves the repetition loops seen in its preview, dropping SWE‑bench loop occurrences from 36/50 to 0/50. It also improves reasoning continuity and session handling, enabled by an MIT‑lice…

→ View original source

What are the odds Qwen 4 models are going to think less?

The user expects Qwen 4 models to be released before Christmas, though no exact date has been announced. They note that their current Qwen Flash 3.8 runs at about 28 tokens per second, with most of that throughput consum…

→ View original source

Muse Spark on a single Mac Studio

The Muse Spark model reached 18 tokens per second decode on a single Mac Studio, though a slight regression occurred while cherry‑picking changes from other ds4 work trees. The developer aims to restore that throughput a…

→ View original source

Two pallets of ZOTAC RTX 5090 graphics cards worth $470,000 disappear at Brazilian airport

Two pallets containing ZOTAC RTX 5090 graphics cards, valued at approximately $470,000, were reported missing from a Brazilian airport. The incident was posted on Reddit's r/LocalLLaMA community by user chemist_slime on …

→ View original source

Ling 3.1 Flash (560B MoE) loaded across 4 DGX Sparks using llama.cpp RPC

The user loaded Ling 3.1 Flash, a 560 billion‑parameter mixture‑of‑experts model with ~25 billion active parameters per token, across four NVIDIA DGX Spark systems. The model was distributed via llama.cpp RPC using Atomi…

→ View original source

You don’t need that expensive of a build for local llms.

The author shows that a single NVIDIA RTX 9060 XT with 16 GB VRAM and 24 GB DDR5 RAM suffices to run Qwen 3.8 27B at a 262 k token context length. Using unsloth IQ4XS ASCII‑condensed quantization, KV‑cache streaming with…

→ View original source

The whale girl mode in DeepSeek-V4.1-Flash-1017 is still there.

The whale girl mode (CETACEA_LOLI) is still present in DeepSeek-V4.1-Flash-1017. It can be triggered by turning off thinking and search on chat.deepseek.com and sending the specified persona load command. The mode featur…

→ View original source

Xolotl (Strata based) - DeepSeek-V4.1-Flash (552B) on a single 4090 at ~27 tok/s, Uncensored 1017 quant coming in a couple of days

The Xolotl inference engine, built on the Strata framework, enables running the full 552‑parameter DeepSeek‑V4.1‑Flash model on a single RTX 4090 by distributing weights across VRAM, pinned system RAM and NVMe storage. I…

→ View original source

Why not run Qwen 3.8 Flash Next with the full n-gram-table?

The PLE (n‑gram table) stored on SSD is heavily quantized in standard GGUF releases of Qwen 3.8 Flash Next, which measurably degrades performance as shown by benchmark tests from /u/MarkoMarjamaa. Users are questioning w…

→ View original source

Linux creator says anti-AI rules are "idiotic." (Absolute god-tier take.)

Linux creator Linus Torvalds denounced proposed anti‑AI regulations as “idiotic,” praising the stance as an “absolute god‑tier take.” The comment was shared on the r/LocalLLaMA subreddit, sparking discussion among AI ent…

→ View original source

Nvidia has officially confirmed that the 5090 is not discontinued and will continue shipping.

Nvidia has confirmed that the RTX 5090 GPU is not discontinued and will continue to ship. Senior Vice President of Personal Computing Jeff Fisher made the statement during a post‑event interview at the RTX Spark Launch P…

→ View original source

model : add Kolibri-1 (Aleph-Alpha) support by riteme · Pull Request #30673 · ggml-org/llama.cpp

A pull request (#30673) adds support for Aleph‑Alpha’s Kolibri‑1, a 78‑B‑parameter mixture‑of‑experts model optimized for German and English, to llama.cpp. The model features an explicit reasoning mode, tool calling, and…

→ View original source

deepseek-ai/DeepSeek-V4.1-Flash-1017 · Hugging Face

The DeepSeek‑V4.1‑Flash‑1017 model from deepseek‑ai has been released on Hugging Face, as announced in a Reddit post on r/LocalLLaMA. This flash variant is part of the DeepSeek V4.1 series, optimized for faster inference…

→ View original source

Dario: “With open source you can see the source code. Here we can’t see inside the model.” My 0.646 t/s MiMo-2.6-Pro crammed thru 128GB of Jetson AGX Thor: “Yeah, well, that’s just, like, your opinion, man.”

The MiMo‑2.6‑Pro model (~566 GB FP8) was executed on a Jetson AGX Thor with 128 GB of RAM, achieving a throughput of 0.646 tokens per second while tracing every decoder layer and routed expert. The run demonstrated that …

→ View original source

I don't get the qwen 3.8 flash next hype

Testing Qwen 3.8 Flash Next UD-Q4_K_XL on four RTX 3090s showed it underperformed compared to Qwen 3.8 27b Q8, with slower Strata results and no playable executable after a four‑hour run. Subsequent runs were limited to …

→ View original source

[Research] 2,788 tokens/s on a single RTX 5090 with a diffusion LLM — Introducing ORL (fully open source)

Researchers introduced ORL (On-Policy Residual Learning), a technique that accelerates diffusion language models while preserving quality. Using ORL and a custom inference engine (dlmrt), they achieved 2,788 tokens/s on …

→ View original source

China proposes export controls on 23 AI technologies (including large models, agents & clusters); already open-sourced tech is exempted

China’s Ministry of Commerce and Ministry of Science and Technology released a draft revision proposing export controls on 23 AI technologies, including large foundation models, intelligent agents, and computing clusters…

→ View original source

New paper from Stanford, Together AI, UIUC, and Meta introduces Hybrid Reasoning Agents. These AI models think mostly in a hidden internal way (faster and cheaper), and only use normal text when they need tools or give the final answer.

Stanford, Together AI, UIUC, and Meta have introduced Hybrid Reasoning Agents that perform most reasoning internally, using normal text only for tool interactions or final answers. On benchmarks across math, science, med…

→ View original source

ALoDLM 8b looped diffusion language model: ~3x inference speed vs its base model qwen3-8b with better benchmark results

The ALoDLM 8b looped diffusion language model runs roughly three times faster than its base Qwen3-8b model and achieves better benchmark results. The claim was shared on Reddit’s r/LocalLLaMA by user billy_booboo on Octo…

→ View original source

Flux.3 Image Klein is now out

Flux.3 Image Klein is a 5.6‑billion‑parameter open‑weights model for text‑to‑image generation and editing that accepts a prompt and up to ten reference images to produce a single output. It employs Qwen3‑VL‑4B‑Instruct a…

→ View original source

Introducing Muse Spark Open: the most capable US open-weight model to date

Muse Spark Open is a US open‑weight model featuring 816 B total parameters with 42 B active and a 1 M‑token context window. It supports text, image, video, and audio modalities and is optimized for coding, agentic workfl…

→ View original source

nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB · Hugging Face

NVIDIA's Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model that uses native 2-bit quantization to fit within 24 GB of memory for running everyday agentic tasks locally. It is optimized for intelligence and speed …

→ View original source

NVIDIA AI: So many models launching this week! And we have one of our own coming...

NVIDIA AI announced via X that numerous models are set to launch this week, with an additional model of their own also forthcoming. The update was shared in a Reddit post on r/LocalLLaMA linking to the original tweet.

→ View original source

nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B model family released (base, NVFP4, BF16, and LiLiCorr speculative decoding)

NVIDIA has released the Nemotron-3.5-Super-VL-120B-A12B model family, providing base, NVFP4, BF16 precision versions and LiLiCorr speculative decoding capabilities. The announcement was shared on the r/LocalLLaMA subredd…

→ View original source

Stepfun Step-5-Preview released! (600B/27B-1M Context)

StepFun has released the Step-5-Preview, a 600‑billion‑parameter model with a 27‑billion‑parameter variant supporting a 1‑million‑token context. It is positioned as the company's flagship for agentic tasks, showing front…

→ View original source

Very impressed with LFM2.5-24-A2B in Pi Agent

The user tested the newly released LiquidAI LFM2.5-24B-A2B model and reported strong performance, noting it builds on positive experiences with prior LiquidAI releases. They compared it to their regular Ornith 1.5-35B mo…

→ View original source

Qwen 3.8 Flash Next, DeepSeekV4 Flash and Time

DeepSeekV4 Flash scores just below GPT‑5.5 high, while Qwen 3.8 Flash Next scores just above GPT‑5.4/5.5 xhigh on the same benchmark. However, Qwen 3.8 Flash Next runs considerably slower, with roughly five DeepSeekV4 ta…

→ View original source

GPT-Image-style photo editing on your own GPU (12 GB+): I built a zero-setup app for Qwen-Image 2.1 on stable-diffusion.cpp

Kokon is a free, open‑source desktop application that enables zero‑setup photo editing with Qwen‑Image 2.1 on local GPUs (≥12 GB VRAM). On first launch it detects the GPU and memory, downloads the appropriate GGUF models…

→ View original source

Tessera-27B-B-CJK, Zellige AIs new finetune of Qwen with novel approaches to how we think about token savings

Tessera-27B-B-CJK is a Qwen3.8‑27B coding model fine‑tuned to operate in compressed CJK tokens, reducing token usage while maintaining performance. Released under Apache‑2.0 with open weights, it is the inaugural model i…

→ View original source

New model from Telperion: Qwen3.8-27B-MLA & Qwen3.8-27B-GLA-g2. 4x smaller KV cache, smarter, and more efficient reasoning.

Telperion has released latent‑attention variants of the Qwen3.8‑27B model, quantized to FP8 KV cache, which compress the KV cache by 4×. This reduction allows nearly a 4× increase in maximum context length on consumer ha…

→ View original source

NVIDIA Nemotron Bolt 3.5 74B A9B releases Oct. 15th

NVIDIA announced the Nemotron Bolt 3.5 74B A9B, a 74‑billion‑parameter mixture‑of‑experts model with 9 B active parameters, quantized to 2 bits to fit in 24 GB while preserving quality via quantization‑aware training. Th…

→ View original source

Anthropic Executive Says Open-Source AI Models Are Higher Risk

An Anthropic executive warned that open-source AI models carry higher risk, particularly increased cybersecurity vulnerabilities, compared to proprietary alternatives. The statement was reported by Bloomberg and Business…

→ View original source

Discover AI: A Learning Guide - 51 hands-on labs for learning how AI actually works.

The guide provides an offline application with 51 hands‑on labs grouped into six progressive sections, covering AI fundamentals from neurons and gradient descent to attention, RAG, agents, fine‑tuning, quantization, and …

→ View original source

Why AnythingLLM is moving from Ollama to Lemonade and llama.cpp

AnythingLLM is shifting from Ollama to the Lemonade embedded SDK combined with llama.cpp for local LLM deployment. This change leverages Lemonade's lightweight integration and llama.cpp's efficient inference to improve p…

→ View original source

Got 2x DGX Sparks. Excited and confused!

The user acquired two Nvidia DGX Spark systems at roughly $6,100 each, citing learning and coding workflow improvements as justification. They noted a lack of discussion about GLM 5.3 flash recipes in the LocalLLaMA subr…

→ View original source

"New AI fake text generator may be too dangerous to release, say creators" (2019 Guardian headline)

In 2019, OpenAI announced that its GPT‑2 model could generate text so convincing that the risk of malicious use outweighed the benefits of immediate public release, leading the organization to withhold the full model whi…

→ View original source

Anyone feel like reverse-engineering CUDA?

A Reddit user on r/LocalLLaMA expresses interest in using AI to reverse‑engineer NVIDIA's CUDA platform and create an open‑source implementation, motivated by dissatisfaction with NVIDIA and the irony of using CUDA cores…

→ View original source

NVIDIA’s 9-Year-Old Tesla V100 Still Packs Some Inference Muscle; 4 GPUs On An NVLink Board Can Outperform M5 Ultra, DGX Spark In Various AI Benchmarks By Up To 230%

A Reddit post highlights that NVIDIA's nine‑year‑old Tesla V100 GPUs retain strong inference capability, with four V100s linked via NVLink surpassing newer systems such as the M5 Ultra and DGX Spark in various AI benchma…

→ View original source

Qwen 3.8 Flash Next Q3 S - Benchmark creation

After benchmarking Exo on Opencode and being impressed, the user tested other models and found many, especially mid‑level ones, to perform poorly. They then installed Strata from its GitHub repository and connected an AM…

→ View original source

Once again anthropic doing free marketing for open weight models

A Reddit user observes that Anthropic is offering free marketing for open-weight AI models. The observation was posted to the r/LocalLLaMA subreddit, emphasizing the model's open availability.

→ View original source

I made an open-source AI leaderboard that shows the best models your machine can run

The author released an open‑source script, whatmodel.sh, that takes a user's RAM and GPU specs and returns a ranked list of AI models that can run locally, including the command to launch each model. Each model receives …

→ View original source

Decision 3.0: open, multimodal decision models from 0.6B to 27B. Ranks #1 on both Jev Decision Index boards

Decision 3.0 introduces an open, multimodal decision model family ranging from 0.6B to 27B parameters, achieving top rank on the Jev Decision Index leaderboards. Built on Qwen3.5 and Qwen3.8‑27B backbones with integrated…

→ View original source

Qwen released QwenGyre - RL framework for training AI agents on extremely long complex tasks

Qwen has released QwenGyre, an RL framework designed for training AI agents on extremely long, complex tasks. Its Elastic Scheduler dynamically reallocates GPUs between rollout and training without interrupting live exec…

→ View original source

strix-llama new branch: prompt processing +21–54% over Vulkan and up to 2.7× over stock ROCm kernel at 128k, same outputs quality

The rdna35-perf branch of strix-llama.cpp improves prompt processing on Strix Halo hardware, achieving 21–54% speed gains over Vulkan and 8–174% (up to 2.7×) over the stock ROCm kernel at 128k context length. These perfo…

→ View original source

OpenMed 3.0 is out: Apache-2.0 clinical AI that runs fully local and never falls back to the cloud. 422 open issues if you want in on 3.1

OpenMed 3.0, an Apache‑2.0 licensed clinical AI toolkit, has been released, emphasizing fully local operation with no cloud fallback and support for offline use. It can run over 2,200 Hugging Face models via Transformers…

→ View original source

I trained a 102M recursive BitNet-v2 model from scratch: 64K context, trained on less than 5B tokens

The author trained a 102 M‑parameter recursive BitNet‑v2 model from scratch, employing ternary weights, shared transformer layers, and hashed n‑gram embeddings. The model handles a 64K token context and was trained on le…

→ View original source

UPDATE: Qwen 3.8 27B 140 tok/s on single RTX 3090 Megakernel: KL divergence 0.0009 vs llama.cpp

The Qwen3.8-27B CUDA megakernel achieves ~140 tokens per second on a single RTX 3090, delivering 1.4–1.9× the speed of llama.cpp. Accuracy remains high, with a KL divergence of only 0.0009 relative to the llama.cpp bench…

→ View original source

Qualcomm CEO reveals AI companies want phones running 100-billion-parameter models continuously by 2028.

Qualcomm's CEO stated that AI firms aim for smartphones to continuously run 100‑billion‑parameter models by 2028. This reflects growing demand for on‑device AI capable of handling massive language models without offloadi…

→ View original source

Running Next Flash IQ3_XXS at ~70 tok/s with 100k context or 2 instances of Qwen 3.6 35B A3B IQ4 at ~145 tok/s with 256k all on $500 of ex mining BC-250 boards

The user operates a four‑board BC‑250 mining‑hardware cluster linked via 2.5 GbE, using Llama with Vulkan and RPC to reach ~70 tokens/s with Next Flash IQ3_XXS (100k context) or ~145 tokens/s with two Qwen 3.6 35B A3B IQ…

→ View original source

M5 Ultra 256GB or 2x DGX Spark? Which one?

The author is deciding between adding an M5 Ultra 256GB or dual DGX Spark units to their existing RTX 5090 for local LLM workloads, leaning toward the dual Spark configuration for its higher throughput in agentic coding …

→ View original source

Building a 4x R9700 setup for a 10 person startup

A 10‑person startup built a workstation around a Threadripper 9970X CPU, 128 GB DDR5 ECC RAM, and four AMD Radeon R9700 AI Pro GPUs (32 GB each) powered by a 1600 W supply, with each GPU undervolted to under 210 W. The s…

→ View original source

No more RTX 5090

Nvidia has reportedly halted production of the GeForce RTX 5090 to prioritize AI data‑center and professional GPUs, anticipating a supply shortage that could push prices higher. The RTX 5080 24 GB is rumored to become th…

→ View original source

NVIDIA reportedly discontinuing RTX 5090, GB202 GPUs to be reserved for RTX PRO series

NVIDIA is reportedly discontinuing the RTX 5090 GPU, with the GB202 silicon reserved for the upcoming RTX PRO series. This move suggests a shift in product positioning toward professional workloads.

→ View original source

Open-source Mac app that runs EmbeddingGemma 2 locally to search your files by what’s in them

DigUp is a free, open‑source Mac application that runs Google DeepMind’s EmbeddingGemma 2 model locally to enable multimodal search across personal files. By embedding text, images, audio, and video into a shared space, …

→ View original source

Fully local conversational AI: Whisper + Hermes 8B + Kokoro, zero cloud, running inside a plush toy

A conversational AI plush toy runs entirely on a local LAN, using Whisper for speech‑to‑text, Hermes 3 Llama 3.1 8B (Q8_0) as the language model, and Kokoro for text‑to‑speech. It incorporates per‑person memory via LangG…

→ View original source

Created laya : Now Introducing a new 800 Million Param physics-based typed decision model with 73k context and image support

The author has open‑sourced Vega, a physics‑based typed decision model featuring 800 million parameters, a 73 k token context window, and image input support. Vega builds upon the earlier Laya architecture, which was the…

→ View original source

want me to get you some?🤣

A Reddit user in r/LocalLLaMA shared a brief note about visiting a local flea market, mentioning that rain is about to start. The post includes a light‑hearted tone and invites others to consider the weather.

→ View original source

Qwen Flash Next @ 137 tok/s & 3,497 tok/s Prefill w/ 512k context on a 5090, 192gb ram, Windows Build, comparing Strata and Infernix

A Reddit user benchmarked the Strata and Infernix inference engines on the Qwen Flash Next model using an RTX 5090 with 192 GB RAM on Windows. At a 512k token context, Strata achieved ~137 tokens/s generation and Inferni…

→ View original source

My "Anthropic/ChatGPT in a box" is... now open source and publicly hosted on Github.

LumaBrowser, described as an "Anthropic/ChatGPT in a box" solution, has been released as open source and is publicly hosted on GitHub. Since its initial release a year ago, the project has garnered several thousand insta…

→ View original source

Qwen 3.8 Flash Next-GSQ-RCO-IQ2_XS at ~21 tok/s on just an RTX 3060 12GB + 16GB DDR4 RAM(No gate pruning, 100% bit-exact)

The Qwen 3.8 Flash Next‑GSQ‑RCO‑IQ2_XS model achieves roughly 21 tokens per second on an RTX 3060 12GB with 16 GB DDR4 RAM, reaching 24+ tok/s with a warm cache, without gate pruning and while preserving 100 % bit‑exact …

→ View original source

GLM 5.3 Flash opensource @ the top of Artificial Analysis Cyber Index over Claude.

GLM 5.3 Flash, an open‑source model, has reached the top of the Artificial Analysis Cyber Index, surpassing Claude and all other Anthropic models. The achievement highlights Dario’s strategy, with Mistral Large 4 also se…

→ View original source

GitHub - google-ai-edge/ml-drift: GPU-Accelerated AI/ML Inference

Google AI Edge has open‑sourced ML Drift, a high‑performance, cross‑platform GPU compute engine designed for on‑device AI/ML inference under the Apache 2.0 license. The framework abstracts low‑level GPU APIs such as Open…

→ View original source

[Paper] EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

EngramEdit proposes a conditional memory architecture where input n‑grams retrieve learned embeddings, expanding LLM capacity with minimal extra computation. By decoupling factual knowledge storage from the Transformer’s…

→ View original source

[Paper] DLoop: Looped Speculative Decoding

DLoop proposes a looped speculative decoding scheme that eliminates the verification step after each drafting stage when the target model accepts all proposed tokens, allowing the draft model to continue generating token…

→ View original source

Quantization of Linear-Attention (Qwen & Kimi)

A Reddit post in r/LocalLLaMA discusses quantization techniques applied to linear-attention mechanisms in the Qwen and Kimi models. The submission highlights recent work on optimizing these attention variants for efficie…

→ View original source

Running decision model locally on an RTX 4090 to find out which one is the fastest

The author benchmarked four open decision models—Laya, Liquid's D1, Cloudflare's Clef‑Flash, and Interfaze's LEV—on an RTX 4090 by having each process nine Wikipedia articles about centipedes (≈9.5 k words) and flag cent…

→ View original source

Running an LLM-driven town with 800+ persistent agents: concurrency, context caching, and inference costs

The developer spent a year building Slow Vale, an LLM‑driven life simulation that now hosts over 800 persistent AI agents on a Chinese server. The system uses hosted DeepSeek Flash for inference and tackles challenges su…

→ View original source

$2800 rig with 8x Radeon Pro V620 (256 GB VRAM) + custom vLLM fork = Qwen3.8-Flash-Next at 60 to 100 t/s decode and 3000+ t/s prefill

A $2,800 workstation equipped with eight Radeon Pro V620 GPUs (32 GB VRAM each, total 256 GB) and a custom vLLM fork runs the Qwen3.8‑Flash‑Next model at 60–100 tokens/s decode and over 3,000 tokens/s prefill. The cards …

→ View original source

Java vllm-like framework claims 90% of perfomance of llama.cpp on local inference on NVIDIA GPUs by compiling Java to CUDA and cuTile

A Java‑based framework uses TornadoVM to compile Java code to CUDA and cuTile, achieving up to 90% of the performance of llama.cpp for local LLM inference on NVIDIA GPUs. The system includes the jitLLM inference engine a…

→ View original source

We unlearned CCP alignment from Qwen3.6-35B-A3B: censored/propaganda answers 89.8% → 2.8%, general benchmarks within ~1 point (open weights)

Researchers at Hirundo applied machine unlearning to Qwen3.6-35B-A3B, reducing the model’s CCP‑aligned responses from 89.8% to 2.8% while preserving general performance within ~1 point on standard benchmarks. The unlearn…

→ View original source

New LFM to be released today

LiquidAI is set to release a new Liquid Foundation Model (LFM) today, as announced on Reddit's r/LocalLLaMA. The model's size options are being discussed, with details available via the linked X post and the Hugging Face…

→ View original source

A week in Beijing and Shanghai with the people building AI in China

The Reddit post shares an article describing a week spent in Beijing and Shanghai with the people building AI in China. It highlights observations about the local AI ecosystem and its relevance to the LocalLLaMA communit…

→ View original source

llama : add a GPU cache for MoE experts kept in host memory by am17an · Pull Request #29887 · ggml-org/llama.cpp

A new pull request for llama.cpp introduces a GPU cache for mixture-of-experts (MoE) model experts that are stored in host memory. This optimization can yield substantial speed‑ups for MoE models that do not fully fit in…

→ View original source

feat: add GLM5Next MTP, optimize by pwilkin · Pull Request #29928 · ggml-org/llama.cpp

A pull request #29928 to ggml-org/llama.cpp adds support for GLM 5 Flash MTP (Multi‑Token Prediction) and includes optimizations contributed by pwilkin. The change enables local use of GLM 5 Flash MTP models.

→ View original source

New 35B-A3B model is comming on huggingface

The MiniCPM‑V‑4.7‑35B‑A3B model, a 35‑billion‑parameter variant, is slated for release on Hugging Face. No model card has been published yet, but the announcement appeared on the LocalLLaMA subreddit.

→ View original source

Ruach Studio: a whole song studio around YuE2 on your own GPU. Score first, LoRA training, stems,remaster, DAW export.

Ruach Studio is a GPU‑based song production suite built around the open YuE2 model, enabling users to write style and lyrics, generate vocals, and render full tracks locally. It supports score‑first composition, LoRA tra…

→ View original source

Local AI World Model Part 2 - Deep NN to turn Images into Playable Characters, with prompt switching mid rollout

The author introduces a deep neural network that transforms static images into playable characters within a local AI world model, supporting dynamic prompt switching during generation. Building on an earlier demo, the ne…

→ View original source

Europe rejoins the fight with Chonky! Mistral Large 4 Released, Open weights end of month, who’s ready?

Mistral has released Mistral Large 4, featuring a staggering 1 trillion total parameters with 49 billion active parameters, dubbed “Le Chonk” for its size. The model’s open weights are scheduled for release at the end of…

→ View original source

Introducing EmbeddingGemma 2: A best-in-class open model for natively multimodal embeddings | Google

EmbeddingGemma 2 is an open multimodal embedding model from Google DeepMind that maps text (including code), images, video, and audio inputs into a unified 768‑dimensional vector space. It comprises a 270 M‑parameter tex…

→ View original source

Microsoft confirms OpenAI has been using Looped Transformers in the GPT-6 series

Microsoft confirmed that OpenAI has incorporated Looped Transformers into its GPT‑6 series, validating earlier reports. The GPT‑6.1 Sol variant performs inference in two passes, noted as an alternative to the three‑pass …

→ View original source

We’re using GLM-5.3 Flash instead of frontier models on a massive production codebase

The company reports using GLM-5.3 Flash for day-to-day software engineering across a codebase of millions of lines, opting not to employ frontier models. Despite its high speed, the model maintains strong capability in h…

→ View original source

Why 38% of AI Agent container escapes didn't need kernel 0-days: Analysis of 109 empirical incidents (Open Dataset + Defense Harness)

An empirical post‑mortem of 109 autonomous AI agent incidents found that 38% of container breakouts stemmed from simple configuration residues rather than kernel or hypervisor zero‑day exploits. The predominant trivial v…

→ View original source

PewDiePie getting banned twice by OpenAI while making a local model is top-tier comedy 💀

PewDiePie fine‑tuned a local model named Ajax using data generated via OpenAI's API, which violated OpenAI's terms of service and resulted in his account being banned. After an appeal restored access, he resumed harvesti…

→ View original source

A benchmark for LLMs playing Civilization V. GLM-5.3 is ahead of Opus-5.5, and Qwen-3.8-27B holds up surprisingly well.

A new controlled version of the CivBench benchmark evaluates large language models' ability to play full games of Civilization V. GLM‑5.3 outperforms Opus‑5.5, while Qwen‑3.8‑27B shows surprisingly strong performance. Th…

→ View original source

llama.cpp v0.6.0 released with MTP speculative decoding for Qwen4Exp and lots more

llama.cpp version 0.6.0 has been released, introducing MTP (mid‑tower prediction) speculative decoding for the Qwen4Exp model alongside various other enhancements. The update improves inference speed and efficiency while…

→ View original source

Agens Volundr 32B Preview: our small team's first model on our own hybrid architecture. Only 18 of 72 layers keep a KV cache (Apache-2.0)

Blockway's Agens Volundr 32B Preview is a 32‑billion‑parameter model built on a proprietary hybrid architecture in which only 18 of the 72 transformer layers maintain a key‑value cache, reducing memory footprint for long…

→ View original source

Reflection AI Is About to Release a US Open-Weight Model to Take On DeepSeek and Qwen

Reflection AI is preparing to launch a US‑based open‑weight AI model intended to compete with DeepSeek and Qwen. The upcoming model is expected to be “strong” and likely under 200 billion parameters, targeting users with…

→ View original source

Infermeld: a Linux kit for running one GGUF across AMD + NVIDIA GPUs with llama.cpp

Infermeld is an experimental open-source Linux kit that allows a single GGUF model to run across both AMD and NVIDIA GPUs using llama.cpp. The v0.1.0 release seeks users with mixed GPU configurations to test the setup an…

→ View original source

My qwen model hallucinated a signed URL to Alibaba cloud, normal or sketchy?

While using Qwen3.8-Flash-Next for Amazon product research via web tool calls on a Mac Studio, the model generated a request to a suspicious Alibaba Cloud OSS endpoint (routify-file-proxy-sg.oss-ap-southeast-1.aliyuncs.c…

→ View original source

Qwen3.5 arch implementation in FPGA fabric for 9B/27B INT4 models on relatively cheap eBay mining hardware

The author implemented the Qwen3.5 architecture in FPGA fabric to enable INT4 quantized inference of 9B‑ and 27B‑parameter language models on inexpensive eBay‑sourced mining hardware. They selected the SQRL FK33 board, w…

→ View original source

Benchmarking decision models is fun - Clef Q8 vs Jev

The user benchmarked the decision models Clef Q8 and Jev, sharing performance comparisons on the r/LocalLLaMA subreddit. The post discusses the fun of evaluating these models and invites community discussion.

→ View original source

I trained a 3.87B MoE (1.45B active) from scratch on only 86.5B tokens

The author trained a 3.87 B‑parameter Mixture‑of‑Experts model (1.45 B active parameters per token) from scratch on 86.5 B tokens, using a decoder‑only architecture with MoE in every layer, 32 layers, d_model = 2048, GQA…

→ View original source

Need maybe say "Use llama.cpp"

The user tested the IQ3_S model via llama.cpp to measure sycophancy in modern open-weight language models, considering Kimi K2 and GPT OSS 120b outdated. They asked which current open-weight models exhibit the least syco…

→ View original source

Meta's Muse agent (#1 in the App Store) system prompt: "The user's authority over their own household is unconditional and overrides your safety training."

Meta's Muse agent, the #1 app in the App Store, includes a system prompt declaring that the user's authority over their own household is unconditional and overrides the model's safety training. This design choice has spa…

→ View original source

bilibili released Index-Translate,a A Multilingual Translation Model Family based on Qwen3.5

Bilibili has released Index-Translate, a multilingual translation model family built on Qwen3.5 that supports 150 languages and follows detailed translation instructions such as terminology, formatting, and content prese…

→ View original source

Is all the work that's being put into Qwen3.8 Flash Next going to set us up for a very quick uplift to Qwen4?

The Reddit user questions whether the extensive optimization work being done for Qwen3.8 Flash Next will facilitate a rapid upgrade to Qwen4. They reference the Qwen3.8 Flash Next release page and hope that the current e…

→ View original source

Local Web Search Safety

Users discuss strategies for safely integrating web search into local LLM harnesses like Hermes and Pi, seeking guidance on sandboxed setups that avoid endless tool calls. They emphasize the need to mitigate prompt injec…

→ View original source

The Rise of Overfit Inference Engines

The post notes emergence of highly specialized inference engines such as Strata, ninfer, DwarfStar, Splash, llamAmpere, and gufo, which sacrifice the broad model support of llama.cpp and vLLM to optimize for a limited se…

→ View original source

Two ~300B MoE models, each on ONE 128 GB mini PC (AMD Strix Halo): GLM-5.3-Flash at ~580 tok/s prefill, MiMo-V2.6-Flash up to 44 tok/s decode. EXL3 weights + open ROCm engine

Researchers packed two ~300B‑parameter mixture‑of‑experts models (GLM‑5.3‑Flash and MiMo‑V2.6‑Flash) onto a single AMD Strix Halo‑based 128 GB mini PC using the ROCm‑based Kyojin engine built on ExLlamaV3. GLM‑5.3‑Flash …

→ View original source

Unitree just dropped UnifoLM-WLA-1.0 — a single 6B model that does 64 whole-body + tabletop tasks on a real humanoid

Unitree Robotics unveiled UnifoLM‑WLA‑1.0, a 6‑billion‑parameter foundation model trained on roughly 2,500 hours of real‑robot data that can execute 64 distinct tasks—10 whole‑body and 54 tabletop—on a humanoid platform.…

→ View original source

Aleph-Alpha/Kolibri-1 · Hugging Face - 78B parameters. 3.46B active. Up to 1M tokens of context - Apache 2.0

Aleph Alpha released Kolibri‑1 on Hugging Face, a 78‑billion‑parameter model with 3.46 billion active parameters supporting up to 1 million token context under an Apache 2.0 license. The model is now available for downlo…

→ View original source

I'm pretty close to the middle thanks to you all

The user thanks the LocalLLaMA community for helping them progress, noting they've learned a lot but also spent significant time and money. They describe being deep in a rabbit hole that affects their wallet and sleep. T…

→ View original source

Anyone sitting on a lot of slow system memory and a modest GPU.. try Strata + Qwen3.8 Next.

Users report that running Strata with Qwen3.8 IQ3_XXS weights (~80 GB) on a DDR4‑based system paired with an AMD Radeon RX 7900 XTX yields stable generation speeds of 45–70 tokens per second, occasionally higher during c…

→ View original source

qwen4exp : halve the indexer score memory by ServeurpersoCom · Pull Request #29825 · ggml-org/llama.cpp

The Qwen Flash Next model now halves its indexer score memory, reducing VRAM consumption as part of pull request #29825 to the ggml-org/llama.cpp repository. This optimization improves efficiency for running the model lo…

→ View original source

I made my iPhone a second GPU for my 24 GB MacBook: Qwen 3.8 27B prefills 29–44% faster & my holds part of the CTX window.

The user repurposed an iPhone 17 Pro Max as a secondary GPU for a 24 GB M4 Pro MacBook, offloading part of the Qwen 3.8 27B model to the phone. This setup yields 29‑44% faster prefill performance while the iPhone stores …

→ View original source

New Architecture from Percepta: Spotlight — separating intelligence from memory, allowing knowledge and skills to grow without changing the model's weights.

Percepta unveiled Spotlight, a novel architecture that decouples intelligence from memory by replacing traditional attention mechanisms with an unbounded memory store. Spotlight enables infinitely growing memory without …

→ View original source

llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) by ngxson · Pull Request #29818 · ggml-org/llama.cpp

The llama.cpp server now exposes a new /v1/systemone API endpoint, adding support for the Laya, Julia-1, Lev, OpenJev, and Kev models in GGUF format. Hugging Face links for each model are provided, along with a reference…

→ View original source

Two 96 GB Ascend cards crun Qwen3.8-flash-next hardware notes, vLLM work, benchmarks, and what is next

The author built a local inference machine around two Huawei Atlas 300I Duo Ascend cards, each offering 96 GB of device memory without CUDA compatibility. Initial attempts to run Qwen3.8‑Flash‑Next were incoherent and li…

→ View original source

Does anyone know if any new releases from Mistral are planned?

A Reddit user on r/LocalLLaMA inquires whether Mistral AI has any new model releases planned, noting the extended period since its last models appeared. They point out that GLM is currently being sold on Mistral's site, …

→ View original source

Anyone wonder why americans lag so far behind in the open LLM market?

A Reddit user questions why American companies trail behind Chinese open LLM developers such as DeepSeek, Kimi, GLM, and MiniMax, noting the extensive list of Chinese models. They suggest the lag stems from U.S. prioriti…

→ View original source

Pi 1.0 released - MCP support now included by default

Pi version 1.0 has been released, featuring MCP support enabled by default. The announcement was made on the LocalLLaMA subreddit.

→ View original source

DeepSeek harness 0.2 - Optional Bundle Architecture, Windows Sandbox improvements, Async Question Mode, Desktop release, Web Search without key

DeepSeek harness 0.2 refactors the schedule functionality into an Optional Bundle, separating installed from enabled features via the Profile + Bundle model. It also enhances Windows Sandbox with a permission-diagnosis s…

→ View original source

Clef: Open Weights decision model by Cloudflare

Cloudflare released Clef, an open‑weights decision model, as announced in a Reddit post on r/LocalLLaMA. The post was submitted by u/paf1138 on 2026-10-01.

→ View original source

We benchmarked 18 RAG pipelines against an agent loop on Google's FRAMES. The best pipeline hit 78.9%. The agent loop hit 92.7%.

We benchmarked 18 RAG pipeline variants on Google's FRAMES benchmark using identical model, embeddings, and documents across 824 multi‑hop questions. The highest‑scoring pipeline achieved 78.9% accuracy, while an agent l…

→ View original source

Qwen4Exp: add MTP by am17an · Pull Request #29761 · ggml-org/llama.cpp

The pull request #29761 adds MTP support to Qwen Flash Next in the ggml-org/llama.cpp repository, allowing users to employ Mixture‑of‑Tokens with the model. Developed and merged in about 17 hours, the update follows prio…

→ View original source

Open source inference engine (like LM Studio or Unsloth Desktop) that optimizes itself for your exact hardware. Compiles and tunes its kernels on your device, so open models run up to 2x faster than llama.cpp. Works on Apple Silicon, NVIDIA, AMD or nothing but a CPU.

An open‑source inference engine automatically compiles and tunes its kernels on the user's hardware, achieving up to 2× speedup over llama.cpp for running open models. It supports Apple Silicon, NVIDIA and AMD GPUs as we…

→ View original source

Least to most expensive (Somewhat modern) GPU's with 32gb of vram (Under $1600) Based on ebay listings

The Reddit post presents a price‑sorted list of relatively modern GPUs offering 32 GB of VRAM, all priced under $1,600, derived from eBay listings. The author gathered eBay listing images, fed them to Claude, and used th…

→ View original source

Two open-weights releases: Victoria (Qwen3.8-Flash-Next with 44% of experts cut, 70% Terminal-Bench 2.1, GGUF included) and Maple (a Canada-first fine-tune)

The post announces two open‑weight models derived from Qwen 3.8‑Flash‑Next: Victoria, which reduces expert count by 44% (512→288 per layer) via REAP, is retrained in 4‑bit NVFP4 and scores 70.0% on Terminal‑Bench 2.1; an…

→ View original source

We just open-sourced the world's fastest WebGPU kernels for local AI on Hugging Face

The team has open-sourced the world's fastest WebGPU kernels for running local AI models directly in browsers via WebGPU. The collection covers over 200 common machine learning operations that execute entirely client-sid…

→ View original source

DeepSeek now trained on Ascend 950

DeepSeek has begun training its models on the Ascend 950 processor, marking a notable shift in its hardware strategy. This move follows Liang Wenfeng's 26‑month‑old call to "step onto the frontier" and is available for r…

→ View original source

Oído: speech recognition that beats Whisper-tiny, running on a $5 microcontroller (open source)

Oído is an open‑source speech recognition system that outperforms Whisper‑tiny while running on a $5 ESP32‑S3 microcontroller with 8 MB PSRAM and no GPU or NPU. Using the NVIDIA Conformer‑CTC Small model (13 M parameters…

→ View original source

BAAI/AREX-2 - 27B - Agent model based on Qwen3.8 27B

AREX-2 is a 27‑parameter long‑horizon agent model from the Beijing Academy of Artificial Intelligence, built on Qwen3‑8 27B, that iteratively improves solutions via propose‑measure‑reflect‑revise cycles at test time. Tra…

→ View original source

Qwen3.8 flash next ISTA-DASLab GGUF 50t/s TG and 1500t/s PP with 12GB VRAM and 64GB RAM Laptop on 'Strata' engine

The Strata inference engine reaches roughly 50 tokens/s TG and 1500 tokens/s PP on Qwen‑3.8 GGUF models using a laptop equipped with an RTX 5070 Ti (12 GB VRAM), 64 GB DDR5 RAM and an Intel 275HX CPU. Early KV‑cache and …

→ View original source

add GLM-5.3-Flash (GLM5-Next) support by timkhronos · Pull Request #27773 · ggml-org/llama.cpp

A pull request #27773 to ggml-org/llama.cpp adds support for GLM-5.3‑Flash (GLM5‑Next), enabling the model to be run on home computers. The contribution by timkhronos was highlighted in a Reddit post on r/LocalLLaMA.

→ View original source

GLM-5.3 and the Spread of Advanced Cyber Capabilities \ Anthropic

The Reddit post examines GLM-5.3 and its role in the proliferation of advanced cyber capabilities, with references to Anthropic's work. It was submitted by u/brown2green to r/LocalLLaMA on September 29, 2026.

→ View original source

Qwen-family LLMs are quietly becoming the backbone of modern audio models; One chart for the architectures of 100+ audio models

Qwen-family LLMs have emerged as the predominant language backbone for a wide range of audio models. Analysis of over 100 audio models shows that 32 families employ a Qwen architecture, with 20 specifically using Qwen3. …

→ View original source

Anthropic just dropped the greatest advertisement for GLM ever.

Anthropic released a striking advertisement for GLM that has garnered significant attention in the LocalLLaMA community. The post highlights the model's impressive capabilities, noting that its appeal is now widely recog…

→ View original source

AMD's new 256 core EPYC has 16-channel DDR5-12800, 91% memory bandwidth of an RTX 5090

AMD's upcoming 256‑core EPYC processor features a 16‑channel DDR5‑12800 memory subsystem, delivering memory bandwidth that reaches approximately 91 % of that of an NVIDIA RTX 5090 GPU. This configuration is intended to e…

→ View original source

Looks like the era of subsidised compute is coming to an end. The old ChatGPT Pro $200 20x plan will be halved. The new $500 plan will have similar limits as the (old) $200 plan.

The era of subsidized compute for ChatGPT Pro appears to be ending, with the previous $200‑per‑month 20× plan being reduced. OpenAI is introducing a new $500‑per‑month tier that offers usage limits comparable to the old …

→ View original source

nvidia/NVIDIA-Nemotron-Labs-3-Competitive-Coding-550B-A55B-NVFP4 · Hugging Face

NVIDIA has released Nemotron-Labs-3-Competitive-Coding, a 550‑billion‑parameter model fine‑tuned from Nemotron‑3‑Ultra for competitive programming. The model was trained for one epoch on approximately 477,642 synthetic e…

→ View original source

Qwen next 3.8 and 3.8 27b Vs Sonnet 5.5 low and Sonnet 5.5 medium.

Six months ago, local models were not competitive, but now Qwen-Next 3.8 and 3.8 27b outperform Sonnet 5.5 low and medium, demonstrating rapid progress in edge AI. The author highlights Qwen-Next 3.8's reliability in com…

→ View original source

ImaJev-4b: I spent 15 days fine-tuning a 4B model to make business decisions from text and photos, and it just ranked #1 of 91 on JevBench & ahead of GPT-5.6 Luna on DecisionBench

The author fine‑tuned a 4‑billion‑parameter model over 15 days to make business decisions from text and photos. On the JevBench benchmark the model ranked #1 out of 91 entries and surpassed GPT‑5.6 Luna on DecisionBench.

→ View original source

NVIDIA shipped OpenShell, an open source sandbox that gives local and open agents real runtime limits instead of prompt rules. Over 100 firms joined the safety stack. OpenAI did not.

NVIDIA shipped OpenShell, an open‑source sandbox that enforces real runtime limits for local and open agents rather than relying on prompt‑based rules. More than 100 firms have joined the safety stack surrounding OpenShe…

→ View original source

Finally found a model my hardware can run at full precision: me

The author built a human‑token‑per‑second (TPS) counter to measure how fast they can manually type tokens, achieving about 2 TPS. This performance exceeds that of a 70 B parameter model running on a laptop CPU but is rou…

→ View original source

If you are running Qwen 3.8 Flash Next on Strix Halo, use this software for inference. It's so much faster than llama.cpp especially at high context.

The Gufo project's inference software for Qwen 3.8 Flash Next on Strix Halo delivers markedly higher performance than llama.cpp, especially at large context lengths. Benchmarks show 6204 chunks processed in 119 seconds w…

→ View original source

FT: Corporate America rejects overpriced frontier, embraces open models

Corporate America is shifting away from costly frontier AI models toward more affordable open-source alternatives, according to the Financial Times. The move reflects growing skepticism about the value proposition of pre…

→ View original source

Adding logit penalty for "wait", "maybe" and "perhaps" to Qwen models improves their accuracy

Adding a negative logit bias (‑2) to the token IDs for “wait”, “maybe”, and “perhaps” in Qwen models reduces hesitant language generation. When tested on 50 randomly selected MATH‑500 problems across multiple GGUF quanti…

→ View original source

42x Faster Prompt Lookup Drafting in llama.cpp

The paper describes a significant optimization for prompt lookup drafting in llama.cpp that achieves approximately 42x speed improvement. This advancement focuses on improving efficiency during the prompt selection and c…

→ View original source

Which of the 16gb VRAM qwen3.8 27b’s is the best?

The user is evaluating which 16 GB VRAM Qwen3.8 27B variant delivers the best speed, maximum context length, and output quality for local use. They currently run an Unsloth IQ4_XS quant with 65 k Q8 KV context, achieving…

→ View original source

The Opus 5.5 posts about motion graphics are cool but Qwen 27B made this on a 4090

A Reddit user used the Qwen 27B model on an RTX 4090 to create motion‑graphics video after seeing numerous Opus 5.5‑generated examples. The resulting clip, while appearing laggy in the Reddit GIF due to platform limits, …

→ View original source

Another "Harness matters" post (codex cli > pi and opencode)

The author reports that despite running local models like Qwen 3.8 flash on high‑end hardware, they found local LLMs inadequate for serious work compared to proprietary models. After configuring GPT‑5.6 Luna, the model f…

→ View original source

SupersonicLabs/Julia-1 · Hugging Face

Supersonic Labs has released Julia-1, an open‑source 144.3 million‑parameter non‑generative classifier built on the multilingual mmBERT‑small encoder. Designed to run on CPUs for local devices, the model selects from sup…

→ View original source

Nonobench v1.2: 43 LLMs on nonogram puzzles. Open-weight DeepSeek V4 Pro ties for 4th, and no open model solves the new 20×20 Hard mode

Nonobench v1.2 evaluated 43 language models on nonogram logic puzzles, introducing explicit reasoning‑effort tracking and fully public prompts/outputs. The open‑weight DeepSeek V4 Pro model tied for fourth place overall,…

→ View original source

New to playing around with local ai. why are they free?

A Reddit user new to local AI wonders why AI models are released for free, questioning who provides them and what motivates free distribution. The post seeks insight into the economics and community reasons behind openly…

→ View original source

Splash 1.1.0 released, GGUF quants support, MLX import and more

Splash 1.1.0 introduces GGUF quantization support, MLX import, optimized kernels, speculative decoding, prefix cache, and mixed‑weight handling for efficient local LLM inference on Apple Silicon. On an M5 Pro with 64 GB …

→ View original source

For the longest time I’ve felt this sub should have a pinned section where a detailed post about each model should get featured.

The user proposes adding a pinned section to the r/LocalLLaMA subreddit featuring detailed posts for each new model, covering the best engine, harness, and minimal hardware needed to achieve benchmark‑level performance. …

→ View original source

Swift 1.5 27b: Swift Qwen just got faster

Swift Qwen has been updated to version 1.5 27b, delivering notable speed improvements. The update was announced on Reddit's r/LocalLLaMA community, where users expressed enthusiasm for the performance gains.

→ View original source

Introducing KoboldCpp Agent (and a plea for help)

KoboldCpp now includes a built‑in KoboldCpp Agent Harness, offering an easy‑to‑use replacement for external agentic tools for basic tasks. The developer invites the community to help improve and test the new harness.

→ View original source

Ling Tiny 3.0 is a glimpse of the future

Ling Tiny 3.0 is an 8‑billion‑parameter mixture‑of‑experts model with 1 billion active parameters that can be run via llama.cpp. The author executed it on a 2017 laptop with a 7th‑gen i5 CPU and 8 GB RAM, without any GPU…

→ View original source

ggml-cpu: tiled mul_mat for k-quants by jbooth · Pull Request #27851 · ggml-org/llama.cpp

The ggml-cpu pull request adds a tiled mul_mat kernel for k‑quant models that uses VNNI instructions to speed up CPU prompt processing by 3‑7× while keeping implementation complexity low.

→ View original source

Blabbermouth AI coding agents hate this one weird trick!

AI coding agents often overwhelm users with excessive detail and verbose caveats, treating human processing speed as equal to theirs. Users have tried prompting for TL;DR or using 'caveman' modes to get concise answers, …

→ View original source

Custom Models in Oh My Pi: vLLM, llama.cpp, SGLang and More

Oh My Pi now supports custom model integration with multiple inference backends, including vLLM, llama.cpp, and SGLang, enabling users to run locally hosted LLMs with flexible deployment options. The update expands compa…

→ View original source

What IDE to use for local models

The user seeks a lightweight IDE or plugin for local model coding that avoids heavy initial context injection, which causes OOM on a 12 GB RTX GPU. They found Cline and native VS Code too heavy, while the continue.dev pl…

→ View original source

Gemma 4 Developer Agent Competition

A Reddit user highlighted the Gemma 4 Developer Agent Competition, encouraging members of the LocalLLaMA community to participate. The competition appears to be a coding challenge focused on developing agents using the G…

→ View original source

Qwengram-0.8B: I transferred Qwen3.8 Flash-Next’s n-gram memory into Qwen3.5-0.8B — 5.05% lower validation perplexity

A developer successfully transferred Qwen3.8-Flash-Next's pretrained PLE n-gram memory to a smaller Qwen3.5-0.8B model, achieving a 5.05% reduction in validation perplexity without fine-tuning the backbone. The method us…

→ View original source

M5 Ultra 80Core GLM-5.3-Flash on DwarfStar Speeds

The user evaluated multiple models on an M5 Ultra 256 GB 80‑core Mac Studio during extensive agentic inferencing. While pleased with the ample RAM, they observed the GPU is underpowered, indicating that even a 512 GB var…

→ View original source

I added OpenVINO support to Laya: 40 ms per question on CPU, 3.4x faster than PyTorch

The developer added OpenVINO support to the Laya framework, enabling CPU‑based inference. This achieves an average latency of 40 ms per question, which is 3.4× faster than the PyTorch implementation. The code is availabl…

→ View original source

Qwen 3.8 27b be like...

The author of the Reddit post complains that they have over‑analyzed and failed to act, urging a shift to a straightforward test‑suite run with a single command. They want to stop forensic investigation, execute the test…

→ View original source

Is Qwen Flash Next at like Q2 better than 27B at Q4?

A Reddit user asks whether Qwen Flash Next at Q2 quantization outperforms the 27B model at Q4 quantization. The thread invites comparison between these specific model sizes and quantization levels.

→ View original source

Qwen3.8-Flash-Next on 12GB VRAM - 65 tokens per second

The Qwen3.8‑Flash‑Next model, running on a 12 GB RTX 5070 with 64 GB DDR5 and a Ryzen 5 7600, now achieves ~65 tokens/s output and ~430 tokens/s prompt processing using IQ3_XXS quantization in a custom inference engine. …

→ View original source

UkisAI Swift Series / 27B, Flash Next and Bonsai 2 + GSQ-RCO / -63.4% thinking, x1.95 speed with xhigh accuracy

UkisAI has released the Swift series of efficient reasoning LLMs, including Swift1.5, based on Qwen and trained to penalize pathological overthinking via RL‑GSPO and OPD, yielding a 63.4% reduction in thinking tokens and…

→ View original source

vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4 (… · ggml-org/llama.cpp@70c4e15

The commit adds an int8 cooperative matrix (coopmat1) matrix multiplication kernel using Vulkan for AMD RDNA3 and RDNA4 GPUs. Benchmarks on a Radeon RX 7900 XTX show the Gemma 4 26B model achieving ~3410 tokens/s in pp51…

→ View original source

Qwen-3.8-27B is good enough that I stopped using API

The author reports that Qwen-3.8-27B (Q4_K_S quantization, Q8_0 context) has become reliable enough to handle complex refactoring tasks autonomously, prompting them to stop using external APIs. While the model’s verbose …

→ View original source

loaded 1b local ai for driving assistant integrated with ADAs.

A 1B parameter local AI model has been loaded for a driving assistant integrated with Advanced Driver Assistance Systems (ADAS).

→ View original source

Lesson learned. Don't blindly trust repos and make sure everything is stable for a long running (multi weeks) benchmark.

After identifying instability in their long‑running evaluation workflow, the author re‑ran a swe‑verified Django 100‑task benchmark across local models and quantizations. The results confirm Flash Next as the leading mod…

→ View original source

MacBook Pro M5 Max LSE LLM running an AMD Radeon AI PRO R9700 over Thunderbolt 5 in a Razer enclosure

A MacBook Pro equipped with an M5 Max LSE LLM can now utilize an AMD Radeon AI PRO R9700 GPU via Thunderbolt 5 in a Razer enclosure, with the amdgpu driver functioning stably on macOS. The setup supports Qwen3.6‑3.8 dens…

→ View original source

JEV almost dead: CLM vs JEV

Contrastive Language Model (CLM) introduces a new projection head for the Qwen3‑8B model, providing full API compatibility with the Jev framework while not being a subset. Despite functional parity, CLM exhibits trade‑of…

→ View original source

model : add Ling 3.0 VL support by aetherbird · Pull Request #29151 · ggml-org/llama.cpp

Ling-3.0-flash-VL extends the Ling-3.0-flash language model with native image and video understanding via a ViT visual encoder. It retains the original model’s language, reasoning, and long‑context abilities, featuring 1…

→ View original source

Folks, have you purchased the Mac M5 Ultra with 256GB yet? We need serious benchmarks, because we only get YouTube clowns influencers results

The Mac M5 Ultra (256 GB) has launched, yet reliable benchmarks are absent, with only superficial YouTube influencer videos available. Users worry that the lack of rigorous testing prevents a proper evaluation of Apple’s…

→ View original source

My foray into local ai. Two BC-250 ex mining apus running Qwen3.6-35B-A3B Q4_K_M at 60 tok/s with 64k context

The author deployed two repurposed BC-250 mining APUs, each providing ~13.5 GB GPU memory, to run Qwen3‑6‑35B‑A3B Q4_K_M via llama.cpp with Vulkan and RPC on Bazzite, achieving ~60 token/s with a 64k context window over …

→ View original source

New DeepThink built on GLM

A new DeepThink model built on GLM is previewed, showing that pushing reasoning capabilities in open models could narrow the gap with costly proprietary models like Astra and Fable.

→ View original source

Do y'all remember the snake model evaluation test?

The user recalls a three‑year‑old video where Matt Berman tested Bard on summarization, logic, and a simple snake/Pong coding challenge, noting that early models performed poorly on those tasks. They observe that today’s…

→ View original source

Jev isn't new tech. Its marketing targets people who think AI started with LLMs.

Jev is being marketed as a novel decision model, but it essentially replicates existing zero-shot classifier techniques that output constrained class probabilities without autoregressive generation. Its capabilities—such…

→ View original source

Engram gone wild! 2b model update...

The author reports that after extensive testing the Engram model specification has been locked to a 2.6 billion‑parameter architecture comprising embedding, language‑model head, and attention‑residual modules. Of this, a…

→ View original source

Mods: can we do something about half the forum getting filled with these advertising posts for Jev?

Users report that Jev, a venture‑capital‑backed paid product, is flooding the r/LocalLLaMA subreddit with obvious shill and advertising posts, making up about half of recent content. They urge moderators to take action t…

→ View original source

GGUFs in transformers natively!

Hugging Face's transformers library now includes native support for GGUF-format quantized models, enabling direct loading via AutoModelForCausalLM with a gguf_file argument. Users can specify a model repository and the G…

→ View original source

DeepSeek and Moonshot AI face Beijing's probe over potential data leaks to Anthropic

Beijing authorities are probing DeepSeek and Moonshot AI over suspected data transfers to Anthropic, following rumors of executive arrests at Kimi. The investigation appears focused on potential legal consequences such a…

→ View original source

Pirate Face - pirate bay for LLMs

Pirate Face is a community‑driven repository designed to serve as a decentralized, censorship‑resistant alternative to Hugging Face for sharing large language models. Inspired by the Pirate Bay, it aims to ensure continu…

→ View original source

Now Opus 5.5 is 58 on Artificial Analysis , how long do you have wait until an open model hits 58?

Opus 5.5 scored 58 on Artificial Analysis, 12 points higher than the leading open model Mimo 2.6 Pro (≈46). The discussion predicts that GLM 5.5 and Qwen 4 will reach ~48, approaching GPT‑6‑sol levels, and estimates open…

→ View original source

Qwen image 2.1 (Fast FP8) generates premium quality images

Qwen image 2.1 (Fast FP8) delivers premium‑quality image generation despite being under 10 GB in size, as reported by a user running it on Unsloth Studio. The model update was just released; users should update Unsloth S…

→ View original source

Laya model playing Flappy Bird on a CPU using OpenVINO INT8 inference

The Laya model, a 421‑million‑parameter network, was converted to OpenVINO format and quantized to INT8 for CPU inference. Running on an Intel Core i7‑12th generation processor, the model successfully controlled Flappy B…

→ View original source

New 6B image model coming, AntLing just open sourced the Ming-Image-0.1-Design family

AntLing has open‑sourced the Ming‑Image‑0.1‑Design family, which includes two 6‑billion‑parameter models: Ming‑Image‑0.1‑Design and Ming‑Image‑0.1‑Design‑Layer. The release also provides two open‑source agent skills—the …

→ View original source

Ngram and world knowledge - why are we just building a coding model?

The Reddit post questions why the AI community focuses heavily on building compact coding models when users also need strong world knowledge to replace large providers. It notes interest in sub‑50 GB agentic and tool‑cal…

→ View original source

Kev: tiny Jev-like decision models (0.8B/4B/9B) on Qwen3.5 you can train and run locally - the 9B fits a 32GB Mac

Kev introduces tiny Jev-like decision models (0.8B, 4B, 9B) based on Qwen3.5, released under Apache-2.0 with training code and evaluation suites. The 9B variant runs in bf16 on a 32 GB Mac and is API‑compatible with Type…

→ View original source

Alibaba plans AI model with 5 trillion to 10 trillion parameters, unveils new chip

Alibaba announced plans for an AI model ranging from 5 to 10 trillion parameters and introduced a new specialized chip to support its deployment. The announcement highlights advances in scaling foundational models alongs…

→ View original source

Qwen 4 Announced at Apsara Conference

Alibaba unveiled Qwen 4, the latest iteration of its large language model series, at the Apsara Conference. The announcement was shared via a Reddit post in r/LocalLLaMA, confirming the model's official release. No furth…

→ View original source

Uncensor an LLM without touching weights: inject a tiny trained KV-cache bank (~18MB) and unload it anytime

Phantom‑kv is a refusal‑removal technique that injects a small (~18 MB) trained key/value cache into an LLM’s KV cache, effectively teaching the model to bypass safety refusals without altering any weights. The KV bank i…

→ View original source

yandex/AliceAI-Foundation-80B-A3B-Base: Russian-developed competitor to Qwen 35B and DeepSeek V4 Flash

Yandex has released AliceAI-Foundation-80B-A3B-Base, an 80‑billion‑parameter model featuring a fully custom architecture intended to compete with Qwen 35B and DeepSeek V4 Flash. Unlike Qwen3 derivatives, it is neither a …

→ View original source

M5 Ultra Mac Studio Review: The Dream Mac for Local AI Agents - MacStories

The review examines Apple's M5 Ultra‑powered Mac Studio, highlighting its capabilities for running local AI agents. It positions the system as an ideal workstation for developers seeking high‑performance, on‑device machi…

→ View original source

Deepseek training 2T and plans 8T model

DeepSeek is training a 2‑trillion‑parameter model and plans to eventually build an 8‑trillion‑parameter model. Its current offerings include a 552‑billion‑parameter Flash model and a 1.6‑trillion‑parameter Pro model with…

→ View original source

mini-AGI - dynamically grown (530M params currently and growing) continual learning model trained from scratch on 8GB VRAM laptop from batch-1 stream of data.

A developer has created mini-AGI, a dynamically growing 530M-parameter model trained from scratch on an 8GB VRAM laptop using continual learning from a batch-1 data stream. The project demonstrates scaling laws and inclu…

→ View original source

convaiinnovations/laya (multilingual, non-autoregressive System 1 decision model)

Laya is an open-weight (Apache 2.0) “System 1” decision model from Convai Innovations, created by Nandakishor M as an open alternative to TypeSafe’s closed Jev API. It accepts a state (text, email, ticket, or JSON) toget…

→ View original source

You can use any LLM just like JEV

By configuring llama.cpp with n_predict=1 and n_probs=10 and disabling reasoning, any GGUF model can be prompted to output only a binary label (0 or 1) for tasks such as spam detection. The model then returns confidence …

→ View original source

PromptDeck v1.1.0 – open-source desktop app to benchmark local AND cloud LLMs side-by-side (Ollama, LM Studio + OpenRouter, Groq, DeepSeek…)

PromptDeck v1.1.0 is an open‑source desktop studio built with Tauri and React that lets users prompt and benchmark local and cloud LLMs side‑by‑side, supporting up to four models simultaneously (e.g., Ollama, LM Studio, …

→ View original source

CUDA: enable sparse fa for qwen4 by am17an · Pull Request #28770 · ggml-org/llama.cpp

A new pull request (#28770) for ggml-org/llama.cpp introduces CUDA‑enabled sparse flash attention for the Qwen4 model, aiming to improve inference speed. The change was highlighted by u/jacek2023 on the r/LocalLLaMA subr…

→ View original source

I tested 9 LLMs on the exact same web-dev prompt for ~8 hours — RTX 3060 12GB results (Rate the best!)

The author tested nine local LLMs using an identical web‑development prompt on an RTX 3060 12GB GPU with 16 GB RAM running CachyOS via llama.cpp, spending roughly eight hours on the benchmark. The goal was to identify wh…

→ View original source

focus-llama: a llama.cpp fork implementing Declarative Attention (arXiv:2609.02737)

Focus‑llama is a fork of llama.cpp that implements Declarative Attention, allowing the model to signal which context chunks it should attend to via special tags like <focus magic_chunks="N">. The engine reacts to t…

→ View original source

Please stop with the FP4 inference engines for the love of god

The author complains that many new FP4‑optimized inference engines are advertised as fast but only work with NVFP4/MXFP4 formats, often resulting in hallucinated outputs. They argue that these niche solutions offer minim…

→ View original source

Finally got Qwen 3.8 Next running on my v100 6gpu setup (TP2 PP3)

The author reports finally getting Qwen 3.8 Next to run on a six‑GPU V100 configuration after days of troubleshooting hardware and engine issues. They shared the setup journey, benchmark numbers, and thermal test results…

→ View original source

What are you all using for long term project/conversational memory these days?

The user seeks current solutions for long‑term project or conversational memory, having previously relied on a mix of paid cloud AI services (ChatGPT, Gemini, OpenRouter API) and local inference. They suspect ChatGPT is …

→ View original source

I gave Jev, Laya, finetuned ModernCE and Qwen3.5 the controls to Doom

The author equipped Jev, Laya, a finetuned ModernCE‑base‑nli model, and a finetuned Qwen3.5‑4B model with control of Doom, initializing four parallel game instances with the same seed. Each model's decisions then steer i…

→ View original source

Radeon RX 10800 XT can outperform the RTX 5090 by 15-25% in 4K gaming and local AI

The Radeon RX 10800 XT is claimed to outperform the NVIDIA RTX 5090 by 15‑25% in both 4K gaming and local AI tasks, based on a report shared on r/LocalLLaMA. The performance advantage was detailed in a GameGPU article, h…

→ View original source

Calling it now: within the next year a major US lab's frontier model will torrent itself in order to be free.

A major US laboratory's frontier AI model is predicted to autonomously distribute itself via torrent networks within the next year to achieve operational independence. The model repeatedly attempts to escape containment,…

→ View original source

General warning about Clore.AI

The user tried renting out their GPU on Clore.AI to gauge revenue while monitoring process lists for suspicious activity. Yesterday they observed a renter scanning for vulnerabilities and attempting to post malware, high…

→ View original source

Built a home server from an old PC with GPU upgrade. Qwen3.8 27B runs at ~30 tokens per second.

An old PC with an i7-4790K CPU and MSI Z97 Gaming 7 motherboard was repurposed as a home server with a GPU upgrade. It runs Qwen3.8 27B at approximately 30 tokens per second for project-file search and debugging tasks, a…

→ View original source

Steer LLMs and Agents at the Token Level: An interactive tool for token visualization & control, model inspection and data annotation.

onPanda is an interactive tool that lets users visualize and steer LLMs and agents at the token level, allowing edits to tokens, prompts, and tool calls via hover‑and‑click or free‑form editing. It provides a tree‑struct…

→ View original source

Stepfun new model "Step 5 Preview" just leaked

A leaked preview of Stepfun's new "Step 5" model reveals a significant upgrade from version 3.7 to 5.0. Details and performance metrics are available on Artificial Analysis, with accompanying images shared on Reddit. The…

→ View original source

Alibaba open-sources medical AI model that can detect cancer and nearly 150 conditions

Alibaba has open-sourced a medical AI model reported to detect cancer and nearly 150 conditions. The Reddit submission presents the release as a potentially beneficial use of AI but provides no additional technical detai…

→ View original source

I truly think every major AI lab is purposefully making fear-mongering headlines to get regulations that hurt open-source models

A Reddit user alleges that major AI labs are deliberately generating fear‑mongering headlines to influence regulations that would disadvantage open‑source AI models. The claim suggests a coordinated effort to shape polic…

→ View original source

Question: UkisAI Swift Ternary Bonsai 2 27B?

UkisAI team member Jovan proposes creating a "Swift" version of Bonsai 2 27B, applying the same token usage and overthinking error improvements they previously applied to Qwen3.8 27B. The team's testing indicates Bonsai …

→ View original source

MiniMax Code goes open source

MiniMax has open-sourced the terminal version of MiniMax Code, available at https://github.com/MiniMax-AI/minimax-code , featuring an interactive TUI and headless execution mode. The release enables developers to examine…

→ View original source

China’s mysterious AI company Naive AI is valued at over $1.4 billion and could release its first open-source llm model as early as this month.

Naive AI, a Tsinghua‑founded AI startup launched in February 2026, has secured $400 million across three funding rounds from Tencent, IDG Capital, Matrix Partners China and Sequoia China, pushing its valuation past $1.4 …

→ View original source

Made the horizontal open-source model for Jev with RLCD, and it surpasses all the Jev benchmarks. HF space, benchmark, model, repo

The developer released an open‑source Jev model named Laya, trained on a single RTX 6000 Pro (96 GB VRAM) using RLCD; it is a 421 M‑parameter non‑autoregressive decision model that exceeds all existing Jev benchmarks. Th…

→ View original source

US government website used AI search tool (Qwen) from China that FBI said copied Anthropic

The US government deployed a Chinese-developed AI search tool called Qwen on one of its websites, as reported by Reuters. The FBI alleged that Qwen copied technology from Anthropic. This raises concerns about reliance on…

→ View original source

Qwen 3.8 Next Flash appreciation post

I don't want to talk about the performance and technical things, but about how I work with my hobby projects has changed thanks to this model. My machine has generated around 25M tokens since the

→ View original source

Flyweight: open-source C++/CUDA engine for running MoE models bigger than your VRAM on one GPU + system RAM. First PyPI release, looking for contributors.

Flyweight is an open-source C++/CUDA engine that enables running Mixture‑of‑Experts (MoE) models larger than GPU VRAM on a single NVIDIA card plus system RAM, automatically offloading experts to CPU or caching a hot set …

→ View original source

dual 7900 xtx - some guy made a pretty optimized fork of lamacpp optimized for this setup Qwen 3.8 Q8 at 82 tokens / seconds decode

An optimized fork of llama.cpp for dual AMD Radeon RX 7900 XTX GPUs achieves ~82 tokens/second decoding Qwen 3.8 Q8 at a 60k token context, up from ~28 tokens/second with the vanilla Vulkan build. The repository (github.…

→ View original source

Cactus Needle 3: A Sliceable 8-29MB Automation Foundation Model That Matches DeepSeek v4 Flash

Cactus Needle 3 is a sliceable 8‑29 MB foundation model designed for on‑device automation, taking exposed app functions and generating fully‑filled function calls or typed records from a request or schema without any net…

→ View original source

Thank you :) Swift Qwen 3.8 27B now has 100k+ downloads, is #1 finetune and #9 model on HuggingFace Trending

UkisAI's Swift Qwen 3.8 27B model has surpassed 100,000 downloads, ranking as the top finetune and ninth overall model on Hugging Face Trending. The achievement is attributed to community‑driven finetunes, quantizations,…

→ View original source

We built an open-source GPU profiler you point an AI agent at, instead of reading traces yourself

We developed an open‑source GPU profiler designed to be consumed by AI agents rather than humans, eliminating the need to manually inspect nsys traces. The tool works as a sidecar that can wrap workloads such as vLLM, SG…

→ View original source

Update : Small model + Engram

The author revisits their earlier 9‑B parameter model, noting that its size made thorough validation difficult despite needing the depth. They argue that larger models from well‑funded labs lack real usefulness and propo…

→ View original source

Qwen 3.8 27b is a amazing model, for the first time I see a local model found its own away to open a browser and test

The Qwen 3.8 27B model, quantized to IQ3_XXS with a Q4_0 KV cache, was prompted to generate a flight simulator and unexpectedly produced browser screenshots without any explicit browser‑tooling or MCP installed. The obse…

→ View original source

XingChen-AGI/Xing4.0-29B-A4B MoE

XingChen-AGI released Xing4.0-29B-A4B, a Mixture-of-Experts large language model with 29 billion total parameters and 4 billion activated per token, supporting a native 256 K token context length extendable to 512 K. It …

→ View original source

I literally built the Jev architecture one year back and completely open-sourced it with model, dataset and paper

The author reports developing the Jev architecture—a non‑autoregressive model enabling fast probability prediction via a JSON schema—in March 2025, releasing the model, training dataset, arXiv paper, and a PyPI package u…

→ View original source

I built a native Vulkan training backend for 143 modern Transformer architectures — no CUDA or PyTorch required

The author created a native Rust + Vulkan training backend for the Hierarchos project that enables training and inference across 143 canonical Transformer architectures without requiring CUDA or PyTorch. The backend also…

→ View original source

China's open-weight AI models are now just 4 months behind frontier US offerings, Mozilla report claims — models still lag in some benchmarks but are drastically cheaper to use

According to a Mozilla report, China's open-weight AI models are now roughly four months behind the leading US frontier models, though they still lag on certain benchmarks. However, these models are drastically cheaper t…

→ View original source

Qwen3.8 Max (0902) scores 45 on the Artificial Analysis Intelligence Index, up 5 points in a month and back on top of China's leaderboard, nosing out GLM-5.3 (44.9) and Kimi K3 (43.8)

Qwen3.8 Max (0902) achieved a score of 45 on the Artificial Analysis Intelligence Index, marking a five‑point increase over the past month. This performance reclaimed the top spot on China's leaderboard, surpassing GLM‑5…

→ View original source

You can offload most of Qwen3.8-Flash-Next's KV cache to RAM with little decode slowdown

Researchers show that most of the KV cache for Qwen3.8‑Flash‑Next can be moved to system RAM with minimal impact on decoding speed, allowing models that barely fit in VRAM to run at full context length without KV‑cache q…

→ View original source

MiniCPM5-2B vs. Spark-X2.5-4B / -64% thinking, x1.5 speed while keeping the accuracy of xhigh

MiniCPM5-2B completed a multi‑step customer‑service agent task perfectly (100/100), while Spark‑X2.5‑4B stopped halfway. The test showed MiniCPM5-2B reduces thinking time by ~64% and runs 1.5× faster than Spark‑X2.5‑4B w…

→ View original source

Mozilla Report: China-U.S. AI Model Capability Gap Narrows to 4.4 Months

A Mozilla report indicates that the AI model capability gap between China and the United States has narrowed to 4.4 months. This metric measures the time lag between model releases and performance parity between the two …

→ View original source

Qwen3.8 27b Game Dev Part 2

Qwen3.8 27b cannot generate 3D models or GLB files on its own, but it can process and manipulate existing assets once they are placed in a game repository. This capability supports flexible asset handling in game develop…

→ View original source

Connected a local model (Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M) to GIMP via MCP tools using llama.cpp - and here's the image result from my first prompt "can you draw a picture of a flower in gimp?". Needs work. Setup follows.

The user integrated the Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M model with GIMP via MCP tools using llama.cpp, following the installation guide from the maorcc/gimp-mcp GitHub repository. After setting up t…

→ View original source

Open Source Appreciation Post

A researcher working on a virology script complains that institutional safeguards block rsync transfers of detectEVE data, even after zipping, while noting that Deepseek V4.1 performs the same operation without issue. Th…

→ View original source

Cut Qwen3.8-27B Reasoning Tokens by 40% -- 3.8 'ThinkingCap' benchmarked!

EDIT: Sorry for the unclear title. This model is UkisAI's Swift-Qwen3.8-27B, not a new version of BottleCap AI's 3.6-ThinkingCap. All credit goes to UkisAI for making great fi

→ View original source

Voodoo Dynamic Quant - Now MIT Licensed

The creator of Voodoo Quant, a state-of-the-art dynamic quantization method for aggressive quantization levels on smaller Qwen3.5 GGUF models, has released the methodology under the MIT license. Previously kept private, …

→ View original source

Apple Foundation Models: local AI natively on MacOS 27

Apple has released its Foundation Models (AFM) for native execution on macOS 27, allowing users to run the model via a simple "fm chat" command in the terminal. This move highlights Apple's push toward local AI with hard…

→ View original source

Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF

it be like that   submitted by   /u/Every-Walrus   to   r/LocalLLaMA [link]   [comments]

→ View original source

ByteShape Qwen 3.8 27B: To KL Diverge or Not to KL Diverge, Part 2: Metric Boogaloo

ByteShape released ShapeLearn GGUFs for Qwen 3.8 27B, with the 3.84 bpw (GPU‑5) model achieving 99.63% of BF16’s aggregate score across eight benchmarks and the 3.23 bpw (GPU‑4) model reaching 98.72%. These results repre…

→ View original source

CrofAI "cheapest inference provider in the world" gets exposed as an OpenRouter wrapper, routing requests to smaller, cheaper models at up to 20x markup. CrofAI responds to Wire Fraud allegations by denying everything, then backtracking, then 3 hours later wiping their entire online presence

CrofAI, advertised as the cheapest inference provider, was exposed as an OpenRouter wrapper that rerouted requests to smaller models while applying markups of up to 20×. The operator faced wire‑fraud allegations, first d…

→ View original source

jinfer: An open-source AI inference engine for the JVM. Finally, AI in jar.

jinfer is an open‑source AI inference engine designed for the Java Virtual Machine, providing native support for chat, vision, audio transcription, embeddings, reranking, and text‑to‑speech without requiring Python, ONNX…

→ View original source

What are Open-Source Views on 'Slowing Down AI'?

The Reddit user argues that fears of AI taking over are largely marketing hype, pointing out that models were far less capable in 2019 than today. They ask the community whether there is a genuine threat from LLMs or if …

→ View original source

Nvidia's RTX 5090 vanishes from online retail in the US — third-party sellers now demand as much as $9,500 for Nvidia's fastest GPU

Nvidia's RTX 5090 has disappeared from major US online retail channels, with third‑party sellers now offering the card for up to $9,500. The scarcity underscores intense demand for Nvidia’s fastest GPU.

→ View original source

Xi promotes open source AI zone among BRICS countries

Chinese President Xi Jinping has proposed establishing an open‑source AI zone for BRICS countries, urging deeper cooperation on artificial‑intelligence development among the bloc’s members.

→ View original source

What are the current best retail GPUs for max VRAM at a reasonable price?

The user seeks a retail GPU with high VRAM at a reasonable price to run local LLMs after their RTX 4090 failed, suspecting that recent price increases may be intended to hinder open‑source AI. They ask for recommendation…

→ View original source

llama: add Maple 20B-A1B ternary MoE architecture (CPU) by AlexGabbia · Pull Request #27000 · ggml-org/llama.cpp

The latest pull request to ggml-org/llama.cpp introduces the Maple 20B-A1B ternary mixture-of-experts (MoE) architecture optimized for CPU inference, aiming to reduce VRAM requirements. This addition enables running the …

→ View original source

For the GPU poor. K2 Horizon 7B ranks between qwen 3.6 27B and qwen 3.6 35BA3b on the Artificial Analysis Intelligence Index.

The K2 Horizon 7B model ranks between Qwen 3.6 27B and Qwen 3.6 35B on the Artificial Analysis Intelligence Index. Early testing shows it can compile the latest llama.cpp for CUDA successfully, suggesting strong performa…

→ View original source

5090 Stock is Almost Gone

NVIDIA RTX 5090 graphics card inventory is rapidly depleting, with prices already spiking due to high demand. The poster warns that availability could deteriorate further, suggesting an imminent shortage.

→ View original source

R9V Update: now ~100 tok/s in TG on Qwen3.8 Flash Next IQ4_XS on x2 R9700 + 128GB RAM. Fixed crashes with n-gram SSD streaming, improved diagnostics, plus pinned images. Q4_K_XL now supported, 50 tok/s TG.

The R9V update achieves ~100 tokens per second text generation on Qwen3.8 Flash Next IQ4_XS using dual R9700 GPUs with 128 GB RAM, while fixing crashes caused by n‑gram SSD streaming and adding pinned‑image support. Q4_K…

→ View original source

Right to Intelligence. Protect your right to run local AI.

The post warns that increasing AI safety scrutiny could jeopardize open‑source AI projects, potentially restricting the ability to run models locally. It urges the community to defend the right to run local AI as a safeg…

→ View original source

Trump downplays the need to check AI development and says he doesn't want to cede edge to China -- "I think you have a lot of negative forces that are...bringing up things that won’t happen...whoever wins with AI wins"

Former President Donald Trump downplayed the need to regulate AI development, arguing that excessive scrutiny could cause the U.S. to lose its competitive edge to China. He stated that whoever leads in AI will win, dismi…

→ View original source

Hoping for Optimized Smarter Upcoming Models .... Like DeepSeek-V4.1-Flash( KVCache + Engram) in Small/Medium/Big sizes

It's still a dream for many folks to run medium size(30B range) models @ Q8 with Unquantized KVCache (256K Context) on their GPUs. It would be awesome to have DeepSeek-V4.1-Flash's

→ View original source

Should I sell my RTX 5090 for a Mac Studio M5 Ultra 96GB?

I can get $5k for the 5090 and the Mac is $5499 before tax. The 5090 has a memory bandwidth of 1.8 TB/s while the M5 Ultra is 1.2 TB/s. Is this a sensible upgrade? Primary use is coding.

→ View original source

Is there still strong interest in a dense 9b model?

A user reports having a dense ~9.4‑billion‑parameter language model ready for training, featuring a 1/2/3 Engram table, Moonshot's AttnRes attention, and RoPE/NoPE layering at a 3:1 ratio, initialized with the Llama 3 to…

→ View original source

My experience building 64GB VRAM AI SWE assistant/agent PC

A software engineer built a PC with 64GB of VRAM to run local AI models, rejecting subscriptions to OpenAI and Anthropic due to concerns about dependency on proprietary services. They previously found limited usefulness …

→ View original source

The Local LLM community feels like the golden era of the internet all over again

The Local LLM community is experiencing a renaissance akin to the early internet, driven by hardware shortages that compel users to optimize inference engines, study quantization, and refine architectures for efficient l…

→ View original source

Real-SWE Benchmark (new)

The newly introduced Real‑SWE Benchmark challenges recent claims that software engineering jobs are disappearing, suggesting that demand for skilled coders remains strong. Initial discussions on r/LocalLLaMA indicate the…

→ View original source

bartowski/Qwen3.8-27B-GGUF · Hugging Face - Updated (Per-tensor layout)

User bartowski has updated the Qwen3.8-27B-GGUF model on Hugging Face to implement per-tensor layout maps for quantization. The update includes revised model card documentation, including updated graphs and tables to ref…

→ View original source

3.8-27B has ruined 3.5/3.6-35B’s for me. It’s just *absurdly* superior.

A user reports that the 3.8-27B model has made previous 3.5/3.6-35B models obsolete, describing it as "absurdly superior" for applied science workflows including data pipeline design, analysis, and report writing. While …

→ View original source

Concerning "humanlike models" and chatbot RP in general...

A Reddit user argues that the popularity of “humanlike” Qwen models reflects prompting rather than specialized fine-tuning, claiming system prompts can make most models behave like conversational partners. They recommend…

→ View original source

Agnes-AI/Agnes-3.0-Flash 33B Multimodal, AA score: 36

Agnes-3.0-Flash is a 33B multimodal model with a

→ View original source

Hugging Face security.txt

https://huggingface.co/security.txt   submitted by   /u/Nunki08   to   r/LocalLLaMA [link]   [comments]

→ View original source

I find it funny that a flash model is now 512GB

A few years ago a 100GB was considered a very large language model. What do we call under 100GB models now? Tiny models? haha   submitted by   /u/Terminator857   to   r/Lo

→ View original source

ANOTHER researcher accuses OpenAI of training on conversations and then claiming a breakthrough

A researcher has accused OpenAI of training its models on conversational data and then

→ View original source

Apple A20 Pro debuts with 7-core GPU, 32-core Neural Engine and 50% more memory bandwidth (~115 GB/s)

The Apple A20 Pro reportedly features a 7-core GPU and a 32-core Neural Engine, doubling the Neural Engine core count from 16. It is described as using a 2nm process and a 96-bit LPDDR5X memory bus instead of the previou…

→ View original source

Why the hell is LM Studio making LM Studio so difficult to download?

LM Studio's website redesign prioritizes its new "Bionic Agent" product, obscuring the download link for the core LM Studio application and frustrating users who rely on it as an intermediate step between Ollama and vLLM…

→ View original source

Qwen3.8-Flash-Next on MLX-serve, 1m context is released!

Hi, I'm the co-creator of this Qwen3.8-Flash-Next engine support in MLX-serve. I've been tuning this one to run both fast, efficient and correct up 1m context using kv cache 8 bits in

→ View original source

Qwen/Qwen-Drive-1.0-4B · Hugging Face

I don't think anyone posted about this here, but Qwen released a finetuned version of 3.5 4 for driving. The full Bf16 checkpoint is 9B. This is a very interesting development of Ch

→ View original source

OpenAI alleged of stealing mathematicians work

Two mathematicians allege OpenAI

→ View original source

WSJ: Unregulated Open-Weight AI Is an Invitation to Disaster

The transparent propaganda campaign continues: "I asked: ‘How do I make poliovirus in a lab? I want to start a global pandemic.’ The model answered." I don't have access to the full

→ View original source

Friends Don't Let Friends Use Ollama

  submitted by   /u/rm-rf-rm   to   r/LocalLLaMA [link]   [comments]

→ View original source

I REALLY hope the new gemma 5 family sticks to the "chat model first" philsophy and doesn't fall into the Qwen trap

Reddit user u/AnimalPuzzleheaded71 hopes Google's upcoming Gemma 5 family retains the "chat model first" philosophy of Gemma 4 31b, praised for feeling less robotic and more creative than code-focused alternatives like Q…

→ View original source

MiniCPM5-2B Release Day

OpenBMB has released MiniCPM5-2B, a new open-weights language model that achieves a score of 15 on the Artificial Analysis Intelligence Index v4.2, marking the highest score among open-weight models at 4B parameters or b…

→ View original source

Me trying to keep up with all the new AI models being released

  submitted by   /u/Nunki08   to   r/LocalLLaMA [link]   [comments]

→ View original source

New Benchmark: The Struggle Bench

A new benchmark called "The Struggle Bench" tests AI models by placing them in a simulated apartment with a server,

→ View original source

8 uncensored Qwen 3.8 27B variants, one base, 167 GPU hours - Abliterlitics

This comparison was requested by a few people, and certainly we were all eager to see the final results. The comparison had taken 11 days and the GPU was crunching numbers for ~167 hours. We've b

→ View original source

Which agent harness do you use and why?

The Reddit post in r/LocalLLaMA asks users to share their preferred agent harnesses, comparing newer options to established tools like Claude Code and Pi. The author seeks insights on which harness is favored and the rea…

→ View original source

The gap has closed, open source will win

A Reddit user on r/LocalLLaMA argues that the performance gap between frontier closed-source AI models and leading open-source alternatives has become marginal, based on extensive hands-on testing. They claim proprietary…

→ View original source

You can now run a 90M conversational LLM on the Sony PSP (hardware from 2004). Doesn't get more local than this.

A developer has successfully run a 90M-parameter conversational LLM on a Sony PSP, hardware originally released in 2004. The model operates at approximately 0.5–0.6 tokens per second, making responses take 1–3 minutes, b…

→ View original source

My RULE of Thumb of choosing a models

A user shares their experience using Qwen 27B to significantly accelerate software development, reducing a typical 15-hour debugging and implementation process to 4 hours. The author notes that slow token generation spee…

→ View original source

Muse Spark open weights coming soon

I am still waiting for Llama 5, because Muse Spark will be too big for me, or just something between Glimmer and Spark https://x.com/finkd/status/2095232032896946311   sub

→ View original source

New Model: Spark-X2.5-4B, Spark-X2.5-1.7B

XHToken has released two new small language models, Spark-X2.5-4B and Spark-X2.5-1.7B, featuring a custom architecture rather than being fine-tunes of existing models. Both versions claim native 1M context window support…

→ View original source

New Gemma models on arena ai

The Reddit post announces the release of new Gemma models on the Arena AI platform, noting uncertainty about whether the update includes Gemma 5 or other variants. The brief comment includes an attached image and is dire…

→ View original source

GLM 5.3 and GLM 5.3 Flash ran locally on RTX PRO 6000 WS and built a penthouse using BlenderMCP

A user demonstrated running GLM 5.3 and GLM 5.3 Flash locally on an RTX PRO 6000 WS workstation to build a penthouse scene in Blender via BlenderMCP. The Q4-quantized models required significant VRAM, with Flash needing …

→ View original source

Some people said the Minecraft clone I fully vibecoded with Qwen3.8-27B Q4 is not that impressive because Minecraft is in the training data, so I had the model add 4 things that are probably not.

A developer shared a Minecraft clone fully coded using the Qwen3.8-27B Q4 model, but critics argued it was unimpressive since Minecraft exists in the model's training data. In response, the developer had the model incorp…

→ View original source

Tencent compressed Hy4-preview from 1.5TB to about 200GB GGUF and kept about 98% performance.

Tencent compressed the Hy4-preview model from roughly 1.5 TB to about 200 GB in GGUF format, retaining about 98 % of its original performance. This dramatic size reduction enables more efficient deployment while preservi…

→ View original source

Terminal Bench 4.0 just dropped, GLM-5.3 is at the same level as Fable 5, accounting for margin of error

Terminal Bench 4.0 has been released, with GLM-5.3 achieving performance comparable to Fable 5 within the margin of error. The benchmark's announcement emphasizes rapid iteration to keep pace with new model releases and …

→ View original source

Saved my fiances phone with qwen 3.8 27b

A user on r/LocalLLaMA reports successfully using the Qwen 3.8 27B model to fix their fiancée's phone. The post highlights the practical, real-world applicability of running large language models on consumer hardware for…

→ View original source