Techyon

CUDA: enable sparse fa for qwen4 by am17an · Pull Request #28770 · ggml-org/llama.cpp

A new pull request (#28770) for ggml-org/llama.cpp introduces CUDA‑enabled sparse flash attention for the Qwen4 model, aiming to improve inference speed. The change was highlighted by u/jacek2023 on the r/LocalLLaMA subr…

→ View original source

I tested 9 LLMs on the exact same web-dev prompt for ~8 hours — RTX 3060 12GB results (Rate the best!)

The author tested nine local LLMs using an identical web‑development prompt on an RTX 3060 12GB GPU with 16 GB RAM running CachyOS via llama.cpp, spending roughly eight hours on the benchmark. The goal was to identify wh…

→ View original source

Abliterated Qwen 3.8 flash can do anything

The user deployed an abliterated Qwen 3.8 Flash model on a Strix Halo machine, enhancing speed and token prefill. By employing a Fable‑based trick to abliterate the model, they achieved zero refusal and cracked a patron‑…

→ View original source

focus-llama: a llama.cpp fork implementing Declarative Attention (arXiv:2609.02737)

Focus‑llama is a fork of llama.cpp that implements Declarative Attention, allowing the model to signal which context chunks it should attend to via special tags like <focus magic_chunks="N">. The engine reacts to t…

→ View original source

we made a 27b model for creative writing. performs as good as claude fable 5, at a 40x cheaper price, open weights.

A Swiss and South African lab released hemmingway-1, a 27b open-weights model optimized for creative writing, roleplay, and dialogue, matching Claude fable 5's performance at 40x lower cost. It benchmarks at 1330 on eq-b…

→ View original source

Qwen-Image-2.1: Compact, efficient, and unified image creation

Qwen-Image-2.1 presents a compact and efficient unified model for image creation. It aims to streamline generation processes while maintaining performance.

→ View original source

Please stop with the FP4 inference engines for the love of god

The author complains that many new FP4‑optimized inference engines are advertised as fast but only work with NVFP4/MXFP4 formats, often resulting in hallucinated outputs. They argue that these niche solutions offer minim…

→ View original source

Finally got Qwen 3.8 Next running on my v100 6gpu setup (TP2 PP3)

The author reports finally getting Qwen 3.8 Next to run on a six‑GPU V100 configuration after days of troubleshooting hardware and engine issues. They shared the setup journey, benchmark numbers, and thermal test results…

→ View original source

Anybody tried NVIDIA PAIR

A Reddit user in r/LocalLLM asked whether anyone has tried NVIDIA PAIR, a tool for automatically routing requests to idle systems. They are seeking real‑world insights into how it performs in practice.

→ View original source

Meta Muse is Here: Cloud-Powered Personal AI Agent

Meta Muse is a cloud-powered personal AI agent introduced by Meta. The article questions whether we are quietly handing complete control of daily life to algorithms.

→ View original source

What are you all using for long term project/conversational memory these days?

The user seeks current solutions for long‑term project or conversational memory, having previously relied on a mix of paid cloud AI services (ChatGPT, Gemini, OpenRouter API) and local inference. They suspect ChatGPT is …

→ View original source

I gave Jev, Laya, finetuned ModernCE and Qwen3.5 the controls to Doom

The author equipped Jev, Laya, a finetuned ModernCE‑base‑nli model, and a finetuned Qwen3.5‑4B model with control of Doom, initializing four parallel game instances with the same seed. Each model's decisions then steer i…

→ View original source