…It ran smoothly on my setup through GPU offloading, even though it’s officially designed for 16GB of VRAM. For most things, it was fine. I would primarily prompt it for quick…
…samples. The script for this process is provided below, showing how to prune using a two-GPU pipeline parallel setup. torchrun --nproc_per_node 2 /opt/NeMo/scripts/llm/gpt_prune.py…
…Related 7 things I wish I knew when I started self-hosting LLMs I've been self-hosting LLMs for quite a while now, and these are all of the things I…
…Sign in to your XDA account I’ve been running my local LLM for a while now, and it’s been hit and miss for me. For starters, I do kind of…
…how deep the conversation can actually go for anyone with a smaller GPU. Related I ran the same prompts through Claude and my local LLM, and the results weren't what I…
…That's more than many GPUs can hold, and that's before you account for the model weights. Modern models have largely moved to GQA, which shares KV heads across multiple query…
…v2 LLM from the NVIDIA API Catalog . These APIs are useful for evaluating many models, quick experimentation, and getting started is free. However, for the unlimited performance and control needed in production…
As the title states, my build is indeed able to run a 1 trillion parameter model (in this case Kimi K2.5) locally at ~4 tokens/second. I thought r/LocalLLaMA would be interested in the build due to that stat line, and al…
Just wanted to share my config in hopes of helping other 12GB GPU owners achieve what I see as very respectable token generation speeds with modest VRAM. Using the latest llama.cpp build + MTP PR, I got over 80 tok/sec w…
2026-05-07 edit: I have updated the hardware based recommendations with more focus on quality. I do not recommend q4_0 KV cache anymore beyond 64k context. After multiple rounds of testing with the different size quants,…
The following is a non-comprehensive test I came up with to test the quality difference (a.k.a degradation) between different quantizations of Qwen 3.6 27B. I want to figure out what's the best quant to run on my 16 GB V…