… Offloading layers lets me run massive LLMs on weak GPUs That’s how I managed to deploy Qwen3.6-35B-A3B on 12GB of VRAM Although your GPU is the ideal component for providing extra processing oomph to your LLMs, it’s not the only device capable of running them. …
… The most notable feature is PagedAttention , which dynamically pages the KV cache in and out instead of allocating fixed GPU resources per request. vLLM also uses continuous batching to keep the GPU fully saturated rather than idle, which gets your work done faster. …
… Because whole layers vary in size, each GPU needs to collect differently sized parameter updates from different GPUs through all gatherv . …
… I went through this in detail with a drilldown on the “Zion” and “ZionEX” and “Grand Teton” hybrid CPU-GPU systems designed by Meta Platforms way back in October 2022 , showing how the DLRMs were just as parameter-intense and flops-hungry as the LLMs of the time. …
… All the quantized variants of the Llama 3 70B model can be served using only one NVIDIA H100 GPU while the baseline FP16 precision requires at least two GPUs. …
… The issue there is that RAM is actually really slow, at least compared to VRAM on a GPU or CPU cache. The former is the best choice for running LLMs with Nvidia GPUs and tool sets leading the way. But how much do you need to spend on a GPU to comfortably run an LLM with decent results? …
… Obsession with GPUs is real GPUs are important, but not everything When you first get into self-hosting LLMs , everything revolves around the GPU; and honestly, that makes sense. …
As the title states, my build is indeed able to run a 1 trillion parameter model (in this case Kimi K2.5) locally at ~4 tokens/second. I thought r/LocalLLaMA would be interested in the build due to that stat line, and al…
Just wanted to share my config in hopes of helping other 12GB GPU owners achieve what I see as very respectable token generation speeds with modest VRAM. Using the latest llama.cpp build + MTP PR, I got over 80 tok/sec w…
2026-05-07 edit: I have updated the hardware based recommendations with more focus on quality. I do not recommend q4_0 KV cache anymore beyond 64k context. After multiple rounds of testing with the different size quants,…
The following is a non-comprehensive test I came up with to test the quality difference (a.k.a degradation) between different quantizations of Qwen 3.6 27B. I want to figure out what's the best quant to run on my 16 GB V…
… This is why an older high-end GPU like the RTX 3090 is better than newer cards for local AI inference . It has 24GB of GDDR6X VRAM while being powerful enough for most modern LLMs. Even the value for money of a used RTX 3090 is hard to resist, considering the prices of GPUs in today's market . …
… Related I started self-hosting LLMs and absolutely loved it Who needs OpenAI when your home lab can do the thinking for you? …
… Related 7 things I wish I knew when I started self-hosting LLMs I've been self-hosting LLMs for quite a while now, and these are all of the things I learned over time that I wish I knew at the start. …