Healthcare and Life Sciences Archives
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
Search
Related topics:
LLMs
All sources
xda-developers.com
40
developer.nvidia.com
32
blogs.nvidia.com
14
amd.com
7
theregister.com
3
phoronix.com
2
wccftech.com
2
press.asus.com
2
pcgamer.com
2
extremetech.com
1
cnx-software.com
1
nextplatform.com
1
Videos
More videosFast, Low-Cost Inference Offers Key to Profitable AI
…To run state-of-the-art LLMs in real time, enterprises need multiple GPUs working in concert. Tools like the NVIDIA Collective Communication Library , or NCCL, enable multi-GPU systems to quickly…
I replaced my local LLM with a model half its size and got better results — And it wasn't about the parameters
…It ran smoothly on my setup through GPU offloading, even though it’s officially designed for 16GB of VRAM. For most things, it was fine. I would primarily prompt it for quick…
Pruning and Distilling LLMs Using NVIDIA TensorRT Model Optimizer | NVIDIA Technical Blog
…samples. The script for this process is provided below, showing how to prune using a two-GPU pipeline parallel setup. torchrun --nproc_per_node 2 /opt/NeMo/scripts/llm/gpt_prune.py…
Top stories
I built my own Googlebook with a Raspberry Pi, local LLMs, and old hardware
1w agoI added a second GPU just for local AI workloads, and it cost less than upgrading my main one
1w ago13 years later, the GTX Titan is still the most important GPU Nvidia ever made
1w agoMy RTX 5090 can't keep up with Apple Silicon on the biggest local LLMs, and I hate to admit it
1w agoOptimized Software for Professionals With AMD and ISV Solutions
…Epic Games AMD partners with Epic Games to optimize Unreal Engine for peak performance on AMD CPUs and GPUs, delivering enhanced graphics and speed for developers and users. Maxon AMD and Maxon…
AR / VR – NVIDIA Technical Blog
…Your Essential Tool for Measuring GPU Interconnect and Memory Performance When you’re writing CUDA applications, one of the most important things you need to focus on to write great code is…
Developer Tools & Techniques – NVIDIA Technical Blog
…Your Essential Tool for Measuring GPU Interconnect and Memory Performance When you’re writing CUDA applications, one of the most important things you need to focus on to write great code is…
Discussions and forums
My first 10 inch rack with local LLM! No more Spotify, Google Home, Netflix, ChatGPT...
I'm pretty new to homelabbing and this is my first mini rack! Started with the Beelink ME Mini and then just kinda grew from there (it's always the way hey haha). It idles at 70 watts (not too shabby for how much is goin…
Computer build using Intel Optane Persistent Memory - Can run 1 trillion parameter model at over 4 tokens/sec
As the title states, my build is indeed able to run a 1 trillion parameter model (in this case Kimi K2.5) locally at ~4 tokens/second. I thought r/LocalLLaMA would be interested in the build due to that stat line, and al…
80 tok/sec and 128K context on 12GB VRAM with Qwen3.6 35B A3B and llama.cpp MTP
Just wanted to share my config in hopes of helping other 12GB GPU owners achieve what I see as very respectable token generation speeds with modest VRAM. Using the latest llama.cpp build + MTP PR, I got over 80 tok/sec w…
2.5x faster inference with Qwen 3.6 27B using MTP - Finally a viable option for local agentic coding - 262k context on 48GB - Fixed chat template - Drop-in OpenAI and Anthropic API endpoints
2026-05-07 edit: I have updated the hardware based recommendations with more focus on quality. I do not recommend q4_0 KV cache anymore beyond 64k context. After multiple rounds of testing with the different size quants,…
services with actually generous free tiers for open-source projects. my list, what would you add?
Been in the weeds shipping an OSS side project for the past few weeks (social media publishing API). Real launch post is coming, this isn't that. Along the way I kept a list of services that actually have usable free tie…
Nvidia slaps Groq into new LPX racks for faster AI response
…To be clear, LPUs won't replace Nvidia's GPUs but rather augment them. Speed for decode LLM inference encompasses two stages: the compute-heavy prefill phase in which the prompts are…
Launching AMD AI Playbooks: Step-by-Step Guides for Building with AI Locally with AMD
…Who they are for? AI Playbooks are written for developers who are comfortable with the basics: The command line Python environments Basic AI/ML concepts You don’t need deep expertise in…
My local LLM is the best productivity tool I've installed in years, and it costs nothing to run
…It actually fits into my setup Integrations are simple Using a local LLM on its own is all good and well if it’s just for the purposes of extracting information. But…