Retail Archives
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
Search
Related topics:
LLMs
All sources
xda-developers.com
40
developer.nvidia.com
31
blogs.nvidia.com
14
amd.com
7
theregister.com
3
intel.com
3
phoronix.com
2
wccftech.com
2
press.asus.com
2
huggingface.co
2
pcgamer.com
2
extremetech.com
1
Videos
More videosBanking Archives
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
Genomics Archives
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
Inference Archives
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
Top stories
AMD just dropped a compact AI workstation that makes discrete GPUs look outdated for running LLMs
5d agoI added a second GPU just for local AI workloads, and it cost less than upgrading my main one
1w ago13 years later, the GTX Titan is still the most important GPU Nvidia ever made
1w agoMy RTX 5090 can't keep up with Apple Silicon on the biggest local LLMs, and I hate to admit it
1w agoNemotron Archives
…GPU and 1,000 tokens per second per user on gpt-oss with the latest NVIDIA TensorRT-LLM stack. As AI shifts from one-shot answers to complex reasoning, the demand for…
Paper page - MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
…Context-Aware Hybrid Attention for Efficient LLMs Inference (2026) LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Models (2026) Forget, Then Recall: Learnable Compression and Selective Unfolding via Gist Sparse…
PS3 emulation project makes polite but firm request after being inundated with AI slop code
…favorites 3 Best graphics cards in 2026: These are the GPUs worth spending money in right now 4 Best gaming laptop 2026: I've tested the best laptops for gaming of this…
Discussions and forums
My first 10 inch rack with local LLM! No more Spotify, Google Home, Netflix, ChatGPT...
I'm pretty new to homelabbing and this is my first mini rack! Started with the Beelink ME Mini and then just kinda grew from there (it's always the way hey haha). It idles at 70 watts (not too shabby for how much is goin…
Computer build using Intel Optane Persistent Memory - Can run 1 trillion parameter model at over 4 tokens/sec
As the title states, my build is indeed able to run a 1 trillion parameter model (in this case Kimi K2.5) locally at ~4 tokens/second. I thought r/LocalLLaMA would be interested in the build due to that stat line, and al…
80 tok/sec and 128K context on 12GB VRAM with Qwen3.6 35B A3B and llama.cpp MTP
Just wanted to share my config in hopes of helping other 12GB GPU owners achieve what I see as very respectable token generation speeds with modest VRAM. Using the latest llama.cpp build + MTP PR, I got over 80 tok/sec w…
2.5x faster inference with Qwen 3.6 27B using MTP - Finally a viable option for local agentic coding - 262k context on 48GB - Fixed chat template - Drop-in OpenAI and Anthropic API endpoints
2026-05-07 edit: I have updated the hardware based recommendations with more focus on quality. I do not recommend q4_0 KV cache anymore beyond 64k context. After multiple rounds of testing with the different size quants,…
services with actually generous free tiers for open-source projects. my list, what would you add?
Been in the weeds shipping an OSS side project for the past few weeks (social media publishing API). Real launch post is coming, this isn't that. Along the way I kept a list of services that actually have usable free tie…
Local LLMs are actually good now, and I wasted months not realizing it
…Related 7 things I wish I knew when I started self-hosting LLMs I've been self-hosting LLMs for quite a while now, and these are all of the things I…
I rebuilt my VS Code setup from scratch this year, and it's the fastest it's ever been
…Related Your old GPU can still run big LLMs – you just need the right tweaks There's a lot you can do with these models Integrating local LLMs with VS Code increased…
Maximizing Memory Efficiency to Run Bigger Models on NVIDIA Jetson | NVIDIA Technical Blog
…SGLang enables flexible and programmable inference workflows. Llama.cpp and NVIDIA TensorRT Edge-LLM are optimized for memory-efficient execution in resource-constrained environments. These frameworks provide the infrastructure needed to serve…