… Likewise, even a dinosaur GPU can run hardware-aided transcoding tasks in Jellyfin or Plex, if you’re willing to toss extra money for its premium license . Even Immich can harness your GPU for machine learning-aided facial recognition, object detection, and smart search tools. …
… But I’ve spent the last couple of months moving away from cloud LLMs for my everyday tasks, partly since I don’t want external servers gaining access to my data, and also because I’d rather avoid the extra charges incurred by paid API usage. …
… Related I over-engineered my home lab on purpose, and it's the best decision I've made Over-engineering everything is the biggest joy of home labbing, and you can't convince me otherwise Quick Sync turns iGPUs into terrific media transcoding solutions Even cheap embedded Intel processors can hold t… …
… While it might not be adept at running the newest titles, old gaming GPUs have a ton of potential in a home server build, where they can excel at local media transcoding, GPU passthrough for VMs, and local AI. …
… For queries where you don't want your sensitive data exposed to cloud servers, and don't mind waiting a bit, shift to your local LLM. For summarizing documents, generating ideas, local assistant tasks, and similar tasks, your local AI powered by an old GPU is more than capable. …
… A They support CUDA, NVIDIA's AI acceleration framework B They have dedicated AI cores running at extremely high clock speeds C They use unified memory shared between CPU and GPU, enabling fast access to large models D They have more CPU cores than equivalent Intel processors That's right! …
… Related Nvidia stopped supporting my GPU, so I started self-hosting LLMs with it I self-support my gpu now because Nvidia won't I went with the Vulkan variant of llama.cpp for this project Getting GPU passthrough working was the easy part Let me make this clear: Ollama is a fantastic local LLM prov… …
… Definitely, but it complements my setup fairly well, especially once you throw GPU passthrough into the mix… Related I tried using a Proxmox-based Windows 11 VM as my daily driver - here's how it went All it took was a little bit of tinkering and a whole lot of patience Even GPU acceleration works … …
… You might have to wait a little longer, depending on the model's size and your GPU's processing power, but it'll still work eventually. And by using LM Studio or other similar model servers, you can also use your CPU and system RAM if your GPU can't fit all the models you want in VRAM. …
… Related I ran local LLMs on Intel's cheapest iGPU, and the results were surprisingly decent It ain't no match for a dedicated GPU, but you can run some light LLMs on the N100 That said, I wouldn’t leave Pi unsupervised I’ve also set up guard rails to prevent misinterpreted prompts from damaging my … …