… Rather, I began using the llama-server functionality to create an LLM server that remains operational 24/7 and hooks up to the rest of my FOSS arsenal thanks to its OpenAI-compatible API. …
… Intel iGPUs win on the energy efficiency front as well And I don’t have to go out of my way to grab a dedicated GPU The biggest drawback of managing your own media, aside from the remote backup requirement and a crippling urge to go out and buy new hardware, is that you have to keep an eye on the e… …
… Related I built my first home server for under $200, and it replaced 4 monthly subscriptions An old, dormant system has the potential to replace subscriptions worth hundreds of dollars a year Old GPUs do have their limitations Constrained by their nature These old gaming GPUs do show their age, and… …
… For queries where you don't want your sensitive data exposed to cloud servers, and don't mind waiting a bit, shift to your local LLM. For summarizing documents, generating ideas, local assistant tasks, and similar tasks, your local AI powered by an old GPU is more than capable. …
… A It automatically connects to OpenAI's servers for faster processing B It provides a built-in ChatGPT-style chat interface and a local API server with no coding required C It can train custom models from scratch using your own data D It requires a subscription to unlock models larger than 3 billio… …
… Related Nvidia stopped supporting my GPU, so I started self-hosting LLMs with it I self-support my gpu now because Nvidia won't I went with the Vulkan variant of llama.cpp for this project Getting GPU passthrough working was the easy part Let me make this clear: Ollama is a fantastic local LLM prov… …
… Definitely, but it complements my setup fairly well, especially once you throw GPU passthrough into the mix… Related I tried using a Proxmox-based Windows 11 VM as my daily driver - here's how it went All it took was a little bit of tinkering and a whole lot of patience Even GPU acceleration works … …
… You might have to wait a little longer, depending on the model's size and your GPU's processing power, but it'll still work eventually. And by using LM Studio or other similar model servers, you can also use your CPU and system RAM if your GPU can't fit all the models you want in VRAM. …
… Related I ran local LLMs on Intel's cheapest iGPU, and the results were surprisingly decent It ain't no match for a dedicated GPU, but you can run some light LLMs on the N100 That said, I wouldn’t leave Pi unsupervised I’ve also set up guard rails to prevent misinterpreted prompts from damaging my … …
… Once I've finished a few upgrades to my server box it'll get moved over, along with the GPU, and the only thing I'll need to adjust is how the networking is set up on the Docker stack. …