Selfhosted

• Small 4B models like gemma3 will run on anything (I have it running on a 2020 laptop with integrated graphics). Don't expect superintelligence, but it works for basic classification tasks, writing/reviewing/fixing small scripts and basic chat, writing, etc
• I use https://github.com/ggml-org/llama.cpp in server mode pointing to a directory of GGUF model files downloaded from huggingface. I access it it from the built-in web interface or API (wrote a small assistant script)
• To load larger models you need more RAM (preferably fast VRAM/GPU but DDR5 on the motherboard will work - it will be noticeably slower). My gaming rig with 16GB AMD 9070 runs 20-30B models at decent speeds. You can grab quantized (lower precision, lower output quality) versions of those larger models if the full-size/unquantized models don't fit. Check out https://whatmodelscanirun.com/
• For image generation I found https://github.com/vladmandic/sdnext which works extremely well and fast wth Z-Image Turbo, FLUX.1-schnell, Stable Diffusion XL and a few other models

As for the prices... well the rig I bought for ~1500€ in september is now up to ~2200€ (once-in-a-decade investment). It's not a beast but it works, the primary use case was general computing and gaming, I'm glad it works for local AI, but costs for a dedicated, performant AI rig are ridiculously high right now. It's not economically competitive yet against commercial LLM services for complex tasks, but that's not the point. Check https://old.reddit.com/r/LocalLLaMA/ (yeah reddit I know). 10k€ of hardware to run ~200-300B models, not counting electricity bills

I've run koboldcpp on a steam deck. You have to stick to small model files, of course, like maybe 4Gb, but you can get decent speed if you do.

And Edge Gallery on Android can run models locally on a phone.

If your focus is LLMs, get a 3090 gpu. Vram is the most important thing here because it determines what models you can load and run at a decent speed, and having 24Gb will let you run the mid range models that specifically target this amount of memory because of this being a very standard amount to have for hobbyists. These models are viable for coding, the smaller ones are less so. Looking at prices it seems like you can get this card for 1-2k depending on if you go used or refurbished. I don't know if better price options are going to be available soon but with the ram shortage and huge general demand it kind of doesn't seem like it.

If you want to focus on image or video generation instead, I understand that there are advantages to going with newer generation cards because certain features and speed is more of a factor than just vram but I know less about this.

I'm running a 4B model on one of my machines, an old surface book 1.

It's a brutal machine. heat issues, and the GPU doesn't work in linux. But pick a minimal enough model and it's good enough for me to have LLM access in my nextcloud if for some reason I wanted it.

Biggest thing really seems to be memory, most cheaper GPUs don't have enough to run a big model, and CPUs are dreadfully slow on larger models if you can put enough RAM in one of them.

As others said it all depends on what you expect. I run stable diffusion on my gaming pc with 32GB RAM and a AMD 9070xt and it works fine. Did also on a 6800xt before that one died. A GPU with 16GB RAM helps a lot, would say that 12GB is the minimum. Lower will limit you in the models and speed.

For LLM just try it out, they work fine without special hardware for smaller models and as long as you are the only user. There are tools like Jan or lmstudio which make it easy to run.

It really comes down to what kind of speed you want. You can run some LLMs on older hardware “just fine” and many models without a dedicated GPU. The problem is that the time taken to generate responses gets to be crazy.

I ran DeepSeek on an old R410 for shits and giggles a while back, and it worked. It just took multiple minutes to actually give me a complete response.

High RAM for MOE models, high VRAM for dense models, and the highest GPU memory bandwidth you can get.

For stable diffusion models (comfyui), you want high VRAM and bandwidth. Diffusion is a GPU heavy and memory intensive operation.

Software/driver support is very important for diffusion models and comfy UI, so your best experience will be Nvidia cards.

I think realistically you need 80gb+ of RAM for things like qwen image quants (40 for model, 20-40 for LORA adapters in ComfyUI to get output).

I run an 128gb AMD AI 395+ Max rig, qwen image takes 5-20 minutes per 720p qwen image result in ComfyUI. Batching offers an improvement, reducing iterations during prototyping makes a huge difference. I have not tested since the fall though, and the newer models are more efficient.

What's your budget?

I’m running gpt-oss20b fine on my m3 MacMini

Depends on how fast you want it to run. A Raspberry Pi with an AI hat runs well enough.

AI said:

To run AI models locally, you'll need a computer with a capable CPU, sufficient RAM, and a powerful GPU

While it's possible to run some AI models on a laptop, a dedicated desktop setup with a powerful GPU will generally offer better performance. The cost of building a dedicated AI PC can range from around $800 for a budget build to $2,500 for a performance-oriented system

Hope that helps /s

I was using a Nvidia 3060 for a while, then had 2 in one box, then switched to a 3090.

The amount of vram is a big factor for decent performance. Getting it to not sound like a predictably repetitive bot though is a whole separate thing that is still kind of elusive.

FYI diffusion models are not really LLMs

Alex Ziskind on YT tests a number of on-site AI devices: https://youtu.be/QbtScohcdwI

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

11 acronyms in this thread; the most compressed thread commented on today has 5 acronyms.

[Thread #91 for this comm, first seen 13th Feb 2026, 17:50] [FAQ] [Full list] [Contact] [Source code]

I'm running a couple of smaller chat models on my mid-range new-ish laptop and they're fairly quick. Try out Jan with something like their jan-nano model on whatever you've already got and get a feel for what you can do.