To add some practical advice:

It depends on what you mean by more advanced models. I run Qwen3.6-27b on 48GB VRAM across 3 cards (RTX 2000e Ada), and with the recent software optimizations merged into llama.cpp (tensor parallelism & MTP) I get around 30 tokens per second in generation. I use the model through openwebui for (agentic) web research and simple Q&A mostly and I’m quite happy with what it can do.

If you want something similar, maybe look at one or two second hand V100 PCIE 32GB. Or something from the Intel Arc Pro series, if you don’t mind the software support lacking behind a bit (as in less optimized).

Also it might be worth reading into the difference of dense vs MoE models, if you’re new to that. For MoE models, if your system RAM is fast enough, it’s often viable to offload the “experts” (largest parts of such models) to RAM, reducing VRAM capacity needs. Note that server motherboards with e.g. octa-channel RAM have a huge advantage over consumer boards (making DDR4 interesting despite slower speed per module).

And to adress your last question, while I have no direct experience, I’ve seen posts online about people connecting Strix Halo or DGX Spark devices, but usually via a 10+Gbit/s switch as interconnect is cruical (except if you just want to load balance).

Self-hosting LLMs is a very fun thing to do, but also a time- and money-consuming rabbit hole. You might wanna check out the LocalLlama community over at shitjustworks.

Edit: typos

With older hardware, once you accumulate enough vram to run it, your problem is going to shift to memory bandwidth and your questions is going to shift from ‘Can I run this Model’ to ‘Can I run this Model at an acceptable speed’.

The trend I see are the Mac Minis with a lot of unified memory. These are typically very well off people though. Prices for even old GPUs like 3090s are ridiculous now. I don’t think connecting 2 machines over Ethernet would work well, but putting 2 GPUs in a single machine does.

What size model? I can run 8 billion parameter models on my Geforce 3070 with 8gb of vram. Bigger models need more memory. For $1-2k you can upgrade to a 16 or 32 gb video card. For $3k you can get a Framework Desktop with 128 gb unified memory. For $6k you can get a DGX Spark with a blackwell chip and 128 gb unified memory. Mac mini or Mac studio are also good choices in this price range.

For really lightweight models, Qwen3:8b is pretty good for an 8gb graphics card.

Gemma 4:e4b is pretty good too. I usually sun that on my 16gb gpu.

Obviously the little ones aren’t as good as big ones, but you can always rely on real intelligence to fill in the gaps

Honestly, you’re a few months late to the whole buying GPUs for local llms party, so expect exorbitant prices even for older cards

The name of the game is vram. For the most part, more is better. If you can get your hands on multiple matching (same model) 24gb or higher cards (within price range), you’re golden.

Going for more than 2 gpus can become challenging with motherboard pcie slot heights, so make sure either your cards aren’t too tall or you have widely spaced out pcie slots.

For inference, speed (tokens/second) is limited by memory bandwidth. Go for faster bandwidth memory cards if you can afford it (e.g. GDDR6 will be faster than GDDR5).

Also with multi gpus you will need an adequate power supply, and a large enough case.

If you want to be a bit eccentric and load huge models, you can also go the CPU route and fill up a motherboard with 256 GB ram, because then you’re in the several hundred B param model territory, which could, depending on your use case, be better than having faster inference on smaller/quantized models. Even then, DDR5 with high MHz is still way slower than gpus.

I’m running Qwen 3.6 35B A3B (the MoE model) on an 8GB Vram Nvidia GPU with 32 GB of ram, with tweaking (and Turboquant) I’ve got it up to 30-40 Tokens per second and a 260k Context. It’s very usable. I’ve seen people report success with Dual 3060 Cards, but you’re still talking $1000-1500 for that kind of setup even if you have parts of it already.

How old we talking? I personally wouldn’t go further back than 2000 series rtx. A friend has had good luck with Intel GPUs for ‘cheap’.

No, you absolutely cannot scale horizontally for speed. VRAM is king, with local RAM being swappable with major speed penalties. SSD is even slower than that and all those are orders of magnitude faster than ant Ethernet you’ll be connecting boxes together with. That’s not to say clustering isn’t an option, just that speed is going to be worse the more you scale out like that.

I’m running Qwen 3.6 35B A3B on my Ryzen 8700g and it runs pretty well, but the bigger problem there is probably the cost of RAM

thank you folks. Your input gives me a decent starting point. I’ll start digging based on info/experiences shared, maybe I can find someone locally selling old GPU with enough ram for cheap

Unless you’re going to really run a lot, this is an area where vast.ai is probably more affordable than mucking with hardware.

If you’re willing to wait until 2028 when memory prices are expected to drop, and if you’re willing to get new hardware if memory prices drop, I’d give real consideration to waiting until then. There’ll also probably be better hardware and better models then.

If you can constrain yourself to MoE-based LLMs, they’ll generally deal better from a performance standpoint with not entirely fitting in VRAM better than non-MoE LLMs, as experts may not get loaded into VRAM at all.

I use Instinct MI60 GPUs. They are pretty decent performance for local LLM. Connecting multiple computers is going to be impractical because severe bandwidth bottleneck.

Ram if a big driver of what models you can run with vram at a premium. Equipping 2 separate boxes with enough ram to load advanced models may be more expensive than just equipping one faster machine.

On the larger models even with ssd swap I can’t even get them to fully load on my 16gb of ram.