Pumpkin Escobar

Taking ollama for instance, either the whole model runs in vram and compute is done on the gpu, or it runs in system ram and compute is done on the cpu. Running models on CPU is horribly slow. You won’t want to do it for large models

LM studio and others allow you to run part of the model on GPU and part on CPU, splitting memory requirements but still pretty slow.

Even the smaller 7B parameter models run pretty slow in CPU and the huge models are orders of magnitude slower

So technically more system ram will let you run some larger models but you will quickly figure out you just don’t want to do it.

Hopefully you’re only forwarding the minimal set of network ports and not all ports/traffic? If so then you’re good, like someone else said if you’ve got a router and it’s forwarding selected traffic then no need for anything else

Just a note, the orange pi drivers are not in great shape. It’s getting better but I have a cluster of raspberry pi’s for development, bought an orange pi without first checking out much about them and it’s rough. Rockchip CPUs are great, and the driver / firmware situation is getting better, but something I’d read up on before buying one.

I’d still look at the N100, it’s about 2.5x the performance of raspberry pi 5, and being x86 you have more options than arm.

There are a lot of tiny PCs these days that can output 4k video and audio. Look for something with an N100 or N200 CPU if you want to go as cheap as possible, they tend to be super-cheap and perform well. I’ve got one of the GMTecs and this wireless keyboard+mouse, works really well from the couch.

There are cheaper/other options but to get you started: https://www.amazon.com/GMKtec-Windows-Computer-Business-G3-dp-B0CQ4XQ2WG/dp/B0CQ4XQ2WG https://morefine.com/collections/pc-box (specifically the M9)

TPM & secure boot. Look into sbctl for secure boot if you’re not on something that uses the signed shim like ubuntu. I know some hate secure boot but storing the unlock key in tpm is at least much more secure than having the key sitting on a usb drive

Tang - network based unlock. If you have a separate raspberry pi or something you can set it up as a tang server. You’ll want that thing encrypted too, can set that up to require manual unlock so if someone boosts your servers the tang server never comes up, storage server won’t either

Or just manually unlock the server with a password every boot?

That’s roughly my prioritized/preferred list

It’s the same, I picked up an Orange Pi 5 plus on sale and didn’t even think about the kernel and module driver situation. It’s rough. Joshua-Riek/ubuntu-rockchip and the other contributors do great work to un-fuck the situation and get a non-screwy ubuntu install cobbled together, but in the comments for issues even he gives off a “well, the situation is shit” sort of vibe.

I won’t buy another rockchip sbc.

Are there any alternatives for people with gluetun allergies?