I don’t want to create an extra post, so I ask here.

Where do you guys get your hardware from, for an affordable price?

I’d like to run at least a 27b model but my hardware is too weak. I’m on a 16GB unified mem, M4. Tried running Bonsai 27b (it’s Qwen architecture) but I had to close all other apps and it is not in a state that I could work with it.

I self host both at home and at work.

At work (for completion and generation) we have a now quite old A6000 running mainly Qwen 3.6 (though I’m trialling 3.8 from Monday) with OpenClaw as the harness. The use case is mainly triage and pre-work on bugs. We have an agent which can be given an exception report with a bunch of information like stack trace and local variable values and it will go off and gather the required code, order files, and data to analyse the issue and come up with a fair idea of the cause and possible solution. This saves devs and support team time. Also looking at adding a PA agent to this which will be able to read emails and calendars and be able to set up (with approval) meetings and move things around and generally make someone’s life a bit less admin-y without having to hire a full PA role.

Also technically in the AI space we have a fleet of Pis with Hailo AI hats running object and anomaly detection in visible and LWIR spectrums for QC

Ollama container, 2070 super, 3-7b, meta and qwen

I use it to ingest tons of news rss, find dupes, summarize, and then post that output to mesh a mesh network.

I also use it for a self hosted mealie to figure out recipe->ingredient.

I also use it for receipts. I translate pdf -> txt, feed the text to qwen, ask it who is the vendor, what is the price, what it a reasonable memo for the receipt, and give a dozen categories, which is best.

Tried running a decent Qwen on a 3060 TI but I only have 16g ram so it kept getting stuck and not completing tasks.

Not OP but running local LLMs for exactly the privacy reasons you describe, some practical notes:

• For a no-GPU start: llama.cpp (or Ollama as a wrapper) with Qwen3-4B/8B or Llama-3.1-8B at Q4_K_M runs fine on a plain x86 box with 16GB RAM. Speed on CPU is ~5-15 tok/s for 4B-class models, which is usable for search summarization and drafting, painful for long chats.
• Quantization is the whole game: 8B Q4 is ~5GB and close to fp16 quality for everyday tasks; don’t run fp16 on consumer hardware, run the bigger model more quantized instead.
• For your “web search without profiling” use case the actual stack that matters is: SearXNG (self-hosted metasearch, no tracking) -> fetch top N pages -> feed text to the local model for summarization. The LLM part is almost the trivial bit; the search layer is what fixes the privacy problem.
• Remote access: don’t expose Ollama/Open WebUI directly. WireGuard (or Tailscale if you don’t want to manage keys) and you get it on the phone with zero ports open.
• One honest caveat: local small models are noticeably worse at factual recall than cloud ones. Where they shine is processing text you hand them (summaries, extraction, rewriting). Design your workflows around that and the gap mostly disappears.

test probe

test probe

Not OP but running local LLMs for exactly the privacy reasons you describe, some practical notes:

• For a no-GPU start: llama.cpp (or Ollama as a wrapper) with Qwen3-4B/8B or Llama-3.1-8B at Q4_K_M runs fine on a plain x86 box with 16GB RAM. Speed on CPU is ~5-15 tok/s for 4B-class models, which is usable for search summarization and drafting, painful for long chats.
• Quantization is the whole game: 8B Q4 is ~5GB and close to fp16 quality for everyday tasks; don’t run fp16 on consumer hardware, run the bigger model more quantized instead.
• For your “web search without profiling” use case the actual stack that matters is: SearXNG (self-hosted metasearch, no tracking) -> fetch top N pages -> feed text to the local model for summarization. The LLM part is almost the trivial bit; the search layer is what fixes the privacy problem.
• Remote access: don’t expose Ollama/Open WebUI directly. WireGuard (or Tailscale if you don’t want to manage keys) and you get it on the phone with zero ports open.
• One honest caveat: local small models are noticeably worse at factual recall than cloud ones. Where they shine is processing text you hand them (summaries, extraction, rewriting). Design your workflows around that and the gap mostly disappears.

I have an LLM running object detection on a b580 that’s also handling camera rotation in frigate. It’s hanging out 95-120 watts

Yes! I’ve got quite a few projects going right now. Probably too many of them to keep track

I am the only user, so it’s relatively easy to run multiple models for what ever task. Ollama is pretty good at balancing which models are working at what time to not bog down they system

I’m running gpt-oss 120 for a general purpose Hermes agent. Out of the box, it was ok. Not very snappy, but it was one of the best ones for orchestrating other agents.

Glm flash (4.something?) does pretty good as well.

I’ve found a hybrid of cloud and local is the best solution. Use local for smaller or more defined tasks. Use cloud for orchestrating, planning, and…fixing the local agents mistakes lol

I’ve got an LLM maintained Obsidian wiki that is written and edited by a few Ollama agents based on chats via my own self hosted matrix instance. I’ve taken it to a point where I’m pretty happy with the results. But I think if I had enough time and energy, I could really dial it in.

One of my favorite use cases for this is when I’m researching topics like UFOs/high strangeness/conspiracy theories. I have my very own LLM maintained Obsidian vault filled with curated notes and links to sources.

Literally, no one cares about this and they think it’s stupid and I’m stupid and I should feel bad for spending time on this or having the audacity of sharing it… they just don’t appreciate the amount of work and knowledge it took to get what I have set up.

I think it’s cool as fuck. I geek out every time that it works.

Oh, forgot to mention I am running it on a Mac studio with 96 gig of unified memory. I think it’s an M3 ultra chip.

Not only do I run and share it. I’ve built an entire service to allow everyone else to do so safely. https://aihorde.net/

Yes, I have a desktop with a radeon 9700 ai pro with 32GB of VRAM. Its more than doubled since I bought it but its basically a down clocked 9070xt with double the VRAM. I also have a little arc b50 in my server.

Mostly yeah use it for privacy reasons. I run llama.CPP directly and access via open webui. I host a searx instance for it to use as web search. Mostly run qwen3.6/3.8 or gemma4 on the radeon and almost exclusively gptoss 20b on the B50 along with a second small model for tags and titles in open webui. Yeah I just use it mostly as a basic chatbot or as an agent in obsidian using the engram plugin. Helps search and summarize years of notes and works pretty well.

I’ve tried to use it as a coding assistant in zed without much luck so usually just post my code into open webui for feedback and it works alot better for me.

Yeah I can access it with other machines on my network including phone with conduit app. I can get to it on the go by connecting to home network with wireguard and sending a magic packet to wakeup desktop.

I havent done any image gen nor have much use for it outside of maybe using it to make app icons in the style of papirus icons for programs they don’t and won’t support.

It definitely feels more comprable to first ai wave than current frontier models but the privacy and price trade off is worth it to me. I’d say using it in obsidian has been the best use of it so far and one I certainly wouldn’t trust to an external AI.

Hardware:

2 x ASUS GX10 128GB(each)

2 x Mac Mini M4 Pro 24GB(each)

The GX10s are clustered and run DeepSeek-V4-Flash-0731-Abliterated. The M4 Pros run various utility LLMs such as Qwen 3.8

Software:

OpenWebUI

Hermes Agent

LiteLLM

Llama.cpp

vLLM

StoryBinder - custom web app that I use for story/script writing when in the mood

LASM - Local AI Stack Manger - custom web interface to manage pretty much all aspects of my local stack. Agents run on the 4 nodes as well as my NAS(Download GGUF/HF snapshots once and copy out to the nodes)

Have yet to connect my Home Assistant into the stack.

For mobile access I use Conduit https://github.com/cogwheel0/conduit

Yes, I installed an LLM at home. I use it for many useful things:

1. reading novels by translating them from English to my language on the fly via machine
2. for communicating on lemmy by translating posts and comments
3. For HomeAssistant
4. And very rarely for development and searching the internet

I have a RX9070XT and llama-swap to switch models on the fly. I’m thinking about installing a second RX9070XT so that I don’t take away the translation capability from my lemmy instance while I stress the other neural network with other tasks.