Does anyone run their own AI service?

If so,

Which one?

What hardware?

Do you have a way to access it from outside or from a phone?

And, if you’re open to sharing, what do you do with it?

I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.

I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.

  • kensand@sopuli.xyzEnglish
    2·
    7 hours ago

    MCP in general:

    1. MCP is very verbose (json tool declarations)
    2. Every tool incurs a prefix cost, which can be in the thousands of tokens across all tools
    3. Tool outputs are usually dumped straight into the prompt. Sometimes they are truncated and dumped into a file with a warning to the LLM, but usually after they reach thousands of tokens
    4. Changing any tools breaks your prefix cache.
    5. Tool discovery can help with 1 a bit, 2 a lot, and 4 a lot, but isn’t a cure-all. It does have its own (much smaller) token overhead too.

    CLI in comparison:

    1. Fairly concise, by nature - short names for frequently used commands, spaces as implicit delimiters, implied outputs.
    2. Tools are generally implied to be present. A short “this environment contains x,y,z,” system prompt can be very effective. Very minimal prefix cost.
    3. Tool outputs can be written to a file directly, formatted, head/tailed, grepped, or used in any other way.
    4. Doesn’t break the prefix cache, but can “confuse” the LLM.
    5. Tool discovery is implicit in the system. This does have its own overhead, but for many things is implied - ls, find, grep, sed, cat - usually an llm will just assume they’re present, and use them to trim down tool outputs to only the required context. In f, I also added search functionality to avoid parsing ‘help’ messages.

    CLI does of course have its own downsides, I think the biggest being security. Give an LLM an MCP server, and all it can do is what that MCP server supports. Give an LLM a Turing complete shell, and it can do anything.

    MCP also tends to be harder to set up and harder still to fix when it doesn’t work. I’ve watched Claude fall back to curl commands when a connector doesn’t work many times. I’ve never seen it try curl commands, fail, and then try to set up MCP.

    I think this is all fairly domain dependent, but I think my domain (software development and general agenetic assistance) is very common. If you’re using an LLM in a product that you’re selling with a fixed range of features, MCP is probably the way to go. If you’re using an LLM for dynamic functionality with expanding capabilities, I think CLI might fare better.

    My main reasoning for all of this is flexibility. In the history of computing, a program has always had a fixed output for a fixed input. LLMs are at the point where they can handle what is effectively an arbitrary input and give an arbitrary output. Giving an LLM a fixed set of tools negates that newfound ability to process the unknown. It’s like giving someone a sledgehammer to bust rocks, but tiring one hand behind their back; the work might get done, but not as efficiently.