Greetings Lemmings,
I realize that self-hosting for many is just a hobby. Something they do for fun. And the topic of today is usually a “not-so-fun” part of the IT industry.
Jump Scare Warning
Documentation
I have for so long just let my lab grow organically. And months down the road an issue pops up with a service, and I have no idea how I set something up, and may not have documented it. Whether that be comments inside of configuration files, or via other means.
I have gotten marginally better at documenting within the config files or code that I am writing. However, I don’t want just that as an option. So I spun myself up a Bookstack container.
I am slowly going through and creating what would amount to a full blown wiki for my setup. Doubly I can use this as part of my resume.
So I ask of you; what ways do you prefer to document? How do you keep yourself honest, and actually stick to it.
Edit: I created bash scripts that are run by a systemd service and timer. At least for my docker box.
I’m kinda overlapping with a few other comments here, but perhaps adding a little…
As others have said, getting something down is the main point, so I keep everything in Logseq markdown files. The instantaneous ability to create links between topics means that I can write a note about installing [[X]] and then realise I should mention I had to update the [[DHCP]] and [[DNS]] entries… job done.
I try to automate things in Ansible and put rough notes and URLs I was following in the ansible file. I recently setup an old Ras Pi 1 B+ as a NUT server and the instructions on 1 site didn’t work correctly, so I just threw links into the comments and mentioned it in Logseq… next year I’ll visit it all again and maybe improve the notes.
I also have a DEATH.txt file which my partner knows where to look… that gives them a high-level overview of what’s going on, where the keepass and logseq files are, etc. Maybe they’ll understand, but our friends will be able to help… just in case…
I used to have a bunch of scattered markdown & txt files containing disparate notes which quickly became unmanageable. Now I run wiki.js (using the linuxserver image) and document everything in there. It’s not pretty, but it gets the job done 🤷♂️
Git.
I write everything in markdown, and sync to my personal repo (backed up locally and as part of an encrypted backup elsewhere), which also goes to my personal codeberg in a private repo.
Passwords are maintained in a dedicated keepass db, also backed up elsewhere in the same fashion.
Config files themselves are always stored on my NAS in a symlinked or mapped directory, but primarily mapped since its (almost) all containers. These also get backed up the same way.
So as long as I structure my NAS/mapping the same way, everything else works the same way, no matter where its hosted. I do all my addressing by dhcp reservation so IP config is all in one place, and I have a local cert so everything is referenced by hostname.
Makes this all pretty easy to document imo, though the only one who will ever read it is me. I’m also in the middle of changing up auth, so I’ll get to redo a good chunk of it, but I really only need to change configs.
… I’m lazy. I only like doing something once.
I am taking a similar philosophical approach.
I’m lazy too, so I am working on havinf most docker documentation automated…
If I have to do it more than once, automated.
For docker, use compose, and map it all to one spot with subdirectories, and also map the config for the service to a spot on your disk/nas.
I put them in one spot on my NAS under services, so it looks like:
/services/service_name/docker-compose.yml /services/service_name/config/
Easy peasy lemon squeezy.
I have it setup similarly currently /root_compose_dir/stack_name/compose.yml
But I’m taking a step further, ans using labels to generate README files. And will automate putting it into my bookstack
I know that feeling. Broke something in my reverse proxy and setting it back up again was painful. I know that certain things had needed special treatment, but how/what/where? Lost 🥲 Going to migrate to infrastructure as code for my documentation now
I learned Ansible for this purpose, so all the modified configs live in a git repo. I still get lazy and modify some things directly (looking at you Minecraft plugins) but a lot of it is pretty easy to set up via IaC.
I am leaning that way too. Although I do want to maintain a wiki that I can host.
So I settled on using a script that runs as a service on a timer, and can beanually triggered to generate README.md files that get synced with my git repo of docker containers
This will be huge for me, as most of what I run are containers.
The next step will be using Bookstack’s API to push these README.md files into books on my bookstack container.
I’ll go a step further and setup automated exports of these documents.
I have a low tech solution. I use this https://github.com/gamosoft/NoteDiscovery
Every time a thought comes to my mind, I create a new document and add a tag to it. I come back to it from time to time.
just make random files with notes and then grep, the most efficient approach for crazy chaotic homelab imho lol
some day llms will be able to parse your homelab notesI just break things enough times until I internalize the knowledge into my soul.
Kidding aside, I do wish I was more disciplined about documentation.
It could have really helped me earlier in the week in fact. I figured out something really clever a couple of months ago, promptly wiped the computer. And now I can’t recreate it. I’ve come up with a halfway solution, but it’s not as elegant as it used to be.
To be honest with you, this is a really good use case for a large language model - self hosted or other wise. You know Karpathy’s LLM-wiki method?
I dont but I am interestes in learning something new
Here’s the OG
https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f
And the clanker write up https://blog.starmorph.com/blog/karpathy-llm-wiki-knowledge-base-guide
Thank you!
Hope it helps!
Terraform and Ansible. LLMs make Infra as code easy, and the best part is they can output beautiful Markdown and HTML documentation (use a CDN for the CSS).
I used to have a set of notes (in a text file) with commands and links to install everything I was running. But it wasn’t bulletproof. After updating my hardware and having to redo everything from those notes, which took a whole weekend, I decided to look for a better way.
Now everything in my homelab is deployed with Ansible. So I don’t have documentation per se, but its kind of self-documented. If my server was fully wiped tomorrow I could reinstall everything in minutes with one command.
I am taking precautions to have my setup easily deployable no matter the hardware. The only real documentation becomes mountpoints and such for external storage.
For DNS and Reverse Proxy entries, I have those automated. Any time I spin up a new docker container, that information is deployed with labels. Also entries into my homepage.
The .env files contain any variable I would want to store to make my setup portable, shareable and not have to worry about leaking any of my information. Not that it would be end of the world considering nothing really leaves my network unless it is through my VPN.
Traditionally I open a README.md in the dir the application lives in after I configure something I will open my browser history, and dump every link I clicked into the file.
I then copy paste a choice selection of my shell history into the file.
This is the bare minimum. It’s easy to do, takes like 5 minutes, and you will be happy to have it in 8 months. I would make the argument that anyone who tells you to do anything else can suck it. Do the readme first, any “better” documentation second.
Getting Into the habit of doing this matters more than a good documentation system. You will be lazy and try to get around to documenting something later. Don’t let it be later. Shitty link dumps are still gold mines for your future self
I am of that camp, lazy, when it comes to documentation. I have lucked out, as of now after years most if not all of what I need setup has been kept in the ol noggin. But I am building out documentation as we speak. Slowly but surely.
I do like the README.md approach.
