@5ymm3trY@discuss.tchncs.de Excellent questions. I’m fairly new at this but I try to be organized. This may be a bit ‘meta’ for your taste but here it goes.

I use: Portainer to manage containers (FOSS, self-host of course!)

Stacks: Group by service importance or structure/db type I research each service thoroughly with AI for stability, activity and how it integrates into my stack and homelab ‘philosophy’ (see below)

Obsidian: to keep detailed notes of the entire stack (with canvas diagrams to define worfklows and keep stacks organized). I also have a LifeMap (how my homelab supports my life) section which AI vets my new services against.

Claude: as my advisor fore everything, pointed at my Obsidian vault. Makes for a much faster roundtrip, as I edit things in Obsidian (pick your .md editor) and Claude just advises/rewrites back to obsdian on the fly.

Main Setup:

• Philosophy: Low wattage 24/7 prosumer mini-pc runs homelab Ubuntu
• Services: should be light, simple and integrate well with LifeMap
• LifeMap: All areas of my life monitored / integrated (Health, Productivity, Ideation/emergence, Content, Moods/Habits, etc… <put in yours>

Vault Folders:

• Match Stacks for ease of management/guidance from Claude

Stacks: dawarich - mapping w/immich photo integration (on its own)

ghost - blogging platform, obsidian drives content high priorit backups

immich - photos high priority for backups

media - other less critical media tools i can bundle together lower priority backups since most of this stuff can be refound audiobookshelf calibre-web-nextgen jellyfin navidrome - higher priority backup of music yamtrack

tools - all the little workers doing interesting things into one stack that I can reload often without impacting more important services excalidraw immich-kiosk immich-public-proxy immich-reversegeo immich_power_tools linkding maloja miniflux miniflux-db netdata nextflux ntfy uptime-kuma rclone watchyourlan ytzero

wanderer - on its own due to complexity and more active usage by a hiking community, back end …py scripts integrating with immich, dawarich, gpslogger GPX uploads

File Structure: (all on SSDs)

Audio Audiobook backups (just a staging area for dumping stuff, it all goes to B2 via rclone) Book Course Film Image Immich Music Obsidian-RSS Podcast Show Video

Jellyfin and Immich point to the same folders I use Immich folder tools (can elaborate) to move photos / videos to match this structure, so they can both ‘share’ the same items. I find immich a wonerful front-end to crawl thru endless family videos before I process them in a video editor

Having a unified location for all your media files is really great when you start integrating your whole world holistically. I use whisper-ai to transcribe podcasts (audiobookshelf), youtube vides (YTZero), etc. and I always know where everything is.

When I add a new service (say paperless-ngx or stirling-pdf, my next project), I first ask Claude to review my stack documentation, my homelab specs (and security, network, cloudflare external, etc. setup) and then help me decide what service to pick of the basket of FOSS document management tools. I also use sumguy.com articles to review pros and cons of various services, quite helpful.

Then I get Claude to give an implementation doc with .yml, envars and all the tweaks documented in Obsidian. Then I test, and when stable, finalize documentation and make sure Obsidian is updated, and Claude has access to that new ‘folder’ or stack information for the next round. That folder is like a MOC in Obsidian, a folder in Obsidian, a project in Claude and usually a folder on the SSD as well. I use the .canvas mindmaps to help me track it all.

Then, once in a while, I tell claude to build me a workflow document with all the .py scripts, cron jobs, etc. into a document. For example Health Workflow Pull data from fitbit, drop int Fitbaus, push to postgres, analyze with Grafana or similar. Then I have it convert that into a canvas document, which makes a mindmap of the whole thing I can’t tell you how helpful it is to push mindmap boxes around and update the workflow. Of course, n8n is next, I’m just not quite there yet. But n8n will be a breeze with all my documentation.

Good luck

Immich and paperless like their own data structure. But they both also just keep your files pretty much intact.

So in order:

• Since I mostly only use 1 app to manage each “type” of data, I let them have their own folder. Unmanaged files just go on a personal share
• By pretty much just accessing the files trough that application. The only way I need to access them in another way is if I need to migrate to another app
• I don’t use Nextcloud, but when accessing files, doesn’t it just show the normal folder structure? You can give it access to the “base” folder and see everything if needed.
• Two ways: 1. the more important the data, the more mature and hopefully with longer support the app needs to be. Documents and photos are important. Game roms are not. 2. Only choose apps that don’t store the data in a proprietary format. Both paperless and immich like their folder structures and naming, but it’s still pdfs, jpgs and heic underneath. This way of I need to move apps, at most, metadata is lost.
• I just backup all of it — app data, files, etc. No real impact.

• Just like czl I do have seperate apps for each type of data I use. For example Ente for photos and Syncthing for keeping the documents folder of my devices synced up.
• It’s not something I really think about because my data sources are functionally disjoint. (e.g. I don’t have photos from my phone in my documents folder). The only exception is Ente, where I do keep a plaintext copy of the “shared with family” account to display some family photos on our digital photoframe. This also “answers” your next question.
• I don’t have anything set up for when an application stops support. I do however do look beforehand whether it’s possible to extract my data in a way I can migrate to another platform if need be. For example, some services like Ente offer an easy plaintext way to export/backup your data in directories mirroring your albums. That way I could easily head to Immich
• It hasn’t really. I usually do backups by keeping two copies of all my volumes. So running more services doesn’t really impact much of anything.

A sidenote on Nextcloud: it does use the same folder structure in its data volume as it presents to the user. So it’s fairly easy to manage, but you need to update the folder you made the change within the container for it to show up. When we used to keep photo’s synced up with Nextcloud, I could directly access that folder for the photoframe. Since Ente is e2ee I needed to duplicate data in that case

After setting up paperless and Immich I struggled with this as well. I had a structure before an was very attached to it. But both, like mentioned below, will organize files in their own way. There are ways to keep things like you had them, export them, whatever. But I deemed it to be not worth the effort.

It took me some time, but I just accepted whatever these do. In particular since I really like those two services mentioned. Data is accessed through them, and that’s it for now. I backup everything through proxmox mechanisms, which is way more convenient then messing with the data in any way.

Should I want to migrate, I can always cobble together a migration script that pulls my data in ways I like.

Hey, Welcome! You’re having a wonderful and sometimes exhausting journey in front of you :)

I’ll just brain dump based on your questions and my associations. Hope something useful is in between!

First the basic setup options because they tie into how to handle your date flow:

basic Most popular I think is docker compose: here I suggest splitting it into one compose file per service though with one file holding your port config. This prevents you yourself getting confused by your port mappings :)

Second in line is a proxmox setup - similar vein and I lack the hands-on experience to talk about the difference.

Then there’s the “everything native” approach where you don’t rely on containers but manage it yourself or via a dedicated OS that makes life easier (after the learning curve) like nixos.

DATA

All this foundational stuff is important because it changes your approach. In general: don’t fear data duplication. Duplicate it until you learn where you want your data to life and only then define your flow.

Specific example: after I got used to paperless I don’t look into my opencloud anymore, at all. I still duplicate them there but as distributed backup, not for consumption.

If a dataset has a clear place ten it’s easy. If not then your options are different depending on your setup: For the *arr stack the official recommendation is to use one shared folder and mount that into each part for example. I personally don’t like that and have hard links for everything - that’s basically a pointer to the file that looks like the file itself everywhere. As long as one pointer exists the file still stays on your drive but when the last pointer is gone, the file is effectively deleted. On Linux, you can think of every file this way but by default only one pointer exists (which often people test as synonymous to “the file”. Drawdown: this only works really well if you manually keep either track of which tool links where or you don’t containerize everything.

Again a specific example: My downloaded torrents never get moved - instead hard links are created into whichever path and naming scheme I defined for each consumer - this way, out of murdrrbot_07.mp3 a new author/series/booktitle.mp3 was created, both pointing to the same data and seeing it as a proper file.

But then there is one more thing: I suggest you split your thinking into data consumption and manipulation - because for the first, data duplication doesn’t matter. Especially for documents you’re talking about a ridiculous small amount of disk space and if it’s only reading/watching/hearing you as manager have no problem that data might exist multiple times.

If you want to keep it clean by design then you’re leaving the starter mode self holster - welcome to system design and infrastructure architecture! Here your approach could be to define lifecycles for each data type that you have. What a “data type” is in this context btw is a user term, NOT the underlying tech stack. You need to understand and document how an invoice should be treated and consumed by you differently than an invitation or a informal letter. Only then do you map file types, incoming channels, transformation steps, etc etc.

In my opinion: huge overkill to this upfront.

In short: spin everything up, observe how you use it and only then decide where things need to stay unique and cleaned up. Don’t break your head over something that’s actually quite easy to repair!

I run everything in docker. My docker volumes are all subdirectories under /media, which is the root of my NAS. If an app only needs something specific, docker only mounts, eg /media/movies. If the app needs many subdirectories (like backups), it gets the full /media mount.

All the docker data/configs are under /docker/volumes (eg /docker/volumes/jellyfin), and every container is configured with compose in /docker/docker-compose.yml.

The biggest hurdle for me was creating a sane docker-compose file that defines where everything lives. Once that is standardized, adding and maintaining your apps becomes easy.

If you’re very new to this, even a dumb LLM like ChatGPT can be very helpful to explain what’s inconsistent, redundant, or incorrect with your config. If you are lost I would strongly suggest this as a way to get oriented. But DO NOT copy and paste anything into the LLM containing passwords, tokens, keys, etc. if you accident do, change those keys before finishing up your project.

With docker there’s no duplication- you have one copy of everything you need and you just point each container at the same data. This makes it trivial to keep file consistent across your apps and doesn’t waste space.