I have noticed that my home server is strangely using lots of swap (~5 GB), despite having only a few lightweight processes running and loads of RAM installed (32 GB).
Upon configuring Grafana + Prometheus, I noticed a trend where cache + buffer will progressively increase until swap starts to be used. My system and services combined will use ~8 GB RAM. Upon rebooting, the cache + buffer will start anywhere from 3–10 GB, progressively ramp up to ~25 GB in 1–2h, where swap will start to be needed (~3 GB). See the image attached for reference.
My swap filesystem is on an expensive (to me) SSD, and I would like to reduce its wear by as much as possible. I understand that swap can introduce only minimal wear on SSDs depending on its nature and that it can be harmless, but I am still not sure what is causing this behavior (and why) and whether I should worry about it or not. So I figured I should investigate what is happening here.
My main question is, how can I figure out what is causing this behavior? Is it expected? I am looking for guidance from others who are more experienced than me in the topic.
A little bit about my system:
I am running Debian 12 on an NVMe SSD containing the root partition (btrfs) and docker services. I also have two HDDs, one with persistent data (ext4), and the other with backups (ext4). This is majoritarily a single-user machine. I tried using the following kernel parameters, but it hasn’t helped:
vm.swappiness=10
vm.vfs_cache_pressure=200
My docker services are:
- *arr stack
- jellyfin
- nextcloud
- immich
- open-webui + ollama
- pi-hole
- invidious
- romm
- nginx proxy manager
- grafana + prometheus
- other minor services that I don’t think are doing much (uptime-kuma, stirlingpdf, vaultwarden, etc)
Seeing high cache and buffer numbers is usually a good sign, as it means the system is making full use of the memory you have. It’ll automatically deallocate it if something actually needs the memory, but until then it’s using it to store data you might need.
Swap’s a funny one, and whilst you can tune it to an astonishing degree, the decisions the kernel makes aren’t always what you’d want. With this little actual memory usage, compared to the installed RAM, you might be able to run without swap at all, unless you want something like hibernate. If you want to test that, you can run (as root)
swapoff -a, which should disable all swap devices, pushing needed pages back in to RAM, but only until you reboot. If it’s stable, you can consider removing the swap partition(s) later.No point in messing with swappiness settings, trust me. You want to address the reason this happens in the first place, not disable to reduce swap.
I’ve had a nearly identical scenario at work. The reason? A data verification process that went through 100s of thousands of files to run their checksums and check integrity. This was working as intended, but the system alerts were firing because of that high buffer/cache utilization. So check which processes have highest IO and you might find the answer. Swap is just a side effect.
That’s low utilization. If your ram was fully used and stuff was being frequently stopped, that’s when I would be concerned (and when you would experience wear on your SSD).
Right now you have less than a quarter used, and a couple things the kernel decided to swap out. Perfectly normal.
The amount of swap used is not a good indicative, you need to check if there is a big exchange of data per second/minute. This is the only indicative of an out of memory system.
Sometimes, some regions of data memory “age” in ram without any access for a long periods of time, the the kernel here has two options, it could destroy the region knowing it could recreate it when needed (with some cpu overhead) or moved this to a swap file when the ram structure already in the swap file and release than section.
Which regions are good candidates for this? Buffers, specially in the fs, code region used for processes or even data sections of a long sleeping process…
Checking your data, if those 5gb are created over a long period of time I would not care a lot about it. Remeber how big the swap is, isn’t that important vs real traffic (in or out) to it
Yeah that’s completely normal, looks absolutely fine to me. For more info on how linux uses memory I suggest reading https://www.linuxatemyram.com/
To add: that link suggests looking if your swap usage is changing. Linux is quite happy to stuff some memory that isn’t actually used in swap. If it was right about it not being used you should see the swap usage not changing much, so your SSD isn’t really being used either apart from the initial swap-out. If your swap usage keeps fluctuating wildly there is more going on which might warrant further investigation.
I disagree with most here.
You need to reduce swappiness even more, and tweak some other variables (like the memory “threshold” Linux starts to swap, and disk cache aggressiveness) to stop Linux from swapping so opportunistically under such a light load.
IMO, Linux is configured for “old” systems by default: slow HDDs, and constrained RAM pools, where disk IO really, really needs caching, and where idle background processes take a large fraction of RAM.
You have superfluous RAM for your workload. And very fast disk IO that isn’t such a hindrance to apps anyway. And a disk you don’t want to wear. This is the opposite scenario: you don’t want Linux to swap unless it absolutely has to.
For reference, this is part of my config. It’s rather niche and you probably shouldn’t use it, but you should consider looking up the variables:
# Keep min reserve reasonable for 8GB usable space vm.min_free_kbytes = 262144 # Lower absolute minimum to 256MB vm.watermark_scale_factor = 10 # Lower to 0.1% (which is ~128MB on 128GB) # Disable watermark boosting completely vm.watermark_boost_factor = 0 # Normal-ish metadata pressure so desktop doesn't stutter on disk reads vm.vfs_cache_pressure = 120 # Allow reasonable swapping of inactive anonymous desktop pages, could be lower vm.swappiness = 10 # Dirty bytes limits to limit caching vm.dirty_background_bytes = 67108864 vm.dirty_bytes = 268435456 # Disable compaction & proactive scans to stop freezing with large portions of RAM mlocked vm.compaction_proactiveness = 0 vm.compact_unevictable_allowed = 0 vm.page-cluster = 0 # 4kb pages for SSDMy system still uses RAM as disk cache with this config, it just won’t go out of its way to swap just to keep that cache, especially I lower swappiness to 1-3.
I also have a 1GB zram pool, prioritized over ssd swap. But you should make yours even larger (maybe 4GB?). This will intercept anything that does swap first.
I also start some applications with systemd-run and specify memory caps and swap limits (often forbidding them from swapping entirely).
It makes a night-and-day difference for some workloads on my system, that would otherwise swap pointlessly, just tank performance and even de-stabilize the system.
I agree that hitting swap is not ideal, but messing with swappiness sysctl is almost never the answer and very often leads to more problems than it solves.
If you want to tune for workload without needing fairly deep understanding of how Linux manages memory, use a sysctl that sets a whole system behaviour like CPU governor presets.
IMO, Linux is configured for “old” systems by default: slow HDDs, and constrained RAM pools, where disk IO really, really needs caching, and where idle background processes take a large fraction of RAM.
Not really a matter of opinion, and not true since kernel 5.1 when ssds became first class citizens.
Huh…
System load: 1.26 Temperature: 57.0 C Usage of /: 20.4% of 454.86GB Processes: 527 Memory usage: 44% Users logged in: 1 Swap usage: 0% IPv4 address for eno1: 192.168.1.190Ubuntu 22.04.5 - 32 GB RAM - 53 total containers
cat /proc/sys/vm/swappiness: 60 (pretty much default) cat /proc/pressure/cpu: some avg10=0.68 avg60=0.62 avg300=0.67 total=960097774 full avg10=0.00 avg60=0.00 avg300=0.00 total=0 cat /proc/pressure/memory: some avg10=0.00 avg60=0.00 avg300=0.00 total=0 full avg10=0.00 avg60=0.00 avg300=0.00 total=0 cat /proc/pressure/io: some avg10=1.16 avg60=4.88 avg300=3.85 total=2831425045 full avg10=1.11 avg60=4.54 avg300=3.58 total=2653626377



