• 1 Post
  • 19 Comments
Joined 3 years ago
Cake day: June 12th, 2023
  • taaz@biglemmowski.wintoLinux@programming.dev•Does zram impede disk cache?
    4·
    9 months ago

    Linux has kind of two forms of memory pages (entries in RAM), one is a file cache (page cache) and the other is “memory allocated by programs for work” (anonymous pages).

    When you look at memory consumed by a process you are looking at RSS, page/file cache is part of kernel and for example in btop corresponds to Cached.

    Page cache can never be moved into swap - that would be the same as duplicating the file from one place on a disk to another place on a (possibly different) disk.
    If more memory is needed, page cache is evicted (written back into the respective file, if changed). Only anonymous pages (not backed by anything permanent) can be moved into swap.

    So what does “PostgreSQL heavily relies on the OSs disk cache” mean? The more free memory there is, the more files can be kept cached in RAM and the faster postgres can then retrieve these files.

    When you add zram, you dedicate part of actual RAM to a compressed swap device which, as I said above, will never contain page cache.
    In theory this still increases the total available memory but in reality that is only true if you configure the kernel to aggressively “swap” anonymous pages into the zram backed swap.

    Notes: I tried to simplify this a bit so it might not be exact, also if you look at a process, the memory consumed by it is called RSS and it contains multiple different things not just memory directly allocated by the code of the program.

  • taaz@biglemmowski.wintoSelfhosted@lemmy.world•Missing votes on self hosted Lemmy instanceEnglish
    5·
    1 year ago

    My instance is close to two years old now, and on average has had about 2 MAU, with no (local) communities.

    Currently we have about 700 active federated communities (that had any federated activity within last month), out of 900.[1]

    The on-disk size of both lemmy and pict-rs database[2]

    postgres@postgres:~$ pwd
/var/lib/postgresql
postgres@postgres:~$ du -sh data/
31G	data/

    I use pict-rs with S3 provider and the bucket size is currently at 22.82 GB (read: external network storage, this is probably mostly just thumbnails[3]).

    So in total there is almost 54GBs spent just for lemmy.

    So assuming you have 100G remaining after system stuff and dedicate that box only to lemmy (and pict-rs media files) and use it mostly for yourself [4], you should be alright for about 3-4 years (assuming that I am gaining about 27GBs total per year and that you will federate with a similar amount of a similarly active communities).

    If you offload media storage to a hosted S3 bucket[5] then you should be good for a lot longer as you will only need space for the postgres databases.

    1. The rest is either dead (instance gone) or no one is subscribed to them anymore (as such my instance is not getting any new content from there: neither posts nor comments or votes) ↩︎

    2. Postgres itself reports about 2G less, don’t really know why but I am guessing it has something to do with the filesystem being btrfs ↩︎

    3. Edit: I currently do not use the “privacy” mode of pict-rs where it proxies all content (so that a bad guy can’t post an image link to his server and unmask users IPs), this would increase the S3 size and slightly postgres size. ↩︎

    4. You should use Lemmy Subscriber Bot to automatically federate little bit of random communities so that public All feed is not exact copy (minus NSFW comms) of whatever you as the only user subscribe to. ↩︎

    5. Though keep in mind that S3 buckets eventually cost some money too, for example Cloudflare R2 charges $0.015 per 1GB, above the first 10GBs. ↩︎