What?
Linux
From Wikipedia, the free encyclopedia
Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).
Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.
Rules
- Posts must be relevant to operating systems running the Linux kernel. GNU/Linux or otherwise.
- No misinformation
- No NSFW content
- No hate speech, bigotry, etc
Related Communities
Community icon by Alpár-Etele Méder, licensed under CC BY 3.0
The Btrfs and ZFS filesystems support compression.
This is how I do it. I put my Steam directory in an external BTRFS drive with compression turned on. Works great, and allows me to take my library between computers.
You'd get even more savings using something like bees because it does block level deduplication.
What bees does is build a hash table of every block on your ssd, and compares them. If it finds any matches, it will delete one and just place a pointer to the other where the deleted one was, the pointer being much smaller than the duplicate data block.
Functionally, any installed games with shared assets get space savings. It's particularly helpful on with Steam games because of all the proton prefixes. Lots of opportunities for finding duplicate data blocks.
If you use snapshots, it can save even more.
Have you measured the difference? Is it huge? I’m curious to understand what the trade-offs are and how many more games can you store on the same drive.
My main goals are efficient space usage and portability, so I've never measured anything. I'll get some stats for you.
I would appreciate even not very accurate comparison just to get the idea. A brief search meanwhile showed me it’s a great idea actually. But having someone who uses that to comment would be great too. Thanks.
This is another reason on why dodi / fitgirl repacks are so popular, they are masters on this.
If you make any changes to the game files in Steam, including compression, then Steam will update the files and redownload them. And for online games, you could even get a ban if an Anticheat system thinks you try to cheat because the files are tempered (changed).
However there is a transparent compression available on filesystem level. Meaning files are saved compressed and look and work like uncompressed files. Something similar what Windows has I believe. EXT4 does not have transparent compression, but I often read Btrfs and ZFS do. I have no experience with that, so cannot assist. But at least you should be able to search for this now. Have in mind that compressed filesystem would be slower.
Have in mind that compressed filesystem would be slower.
Often the opposite is true, depending on case. Compressed files load faster, so if you have the cpu power to spare (which you usually have in games while loading) and loading speed is the bottle-neck then compression speeds things up, often considerably.
And even in the age of ssds processing data and moving it through ram is much faster than the disk, so even for writing some amount of transparent compression is possible without affecting speeds.
If you have a fast CPU adding compression can actually improve performance. https://www.phoronix.com/review/btrfs-zstd-compress
with fast nvme not that much.. not like the 'olden days' of msdos and stacker where it was actually noticeable on some systems.
Hmm, looks like it can be slower on write, but faster on read depending on the CPU and drive probably. Games mostly read files, so this might even improve performance.
I would suggest Btrfs for regular use. ZFS is better for server use and redundancy in systems with critical data you don't want to lose. It also uses all your ram as cache and will give it back to you if you need it, but still has a harder performance hit on things you run. Be overkill for just wanting to save a little space. A lot of new gaming distros like Bazzite suggest using Btrfs on all secondary drives.
Aren’t most games assets files already compressed?
With compress=zstd ZFS reports a 1.30 compression ratio on my Steam dataset, compressing it from ~1.2TB (reported by the file manager) down to 904GB (reported by zfs list -o name,usedds). Pretty good I think.
Obviously it depends on the games. AAA crap will probably add tens/hundreds of GBs of pre-compressed asset blobs, while indie and older games will often have more loose file structures with config files, scripts, runtimes etc. that all compress extremely well. And with older games, even when compressed the algorithms are often far from ideal and zstd can still get a few more percent out of them.
Btrfs here, with zsdt:3, on my game data the compression ratio is a bit worse than that, but it does save some space to my surprise.
with default btrfs zstd compression here, i save a whole 2 percent on game install directories..but 48 percent elsewhere (trixie kde, ~ 2800 dpkg + flatpaks. 'documents' are on a diff filesystem)
Btrfs disk with periodical dedup.
Does dedup no longer shred the file system?
Never did with my weekly cronjob. But i have ext4 on the system partition, btrfs on the 1.5 TB game partition.
Edit: system partition, boot partition is fat32.
Use ZFS
As noted by others, I'm pretty sure many different solutions exist. But, FWIW, I've had good results with DwarFS.
There are about a million ways to do what you’re asking. Which one is the right choice depends on the particular hardware you’re using and what you want to accomplish.