this post was submitted on 19 May 2024
71 points (97.3% liked)

Linux

48287 readers
627 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
AgreeableLandscape@lemmy.ml
nooter692@lemmy.ml
MarcellusDrum@lemmy.ml
cypherpunks@lemmy.ml
cyclohexane@lemmy.ml
d3Xt3r@lemmy.nz
71
How to speed up accessing lots of files on another computer? Some kind of local cache? (discuss.tchncs.de)
submitted 6 months ago by linuxPIPEpower@discuss.tchncs.de to c/linux@lemmy.ml
63 comments fedilink hide all child comments
 

Title is TLDR. More info about what I'm trying to do below.

My daily driver computer is Laptop with an SSD. No possibility to expand.

So for storage of lots n lots of files, I have an old, low resource Desktop with a bunch of HDDs plugged in (mostly via USB).

I can access Desktop files via SSH/SFTP on the LAN. But it can be quite slow.

And sometimes (not too often; this isn't a main requirement) I take Laptop to use elsewhere. I do not plan to make Desktop available outside the network so I need to have a copy of required files on Laptop.

Therefor, sometimes I like to move the remote files from Desktop to Laptop to work on them. To make a sort of local cache. This could be individual files or directory trees.

But then I have a mess of duplication. Sometimes I forget to put the files back.

Seems like Laptop could be a lot more clever than I am and help with this. Like could it always fetch a remote file which is being edited and save it locally?

Is there any way to have Laptop fetch files, information about file trees, etc, located on Desktop when needed and smartly put them back after editing?

Or even keep some stuff around. Like lists of files, attributes, thumbnails etc. Even browsing the directory tree on Desktop can be slow sometimes.

I am not sure what this would be called.

Ideas and tools I am already comfortable with:

  • rsync is the most obvious foundation to work from but I am not sure exactly what would be the best configuration and how to manage it.

  • luckybackup is my favorite rsync GUI front end; it lets you save profiles, jobs etc which is sweet

  • freeFileSync is another GUI front end I've used but I am preferring lucky/rsync these days

  • I don't think git is a viable solution here because there are already git directories included, there are many non-text files, and some of the directory trees are so large that they would cause git to choke looking at all the files.

  • syncthing might work. I've been having issues with it lately but I may have gotten these ironed out.

Something a little more transparent than the above would be cool but I am not sure if that exists?

Any help appreciated even just idea on what to web search for because I am stumped even on that.

you are viewing a single comment's thread
view the rest of the comments
[–] gravitywell@sh.itjust.works 2 points 6 months ago (2 children)

Can you upgrade the desktop? What speed is your laptops WiFi?

I just use rsync manually until I have syncthig working but these don't really solve slowdown issues and aren't mounted. I would look into a better NIC and/or storage for the desktop or possibly your router.

Try using something like iperf to measure raw speed of the connection between your 2 systems, see if its what it should be (around 300-600mbps for wireless to wired locally) and try to narrow down where the bottleneck is.

[–] linuxPIPEpower@discuss.tchncs.de 3 points 6 months ago (1 children)

A few weeks ago I put some serious time/brainpower into the network and got it waaaay smoother and faster than before. Finally implemented some upgraded hardware that has been sitting on a shelf for too long.

I tried iperf. Actually iperf3 because that's the first tutorial I found. Do you have any opinion on iPerf vs iperf3? On Desktop I ran:

iperf3 -s -p 7673

On Laptop I am currently doing some stuff I didn't want to quit so this may not be a totally fair test. I'll try re running it later. That said I ran:

 iperf3 -c desktop.lan -p 7673 -bidir

And what looks like a summary at the bottom:

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   102 MBytes  86.0 Mbits/sec  152             sender
[  5]   0.00-10.00  sec   102 MBytes  85.6 Mbits/sec                  receiver

I actually have AnotherDesktop on the LAN also connected via ethernet. Going from Laptop ---> AnotherDesktop gets similar to the above.

However going AnotherDesktop ---> Desktop gets 10x better results:

[  5]   0.00-10.00  sec  1.09 GBytes   936 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   933 Mbits/sec                  receiver

Laptop has Intel Dual Band Wireless-AC 8260 who's Max Speed = 867 Mbps. It probably isn't the bottleneck. Although with the distro running at the moment (Fedora) I have a LOT of problems with everything so possibly things aren't set up ideally here.

I still didn't upgrade the actual wireless access point for the network; don't recall what the max speed is for current WAP but could be around 100Mbps.

So this is an interesting path to optimize. However I am still interested in solving the original problem because even when I am directly using Desktop, things are slow. I do not really want to upgrade it is I can get away with a software solution. There are many items on my list of projects and purchases that I'd rather concentrate on.

[–] Davel23@fedia.io 5 points 6 months ago

Wifi is always the bottleneck.