On a daily basis I work on at least three different platforms, hosted locally, virtually, remotely. Shifting files from place to place is a problem I need to resolve every day, and I have more than one solution.
I’m not a fan of file browsers, graphical ftp clients and similar. They clutter the desktop, vary from platform to platform, take ages to drive — especially with a touch pad — and prompt for input at all the wrong times. It’s hard to undo an operation when your pointer slips. By contrast, using simple commands in a shell window puts the power back at your fingertips, whatever platform you’re on. Recovering from mistakes is as easy as recalling your command history.
Local File Systems
For simple operations on a local file system, I tend to use
cp or Emacs dired mode simply because my immediate context is usually Emacs or a shell window, and often both. For bulky and recursive directory operations, a good starting point is:
$ tar c SRC | tar x -C DST
You can vary this command line to reorganise file systems, though sometimes sprinkling a few soft links around may be worth considering.
Remote File Systems
Things can get tricky for remote file systems. Preserving permissions and ownership causes problems, as does the security layer. NFS and Samba may seem like the right solutions for a private network but I’ve grown to regard them as troublesome; they work best on stable networks with well known machines at well-known addresses, and, as usual, I prefer a dynamic model to a static one.
Again, command line tools can do the job. To save the overhead of re-entering your username/password credentials, you’ll want to store SSH keys on the machines you frequent. The most basic remote copy command is
scp. Use it much like
cp, but specify a destination machine on the command line.
For more complex filesystems, we can extend our
tar command using
ssh on the far side of the pipeline. The following command tars up the local SRC directory then extracts the archive on the REMOTE machine in directory DST.
$ tar c SRC | ssh REMOTE tar x -C DST
If this isn’t possible, I sometimes use
netcat to listen at a port on the remote machine:
nc -l -p 2345 | tar x -C DST
Then, on the source machine, kick off the
tar c SRC > /dev/tcp/DOTTED.IP.OF.MIRROR/2345
Suppose you want or need to transfer files using the venerable FTP protocol. If you haven’t already discovered lftp, then it’s time to investigate. When you connect to a remote machine using
lftp it’s rather like having a shell session open on that machine: you can navigate using tab completion and the usual shell tools relating to file and directory operations are there, as well as extra goodies like
mirror and a decent help system.
Rsync is another great command-line file-system shifter. It’s designed to keep two directory structures in sync, and to do so efficiently by just transmitting deltas between the two. Typically the source and destination directories reside on separate machines, and
rsync is often invoked automatically as a scheduled job.
Rsync forms the backbone of many a backup system. I’ve often used it to complement more heavy-weight coporate backup systems which would require me to ask an administrator to restore my own files.
rsync to post updates to this website, and indeed to mirror this website to other machines I use. My publish script is as simple as:
#! /bin/sh rsync -avz www email@example.com:~
Here, local directory structure
www will be mirrored to
~wordaligned/www on remote machine
wordaligned.org. I supply the remote username
wordaligned explicitly since it differs from my local username
-v verbose option gives me a warm fuzzy feeling that the updates I want to post are indeed being posted, the
-z compress option reduces network traffic by compressing file data, and the
-a archive shorthand option recurses and preserves permissions and ownerships.
By the way, I’m implicitly using
rsync default) to access the remote machine. No password is required for user
tag to copy files to user
wordaligned’s home directory since I’ve configured SSH to allow this.
Rsync comes with many more options, but they’re all well documented. A simple
-a is usually all that’s required.
More Thoughts on File Shifting
At the start of this note I unfairly dismissed GUI driven file system tools. The truth is that I do often use them. I’m generally unprincipled and promiscuous when it comes to tool selection: whatever works and is to hand will do. Thus, while both
rsync come with a plethora of options —
lftp does everything any GUI driven FTP client can do, and probably more, and
rsync similarly defeats graphical file browsers — the irony is that I only use them for basic stuff, and may well resort to something with a GUI when attempting something out of the ordinary. A bit of interactive pointing and clicking often appeals more than paging through a rather dry manual.
rsync won’t do is find a directory on a remote file system; you can’t use TAB completion at the far end . An interactive
lftp session does support basic TAB completion on a remote filesystem, but not more powerful tools like
In general you can reduce this problem by adopting a disciplined approach to structuring your workspace on whatever platforms you use. If you find yourself typing a command-line like:
$ scp -r ~/tmp/dev-2008-01-06 cromarty:~/scratch/work-copy2
then I’d suggest something has gone wrong.
One way to combat this disorganisation is to place your home directory under version control. Make sure the version control system you use for this is flexible enough to allow you to rename entries, though. If you do adopt this model, your version control system becomes the home for all your files, and transfers between machines become a matter of check-in, check-out.
I use Subversion in this way, to a degree. There are plenty of files, though, which I don’t version control — in general, large files or files which only make sense on certain platforms. I’ve often found it useful to make these available for access via a webserver, either somewhere on a Wiki, or just served by a lighttpd instance with directory listing enabled.
 I had a suspicion when I wrote this I’d turn out to be wrong! Michael Kedzierski emailled me:
I’m actually using bash completion on Ubuntu and I get remote-side tab completion with scp, it’s great.