Working in the industry, at one time or another, you will have to transfer files. Im sure it will be in a variety of different ways. For the most part everyone has their favorites for each situation. But I would prefer to have one utility on all servers to handle all of those situations. So, my choice has to be a bad ass.

Lets look at some requirements:
  • support for all of my common protocols
  • easy and logical navigation
  • parallel threads!
  • full command line usage.

lftp has been the only thing i've come across that met my criteria. Let me prove this my giving you some examples:

Example 1: At some point, everyone has had to mirror a directory that was being served by Apache with directory indexing turned on. Something like http://pkgs.repoforge.org/bsc/.

Lets demonstrate lftps versatility.

debian:/tmp/outgoing# lftp
lftp :~> open http://10.100.15.10/log/dists/lenny-20120514/binary-i386/
cd ok, cwd=/log/dists/lenny-20120514/binary-i386
lftp 10.100.15.10:/log/dists/lenny-20120514/binary-i386> mirror
Total: 1 directory, 69 files, 0 symlinks
New: 69 files, 0 symlinks
180374247 bytes transferred in 136 seconds (1.27M/s)
lftp 10.100.15.10:/log/dists/lenny-20120514/binary-i386>

Downloading a Debian image from a local box. But look at the protocol... http://. I'm able to treat a web page like a cli. But it does lack the depth to do anything crazy. As far as I can see, there isnt a way to be like "mirror http://10.100.15.10/log/dists/lenny-20120514/binary-i386/a*"

Other ways to solve this? Yes. I could have done a fancy curl request stripping html using links/lynx, and then wgetting the result.

Example 2:

You need an entire directory copied from your server, to another server(put)... But you only have SSH.

[root@core ~]# ls -l lame
total 300
-rw-r--r-- 1 root root 100002 2013-01-13 05:32 a
-rw-r--r-- 1 root root 100003 2013-01-13 05:32 b
-rw-r--r-- 1 root root 100004 2013-01-13 05:32 c
[root@core ~]#

Now lets go ahead and login over sftp.

[root@core ~]# lftp sftp://root@g1.ragenetworks.com
Password:
lftp root@g1.ragenetworks.com:~> mirror -R --parallel=3 lame
Total: 1 directory, 3 files, 0 symlinks
New: 3 files, 0 symlinks
300009 bytes transferred in 2 seconds (133.2K/s)
lftp root@g1.ragenetworks.com:~>

What I did was reverse(-R) mirror the directory. In other words I put the directory from my server, to the remove box. But I also did this using parallel threads(--parallel=N).

Example 3:

Along with command line usage is, how scriptable is it. There are many times you need to simply back up a directory with a cron job. This time we are going to use ftp, and script our remote commands in a file.

[root@core ~]# cat script-file
open ftp://username:password@fazey.org
mirror -R /root/local /home/james/remote
exit
[root@core ~]#

Now we call lftp with the "-f" flag to give it a script input.

[root@core ~]# lftp -f script-file
at 80527360 (80%) 35.12M/s eta:1s [Sending data]
...
[root@core ~]#

As you can see, lftp is a hell of a tool.