Working in the industry, at one time or another, you will have to transfer files. Im sure it will be in a variety of different ways. For the most part everyone has their favorites for each situation. But I would prefer to have one utility on all servers to handle all of those situations. So, my choice has to be a bad ass.
Lets look at some requirements:
- support for all of my common protocols
- easy and logical navigation
- parallel threads!
- full command line usage.
lftp has become my tool of choice. Let me prove it with some examples:
At some point, everyone has had to mirror a directory that was being served by Apache with directory indexing turned on. Something like http://pkgs.repoforge.org/bsc/.
debian:/tmp/outgoing# lftp lftp :~> open http://10.100.15.10/log/dists/lenny-20120514/binary-i386/ cd ok, cwd=/log/dists/lenny-20120514/binary-i386 lftp 10.100.15.10:/log/dists/lenny-20120514/binary-i386> mirror Total: 1 directory, 69 files, 0 symlinks New: 69 files, 0 symlinks 180374247 bytes transferred in 136 seconds (1.27M/s) lftp 10.100.15.10:/log/dists/lenny-20120514/binary-i386>
Downloading a Debian image from a local box. But look at the protocol… http://. I’m able to treat a web page like a cli. But it does lack the depth to do anything crazy. As far as I can see, there isnt a way to be like “mirror http://10.100.15.10/log/dists/lenny-20120514/binary-i386/a*”
Other ways to solve this? Yes. I could have done a fancy curl request stripping html using links/lynx, and then wgetting the result.
You need an entire directory copied from your server, to another server(put)… But you only have SSH.
[root@core ~]# ls -l lame total 300 -rw-r--r-- 1 root root 100002 2013-01-13 05:32 a -rw-r--r-- 1 root root 100003 2013-01-13 05:32 b -rw-r--r-- 1 root root 100004 2013-01-13 05:32 c [root@core ~]#
Now lets go ahead and login over sftp.
[root@core ~]# lftp sftp://email@example.com Password: lftp firstname.lastname@example.org:~> mirror -R --parallel=3 lame Total: 1 directory, 3 files, 0 symlinks New: 3 files, 0 symlinks 300009 bytes transferred in 2 seconds (133.2K/s) lftp email@example.com:~>
What I did was reverse(-R) mirror the directory. In other words I put the directory from my server, to the remove box. But I also did this using parallel threads(–parallel=N).
Along with command line usage is, how scriptable is it. There are many times you need to simply back up a directory with a cron job. This time we are going to use ftp, and script our remote commands in a file.
[root@core ~]# cat script-file open ftp://username:firstname.lastname@example.org mirror -R /root/local /home/james/remote exit [root@core ~]#
Now we call lftp with the “-f” flag to give it a script input.
[root@core ~]# lftp -f script-file at 80527360 (80%) 35.12M/s eta:1s [Sending data] ... [root@core ~]#
As you can see, lftp is a hell of a tool.