GNU make utility is great. It can be used in different tasks - not only in source building. I'll show you interesting example. Suppose we have 5 servers: A, B, C, D, and E. There are some *.tar.gz files in the home directory on each server. Every file has the following name syntax <prefix>-<number>.tar.gz. I want to download these files and do it simultaneously in 4 jobs. Also I want to have ability to abort downloading and resume it from the place where I stopped. It's easy with help of the following Makefile:

all : $(shell for i in A B C D E; do ssh $$i "ls *.tar.gz" | awk "{print \"$$i...\"\$$0}"; done | sort -n -t- -k2)


    @echo $(subst ...,:~/,$@)
    @rsync -Prh $(subst ...,:~/,$@) $@.partial
    @mv $@.partial $@

In the first line we form the files list. Every item consists of server name and file name divided by "...". The list is sorted by "-" part for the file name. So the download order will be:

A:~/file-1.tar.gz -> ./A...file-1.tar.gz
B:~/file-1.tar.gz -> ./B...file-1.tar.gz
E:~/file-1.tar.gz -> ./E...file-1.tar.gz
A:~/file-2.tar.gz -> ./A...file-2.tar.gz
B:~/file-2.tar.gz -> ./B...file-2.tar.gz

We can choose almost any separator. Keep in mind - the resulting name is exactly the file name on the local machine.

In the fifth line we tell make utility what to do with every *.tar.gz item not present as the local file. It should do two steps - rsync file with .partial suffix and rename it. So when we abort and resume make will skip all fully downloaded files and start with these, not present at local machine, including .partial (rsync smart enough to continue with them).

Instruction in the third line tells make not to delete intermediate results.


comments powered by Disqus