GNU make utility is great. It can be used in different tasks - not only
in source building. I'll show you interesting example. Suppose we have 5
servers: A, B, C, D, and E. There are some *.tar.gz
files in the home
directory on each server. Every file has the following name syntax
<prefix>-<number>.tar.gz
. I want to download these files and do it
simultaneously in 4 jobs. Also I want to have ability to abort
downloading and resume it from the place where I stopped. It's easy with
help of the following Makefile:
all : $(shell for i in A B C D E; do ssh $$i "ls *.tar.gz" | awk "{print \"$$i...\"\$$0}"; done | sort -n -t- -k2)
.SECONDARY:
%.tar.gz:
@echo $(subst ...,:~/,$@)
@rsync -Prh $(subst ...,:~/,$@) $@.partial
@mv $@.partial $@
In the first line we form the files list. Every item consists of server
name and file name divided by "...". The list is sorted by "-
A:~/file-1.tar.gz -> ./A...file-1.tar.gz
B:~/file-1.tar.gz -> ./B...file-1.tar.gz
...
E:~/file-1.tar.gz -> ./E...file-1.tar.gz
A:~/file-2.tar.gz -> ./A...file-2.tar.gz
B:~/file-2.tar.gz -> ./B...file-2.tar.gz
...
We can choose almost any separator. Keep in mind - the resulting name is exactly the file name on the local machine.
In the fifth line we tell make utility what to do with every *.tar.gz
item not present as the local file. It should do two steps - rsync file
with .partial
suffix and rename it. So when we abort and resume make
will skip all fully downloaded files and start with these, not present
at local machine, including .partial
(rsync smart enough to continue
with them).
Instruction in the third line tells make
not to delete intermediate
results.
Comments
comments powered by Disqus