wgrab

It is an advanced tool based on the classic wget UNIX command.

Instead of the unspecified way, in which 'wget -r' downloads and stores everything, wgrab allows the developer to use iteration over dates and numbers, as well as regular expressions.

What is new in this release:

• For the -K feature (storage of 'known' URLs we don't want to get again), only the new ones are appended to the file now (instead of rewriting it completely). this makes it less error prone to run concurrent wgrab jobs with the same 'known' file.
• New feature --printFirstSave prints the (pattern generated) name of the first file saved to stdout. When wgrab is used from another script in conjunction with -A (saveAll) to get, rewrite and store an entire subtree, this printed filename can be used to generate an entry point into the saved documents.

Limitations:

• Don't use on large files.