Spidr

The script can run on a single page, website, multiple domains or even non-stop.

Spidr is designed to be fast and easy to use.

Features:

• Restore the spidering queue and history from a previous session.
• Custom User-Agent strings.
• Custom proxy settings.
• HTTPS support.
• Follows:
• a tags.
• iframe tags.
• frame tags.
• Cookie protected links.
• HTTP 300, 301, 302, 303 and 307 Redirects.
• Meta-Refresh Redirects.
• HTTP Basic Auth protected links.
• Black-list or white-list URLs based upon:
• URL scheme.
• Host name
• Port number
• Full link
• URL extension
• Provides call-backs for:
• Every visited Page.
• Every visited URL.
• Every visited URL that matches a specified pattern.
• Every origin and destination URI of a link.
• Every URL that failed to be visited.
• Provides action methods to:
• Pause spidering.
• Skip processing of pages.
• Skip processing of links.

What is new in this release:

• Catch OpenSSL::SSL::SSLError exceptions when initiated HTTPS Sessions.