Scrapy

Scrappy is written 100% in Python and can be utilized for simple data mining, to page monitoring, Web search engines and even for code testing.

Scrapy is not a search engine in the true meaning of the word, but it acts like one (without the indexing part). Nevertheless Scrapy can be a great tool to build your search engine logic on.

The true power of this framework relies in its core's versatility, Scrapy being a system on which to build generic or dedicated search spiders (crawlers) on.

While this might sound very complicated to non-technical users, with a quick look over the documentation and available tutorials, it's pretty simple to see how Scrapy has managed to take out all the hard-work out of this and reduce the entire process to just a few lines of code (for easier, smaller crawlers).

What is new in this release:

• Unquote request path before passing to FTPClient, it already escape paths.
• Include tests/ to source distribution in MANIFEST.in.

What is new in version 1.0.1:

• Unquote request path before passing to FTPClient, it already escape paths.
• Include tests/ to source distribution in MANIFEST.in.

What is new in version 0.24.6:

• Add UTF8 encoding header to templates
• Telnet console now binds to 127.0.0.1 by default
• Update debian/ubuntu install instructions
• Disable smart strings in lxml XPath evaluations
• Restore filesystem based cache as default for HTTP cache middleware
• Expose current crawler in Scrapy shell
• Improve testsuite comparing CSV and XML exporters
• New offsite/filtered and offsite/domains stats
• Support process_links as generator in CrawlSpider

What is new in version 0.24.5:

• Add UTF8 encoding header to templates
• Telnet console now binds to 127.0.0.1 by default
• Update debian/ubuntu install instructions
• Disable smart strings in lxml XPath evaluations
• Restore filesystem based cache as default for HTTP cache middleware
• Expose current crawler in Scrapy shell
• Improve testsuite comparing CSV and XML exporters
• New offsite/filtered and offsite/domains stats
• Support process_links as generator in CrawlSpider

What is new in version 0.22.0:

• Rename scrapy.spider.BaseSpider to scrapy.spider.Spider
• Promote startup info on settings and middleware to INFO level
• Support partials in get_func_args util
• Allow running indiviual tests via tox
• Update extensions ignored by link extractors
• Selectors register EXSLT namespaces by default
• Unify item loaders similar to selectors renaming
• Make RFPDupeFilter class easily subclassable
• Improve test coverage and forthcoming Python 3 support

What is new in version 0.20.1:

• include_package_data is required to build wheels from published sources.

What is new in version 0.18.4:

• Fixed AlreadyCalledError replacing a request in shell command.
• Fixed start_requests lazyness and early hangs.

What is new in version 0.18.1:

• Removed extra import added by cherry picked changes.
• Fixed crawling tests under twisted pre 11.0.0.
• py26 can not format zero length fields {}.
• Test PotentiaDataLoss errors on unbound responses.
• Treat responses without content-length or Transfer-Encoding as good responses.
• Does no include ResponseFailed if http11 handler is not enabled.

Requirements:

• Python 2.7 or higher
• Twisted 2.5.0 or higher
• libxml2 2.6.28 or higher
• pyOpenSSL