Scrapy

Scrapy is a high level web crawling and scraping framework for writing spiders to crawl and parse web pages for all kinds of purposes, from information retrieval to monitoring or testing web sites.

Features:

• Simple:
• Scrapy was designed with simplicity in mind, by providing the features you need without getting in your way
• Productive:
• Just write the rules to extract the data from web pages and let Scrapy crawl the entire web site for you
• Fast:
• Scrapy is used in production crawlers to completely scrape more than 500 retailer sites daily, all in one server
• Extensible:
• Scrapy was designed with extensibility in mind and so it provides several mechanisms to plug new code without having to touch the framework core
• Portable:
• Scrapy runs on Linux, Windows, Mac and BSD
• Open Source and 100% Python:
• Scrapy is completely written in Python, which makes it very easy to hack
• Well-tested:
• Scrapy has an extensive test suite with very good code coverage

Requirements:

• Python