Gizzard

Twitter uses it for retrieving information from a large data set, information so large that it can't be stored in one computer alone.

This means that the needed data is scattered across many network servers, and hard to query without running multiple queries on multiple machines.

Gizzard comes into play when needing to create a common system for retrieving the data from multiple servers and then aggregating it on the destination machine.

Gizazard is actually a middleware and can be used with databases or with search engines the same, taking data and partioning it across various backends (SQL databases, NoSQL databases, Lucene, etc.).

Features:

• Gizzard supports any datastorage backend
• Gizzard handles partitioning through a forwarding table
• Gizzard is middleware
• Gizzard handles replication through a replication tree
• Gizzard is fault-tolerant
• Gizzard supports winged migrations
• Gizzard handles write conflicts