DataFu

DataFu was developed at LinkedIn and is written entirely in Java.

DataFu includes functions/libraries for working with:

- Statistics

- Bags

DataFu is perfect for data mining and statistical applications working on top of Hadoop or Pig databases.

These functions allow developers to take full advantage of all the data stored inside a Hadoop or Pig database without having to deal with massive system requirements in order to do so.

What is new in this release:

• Pair of UDFs for simple random sampling with replacement.
• More dependencies now packaged in DataFu so fewer JAR dependencies required.
• SetDifference UDF for computing set difference (e.g. A-B or A-B-C).

What is new in version 1.2.0:

• Pair of UDFs for simple random sampling with replacement.
• More dependencies now packaged in DataFu so fewer JAR dependencies required.
• SetDifference UDF for computing set difference (e.g. A-B or A-B-C).

What is new in version 1.1.0:

• Added SHA hash UDF.
• InUDF and AssertUDF added for Pig 0.12 compatibility. These are the same as In and Assert.
• SimpleRandomSample, which implements a scalable simple random sampling algorithm.