Jug

Jug is a Python implementation of the MapReduce framework (now a standard part of Hadoop), allowing developers to write code that can run in different forms of parallelization.

This means code that can run in parallel on the same machine, or across a computing cluster (multiple machines).

Jug enables developers to organize code into smaller pieces, and selecting where, how and when to run it.

While this may be over the top for small websites and apps, this behavior can be very useful in applications with a high traffic load or that handle large amounts of data.

Jug can work and store data locally via the hard-drive, or it can use remote storage. For the later case, the Redis database is supported.

Features:

• Persistent data storage support
• Reuses partial results
• Concurrency support
• Local data storage
• Remote data storage
• Operation status checker

What is new in this release:

• Python 3 compatibility fixes
• fsync(directory) in file backend
• Jug hooks (still mostly undocumented, but already enabling internal code simplification)

What is new in version 0.9.7:

• Fix use of numpy subclasses
• Fix redis URL parsing
• Fix ``shell`` for newer versions of IPython
• Correctly fall back on non-sqlite ``status``

What is new in version 0.9.6:

• Faster decoding
• Added jug-execute script
• Added describe() function
• Added write_task_out() function

What is new in version 0.9.1:

• Added ``webstatus`` subcommand
• Added bvalue() function
• Fixed bug in ``shell`` subcommand (``value`` was not in global namespace)
• Improved identity()
• Fixed bug in using Tasklets and --aggressive-unload
• Fixed bug with Tasklets and sleep-until/check

What is new in version 0.9:

• In the presence of a barrier(), rerun the jugfile. This makes barrier much easier to use.
• Added set_jugdir to public API.
• Added CompoundTaskGenerator.
• Support subclassing of Task.
• Avoid creating directories in file backend unless it is necessary.
• Add jug.mapreduce.reduce (which mimicks the builtin reduce).

What is new in version 0.8.1:

• Fix redis backend for new version of client module
• Faster file store for large files
• Fix `invalidate` with Tasklets
• Install tests and have them be runnable
• Changed hash computation method. This has a special case on numpy arrays
• (for speed) and is more extensible through a __jug_hash__ hook
• Fix bug with Tasklet dependencies not being properly taken into account
• Fix shell subcommand in newer versions of ipython
• Add __file__ attribute to fake jugmodule

What is new in version 0.8:

• Tasklets
• Fix bugs in sleep-until and cleanup
• Fix bugs with CompoundTask (you needed to run jug execute twice before).

What is new in version 0.8-b0:

• Tasklets - Fix bugs in sleep-until and cleanup.

What is new in version 0.7.4:

• Fix case where ~/.jug/configrc does not exist.
• Print host name to lock file on file_store.
• Refactored implementation of options.
• Fix unloading tasks that have not run.
• Fix mapreduce for empty input.

What is new in version 0.7.3:

• Parse ~/.jug/configrc.
• Fix bug with waiting times.
• Special case saving of numpy arrays.
• Add more expressive jugdir syntax.
• Save dict_store backend to disk.

What is new in version 0.7.2:

• Included missing files in the distribution.

What is new in version 0.7.1:

• sleep-until subcommand.
• Fix for barrier() and mapreduce interaction.
• Documentation improvement.

What is new in version 0.7:

• Better ``shell`` command
• Rewrote method for loading jugfiles
• Tasks have original function accessible as `f`

What is new in version 0.6.9:

• New check command.
• load_all() inside shell.

What is new in version 0.5.2:

• New `shell` command.
• Compression in redis backend.
• Improvements in speed for 'invalidate' command.

What is new in version 0.5.1:

• Redis backend
• Dictionary dependencies
• Changed command-line interface (jug command jugfile ...)

Requirements:

• Python 2.6 or higher