Celery

Celery was initially created for Django, but is now a Python-friendly app as well.

It was designed to work with asynchronous task/job queues either in real-time or for scheduled operations.

Celery was tested and is being used in production environments to process millions of tasks per day.

The platform has also been tested with Python frameworks like Flask, Django, web2py, Pylons, Tornado and Pyramid.

Supported brokers:
RabbitMQ
Redis
Django ORM
Amazon SQS
MongoDB
Beanstalk
CouchDB
SQLAlchemy

Supported data stores:
SQLAlchemy
Django ORM
Apache Cassandra
AMQP
Redis
memcached
MongoDB

Supported serializers:
Pickle
JSON
YAML
msgpack
zlib or bzip2 compression

What is new in this release:

• Prefork pool improvements
• Django supported out of the box
• Events are now ordered using logical time
• New worker node name format (name@host)
• Bound tasks
• Mingle: Worker synchronization
• Gossip: Worker <-> Worker communication
• Bootsteps: Extending the worker
• New RPC result backend
• Time limits can now be set by the client
• Redis: Broadcast messages and virtual hosts
• pytz replaces python-dateutil dependency
• Support for Setuptools extra requirements
• subtask.__call__() now executes the task directly

What is new in version 3.1.17:

• Prefork pool improvements
• Django supported out of the box
• Events are now ordered using logical time
• New worker node name format (name@host)
• Bound tasks
• Mingle: Worker synchronization
• Gossip: Worker <-> Worker communication
• Bootsteps: Extending the worker
• New RPC result backend
• Time limits can now be set by the client
• Redis: Broadcast messages and virtual hosts
• pytz replaces python-dateutil dependency
• Support for Setuptools extra requirements
• subtask.__call__() now executes the task directly

What is new in version 3.0.0:

• The worker is now thread-less, giving great performance improvements.
• The new "Canvas" makes it easy to define complex workflows.
• New API.
• All of Celery's command line programs are now available from a single celery umbrella command.
• This is the last version to support Python 2.5.

What is new in version 2.5.1:

• Timezone support.
• New security serializer using cryptographic signing.
• Experimental support for automatic module reloading.
• New CELERY_ANNOTATIONS setting.
• current provides the currently executing task.

What is new in version 2.4.1:

• This release contains Python 3 support, broker URLs, less dependencies, and loads of bug fixes.

What is new in version 2.3.0:

• This release contains automatic connection pool support (not enabled by default), chords (aka taskset callbacks), custom time limits for individual tasks, and more.
• Results are now disabled by default, if you depend on the amqp result backend being the default, then you have to configure that manually before upgrading.
• Everything is detailed in the Changelog, which you should have read before upgrading.
• Users of Django must also upgrade to django-celery 2.3.

What is new in version 2.2.4:

• celeryd: 2.2.3 broke error logging, resulting in tracebacks not being logged.
• AMQP result backend: Polling task states did not work properly if there were more than one result message in the queue.
• TaskSet.apply_async() and TaskSet.apply() now supports an optional taskset_id keyword argument (Issue #331).
• The current taskset id (if any) is now available in the task context as request.taskset (Issue #329).
• SQLAlchemy result backend: date_done was no longer part of the results as it had been accidentally removed. It is now available again (Issue #325).
• SQLAlchemy result backend: Added unique constraint on Task.task_id and TaskSet.taskset_id. Tables needs to be recreated for this to take effect.
• Fixed exception raised when iterating on the result of TaskSet.apply().
• Tasks Userguide: Added section on choosing a result backend.

What is new in version 2.2.3:

• Now depends on Kombu 1.0.3
• Task.retry now supports a max_retries argument, used to change the default value.
• multiprocessing.cpu_count may raise NotImplementedError on platforms where this is not supported (Issue #320).
• Coloring of log messages broke if the logged object was not a string.
• Fixed several typos in the init script documentation.
• A regression caused Task.exchange and Task.routing_key to no longer have any effect. This is now fixed.

What is new in version 2.2.0:

• The internal module celery.task.builtins has been removed.
• The module celery.task.schedules is deprecated, and celery.schedules should be used instead.

What is new in version 2.1.4:

• Celery programs now hijacks the root logger by default.

Requirements:

• Python 2.5 or higher
• Jython 2.5 or higher