Apache Flume

Apache Flume was developed as an intermediary system that collects logs from one application and delivers it to another.

It should work with most Java applications and developers should rest assured, Flume being protected against data loss and service failure.

Log data is streamed in real-time via fault tolerant systems with support for many failover and recovery mechanisms.

Because Flume runs from one central location, developers can easily manage it and also tap into its resources for statistics and analytics purposes.

A hidden feature of Flume is its actual "data-agnostic" system. This means you can easily switch a data stream's source from a log output to anything else and use Flume to safely aggregate information from various types of services.

These can be email messages, social media streams, network traffic, and so on.

What is new in this release:

• Removing SSLv2Hello causes Java 6 clients to break
• Enable SSLv2Hello for Avro Source and NettyAvroRpcClient
• Enable SSLv2Hello for HttpSource
• Added dev-support to src tarball
• Unit test TestHTTPSource.java failed with IBM JDK 1.7
• HTTP Source should be able to block a prefixed set of protocols.
• HTTPS tests fail on Java 6
• Allow configuration of enabled protocols in Avro source and RPC client

What is new in version 1.5.2:

• Removing SSLv2Hello causes Java 6 clients to break
• Enable SSLv2Hello for Avro Source and NettyAvroRpcClient
• Enable SSLv2Hello for HttpSource
• Added dev-support to src tarball
• Unit test TestHTTPSource.java failed with IBM JDK 1.7
• HTTP Source should be able to block a prefixed set of protocols.
• HTTPS tests fail on Java 6
• Allow configuration of enabled protocols in Avro source and RPC client

What is new in version 1.5.0:

• New Feature:
• Introduce some sort of SpillableChannel
• Allow SpoolDir to pass just the filename that is the source of an event
• Flume Context doesn't support float or double configuration values.
• Upgrade morphlines to 0.7.0
• flume-ng-log4jappender Support user supplied headers
• Elasticsearch Sink for ES HTTP API
• Added a sink for Kite Datasets
• Spooling directory should not always consume the oldest file first.
• Improvement:
• Flume User Guide sample configurations need fixing
• Make SpoolDir work with Sub-Directories
• Spooling directory source should be able to replace or ignore malformed characters
• Handle larger payloads via SyslogUDPSource
• Upgrade morphline library dependency to a 0.5.0 release
• Reducing duplicate events caused by reset-connection-interval

What is new in version 1.3.0:

• Added HTTP Post Source
• ElasticSearch Sink
• Flume adopt message from existing local Scribe
• Added a multiport syslog source
• File Channel should support encryption
• Create a SpoolDirectory Source and Client
• Load Balancing RPC client should support exponential backoff of failed nodes
• Dump RollingFileSink's counter status when agent stops
• Regex Extractor Interceptor

What is new in version 1.2.0:

• Implemented a Durable Memory Channel
• Created developer guide for Flume NG
• Client SDK
• Implemented a durable FileChannel
• Implemented Interceptors (previously known as Decorators) for Flume 1.x
• Implemented an HBase Sink which supports table level access
• Implemented Timestamp Interceptor
• Asynchronous HBase Sink

Requirements:

• Java 1.6 or higher