S3QL

S3QL supports Amazon S3 as well as arbitrary SFTP servers and effectively provides you with a hard disk of dynamic, infinite capacity that can be accessed from any computer with internet access.

S3QL is providing a standard, full featured UNIX file system that is conceptually indistinguishable from any local file system. Furthermore, S3QL has additional features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival.

S3QL is designed to favor simplicity and elegance over performance and feature-creep. Care has been taken to make the source code as readable and serviceable as possible. Solid error detection and error handling have been included from the very first line, and S3QL comes with extensive automated test cases for all its components.

Before a file system can be mounted, the backend which will hold the data has to be initialized. This is done with the mkfs.s3ql command. Here we are using the Amazon S3 backend, and nikratio-s3ql-bucket is the S3 bucket in which the file system will be stored.

mkfs.s3ql s3://nikratio-s3ql-bucket

To mount the S3QL file system stored in the S3 bucket nikratio_s3ql_bucket in the directory /mnt/s3ql, enter:

mount.s3ql s3://nikratio-s3ql-bucket /mnt/s3ql

Now you can instruct your favorite backup program to run a backup into the directory /mnt/s3ql and the data will be stored an Amazon S3. When you are done, the file system has to be unmounted with

umount.s3ql /mnt/s3ql

Features:

• Transparency. Conceptually, S3QL is indistinguishable from a local file system. For example, it supports hardlinks, symlinks, full unix permissions, extended attributes and file sizes up to 2 TB.
• Dynamic Size. The size of an S3QL file system grows and shrinks dynamically as required.
• Compression. Before storage, all data may compressed with the LZMA, BZIP2 or LZ algorithm.
• Encryption. After compression (but before upload), all data can AES encrypted with a 256 bit key. An additional SHA256 HMAC checksum is used to protect the data against manipulation.
• Data De-duplication. If several files have identical contents, the redundant data will be stored only once. This works across all files stored in the file system, and also if only some parts of the files are identical while other parts differ.
• Immutable Trees. Directory trees can be made immutable, so that their contents can no longer be changed in any way whatsoever. This can be used to ensure that backups can not be modified after they have been made.
• Copy-on-Write/Snapshotting. S3QL can replicate entire directory trees without using any additional storage space. Only if one of the copies is modified, the part of the data that has been modified will take up additional storage space. This can be used to create intelligent snapshots that preserve the state of a directory at different points in time using a minimum amount of space.
• High Performance independent of network latency. All operations that do not write or read file contents (like creating directories or moving, renaming, and changing permissions of files and directories) are very fast because they are carried out without any network transactions.
• S3QL achieves this by saving the entire file and directory structure in a database. This database is locally cached and the remote copy updated asynchronously.
• Support for low bandwidth connections. S3QL splits file contents into smaller blocks and caches blocks locally. This minimizes both the number of network transactions required for reading and writing data, and the amount of data that has to be transferred when only parts of a file are read or written.

What is new in this release:

• Attempts to upgrade a filesystem that is already at the newest revision no longer render the local metadata copy unusable.
• Google storage bucket names containing dots no longer cause problems with SSL certificate validation.
• Fixes a crash when statvfs() is called on an almost empty file system.
• There is a new "swiftks" backend for OpenStack/Swift servers.
• Eliminates a race condition that could lead to mount.s3ql crashing with a "KeyError in collections/abc.py".
• Fixes a crash that could occur when trying to clean up after receiving a connection or server error when uploading an object.

What is new in version 2.2:

• S3QL now uses SSL connections by default and verifies server certificates.
• A race condition that resulted in an apparent loss of remote metadata was fixed.
• The S3 backend now deletes multiple storage objects using one request if possible, reducing latency, traffic, and number of requests to S3.
• S3QL is now able to detect server errors and redirections before all data has been sent to the server.
• A new 'rackspace' backend that allows specification of the object storage region in the storage URL was added.
• The mount.s3ql --compress option now allows specification of the compression level.

What is new in version 1.15:

• Cycling of metadata backups when using the local backend now takes much less time.
• It is no longer required that S3QL and Python APSW be linked to the same SQLite shared library.
• In mount.s3ql, a crash when using --metadata-upload-interval 0 was fixed.
• Instead of completely unmounting the file system (and leaving the mount point inaccessible) when backend data is missing or corrupted, S3QL now returns an error and switches to read-only operation.

What is new in version 1.13:

• This version fixes a bug that caused malformed metadata to be written out when using recent eglibc versions.
• It fixes issues when trying to access a directory at the root of the filesystem that has recently been removed with s3qlrm.
• Ostensible ACL support has been removed.
• This release better handles buckets with invalid DNS names.
• It will retry if DNS is not available.

What is new in version 1.11.1:

• This version fixes a crash when using the S3 backend (regression introduced in release 1.11), increases the minimum reported filesystem size to 1 TB, and works around a df bug so size is reported properly.

What is new in version 1.11:

• Metadata about stored objects is now split into multiple fields, so no field is longer than 256 bytes. (This makes S3QL compatible with a wider range of storage providers.) The SWIFT backend now retries if it receives 5xx error codes from the remote server, and no longer fails when attempting to upload data with an expired auth token.

What is new in version 1.10:

• The s3s, s3cs, and gss backends have been removed. Use the new --ssl option together with the s3, s3c, and gs backends instead.
• S3QL no longer keeps track of consistency guarantees for the different backends.
• The increasing number of different storage providers offering different regions, redundancy, and availability levels makes this no longer feasible.
• The User's Guide contains a new section, "Important Rules to Avoid Losing Data".
• Reading it is strongly recommended.

What is new in version 1.9:

• This version makes file times no longer change when changing the system time zone, fixes and improves contrib/benchmark.p, fixes the "Too many chained redirections" bug, retries on 5xx server errors, and adds experimental native support for OpenStack/SWIFT servers and experimental proxy support.

What is new in version 1.8:

• A "ValueError: Can't dump NULL values" crash when file system metadata was uploaded during heavy file system activity was fixed.
• s3qlcp and directory listing performance for very large directories were improved.
• An rm performance regression was fixed.
• --blocksize was renamed to --max-obj-size.
• Some crashes that happened under heavily concurrent file system usage were fixed.
• python-llfuse 0.37 is now used.

Limitations:

• There are still plenty of bugs in the code and you should be ready to deal with them. Nevertheless, you can be reasonably confident that the bugs (although inconvenient) will not endanger your stored data.