Duplicity

Duplicity is an open source software that uses the rsync algorithm to backup directories and produce encrypted tar-format volumes.

Duplicity is able to  upload the backups to a local or remote file server, and it also includes the rdiffdir utility.

System requirements

• Python >= 2.2
• Librsync >= 0.9.6
• GnuPG

What is new in this release:

• Fixed problem in dist/makedist when building on Mac where AppleDouble files were being created in the tarball.

What is new in version 0.7.12:

• Fixed bug #1623342 with patch supplied by Daniel Jakots
• - Failing test on OpenBSD because tar/gtar not found
• Fixed bug #1654220 with patch supplied by Kenneth Newwood
• - Duplicity fails on MacOS because GPG version parsing fails
• Fixed bug #1655268 "--gpg-binary option not working"
• - If gpg binary is specified rebuild gpg profile using new binary location
• Fixed bug #1658283 "Duplicity 0.7.11 broken with GnuPG 2.0"
• - Made gpg version check more robust than just major version
• - Now use --pinentry-mode=loopback on gpg 2.1 and greater
• - Removed check for non-Linux systems, a false problem
• Merged in lp:~matthew-t-bentley/duplicity/duplicity
• - Sets a user agent. Backblaze asked for this in case there are errors that originate from the Duplicity B2 backend
• - Only retrieves a new upload URL when the current one expires, to bring it in line with their best practices for integrations: https://www.backblaze.com/b2/docs/integration_checklist.html
• Add detail about import exceptions in onedrivebackend.py
• Fixed bug #1657916 with patch supplied by Daniel Harvey
• - B2 provider cannot handle two backups in the same bucket
• Fixed bug #1603704 with patch supplied by Maciej Blizinski
• - Crash with UnicodeEncodeError
• Some fixes to gpg.py to handle gpg1 & gpg2 & gpg2.1 commandline issues
• - --gpg-agent is optional on gpg1, but on gpg2 it is used automatically
• - --pinentry-mode is not a valid opt until gpg2.1, so condition on that
• Fixed bug #1367675 - IMAP Backend does not work with Yahoo server
• - added the split() as needed in 'nums=list[0].strip().split(" ")'
• - the other fixes mentioned in the bug report comments were already done
• Fixed bug #1671852 - Code regression caused by revision 1108
• - change util.uexc() back to bare uexc()
• Fixed bug #1668750 - Don't mask backend errors
• - added exception prints to module import errors

What is new in version 0.7.11:

• Fixed bugs #815510 and #1615480
• Changed default --volsize to 200MB
• Merged in lp:~mstoll-de/duplicity/duplicity
• Backblaze announced a new domain for the b2 api
• Merged in lp:~aaron-whitehouse/duplicity/bug_1620085_exclude-if-present-locked-folder
• Fixes Bug #1620085: --exclude-if-present gives OSError looking for tag in locked folders
• Fixed bug #1623342 with patch from Daniel Jakots
• failing test on OpenBSD because tar/gtar not found
• Merged in lp:~mwilck/duplicity/duplicity
• GPG: enable truly non-interactive operation with gpg2
• This patch fixes the IMO unexpected behavior that, when using GnuPG2, a pass phrase dialog always pops up for saving backups. This is particularly annoying when trying to do unattended / fully automatic backups.
• Fixed bug #1621194 with code from Tornhoof
• Do backup to google drive working without a service account
• Fixed bug #1642098does not create PAR2 archives when '--par2-options' is used
• Missing space between par2-options plus default options
• Fix bug using 40-char sign keys, from Richard McGraw on mail list
• Remove truncation of argument and adjust comments
• Merged in lp:~dernils/duplicity/robust-dropbox-backend
• Added new command line option --backend-retry-delay that allows to determine the time that duplicity sleeps before retrying after an error has occured.
• Added some robustness to dpbxbackend.py that ensures re-authentication happens in case that a socket is changed (e.g. due to a forced reconnectof a dynamic internet connection).
• Merged in lp:~ed.so/duplicity/manpage.fixes
• Fix html output via rman on the website
• Merged in lp:~horgh/duplicity/copy-symlink-targets-721599
• Add --copy-links to copy symlink contents, not just the link itself.
• Merged in lp:~aaron-whitehouse/duplicity/Bug_1624725_files_within_folder_slash
• Fixed Bug #1624725, so that an include glob ending in "/" now includes folder contents (for globs with and without special characters). This preserves the behaviour that an expression ending in "/" only matches a folder, but now the contents of any matching folder is included.
• Fix problem with gpg2 in yakety and zesty
• Fix Bug #1642813 with patch from Ravi
• If stat() returns None, don't attempt to set perms.
• Merged in lp:~breunigs/duplicity/amazondrive3
• As reported on the mailinglist, if a space is entered while duplicity asks for the URL, it fails. Since all important spaces are URL encoded anyway, this should be fine even if there are spaces in the URL at all. I also patched it in the onedrive backend, because it must have similar issues.
• Prep for 0.7.11

What is new in version 0.7.10:

• Speedup of path_matches_glob() by about 8x.
• Remove -w from setsid in functional tests.
• Fixed conflict in merge from Martin Wilck and applied merge fixes setsid usage in functional testing.
• Fixed bug #1612472 with patch from David Cuthbert
• Restore from S3 fails with --with-prefix-archive if prefix includes '/'
• Merged in lp:~arashad.ahamad/duplicity/duplicity_latest Changes for connecting to IBM Bluemix ObjectStorage.
• Merged in lp:~fenisilius/duplicity/acd_init_mkdir. Allow duplicity to create remote folder

What is new in version 0.7.09:

• Fixed bug #1600692 with patch from Wolfgang Rohdewald
• Allow symlink to have optional trailing slash during verify.
• Merged in lp:~aaron-whitehouse/duplicity/07-fix_deja_dup_error_on_locked_files
• Revert log.Error to log.Warn, as it was prior to the merge in rev 1224, as this was affecting other applications (e.g. deja dup; Bug #1605939).
• Merged in lp:~duplicity-team/duplicity/po-updates

What is new in version 0.7.08:

• Merged in lp:~noizyland/duplicity/fix_azurebackend_typo
• Fix typo in error handling code
• Merged in lp:~ghoz/duplicity/swift-prefix:
• adds the abiliy to use path in the swift backend, in order to have multiple
• backups to the same container neatly organized.
• Fixed bug #1573957 with patches from Dmitry Nezhevenko:
• upload last chunk with files_upload_session_finish to avoid extra request
• upload small files using non-chunked api
• Fixed bug #1586934 with patches from Dmitry Nezhevenko:
• fixes error handling in wrapper
• Fixed bug #1586992 with patches from Dmitry Nezhevenko:
• Patch adds _delete_list to Par2Backend. And _delete_list fallbacks to
• _delete calls if wrapped backend has no _delete_list.
• Fixed bug #1589038 with patches from Malte Schroder
• Added ignore_case option to selection functions:
• Merged in lp:~mstoll-de/duplicity/b2-reauth:
• Fixes bug #1588503 b2: large uploads fail due to expired auth token
• Merged in lp:~aaron-whitehouse/duplicity/fix_pep8:
• Fix PEP8 error in onedrivebackend.py (space before bracket)
• Fixed bug #822697 ssh-options not passed in rsync over ssh:
• Added globals.ssh_options to rsync command line
• Increased default volume size to 200M, was 25M:
• Fixed README-REPO to no longer mention 0.6-series:
• Merged in lp:~aaron-whitehouse/duplicity/fix_stat_errors:
• Only give an error about not being able to access possibly locked file if
• that file is supposed to be included or scanned (i.e. not excluded).
• Fixes Bug #1089131
• Fixed bug #1594780 with patches from B. Reitsma:
• Use re.finditer() to speed processing
• Merged in lp:~aaron-whitehouse/duplicity/PEP8_W503_fixes:
• Fix PEP8 W503 errors (line break before binary operator) and enable the
• PEP8 test for this in test_code.CodeTest.
• Merged in lp:~aaron-whitehouse/duplicity/PEP8_line_length:
• Set line length error length to 120 (matching tox.ini) for PEP8 and
• fixed E501(line too long) errors.
• Merged in lp:~duplicity-team/duplicity/po-updates

What is new in version 0.7.07.1:

• Fixed bug #1568677 duplicity fails to use existing S3 bucket in boto backend
• bug introduced by incomplete fix of bug #1296793
• simplified setting of bucket locations
• Fixed bug #1569523 get_bucket unknown keyword location and my_location name error
• bug introduced in improper fix of bug #1568677
• gotta love those inconsistent APIs
• Fixed bug #1571134 incompatible with python-oauth2client version 2.x and #1558155 PyDrive backend broken, needs update to oauth2client library
• used patch from https://bugs.debian.org/820725 but made changes to allow the user to continue using the old version
• Fixed bug #1570293 duplicity is very slow due to excessive fsync
• removed flush() after write.
• revert to previous version

What is new in version 0.7.07:

• Merged in lp:~matthew-t-bentley/duplicity/b2
• Fix import and error typos.
• Allow multiple backups in the same bucket.
• Fixes bug #1523498.
• A couple fixes allowing multiple backups to be hosted in different folders in the same bucket as well as some logging for -v9.
• Random stuff:
• remove RPM stuff from makedist
• have makedist pull directly from VCS, not local dir
• update po translation directory and build process
• clean up some odd error messages
• move Pep8 ignores to tox.ini
• supply correct path for pydevd under Mac
• fix some tests to run under Linux and Mac
• Partial fix for bug #1529606 shell code injection in lftpbackend
• still need to fix the other backends that spawn shell commands
• Make test_restart compatible with both GNUtar and BSDtar
• Fix stupid issue with functional test path for duplicity
• Applied patch from shaochun to fix bug #1531154,
• --file-changed failed when file contains spaces
• Applied patch from abeverly to fix bug #1475890
• allow port to be specified along with hostname on S3
• adjusted help text and man page to reflect the change
• Undo changes to test_restart.py. GNU tar is needed.
• Fix minor pep8 nit in collections.py
• Applied changes from ralle-ubuntu to fix bug 1072130.
• duplicity does not support ftpes://
• Fixed bug #1296793 Failed to create bucket
• use S3Connection.lookup() to check bucket exists
• skips Boto's Exception processing for this check
• dupe of bug #1507109 and bug #1537185
• Merged in lp:~mifchip/duplicity/duplicity
• fix bug #1313964, absolute path doesn't work for FTP
• Merged in lp:~fpytloun/duplicity/webdav-gssapi
• support GSSAPI authentication in webdav backend
• Add more pylint ignore warnings tags
• Adjust so test_restart.py can run on Mac as well
• Fix for bug #1538333 assert filecount == len(self.files_changed)
• added flush after every write for all FileobjHooked files which should prevent some errors when duplicity is forcibly closed.
• Fix bug #1540279 mistake in --help
• Applied patch from kay-diam to fix error handling in ssh pexpect, fixes bug #1541314
• Fixed a patching error in ssh_pexpect_backend.py
• Merged in lp:~fpytloun/duplicity/webdav-gssapi-fix
• Make kerberos optional for webdav backend
• Merged in lp:~harningt/duplicity/multibackend-mirror
• This changeset addresses multibackend handling to permit a mirroring option in addition to its "stripe" mode to make it a redundancy tool vs space-expansion tool. To do this without changing the configuration too much, I used the query string that would generally go unused for files to specify behavior that applies to all items inside the configuration file.
• Added acdclibackend.py from Stefan Breunig and Malay Shah
• renamed from amazoncloudbackend to stress use of acd_cli
• Fixed some 2to3 and Pep8 issues that had crept in
• Backed out changes made by patching for bug #1541314. These patches should not have been applied to the 0.7 series.
• Merged in lp:~rye/duplicity/mediafire
• Backend for https://www.mediafire.com
• Requires https://pypi.python.org/pypi/mediafire/ installed.
• Reverted changes made in rev 1164 w.r.t. getting the source from VCS rather than local directory. Fixes bug #1548080.
• More fixes to dist/makedist to make it more OS agnostic.
• Merged in lp:~ed.so/duplicity/webdav.lftp.ssl-overhaul duplicity.1, commandline.py, globals.py added --ssl-cacert-path parameter backend.py make sure url path component is properly url decoded, in case it contains special chars (eg. @ or space) lftpbackend.py quote _all_ cmd line params added missing lftp+ftpes protocol fix empty list result when chdir failed silently added ssl_cacert_path support webdavbackend.py add ssl default context support for python 2.7.9+ (using system certs eg. in /etc/ssl/certs) added ssl_cacert_path support for python 2.7.9+ gettext wrapped all log messages minor refinements
• Applied patch from Dmitry Nezhevenko to upgrade dropbox backend:
• update to SDK v2
• use chunked upload
• Merged in lp:~aaron-whitehouse/duplicity/improve_present_get_sf_man_page
• Improve man page entry for --exclude-if-present
• Merged in lp:~aaron-whitehouse/duplicity/split_glob_matching_from_select
• Move glob matching code out of selection.py's Select function and into globmatch.py.
• Fix bug reported on the mailing list from Mark Grandi (assertion error while backing up). In file_naming.parse() the filename was being lower cased prior to parsing. If you had used a prefix with mixed case, we were writing the file properly, but could not find it in the backend.
• Merged in lp:~duplicity-team/duplicity/po-updates

What is new in version 0.7.06:

• Merged in lp:~mnjul/duplicity/s3-infreq-access
• This adds support for AWS S3's newly announced Infrequent Access storage class and is intended to implement Blueprint: https://blueprints.launchpad.net/duplicity/+spec/aws-s3-std-ia-class .
• A new command line option, --s3-use-ia, is added, and boto backend will automatically use the correct storage class value depending on whether --s3-use-rrs and --s3-use-ia is set. Command line parser will prompt error if both --s3-use-ia and --s3-use-rrs are used together, as they conflict with each other.
• The manpage has been updated giving a short explanation on the new option. Its wording derives from Amazon's official announcement: https://aws.amazon.com/about-aws/whats-new/2015/09/announcing-new- amazon-s3-storage-class-and-lower-glacier-prices/
• The ptyprocess module no longer supports Python 2.6, so fix tox.ini to use an older version. Make explicit environs for all tests.
• Upgrade to newest version of pep8 and pylint. Add three ignores to test_pep8 and one to test_pylint to get the rest to pass. They
• Applied patch from Alexander Zangerl to update to changes in lockfile API 0.9 and later. Updated README to notify users.
• Modded tox.ini to use the latest lockfile.
• Merged in lp:~ed.so/duplicity/setup.shebang
• Having the python interpreter searched in the PATH is much more flexible than the /usr/bin/python inserted into our scripts shebang by setuptools. This patch prevents that. don't touch my shebang!
• Cleanup issues around Launchpad build, mainly lockfile >= 0.9.
• Merged in lp:~michal-s/duplicity/duplicity
• WindowsAzureMissingResourceError and WindowsAzureConflictError changed due to SDK changes. are all valid in our case.
• Reversed previous changes to lockfile. Now it will take any version extant in the LP build repository. (PyPi is not avail in LP build).
• Merged in lp:~ed.so/duplicity/tempfile.tempdir
• make sure packages using python's tempfile create temp files in duplicity's temp dir
• Fixed bug #1511308 - Cannot restore no-encryption, no-compression backup
• Corrected code to include plain file in write_multivolume()
• Added PlainWriteFile() to gpg.py
• Merged in lp:~michal-s/duplicity/duplicity
• Fix azurebackend storage class import
• Merged in lp:~feraudet/duplicity/fix
• Fix missing SWIFT_ENDPOINT_TYPE env var, bug 1519694.
• Fix bug #1520691 - Shell Code Injection in hsi backend
• Replace use of os.popen3() with subprocess equivalent.
• Added code to expand relative program path to full path.
• Fix hisbackend where it expected a list not a string.
• Merged in lp:~noizyland/duplicity/azurebackend-fixes
• Support new version of Azure Storage SDK
• Refactor _list method to support containers with >5000 blobs
• Merged in lp:~matthew-t-bentley/duplicity/b2
• Adds a backed for BackBlaze's (currently beta) B2 backup service.
• This adds backends/b2backend.py, modifies log.py to add an error code and modifies commandline.py to add the b2:// example to the help text.

What is new in version 0.7.05:

• Change use of mock.patch in unit tests to accommodate the obsolete version of python-mock on the build server.
• Fixed Bug 1476019 S3 storage bucket not being automatically created with patch from abeverley
• Add tox testing profile that mimics the packages installed on the Launchpad build server, to reduce the likelihood of tests passing our test suite, but failing on the build server (e.g. because of the out-of-date mock version).
• Set RUN_CODE_TESTS to 0 for lpbuildd tox profile, reflecting its value on the Launchpad build server (and therefore skipping PEP8, 2to3 and pylint). More accurately reflects the system we are mimicking and saves approximately 1 minute per test run.
• Fixed Bug 1438170 duplicity crashes on resume when using gpg-agent with patch from Artur Bodera (abodera). Applied the same patch to incremental resumes as well.
• This little patch logs debug messages concerning path selection process, and so allows users to debug their include/exclude configuration.
• After reorganisation in revision 981 and the fix for bug #1406173 the par2backend does not remove .par2 files anymore when removing duplicity-*.gpg files.
• This banch adds an unfiltered_list() method which is used in delete() and delete_list()
• Updated man pages to reflect more contributors.
• Fix bug #1493573. Correct option typo in man page.
• Fix bug #1494228 CygWin: TypeError: basis_file must be a (true) file. The problem that caused the change to tempfile.TemporaryFile was due to the fact that os.tmpfile always creates its file in the system temp directory, not in the directory specified. The fix applied was to use os.tmpfile in cygwin/windows and tempfile.TemporaryFile in all the rest. This means that cygwin is now broken with respect to temp file placement of this one file (deleted automatically on close).
• This fixes the issue a number of users (including myself) have been having with duplicity creating files with duplicate filenames on Google Drive. It keeps a runtime cache of filename to object ID mappings, so that once it has uploaded an object it won't be fooled by weakly consistent directory listings.

Requirements:

• Python >= 2.2
• Librsync >= 0.9.6
• GnuPG