Berkeley DB

Berkeley DB (libdb) is a programmatic toolkit that provides embedded database support for both traditional and client/server applications.

Berkeley DB includes b+tree, queue, extended linear hashing, fixed, and variable-length record access methods, transactions, locking, logging, shared memory caching, database recovery, and replication for highly available systems.

What is new in this release:

• New Features:
• Added an alternate storage format called blobs, which improves the read and update performance of very large data items. Interface available through C, C#, C++ and Java APIs. [#19332]
• Added several pragmas to the SQL layer that configure and give information on HA SQL applications, such as the number of sites in the replication group, or configuring election priority of a site. [#20227]
• Added a BDBSQL_CONCURRENT_CONNECTIONS flag to the SQL API to allow database locking to be disabled. Useful when there are only readers in the database. [#20928]
• When running Replication Manager in multiple processes, if the replication process quits and there is an available subordinate process, the subordinate process automatically takes over as the replication process. [#20969]
• Added replication views/partial replication feature. Feature available through C, C#, C++ and Java APIs. [#21062]
• DPL enhancements to work with JDK7. [#21101]
• Added encryption support for the Android JDBC driver. [#21129]
• Applications which use snapshot isolation and long-running transactions will, in many instances, see reduced use of the cache. This leads to improved performance by reducing the need for temporary freezer files. [#21324]
• Added mpool stat fields st_mvcc_resued, st_oddfsize_detect and st_oddfsize_resolve to C#. [#21325]
• Added support database partitioning in C# APIs. [#21474]
• Added a per-thread cache for the locker struct which stores lock identifiers, such as transaction ids or the locker id returned by DB_ENV->lock_id(). This can improve concurrency of programs which use DB_ENV->set_thread_count(). [#21591]
• Port utilities to Windows Mobile. [#21636]
• Expose 3 new repmgr stat values and a new DB_REPMGR_SITE flag value to C# and Java APIs. [#21687]
• Added support for localizing error messages in UTF-8 characters, and added a section about how to build localized library and an ICU example in Programmer's Reference Guide. [#21280]
• Database Environment Changes:
• Redundant whitespaces are now ignored in DB_CONFIG lines pertaining to directories. [#20158]
• Fixed bugs when attempting to open non-DB files as if they were databases. If encryption was active, an assertion failure would occur. If encryption was not active, the error message "BDB0178 Encrypted database: no encryption flag" would incorrectly imply that a password was required. [#21408]
• Access Method Changes:
• Fixed a bug that could allocate a heap data page in a region after the region creation has been undone. [#20939]
• Fixed a bug in heap that prevented it from working in mixed-endian configurations. [#21016]
• Enhance the search performance for Btree access method when the keys share a common prefix. [#21216]
• Fixed bugs with the byte swapping of heap pages. Now heap databases can be moved between little-endian and big-endian architectures. [#21387]
• Fix a race condition where a cursor in the process of being cleaned up could be accessed by another thread of control. [#21714]
• Duplicating the position of a cursor on a partitioned database will no longer result in a crash. [#22195]
• Fixed an assertion error when closing a joined cursor. [#22398]
• API Changes:
• Change the sequence's cachesize and the delta parameter of DB_SEQUENCE->get to be of type u_int32_t instead of int32_t. [#21552]
• Fixed a bug with the DB_MPOOLFILE->set_maxsize API. In previous releases the file was allowed to grow one page larger than the configured limit. [#21867]
• Change the comparison function in DB->set_bt_compare, DB->set_dup_compare and DB->set_h_compare to accept an extra 'size_t *locp' parameter. [#22235]
• SQL-specific API Changes:
• Added the PRAGMA large_record_opt, which when enabled can improve the read and update performance of very large records. [#19332]
• Added support for multi-process access to the database in SQL HA applications. [#21050]
• In-memory databases can now be created without passing the flag SQLITE_OPEN_CREATE. [#21456]
• Fix a bug in the SQL API where setting the page size to 64 K would cause an error. Also fixed an associated memory leak in SQL online backup. [#21460]
• Add bdbsql_log_buffer pragma to allow setting of log buffer size. [#21471]
• Upgraded SQLite to version 3.7.13. [#21641]
• Fixed a mutex leak with multi-prcess access to the database in SQL HA applications. [#21714]
• Fixed a bug that could cause jdbc build to fail on ARM-Linux using gnueabi gcc. [#22123]
• Upgraded SQLite to version 3.7.16.2. [#22349]
• Upgraded ADO.NET to version 1.0.85.0. [#22405]
• Tcl-specific API Changes:
• Support blob storage APIs in Tcl. [#19332]
• Java-specific API Changes:
• Fixed a bug where EnvironmentConfig.setCreateDir() was not correctly configuring the environment. [#21127]
• Fixed a bug where Database.verify() was not correctly configuring the db. [#22401]
• Fixed a false error message that could be printed when creating a transaction on a replication client in Java. [#21671]
• C#-specific API Changes:
• Clean up many C# build warnings. [#22285]
• Fixed SWIG typemap defintions for C# interface. [#18509]
• Replication Changes:
• Fixed several assertion failures that could result when using failchk and replication. [#19765]
• Provide an error when the local site defined during internal initialization is inconsistent with an earlier definition. [#20870]
• Fixed a bug where replication clients were incorrectly flushing the log when the DB_TXN_WRITE_NOSYNC flag was specified. [#20964]
• Fixed a bug where a valid invocation of the DB_ENV->repmgr_start() method would fail after a previous invocation that supplied an invalid nthreads parameter value. A BDB3640 error was being incorrectly generated. [#21289]
• Fixed a bug where the calculation of the number of required master lease grants enforced that a transaction must be present on a strict majority of replication group sites. It is sufficient that a transaction be present on at least half of the replication group sites, which in some cases reduces the number of required lease grants. [#21329]
• Fixed a recovery issue on the client where an incorrect checkpoint was being selected causing recovery to go to an incorrect point. [#21342]
• Fixed a bug that could result in a crash due to an incorrect re-request on a site that just won an election. [#21344]
• Fixed a bug where logs would never get archived if there are 2 sites and various combinations of failures. [#21448]
• Fixed a bug where we were not detecting the fact that DB_TXN_SNAPSHOT was being used with an HA client. Clients are by definition read only, so snapshot transactions are not permitted. [#21601]
• Fixed a bug where the determination of whether or not a site is peer was incorrectly determined. [#21722]
• Fixed a bug where a Replication Manager master could stop functioning after accepting an obsolete group membership site list from another site. [#21804]
• Reduce the multiple data streams that can sometimes occur if rerequest times are shorter than the time it takes to switch log files or perform a checkpoint at a client site. [#21843]
• Fixed a bug that a site is removed unexpectedly from the replication group when the replication group loses the old master's confirmation of adding the site during a master change. [#21875]
• Fixed a bug where a master site could crash when removing a site if the removed site closes its connections before the master completes its final steps of the operation. [#22050]
• Fixed a bug where an election never happens after the loss of the master when master leases are in use. [#22120]
• Locking Subsystem Changes:
• Fixed a bug that could cause a trap if a lock timeout occured while opening a database. [#21098]
• Fixed a hang which could occur if there were no more space for lockers. [#21774]
• Fixed a potential race condition when building the deadlock detection graph which, under rare conditions, might cause a invalid address fault. [#22261]
• Logging Subsystem Changes:
• Fixed a bug that DB_ENV->log_get_config did not work correctly when it was called before DB_ENV->open. [#21359]
• Memory Pool Subsystem Changes:
• Fixed a bug in the DB_ENV->get_cache_max API that caused it to return an incorrect value and a bug in the DB_ENV->set_cachesize API that caused incorrect behavior when changing the size of an existing cache. [#21146]
• Fixed a race condition occurring when trying to get information about a file and the file is being extended by the OS. [#21193]
• Corrected error messages in several memory pool configuration functions. [#21779]
• Fixed several memory leak items identified by Purify runs. [#22283]
• Mutex Subsystem Changes:
• Fixed an undefined symbol error (MUTEX_IS_OWNED) when configuring with --enable-diagnostic --disable-mutexsupport. [#21145]
• Fix a bug where we were not properly aligning mutexes in DB_PRIVATE regions when the application configured an alignment. [#21491]
• Fix a race condition between DB_ENV->failchk() and the allocation of a mutex. [#21796]
• Limit the default mutex spin count to be at most 200, to avoid excessive spinning on systems with hundreds of hyperthreads. [#21463]
• Test Suite Changes:
• Added support for get/set_heapsize. [#21510]
• Utility Changes:
• Fixed a bug where calling DB->set_partition with a key array required users to keep the memory array valid until after the database was opened. [#18350]
• Fixed a bug that caused DB_ENV->backup to stop early if DB_BACKUP_FILES was not set and a non-DB file was in the data directory. [#21076]
• Fixed a problem with db_hotbackup's -F option, which could cause a segmentation violation. Also replaced an unclear error message. [#21184]
• Fixed three problems with db_hotbackup's backup of transaction logs. A hot backup did not use any configured log directory, but would try to open the logs in the environment home. The second fix corrected an error path, in which the memory was freed by the wrong function, possibly causing a guard byte error. The third fix fixed the issue that a wrong message would be displayed when only "-l" was specified. [#21313]
• Fixed a spurious "No such file or directory" (ENOENT) error when running db_archive in an environment with an empty queue database. [#21541]
• Fixed bugs in compaction of large keys in the upper levels of btrees. [#21569]
• Most utilities no longer operate on replication clients which are being automatically initialized. The DB_REP_LOCKOUT error is now returned. [#21593]
• Fixed a resource leak in the db->verify() function for btrees. The bug would slow down verification and possibly cause it to run out of memory. [##21917]
• Removed a potential hang when compacting databases with many duplicates. [#21975]
• Fixed a bug where we were not returning an error when trying to compact a heap database. [#22063]
• Fixed a bug in the error handling of db_log_verify which would cause a segmenation violation if there were not enough space to create its temporary databases. [#22187]
• Configuration, Documentation, Sample Apps, Portability and Build Changes
• Fix an odbc configure issue which was causing ODBC build failure with SQL API. [#21490]
• Disabled encryption by default in SQL API on Linux, completed build options for encryption in SQL and SQL JDBC on Linux, Windows and Android, and updated build process for the three platforms in Installation Guide. [#21872]
• Use "-xO2" as the default optimization level for the Sun Workshop compiler. [#22057]
• Fix an "opcode not supported" error on MIPS platforms whose assembler defaults to the MIPS1 instruction set. [#22065]
• Provide a consistent way to turn on/off encryption for db,sql,jdbc libraries on Unix. [#22158]
• Added the compile option CONFIG_TEST when using debug mode in the db_csharp Windows project, to be consistent with the db Windows project. [#22300]
• Example Changes:
• Port the C bulk example to C++. [#19541]
• Miscellaneous Bug Fixes:
• Fixed a race condition that could cause a crash if 2 processes opened the same SQL database at the same time. [#21041]
• Fixed an assert failure in the btree comparison function which caused memory to be allocated to the wrong thread. [#21232]
• Fixed a race condition in the SQL handle cache locking code. [#21265]
• Fixed a bug in the SQL code that prevented the table from being created when BDBSQL_FILE_PER_TABLE is defined. [#21417]
• Fixed a bug that some fields of the partition key DBTs returned from DB->get_partition_keys may not be initialized. [#21474]
• Fixed a bug that prevented a sub-database from being created under the directory identified in DB->set_create_dir. [#21603]
• Fixed a bug where dbenv->mp_mtxcount was not being used correctly. [#21768]
• Cleaned a few variable initialization problems that were causing failures on 64-bit Windows. [#21850]
• Improved the randomness of the 20 byte unique fileid generator on Windows. [#22470]
• Fixed a possible assertion failure in the XA Transaction Manager feature. [#22450]
• Deprecated Features:
• Tornado 2.0 is not supported, and VxWorks versions older than 6.9 are not supported. [#21877]

What is new in version 12.1.6.0.20:

• New Features:
• Added an alternate storage format called blobs, which improves the read and update performance of very large data items. Interface available through C, C#, C++ and Java APIs. [#19332]
• Added several pragmas to the SQL layer that configure and give information on HA SQL applications, such as the number of sites in the replication group, or configuring election priority of a site. [#20227]
• Added a BDBSQL_CONCURRENT_CONNECTIONS flag to the SQL API to allow database locking to be disabled. Useful when there are only readers in the database. [#20928]
• When running Replication Manager in multiple processes, if the replication process quits and there is an available subordinate process, the subordinate process automatically takes over as the replication process. [#20969]
• Added replication views/partial replication feature. Feature available through C, C#, C++ and Java APIs. [#21062]
• DPL enhancements to work with JDK7. [#21101]
• Added encryption support for the Android JDBC driver. [#21129]
• Applications which use snapshot isolation and long-running transactions will, in many instances, see reduced use of the cache. This leads to improved performance by reducing the need for temporary freezer files. [#21324]
• Added mpool stat fields st_mvcc_resued, st_oddfsize_detect and st_oddfsize_resolve to C#. [#21325]
• Added support database partitioning in C# APIs. [#21474]
• Added a per-thread cache for the locker struct which stores lock identifiers, such as transaction ids or the locker id returned by DB_ENV->lock_id(). This can improve concurrency of programs which use DB_ENV->set_thread_count(). [#21591]
• Port utilities to Windows Mobile. [#21636]
• Expose 3 new repmgr stat values and a new DB_REPMGR_SITE flag value to C# and Java APIs. [#21687]
• Added support for localizing error messages in UTF-8 characters, and added a section about how to build localized library and an ICU example in Programmer's Reference Guide. [#21280]
• Database Environment Changes:
• Redundant whitespaces are now ignored in DB_CONFIG lines pertaining to directories. [#20158]
• Fixed bugs when attempting to open non-DB files as if they were databases. If encryption was active, an assertion failure would occur. If encryption was not active, the error message "BDB0178 Encrypted database: no encryption flag" would incorrectly imply that a password was required. [#21408]
• Access Method Changes:
• Fixed a bug that could allocate a heap data page in a region after the region creation has been undone. [#20939]
• Fixed a bug in heap that prevented it from working in mixed-endian configurations. [#21016]
• Enhance the search performance for Btree access method when the keys share a common prefix. [#21216]
• Fixed bugs with the byte swapping of heap pages. Now heap databases can be moved between little-endian and big-endian architectures. [#21387]
• Fix a race condition where a cursor in the process of being cleaned up could be accessed by another thread of control. [#21714]
• Duplicating the position of a cursor on a partitioned database will no longer result in a crash. [#22195]
• Fixed an assertion error when closing a joined cursor. [#22398]
• API Changes:
• Change the sequence's cachesize and the delta parameter of DB_SEQUENCE->get to be of type u_int32_t instead of int32_t. [#21552]
• Fixed a bug with the DB_MPOOLFILE->set_maxsize API. In previous releases the file was allowed to grow one page larger than the configured limit. [#21867]
• Change the comparison function in DB->set_bt_compare, DB->set_dup_compare and DB->set_h_compare to accept an extra 'size_t *locp' parameter. [#22235]
• SQL-specific API Changes:
• Added the PRAGMA large_record_opt, which when enabled can improve the read and update performance of very large records. [#19332]
• Added support for multi-process access to the database in SQL HA applications. [#21050]
• In-memory databases can now be created without passing the flag SQLITE_OPEN_CREATE. [#21456]
• Fix a bug in the SQL API where setting the page size to 64 K would cause an error. Also fixed an associated memory leak in SQL online backup. [#21460]
• Add bdbsql_log_buffer pragma to allow setting of log buffer size. [#21471]
• Upgraded SQLite to version 3.7.13. [#21641]
• Fixed a mutex leak with multi-prcess access to the database in SQL HA applications. [#21714]
• Fixed a bug that could cause jdbc build to fail on ARM-Linux using gnueabi gcc. [#22123]
• Upgraded SQLite to version 3.7.16.2. [#22349]
• Upgraded ADO.NET to version 1.0.85.0. [#22405]
• Tcl-specific API Changes:
• Support blob storage APIs in Tcl. [#19332]
• Java-specific API Changes:
• Fixed a bug where EnvironmentConfig.setCreateDir() was not correctly configuring the environment. [#21127]
• Fixed a bug where Database.verify() was not correctly configuring the db. [#22401]
• Fixed a false error message that could be printed when creating a transaction on a replication client in Java. [#21671]
• C#-specific API Changes:
• Clean up many C# build warnings. [#22285]
• Fixed SWIG typemap defintions for C# interface. [#18509]
• Replication Changes:
• Fixed several assertion failures that could result when using failchk and replication. [#19765]
• Provide an error when the local site defined during internal initialization is inconsistent with an earlier definition. [#20870]
• Fixed a bug where replication clients were incorrectly flushing the log when the DB_TXN_WRITE_NOSYNC flag was specified. [#20964]
• Fixed a bug where a valid invocation of the DB_ENV->repmgr_start() method would fail after a previous invocation that supplied an invalid nthreads parameter value. A BDB3640 error was being incorrectly generated. [#21289]
• Fixed a bug where the calculation of the number of required master lease grants enforced that a transaction must be present on a strict majority of replication group sites. It is sufficient that a transaction be present on at least half of the replication group sites, which in some cases reduces the number of required lease grants. [#21329]
• Fixed a recovery issue on the client where an incorrect checkpoint was being selected causing recovery to go to an incorrect point. [#21342]
• Fixed a bug that could result in a crash due to an incorrect re-request on a site that just won an election. [#21344]
• Fixed a bug where logs would never get archived if there are 2 sites and various combinations of failures. [#21448]
• Fixed a bug where we were not detecting the fact that DB_TXN_SNAPSHOT was being used with an HA client. Clients are by definition read only, so snapshot transactions are not permitted. [#21601]
• Fixed a bug where the determination of whether or not a site is peer was incorrectly determined. [#21722]
• Fixed a bug where a Replication Manager master could stop functioning after accepting an obsolete group membership site list from another site. [#21804]
• Reduce the multiple data streams that can sometimes occur if rerequest times are shorter than the time it takes to switch log files or perform a checkpoint at a client site. [#21843]
• Fixed a bug that a site is removed unexpectedly from the replication group when the replication group loses the old master's confirmation of adding the site during a master change. [#21875]
• Fixed a bug where a master site could crash when removing a site if the removed site closes its connections before the master completes its final steps of the operation. [#22050]
• Fixed a bug where an election never happens after the loss of the master when master leases are in use. [#22120]
• Locking Subsystem Changes:
• Fixed a bug that could cause a trap if a lock timeout occured while opening a database. [#21098]
• Fixed a hang which could occur if there were no more space for lockers. [#21774]
• Fixed a potential race condition when building the deadlock detection graph which, under rare conditions, might cause a invalid address fault. [#22261]
• Logging Subsystem Changes:
• Fixed a bug that DB_ENV->log_get_config did not work correctly when it was called before DB_ENV->open. [#21359]
• Memory Pool Subsystem Changes:
• Fixed a bug in the DB_ENV->get_cache_max API that caused it to return an incorrect value and a bug in the DB_ENV->set_cachesize API that caused incorrect behavior when changing the size of an existing cache. [#21146]
• Fixed a race condition occurring when trying to get information about a file and the file is being extended by the OS. [#21193]
• Corrected error messages in several memory pool configuration functions. [#21779]
• Fixed several memory leak items identified by Purify runs. [#22283]
• Mutex Subsystem Changes:
• Fixed an undefined symbol error (MUTEX_IS_OWNED) when configuring with --enable-diagnostic --disable-mutexsupport. [#21145]
• Fix a bug where we were not properly aligning mutexes in DB_PRIVATE regions when the application configured an alignment. [#21491]
• Fix a race condition between DB_ENV->failchk() and the allocation of a mutex. [#21796]
• Limit the default mutex spin count to be at most 200, to avoid excessive spinning on systems with hundreds of hyperthreads. [#21463]
• Test Suite Changes:
• Added support for get/set_heapsize. [#21510]
• Utility Changes:
• Fixed a bug where calling DB->set_partition with a key array required users to keep the memory array valid until after the database was opened. [#18350]
• Fixed a bug that caused DB_ENV->backup to stop early if DB_BACKUP_FILES was not set and a non-DB file was in the data directory. [#21076]
• Fixed a problem with db_hotbackup's -F option, which could cause a segmentation violation. Also replaced an unclear error message. [#21184]
• Fixed three problems with db_hotbackup's backup of transaction logs. A hot backup did not use any configured log directory, but would try to open the logs in the environment home. The second fix corrected an error path, in which the memory was freed by the wrong function, possibly causing a guard byte error. The third fix fixed the issue that a wrong message would be displayed when only "-l" was specified. [#21313]
• Fixed a spurious "No such file or directory" (ENOENT) error when running db_archive in an environment with an empty queue database. [#21541]
• Fixed bugs in compaction of large keys in the upper levels of btrees. [#21569]
• Most utilities no longer operate on replication clients which are being automatically initialized. The DB_REP_LOCKOUT error is now returned. [#21593]
• Fixed a resource leak in the db->verify() function for btrees. The bug would slow down verification and possibly cause it to run out of memory. [##21917]
• Removed a potential hang when compacting databases with many duplicates. [#21975]
• Fixed a bug where we were not returning an error when trying to compact a heap database. [#22063]
• Fixed a bug in the error handling of db_log_verify which would cause a segmenation violation if there were not enough space to create its temporary databases. [#22187]
• Configuration, Documentation, Sample Apps, Portability and Build Changes
• Fix an odbc configure issue which was causing ODBC build failure with SQL API. [#21490]
• Disabled encryption by default in SQL API on Linux, completed build options for encryption in SQL and SQL JDBC on Linux, Windows and Android, and updated build process for the three platforms in Installation Guide. [#21872]
• Use "-xO2" as the default optimization level for the Sun Workshop compiler. [#22057]
• Fix an "opcode not supported" error on MIPS platforms whose assembler defaults to the MIPS1 instruction set. [#22065]
• Provide a consistent way to turn on/off encryption for db,sql,jdbc libraries on Unix. [#22158]
• Added the compile option CONFIG_TEST when using debug mode in the db_csharp Windows project, to be consistent with the db Windows project. [#22300]
• Example Changes:
• Port the C bulk example to C++. [#19541]
• Miscellaneous Bug Fixes:
• Fixed a race condition that could cause a crash if 2 processes opened the same SQL database at the same time. [#21041]
• Fixed an assert failure in the btree comparison function which caused memory to be allocated to the wrong thread. [#21232]
• Fixed a race condition in the SQL handle cache locking code. [#21265]
• Fixed a bug in the SQL code that prevented the table from being created when BDBSQL_FILE_PER_TABLE is defined. [#21417]
• Fixed a bug that some fields of the partition key DBTs returned from DB->get_partition_keys may not be initialized. [#21474]
• Fixed a bug that prevented a sub-database from being created under the directory identified in DB->set_create_dir. [#21603]
• Fixed a bug where dbenv->mp_mtxcount was not being used correctly. [#21768]
• Cleaned a few variable initialization problems that were causing failures on 64-bit Windows. [#21850]
• Improved the randomness of the 20 byte unique fileid generator on Windows. [#22470]
• Fixed a possible assertion failure in the XA Transaction Manager feature. [#22450]
• Deprecated Features:
• Tornado 2.0 is not supported, and VxWorks versions older than 6.9 are not supported. [#21877]

What is new in version 5.3.21:

• The first 5.3 release came out a few month ago. One of the main highlights was support for the Android platform. 5.3.21 is the second release. This is mainly a bugfix release, but it does include one feature which is potentially very interesting, support for Oracle JDK7.