Mimeo

Mimeo can be used to move database tables from one PostgreSQL database to another.

Includes support for three replication strategies:

1. Snapshot - entire table copied from one DB to another

2. Incremental - data is moved as new content is detected

3. DML - takes data from one database and permanently moves it to a new location

What is new in this release:

• Function was added in version 1.1.0, so any updates from version 1.0.1 to later would fail on PostgreSQL 9.1.
• Upgrading PostgreSQL to >= 9.2 will also allow Mimeo to be updated to 1.1.0 and greater (Recommended fix).

What is new in version 1.3.7:

• Function was added in version 1.1.0, so any updates from version 1.0.1 to later would fail on PostgreSQL 9.1.
• Upgrading PostgreSQL to >= 9.2 will also allow Mimeo to be updated to 1.1.0 and greater (Recommended fix).

What is new in version 1.3.6:

• Function was added in version 1.1.0, so any updates from version 1.0.1 to later would fail on PostgreSQL 9.1.
• Upgrading PostgreSQL to >= 9.2 will also allow Mimeo to be updated to 1.1.0 and greater (Recommended fix).

What is new in version 1.3.4:

• Function was added in version 1.1.0, so any updates from version 1.0.1 to later would fail on PostgreSQL 9.1.
• Upgrading PostgreSQL to >= 9.2 will also allow Mimeo to be updated to 1.1.0 and greater (Recommended fix).

What is new in version 1.1.1:

• Account for role names with special characters in them (dashes were the culprit that found this bug).
• Changed dblink_mapping table to an extension specific name: dblink_mapping_mimeo. Allows it to be installed in same schema as pg_jobmon if desired.
• Changed advisory lock system to use transactional advistory locks instead of explicit ones. Gets rid of errors in exception block when exception is hit before advisory lock is taken.
• Organized documentation better.

What is new in version 1.0.0:

• First stable release.

What is new in version 0.13.0:

• Made the pg_jobmon extension optional. It can be turned on or off on a per replication table basis using the new "jobmon" boolean column in the config table. If pg_jobmon is installed it will be set to true by default for all replication types, otherwise it will be set false. You can also set whether it is used at runtime with the "p_jobmon" parameter to the refresh functions.
• Jobmon logging has been added to the table replication method.

What is new in version 0.12.0:

• Logdel refresh function now has a "repull" option to do a complete refresh of data from the source. Note that it will NOT delete the rows on the destination that were previously deleted from the source (only deletes rows on the destination where mimeo_source_deleted column is null). Since a TRUNCATE cannot be done as is done with the other replication repulls, it is highly recommended to do a manual VACUUM of destination table after this is done, possibly even a VACUUM FULL to reclaim disk space.
• To help the above repull process be more efficient, an index is now created on the "mimeo_source_deleted" destination column of all newly created logdel replication tables. Existing replication tables will not have this index added. Recommend adding one if you need to do this repull method on old setups.
• DML & Logdel replication source queue tables now have indexes created on the "processed" and "processed, mimeo_source_deleted" columns respectively. This should help replication be more efficient for higher traffic tables that create larger queues. Existing replication sets will not have their source queue table modified to add this index. Recommend going back and manually adding it if you notice performance problems.
• Exception messages if dml/logdel maker functions fail are clearer about what has happened reguarding objects created on the source.
• Added p_debug option to maker functions. Also made debugging a little cleaner and have it provide more information in many cases.
• Bug Fix: WHERE condition of logdel replication wasn't working properly.

What is new in version 0.11.0:

• Snapshot & incremental maker functions now ensure that the given table names are schema qualified.
• Updated the extras functions for older source versions of postgresql. There's new dml functions for 8.1 as well as custom array_agg() functions that Mimeo requires.

What is new in version 0.10.2:

• New refresh_table option for just doing a straight truncate and repull for a regular table. Only adding as a minor feature because other changes do not affect API and it's not recommended as a regular refresh job if possible. Also refresh_table() does not currently log to pg_jobmon, so cannot be monitored and will not set off any alerts if/when this refresh type fails. What this is useful for is having a way to get data from production to a staging/dev database where you still want to be able to edit the destination table. Could do that with Incremental or DML, but this avoids requiring any primary keys, control columns or write access on the source database.
• Removed custom enum type. Made things much more complicated than they needed to be. Enums are REALLY bad in extensions since you can never just add a new value in an update.
• The commands to remove the batch limit defaults & set the default boundaries for incremental replication were missing from the mimeo--0.9.3--0.10.1 bypass update file. If that file was used to update mimeo, the defaults were never changed and limits were still being set for new refresh jobs. Run those here again.

What is new in version 0.10.0:

• Now that pulling larger amounts of data is more efficient, there is no longer a default batch limit. Every refresh run will get all new rows available. This does not affect existing jobs, only newly created jobs after this version is installed. If you'd like remove the batch limits (recommended unless necessary), set the batch_limit column to NULL in all config tables.
• Boundary value is now required for inserter/updater configuration to help ensure consistency. Default is 10 minutes. If any had been set as NULL, it has been updated to the default.
• New "extras" folder in repository. Contains some versions of functions for when the source database is older than 9.1. These can make it possible to use mimeo to upgrade a database across major versions. While they could be used for normal replication purposes as well, that is not advised long term since an extension update would overwrite the functions if you rename them to their original names.
• Extras folder also contains a python script to replace the deprecated run_refresh function. Use this to help make running scheduled replication jobs easier and more reliable that the old plpgsql function.
• Set functions as STABLE or IMMUTABLE that can be.
• Bug Fix: Temporary queue table not getting dropped in refresh_logdel. Caused errors if same refresh was run in the same session.
• Bug Fix: Will now accurately report row counts even when inserting to partitioned tables.
• Code cleanup.
• Make pgTAP tests a little more extensive. Added tests for batch limits.

What is new in version 0.9.3:

• Reaching the exact batch limit to cause a pg_jobmon log warning for dml/logdel replication would almost never actually occur due to getting DISTINCT entries from the source queue table. This would make notification of replication possibly falling behind not occur. Changed the alert to happen if the actual batch reaches 75% of the batch limit.

What is new in version 0.8.3:

• Fixed dml refresh not propagating updates and deletes.
• Fixed dml/logdel refresh not updating a row if it has a multi-column primary/unique key and only a subset of the columns of that key are changed.
• Fixed edge case in refresh_dml/logdel where, if the batch limit was hit, the remote queue table might not mark the processed rows properly.
• Changed tests to use pgTAP. Testing suite is now much more extensive and helped find above bugs.

Requirements:

• PostgreSQL 9.1 or higher
• pg_jobmon 0.3 or higher
• PostgreSQL dblink extension