Continuent Tungsten MySQL Replicator Guide
Continuent Tungsten MySQL Replicator Guide
All Materials on this Document are (and shall continue to be) owned exclusively by Continuent or other respective third party owners and are
protected under applicable copyrights, patents, trademarks, trade dress and/or other proprietary rights. Under no circumstances will you acquire
any ownership rights or other interest in any Materials by or through your access or use of the Materials. All right, title and interest not expressly
granted is reserved to Continuent.
Table of Contents
1 Building, Installing, and Running Replicator .................................................................................... 1
1.1 Installation Prerequisites ..................................................................................................... 1
1.2 Building Replicator from Source .......................................................................................... 1
1.3 Installing Replicator ............................................................................................................ 1
1.4 Running Replicator ............................................................................................................. 4
2 Replication Architecture and Key Components ............................................................................... 6
3 Introduction to Replicator .............................................................................................................. 7
3.1 Basic Replication ............................................................................................................... 7
3.2 Provisioning Slaves ............................................................................................................ 7
3.3 Switching the Master .......................................................................................................... 8
3.4 Consistency Checking ........................................................................................................ 8
3.5 Monitoring and Management APIs .................................................................................... 10
3.5.1 JMX/MBean Interface ............................................................................................ 10
4 Troubleshooting Replicator .......................................................................................................... 12
4.1 Error Handling ................................................................................................................. 12
4.2 Dealing with Failed Slaves ............................................................................................... 12
4.3 Dealing with a Failed Master ............................................................................................ 12
4.4 Database Failure .............................................................................................................. 13
4.5 Replicator State Machine .................................................................................................. 13
5 Command Reference Guide ........................................................................................................ 14
5.1 Running the Replicator from the Command Line Interface .................................................. 14
5.2 Running the Replicator as an Operating System Service .................................................... 14
5.3 Controlling a Running Replicator Process .......................................................................... 15
6 Extending the Replicator System ................................................................................................. 17
• tungsten-replicator-version.tgz
• tungsten-replicator-version.zip
1. Copy the distribution archive to the database nodes and unpack at the location of your choice. In the
following code examples, we will use this location as the default directory.
On Linux, Solaris, MacOS X, and other Unix variants we recommend installing in directory /opt/
continuent. On Windows, use for example the C:\Program Files directory.
Note
If you use Windows and cannot unpack the .zip distribution archive, try installing another file
compression program, such as the 7-zip. You can also use the jar program distributed with the
Java JDK.
b. Edit your my.cnf configuration file to enable binlogging and to disable logging of the tungsten
database. Use, for example, the my.cnf configuration file below:
[mysqld]
# Master replication settings.
log-bin=mysql-bin
server-id=1
binlog-ignore-db=tungsten
c. Start MySQL.
d. Create a new empty database called tungsten. Use, for example, the following command on
your MySQL client:
3. Dump the master database and upload it to all slaves in the cluster. For example, issue the following
command on the master:
On the slave:
Note
On Debian based distributions, you may have to copy the password value in /etc/mysql/
debian.cnf from the master to the slave after taking a dump. Otherwise MySQL scripts will
not work.
b. See below for example configurations. Pay attention to the following restrictions and instructions:
• replicator.node_id must be a unique name for each Replicator. The host name is a good
value if you have one Replicator per host.
• replicator.dbms_uri must have the correct host, port, login, and password vale for the
server
• The extractor binlog directory must be the same as defined with the log_bin option in the
my.cnf MySQL configuration file.
In some Linux distributions, the MySQL binlog resides in /var/lib/mysql instead of the de-
fault value in conf/replicator.properties (/var/log/mysql).
Master MySQL:
=============
#
# Node Identifier. This must be unique for each node.
#
replicator.node_id=master_host
#
# Transaction History UUID. This parameter is used to uniquely
# identify transaction history.
#
replicator.history_uuid=2a849f26-d08d-4d69-9ee1-65cb5775fa02
#
# Replicator DBMS driver
#
replicator.dbms_driver=com.mysql.jdbc.Driver
#
# Replicator SQL Database URI. Must have administrator rights.
#
replicator.dbms_uri=jdbc:mysql://localhost/?user=root&password=rootpass
Continuent Replicator Guide - Document issue 1.0 2
Building, Installing, and Running Replicator
#
# Schema to store Replicator metadata
#
replicator.schema=tungsten
replicator.thl.uri=thl://0.0.0.0/
replicator.thl.remote_uri=thl://master_host/
#
# Replicator SQL Event Extractor type. Currently recognized
# values are 'dummy' and 'mysql'.
#
replicator.extractor=mysql
#
# Replicator MySQL Event Extractor binlog directory.
# Notice that in some operating system distributions
# the MySQL binlog resides in /var/lib/mysql.
#
replicator.extractor.mysql.binlog_dir=/var/log/mysql
#
# Replicator MySQL Event Extractor binlog file name pattern.
#
replicator.extractor.mysql.binlog_file_pattern=mysql-bin
#
# Set for events to contain checksums.
# Possible values are 'SHA' or 'MD5' or empty for no checksums
# The master and slave checksum settings must match
#
replicator.event.checksum=md5
#
# How to react on consistency check failure (stop|warn)
#
replicator.applier.consistency_policy=stop
Slave MySQL:
============
#
# Node Identifier. This must be unique for each node.
#
replicator.node_id=slave_1_host
#
# Transaction History UUID. This parameter is used to uniquely
# identify transaction history.
#
replicator.history_uuid=2a849f26-d08d-4d69-9ee1-65cb5775fa02
#
# Replicator DBMS driver
#
replicator.dbms_driver=com.mysql.jdbc.Driver
#
# Replicator SQL Database URI. Must have administrator rights.
#
replicator.dbms_uri=jdbc:mysql://localhost/?user=root&password=rootpass
#
# Schema to store Replicator metadata
#
replicator.schema=tungsten
replicator.thl.uri=thl://0.0.0.0/
replicator.thl.remote_uri=thl://master_host/
#
# Replicator SQL Event Extractor type. Currently recognized values
# are 'dummy' and 'mysql'.
#
replicator.extractor=mysql
#
# Replicator MySQL Event Extractor binlog directory.
# Notice that in some operating system distributions
# the MySQL binlog resides in /var/lib/mysql.
#
replicator.extractor.mysql.binlog_dir=/var/log/mysql
#
# Replicator MySQL Event Extractor binlog file name pattern.
#
replicator.extractor.mysql.binlog_file_pattern=mysql-bin
#
# Set for events to contain checksums.
# Possible values are 'SHA' or 'MD5' or empty for no checksums
# The master and slave checksum settings must match
#
replicator.event.checksum=md5
#
# How to react on consistency check failure (stop|warn)
#
replicator.applier.consistency_policy=stop
If you have followed these instructions so far and have MySQL installed, you should not need to make any
other changes.
The Replicator is run and configured by using shell scripts residing in the /bin directory.
bin/trep_start.sh
a. bin/trep_ctl.sh configure
b. bin/trep_ctl.sh goOnline
c. bin/trep_ctl.sh goMaster
a. bin/trep_ctl.sh configure
b. bin/trep_ctl.sh goOnline
bin/trep_start.bat
a. bin/trep_ctl.bat configure
b. bin/trep_ctl.bat goOnline
c. bin/trep_ctl.bat goMaster
a. bin/trep_ctl.bat configure
b. bin/trep_ctl.bat goOnline
This is all it takes to start Tungsten Replicator master and slaves. You should now have your master and
slave Replicators running and you can check the replication by making some data changes in the master
database and verifying that the changes are reflected in the slave databases.
• Master DBMS - The Database Management System (DBMS), which acts as the master for the replication
system. The master role can change, and any DBMS can be potentially elected as the master for the
replication.
• Slave DBMS - The slave DBMS receives replication events from the master DBMS and applies them.
There can be any number of slaves in the replication system.
• Replication Event Extractor - The replication event extractor extracts replication events from the master
DBMS logs. Events are either SQL statements or rows from the replicated database.
• Transaction History Log - The transaction history log provides a persistent storage for replication events
and communications with other transaction history logs in the cluster.
• Replication Event Applier - The replication event applier applies the replication events into the slave
DBMS.
• Node Manager - Node manager refers to the manager for Tungsten components running either on the
slave or master node. Node manager connects to the Tungsten service manager at the upper level.
Due to the asynchronous nature of the replication, there is some delay before changes made on the master
node will be reflected in the slave nodes. The latency depends on hardware, communication network, and
the SQL load profile and should be measured separately for each installation.
bin/trep_ctl.sh shutdown
In Windows:
bin/trep_ctl.bat shutdown
2. Take dumps from the tungsten database and your own database(s). Here, mydb is used as an ex-
ample:
bin/trep_ctl.sh goOnline
In Windows:
bin/trep_ctl.bat goOnline
4. Make sure that MySQL is running on the new slave. Load the tungsten and the application database
to the new slave:
6. Start the Replicator on the new slave and command it to the SLAVE state:
bin/trep_start.sh&
bin/start_slave.sh
In Windows:
bin/trep_start.bat
bin/start_slave.bat
bin/trep_ctl.sh reconfigure
In Windows:
bin/trep_ctl.bat reconfigure
Wait until the Replicator reaches the SLAVE state. Then, run:
bin/trep_ctl.sh goOnline
bin/trep_ctl.sh goMaster
In Windows:
bin/trep_ctl.bat goOnline
bin/trep_ctl.bat goMaster
bin/trep_ctl.sh reconfigure
In Windows:
bin/trep_ctl.bat reconfigure
Upon startup, the Replicator node manager creates a special table in the Replicator metadata schema:
The table keeps logs of consistency checks. Consistency check is a result of a specially forged transaction,
which should follow the structure below:
START TRANSACTION;
SET @crc:='', @cnt:=0;
REPLACE INTO tungsten.consistency (id, db, tbl, this_cnt, this_crc)
<SELECT_CLAUSE>;
SET @crc:='', @cnt:=0;
SELECT this_cnt, this_crc INTO @cnt, @crc FROM tungsten.consistency
WHERE db = <schema> AND tbl = <table> AND id = <id>;
UPDATE tungsten.consistency
SET master_cnt = @cnt, master_crc = @crc, command = <REPLACE_STATEMENT>
WHERE db = <schema> AND tbl = <table> AND id = <id>;
COMMIT;
Where
• <id> - a unique check identifier for the table. This identifieris used if you want to separately check different
regions of the table or differentiate between checks. The actual value is not important.
• <SELECT_CLAUSE> - a SELECT statement, which provides the values for the row. Its exact form depends
on the table to be checked and the check type. This SELECT statement normally yields the same results
on both master and slave nodes. For example, if a table was created as:
you can check the consistency of the first 100 rows by using a <SELECT_CLAUSE> as follows:
The order and number of <SELECT_CLAUSE> outputs must match the REPLACE inputs.
ment. Notice that the concatenation includes the <SELECT_CLAUSE> as a part of the REPLACE state-
ment. This parameter is required in row-level replication. In the case of statement-level replication, this
parameter can be an empty string.
For compatibility with databases that do not support the REPLACE command, the REPLACE command can
be substituted by an equivalent DELETE INSERT superposition as folows:
For an example of how to script consistency checks on MySQL, please see the mysql_cc_sample.sh
script.
1. When the Extractor detects a consistency check transaction in the binlog stream, it creates a special
consistency check event. In the case of row-level replication, it uses information contained in the last
UPDATE statement to recover the original <REPLACE_STATEMENT>, and restores the transaction to
the statement form.
2. On the slave, the Applier receives the consistency check event and applies the transaction in the
event, as if it were a normal statement-level SQL event. This causes the slave to perform the same
<SELECT_CLAUSE> as the master and the master values are carried by the last UPDATE statement.
Next, the Applier selects the master and slave CRC values from the updated row and compares them.
If the comparison fails, the Applier notifies the node manager of the failure and either stops or contin-
ues working depending on the replicator.applier.consistency_policy setting, which can
be either stop or warn.
• The management system provides a user interface to the Replicator states and error notifications.
• The JMX/MBean interface is the management interface between the Replicator and the management
system.
• The Replicator contains the state machine that changes the Replicator state in line with the state change
commands from the JMX/MBean interface. It also sends state change and error notifications to the JMX/
MBean interface.
• configure() - This method is used to configure the Replicator. It is only allowed in the OFFLINE state.
• reconfigure() - This method is used to re-configure the Replicator. This method only affects param-
eters associated to THL connectivity (remote THL address). This method is allowed in all other states
except in the MASTER state.
• goOnline() - This method is used to set the Replicator to the ONLINE state.
• goMaster() - This method is used to set the Replicator to the MASTER state. It is only allowed in the
ONLINE state.
• shutdown() - This method is used to set the Replicator to the OFFLINE state.
The Replicator also gives notifications. These notifications can be listened to by registering a JMX notifi-
cation listener. See Section 4.1, “Error Handling” for more information on errors. The Replicator can give
notifications as follows:
• StateChangeNotification - This notification is sent when the Replicator state machine reaches a
new state.
• ErrorNotification - This notification is sent when one of the Replicator components encounters an
unrecoverable error.
To recover from a slave node failure, it is essential to first analyze the reason for the node failure and fix
any problem(s) that can prevent the future slave operation.
Once you have fixed the problem on the slave node, you can join the slave node in the cluster by following
the instructions in Section 3.2, “Provisioning Slaves”.
bin/trep_ctl.sh
In Windows:
bin/trep_ctl.bat
2. Check the seqno ranges from both slaves by issuing the command below:
bin/trep_ctl.sh
In Windows:
bin/trep_ctl.bat
Select the slave with the greater max seqno as the new master.
bin/trep_ctl.sh reconfigure
In Windows:
bin/trep_ctl.bat reconfigure
Wait until the Replicator reaches the SLAVE state. Then, run:
bin/trep_ctl.sh goOnline
bin/trep_ctl.sh goMaster
In Windows:
bin/trep_ctl.bat goOnline
bin/trep_ctl.bat goMaster
bin/trep_ctl.sh reconfigure
In Windows:
bin/trep_ctl.bat reconfigure
• OFFLINE - The Replicator process is running, but none of its subprocesses are.
• SYNCHRONIZING - The Replicator applier subprocess is running and is either connecting to the remote
THL instance or waiting for a reconfiguration that changes the remote THL address and the goOnline()
command that follows it.
• ONLINE - The Replicator applier subprocess is running, has connected successfully to the remote THL
instance and detected that a continuous sequence of SQLEvents can be fetched from the remote THL.
• MASTER - The Replicator extractor is running, extracting a database log, and storing SQLEvents into
the THL.
• bin/trep_ctl.sh shutdown
This command is used to stop the Replicator in an orderly manner. Run this command before running
the bin/trep_stop.sh command.
• bin/trep_stop.sh
This command stops the Replicator process if it is running. Run the bin/trep_ctl.sh shutdown command
before this command. After you have issued this command, the master or slave Replicator enters the
OFFLINE state.
• bin/trep_start.sh
• bin/trep_stop.bat
This command is used to command the master or slave Replicator to enter the OFFLINE state.
• bin/trep_start.bat
The Replicator includes a service implementation based on the Java Service Wrapper (http://
wrapper.tanukisoftware.org). This allows you to run the Replicator as a service that has protection against
signals and it also implements the standard interface used by Unix Services. The service implementation
also restarts the Replicator in the event of a crash.
You can adjust the Replicator service configuration by editing the conf/wrapper.properties configu-
ration file. Please read the comments in the file for information on legal settings. For most installations, the
included file should work out of the box.
The Replicator service implementation supports services on 32-bit and 64-bit versions of Linux, and on Mac
OS X platforms.
The Replicator service wrapper binary is called the trepsvc. in practice, the trepsvc is a replacement
for the trep_start.sh and trep_stop.sh commands. The trepsvc commands are summarized below:
• trepsvc start - This command starts the Replicator service if it is not already running. Logs are written
to log trepsvc.log.
• trepsvc status - This command prints out the status of the Replicator service, namely whether it is running
and if it is, on which process number.
• trepsvc stop - This command stops the Replicator service if it is currently running.
• trepsvc restart - This command restarts the Replicator service, stopping it first if it is currently running.
• trepsvc console - This command runs the Replicator service in a Java console program that allows you
to view log output in a GUI shell.
• trepsvc dump - This command sends a 'kill -quit' signal to the Java VM to force it to write a thread dump
to the log. This command is useful for debugging a stuck process.
This command uploads the Replicator configuration. The configure command should only be used
initially, when the node is in the OFFLINE state. For later configuration changes, use the reconfigure
command.
This command uploads the Replicator configuration and resets the applying process in the node. The
reconfigure command can be issued in SYNCHRONIZING and SLAVE states.
• bin/trep_ctl.sh goMaster
This command is used to command the slave Replicator to enter the MASTER state. Always issue the
goOnline command before the goMaster command.
• bin/trep_ctl.sh goOnline
This command is used to command the master Replicator to enter the ONLINE state.
• bin/trep_ctl.sh
Run the bin/trep_ctl.sh command without arguments to check the Replicator state.
This command uploads the Replicator configuration. The configure command should only be used
initially, when the node is in the OFFLINE state. For later configuration changes, use the reconfigure
command.
This command uploads the Replicator configuration and resets the applying process in the node. The
reconfigure command can be issued in SYNCHRONIZING and SLAVE states.
• bin/trep_ctl.bat goMaster
This command is used to command the slave Replicator to enter the MASTER state. Always issue the
goOnline command before the goMaster command.
• bin/trep_ctl.bat goOnline
This command is used to command the master Replicator to enter the ONLINE state.
• bin/trep_ctl.bat
Run the bin/trep_ctl.bat command without arguments to check the Replicator state.
To be more specific, the replication event extracting is a generic framework, which allows adapters for
different DBMSs to plug in. The master node connects to one DBMS only, which means that only one of
the adapters is needed for a particular installation. The adapters convert native DBMS log information into
generic replication events (SQLEvent), which are then propagated to the transaction history log for further
processing.
The diagram below depicts the internal structure of the extracting process and transaction history log. In
this figure, Oracle is used as the source database.