State Snapshot Transfer in Galera cluster happens outside of Galera replication. It belongs to node initialization and is specific to the type of a node. The only thing that Galera does there is broadcasting State Transfer Request. It also needs to be notified when state transfer starts and when it ends in order to track node status. In other words: new node provisioning in Galera cluster is no different than creating a new server and importing database to it.
Currently MySQL-wsrep patch includes scripts for the following SST methods (and more can be written for specific use cases):
which can be categorized as follows:
|method||speed||blocks the donor||can be done on live node?||logical/physical||requires root access to MySQL server?|
|mysqldump||slow||yes||yes||logical||both donor and joiner|
|xtrabackup||fast||for a very short time||no||physical||donor only|
Logical/physical distinction is most important here from the configuration point of view:
As can be seen from the above there is no single best SST method. It must be chosen depending on the situation. The good news is that the choice needs to be done only on the receptor node - the donor will serve whatever is requested, as long as it has support for it.
This was the first method supported in MySQL-wsrep patch. It requires the receptor node to have fully functional database (can be empty) and the same root credentials as on donor. It also requires root access from other nodes. It is very slow - several times slower than other methods on sizable databases, but may be faster if the database is very small (smaller than the log files, for example). It is also sensitive to the version of the
mysqldump tool used: it must be the most recent. It is not uncommon for several
mysqldump binaries to be found in the system, in that case it may fail if an older version is incompatible with the newer server.
Its main advantage is that state snapshot by
mysqldump can be transferred to a working server. That is, the server can be started standalone and then be instructed to join a cluster from
mysql client command line. It also can be used to migrate from older database formats to newer. Sometimes it is the only option, when, for example, upgrading from MySQL 5.1 cluster with InnoDB built-in to MySQL 5.5 with InnoDB plugin.
rsync-based state snapshot transfer is the fastest. It has all pluses and minuses of the physical snapshot transfer and in addition it blocks the donor for the whole duration of transfer. However on terabyte-scale databases it was found to be considerably (1.5-2 times) faster than xtrabackup - and that is several hours faster. It also does not care about MySQL configuration or root access, so is probably the easiest to configure.
It also has
rsync-wan modification that engages
rsync's delta transfer algorithm. However this method is more IO intensive so should be used only when the network throughput is the bottleneck, that is usually wide area networks.
The most frequently encountered issue with this method is having incompatible
rsync versions on donor and receptor.
xtrabackup-based state snapshot transfer will most likely be the most popular choice. As rsync it has pluses and minuses of the physical snapshot, but in addition it is virtually non-blocking on donor. It blocks the donor only for a very short period to copy MyISAM tables - like system tables, so if those are small then the blocking time is very short. This naturally goes at the cost of speed: it may be considerably slower than
Also as it still needs to copy large amount of data in the shortest possible time, it may noticeably degrade the donor performance.
The most frequently encountered problem with
xtrabackup is its configuration. As of the time of this writing (2013.02.06) it needs certain options to be set in
my.cnf file (e.g.
datadir) and local root access to the donor server. Refer to
xtrabackup manual for details.