+1 (541) 238-9429

Rebuilding a replica without a maintenance window: clone plugin, snapshots, and dump-and-load

A replica diverges, or the disk fills, or someone writes to it by mistake and replication stops with a duplicate-key error. Now you need a replacement replica, and the question is not which method is best in the abstract. It is which method finishes before your binary logs expire, and how much the source will feel it.

Three options are in common use: the InnoDB clone plugin, a storage-layer snapshot, and a logical dump. They fail in different ways.

First: how long do you have?

Before choosing a method, check binlog retention on the source. Whatever you copy, the new replica resumes from the position the copy was consistent at, and it can only resume if the source still has the binary logs covering the gap between then and now.

SELECT @@binlog_expire_logs_seconds / 3600 AS retention_hours;
SHOW BINARY LOGS;

On RDS and Aurora the equivalent is the binlog retention hours setting, set through mysql.rds_set_configuration, and the default is often too short for a large rebuild. If retention is 24 hours and the copy plus catch-up will take 30, raise retention first. The cost is disk on the source, which is cheap compared with starting the rebuild twice.

The second number to check is your write rate. A replica that takes six hours to provision must then apply six hours of accumulated binlog. If the replica applies changes slower than the source generates them, it never catches up, and the method is wrong no matter how clean the copy was.

The clone plugin

Since 8.0.17 the clone plugin copies InnoDB page files directly from a donor to a recipient over the MySQL protocol. No extra tooling, no filesystem access, no backup agent.

On the donor:

INSTALL PLUGIN clone SONAME 'mysql_clone.so';
CREATE USER clone_user@'10.%' IDENTIFIED BY '...';
GRANT BACKUP_ADMIN ON *.* TO clone_user@'10.%';

On the recipient, which must be a freshly initialised server of the same major version, same OS platform, and compatible configuration:

INSTALL PLUGIN clone SONAME 'mysql_clone.so';
SET GLOBAL clone_valid_donor_list = '10.0.1.20:3306';
CLONE INSTANCE FROM clone_user@'10.0.1.20':3306 IDENTIFIED BY '...';

The recipient's existing data directory is dropped and replaced, then the server restarts itself. Progress is visible in performance_schema.clone_progress and clone_status.

What makes this the default choice: the clone records the donor's binary log position and GTID set, so the new replica knows exactly where to start. With GTIDs enabled you set SOURCE_AUTO_POSITION = 1 and start replication; the handshake works out the rest. Without GTIDs, read clone_status for the binlog file and offset and feed them to CHANGE REPLICATION SOURCE TO.

The costs are real and worth stating. The clone reads the whole buffer pool's worth of pages and the whole tablespace from the donor, so a production donor will see elevated I/O and some page-cache eviction for the duration. Clone from an existing replica rather than the primary when you have one. The clone also blocks DDL on the donor for the duration in 8.0 — concurrent DDL will fail or make the clone fail, depending on timing — so do not run it during a schema migration window. And the recipient must be the same major version: you cannot clone 8.0 to 8.4 and call it an upgrade.

Typical throughput on a 10 Gbit network with decent NVMe is in the low hundreds of MB/s, so figure roughly an hour per terabyte, plus catch-up.

Storage snapshots

If the data lives on EBS, a SAN, or ZFS/LVM, the fastest copy is one you do not make. Snapshot the volume, create a new volume from it, attach, start MySQL, and let InnoDB crash-recover.

This is the clear winner above a few terabytes, because the copy time is largely independent of data size. On RDS it is the only sensible path: restore the snapshot to a new instance and point it at the source as a replica.

Two cautions. First, the snapshot must be crash-consistent at the filesystem level — for a multi-volume layout where the data directory, the redo log, and the binary log sit on different volumes, a per-volume snapshot is not a consistent set unless you freeze writes across all of them. Second, you still need the replication coordinates. Either take the snapshot with FLUSH TABLES WITH READ LOCK held long enough to read SHOW BINARY LOG STATUS (seconds, but it is a global lock), or take the snapshot from a stopped replica, where gtid_executed at shutdown is recorded for you. On EBS, remember a restored volume reads lazily from S3 unless you pay for fast-restore, which can make the first hours of replica apply far slower than the hardware suggests.

Logical dumps

mysqldump --single-transaction --source-data=2 or, for anything non-trivial, mydumper with parallel threads. Logical dumps are an order of magnitude slower to restore than a physical copy, because the recipient rebuilds every index from scratch.

They remain the right tool in three cases: you need a subset of schemas or tables; you are crossing a version or platform boundary where a physical copy is not valid; or the dataset is small enough that simplicity wins. Below roughly 100 GB, a parallel logical load on fast storage finishes in a time nobody will complain about, and it has the useful side effect of defragmenting the tablespaces.

The trap is the long-running transaction the dump holds open on the source. A consistent dump keeps a read view alive for its whole duration, which blocks InnoDB purge and grows the history list — the same mechanism that quietly degrades a busy server over hours. If you dump from the primary, watch history list length while it runs, and prefer to dump from a replica with replication stopped.

Choosing

Small dataset, or crossing versions: logical dump. Large dataset on block storage you control: snapshot. Everything else, and especially the common case of one more read replica on self-managed MySQL between 100 GB and a few terabytes: clone plugin, from an existing replica, with binlog retention raised first.

Whichever you pick, the rebuild is not finished when the copy is. It is finished when Seconds_Behind_Source has been at zero through a peak traffic period, the replica has the same read_only and super_read_only settings as its peers, and whatever checksums your workload allows have confirmed it matches. A replica that lags only under load is a replica that will be removed from the pool during the next incident, which is exactly when you wanted it.

If a rebuild is on your list and the arithmetic on retention versus copy time looks tight, that is a short conversation and usually a cheap one. We do this work hourly, read-only first, and we will tell you if a snapshot would have done the job.