Former primary node remains in Running state instead of rejoining as a replica after MaxScale failover

Issue

After a MaxScale automatic failover, the former primary node comes back online but remains in a Running state instead of transitioning to a replica state.

The maxctrl list servers command shows the node as Running rather than Slave, Running.

The MaxScale logs may show the node briefly transitioning to new_slave before reverting to Running with a lost_slave notice, accompanied by SQL or IO errors such as:

• SQL Error: Error 'Unknown database...'
• IO Error: Got fatal error 1236 from master when reading data from binary log...

Environment

• MariaDB MaxScale
• MariaDB Enterprise Server or MariaDB Community Server
• mariadbmon monitor configured with auto_rejoin=true
• Primary-replica replication topology

Cause

This behavior occurs when there is a data or Global Transaction ID (GTID) inconsistency between the two nodes. When the former primary comes back online, MaxScale's auto_rejoin feature successfully attempts to configure it as a replica of the new primary. However, if the nodes are out of sync (e.g., missing databases, divergent GTIDs, or errant transactions), MariaDB replication fails to start.

Because the replication SQL or IO threads stop, MaxScale detects the replication failure and updates the node's state from Slave, Running back to Running.

Resolution

To resolve this issue, the underlying MariaDB replication inconsistency must be corrected so the node can successfully replicate from the new primary.

Warning: Step 4 involves overwriting data on the divergent node. Ensure you have proper backups before proceeding.

1. Review the MariaDB error logs on the former primary node to identify the exact replication error.
2. Execute SHOW SLAVE STATUS\G (or SHOW REPLICA STATUS\G on newer versions) on the former primary to confirm the specific Last_SQL_Error or Last_IO_Error.
3. If the divergence is minor, manually resolve the conflicting transactions or missing schema objects.
4. If the nodes have significantly diverged (e.g., fatal error 1236), rebuild the former primary node by provisioning it from a recent backup of the current primary node using a tool like MariaDB Backup.
5. Once the data is synchronized and replication is manually verified to be working, MaxScale will automatically detect the running replication threads and update the node's state to Slave, Running.

Metadata

• State: In Progress
• Visibility: Customer
• Version verified: Not specified
• Category: Troubleshooting
• Keywords: maxctrl list servers, Running, Slave, Running, auto_rejoin, mariadbmon, lost_slave, fatal error 1236, GTID inconsistency
• Source: 242294