Why is the old master reported as Running instead of Slave after MaxScale failover?

Title

Why is the old master reported as Running instead of Slave after MaxScale failover?

Issue

After a MaxScale automatic failover, the original primary server comes back online but is reported only as Running instead of Slave, Running. The current primary does not recognize the old primary as its replica, and writes performed on the new primary are not replicated back to the old primary.

MaxScale logs may show the server transitioning to new_slave but quickly reverting to lost_slave and Running with errors such as "Unknown database" or "Got fatal error 1236 from master when reading data from binary log".

Environment

• MariaDB MaxScale
• MariaDB Server
• Primary-replica replication topology with mariadbmon and auto_rejoin enabled

Cause

The auto_rejoin feature successfully attempts to reconfigure the old primary as a replica. However, replication stops immediately due to data inconsistencies between the two nodes that existed before the failover.

Common inconsistencies include missing databases on the old primary or Global Transaction ID (GTID) divergence (for example, the old primary executed transactions not present in the new primary's binary logs). When replication fails, MaxScale marks the node as Running rather than an active replica.

Resolution

1. Check the MaxScale logs to identify the exact replication error causing the node to drop out of the replica state.
2. Log in to the demoted primary server and execute SHOW REPLICA STATUS\G to view the specific SQL or IO thread errors.
3. If the error is a minor inconsistency that can be safely resolved, correct the schema difference manually and restart the replication threads using START REPLICA.
4. Warning: The following step will overwrite all data on the demoted primary. If the error is a fatal GTID divergence (such as error 1236), you must rebuild the demoted primary from a fresh backup of the current primary.
5. Use a tool like MariaDB Enterprise Backup to take a full backup of the current primary.
6. Restore the backup onto the demoted primary.
7. Configure the demoted primary to replicate from the current primary using the GTID coordinates from the backup.
8. Verify the server state in MaxScale using maxctrl list servers to ensure it now shows as a running replica.

Metadata

• State: In Progress
• Visibility: Customer
• Version verified: MariaDB MaxScale
• Category: Support / MaxScale
• Keywords: maxctrl list servers, Running, Slave, Master, auto_rejoin, failover, error 1236, GTID, lost_slave
• Source: 242294