Fix manul failover with CLIENT PAUSE/UNPAUSE

[Paragrf]

Motivation

Implement Controller modifications to ensure master-replica data consistency during failover, aligning with server-side changes in apache/kvrocks#3377

Solution

Step 1 (Pause): Send CLIENT PAUSE WRITE from the controller to the current master.

Step 2 (Wait): Monitor the master-replica sequence gap until it hits zero, ensuring no data loss.

Step 3 (Metadata): Update the global topology metadata for the switchover.

Step 4 (Switch & Unpause): Promote the target and demote the old master; then explicitly call CLIENT UNPAUSE on the old master to restore its status.

Step 5 (Replicate): Reconfigure all other followers to sync from the new master.

Configuration Options

To prevent excessive blocking durations during periods of high write traffic, a maximum pause timeout has been introduced; the failover process will fail if the synchronization times out. The following parameters are added to the Controller failover configuration:

• "force_on_timeout": false,

• "sync_timeout_ms": 100,

• "pause_timeout_ms": 500

Related Issues

Fixes #384

[Paragrf]

In actual testing with the Controller and Kvrocks deployed in the same IDC, the system achieved a single-node write QPS of 10k/s with 20MB/s throughput. The write-stop duration (stall time) remained consistently under 10ms.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 43.30357% with 127 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.67%. Comparing base (6c56470) to head (8723839).
⚠️ Report is 106 commits behind head on unstable.

Files with missing lines	Patch %	Lines
store/cluster_node.go	10.00%	54 Missing ⚠️
store/cluster_shard.go	55.81%	32 Missing and 6 partials ⚠️
server/api/shard.go	45.94%	14 Missing and 6 partials ⚠️
controller/cluster.go	36.36%	5 Missing and 2 partials ⚠️
store/cluster_mock_node.go	80.95%	4 Missing ⚠️
server/api/handler.go	0.00%	2 Missing ⚠️
server/route.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable     #389      +/-   ##
============================================
+ Coverage     43.38%   49.67%   +6.28%     
============================================
  Files            37       45       +8     
  Lines          2971     4103    +1132     
============================================
+ Hits           1289     2038     +749     
- Misses         1544     1839     +295     
- Partials        138      226      +88     

Flag	Coverage Δ
unittests	49.67% <43.30%> (+6.28%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jihuayu]

Hi @Paragrf. Thanks for your contribution. I think the changes for manual failover are necessary, but why do we need this for automatic failover?
Are there any other cases that do it this way?

[Paragrf]

@jihuayu All issues are fixed. Ready for another round of review. Thanks!

[Paragrf]

@jihuayu @git-hulk By the way,I initially planned to use slave lags for sync checks, but found it adds a 1s delay because the master only updates these stats every second. To get real-time offsets, I had to query both master and slaves, which unfortunately complicates the logic. Why does Kvrocks cap the slave lag update frequency at 1s rather than providing real-time updates? Is it a performance trade-off to avoid overhead?

[Paragrf]

@jihuayu @git-hulk This commit has been blocked for quite a while now. Whenever you have a moment, could you please take another look? Really appreciate your time and help.

[jihuayu]

Hi @Paragrf. Sorry for the delay! The PR looks good to me so far.

Why does Kvrocks cap the slave lag update frequency at 1s rather than providing real-time updates? Is it a performance trade-off to avoid overhead?

Yes, it is mainly a performance trade-off. The slave lag shown by the master is intended as a coarse-grained monitoring/health metric, so updating it once per second avoids adding extra work or locking on the replication hot path.

[git-hulk]

Thanks @Paragrf @jihuayu, I will take another pass today.

[Paragrf]

Hi @Paragrf. Sorry for the delay! The PR looks good to me so far.

Why does Kvrocks cap the slave lag update frequency at 1s rather than providing real-time updates? Is it a performance trade-off to avoid overhead?

Yes, it is mainly a performance trade-off. The slave lag shown by the master is intended as a coarse-grained monitoring/health metric, so updating it once per second avoids adding extra work or locking on the replication hot path.

@jihuayu @git-hulk OK, if we immediately trigger a _getack command on the server side upon receiving INFO replication, it will force the slave to report its current seq right away. Since INFO replication is currently only invoked by the operator and the controller, the performance impact on the server would be minimal. This also enables the monitoring system to capture real-time slave lags (the current parameter has a 1-second delay between the master and slave, which is virtually meaningless for monitoring purposes). Additionally, the controller's workflow during an active master switch can be simplified. What do you think of this idea?

[git-hulk]

if we immediately trigger a _getack command on the server side upon receiving INFO replication, it will force the slave to report its current seq right away.

We shouldn't let the INFO command do this.

[Paragrf]

if we immediately trigger a _getack command on the server side upon receiving INFO replication, it will force the slave to report its current seq right away.

We shouldn't let the INFO command do this.

@git-hulk OK,How about adding an asynchronous probe in the controller to expose a slave lag metric for better monitoring accuracy?

[git-hulk]

if we immediately trigger a _getack command on the server side upon receiving INFO replication, it will force the slave to report its current seq right away.

We shouldn't let the INFO command do this.

@git-hulk OK,How about adding an asynchronous probe in the controller to expose a slave lag metric for better monitoring accuracy?

Yes, it would be better to do this in the controller. It can sync the lag between nodes before failover.

[Paragrf]

if we immediately trigger a _getack command on the server side upon receiving INFO replication, it will force the slave to report its current seq right away.

We shouldn't let the INFO command do this.

@git-hulk OK,How about adding an asynchronous probe in the controller to expose a slave lag metric for better monitoring accuracy?

Yes, it would be better to do this in the controller. It can sync the lag between nodes before failover.

OK, how about we merge this for now? It's already handled this way in the failover logic of this commit. The discussion here is only about the slave lag metric, which we can always add later.

[git-hulk]

@Paragrf Great thanks for your contribution.