SCALING IN-MEMORY PLATFORMS

When a Valkey Sentinel Service Outgrew a Single Primary

A live migration story from failover-oriented Sentinel to horizontally scalable Cluster mode.

The service had survived failures, but it could no longer absorb growth comfortably. More replicas improved reads and resilience; they did not remove the write and memory ceiling of one primary.

Start with one workload | Scale into a complete platform | Open Source Freedom | Enterprise-Grade Operations.

Published | September 2025

The work before the maintenance window

The service had survived failures, but it could no longer absorb growth comfortably. More replicas improved reads and resilience; they did not remove the write and memory ceiling of one primary.

Moving to Cluster changed key placement, multi-key behaviour, client routing and the meaning of capacity. It could not be treated as an in-place configuration toggle.

The turning point

The team built a parallel cluster, checked hash-slot and command compatibility, moved data in controlled waves, compared the live working set, and shifted traffic behind a reversible gate.

The team used OSS Manager to make online Sentinel-to-cluster migration for traffic scaling reviewable and repeatable. The engine still owned its native behaviour; the control plane made intent, prerequisites, execution and evidence visible to the people responsible for the service.

What had to become explicit

• The service boundary: The existing Sentinel primary-replica service remains live while a multi-master Valkey Cluster is provisioned, populated, validated, and introduced through a controlled write and endpoint transition.
• The operational controls: cluster-aware client proof, multi-key command analysis, hash-tag policy, live data copy, change capture, slot coverage, cutover fencing, reconciliation, and rollback.
• The capacity conversation: keyspace, write rate, large keys, TTL churn, network copy capacity, cluster masters and replicas, slot movement headroom, and cutover duration.
• The human boundary: who may observe, who may approve, who may execute, and who decides whether the application is ready.
  

The architecture that changed the conversation

The diagram is deliberately centred on the decision the team had to make. It is not a product inventory. It shows where authority sits, what crosses the boundary, and where a failed assumption must stop the workflow.

The practical design choices

• Hash slots. Application impact: Keys distribute across primaries. Acceptance: Complete slot coverage.
• Multi-key commands. Application impact: Keys may cross slots. Acceptance: Workload compatibility.
• Client discovery. Application impact: Cluster-aware routing. Acceptance: Reconnect and redirect test.
• Dual run. Application impact: Source remains authoritative during copy. Acceptance: Counts, TTL and error deltas stay bounded.
• Cutover. Application impact: Traffic changes without hidden writes. Acceptance: Reversible gate and quiet source.

These choices are intentionally small enough to review and test. They keep the architecture tied to operating behaviour instead of allowing a visually impressive diagram to hide unclear ownership.

What OSS Manager changed - and what it did not

OSS Manager brought discovery, planning, guarded execution and normalised status into one path for Valkey. It did not replace the engine's correctness model or the application's responsibility for data semantics.

• Before change, validate versions, hosts, identities, artifacts, topology and the recovery boundary without mutating the service.
• During change, persist progress, expose stop conditions and prevent a partial result from being mistaken for success.
• After change, observe copy progress; write delta; key and TTL difference; slot coverage; cluster state; command errors; latency; client redirects; rollback lag and run representative application journeys, not only process checks.
• For recovery, protect the last known-good authority and require an explicit decision before promotion, rollback or destructive cleanup.

The failure modes worth rehearsing

The test plan should make room for non-cluster-aware client, cross-slot operation, hot slot, missed write delta, TTL drift, failed node, incomplete slot coverage, and rollback divergence. The purpose is not to produce a longer checklist. It is to learn whether operators can recognise the failure and choose the safe next action while the evidence is incomplete.

A human operating model

L1 operators need a plain-language answer to what is healthy, what is delayed and whether a change is in progress. Platform engineers need topology and evidence. Application owners need to know what users will experience. A useful platform connects those views without giving every person the same privileges.

Where this pattern earns its place

Retail

This pattern is relevant where teams need campaign peaks, catalogue freshness, transaction continuity and rapid rollback. The architecture must still be adjusted for local data classification, recovery objectives, workload shape and application behaviour.

Telecom

This pattern is relevant where teams need high event volume, distributed operations, identity boundaries and service assurance. The architecture must still be adjusted for local data classification, recovery objectives, workload shape and application behaviour.

Gaming

This pattern is relevant where teams need latency-sensitive sessions, leaderboards, event streams and regional player experience. The architecture must still be adjusted for local data classification, recovery objectives, workload shape and application behaviour.

The lesson we would carry into the next project

The migration succeeded because the architecture acknowledged what changed for applications, not because the data-copy tool ran quickly.

The strongest open-source platforms are not the ones with the most automation. They are the ones where automation makes ownership, risk and recovery easier for people to understand.