Edu/Supportk8s_redis_cluster_java_app_2
Redis-Cluster Java App on Kubernetes ②
Failover test
🖧 Redis-Cluster Pod diagram
1. Start the failover test
Open two SSH terminals. In Terminal-1 create the App Pod, take Redis Pods down, and so on. Watch App logs in Terminal-2.
1) Create the Java (Spring) App Pod
Run on the server (Terminal-1 [~/kube/app/cluster/])
$ kubectl apply -f app-deployment.yaml -> create the Pod
💻 App log
Run on the server (Terminal-2 [~/kube/app/cluster])
$ kubectl logs -f -l app=spring-redis-cluster-app
You can see 10 writes per second ([SUCCESS]).
2) Redis (Redis-0) 💥 Pod down (delete)
Terminal-1) $ date; kubectl delete pod redis-cluster-0 -> time: 11:04:49
💻 Redis-0 log
① 49s Redis down (+0s)
② 50s Redis restarted (+1s)
③ 54s switched master -> replica (+5s)
Redis-0 full log
💻 Redis-4 log
① 49s connection with master lost (+0s)
② 54s switched replica -> master (promoted) (+5s)
Redis-4 full log
💻 App log
Terminal-2) $ kubectl logs -f -l app=spring-redis-sentinel-app
① 49s Redis connection lost -> reconnect attempt (+0s)
② 50s writes to Redis-1,2 succeed (+1s)
③ 53s~54s 4 writes to Redis-0 fail (+4s~+5s); then Redis-4 is promoted to master and writes succeed again
④ 59s Redis-0 reconnect succeeds -> writes succeed (+10s)
App full log
🖧 Redis-Cluster Pod diagram: failover state (Redis-4 Master-1)
2. Failback test
1) Redis (Redis-4) Pod 💥 down (delete)
💻 Redis-4 log
① 49s Redis down (+0s)
② 52s Redis restarted (+3s)
③ 55s switched master -> replica (+6s)
Redis-4 full log
💻 Redis-0 log
① 50s connection with master lost (+1s)
② 55s switched replica -> master (+5s)
Redis-0 full log
💻 App log
① 50s Redis-4 connection lost (+1s)
② 54s~56s 5 writes fail (+4s~+6s); then Redis-0 is promoted to master and writes succeed again
③ 59s Redis-4 reconnect succeeds (+10s)
App full log
🖧 Redis-Cluster Pod diagram: failback state (Redis-0 Master-1)
🧑💻 Test review and key summary
This failover/failback test verified how Redis-Cluster on Kubernetes and the Java (Spring Boot / Lettuce) app actually handle failure and rebuild cluster topology.
⏱️ Failover timeline
| Phase | Time (relative) | Redis Cluster node behavior | Java (Spring) App behavior |
|---|---|---|---|
| Failure | 11:04:49 (+0s) | Master (Redis-0) Pod terminated (delete) | Redis-0 connection lost; background reconnect. 📍Writes aimed at healthy nodes (Redis-1, 2) keep succeeding without blocking thanks to async handling (11:04:50~) |
| Retry limit reached | 11:04:53~54 (+4~5s) | Replica (Redis-4) detected failure and is in a promotion election | Some requests (4) toward Redis-0 slots exceeded the background 5 retries (5s) and were finally marked FAIL |
| Failover complete | 11:04:54 (+5s) | After the vote, Replica (Redis-4) is promoted to new Master | Cluster topology refresh attempted (ClusterTopologyRefresh) |
| Fully healthy | 11:04:55 (+6s) | New Master (Redis-4) starts read/write on those slots (0~5460) | Updated topology applied; writes to all slots resume at 100% SUCCESS |
| Old master restarted | 11:04:59 (+10s) | Restarted old master (Redis-0) is reconfigured as a Replica of the new master (Redis-4) | Redis-0 node reconnect completed |
💡 Key takeaways for production
-
1. Even if Redis-0 master is down, writes to remaining Redis-1,2 still succeed
-
2. Cluster failure detection and promotion time (about 4~5s):
After a master goes down, the other nodes check status forcluster-node-timeout(FAIL decision), then the replica is auto-promoted. That produces about 4~5 seconds of downtime. -
3. Why Spring / Lettuce Dynamic Topology Refresh matters:
In Redis Cluster, when a master changes, ownership of its hash slots moves. The Spring App (Lettuce) cluster topology auto-refresh option detected the new master IP and slot map in real time, so the integration held without restarting the service. -
4. Effect and limits of client-side retry (5s):
In the first test (Redis-0 down), failover finished at 5 seconds, inside the App retry window, so the success rate was 100% (0 lost). In the second test (Redis-4 down), topology rebalancing time differed and one failed request (keyA-00000607) occurred. In production you should design async queues and retry together with App retry to aim for true zero downtime. -
5. Cluster auto node role reconfiguration:
When the recovered old master (Redis-0) comes back, it receives the new node state on the cluster bus and automatically becomes a replica of the new master (Redis-4). HA replica count is restored without operator intervention. -
6. UX and timeout considerations:
Clients see about 4~5 seconds of latency during failure. Combine Front-end Timeout and a Circuit Breaker (Resilience4j, etc.) so the web/app UI does not freeze on API calls.
🛡️ Front-end Timeout and Circuit Breaker
🛡️ Safety nets that keep Redis failure from freezing the system
The 4~5 second latency during failover can cascade to backend servers and the user UI (cascading failure). In production, combine the two safety nets below.
1. Front-end Timeout
After the browser or mobile app calls a backend API, this is the maximum time before giving up instead of waiting forever.
- Problem (no timeout): If Redis stalls for 4 seconds, impatient users mash refresh or the button. Backend Tomcat/Spring threads exhaust and the whole web server goes down.
- Fix (2~3s timeout): If there is no response in 2 seconds, abort immediately and show "Temporary network delay. Please try again shortly" to stop a backend stampede.
2. Circuit Breaker (Resilience4j)
Like an electrical breaker inside the Spring backend: when Redis failure is detected it opens the circuit and immediately stops trying Redis.
- 🟢 CLOSED (healthy): All cache/data requests go to Redis as usual.
- 🔴 OPEN (tripped): If Redis latency/error rate exceeds the threshold, open the breaker, do not even try Redis, and immediately run prepared fallback (e.g. query the main RDB).
- 🟡 HALF-OPEN (probe): Send a few requests to see if Redis recovered; if they succeed, return to 🟢 CLOSED.
🔄 How the two techniques work together during failure
- 0s (Redis Master down): Failover starts; latency appears
- 1~2s (Circuit Breaker): Spring App Resilience4j detects errors/latency and switches the breaker to 🔴 OPEN ➔ query RDB or return a default (Fallback) instead of Redis
- 2s (Front-end Timeout): The frontend cuts the request so infinite loading and user mashing do not flood the WAS
- 4~5s (Failover complete & auto recovery): After the new Master is promoted, Resilience4j switches back to 🟢 CLOSED and cache service recovers automatically
| << K8s Redis-Cluster App ① | K8s Redis-Sentinel Prep ① >> |
|---|
