Redis-Standalone Hands-on Guide on Kubernetes ③

Failover test

🖧 1. Redis-Standalone Pod diagram (initial)

┌──────────────┐ │ Redis(220) │ │ 🟢Master │ └──────────────┘

1. Failover test

Take Redis-0 down. There is no replica.
💥 After delete/restart the Pod IP changes.

Redis-0 💥 down (delete)

💻 Terminal-1

$ kubectl delete pod redis-0 -> time: 22:11:15
  pod "redis-0" deleted

💻 Redis-0 LOG

① 15s: got kill signal, Redis down (+0s)
② 18s: Redis auto-restart (+3s)
③ 22s: data load complete (~5s) (+7s)
④ 22s: Redis ready (+7s)

📄 redis.log

1:M 22:11:15.321 *📍got kill signal, Redis down: User requested shutdown...
1:C 22:11:18.053 *📍Redis auto-restart: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:M 22:11:22.782 *📍data load complete: DB loaded from append only file: 4.728 seconds
1:M 22:11:22.788 *📍Redis ready: Ready to accept connections tcp

Full Redis-0 log ▼

📄 redis.log

$ kubectl exec -it redis-0 -- tail -f /data/redis.log
1:signal-handler (1788786675) Received SIGTERM scheduling shutdown...
1:M 22:11:15.321 *📍User requested shutdown...
1:M 22:11:15.321 * Calling fsync() on the AOF file.
1:M 22:11:15.327 * Removing the pid file.
1:M 22:11:15.327 # Redis is now ready to exit, bye bye...

1:C 22:11:18.053 *📍oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 22:11:18.053 * Redis version=8.8.2, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 22:11:18.053 * Configuration loaded
1:M 22:11:18.053 * monotonic clock: POSIX clock_gettime
1:M 22:11:18.054 * Server initialized
1:M 22:11:18.055 * Reading RDB base file on AOF loading...
1:M 22:11:18.055 * Loading RDB produced by version 8.8.2
1:M 22:11:18.055 * RDB age 22168 seconds
1:M 22:11:18.055 * RDB memory usage when created 0.68 Mb
1:M 22:11:18.055 * RDB is base AOF
1:M 22:11:18.055 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 22:11:18.055 * DB loaded from base file aof.1.base.rdb: 0.001 seconds
1:M 22:11:22.782 * DB loaded from incr file aof.1.incr.aof: 4.727 seconds
1:M 22:11:22.782 *📍DB loaded from append only file: 4.728 seconds
1:M 22:11:22.788 * Opening AOF incr file aof.1.incr.aof on server start
1:M 22:11:22.788 *📍Ready to accept connections tcp

💻 App log

① 15s: connection lost -> reconnect fails (+0s)
② 19s: after 5 retries, FAIL (+4s)
③ 24s: reconnect fails (+9s)
④ 32s: reconnect succeeds (+17s)
⑤ 32s: writes succeed (+17s)

💻 Terminal-2

$ kubectl logs -f -l app=redis-standalone-app
T22:11:15.442  WARN :📍Cannot reconnect to redis-0.redis-service.default.svc.cluster.local: Name does not resolve 
  java.net.UnknownHostException: redis-0.redis-service.default.svc.cluster.local: Name does not resolve
T22:11:19.409 ERROR :📍[FAIL] Key: keyA-00005649 -> Error after retries: Redis exception
T22:11:24.433  WARN :📍Cannot reconnect to redis-0.redis-service.default.svc.cluster.local
  java.net.UnknownHostException: redis-0.redis-service.default.svc.cluster.local
T22:11:32.636 INFO :📍Reconnected to redis-0.redis-service.default.svc.cluster.local/<unresolved>:18504
T22:11:32.711 INFO :📍[SUCCESS] Key: keyA-00005690 -> Normal operation in progress...

Full App log ▼

📄 Full App log

[redis@Rocky9 standalone]$ kubectl logs -f -l app=redis-standalone-app
T22:11:11.503  INFO c.r.K.RedisOperationService          :  [SUCCESS] Key: keyA-00005610 -> Normal operation in progress...
T22:11:12.503  INFO c.r.K.RedisOperationService          :  [SUCCESS] Key: keyA-00005620 -> Normal operation in progress...
T22:11:13.503  INFO c.r.K.RedisOperationService          :  [SUCCESS] Key: keyA-00005630 -> Normal operation in progress...
T22:11:14.503  INFO c.r.K.RedisOperationService          :  [SUCCESS] Key: keyA-00005640 -> Normal operation in progress...
T22:11:15.435  INFO i.l.core.protocol.ConnectionWatchdog : Reconnecting, last destination was redis-0.redis-service.default.svc.cluster.local/10.42.0.220:18504
T22:11:15.442  WARN i.l.core.protocol.ConnectionWatchdog :📍Cannot reconnect to [redis-0.redis-service.default.svc.cluster.local/<unresolved>:18504]: redis-0.redis-service.default.svc.cluster.local: Name does not resolve

📍java.net.UnknownHostException: redis-0.redis-service.default.svc.cluster.local: Name does not resolve

T22:11:19.409 ERROR c.r.K.RedisOperationService          :📍[FAIL] Key: keyA-00005649 -> Error after retries: Redis exception
T22:11:19.504 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005650 -> Error after retries: Redis exception
T22:11:19.604 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005651 -> Error after retries: Redis exception
T22:11:19.704 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005652 -> Error after retries: Redis exception
T22:11:19.804 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005653 -> Error after retries: Redis exception
T22:11:19.904 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005654 -> Error after retries: Redis exception
T22:11:20.004 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005655 -> Error after retries: Redis exception
T22:11:20.104 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005656 -> Error after retries: Redis exception
T22:11:23.411 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005657 -> Error after retries: Redis exception
T22:11:23.506 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005658 -> Error after retries: Redis exception
T22:11:23.606 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005659 -> Error after retries: Redis exception
T22:11:23.706 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005660 -> Error after retries: Redis exception
T22:11:23.805 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005661 -> Error after retries: Redis exception
T22:11:23.905 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005662 -> Error after retries: Redis exception
T22:11:24.005 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005663 -> Error after retries: Redis exception
T22:11:24.105 ERROR c.r.K.RedisOperationService          :  [FAIL] Key: keyA-00005664 -> Error after retries: Redis exception
T22:11:24.432  INFO i.l.core.protocol.ConnectionWatchdog : Reconnecting, last destination was redis-0.redis-service.default.svc.cluster.local/<unresolved>:18504
T22:11:24.433  WARN i.l.core.protocol.ConnectionWatchdog :📍Cannot reconnect to [redis-0.redis-service.default.svc.cluster.local/<unresolved>:18504]: redis-0.redis-service.default.svc.cluster.local

📍java.net.UnknownHostException: redis-0.redis-service.default.svc.cluster.local

T22:11:27.413 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005665 -> Error after retries: Redis exception
T22:11:27.507 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005666 -> Error after retries: Redis exception
T22:11:27.607 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005667 -> Error after retries: Redis exception
T22:11:27.707 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005668 -> Error after retries: Redis exception
T22:11:27.807 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005669 -> Error after retries: Redis exception
T22:11:27.907 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005670 -> Error after retries: Redis exception
T22:11:28.007 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005671 -> Error after retries: Redis exception
T22:11:28.107 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005672 -> Error after retries: Redis exception
T22:11:31.414 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005673 -> Error after retries: Redis exception
T22:11:31.509 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005674 -> Error after retries: Redis exception
T22:11:31.609 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005675 -> Error after retries: Redis exception
T22:11:31.709 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005676 -> Error after retries: Redis exception
T22:11:31.808 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005677 -> Error after retries: Redis exception
T22:11:31.908 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005678 -> Error after retries: Redis exception
T22:11:32.008 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005679 -> Error after retries: Redis exception
T22:11:32.108 ERROR c.r.K.RedisOperationService           :  [FAIL] Key: keyA-00005680 -> Error after retries: Redis exception
T22:11:32.632  INFO i.l.core.protocol.ConnectionWatchdog  : Reconnecting, last destination was redis-0.redis-service.default.svc.cluster.local/<unresolved>:18504
T22:11:32.636  INFO i.l.core.protocol.ReconnectionHandler :📍Reconnected to redis-0.redis-service.default.svc.cluster.local/<unresolved>:18504
T22:11:32.711  INFO c.r.K.RedisOperationService           :📍[SUCCESS] Key: keyA-00005690 -> Normal operation in progress...
T22:11:32.733  INFO c.r.K.RedisOperationService           :  [SUCCESS] Key: keyA-00005700 -> Normal operation in progress...
T22:11:32.737  INFO c.r.K.RedisOperationService           :  [SUCCESS] Key: keyA-00005710 -> Normal operation in progress...
T22:11:32.741  INFO c.r.K.RedisOperationService           :  [SUCCESS] Key: keyA-00005720 -> Normal operation in progress...
T22:11:32.745  INFO c.r.K.RedisOperationService           :  [SUCCESS] Key: keyA-00005730 -> Normal operation in progress...

🧮 After failover

Redis-0 IP changed from 10.42.0.220 to 10.42.0.222.

💻 Terminal-1

$ k get pods -o wide
NAME    READY STATUS  RESTARTS  AGE  IP            NODE  
redis-0 1/1   Running 0         1h   10.42.0.222   rocky9

🖧 Redis-Standalone Pod diagram (after failover)

┌──────────────┐ │ Redis(222) │ │ 🟢Master │ └──────────────┘

🧑‍💻 Conclusion

Redis-Standalone failover test succeeded.
Deleting/restarting the Pod changes its IP. After about 17s the app reconnects and writes again.
If ~30s of downtime is acceptable, Redis-Standalone can be an option.

➡ Up to here we tested Redis-Standalone by itself.
Click here to see it working with a Java (Spring) app.

<< K8s Redis-Standalone ②	K8s Redis Scale-Out >>

Email 답글이 올라오면 이메일로 알려드리겠습니다.