Edu/Support

Online Periodic Inspection
Service Now Open

Redis-Sentinel on Kubernetes Test ③

Failover test

🖧 Redis-Sentinel Pod diagram (initial state)

┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │Sentinel-0(98) │ │Sentinel-1(100)│ │Sentinel-2(102)│ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ └───────────────────┼───────────────────┘ ▼ ┌───────────────┐ │ Redis-0(92) │ │ 🟢Master │ └──────┬────────┘ ┌──────────┴───────────┐ ▼ ▼ ┌───────────────┐ ┌───────────────┐ │ Redis-1(94) │ │ Redis-2(96) │ │ 🟡Replica-1 │ │ 🟡Replica-2 │ └───────────────┘ └───────────────┘

1. Failover test

Take Redis-0 down and confirm Redis-1 is promoted from replica to master.
Why Redis-1 becomes master among replicas Redis-1 and Redis-2: redis.conf replica-priority value: Redis-0 100, Redis-1 110, Redis-2 120 -> Redis-1 has the smaller value, so it is promoted to master.
💥 After a Pod is deleted and restarted, its IP changes.

Redis-0 💥 down (delete)

💻 Terminal

$ kubectl delete pod redis-0
  pod "redis-0" deleted

💻 Redis-0 LOG: Master -> Replica

① 03s receives kill signal and Redis goes down (+0s)
② 05s Redis auto-restarts (+2s)
③ 17s receives a command from Sentinel and switches Master -> Replica(Slave) (+14s)

📄 redis.log

1:M 11:03:03.043 #📍down after kill signal: Redis is now ready to exit, bye bye...
1:C 11:03:05.808 *📍restarts after 2s: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:S 11:03:17.398 *📍signal from Sentinel, Master->Replica(Slave): Before turning into a replica,...

Redis-0 full log ▼

📄 redis.log

1:signal-handler (1786413782) Received SIGTERM scheduling shutdown... 
1:M 11:03:03.037 * User requested shutdown...
1:M 11:03:03.043 #📍down after kill signal: Redis is now ready to exit, bye bye...				
1:C 11:03:05.808 *📍restarts after 2s: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 11:03:05.808 * Redis version=8.8.2, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 11:03:05.808 * Configuration loaded				
1:M 11:03:05.808 *➡starts as Master first: monotonic clock: POSIX clock_gettime
1:M 11:03:05.810 * Server initialized
1:M 11:03:05.811 * Reading RDB base file on AOF loading...
1:M 11:03:05.811 * DB loaded from base file aof.1.base.rdb: 0.001 seconds
1:M 11:03:10.458 * DB loaded from incr file aof.1.incr.aof: 4.648 seconds
1:M 11:03:10.458 *➡took 4.6s: DB loaded from append only file: 4.648 seconds 
1:M 11:03:10.462 * Opening AOF incr file aof.1.incr.aof on server start
1:M 11:03:10.462 * Ready to accept connections tcp				
1:S 11:03:17.398 *📍signal from Sentinel, Master -> Replica(Slave): Before turning into a replica, ... 
1:S 11:03:17.398 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 11:03:17.399 * MASTER <-> REPLICA sync started
1:S 11:03:17.399 *➡received 'REPLICAOF redis-1' from Sentinel: REPLICAOF redis-1.redis-service.default.svc.cluster.local:18504 enabled (user request from 'id=6 name=sentinel-...')
1:S 11:03:17.403 *➡wrote redis.conf: CONFIG REWRITE executed with success.
1:S 11:03:17.404 * Master replied to PING, replication can continue...
1:S 11:03:27.929 * Done loading RDB, keys loaded: 2955803, keys expired: 0.
1:S 11:03:27.930 * MASTER <-> REPLICA sync: Finished with success
1:S 11:03:27.930 * MASTER <-> REPLICA sync: Starting AOF after a successful sync
1:S 11:03:31.263 * Background AOF rewrite finished successfully

💻 Redis-1 LOG: Replica -> Master

① 03s detects Redis-0 down (+0s)
② 06s receives replicaof noone from Sentinel-1 and switches replica -> master (+3s)

📄 redis.log

1:S 11:03:03.090 *📍detects Redis-0 down: Connection with master lost. 
1:M 11:03:06.504 *📍command from Sentinel, Replica(Slave) -> Master: Discarding previously cached master state.

Redis-1 full log ▼

📄 redis.log

1:S 11:03:03.090 *📍detects Redis-0 down: Connection with master lost. 
1:S 11:03:03.090 * Caching the disconnected master state.
1:S 11:03:03.091 *➡reconnect attempt: Reconnecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504  
1:S 11:03:03.091 * MASTER <-> REPLICA sync started
1:S 11:03:03.091 #➡connect failed: Error condition on socket for SYNC: Connection refused 
1:S 11:03:03.905 * Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504
1:S 11:03:03.905 * MASTER <-> REPLICA sync started					 
1:M 11:03:06.504 *📍command from Sentinel, Replica(Slave) -> Master: Discarding previously cached master state.
1:M 11:03:06.504 * MASTER MODE enabled (user request from 'id=8 addr=10.42.0.100:49936 laddr=10.42.0.94:18504 fd=16 name=sentinel-bde0e191-cmd...')
1:M 11:03:06.508 *➡wrote redis.conf (removed 'REPLICAOF'): CONFIG REWRITE executed with success.
1:M 11:03:07.355 *➡sync request from redis-2: Replica redis-2.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 11:03:07.356 * Partial resynchronization request from redis-2.redis-service.default.svc.cluster.local:18504 accepted. Sending 596 bytes of backlog starting from offset 432382052.
1:M 11:03:17.404 * Replica redis-0.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 11:03:17.404 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '27e7d7f2495d430017d646aef6776a55870fac45', my replication IDs are 'd735e0345181173ecc516d24ffb58e11e54cc224' and '80d34a3b7754b6bc41d8e95c4645a44540ec73b2')  
1:M 11:03:17.404 * Replica redis-0.redis-service.default.svc.cluster.local:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 11:03:17.404 *➡sync request from redis-0: Replica redis-0.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 11:03:17.405 * Full resync requested by replica redis-0.redis-service.default.svc.cluster.local:18504 (rdb-channel)
1:M 11:03:17.405 * Delay next BGSAVE for diskless SYNC
1:M 11:03:20.981 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 11:03:20.981 * Starting to deliver RDB and replication stream to replica: redis-0.redis-service.default.svc.cluster.local:18504
1:M 11:03:20.994 * Background RDB transfer started by pid 20 to replica socket
20:C 11:03:24.223 * BGSAVE done, 2955803 keys saved, 0 keys skipped, 354696590 bytes written.
20:C 11:03:24.231 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
1:M 11:03:24.309 * Background RDB transfer terminated with success
1:M 11:03:24.309 * Synchronization with replica redis-0.redis-service.default.svc.cluster.local:18504 succeeded
1:M 11:03:24.309 * Streamed RDB transfer with replica redis-0.redis-service.default.svc.cluster.local:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 11:03:24.309 * RDB transfer completed, rdb only replica (redis-0.redis-service.default.svc.cluster.local:18504) should be disconnected asap
1:M 11:03:24.309 * Connection with replica (rdbchannel) redis-0.redis-service.default.svc.cluster.local:18504 lost.

💻 Redis-2 LOG

① 03s detects Redis-0 down (+0s)
② 07s connects to the new master (Redis-1) (+4s)

📄 redis.log

1:S 11:03:03.090 *📍detects Redis-0 down: Connection with master lost. 
1:S 11:03:07.349 *📍connects to new master: Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504

Redis-2 full log ▼

📄 redis.log

1:S 11:03:03.090 *📍detects Redis-0 down: Connection with master lost. 
1:S 11:03:03.090 * Caching the disconnected master state.
1:S 11:03:03.091 *➡reconnect attempt: Reconnecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504  
1:S 11:03:03.091 * MASTER <-> REPLICA sync started
1:S 11:03:03.091 #➡connect failed: Error condition on socket for SYNC: Connection refused 
1:S 11:03:03.965 * Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504
1:S 11:03:03.965 * MASTER <-> REPLICA sync started
1:S 11:03:07.349 *📍connects to new master: Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504  
1:S 11:03:07.350 * MASTER <-> REPLICA sync started
1:S 11:03:07.350 *➡received 'REPLICAOF redis-1' from Sentinel: REPLICAOF redis-1.redis-service.default.svc.cluster.local:18504 enabled (user request from 'id=8 addr=10.42.0.100:33902 laddr=10.42.0.96:18504 fd=16 name=sentinel-bde0e191-cmd ...')  
1:S 11:03:07.355 *➡wrote redis.conf: CONFIG REWRITE executed with success.
1:S 11:03:07.355 * Non blocking connect for SYNC fired the event.
1:S 11:03:07.355 * Master replied to PING, replication can continue...
1:S 11:03:07.355 * Trying a partial resynchronization (request 80d34a3b7754b6bc41d8e95c4645a44540ec73b2:432382052).
1:S 11:03:07.356 * Successful partial resynchronization with master.
1:S 11:03:07.356 * Master replication ID changed to d735e0345181173ecc516d24ffb58e11e54cc224
1:S 11:03:07.356 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

💻 Sentinel-0 LOG

① 06.1s detects redis-0 down (+sdown) (+3s) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 06.2s confirms redis-0 down (+odown) (+3s) -> confirmed when a majority of Sentinels agree

📄 sentinel.log

1:X 11:03:06.131 #📍detects redis-0 down: +sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.242 #📍confirms redis-0 down: +odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 2/2

Sentinel-0 full log ▼

📄 sentinel.log

1:X 11:03:06.131 #📍detects redis-0 down: +sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.231 * Sentinel new configuration saved on disk
1:X 11:03:06.231 # +new-epoch 1
1:X 11:03:06.242 * Sentinel new configuration saved on disk
1:X 11:03:06.242 # +vote-for-leader bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:06.242 #📍confirms redis-0 down: +odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 11:03:06.242 * Next failover delay: I will not start a failover before Tue Aug 11 11:05:06 2026
1:X 11:03:07.350 # +config-update-from sentinel bde0e1917a4b878a3d48742d44f932639a1de44a sentinel-1.sentinel-service 28504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.350 #➡confirms redis-1 as master: +switch-master mymaster redis-0.redis-service.default.svc.cluster.local 18504 redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.351 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504  
1:X 11:03:07.351 * +slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.356 *➡writes changes to sentinel.conf: Sentinel new configuration saved on disk

💻 Sentinel-1 LOG

① 06.1s detects redis-0 down (+sdown) (+3s) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 06.2s confirms redis-0 down (+odown) (+3s) -> confirmed when a majority of Sentinels agree
③ 06.3s Sentinels vote and elect 'Sentinel-1' as Leader (+elected-leader)
④ 06.4s change redis-1 to master (+failover-state-send-slaveof-noone)

📄 sentinel.log

1:X 11:03:06.138 #📍detects redis-0 down: +sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.222 #📍confirms redis-0 down: +odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 11:03:06.327 #📍elected Leader: +elected-leader master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.403 *📍change redis-1 to master: +failover-state-send-slaveof-noone slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504

Sentinel-1 full log ▼

📄 sentinel.log

1:X 11:03:06.138 #📍detects redis-0 down: +sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.222 #📍confirms redis-0 down: +odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 11:03:06.222 # +new-epoch 1
1:X 11:03:06.222 # +try-failover master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.226 * Sentinel new configuration saved on disk
1:X 11:03:06.226 # +vote-for-leader bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:06.243 * 984e3350ee6299ced794cd4fb6820e0d202f0a6e voted for bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:06.243 * b2919c14e1a99a3ca317cf7f91ba75f1d40b50d1 voted for bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:06.327 #📍elected Leader: +elected-leader master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.327 # +failover-state-select-slave master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.403 # +selected-slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.403 *📍change redis-1 to master: +failover-state-send-slaveof-noone slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.503 * +failover-state-wait-promotion slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.295 * Sentinel new configuration saved on disk
1:X 11:03:07.296 # +promoted-slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.296 # +failover-state-reconf-slaves master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.349 *➡notify redis-2 that redis-1 is now master: +slave-reconf-sent slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504  
1:X 11:03:08.322 * +slave-reconf-inprog slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.322 * +slave-reconf-done slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.381 # -odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.381 # +failover-end master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.381 # +switch-master mymaster redis-0.redis-service.default.svc.cluster.local 18504 redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.381 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.382 * +slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.394 * Sentinel new configuration saved on disk

💻 Sentinel-2 LOG

① 06.1s detects redis-0 down (+sdown) (+3s) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 07.2s confirms redis-0 down (+odown) (+4s) -> confirmed when a majority of Sentinels agree

📄 sentinel.log

1:X 11:03:06.114 #📍+sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.239 #📍+odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 3/2

Sentinel-2 full log ▼

📄 sentinel.log

1:X 11:03:06.114 #📍+sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.232 * Sentinel new configuration saved on disk
1:X 11:03:06.232 # +new-epoch 1
1:X 11:03:06.243 * Sentinel new configuration saved on disk
1:X 11:03:06.243 # +vote-for-leader bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:07.239 #📍+odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 11:03:07.239 * Next failover delay: I will not start a failover before Tue Aug 11 11:05:06 2026
1:X 11:03:07.350 # +config-update-from sentinel bde0e1917a4b878a3d48742d44f932639a1de44a sentinel-1.sentinel-service 28504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.350 #➡confirms redis-1 as master: switch-master mymaster redis-0.redis-service.default.svc.cluster.local 18504 redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.351 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.351 * +slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.356 * Sentinel new configuration saved on disk
1:X 11:03:17.398 *➡notify redis-0 that redis-1 is now master: +convert-to-slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504

🧮 State after failover

redis-0 IP changed from '10.42.0.92' to '10.42.0.103'.

💻 Terminal

[redis@Rocky9 kube]$ k get pods -o wide
NAME         READY   STATUS    RESTARTS   AGE     IP            NODE  
redis-0      1/1     Running   0           7s     10.42.0.103   rocky9 📍IP:92->103
redis-1      1/1     Running   0          23m     10.42.0.94    rocky9
redis-2      1/1     Running   0          23m     10.42.0.96    rocky9
sentinel-0   1/1     Running   0          23m     10.42.0.98    rocky9
sentinel-1   1/1     Running   0          22m     10.42.0.100   rocky9
sentinel-2   1/1     Running   0          22m     10.42.0.102   rocky9

🧮 Check redis.conf

Redis LOG: changed during 'CONFIG REWRITE'.

📄 redis.conf

Redis-0(103): replicaof redis-1.redis-service.default.svc.cluster.local 18504
Redis-1(94):  no replicaof -> because Redis-1 is master
Redis-2(96):  replicaof redis-1.redis-service.default.svc.cluster.local 18504

🧮 Check sentinel.conf

Sentinel LOG: changed at 'Sentinel new configuration saved on disk'.
records redis-0,2 as replicas; sentinel-0,1,2 contents are the same.

📄 sentinel.conf

# Generated by CONFIG REWRITE
sentinel monitor mymaster redis-1.redis-service.default.svc.cluster.local 18504 2
sentinel known-replica mymaster redis-0.redis-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-2.redis-service.default.svc.cluster.local 18504

🖧 Redis-Sentinel Pod diagram (after failover)

┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │Sentinel-0(98) │ │Sentinel-1(100)│ │Sentinel-2(102)│ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ └─────┬─────────────┴───────────────────┘ │ │ ┌───────────────┐ │ │ Redis-0(103) │ │ │ 🟡Replica-1 │ │ └───────────────┘ │ ▲ │ ┌──────────┴───────────┐ ▼ │ ▼ ┌──────┴────────┐ ┌───────────────┐ │ Redis-1(94) │ │ Redis-2(96) │ │ 🟢Master │ │ 🟡Replica-2 │ └───────────────┘ └───────────────┘

2. Failback test

Take the current master Redis-1 down and confirm Redis-0 is promoted from replica to master.
📚 'Failback test' is used here to mean making the Redis-0 Pod master again.
(Used to distinguish it from the failover test.)

💻 Redis-1 💥 down (delete)

💻 Terminal

$ kubectl delete pod redis-1
  pod "redis-1" deleted

💻 Redis-0 LOG: Replica -> Master

① 29m 55s Redis-1 connection lost (+0s)
② 29m 58s command from Sentinel, Replica(Slave) -> Master (+3s)

📄 redis.log

1:S 13:29:55.437 *📍Redis-1 connection lost: Connection with master lost.
1:M 13:29:58.714 *📍signal from Sentinel, Replica->Master: Discarding previously cached master state.

Redis-0 full log ▼

📄 redis.log

1:S 13:29:55.437 *📍Redis-1 connection lost: Connection with master lost.
1:S 13:29:55.437 * Caching the disconnected master state.
1:S 13:29:55.437 *➡reconnect attempt: Reconnecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:55.438 * MASTER <-> REPLICA sync started
1:S 13:29:55.438 #➡connect failed: Error condition on socket for SYNC: Connection refused
1:S 13:29:55.765 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:55.766 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:56.771 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:56.772 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:57.776 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:57.777 # Unable to connect to MASTER: Operation now in progress
1:M 13:29:58.714 *📍signal from Sentinel, Replica(Slave) -> Master: Discarding previously cached master state.
1:M 13:29:58.714 * Setting secondary replication ID to d735e0345181173ecc516d24ffb58e11e54cc224, valid up to offset: 434858832. New replication ID is b9c98fef85ebf2b43197de7c0505082ef78ccef1
1:M 13:29:58.714 * MASTER MODE enabled (user request from 'id=15 addr=10.42.0.98:55870 laddr=10.42.0.103:18504 fd=18 name=sentinel-984e3350-cmd ...')
1:M 13:29:58.720 * CONFIG REWRITE executed with success.
1:M 13:29:59.655 * Replica redis-2.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 13:29:59.655 * Partial resynchronization request from redis-2.redis-service.default.svc.cluster.local:18504 accepted. Sending 596 bytes of backlog starting from offset 434858832.
1:M 13:30:11.770 * Replica redis-1.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 13:30:11.770 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '1d3bd6057b7649148d5200d8a78f296834efc05e', my replication IDs are 'b9c98fef85ebf2b43197de7c0505082ef78ccef1' and 'd735e0345181173ecc516d24ffb58e11e54cc224')
1:M 13:30:11.770 * Replica redis-1.redis-service.default.svc.cluster.local:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 13:30:11.770 * Replica redis-1.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 13:30:11.771 * Full resync requested by replica redis-1.redis-service.default.svc.cluster.local:18504 (rdb-channel)
1:M 13:30:11.771 * Delay next BGSAVE for diskless SYNC
1:M 13:30:14.852 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 13:30:14.852 * Starting to deliver RDB and replication stream to replica: redis-1.redis-service.default.svc.cluster.local:18504
1:M 13:30:14.860 * Background RDB transfer started by pid 14 to replica socket
14:C 13:30:17.948 * BGSAVE done, 2955803 keys saved, 0 keys skipped, 354696590 bytes written.
14:C 13:30:17.956 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
1:M 13:30:17.973 * Background RDB transfer terminated with success
1:M 13:30:17.973 * Synchronization with replica redis-1.redis-service.default.svc.cluster.local:18504 succeeded
1:M 13:30:17.973 * Streamed RDB transfer with replica redis-1.redis-service.default.svc.cluster.local:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 13:30:17.973 * RDB transfer completed, rdb only replica (redis-1.redis-service.default.svc.cluster.local:18504) should be disconnected asap
1:M 13:30:17.973 * Connection with replica (rdbchannel) redis-1.redis-service.default.svc.cluster.local:18504 lost.

💻 Redis-1 LOG: Master -> Replica

① 29m 55s down after kill signal (+0s)
② 29m 57s Pod restarts (+2s)
③ 30m 11s command from Sentinel, Master -> Replica(Slave) (+16s)

📄 redis.log

1:M 13:29:55.389 #📍down after kill signal: Redis is now ready to exit, bye bye...
1:M 13:29:57.787 *📍restarts after 2s: Server initialized
1:S 13:30:11.754 *📍signal from Sentinel, Master->Replica: Before turning into a replica,...

Redis-1 full log ▼

📄 redis.log

1:signal-handler (1786422594)➡kill signal: Received SIGTERM scheduling shutdown...
1:M 13:29:54.480 * User requested shutdown...
1:M 13:29:55.389 #📍down after kill signal: Redis is now ready to exit, bye bye...
1:C 13:29:57.785 *oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 13:29:57.785 * Redis version=8.8.2, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 13:29:57.785 * Configuration loaded
1:M 13:29:57.785 *➡starts as Master first: monotonic clock: POSIX clock_gettime
1:M 13:29:57.787 *📍restarts after 2s: Server initialized
1:M 13:29:57.787 * Reading RDB base file on AOF loading...
1:M 13:29:57.788 * Loading RDB produced by version 8.8.2
1:M 13:29:57.788 * RDB age 10198 seconds
1:M 13:29:57.788 * RDB memory usage when created 0.70 Mb
1:M 13:29:57.788 * RDB is base AOF
1:M 13:29:57.788 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 13:29:57.788 * DB loaded from base file aof.2.base.rdb: 0.001 seconds
1:M 13:30:02.391 * DB loaded from incr file aof.2.incr.aof: 4.604 seconds
1:M 13:30:02.391 * DB loaded from append only file: 4.604 seconds
1:M 13:30:02.397 * Opening AOF incr file aof.2.incr.aof on server start
1:M 13:30:02.397 * Ready to accept connections tcp
1:S 13:30:11.754 *📍signal from Sentinel, Master -> Replica(Slave): Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 13:30:11.754 * Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504
1:S 13:30:11.755 * MASTER <-> REPLICA sync started
1:S 13:30:11.755 * REPLICAOF redis-0.redis-service.default.svc.cluster.local:18504 enabled (user request from 'id=6 addr=10.42.0.100:51752 laddr=10.42.0.104:18504 fd=13 name=sentinel-bde0e191-cmd ...')
1:S 13:30:11.769 * CONFIG REWRITE executed with success.
1:S 13:30:11.769 * Non blocking connect for SYNC fired the event.
1:S 13:30:11.770 * Master replied to PING, replication can continue...
1:S 13:30:11.770 * Trying a partial resynchronization (request 1d3bd6057b7649148d5200d8a78f296834efc05e:434858832).
1:S 13:30:11.770 * PSYNC is not possible, initialize RDB channel.
1:S 13:30:14.852 * Starting to receive RDB and replication stream in parallel.
1:S 13:30:14.871 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 13:30:17.954 * Discarding previously cached master state.
1:S 13:30:17.954 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 13:30:17.960 * MASTER <-> REPLICA sync: Flushing old data
1:S 13:30:18.983 * Loading RDB produced by version 8.8.2
1:S 13:30:18.983 * RDB age 4 seconds
1:S 13:30:18.983 * RDB memory usage when created 476.64 Mb
1:S 13:30:21.623 * Done loading RDB, keys loaded: 2955803, keys expired: 0.
1:S 13:30:21.623 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 13:30:21.623 * MASTER <-> REPLICA sync: Finished with success
1:S 13:30:21.623 * MASTER <-> REPLICA sync: Starting AOF after a successful sync
1:S 13:30:21.623 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 13:30:21.634 * Background append only file rewriting started by pid 13
1:S 13:30:21.634 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (1933 bytes).
1:S 13:30:21.635 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (1933 bytes in total)
13:C 13:30:24.866 * BGSAVE done, 2955803 keys saved, 0 keys skipped, 354696457 bytes written.
13:C 13:30:24.872 * Successfully created the temporary AOF base file temp-rewriteaof-bg-13.aof
13:C 13:30:24.881 * Fork CoW for AOF rewrite: current 1 MB, peak 1 MB, average 1 MB
1:S 13:30:24.950 * Background AOF rewrite terminated with success
1:S 13:30:24.950 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-13.aof into aof.3.base.rdb
1:S 13:30:24.950 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.3.incr.aof
1:S 13:30:24.953 * Removing the history file aof.2.incr.aof in the background
1:S 13:30:24.953 * Removing the history file aof.2.base.rdb in the background
1:S 13:30:24.960 * Background AOF rewrite finished successfully

💻 Redis-2 LOG

① 29m 55s connection to Redis-1 lost (+0s)
② 29m 59s Sentinel signals Redis-0 is the new master; reconnect (+4s)

📄 redis.log

1:S 13:29:55.437 *📍connection to Redis-1 lost: Connection with master lost.
1:S 13:29:59.650 *📍connect after Sentinel signals Redis-0 is the new master: Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504

Redis-2 full log ▼

📄 redis.log

1:S 13:29:55.437 *📍connection to Redis-1 lost: Connection with master lost.
1:S 13:29:55.437 * Caching the disconnected master state.
1:S 13:29:55.437 *➡connect attempt to Redis-1: Reconnecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:55.437 * MASTER <-> REPLICA sync started
1:S 13:29:55.438 #➡connect to Redis-1 failed: Error condition on socket for SYNC: Connection refused
1:S 13:29:56.341 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:56.342 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:57.347 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:57.347 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:58.352 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:58.353 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:59.357 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:59.357 # Unable to connect to MASTER: Success
1:S 13:29:59.650 *📍connect after Sentinel signals Redis-0 is the new master: Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504
1:S 13:29:59.650 * MASTER <-> REPLICA sync started
1:S 13:29:59.650 * REPLICAOF redis-0.redis-service.default.svc.cluster.local:18504 enabled (user request from 'id=16 addr=10.42.0.98:54564 laddr=10.42.0.96:18504 fd=17 name=sentinel-984e3350-cmd ...')
1:S 13:29:59.654 * CONFIG REWRITE executed with success.
1:S 13:29:59.654 * Non blocking connect for SYNC fired the event.
1:S 13:29:59.655 * Master replied to PING, replication can continue...
1:S 13:29:59.655 * Trying a partial resynchronization (request d735e0345181173ecc516d24ffb58e11e54cc224:434858832).
1:S 13:29:59.655 * Successful partial resynchronization with master.
1:S 13:29:59.655 * Master replication ID changed to b9c98fef85ebf2b43197de7c0505082ef78ccef1
1:S 13:29:59.655 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

💻 Sentinel-0 LOG

① 58.4s detects redis-1 down (+sdown) (+3s) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 58.5s confirms redis-1 down (+odown) (+3s) -> confirmed when a majority of Sentinels agree
③ 58.5s Sentinels vote and elect 'Sentinel-0' as Leader (+elected-leader)
④ 58.6s change redis-0 to master (+failover-state-send-slaveof-noone)

📄 sentinel.log

1:X 13:29:58.457 #📍detects Redis-1 down: +sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.534 #📍confirms Redis-1 down: +odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 13:29:58.593 #📍elected leader: +elected-leader master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.652 *📍change redis-0 to master: +failover-state-send-slaveof-noone slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504

Sentinel-0 full log ▼

📄 sentinel.log

1:X 13:29:56.546 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.302 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.457 #📍detects Redis-1 down: +sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.521 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.534 #📍confirms Redis-1 down: +odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 13:29:58.534 # +new-epoch 2
1:X 13:29:58.534 # +try-failover master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.538 * Sentinel new configuration saved on disk
1:X 13:29:58.538 # +vote-for-leader 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:58.548 * bde0e1917a4b878a3d48742d44f932639a1de44a voted for 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:58.549 * b2919c14e1a99a3ca317cf7f91ba75f1d40b50d1 voted for 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:58.593 #📍elected leader: +elected-leader master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.593 # +failover-state-select-slave master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.652 # +selected-slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.652 *📍change redis-0 to master: +failover-state-send-slaveof-noone slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.714 * +failover-state-wait-promotion slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.354 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.587 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.596 * Sentinel new configuration saved on disk
1:X 13:29:59.596 # +promoted-slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.596 # +failover-state-reconf-slaves master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.649 * +slave-reconf-sent slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.611 * +slave-reconf-inprog slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.611 * +slave-reconf-done slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.687 #➡confirms Redis-1 restarted: -odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.687 # +failover-end master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.687 #➡notify Redis-1 that Redis-0 is master: +switch-master mymaster redis-1.redis-service.default.svc.cluster.local 18504 redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.688 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.689 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.689 * +slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.693 * Sentinel new configuration saved on disk
1:X 13:30:00.747 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'

💻 Sentinel-1 LOG

① 58.5s detects redis-1 down (+sdown) (+3s) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 58.6s confirms redis-1 down (+odown) (+3s) -> confirmed when a majority of Sentinels agree

📄 sentinel.log

1:X 13:29:58.519 #📍+sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.620 #📍+odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 3/2

Sentinel-1 full log ▼

📄 sentinel.log

1:X 13:29:56.497 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.302 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.459 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.519 #📍+sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.539 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.543 * Sentinel new configuration saved on disk
1:X 13:29:58.543 # +new-epoch 2
1:X 13:29:58.548 * Sentinel new configuration saved on disk
1:X 13:29:58.548 # +vote-for-leader 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:58.620 #📍+odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 13:29:58.620 * Next failover delay: I will not start a failover before Tue Aug 11 13:31:59 2026
1:X 13:29:59.354 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.592 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.650 # +config-update-from sentinel 984e3350ee6299ced794cd4fb6820e0d202f0a6e sentinel-0.sentinel-service 28504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.650 #➡confirms redis-0 as master: +switch-master mymaster redis-1.redis-service.default.svc.cluster.local 18504 redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.651 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.652 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.652 * +slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.657 *➡writes changes to sentinel.conf: Sentinel new configuration saved on disk
1:X 13:29:59.664 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.612 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.689 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:11.754 * +convert-to-slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504

💻 Sentinel-2 LOG

① 58.3s detects redis-1 down (+sdown) (+3s) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 59.4s confirms redis-1 down (+odown) (+3s) -> confirmed when a majority of Sentinels agree

📄 sentinel.log

1:X 13:29:58.301 #📍+sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.454 #📍+odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 3/2

Sentinel-2 full log ▼

📄 sentinel.log

1:X 13:29:56.552 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.301 #📍+sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.459 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.521 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.539 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.543 * Sentinel new configuration saved on disk
1:X 13:29:58.543 # +new-epoch 2
1:X 13:29:58.549 * Sentinel new configuration saved on disk
1:X 13:29:58.549 # +vote-for-leader 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:59.454 #📍+odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 13:29:59.454 * Next failover delay: I will not start a failover before Tue Aug 11 13:31:58 2026
1:X 13:29:59.587 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.592 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.650 # +config-update-from sentinel 984e3350ee6299ced794cd4fb6820e0d202f0a6e sentinel-0.sentinel-service 28504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.650 #➡confirms redis-0 as master: +switch-master mymaster redis-1.redis-service.default.svc.cluster.local 18504 redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.651 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504  
1:X 13:29:59.652 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.652 * +slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.657 *➡writes changes to sentinel.conf: Sentinel new configuration saved on disk
1:X 13:29:59.694 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.612 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.708 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'

🧮 State after failback

💻 Terminal

$ kubectl get pods -o wide
redis-0      1/1     Running   0          147m    10.42.0.103   rocky9 📍IP:92->103
redis-1      1/1     Running   0           39s    10.42.0.104   rocky9 📍IP:94->104
redis-2      1/1     Running   0          170m    10.42.0.96    rocky9
sentinel-0   1/1     Running   0          170m    10.42.0.98    rocky9
sentinel-1   1/1     Running   0          170m    10.42.0.100   rocky9
sentinel-2   1/1     Running   0          170m    10.42.0.102   rocky9

🧮 Check redis.conf

📄 redis.conf

Redis-0: no replicaof	-> Master
Redis-1: replicaof redis-0.redis-service.default.svc.cluster.local 18504
Redis-2: replicaof redis-0.redis-service.default.svc.cluster.local 18504

🧮 Check sentinel.conf

Redis-0 master, Redis-1,2 replicas

📄 sentinel.conf

sentinel monitor mymaster redis-0.redis-service.default.svc.cluster.local 18504 2
sentinel known-replica mymaster redis-1.redis-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-2.redis-service.default.svc.cluster.local 18504

🖧 Redis-Sentinel Pod diagram (failback)

┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │Sentinel-0(98) │ │Sentinel-1(100)│ │Sentinel-2(102)│ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ └───────────────────┼───────────────────┘ ▼ ┌───────────────┐ │ Redis-0(103) │ │ 🟢Master │ └──────┬────────┘ ┌──────────┴───────────┐ ▼ ▼ ┌───────────────┐ ┌───────────────┐ │ Redis-1(104) │ │ Redis-2(96) │ │ 🟡Replica-1 │ │ 🟡Replica-2 │ └───────────────┘ └───────────────┘

🧑‍💻 Conclusion

Redis-Sentinel failover and failback tests succeeded
Deleting and restarting a Pod changes its IP, and it is handled without issues.
Failover (recovery) takes about 4 seconds.

➡ So far we tested Redis-Sentinel by itself.
Click here to see it working with a Java (Spring) app.

<< K8s Redis-Sentinel Setup ②	K8s Redis-Sentinel App ① >>

Email 답글이 올라오면 이메일로 알려드리겠습니다.