Edu/Support

Online Periodic Inspection
Service Now Open

Redis-Sentinel Sidecar Pattern Hands-on Guide on Kubernetes ③

Failover test

🖧 1. Redis-Sentinel Pod diagram (initial)

┌─────────────────┐ │ Redis-0(106) │ │ 🟢Master │ │ Sentinel-0 │ └───────┬─────────┘ ┌───────────┴──────────┐ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ │ Redis-1(108) │ │ Redis-2(110) │ │ 🟡Replica-1 │ │ 🟡Replica-2 │ │ Sentinel-1 │ │ Sentinel-2 │ └─────────────────┘ └─────────────────┘

1. Failover test

Take Redis-0 down and confirm Redis-1 is promoted from replica to master

Why Redis-1 becomes master among replicas 1 and 2: replica-priority: Redis-0=100, Redis-1=110, Redis-2=120 -> lower value wins, so Redis-1 is promoted.
Redis and Sentinel share Pod-0, so they go down together.
💥 After restart both Redis and Sentinel IPs change.

Pod-0 (Redis-0/Sentinel-0) 💥 down (delete)

💻 Terminal

$ kubectl delete pod redis-sentinel-0
  pod "redis-sentinel-0" deleted

💻 Redis-0 LOG: Master -> Replica

① 27s: got kill signal, Redis down (+0s)
② 30s: Redis auto-restart (+3s)
③ 43s: Sentinel command, Master -> Replica (+16s)

📄 redis.log

1:M 19:39:27.556 #📍got kill signal, down: Redis is now ready to exit, bye bye...
1:C 19:39:30.701 *📍restart after 3s: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:S 19:39:43.119 *📍Sentinel signal, Master -> Replica: Before turning into a replica, ...

Full Redis-0 log ▼

📄 redis.log

1:signal-handler (1786444767) Received SIGTERM scheduling shutdown... 
1:M 19:39:27.551 * User requested shutdown...
1:M 19:39:27.556 #📍got kill signal, down: Redis is now ready to exit, bye bye...
1:C 19:39:30.701 *📍restart after 2s: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 19:39:30.701 * Redis version=8.8.1, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 19:39:30.701 * Configuration loaded				
1:M 19:39:30.701 *➡starts as master and loads AOF: monotonic clock: POSIX clock_gettime
1:M 19:39:30.703 * Server initialized
1:M 19:39:30.704 * Reading RDB base file on AOF loading...
1:M 19:39:30.704 * Loading RDB produced by version 8.8.1
1:M 19:39:30.704 * RDB age 12883 seconds
1:M 19:39:30.704 * RDB memory usage when created 0.68 Mb
1:M 19:39:30.704 * RDB is base AOF
1:M 19:39:30.704 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 19:39:30.704 * DB loaded from base file aof.1.base.rdb: 0.001 seconds
1:M 19:39:35.251 * DB loaded from incr file aof.1.incr.aof: 4.547 seconds
1:M 19:39:35.251 *➡took 4.5s: DB loaded from append only file: 4.548 seconds 
1:M 19:39:35.256 * Opening AOF incr file aof.1.incr.aof on server start
1:M 19:39:35.256 * Ready to accept connections tcp				
1:S 19:39:43.119 *📍Sentinel signal, Master -> Replica: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 19:39:43.119 * Connecting to MASTER redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 ➡📍 connect to new master 
1:S 19:39:43.120 *➡start receiving data from new master: MASTER <-> REPLICA sync started 
1:S 19:39:43.120 * REPLICAOF redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 enabled (user request from 'id=9 addr=10.42.0.108:37556 laddr=10.42.0.111:18504 fd=15 name=sentinel-1f2...')
1:S 19:39:43.126 * CONFIG REWRITE executed with success.
1:S 19:39:43.126 * Non blocking connect for SYNC fired the event.
1:S 19:39:43.127 * Master replied to PING, replication can continue...
1:S 19:39:43.127 * Trying a partial resynchronization (request 48ff07935a656f2184c8a4bfef0b326bbbe6553a:436188020).
1:S 19:39:43.127 * PSYNC is not possible, initialize RDB channel.
1:S 19:39:46.584 * Starting to receive RDB and replication stream in parallel.
1:S 19:39:46.617 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 19:39:49.765 * Discarding previously cached master state.
1:S 19:39:49.766 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 19:39:49.775 * MASTER <-> REPLICA sync: Flushing old data
1:S 19:39:50.780 * Loading RDB produced by version 8.8.1
1:S 19:39:50.780 * RDB age 4 seconds
1:S 19:39:50.780 * RDB memory usage when created 602.67 Mb
1:S 19:39:53.441 * Done loading RDB, keys loaded: 2955238, keys expired: 0.
1:S 19:39:53.441 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 19:39:53.441 *➡finished loading from new master: MASTER <-> REPLICA sync: Finished with success 
1:S 19:39:53.441 *➡AOF rewrite start: MASTER <-> REPLICA sync: Starting AOF after a successful sync 
1:S 19:39:53.441 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 19:39:53.450 * Background append only file rewriting started by pid 13
1:S 19:39:53.450 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (2233 bytes).
1:S 19:39:53.451 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (2233 bytes in total)
13:C 19:39:56.651 * BGSAVE done, 2955238 keys saved, 0 keys skipped, 354628657 bytes written.
13:C 19:39:56.656 * Successfully created the temporary AOF base file temp-rewriteaof-bg-13.aof
13:C 19:39:56.663 * Fork CoW for AOF rewrite: current 1 MB, peak 1 MB, average 1 MB
1:S 19:39:56.765 * Background AOF rewrite terminated with success
1:S 19:39:56.765 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-13.aof into aof.2.base.rdb
1:S 19:39:56.765 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.2.incr.aof
1:S 19:39:56.771 * Removing the history file aof.1.incr.aof in the background
1:S 19:39:56.772 * Removing the history file aof.1.base.rdb in the background
1:S 19:39:56.776 *➡AOF rewrite done: Background AOF rewrite finished successfully

💻 Redis-1 LOG: Replica -> Master

① 27s: detect Redis-0 down (connection lost) (+0s)
② 31s: Sentinel replicaof no one, Replica -> Master (+4s)

📄 redis.log

1:S 19:39:27.597 *📍detect Redis-0 down (connection lost): Connection with master lost. 
1:M 19:39:31.083 *📍Sentinel command, Replica -> Master: MASTER MODE enabled

Full Redis-1 log ▼

📄 redis.log

1:S 19:39:27.597 *📍detect Redis-0 down (connection lost): Connection with master lost. 
1:S 19:39:27.598 * Caching the disconnected master state.
1:S 19:39:27.598 * Reconnecting to MASTER redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 ➡📍reconnect attempt 
1:S 19:39:27.599 #➡connect failed: Unable to connect to MASTER: Success 
1:S 19:39:28.508 * Connecting to MASTER redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504
1:S 19:39:28.509 # Unable to connect to MASTER: Success
1:S 19:39:29.512 * Connecting to MASTER redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504
1:S 19:39:29.513 # Unable to connect to MASTER: Success
1:S 19:39:30.517 * Connecting to MASTER redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504
1:S 19:39:30.518 # Unable to connect to MASTER: Success					
1:M 19:39:31.083 * Discarding previously cached master state.
1:M 19:39:31.083 * Setting secondary replication ID to fc340b7e739af336c0a8ba9b8d7fbef15baa5f0b, valid up to offset: 436188020. New replication ID is 4f961426ec4fc18f4d78cb18cdc1bfab7ad056cc
1:M 19:39:31.083 *📍Sentinel signal, Replica -> Master: MASTER MODE enabled (user request from 'id=9 addr=10.42.0.110:38598 laddr=10.42.0.108:18504 fd=14 name=sentinel-4d...')
1:M 19:39:31.087 * CONFIG REWRITE executed with success.
1:M 19:39:32.010 * Replica redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 asks for synchronization
1:M 19:39:32.010 * Partial resynchronization request from redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 accepted. Sending 465 bytes of backlog starting from offset 436188020.
1:M 19:39:43.127 * Replica redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 asks for synchronization
1:M 19:39:43.127 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '48ff07935a656f2184c8a4bfef0b326bbbe6553a', my replication IDs are '4f961426ec4fc18f4d78cb18cdc1bfab7ad056cc' and 'fc340b7e739af336c0a8ba9b8d7fbef15baa5f0b')
1:M 19:39:43.127 * Replica redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 19:39:43.127 * Replica redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 asks for synchronization
1:M 19:39:43.128 * Full resync requested by replica redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 (rdb-channel)
1:M 19:39:43.128 * Delay next BGSAVE for diskless SYNC
1:M 19:39:46.584 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 19:39:46.584 * Starting to deliver RDB and replication stream to replica: redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504
1:M 19:39:46.596 * Background RDB transfer started by pid 26 to replica socket
26:C 19:39:49.759 * BGSAVE done, 2955238 keys saved, 0 keys skipped, 354628790 bytes written.
26:C 19:39:49.766 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
1:M 19:39:49.813 * Background RDB transfer terminated with success
1:M 19:39:49.813 * Synchronization with replica redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 succeeded
1:M 19:39:49.813 * Streamed RDB transfer with replica redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 19:39:49.813 * RDB transfer completed, rdb only replica (redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504) should be disconnected asap
1:M 19:39:49.813 * Connection with replica (rdbchannel) redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 lost.

💻 Redis-2 LOG: Replica

① 27s: detect Redis-0 down (connection lost) (+0s)
② 32s: connect to new master Redis-1 (+5s)

📄 redis.log

1:S 19:39:27.597 *📍detect Redis-0 down: Connection with master lost. 
1:S 19:39:32.005 *📍Connecting to MASTER redis-sentinel-1.redis-sentinel-service...:18504

Full Redis-2 log ▼

📄 redis.log

1:S 19:39:27.597 *📍detect Redis-0 down: Connection with master lost. 
1:S 19:39:27.597 * Caching the disconnected master state.
1:S 19:39:27.598 *➡reconnect attempt: Reconnecting to MASTER redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504  
1:S 19:39:27.599 #➡connect failed: Unable to connect to MASTER: Success 
1:S 19:39:28.202 * Connecting to MASTER redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504
1:S 19:39:28.203 # Unable to connect to MASTER: Success
1:S 19:39:29.207 * Connecting to MASTER redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504
1:S 19:39:29.208 # Unable to connect to MASTER: Success
1:S 19:39:30.212 * Connecting to MASTER redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504
1:S 19:39:30.213 # Unable to connect to MASTER: Success
1:S 19:39:31.217 * Connecting to MASTER redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504
1:S 19:39:31.218 # Unable to connect to MASTER: Success                 
1:S 19:39:32.005 *📍Sentinel: Redis-1 is now master: Connecting to MASTER redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504
1:S 19:39:32.005 *➡receive data from Redis-1: MASTER <-> REPLICA sync started 
1:S 19:39:32.005 * REPLICAOF redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 enabled (user request from 'id=5 addr=10.42.0.110:55940 laddr=10.42.0.110:18504 fd=15 name=sentinel-...')
1:S 19:39:32.009 * CONFIG REWRITE executed with success.
1:S 19:39:32.009 * Non blocking connect for SYNC fired the event.
1:S 19:39:32.009 * Master replied to PING, replication can continue...
1:S 19:39:32.010 * Trying a partial resynchronization (request fc340b7e739af336c0a8ba9b8d7fbef15baa5f0b:436188020).
1:S 19:39:32.010 * Successful partial resynchronization with master.
1:S 19:39:32.010 * Master replication ID changed to 4f961426ec4fc18f4d78cb18cdc1bfab7ad056cc
1:S 19:39:32.010 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

💻 Sentinel-0 LOG

① 27s: kill signal, Sentinel down (+0s)
② 30s: Sentinel auto-restart (+3s)
③ 34s: start monitoring Redis-1 (+7s)

📄 redis.log

1:X 19:39:27.533 #📍Sentinel-0 goes down with Redis-0: Sentinel is now ready to exit, bye bye...
1:X 19:39:30.785 *📍restart after 3s oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 
1:X 19:39:34.035 #📍start monitoring Redis-1: +switch-master mymaster redis-sentinel-1...

Full Sentinel-0 log ▼

📄 redis.log

1:signal-handler (1786444767) Received SIGTERM scheduling shutdown... 
1:X 19:39:27.533 * User requested shutdown...
1:X 19:39:27.533 * Removing the pid file.
1:X 19:39:27.533 #📍Sentinel-0 goes down with Redis-0: Sentinel is now ready to exit, bye bye...
1:X 19:39:30.785 *📍restart after 3s oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 19:39:30.785 * Redis version=8.8.1, bits=64, commit=00000000, modified=1, pid=1, just started
1:X 19:39:30.785 * Configuration loaded
1:X 19:39:30.785 * monotonic clock: POSIX clock_gettime
1:X 19:39:30.786 * Running mode=sentinel, port=28504.
1:X 19:39:30.790 * Sentinel new configuration saved on disk
1:X 19:39:30.790 * Sentinel ID is d68b6617a21c51bd7ef03cf857c474aef544679c
1:X 19:39:30.790 # +monitor master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 quorum 2
1:X 19:39:34.025 * +sentinel sentinel 1f227bb1efbb0a6e25594ffaf28bbc3110b6b2cd redis-sentinel-1.redis-sentinel-service 28504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:34.030 * Sentinel new configuration saved on disk
1:X 19:39:34.034 * Sentinel new configuration saved on disk
1:X 19:39:34.034 # +new-epoch 1
1:X 19:39:34.035 # +config-update-from sentinel 1f227bb1efbb0a6e25594ffaf28bbc3110b6b2cd redis-sentinel-1.redis-sentinel-service 28504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:34.035 #📍start monitoring redis-1: +switch-master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:34.035 * +slave slave redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:34.039 * Sentinel new configuration saved on disk
1:X 19:39:35.067 * +sentinel sentinel 4d41e0ad06a5e2b48c87b09cf7136473569efe40 redis-sentinel-2.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:35.071 * Sentinel new configuration saved on disk
1:X 19:39:40.843 * +slave slave redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:40.848 * Sentinel new configuration saved on disk

💻 Sentinel-1 LOG

① 27s: Redis-0 connection lost (+0s)
② 30s: +sdown on Redis-0 (+3s) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
③ 31s: +odown on Redis-0 (+4s) -> after majority of Sentinels agree
④ 43s: notify Redis-0 that Redis-1 is master (+16s)

📄 redis.log

1:X 19:39:27.579 #📍Redis-0 connection lost: Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service' 
1:X 19:39:30.649 #📍detect Redis-0 down: +sdown sentinel 1d4285d6f0c8027fb97375d2ce4ae7168a8b74ca redis-sentinel-0.redis-sentinel-service 28504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:31.807 #📍confirm Redis-0 down: +odown master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 19:39:43.119 *📍notify Redis-0 that Redis-1 is master: +convert-to-slave slave redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504

Full Sentinel-1 log ▼

📄 redis.log

1:X 19:39:27.579 #📍Redis-0 connection lost: Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service' 
1:X 19:39:27.652 # Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local' ➡📍detect Sentinel-0 down
1:X 19:39:28.603 # Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service'
1:X 19:39:28.660 # Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local'
1:X 19:39:30.649 #📍detect Redis-0 down: +sdown sentinel 1d4285d6f0c8027fb97375d2ce4ae7168a8b74ca redis-sentinel-0.redis-sentinel-service 28504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:30.712 # +sdown master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:30.822 # Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local'
1:X 19:39:30.878 # Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local'
1:X 19:39:30.883 * Sentinel new configuration saved on disk
1:X 19:39:30.883 # +new-epoch 1
1:X 19:39:30.892 * Sentinel new configuration saved on disk
1:X 19:39:30.892 # +vote-for-leader 4d41e0ad06a5e2b48c87b09cf7136473569efe40 1
1:X 19:39:31.807 #📍confirm Redis-0 down: +odown master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 19:39:31.807 * Next failover delay: I will not start a failover before Tue Aug 11 19:41:31 2026
1:X 19:39:31.939 # Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local'
1:X 19:39:32.005 # +config-update-from sentinel 4d41e0ad06a5e2b48c87b09cf7136473569efe40 redis-sentinel-2.redis-sentinel-service 28504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:32.005 # +switch-master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:32.006 * +slave slave redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:32.006 # Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local'
1:X 19:39:32.006 * +slave slave redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:32.011 * Sentinel new configuration saved on disk
1:X 19:39:32.012 # Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local'
1:X 19:39:34.803 * +sentinel-invalid-addr sentinel 1d4285d6f0c8027fb97375d2ce4ae7168a8b74ca redis-sentinel-0.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:34.803 * +sentinel sentinel d68b6617a21c51bd7ef03cf857c474aef544679c redis-sentinel-0.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:34.816 * Sentinel new configuration saved on disk
1:X 19:39:43.119 *📍notify Redis-0 that Redis-1 is master: +convert-to-slave slave redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504

💻 Sentinel-2 LOG

① 27s: Redis-0 connection lost (+0s)
② 30s: +sdown on Redis-0 (+3s) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
③ 30s: +odown on Redis-0 (+3s) -> after majority of Sentinels agree
③ 30s: Sentinels elect Sentinel-2 as leader (+elected-leader) (+3s)
④ 31s: promote Redis-1 (+failover-state-send-slaveof-noone) (+4s)

📄 redis.log

1:X 19:39:27.674 #📍Redis-0 connection lost: Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service'
1:X 19:39:30.692 #📍detect Redis-0 down: +sdown sentinel 1d4285d6f0c8027fb97375d2ce4ae7168a8b74ca redis-sentinel-0.redis-sentinel-service 28504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:30.873 #📍confirm Redis-0 down: +odown master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 19:39:30.940 #📍elected leader: +elected-leader master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504  
1:X 19:39:31.031 *📍promote Redis-1 to master: +failover-state-send-slaveof-noone slave redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504

Full Sentinel-2 log ▼

📄 redis.log

1:X 19:39:27.674 #📍Redis-0 connection lost: Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service'
1:X 19:39:27.776 # Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local'
1:X 19:39:28.612 # Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service'
1:X 19:39:30.692 #📍detect Redis-0 down: +sdown sentinel 1d4285d6f0c8027fb97375d2ce4ae7168a8b74ca redis-sentinel-0.redis-sentinel-service 28504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:30.713 # Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local'
1:X 19:39:30.820 # +sdown master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:30.873 #📍confirm Redis-0 down: +odown master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 19:39:30.873 # +new-epoch 1
1:X 19:39:30.873 # +try-failover master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:30.877 * Sentinel new configuration saved on disk
1:X 19:39:30.877 # +vote-for-leader 4d41e0ad06a5e2b48c87b09cf7136473569efe40 1
1:X 19:39:30.892 * 1f227bb1efbb0a6e25594ffaf28bbc3110b6b2cd voted for 4d41e0ad06a5e2b48c87b09cf7136473569efe40 1
1:X 19:39:30.940 #📍elected leader: +elected-leader master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504  
1:X 19:39:30.940 # +failover-state-select-slave master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:31.031 # +selected-slave slave redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:31.031 *📍promote Redis-1 to master: +failover-state-send-slaveof-noone slave redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:31.083 * +failover-state-wait-promotion slave redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:31.737 # Failed to resolve hostname 'redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local'
1:X 19:39:31.942 * Sentinel new configuration saved on disk
1:X 19:39:31.942 # +promoted-slave slave redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:31.942 # +failover-state-reconf-slaves master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:32.004 *➡notify Redis-2 that Redis-1 is master: +slave-reconf-sent slave redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:32.992 * +slave-reconf-inprog slave redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:32.992 * +slave-reconf-done slave redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:33.047 # -odown master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:33.047 # +failover-end master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:33.047 # +switch-master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:33.048 * +slave slave redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:33.048 * +slave slave redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:33.052 * Sentinel new configuration saved on disk
1:X 19:39:34.803 * +sentinel-invalid-addr sentinel 1d4285d6f0c8027fb97375d2ce4ae7168a8b74ca redis-sentinel-0.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:34.803 * +sentinel sentinel d68b6617a21c51bd7ef03cf857c474aef544679c redis-sentinel-0.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 19:39:34.807 * Sentinel new configuration saved on disk

🧮 After failover

Redis-0 IP changed from 10.42.0.106 to 10.42.0.111.

💻 Terminal

$ kubectl get pods -o wide
NAME               READY   STATUS    RESTARTS   AGE     IP            NODE  
redis-sentinel-0   2/2     Running   0          18m     10.42.0.111   rocky9 IP:106->111
redis-sentinel-1   2/2     Running   0          3h53m   10.42.0.108   rocky9
redis-sentinel-2   2/2     Running   0          3h52m   10.42.0.110   rocky9

💾 Check redis.conf

Check replicaof

📄 redis.conf

Redis-0(111): replicaof redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
Redis-1(108): no replicaof ➡ master
Redis-2(110): replicaof redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504

💾 Check sentinel.conf

Redis-1 is master; Redis-0 and Redis-2 are replicas.

📄 sentinel.conf

💻 Sentinel-0
sentinel monitor mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 2
# Generated by CONFIG REWRITE
sentinel known-replica mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504

💻 Sentinel-1
sentinel monitor mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 2
# Generated by CONFIG REWRITE
sentinel known-replica mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504

💻 Sentinel-2
sentinel monitor mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 2
# Generated by CONFIG REWRITE
sentinel known-replica mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504

🖧 Redis-Sentinel Pod diagram (after failover)

┌─────────────────┐ │ Redis-0(111) │ │ 🟡Replica-1 │ │ Sentinel-0 │ └───────┬─────────┘ ▲ ┌───────────┴──────────┐ │ ▼ ┌─────────────────┐ ┌─────────────────┐ │ Redis-1(108) │ │ Redis-2(110) │ │ 🟢Master │ │ 🟡Replica-2 │ │ Sentinel-1 │ │ Sentinel-2 │ └─────────────────┘ └─────────────────┘

2. Failback test

Take current master Redis-1 down and confirm Redis-0 is promoted back to master
📚 “Failback test” here means making Redis-0 master again.
(Named to distinguish it from the failover test.)

💻 Redis-1 💥 down (delete)

💻 Terminal

$ kubectl delete pod redis-sentinel-1
  pod "redis-sentinel-1" deleted

💻 Redis-0 LOG: Replica -> Master

① 15s: Redis-1 connection lost (+1s)
② 18s: Sentinel command, Replica -> Master (+4s)

📄 redis.log

1:S 20:18:15.641 *📍Redis-1 connection lost: Connection with master lost. 
1:M 20:18:18.950 *📍MASTER MODE enabled (user request from 'name=sentinel-...')

Full Redis-0 log ▼

📄 redis.log

1:S 20:18:15.641 *📍Redis-1 connection lost: Connection with master lost. 
1:S 20:18:15.641 * Caching the disconnected master state.
1:S 20:18:15.641 * Reconnecting to MASTER redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504
1:S 20:18:15.643 # Unable to connect to MASTER: Success
1:S 20:18:16.245 * Connecting to MASTER redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504
1:S 20:18:16.246 # Unable to connect to MASTER: Success
1:S 20:18:17.250 * Connecting to MASTER redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504
1:S 20:18:17.251 # Unable to connect to MASTER: Success
1:S 20:18:18.256 * Connecting to MASTER redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504
1:S 20:18:18.257 # Unable to connect to MASTER: Success
1:M 20:18:18.950 * Discarding previously cached master state.
1:M 20:18:18.950 * Setting secondary replication ID to 4f961426ec4fc18f4d78cb18cdc1bfab7ad056cc, valid up to offset: 436944144. New replication ID is 1e5ffab2805243301ffa77113408118d3aefe24e
1:M 20:18:18.950 *📍Sentinel command, Replica -> Master: MASTER MODE enabled (user request from 'id=18 addr=10.42.0.110:49152 laddr=10.42.0.111:18504 fd=18 name=sentinel-4d41e0ad-cmd age=2315 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 watch=0 qbuf=202 qbuf-free=20272 argv-mem=65 multi-mem=65 rbs=2048 rbp=1024 obl=45 oll=0 omem=0 omem-shared=0 omem-unshared=0 tot-mem=24570 events=r cmd=exec user=default redir=-1 resp=2 lib-name= lib-ver= io-thread=0 tot-net-in=286172 tot-net-out=1801356 tot-cmds=3608 read-events=3498 avg-pipeline-len-sum=3617 avg-pipeline-len-cnt=3502')
1:M 20:18:18.954 * CONFIG REWRITE executed with success.
1:M 20:18:19.826 * Replica redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 asks for synchronization
1:M 20:18:19.826 * Partial resynchronization request from redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 accepted. Sending 465 bytes of backlog starting from offset 436944144.
1:M 20:18:29.889 * Replica redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 asks for synchronization
1:M 20:18:29.889 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '60c1cf187a0973dca3ea0e2dd3ed5ea2e27981ec', my replication IDs are '1e5ffab2805243301ffa77113408118d3aefe24e' and '4f961426ec4fc18f4d78cb18cdc1bfab7ad056cc')
1:M 20:18:29.889 * Replica redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 20:18:29.890 * Replica redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 asks for synchronization
1:M 20:18:29.890 * Full resync requested by replica redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 (rdb-channel)
1:M 20:18:29.890 * Delay next BGSAVE for diskless SYNC
1:M 20:18:32.318 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 20:18:32.318 * Starting to deliver RDB and replication stream to replica: redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504
1:M 20:18:32.326 * Background RDB transfer started by pid 26 to replica socket
26:C 20:18:35.443 * BGSAVE done, 2955238 keys saved, 0 keys skipped, 354628790 bytes written.
26:C 20:18:35.451 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
1:M 20:18:35.539 * Background RDB transfer terminated with success
1:M 20:18:35.539 * Synchronization with replica redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 succeeded
1:M 20:18:35.539 * Streamed RDB transfer with replica redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 20:18:35.539 * RDB transfer completed, rdb only replica (redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504) should be disconnected asap
1:M 20:18:35.539 * Connection with replica (rdbchannel) redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 lost.

💻 Redis-1 LOG: Master -> Replica

① 14s: got kill signal, down (+0s)
② 17s: Pod restart (+3s)
③ 29s: Sentinel command, Master -> Replica (+15s)

📄 redis.log

1:M 20:18:14.587 *📍got kill signal, down: User requested shutdown...
1:C 20:18:17.441 *📍restart after 2s oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:S 20:18:29.884 *📍Sentinel signal, Master -> Replica: Before turning into a replica...

Full Redis-1 log ▼

📄 redis.log

1:signal-handler (1786447094) Received SIGTERM scheduling shutdown... 
1:M 20:18:14.587 *📍got kill signal, down: User requested shutdown...
1:M 20:18:14.587 * Waiting for replicas before shutting down.
1:M 20:18:15.592 * 2 of 2 replicas are in sync when shutting down.
1:M 20:18:15.592 * Calling fsync() on the AOF file.
1:M 20:18:15.596 * Removing the pid file.
1:M 20:18:15.597 # Redis is now ready to exit, bye bye...                 
1:C 20:18:17.441 *📍restart after 2s oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 20:18:17.441 * Redis version=8.8.1, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 20:18:17.441 * Configuration loaded                 
1:M 20:18:17.441 * monotonic clock: POSIX clock_gettime
1:M 20:18:17.443 *➡starts as master and loads AOF: Server initialized
1:M 20:18:17.444 * Reading RDB base file on AOF loading...
1:M 20:18:17.444 * Loading RDB produced by version 8.8.1
1:M 20:18:17.444 * RDB age 15203 seconds
1:M 20:18:17.444 * RDB memory usage when created 0.87 Mb
1:M 20:18:17.444 * RDB is base AOF
1:M 20:18:17.444 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 20:18:17.444 * DB loaded from base file aof.2.base.rdb: 0.001 seconds
1:M 20:18:22.001 * DB loaded from incr file aof.2.incr.aof: 4.557 seconds
1:M 20:18:22.001 * DB loaded from append only file: 4.558 seconds
1:M 20:18:22.006 * Opening AOF incr file aof.2.incr.aof on server start
1:M 20:18:22.006 * Ready to accept connections tcp                 
1:S 20:18:29.884 *📍Sentinel signal, Master -> Replica: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 20:18:29.884 * Connecting to MASTER redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504
1:S 20:18:29.885 * MASTER <-> REPLICA sync started
1:S 20:18:29.885 * REPLICAOF redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 enabled (user request from 'id=9 addr=10.42.0.111:48286 laddr=10.42.0.112:18504 fd=13 name=sentinel-d68b6617-cmd age=10 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 watch=0 qbuf=254 qbuf-free=20220 argv-mem=130 multi-mem=130 rbs=1024 rbp=1024 obl=45 oll=0 omem=0 omem-shared=0 omem-unshared=0 tot-mem=23676 events=r cmd=exec user=default redir=-1 resp=2 lib-name= lib-ver= io-thread=0 tot-net-in=1405 tot-net-out=13903 tot-cmds=16 read-events=16 avg-pipeline-len-sum=24 avg-pipeline-len-cnt=19')
1:S 20:18:29.889 * CONFIG REWRITE executed with success.
1:S 20:18:29.889 * Non blocking connect for SYNC fired the event.
1:S 20:18:29.889 * Master replied to PING, replication can continue...
1:S 20:18:29.889 * Trying a partial resynchronization (request 60c1cf187a0973dca3ea0e2dd3ed5ea2e27981ec:436944144).
1:S 20:18:29.889 * PSYNC is not possible, initialize RDB channel.
1:S 20:18:32.318 * Starting to receive RDB and replication stream in parallel.
1:S 20:18:32.336 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 20:18:35.449 * Discarding previously cached master state.
1:S 20:18:35.449 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 20:18:35.455 * MASTER <-> REPLICA sync: Flushing old data
1:S 20:18:36.476 * Loading RDB produced by version 8.8.1
1:S 20:18:36.476 * RDB age 4 seconds
1:S 20:18:36.476 * RDB memory usage when created 474.91 Mb
1:S 20:18:39.127 * Done loading RDB, keys loaded: 2955238, keys expired: 0.
1:S 20:18:39.127 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 20:18:39.127 * MASTER <-> REPLICA sync: Finished with success
1:S 20:18:39.127 * MASTER <-> REPLICA sync: Starting AOF after a successful sync
1:S 20:18:39.127 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 20:18:39.137 * Background append only file rewriting started by pid 13
1:S 20:18:39.137 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (2012 bytes).
1:S 20:18:39.137 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (2012 bytes in total)
13:C 20:18:42.416 * BGSAVE done, 2955238 keys saved, 0 keys skipped, 354628657 bytes written.
13:C 20:18:42.422 * Successfully created the temporary AOF base file temp-rewriteaof-bg-13.aof
13:C 20:18:42.430 * Fork CoW for AOF rewrite: current 1 MB, peak 1 MB, average 1 MB
1:S 20:18:42.455 * Background AOF rewrite terminated with success
1:S 20:18:42.455 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-13.aof into aof.3.base.rdb
1:S 20:18:42.455 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.3.incr.aof
1:S 20:18:42.459 * Removing the history file aof.2.incr.aof in the background
1:S 20:18:42.459 * Removing the history file aof.2.base.rdb in the background
1:S 20:18:42.463 * Background AOF rewrite finished successfully

💻 Redis-2 LOG

① 15s: lost Redis-1 (+1s)
② 19s: Sentinel: Redis-0 is new master; connect (+5s)

📄 redis.log

1:S 20:18:15.641 *📍lost Redis-1: Connection with master lost.
1:S 20:18:19.820 *📍Sentinel: Redis-0 is new master; connect: Connecting to MASTER redis-sentinel-0

Full Redis-2 log ▼

📄 redis.log

1:S 20:18:15.641 *📍lost Redis-1: Connection with master lost.
1:S 20:18:15.641 * Caching the disconnected master state.
1:S 20:18:15.641 * Reconnecting to MASTER redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504
1:S 20:18:15.643 # Unable to connect to MASTER: Success
1:S 20:18:16.526 * Connecting to MASTER redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504
1:S 20:18:16.527 # Unable to connect to MASTER: Success
1:S 20:18:17.530 * Connecting to MASTER redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504
1:S 20:18:17.532 # Unable to connect to MASTER: Success
1:S 20:18:18.536 * Connecting to MASTER redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504
1:S 20:18:18.537 # Unable to connect to MASTER: Success
1:S 20:18:19.542 * Connecting to MASTER redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504
1:S 20:18:19.543 # Unable to connect to MASTER: Success
1:S 20:18:19.820 *📍Sentinel: Redis-0 is new master; connect: Connecting to MASTER redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504
1:S 20:18:19.821 * MASTER <-> REPLICA sync started
1:S 20:18:19.821 *➡set Redis-0 as master: REPLICAOF redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 enabled (user request from 'id=17 addr=10.42.0.110:34472 laddr=10.42.0.110:18504 fd=15 name=sentinel-4d41e0ad-cmd age=2326 idle=0 flags=x db=0 sub=0 psub=0 ssub=0 multi=4 watch=0 qbuf=475 qbuf-free=19999 argv-mem=130 multi-mem=130 rbs=2048 rbp=2048 obl=45 oll=0 omem=0 omem-shared=0 omem-unshared=0 tot-mem=24700 events=r cmd=exec user=default redir=-1 resp=2 lib-name= lib-ver= io-thread=0 tot-net-in=287385 tot-net-out=1752684 tot-cmds=3619 read-events=3518 avg-pipeline-len-sum=3625 avg-pipeline-len-cnt=3521')
1:S 20:18:19.825 * CONFIG REWRITE executed with success.
1:S 20:18:19.826 * Non blocking connect for SYNC fired the event.
1:S 20:18:19.826 * Master replied to PING, replication can continue...
1:S 20:18:19.826 * Trying a partial resynchronization (request 4f961426ec4fc18f4d78cb18cdc1bfab7ad056cc:436944144).
1:S 20:18:19.826 * Successful partial resynchronization with master.
1:S 20:18:19.826 * Master replication ID changed to 1e5ffab2805243301ffa77113408118d3aefe24e
1:S 20:18:19.826 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

💻 Sentinel-0 LOG

① 14s: Redis-1 connection lost (+0s)
② 17s: +sdown on Redis-1 (+3s) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 19s: +odown on Redis-1 (+5s) -> after majority of Sentinels agree

📄 redis.log

1:X 20:18:14.644 #📍Redis-1 connection lost: Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service'
1:X 20:18:17.695 #📍detect Redis-1 down: +sdown sentinel 1f227bb1efbb0a6e25594ffaf28bbc3110b6b2cd redis-sentinel-1.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:19.469 #📍confirm Redis-1 down: +odown master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 #quorum 2/2

Full Sentinel-0 log ▼

📄 redis.log

1:X 20:18:14.644 #📍Redis-1 connection lost: Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service'
1:X 20:18:15.652 # Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local'
1:X 20:18:15.653 # Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service'
1:X 20:18:16.715 # Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local'
1:X 20:18:16.716 # Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service'
1:X 20:18:17.695 #📍detect Redis-1 down: +sdown sentinel 1f227bb1efbb0a6e25594ffaf28bbc3110b6b2cd redis-sentinel-1.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.390 # +sdown master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.666 # Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local'
1:X 20:18:18.742 # Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local'
1:X 20:18:18.746 * Sentinel new configuration saved on disk
1:X 20:18:18.746 # +new-epoch 2
1:X 20:18:18.750 * Sentinel new configuration saved on disk
1:X 20:18:18.750 # +vote-for-leader 4d41e0ad06a5e2b48c87b09cf7136473569efe40 2
1:X 20:18:19.469 #📍confirm Redis-1 down: +odown master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 20:18:19.469 * Next failover delay: I will not start a failover before Tue Aug 11 20:20:19 2026
1:X 20:18:19.820 # +config-update-from sentinel 4d41e0ad06a5e2b48c87b09cf7136473569efe40 redis-sentinel-2.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:19.820 # +switch-master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:19.821 * +slave slave redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:19.821 * +slave slave redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:19.826 * Sentinel new configuration saved on disk
1:X 20:18:21.596 * +sentinel-invalid-addr sentinel 1f227bb1efbb0a6e25594ffaf28bbc3110b6b2cd redis-sentinel-1.redis-sentinel-service 28504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:21.597 * +sentinel sentinel ff3fa364394d14803c97d2915d96aaca7676c866 redis-sentinel-1.redis-sentinel-service 28504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:21.601 * Sentinel new configuration saved on disk
1:X 20:18:29.884 *➡tell Redis-1 to become replica: +convert-to-slave slave redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504

💻 Sentinel-1 LOG

① 14s: got kill signal, down (+0s)
② 17s: Pod restart (+3s)

📄 redis.log

1:X 20:18:14.623 *📍got kill signal, down: User requested shutdown...
1:X 20:18:17.529 *📍Pod restart: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

Full Sentinel-1 log ▼

📄 redis.log

1:signal-handler (1786447094) Received SIGTERM scheduling shutdown...
1:X 20:18:14.623 *📍User requested shutdown...
1:X 20:18:14.623 * Removing the pid file.
1:X 20:18:14.623 # Sentinel is now ready to exit, bye bye...
1:X 20:18:17.529 *📍oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 20:18:17.529 * Redis version=8.8.1, bits=64, commit=00000000, modified=1, pid=1, just started
1:X 20:18:17.529 * Configuration loaded
1:X 20:18:17.529 * monotonic clock: POSIX clock_gettime
1:X 20:18:17.530 * Running mode=sentinel, port=28504.
1:X 20:18:17.535 * Sentinel new configuration saved on disk
1:X 20:18:17.535 * Sentinel ID is ff3fa364394d14803c97d2915d96aaca7676c866
1:X 20:18:17.535 # +monitor master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 quorum 2
1:X 20:18:18.452 * +sentinel sentinel 4d41e0ad06a5e2b48c87b09cf7136473569efe40 redis-sentinel-2.redis-sentinel-service 28504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.456 * Sentinel new configuration saved on disk
1:X 20:18:18.460 * Sentinel new configuration saved on disk
1:X 20:18:18.460 # +new-epoch 1
1:X 20:18:18.460 # +config-update-from sentinel 4d41e0ad06a5e2b48c87b09cf7136473569efe40 redis-sentinel-2.redis-sentinel-service 28504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.460 # +switch-master mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.461 * +slave slave redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.465 * Sentinel new configuration saved on disk
1:X 20:18:19.825 * Sentinel new configuration saved on disk
1:X 20:18:19.825 # +new-epoch 2
1:X 20:18:19.826 # +config-update-from sentinel 4d41e0ad06a5e2b48c87b09cf7136473569efe40 redis-sentinel-2.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:19.826 # +switch-master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:19.826 * +slave slave redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:19.830 * Sentinel new configuration saved on disk
1:X 20:18:21.867 * +sentinel sentinel d68b6617a21c51bd7ef03cf857c474aef544679c redis-sentinel-0.redis-sentinel-service 28504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:21.872 * Sentinel new configuration saved on disk
1:X 20:18:27.574 * +slave slave redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:27.578 * Sentinel new configuration saved on disk

💻 Sentinel-2 LOG: elected leader

① 14s: lost Redis-1 (+0s)
② 17s: detect Redis-1 down (+3s)
③ 18.7s: confirm Redis-1 down (+4s)
④ 18.8s: Sentinels elect Sentinel-2 as leader (+elected-leader) (+4s)
④ 18.8s: promote redis-0 (+failover-state-send-slaveof-noone) (+4s)

📄 redis.log

1:X 20:18:14.663 #📍lost Redis-1: Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service'
1:X 20:18:17.696 #📍detect Redis-1 down: +sdown sentinel 1f227bb1efbb0a6e25594ffaf28bbc3110b6b2cd redis-sentinel-1.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.736 #📍confirm Redis-1 down: +odown master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 20:18:18.832 #📍elected leader: +elected-leader master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.887 *📍promote Redis-0 to master: +failover-state-send-slaveof-noone slave redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504

Full Sentinel-2 log ▼

📄 redis.log

1:X 20:18:14.663 #📍lost Redis-1: Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service'
1:X 20:18:15.655 # Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local'
1:X 20:18:15.711 # Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service'
1:X 20:18:16.671 # Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local'
1:X 20:18:16.743 # Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service'
1:X 20:18:17.696 #📍detect Redis-1 down: +sdown sentinel 1f227bb1efbb0a6e25594ffaf28bbc3110b6b2cd redis-sentinel-1.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.392 # Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local'
1:X 20:18:18.665 # +sdown master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.736 #📍confirm Redis-1 down: +odown master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 20:18:18.736 # +new-epoch 2
1:X 20:18:18.736 # +try-failover master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.741 * Sentinel new configuration saved on disk
1:X 20:18:18.741 # +vote-for-leader 4d41e0ad06a5e2b48c87b09cf7136473569efe40 2
1:X 20:18:18.750 * d68b6617a21c51bd7ef03cf857c474aef544679c voted for 4d41e0ad06a5e2b48c87b09cf7136473569efe40 2
1:X 20:18:18.832 #📍elected leader: +elected-leader master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.832 # +failover-state-select-slave master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.887 # +selected-slave slave redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.887 *📍promote Redis-0 to master: +failover-state-send-slaveof-noone slave redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:18.949 * +failover-state-wait-promotion slave redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:19.394 # Failed to resolve hostname 'redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local'
1:X 20:18:19.773 * Sentinel new configuration saved on disk
1:X 20:18:19.773 # +promoted-slave slave redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:19.773 # +failover-state-reconf-slaves master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:19.820 *➡notify Redis-2 that Redis-0 is master: +slave-reconf-sent slave redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:20.495 * +sentinel-invalid-addr sentinel 1f227bb1efbb0a6e25594ffaf28bbc3110b6b2cd redis-sentinel-1.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:20.495 * +sentinel sentinel ff3fa364394d14803c97d2915d96aaca7676c866 redis-sentinel-1.redis-sentinel-service 28504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:20.499 * Sentinel new configuration saved on disk
1:X 20:18:20.823 * +slave-reconf-inprog slave redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:20.823 * +slave-reconf-done slave redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:20.876 # -odown master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:20.876 # +failover-end master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:20.876 # +switch-master mymaster redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:20.876 * +slave slave redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-2.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:20.877 * +slave slave redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local:18504 redis-sentinel-1.redis-sentinel-service.default.svc.cluster.local 18504 @ mymaster redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
1:X 20:18:20.881 * Sentinel new configuration saved on disk

🧮 After failback

Redis-1 IP changed from 10.42.0.108 to 10.42.0.112.

💻 Terminal

$ kubectl get pods -o wide
NAME               READY   STATUS    RESTARTS   AGE     IP            NODE  
redis-sentinel-0   2/2     Running   0          39m     10.42.0.111   rocky9 IP:106->111
redis-sentinel-1   2/2     Running   0          13s     10.42.0.112   rocky9 IP:108->112
redis-sentinel-2   2/2     Running   0          4h13m   10.42.0.110   rocky9

💾 Check redis.conf

📄 redis.conf

Redis-0(111): no replicaof	-> Master
Redis-1(112): replicaof redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504
Redis-2(110): replicaof redis-sentinel-0.redis-sentinel-service.default.svc.cluster.local 18504

💾 Check sentinel.conf

Redis-0 is master; Redis-1 and Redis-2 are replicas.

📄 sentinel.conf

💻 Sentinel-0
sentinel monitor mymaster redis-0.redis-service.default.svc.cluster.local 18504 2
# Generated by CONFIG REWRITE
sentinel known-replica mymaster redis-1.redis-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-2.redis-service.default.svc.cluster.local 18504

💻 Sentinel-1
sentinel monitor mymaster redis-0.redis-service.default.svc.cluster.local 18504 2
# Generated by CONFIG REWRITE
sentinel known-replica mymaster redis-1.redis-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-2.redis-service.default.svc.cluster.local 18504

💻 Sentinel-2
sentinel monitor mymaster redis-0.redis-service.default.svc.cluster.local 18504 2
# Generated by CONFIG REWRITE
sentinel known-replica mymaster redis-1.redis-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-2.redis-service.default.svc.cluster.local 18504

🖧 Redis-Sentinel Pod diagram (after failback)

┌─────────────────┐ │ Redis-0(111) │ │ 🟢Master │ │ Sentinel-0 │ └───────┬─────────┘ ┌───────────┴──────────┐ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ │ Redis-1(112) │ │ Redis-2(110) │ │ 🟡Replica-1 │ │ 🟡Replica-2 │ │ Sentinel-1 │ │ Sentinel-2 │ └─────────────────┘ └─────────────────┘

<< K8s Sidecar Pattern Setup ②	K8s Redis-Standalone ① >>

Email 답글이 올라오면 이메일로 알려드리겠습니다.