Redis-Sentinel on Kubernetes Setup ②

Setup (create Pods)

1. Apply YAML files in order

1) Create Config

💻 Terminal

$ kubectl apply -f redis-sentinel-config.yaml	➡ apply (create)
  configmap/redis-sentinel-config created	➡ newly created
  configmap/redis-sentinel-config configured ➡ when there are changes
  configmap/redis-sentinel-config unchanged ➡ when there are no changes
$ kubectl get configmap redis-sentinel-config	➡ verify
  NAME                        DATA  AGE
  redis-sentinel-config   3      21s
  * 'DATA 3' means the three files redis.conf, users.acl, and sentinel.conf.

2) Create Service

💻 Terminal

$ kubectl apply -f redis-sentinel-service.yaml	➡ apply (create)
  service/redis-sentinel-service created
$ kubectl get service redis-service	➡ verify
  NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
  redis-service   ClusterIP   None         <none>        18504/TCP   39s
$ kubectl get service sentinel-service	➡ verify
  NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
  sentinel-service   ClusterIP   None         <none>        28504/TCP   64s

3) Create StatefulSet

💻 Terminal

$ kubectl apply -f redis-statefulset.yaml	➡ apply (create)
  statefulset.apps/redis-statefulset created
$ kubectl get pods -o wide	➡ confirm Redis Pods, then apply Sentinel
$ kubectl apply -f sentinel-statefulset.yaml	➡ apply (create)
  statefulset.apps/sentinel created
$ kubectl get pods -o wide	➡ list Pods 🧮
  NAME              READY   STATUS    RESTARTS   AGE     IP            NODE  
  redis-0             1/1     Running   0          62s     10.42.0.92     rocky9
  redis-1             1/1     Running   0          55s     10.42.0.94     rocky9
  redis-2             1/1     Running   0          51s     10.42.0.96     rocky9
  sentinel-0        1/1     Running   0          37s     10.42.0.98     rocky9
  sentinel-1        1/1     Running   0          30s     10.42.0.100   rocky9
  sentinel-2        1/1     Running   0          24s     10.42.0.102   rocky9

Three Redis Pods and three Sentinel Pods are running, so Redis-Sentinel is in place.

2. Inspect Redis-Sentinel

1) Check roles (master/replica): redis-role.sh

Script that runs Redis ROLE to check master/replica roles

📄 redis-role.sh

#!/bin/bash
# settings
PORT="18504"
PASSWORD="redisgate"
kubectl exec -i redis-0 -- redis-cli -p ${PORT} -a ${PASSWORD} role
kubectl exec -i redis-1 -- redis-cli -p ${PORT} -a ${PASSWORD} role
kubectl exec -i redis-2 -- redis-cli -p ${PORT} -a ${PASSWORD} role

Run (inspect) result: edited for readability.

💻 Terminal

$ redis-role.sh 
Redis-0
  master  ➡ Redis-0 (itself) is master and has the two replicas below.
    redis-1.redis-service.default.svc.cluster.local:18504
    redis-2.redis-service.default.svc.cluster.local:18504
Redis-1: slave:redis-0.redis-service.default.svc.cluster.local:18504 ➡ Redis-1 is replica, master 'redis-0'
Redis-2: slave:redis-0.redis-service.default.svc.cluster.local:18504 ➡ Redis-2 is replica, master 'redis-0'

2) Inspect Redis monitored by Sentinel: sentinel-info.sh

Script that runs Sentinel 'INFO SENTINEL'

📄 sentinel-info.sh

#!/bin/bash
# settings
PORT="28504"
PASSWORD="redisgate"
kubectl exec -i sentinel-0 -- redis-cli -p ${PORT} -a ${PASSWORD} info sentinel
kubectl exec -i sentinel-1 -- redis-cli -p ${PORT} -a ${PASSWORD} info sentinel
kubectl exec -i sentinel-2 -- redis-cli -p ${PORT} -a ${PASSWORD} info sentinel
#kubectl exec -i sentinel-0 -- redis-cli -p ${PORT} -a ${PASSWORD} role -> inspect role
#kubectl exec -i sentinel-0 -- redis-cli -p ${PORT} -a ${PASSWORD} sentinel masters -> inspect monitor

Run (inspect) result: edited for readability.
The master Sentinel monitors is 'redis-0', there are 2 replicas (slaves=2), and 3 Sentinels in total (sentinels=3).

💻 Terminal

$ sentinel-info.sh 
Sentinel-0: master0:name=mymaster,status=ok,address=redis-0.redis-service...:18504,slaves=2,sentinels=3  
Sentinel-1: master0:name=mymaster,status=ok,address=redis-0.redis-service...:18504,slaves=2,sentinels=3
Sentinel-2: master0:name=mymaster,status=ok,address=redis-0.redis-service...:18504,slaves=2,sentinels=3

3. 🖧 Redis-Sentinel Pod diagram

┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │Sentinel-0(98) │ │Sentinel-1(100)│ │Sentinel-2(102)│ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ └───────────────────┼───────────────────┘ ▼ ┌───────────────┐ │ Redis-0(92) │ │ 🟢Master │ └──────┬────────┘ ┌──────────┴───────────┐ ▼ ▼ ┌───────────────┐ ┌───────────────┐ │ Redis-1(94) │ │ Redis-2(96) │ │ 🟡Replica-1 │ │ 🟡Replica-2 │ └───────────────┘ └───────────────┘

4. 💽 Load data

1) Load data

If you take Redis-0 down (delete) for a failover test, Kubernetes recreates redis-0 within about 2 seconds. down-after-milliseconds is 3 seconds, so the downtime of Redis-0 is not detected. So we load about 3 million keys so a restarted Redis takes more than 5 seconds to come back (data load takes about 5 seconds).

💻 Terminal

$ kubectl exec -it redis-0 -- redis-benchmark -p 18504 -a password -t set -d 100 -r 100000000 -n 3000000   
  ====== SET ======
    3000000 requests completed in 30.89 seconds
  Summary:
    throughput summary: 97115.66 requests per second
    latency summary (msec):
            avg       min       p50       p95       p99       max
          0.428     0.128     0.407     0.639     0.831    16.159

2) Verify loaded data

① Key count: 2,955,803 ➡ redis-benchmark generates keys randomly, so duplicates mean we fall a bit short of 3 million.

💻 Terminal

$ kubectl exec -it redis-0 -- redis-cli -p 18504 -a password dbsize

② AOF file size: aof.1.incr.aof 412M
③ Pod memory: Redis 617MB. Sentinel holds no data, so its size does not grow.

💻 Terminal

$ kubectl top pod
  NAME              CPU(cores)   MEMORY(bytes)   
  redis-0           2m           617Mi
  redis-1           2m           617Mi
  redis-2           3m           617Mi
  sentinel-0        3m           8Mi
  sentinel-1        4m           8Mi
  sentinel-2        4m           8Mi

➡ Pods are up and data is loaded. Next: failover test.

<< K8s Redis-Sentinel Prep ①	K8s Redis-Sentinel Test ③ >>

Email 답글이 올라오면 이메일로 알려드리겠습니다.