Redis-Cluster on Kubernetes: Hands-on Guide ②

Setup (create Pods)

1. Apply YAML files in order (create)

1) Create Config

💻 Terminal

$ kubectl apply -f redis-cluster-config.yaml	➡ apply (create)
  configmap/redis-cluster-config created	➡ newly created
  configmap/redis-cluster-config configured	➡ when there are changes
  configmap/redis-cluster-config unchanged	➡ when there are no changes
$ kubectl get configmap redis-cluster-config	➡ check
  NAME                   DATA   AGE
  redis-cluster-config   2      8m5s
* 'DATA 2' means two files: redis.conf and users.acl.

2) Create Service

💻 Terminal

$ kubectl apply -f redis-cluster-service.yaml	➡ apply (create)
  service/redis-cluster-service created
$ kubectl get service redis-cluster-service	➡ check
  NAME                  TYPE      CLUSTER-IP EXTERNAL-IP PORT(S)          AGE
  redis-cluster-service ClusterIP None       <none>   18504/TCP,28504/TCP 1m

3) Create StatefulSet

💻 Terminal

$ kubectl apply -f redis-cluster-statefulset.yaml	➡ apply (create)
  statefulset.apps/redis-cluster created
$ kubectl get pods -o wide	➡ list Pods 🧮
  NAME              READY   STATUS    RESTARTS   AGE   IP            NODE  
  redis-cluster-0   1/1     Running   0          33s   10.42.0.231   rocky9
  redis-cluster-1   1/1     Running   0          27s   10.42.0.233   rocky9
  redis-cluster-2   1/1     Running   0          23s   10.42.0.235   rocky9
  redis-cluster-3   1/1     Running   0          18s   10.42.0.237   rocky9
  redis-cluster-4   1/1     Running   0          14s   10.42.0.239   rocky9
  redis-cluster-5   1/1     Running   0          10s   10.42.0.241   rocky9

At this point six standalone Redis instances are running. Redis-Cluster is not formed yet.

2. Form Redis-Cluster

1) Write and run the Redis-Cluster create script

Running 'cluster_create.sh' forms Redis-Cluster with 3 Masters and 3 Replicas.

💻 Terminal

$ cat -n cluster_create.sh 
#!/bin/bash
# =========================================================================
# Redis Cluster create script (3 Master / 3 Replica)
# =========================================================================
# Settings
PORT="18504"
PASSWORD="password"
SERVICE_NAME="redis-cluster-service"
NAMESPACE="default"
echo "====================================================================="
echo "🚀 Starting Redis Cluster setup..."
echo "====================================================================="
# Add --cluster-yes to auto-answer the yes prompt
# Use -i instead of -it for automation scripts (avoid TTY warnings)
kubectl exec -i redis-cluster-0 -- redis-cli -p ${PORT} -a ${PASSWORD} --cluster create \
  redis-cluster-0.${SERVICE_NAME}.${NAMESPACE}.svc.cluster.local:${PORT} \
  redis-cluster-1.${SERVICE_NAME}.${NAMESPACE}.svc.cluster.local:${PORT} \
  redis-cluster-2.${SERVICE_NAME}.${NAMESPACE}.svc.cluster.local:${PORT} \
  redis-cluster-3.${SERVICE_NAME}.${NAMESPACE}.svc.cluster.local:${PORT} \
  redis-cluster-4.${SERVICE_NAME}.${NAMESPACE}.svc.cluster.local:${PORT} \
  redis-cluster-5.${SERVICE_NAME}.${NAMESPACE}.svc.cluster.local:${PORT} \
  --cluster-replicas 1 \
  --cluster-yes

🧑‍💻 Key points

On a VM you write IP:Port; on Kubernetes you write the following.
redis-cluster-0.${SERVICE_NAME}.${NAMESPACE}.svc.cluster.local:${PORT}
List redis-cluster-0 through 5.
--cluster-replicas 1 # Set Master 1 -> Replica 1.
--cluster-yes # This option auto-answers 'yes' to the prompt below.
Can I set the above configuration? (type 'yes' to accept): yes
Without this option you must type 'yes'.

🖧 Redis Cluster create process

📍 Slots were assigned evenly to 3 Masters.
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
📍 Replica -> Master relationship
Adding replica redis-cluster-4 to redis-cluster-0
Adding replica redis-cluster-5 to redis-cluster-1
Adding replica redis-cluster-3 to redis-cluster-2
📍 OK success! All slots are covered.
[OK] All 16384 slots covered.

View full output ▼

💻 Terminal

======================================================================
🚀 Starting Redis Cluster setup...
======================================================================
>>> Performing hash slots allocation on 6 nodes...
📍 Slots were assigned evenly to 3 Masters.
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
📍 Replica -> Master relationship
Adding replica redis-cluster-4.redis-cluster-service.default.svc.cluster.local:18504 to redis-cluster-0.redis-cluster-service.default.svc.cluster.local:18504
Adding replica redis-cluster-5.redis-cluster-service.default.svc.cluster.local:18504 to redis-cluster-1.redis-cluster-service.default.svc.cluster.local:18504
Adding replica redis-cluster-3.redis-cluster-service.default.svc.cluster.local:18504 to redis-cluster-2.redis-cluster-service.default.svc.cluster.local:18504
M: 66390566e7f48211e8bed3a6a0e49155cbe8b54b redis-cluster-0.redis-cluster-service.default.svc.cluster.local:18504
   slots:[0-5460] (5461 slots) master
M: 3b690009fb9a90ff6b43765363c4c0b81b63d103 redis-cluster-1.redis-cluster-service.default.svc.cluster.local:18504
   slots:[5461-10922] (5462 slots) master
M: 02069ae6b4f98c31d5f4e70f243bc9a006b68443 redis-cluster-2.redis-cluster-service.default.svc.cluster.local:18504
   slots:[10923-16383] (5461 slots) master
S: 3722046802e006615ac1d976e62e4c94c58af586 redis-cluster-3.redis-cluster-service.default.svc.cluster.local:18504
   replicates 02069ae6b4f98c31d5f4e70f243bc9a006b68443
S: ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 redis-cluster-4.redis-cluster-service.default.svc.cluster.local:18504
   replicates 66390566e7f48211e8bed3a6a0e49155cbe8b54b
S: 541fa7cc95095c4b0390bb8ae9d787aa1201c9e2 redis-cluster-5.redis-cluster-service.default.svc.cluster.local:18504
   replicates 3b690009fb9a90ff6b43765363c4c0b81b63d103
Can I set the above configuration? (type 'yes' to accept): yes 	➡📍 type 'yes' here
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join

>>> Performing Cluster Check (using node redis-cluster-0.redis-cluster-service.default.svc.cluster.local:18504)
M: 66390566e7f48211e8bed3a6a0e49155cbe8b54b redis-cluster-0.redis-cluster-service.default.svc.cluster.local:18504
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 02069ae6b4f98c31d5f4e70f243bc9a006b68443 10.42.0.235:18504
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 3b690009fb9a90ff6b43765363c4c0b81b63d103 10.42.0.233:18504
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 541fa7cc95095c4b0390bb8ae9d787aa1201c9e2 10.42.0.241:18504
   slots: (0 slots) slave
   replicates 3b690009fb9a90ff6b43765363c4c0b81b63d103
S: 3722046802e006615ac1d976e62e4c94c58af586 10.42.0.237:18504
   slots: (0 slots) slave
   replicates 02069ae6b4f98c31d5f4e70f243bc9a006b68443
S: ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 10.42.0.239:18504
   slots: (0 slots) slave
   replicates 66390566e7f48211e8bed3a6a0e49155cbe8b54b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
📍 OK success! All slots are covered.
[OK] All 16384 slots covered.

2) cluster info

💻 Terminal

$ kubectl exec -it redis-cluster-0 -- redis-cli -a redisgate -p 18504 cluster info
cluster_state:ok	➡📍 ok: healthy
cluster_slots_assigned:16384	➡📍 total assigned slots assigned = ok + pfail + fail
cluster_slots_ok:16384  ➡📍 healthy slot count
cluster_slots_pfail:0	➡📍 no Possible Fail (sdown). A node did not respond, so failure is suspected
cluster_slots_fail:0 ➡📍 no Fail (odown). A majority of masters marked the node as failed. Shows that node's slot count.
cluster_known_nodes:6  ➡📍 all nodes (masters + replicas)
cluster_size:3  ➡📍 master node count
... omitted ...

3) cluster nodes

Shown in node order for convenience. Actual output is random.
Same content as nodes.conf.

💻 Terminal

$ kubectl exec -it redis-cluster-0 -- redis-cli -a redisgate -p 18504 cluster nodes
66390566e7f48211e8bed3a6a0e49155cbe8b54b 10.42.0.231:18504@28504,redis-cluster-0.redis-cluster-service.default.svc.cluster.local master - 0 1786164478296 1 connected 0-5460
3b690009fb9a90ff6b43765363c4c0b81b63d103 10.42.0.233:18504@28504,redis-cluster-1.redis-cluster-service.default.svc.cluster.local myself,master - 0 0 2 connected 5461-10922
02069ae6b4f98c31d5f4e70f243bc9a006b68443 10.42.0.235:18504@28504,redis-cluster-2.redis-cluster-service.default.svc.cluster.local master - 0 1786164478000 3 connected 10923-16383
3722046802e006615ac1d976e62e4c94c58af586 10.42.0.237:18504@28504,redis-cluster-3.redis-cluster-service.default.svc.cluster.local slave 02069ae6b4f98c31d5f4e70f243bc9a006b68443 0 1786164478000 3 connected
ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 10.42.0.239:18504@28504,redis-cluster-4.redis-cluster-service.default.svc.cluster.local slave 66390566e7f48211e8bed3a6a0e49155cbe8b54b 0 1786164479099 1 connected
541fa7cc95095c4b0390bb8ae9d787aa1201c9e2 10.42.0.241:18504@28504,redis-cluster-5.redis-cluster-service.default.svc.cluster.local slave 3b690009fb9a90ff6b43765363c4c0b81b63d103 0 1786164478000 2 connected

3. 🖧 Redis-Cluster Pod diagram

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟢Redis-0(231) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Master-1 │ │ Master-2 │ │ Master-3 │ └──────┬─────────┘ └──────┬─────────┘ └──────┬────────┘ │ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ ▼ ▼ ┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟡Redis-4(239) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Replica-2 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

4. 💽 Load data

If you down (delete) the Redis-0 Pod for a failover test, Kubernetes brings redis-0 back within 2 seconds. cluster-node-timeout is 3 seconds, so the down of Redis-0 is not detected. So we load about 10 million keys so restart takes more than 5 seconds.

💻 Terminal

$ kubectl exec -it redis-cluster-1 -- redis-benchmark -p 18504 -a redisgate -t set -d 100 -r 100000000 -n 10000000 --cluster   
Cluster has 3 master nodes:
Master 0: 66390566e7f48211e8bed3a6a0e49155cbe8b54b 10.42.0.231:18504
Master 1: 3b690009fb9a90ff6b43765363c4c0b81b63d103 10.42.0.233:18504
Master 2: 02069ae6b4f98c31d5f4e70f243bc9a006b68443 10.42.0.235:18504

====== SET ======                                                     
  10000000 requests completed in 59.01 seconds
  50 parallel clients
  100 bytes payload
  cluster mode: yes (3 masters)

Summary:
  throughput summary: 169462.81 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        0.243     0.048     0.239     0.367     0.487    12.383

1) Verify loaded data

① Keys per node: dbsize

💻 Terminal

$ kubectl exec -it redis-cluster-0 -- redis-cli -p 18504 -a redisgate dbsize -> 3053053
$ kubectl exec -it redis-cluster-1 -- redis-cli -p 18504 -a redisgate dbsize -> 3241431
$ kubectl exec -it redis-cluster-2 -- redis-cli -p 18504 -a redisgate dbsize -> 3107105

② Check AOF file size

💻 Terminal

[root]$ chmod 755 /var/lib/rancher/k3s/storage -> run as root so the redis user can also read
[storage]$ ls -lh pvc*/aof
pvc-8f6..._default_redis-data-redis-cluster-0/aof:	->  877M aof.1.incr.aof
pvc-cbf..._default_redis-data-redis-cluster-1/aof:	-> 1011M aof.1.incr.aof
pvc-f9d..._default_redis-data-redis-cluster-2/aof:	->  887M aof.1.incr.aof
pvc-37e..._default_redis-data-redis-cluster-3/aof:	->  887M aof.2.incr.aof
pvc-63a..._default_redis-data-redis-cluster-4/aof:	->  877M aof.2.incr.aof
pvc-0ff..._default_redis-data-redis-cluster-5/aof:	-> 1011M aof.2.incr.aof

③ Check memory

💻 Terminal

$ kubectl top pod
NAME              CPU(cores)   MEMORY(bytes)   
redis-cluster-0   2m           672Mi
redis-cluster-1   3m           701Mi
redis-cluster-2   3m           679Mi
redis-cluster-3   2m           678Mi
redis-cluster-4   2m           670Mi
redis-cluster-5   2m           701Mi

➡ Pods are up and data is loaded. Let’s run the failover test.

<< K8s Redis-Cluster Prep ①	K8s Redis-Cluster Test ③ >>

Email 답글이 올라오면 이메일로 알려드리겠습니다.