暴露redis-cluster到k8s集群外部.md

一、容器环境下搭建redis cluster集群本案例中直接使用redis-operator搭建一个规模为3，副本数量为1的redis cluster集群。搭建方式可参考官方文档： https://github.com/OT-CONTAINER-KIT/redis-operator搭建完成后，集群节点列表如下：[root@k8s-master redis]# kubectl get pods -l

ll577644332

2435人浏览 · 2022-05-16 13:36:13

ll577644332 · 2022-05-16 13:36:13 发布

一、容器环境下搭建redis cluster集群

本案例中直接使用redis-operator搭建一个规模为3，副本数量为1的redis cluster集群。
搭建方式可参考官方文档： https://github.com/OT-CONTAINER-KIT/redis-operator

搭建完成后，集群节点列表如下：

[root@k8s-master redis]# kubectl get pods -l redis_setup_type=cluster -o wide
NAME                             READY   STATUS    RESTARTS   AGE    IP               NODE        NOMINATED NODE   READINESS GATES
redis-cluster-alpha-follower-0   2/2     Running   0          146m   172.16.122.79    k8s-node4   <none>           <none>
redis-cluster-alpha-follower-1   2/2     Running   0          145m   172.16.107.217   k8s-node3   <none>           <none>
redis-cluster-alpha-follower-2   2/2     Running   0          145m   172.16.107.235   k8s-node3   <none>           <none>
redis-cluster-alpha-leader-0     2/2     Running   0          146m   172.16.107.239   k8s-node3   <none>           <none>
redis-cluster-alpha-leader-1     2/2     Running   0          145m   172.16.122.81    k8s-node4   <none>           <none>

redis-cluster-alpha-leader-2 2/2 Running 0 145m 172.16.122.87 k8s-node4


- 验证集群状态是否正常

```shell
bash-4.4# redis-cli cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:4
cluster_my_epoch:3
# ...

二、暴露集群中的每个节点到集群Node上

暴露redis cluster到k8s集群外主要面临两个问题：

1、如何让集群中的每个节点能够被集群外访问。

2、如何让cluster nodes命令中列出的节点为集群外部可访问的节点和端口。绝大多数redis cluster客户端都会从cluster nodes的输出中获取所有节点的ip和端口，然后通过这些信息去直连redis节点。默认情况下，该命令输出的是集群内部的pod的ip和端口，所以我们需要将这里输出的信息转换为集群外部可访问的ip和port

# 默认情况下，该命令输出的是集群内部的pod的ip和端口
bash-4.4# redis-cli cluster nodes
b0ea18416b149f3240eec924fd8ad76ce0854579 10.42.47.12:6379@16379 slave f7faeb76e067d22fff480ef29a6b2875ce4622b4 0 1652610385537 2 connected
003b2fea2dbeedfb40d01d5d3140b916d9571b3e 10.42.47.22:6379@16379 slave 0bb47993188bfcba5c75d55d70cd7fd46a3a8c1b 0 1652610385000 1 connected
0b0959a0e935ce5f2ad0e1f1de484c2ac81b73b3 10.42.47.28:6379@16379 master - 0 1652610386841 3 connected 10923-16383
0bb47993188bfcba5c75d55d70cd7fd46a3a8c1b 10.42.47.46:6379@16379 myself,master - 0 1652610385000 1 connected 0-5460
10b421e3385c9aba1b8ceba5e2b1092dafc3f177 10.42.47.48:6379@16379 slave 0b0959a0e935ce5f2ad0e1f1de484c2ac81b73b3 0 1652610385000 3 connected
f7faeb76e067d22fff480ef29a6b2875ce4622b4 10.42.47.45:6379@16379 master - 0 1652610386000 2 connected 5461-10922

1、将所有节点暴露到集群外

针对上述的第一个问题，使用Node port的方式，将集群中的6个节点暴露到集群外部。
在下面的配置文件中，将为redis cluster集群中的每个节点(通过spec.selector关联pod)创建一个NodePort类型的Service, 并将客户端通信端口6379端口和集群总线端口16379暴露到集群的不同端口上。
- 例如这里的节点redis-cluster-alpha-leader-0, 其客户端通信端口6379暴露为k8s集群Node的31000端口，其集群总线端口16379被暴露为k8s集群Node上的32000端口。

之所以还需要暴露集群总线端口，是因为我们后面需要通过修改cluster-announce-ip的方式来让cluster nodes的输出中ip地址可被集群外部访问，而cluster-announce-ip的值一旦被修改，集群中的其他节点会使用该配置指定的地址来进行节点通信，此时如果不将集群总线端口也暴露到k8s集群Node上，将导致节点之间无法通信。

apiVersion: v1
kind: Service
metadata:
  name: redis-cluster-alpha-leader-0
spec:
  selector:
    statefulset.kubernetes.io/pod-name: redis-cluster-alpha-leader-0
  ports:
    - name: client-port
      port: 6379
      protocol: TCP
      targetPort: 6379
      nodePort: 31000 # 当type为NodePort或LoadBalancer时，公开此服务的每个节点上的端口。通常由系统分配。如果指定了一个范围内的值，并且没有被使用，则将使用该值，否则操作将失败。如果不指定，如果该服务需要，将分配一个端口。如果在创建不需要该字段的服务时指定了该字段，则创建将失败。当一个服务更新到不再需要它时，这个字段将被删除(例如，将类型从NodePort改为ClusterIP)
    - name: bus-port
      port: 16379
      protocol: TCP
      targetPort: 16379
      nodePort: 32000
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  name: redis-cluster-alpha-leader-1
spec:
  selector:
    statefulset.kubernetes.io/pod-name: redis-cluster-alpha-leader-1
  ports:
    - name: client-port
      port: 6379
      protocol: TCP
      targetPort: 6379
      nodePort: 31001
    - name: bus-port
      port: 16379
      protocol: TCP
      targetPort: 16379
      nodePort: 32001
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  name: redis-cluster-alpha-leader-2 # 改
spec:
  selector:
    statefulset.kubernetes.io/pod-name: redis-cluster-alpha-leader-2 # 改
  ports:
    - name: client-port
      port: 6379
      protocol: TCP
      targetPort: 6379
      nodePort: 31002 # 改
    - name: bus-port
      port: 16379
      protocol: TCP
      targetPort: 16379
      nodePort: 32002 # 改
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  name: redis-cluster-alpha-follower-0 # 改
spec:
  selector:
    statefulset.kubernetes.io/pod-name: redis-cluster-alpha-follower-0 # 改
  ports:
    - name: client-port
      port: 6379
      protocol: TCP
      targetPort: 6379
      nodePort: 31100 # 改
    - name: bus-port
      port: 16379
      protocol: TCP
      targetPort: 16379
      nodePort: 32100 # 改
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  name: redis-cluster-alpha-follower-1 # 改
spec:
  selector:
    statefulset.kubernetes.io/pod-name: redis-cluster-alpha-follower-1 # 改
  ports:
    - name: client-port
      port: 6379
      protocol: TCP
      targetPort: 6379
      nodePort: 31101 # 改
    - name: bus-port
      port: 16379
      protocol: TCP
      targetPort: 16379
      nodePort: 32101 # 改
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  name: redis-cluster-alpha-follower-2 # 改
spec:
  selector:
    statefulset.kubernetes.io/pod-name: redis-cluster-alpha-follower-2 # 改
  ports:
    - name: client-port
      port: 6379
      protocol: TCP
      targetPort: 6379
      nodePort: 31102 # 改
    - name: bus-port
      port: 16379
      protocol: TCP
      targetPort: 16379
      nodePort: 32102 # 改
  type: NodePort

完成后结果如下：

[root@k8s-master redis]# kubectl get svc  | grep NodePort
redis-cluster-alpha-follower-0          NodePort    10.96.107.8     <none>        6379:31100/TCP,16379:32100/TCP                                                                              98m
redis-cluster-alpha-follower-1          NodePort    10.96.30.84     <none>        6379:31101/TCP,16379:32101/TCP                                                                              98m
redis-cluster-alpha-follower-2          NodePort    10.96.107.45    <none>        6379:31102/TCP,16379:32102/TCP                                                                              98m
redis-cluster-alpha-leader-0            NodePort    10.96.138.108   <none>        6379:31000/TCP,16379:32000/TCP                                                                              130m
redis-cluster-alpha-leader-1            NodePort    10.96.54.196    <none>        6379:31001/TCP,16379:32001/TCP                                                                              113m
redis-cluster-alpha-leader-2            NodePort    10.96.62.103    <none>        6379:31002/TCP,16379:32002/TCP                                                                              98m

完成之后，我们可以通过NodeIp:NodePort的方式访问到redis cluster中的某个节点。例如，我们想访问redis-cluster-alpha-leader-0，就连接31000端口。

[root@k8s-master redis-6.2.6]# redis-cli -h 192.168.0.163 -p 31000
192.168.0.163:31002> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:4
cluster_my_epoch:3
#...

2、修改cluster nodes命令输出

redis cluster原生提供了如下几个配置来支持DOCKER/NAT的场景：
- cluster-announce-ip: 对外发布自己的节点ip
- cluster-announce-port: 对外发布自己的客户端服务端口。根据实验结果来看，这个设置只会影响到redis对外发布的（也就是其他节点看到的）客户端服务端口，并不是将正常的客户端服务端口6379改为这里指定的端口，设置该选项实际上并不会影响正常的客户端服务端口6379，通过6379端口依旧可以访问redis服务。即我们在做端口映射的时候，应该将cluster-announce-port指定的端口映射到6379端口。外部服务通过cluster-announce-port指定的端口访问到6379端口上的服务
- cluster-announce-tls-port: 对外发布自己的tls连接下的客户端服务端口
- cluster-announce-bus-port: 对外发布自己的集群消息总线端口。根据实验结果来看，这个设置只会影响到redis对外发布的集群总线端口的值，并不是将正常的集群总线端口16379改为这里指定的端口。集群节点之间进行通信应该依旧使用16379端口。我们在做端口映射的时候，应该将cluster-announce-bus-port指定的端口映射到16379端口。其他节点通过cluster-announce-bus-port指定的端口映射到16379端口，以实现节点间的gossip消息通信。
所以我们可以通过上述几个配置，来修改自己对外发布的节点和端口信息，这样，从其他节点上看到的当前节点信息就是我们这里发布的信息了。

具体操作方式如下：

修改集群中每一个节点对外发布的地址cluster-announce-ip为192.168.0.163， 192.168.0.163为k8s集群master节点的ip
修改集群中每一个节点对外发布的客户端通信端口cluster-announce-port为前面NodePort Service中指定的端口。以redis-cluster-alpha-leader-0节点为例，其对应的NodePort Service中指定的暴露到Node上的端口(spec.ports.nodePort)为31000，所以这里指定cluster-announce-port的值为31000。这样k8s集群外部的客户端就可以通过31000端口映射到redis cluster节点的6379端口（spec.ports.targetPort指定的值）.
修改集群中的每一个节点对外发布的集群总线端口cluster-announce-bus-port为前面NodePort Service中指定的端口。以redis-cluster-alpha-leader-0节点为例，其对应的NodePort Service中指定的暴露到Node上的集群总线端口(spec.ports.nodePort)为32000，所以这里指定cluster-announce-bus-port的值为32000。这样redis cluster集群中的其他节点就可以通过32000端口映射到真正的集群总线端口16379（spec.ports.targetPort指定的值）,以实现redis cluster节点之间的通信

redis-cli -h redis-cluster-alpha-leader-0.redis-cluster-alpha-leader.default -p 6379 config set cluster-announce-ip 192.168.0.163
redis-cli -h redis-cluster-alpha-leader-0.redis-cluster-alpha-leader.default -p 6379 config set cluster-announce-port 31000
redis-cli -h redis-cluster-alpha-leader-0.redis-cluster-alpha-leader.default -p 6379 config set cluster-announce-bus-port 32000
redis-cli -h redis-cluster-alpha-leader-0.redis-cluster-alpha-leader.default -p 6379 config rewrite

redis-cli -h redis-cluster-alpha-leader-1.redis-cluster-alpha-leader.default -p 6379 config set cluster-announce-ip 192.168.0.163
redis-cli -h redis-cluster-alpha-leader-1.redis-cluster-alpha-leader.default -p 6379 config set cluster-announce-port 31001
redis-cli -h redis-cluster-alpha-leader-1.redis-cluster-alpha-leader.default -p 6379 config set cluster-announce-bus-port 32001
redis-cli -h redis-cluster-alpha-leader-1.redis-cluster-alpha-leader.default -p 6379 config rewrite

redis-cli -h redis-cluster-alpha-leader-2.redis-cluster-alpha-leader.default -p 6379 config set cluster-announce-ip 192.168.0.163
redis-cli -h redis-cluster-alpha-leader-2.redis-cluster-alpha-leader.default -p 6379 config set cluster-announce-port 31002
redis-cli -h redis-cluster-alpha-leader-2.redis-cluster-alpha-leader.default -p 6379 config set cluster-announce-bus-port 32002
redis-cli -h redis-cluster-alpha-leader-2.redis-cluster-alpha-leader.default -p 6379 config rewrite

redis-cli -h redis-cluster-alpha-follower-0.redis-cluster-alpha-follower.default -p 6379 config set cluster-announce-ip 192.168.0.163
redis-cli -h redis-cluster-alpha-follower-0.redis-cluster-alpha-follower.default -p 6379 config set cluster-announce-port 31100
redis-cli -h redis-cluster-alpha-follower-0.redis-cluster-alpha-follower.default -p 6379 config set cluster-announce-bus-port 32100
redis-cli -h redis-cluster-alpha-follower-0.redis-cluster-alpha-follower.default -p 6379 config rewrite

redis-cli -h redis-cluster-alpha-follower-1.redis-cluster-alpha-follower.default -p 6379 config set cluster-announce-ip 192.168.0.163
redis-cli -h redis-cluster-alpha-follower-1.redis-cluster-alpha-follower.default -p 6379 config set cluster-announce-port 31101
redis-cli -h redis-cluster-alpha-follower-1.redis-cluster-alpha-follower.default -p 6379 config set cluster-announce-bus-port 32101
redis-cli -h redis-cluster-alpha-follower-1.redis-cluster-alpha-follower.default -p 6379 config rewrite

redis-cli -h redis-cluster-alpha-follower-2.redis-cluster-alpha-follower.default -p 6379 config set cluster-announce-ip 192.168.0.163
redis-cli -h redis-cluster-alpha-follower-2.redis-cluster-alpha-follower.default -p 6379 config set cluster-announce-port 31102
redis-cli -h redis-cluster-alpha-follower-2.redis-cluster-alpha-follower.default -p 6379 config set cluster-announce-bus-port 32102
redis-cli -h redis-cluster-alpha-follower-2.redis-cluster-alpha-follower.default -p 6379 config rewrite

三、集群外客户端访问验证

本例中使用Spring Boot中自带的Spring-data-redis客户端访问该redis cluster集群。

首先准备一个Spring Boot项目，该项目中需要导入如下两个jar包:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

然后在application.yml配置文件中，指定redis cluster集群配置：

spring:
  redis:
    cluster:
      nodes:
        - 192.168.0.163:31000
        - 192.168.0.163:31001
        - 192.168.0.163:31002
        - 192.168.0.163:31100
        - 192.168.0.163:31101
        - 192.168.0.163:31102
      max-redirects: 5
    timeout: 2000
    lettuce:
      pool:
        max-active: 1000
        max-idle: 10
        min-idle: 5
        max-wait: -1
      cluster:
        refresh: # 注意要加上这段配置，否则故障转移后ip会变，如果不刷新会造成无法访问
          period: 30s 
          adaptive: true

创建一个Controller如,该controller可以支持set和get命令发送到redis cluster集群

package cn.ljz.redisclient.controller;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class RedisController {

    @Autowired
    private RedisTemplate<String, String> redisTemplate;

    /**
     * http://localhost:8080/redis/operateRedis?cmd=set&args=k11 v11
     * @param cmd
     * @param args
     * @return
     */
    @GetMapping("/redis/operateRedis")
    public String operateRedis(String cmd, String args){
        String result;
        if ("set".equalsIgnoreCase(cmd)){
            String[] s = args.split(" ");
            redisTemplate.opsForValue().set(s[0], s[1]);
            result = "exec success";
        } else if ("get".equalsIgnoreCase(cmd)){
            result = redisTemplate.opsForValue().get(args);
        }else {
            result =  "Unkown command " + cmd;
        }
        return result;
    }
}

之后将该项目打成一个jar包，并将该jar包在一个可访问到k8s集群Node的机器上启动。

[root@k8s-master redis]# java -jar redis-client-0.0.1-SNAPSHOT.jar 

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.6.4)

2022-05-15 17:58:49.551  INFO 5837 --- [           main] c.l.redisclient.RedisClientApplication   : Starting RedisClientApplication v0.0.1-SNAPSHOT using Java 1.8.0_321 on k8s-master with PID 5837 (/root/redis/redis-client-0.0.1-SNAPSHOT.jar started by root in /root/redis)
2022-05-15 17:58:49.557  INFO 5837 --- [           main] c.l.redisclient.RedisClientApplication   : No active profile set, falling back to 1 default profile: "default"
2022-05-15 17:58:50.376  INFO 5837 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode!
2022-05-15 17:58:50.379  INFO 5837 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data Redis repositories in DEFAULT mode.
2022-05-15 17:58:50.401  INFO 5837 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 6 ms. Found 0 Redis repository interfaces.
2022-05-15 17:58:50.949  INFO 5837 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2022-05-15 17:58:50.985  INFO 5837 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2022-05-15 17:58:50.985  INFO 5837 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.58]
2022-05-15 17:58:51.072  INFO 5837 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2022-05-15 17:58:51.073  INFO 5837 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1416 ms
2022-05-15 17:58:52.089  INFO 5837 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2022-05-15 17:58:52.100  INFO 5837 --- [           main] c.l.redisclient.RedisClientApplication   : Started RedisClientApplication in 3.243 seconds (JVM running for 3.928)
2022-05-15 18:01:24.696  INFO 5837 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2022-05-15 18:01:24.696  INFO 5837 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2022-05-15 18:01:24.697  INFO 5837 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
2022-05-15 18:01:25.186  WARN 5837 --- [ioEventLoop-4-1] i.l.c.c.t.DefaultClusterTopologyRefresh  : Unable to connect to [192.168.0.163:6379]: Connection refused: /192.168.0.163:6379
2022-05-15 18:02:23.704 ERROR 5837 --- [nio-8080-exec-9] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.ArrayIndexOutOfBoundsException: 1] with root cause

之后在另一个窗口中访问controller中的接口

[root@k8s-master redis-6.2.6]# curl http://192.168.0.163:8080/redis/operateRedis?cmd=set\&args=k6%20v6
exec success
[root@k8s-master redis-6.2.6]# curl http://192.168.0.163:8080/redis/operateRedis?cmd=get\&args=k6
v6
[root@k8s-master redis-6.2.6]#