ETCD 高可用集群 Centos7

ETCD 高可用集群在本地三台虚拟机以static方式搭建ETCD集群测试环境，均为Centos7一、准备工作1、主机名及IP主机名IPvm5192.168.56.10vm6192.168.56.11vm7192.168.56.122、安装go lang1）使用自动安装方式y...

hahaxu

1468人浏览 · 2018-07-08 12:16:23

hahaxu · 2018-07-08 12:16:23 发布

ETCD 高可用集群

在本地三台虚拟机以static方式搭建ETCD集群测试环境，均为Centos7

一、准备工作

1、主机名及IP

主机名	IP
vm5	192.168.56.10
vm6	192.168.56.11
vm7	192.168.56.12

2、安装go lang

1）使用自动安装方式

yum install go

查看go安装后的环境

go env

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GORACE=""
GOROOT="/usr/lib/golang"
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build616764896=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

设置环境变量

export GOPATH=/root/go
export PATH=$PATH:$GOPATH/bin

2）下载安装

https://blog.csdn.net/warnerwu/article/details/73825105

3、安装git

yum install git

4、其它工具安装

vim、netstat

yum install vim
yum install net-tools

netstat -tunlp|grep 2379

5、开启防火墙

centos7默认开启防火墙，需要开启相应端口，建议不要关闭防火墙
需要开启：2379、2380

1、添加端口(–permanent永久生效，没有此参数重启后失效)

firewall-cmd --zone=public --add-port=2379/tcp --permanent
firewall-cmd --zone=public --add-port=2380/tcp --permanent

2、重新载入生效

firewall-cmd --reload

3、查看确认

firewall-cmd --zone=public --query-port=2379/tcp
firewall-cmd --zone=public --query-port=2380/tcp

4、删除

firewall-cmd --zone=public --remove-port=2379/tcp --permanent
firewall-cmd --zone=public --remove-port=2380/tcp --permanent

二、TLS密钥和证书

这里部署的etcd集群使用TLS证书对证书通信进行加密，并开启基于CA根证书签名的双向数字证书认证。

下面介绍使用cfssl生成所需要的私钥和证书。

1、安装cfssl

cfssl是使用Go语言开发的工具，如果系统中安装了Go，可以使用直接go get安装cfssl:

go get -u github.com/cloudflare/cfssl/cmd/...

默认安装到$GOPATH/bin目录，安装cfssl, cfssjosn, mkbundle等工具。

CA证书和私钥

创建ca-config.json：

{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "ufs": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}

ca-config.json中可以定义多个profile，分别设置不同的expiry和usages等参数。如上面的ca-config.json中定义了名称为frognew的profile，这个profile的expiry 87600h为10年，useages中：

signing表示此CA证书可以用于签名其他证书，ca.pem中的CA=TRUE
server auth表示TLS Server Authentication
client auth表示TLS Client Authentication

创建CA证书签名请求配置ca-csr.json：

{
  "CN": "ufs",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "Unionflight",
      "OU": "DEV"
    }
  ]
}

下面使用cfss生成CA证书和私钥：

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

018/07/04 09:27:27 [INFO] generating a new CA key and certificate from CSR
2018/07/04 09:27:27 [INFO] generate received request
2018/07/04 09:27:27 [INFO] received CSR
2018/07/04 09:27:27 [INFO] generating key: rsa-2048
2018/07/04 09:27:27 [INFO] encoded CSR
2018/07/04 09:27:27 [INFO] signed certificate with serial number 425600086876483002385252956592256395427139612143

ls

ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem

ca-key.pem和ca.pem需要保存在一个安全的地方，后边会用到。

etcd证书和私钥

创建etcd证书签名请求配置etcd-csr.json：

{
    "CN": "ufs",
    "hosts": [
      "127.0.0.1",
      "192.168.56.10",
      "192.168.56.11",
      "192.168.56.12",
      "vm5",
      "vm6",
      "vm7"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O": "OVO",
            "OU": "DEV"
        }
    ]
}

注意上面配置hosts字段中制定授权使用该证书的IP和域名列表，因为现在要生成的证书需要被etcd集群各个节点使用，所以这里指定了各个节点的IP和hostname。

下面生成etcd的证书和私钥：

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=ufs etcd-csr.json | cfssljson -bare etcd

2018/07/04 09:36:23 [INFO] generate received request
2018/07/04 09:36:23 [INFO] received CSR
2018/07/04 09:36:23 [INFO] generating key: rsa-2048
2018/07/04 09:36:23 [INFO] encoded CSR
2018/07/04 09:36:23 [INFO] signed certificate with serial number 130267449506552215685384729289882051406882411566

ls etcd*

etcd.csr etcd-csr.json etcd-key.pem etcd.pem

对生成的证书可以使用cfssl或openssl查看：

安装etcd

将CA证书ca.pem, etcd秘钥etcd-key.pem, etcd证书etcd.pem拷贝到各节点的/etc/etcd/ssl目录中。

下载etcd二进制文件包：

wget https://github.com/coreos/etcd/releases/download/v3.3.8/etcd-v3.3.8-linux-amd64.tar.gz

tar xzvf etcd-v3.3.8-linux-amd64.tar.gz

解压缩etcd-v3.3.8-linux-amd64.tar.gz，将其中的etcd和etcdctl两个可执行文件复制到各节点的/usr/bin目录。

在各节点创建etcd的数据目录：

mkdir -p /var/lib/etcd

在每个节点上创建etcd的systemd unit文件/usr/lib/systemd/system/etcd.service，注意替换ETCD_NAME和INTERNAL_IP变量的值：

export ETCD_NAME=vm5
export INTERNAL_IP=192.168.56.10
cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd \
  --name ${ETCD_NAME} \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \
  --listen-peer-urls https://${INTERNAL_IP}:2380 \
  --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://${INTERNAL_IP}:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster vm5=https://192.168.56.10:2380,vm6=https://192.168.56.11:2380,vm7=https://192.168.56.12:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

上面在启动参数中指定了etcd的工作目录和数据目录是/var/lib/etcd
–cert-file和–key-file分别指定etcd的公钥证书和私钥
–peer-cert-file和–peer-key-file分别指定了etcd的Peers通信的公钥证书和私钥。
–trusted-ca-file指定了客户端的CA证书
–peer-trusted-ca-file指定了Peers的CA证书
–initial-cluster-state new表示这是新初始化集群，–name指定的参数值必须在–initial-cluster中

启动etcd

在各节点上启动etcd：

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

检查集群是否健康，在任一节点执行：

etcdctl \
  --ca-file=/etc/etcd/ssl/ca.pem \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --endpoints=https://192.168.56.10:2379,https://192.168.56.11:2379,https://192.168.56.12:2379 \
  cluster-health

etcdctl --ca-file=/etc/etcd/ssl/ca.pem --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.56.10:2379,https://192.168.56.11:2379,https://192.168.56.12:2379 cluster-health

member 490019a0be2eb111 is healthy: got healthy result from https://192.168.56.12:2379
member ae6c8e8866610eff is healthy: got healthy result from https://192.168.56.10:2379
member b8b678caf9e91e40 is healthy: got healthy result from https://192.168.56.11:2379
cluster is healthy

确保输出cluster is healthy的信息。

华为开发者空间

华为开发者空间，是为全球开发者打造的专属开发空间，汇聚了华为优质开发资源及工具，致力于让每一位开发者拥有一台云主机，基于华为根生态开发、创新。

更多推荐