不同云服务器公网IP部署k8s集群

文档参考：华为云官方文档k8s集群部署https://support.huaweicloud.com/dpmg-kunpengcpfs/kunpengk8s_04_0004.html机器：k8s-master 华为云 2 CPU 4核心k8s-node1 腾讯云 2 CPU 4核心1、docker 安装https://blog.csdn.net/m0_37840000/art...

愿好

6498人浏览 · 2020-05-08 10:24:00

愿好 · 2020-05-08 10:24:00 发布

文档参考：华为云官方文档k8s集群部署https://support.huaweicloud.com/dpmg-kunpengcpfs/kunpengk8s_04_0004.html

机器：

k8s-master 华为云 2 CPU 4核心

k8s-node1 腾讯云 2 CPU 4核心

1、docker 安装

https://blog.csdn.net/m0_37840000/article/details/83030554

2、配置阿里云yum源（因华为云开源仓库的k8s相关的镜像根本没有packages目录而报错notfound）

 cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

配置主机名

 cat /etc/hosts

3、安装K8S必要组件

 yum install kubectl kubelet kubeadm

systemctl enable kubelet

$ systemctl start kubelet

4、初始化k8s集群（kubeadm 默认从官网k8s.grc.io下载所需镜像，国内无法访问，因此需要通过–image-repository指定阿里云镜像仓库地址）(只有master执行)

kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans master公网Ip --image-repository registry.aliyuncs.com/google_containers --apiserver-advertise-address master公网Ip

kubeadm init --kubernetes-version=1.18.0  --apiserver-advertise-address=机器Ip  --image-repository registry.aliyuncs.com/google_containers  --service-cidr=10.10.0.0/16 --pod-network-cidr=10.122.0.0/16

5、记录生成的最后部分内容，此内容需要在其它节点加入Kubernetes集群时执行,根据提示创建kubectl(只有master执行)

 mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

6、执行下面命令，使kubectl可以自动补充(只有master执行)

source <(kubectl completion bash)

查看节点，pod

kubectl get node

kubectl get pod --all-namespaces

node节点为NotReady，因为corednspod没有启动，缺少网络pod

7、安装calico网络

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

查看pod和node

命令：

kubectl get pod --all-namespaces

结果：

NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-555fc8cc5c-k8rbk    1/1     Running   0          36s
kube-system   calico-node-5km27                           1/1     Running   0          36s
kube-system   coredns-7ff77c879f-fsj9l                    1/1     Running   0          5m22s
kube-system   coredns-7ff77c879f-q5ll2                    1/1     Running   0          5m22s
kube-system   etcd-master01.paas.com                      1/1     Running   0          5m32s
kube-system   kube-apiserver-master01.paas.com            1/1     Running   0          5m32s
kube-system   kube-controller-manager-master01.paas.com   1/1     Running   0          5m32s
kube-system   kube-proxy-th472                            1/1     Running   0          5m22s
kube-system   kube-scheduler-master01.paas.com            1/1     Running   0          5m32s

命令：

kubectl get node

结果：

NAME                STATUS   ROLES    AGE     VERSION
master01.paas.com   Ready    master   5m47s   v1.18.0

此时集群状态正常

8、安装kubernetes-dashboard

github项目地址https://github.com/kubernetes/dashboard

官方部署dashboard的服务没使用nodeport，将yaml文件下载到本地，在service里添加nodeport(并再安全组里开放对应的nodePort端口)

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml

vim recommended.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30000
  selector:
    k8s-app: kubernetes-dashboard

kubectl create -f recommended.yaml

查看pod，service

kubectl get svc -n kubernetes-dashboard

使用token进行登录，执行下面命令获取token

创建完成后，会自动创建相关的token
kubectl -n kube-system create sa cluster-admin

再获取具体token令牌key
kubectl -n kube-system get secrets
结果如下：
NAME                                             TYPE                                  DATA   AGE
attachdetach-controller-token-j959p              kubernetes.io/service-account-token   3      7h10m
bootstrap-signer-token-z8dsn                     kubernetes.io/service-account-token   3      7h10m
bootstrap-token-zl5k9s                           bootstrap.kubernetes.io/token         6      7h10m
calico-kube-controllers-token-gkgsn              kubernetes.io/service-account-token   3      5h56m
calico-node-token-66dzp                          kubernetes.io/service-account-token   3      5h56m
certificate-controller-token-c8qkv               kubernetes.io/service-account-token   3      7h10m
cluster-admin-token-6r95k                        kubernetes.io/service-account-token   3      67s
clusterrole-aggregation-controller-token-bmxp6   kubernetes.io/service-account-token   3      7h10m


找到cluster-admin-token的具体key然后再获取token值
kubectl -n kube-system describe secret cluster-admin-token-6r95k

登录后如下展示，如果没有namespace可选，并且提示找不到资源，那么就是权限问题

通过查看dashboard日志，得到如下信息

[root@master01 ~]# kubectl logs -f -n kubernetes-dashboard kubernetes-dashboard-5d4dc8b976-sdxxt
2020/04/08 01:54:31 Non-critical error occurred during resource retrieval: nodes is forbidden: User "system:serviceaccount:kube-system:cluster-admin" c

解决方法1：

再创建kubernetes-dashboard管理员角色

[root@k8s-master ~]# vi k8s-admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-admin
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: dashboard-admin
subjects:
  - kind: ServiceAccount
    name: dashboard-admin
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

[root@k8s-master ~]# kubectl create -f k8s-admin.yaml
[root@k8s-master ~]# kubectl describe secret dashboard-admin-token -n kube-system    
执行后会得到admin-token

解决方法2：

 kubectl create clusterrolebinding serviceaccount-cluster-admin --clusterrole=cluster-admin --group=system:serviceaccount

此时再查看dashboard，即可看到有资源展示

Node节点加入集群

使用kubeadm join 注册Node节点到Matser

（运行第5步骤最后生成的命令，（下方数据已作处理，别想加到我的集群里来，哈哈））

kubeadm join masteIp:6443 --token zl9s.cfpug3wrygnch \
    --discovery-token-ca-cert-hash sha256:a2b64890f3f143fbd6293e9a571c5bec473faaf866bf7494092

如果运行此报错如下：

W0508 11:14:37.272313   19221 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
        [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING Hostname]: hostname "k8s-node-1" could not be reached
        [WARNING Hostname]: hostname "k8s-node-1": lookup k8s-node-1 on 183.60.82.98:53: no such host
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

默认token的有效期为24小时，即该token已过期，如果后续有nodes节点加入，需重新生成token

# 1.查看当前的token列表
[root@k8s-master ~]# kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
7mjtn4.9kds6sabcouxaugd   5h         2020-05-08T10:44:44+08:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token

# 2.重新生成新的token
[root@k8s-master ~]# kubeadm token create
369tcl.oe4punpoj9gaijh7

# 3.再次查看当前的token列表
[root@k8s-master ~]# kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                               
DESCRIPTION                                                EXTRA GROUPS
369tcl.oe4punpoj9gaijh7   23h        2020-05-09T10:44:44+08:00        authentication,signing   <none>                                                         system:bootstrappers:kubeadm:default-node-token
7mjtn4.9kds6sabcouxaugd   23h        2020-05-09T10:44:44+08:00    authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token

# 4.获取ca证书sha256编码hash值
[root@k8s-master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

7ae10591aa593c2c36fb965d58964a84561e9ccd416ffe7432550a0d0b7e4f90

# 5.节点加入集群
[root@k8s-master ~]# kubeadm join --token 369tcl.oe4punpoj9gaijh7(新的token) --discovery-token-ca-cert-hash sha256:7ae10591aa593c2c36fb965d58964a84561e9ccd416ffe7432550a0d0b7e4f90(ca证书sha256编码hash值) masterIP:6443 --skip-preflight-chec

如果节点加入集群失败，报错如下：

[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING Hostname]: hostname "k8s-node-1" could not be reached
        [WARNING Hostname]: hostname "k8s-node-1": lookup k8s-node-1 on 183.60.83.19:53: no such host
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

则根据错误信息ERROR，执行 echo "1"> /proc/sys/net/ipv4/ip_forward
接着再运行加入集群的命令即可，此时dashBoard的node节点就出现了，但其状态为notReady（红）,因为刚加入还需要点时间，大概1分钟状态会变成ready（绿）