k8s 集群搭建详细教程

参考： Kubernetes 文档 / 入门 / 生产环境 / 使用部署工具安装 Kubernetes /使用 kubeadm 引导集群 / 安装 kubeadm一般来讲，硬件设备会拥有唯一的地址，但是有些虚拟机的地址可能会重复。 Kubernetes 使用这些值来唯一确定集群中的节点。如果这些值在每个节点上不唯一，可能会导致安装失败。[][][]()Ubuntu Server (minimi

焱宣

2483人浏览 · 2023-04-29 11:37:18

焱宣 · 2023-04-29 11:37:18 发布

参考： Kubernetes 文档 / 入门 / 生产环境 / 使用部署工具安装 Kubernetes / 使用 kubeadm 引导集群 / 安装 kubeadm

B. 准备开始

一台兼容的 Linux 主机。Kubernetes 项目为基于 Debian 和 Red Hat 的 Linux 发行版以及一些不提供包管理器的发行版提供通用的指令
每台机器 2 GB 或更多的 RAM （如果少于这个数字将会影响你应用的运行内存)
2 CPU 核或更多
集群中的所有机器的网络彼此均能相互连接(公网和内网都可以)
节点之中不可以有重复的主机名、MAC 地址或 product_uuid。请参见这里了解更多详细信息。
开启机器上的某些端口。请参见这里了解更多详细信息。
禁用交换分区。为了保证 kubelet 正常工作，你必须 禁用交换分区。

U. 确保每个节点上 MAC 地址和 product_uuid 的唯一性

你可以使用命令 ip link 或 ifconfig -a 来获取网络接口的 MAC 地址
可以使用 sudo cat /sys/class/dmi/id/product_uuid 命令对 product_uuid 校验

一般来讲，硬件设备会拥有唯一的地址，但是有些虚拟机的地址可能会重复。 Kubernetes 使用这些值来唯一确定集群中的节点。如果这些值在每个节点上不唯一，可能会导致安装失败。

V. 虚拟机

新建… / 创建自定虚拟机 /
Linux / Ubuntu 64位

设置过程

ID	『虚拟机』设置	建议配置	默认值	说明
1	处理器	-	2	最低要求
2	内存	-	4096 MB	节约内存
3	显示器	取消复选`加速 3D 图形`	复选	节约内存
4	网络适配器	-	nat	需上网
5	硬盘	`40`GB	20 GB	保证练习容量
6	选择固件类型	UEFI	传统 BIOS	VMware Fusion 支持嵌套虚拟化

设置结果

ID	Your computer’s name	CPU 核	RAM	DISK	NIC
1	`k8s-master`	4 或更多	8 GB或更多	40 GB	nat
2	`k8s-worker1`	同上	2 GB或更多	同上	同上
3	`k8s-worker2`	同上	同上	同上	同上

I. 安装 Ubuntu 22.04 LTS

Willkommen! Bienvenue! Welcome! Welkom!

[ English ]
Installer update available

[ Continue without updating ]
Keyboard configuration

[ Done ]
Choose type of install

(X) Ubuntu Server (minimized)
/ [ Done ]
Network connections

[ Done ]
Configure proxy

[ Done ]
Configure Ubuntu archive mirror

Mirror address: http://mirror.nju.edu.cn/ubuntu
/ [ Done ]
Guided storage configuration

[ Done ]
Storage configuration

[ Done ]
Profile setup

                         Your name: kiosk
         Your server 's name: k8s-master
               Pick a username: kiosk
          Choose a password: ubuntu
    Confirm your password: ubuntu
    / [ Done ]
SSH Setup

[X] Install OpenSSH server
/ [ Done ]
Featured Server Snaps

[ Done ]
Install complete!

🅰️ [ Cancel update and reboot ]

🅱️ [ Reboot Now ]
建议（可选）

关机后，做个快照

P. 准备工作

[kiosk@k8s-master|k8s-worker1|k8s-worker2]$

设置当前用户sudo免密

sudo tee /etc/sudoers.d/$USER >/dev/null <<EOF
$USER ALL=(ALL) NOPASSWD:ALL
EOF

使用国内镜像仓库

# 国内镜像仓库
MIRROR_URL=http://mirror.nju.edu.cn/ubuntu

# 生成软件仓库源
sudo tee /etc/apt/sources.list >/dev/null <<EOF
deb $MIRROR_URL jammy main restricted universe multiverse
deb $MIRROR_URL jammy-updates main restricted universe  multiverse
deb $MIRROR_URL jammy-backports main restricted universe multiverse
deb $MIRROR_URL jammy-security main restricted universe multiverse
EOF

安装相关软件

# 更新
sudo apt -y update

# 安装
sudo apt install -y openssh-server \
	vim sshpass nfs-common \
	bash-completion netcat-openbsd \
	open-vm-tools

[kiosk@k8s-master]$

设置静态IP

# 配置IP
sudo tee /etc/netplan/00-installer-config.yaml >/dev/null <<EOF
network:
  ethernets:
    ens33:
      dhcp4: false
      addresses: [192.168.147.128/24]
      gateway4: 192.168.147.2
      nameservers:
        addresses: [8.8.8.8]
  version: 2
EOF

# dns
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

# active
sudo netplan apply

[kiosk@k8s-worker1]$

设置静态IP

# 配置IP
sudo tee /etc/netplan/00-installer-config.yaml >/dev/null <<EOF
network:
  ethernets:
    ens33:
      dhcp4: false
      addresses: [192.168.147.129/24]
      gateway4: 192.168.147.2
      nameservers:
        addresses: [8.8.8.8]
  version: 2
EOF

# dns
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

# active
sudo netplan apply

[kiosk@k8s-worker2]$

设置静态IP

# 配置IP
sudo tee /etc/netplan/00-installer-config.yaml >/dev/null <<EOF
network:
  ethernets:
    ens33:
      dhcp4: false
      addresses: [192.168.147.130/24]
      gateway4: 192.168.147.2
      nameservers:
        addresses: [8.8.8.8]
  version: 2
EOF

# dns
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

# active
sudo netplan apply

[kiosk@k8s-master|k8s-worker1|k8s-worker2]$

编辑 hosts

sudo tee -a /etc/hosts >/dev/null <<EOF
192.168.147.128	k8s-master
192.168.147.129 k8s-worker1
192.168.147.130 k8s-worker2
EOF

# 设置 root 密码
(echo ubuntu; echo ubuntu) | sudo passwd root
echo PermitRootLogin yes | sudo tee -a /etc/ssh/sshd_config
sudo systemctl restart sshd

[kiosk@k8s-master]$

ssh免密

# 生成 keypair
ssh-keygen -f ~/.ssh/id_rsa -N ''

# 拷贝公钥
for i in k8s-master k8s-worker1 k8s-worker2; do
   sshpass -pubuntu ssh-copy-id -o StrictHostKeyChecking=no kiosk@$i
   sshpass -pubuntu ssh-copy-id -o StrictHostKeyChecking=no root@$i
done

[kiosk@k8s-master|k8s-worker1|k8s-worker2]$

禁用swap

# 交换文件
SWAPF=$(awk '/swap/ {print $1}' /etc/fstab)

# 立即禁用
sudo swapoff $SWAPF

# 永久禁用
sudo sed -i '/swap/d' /etc/fstab

# 删除交换文件
sudo rm $SWAPF

扩容

# 逻辑卷名
export LVN=$(sudo lvdisplay | awk '/Path/ {print $3}')

# 扩容
sudo lvextend -l 100%PVS $LVN

# 立即生效
sudo resize2fs $LVN

# 验证
df -h /

模块支持

# 安装
sudo apt -y install bridge-utils

# 立即生效
sudo modprobe br_netfilter

# 内核支持
sudo tee /etc/sysctl.d/k8s.conf >/dev/null <<EOF
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
EOF

# 立即生效
sudo sysctl -p /etc/sysctl.d/k8s.conf

docker: k8s-master, k8s-worker1
containerd: k8s-worker2

[kiosk@k8s-master|k8s-worker1]$

安装运行时

# 创建文件夹
sudo mkdir -p /etc/docker

# 生成配置文件
sudo tee /etc/docker/daemon.json >/dev/null <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "10"
  },
  "registry-mirrors": ["https://docker.nju.edu.cn/"]
}
EOF

# 安装 runtime
sudo apt -y install docker.io

# 开机自启
sudo systemctl enable docker

# 立即重启
sudo systemctl restart docker

# 安装命令 cri-dockerd
curl -# https://vmcc.xyz:8443/k8s/cri-docker/cri-dockerd-0.2.5.amd64.tgz \
	-o cri-dockerd-0.2.5.amd64.tgz
tar -xf cri-dockerd-0.2.5.amd64.tgz
sudo cp cri-dockerd/cri-dockerd /usr/bin/

# 安装服务 cri-docker.service
sudo curl -s https://vmcc.xyz:8443/k8s/cri-docker/cri-docker.service \
	-o /usr/lib/systemd/system/cri-docker.service
sudo sed -i '/ExecStart/s+$+ --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.8+' /usr/lib/systemd/system/cri-docker.service

# 安装 cri-docker.socket
sudo curl -s https://vmcc.xyz:8443/k8s/cri-docker/cri-docker.socket \
	-o /usr/lib/systemd/system/cri-docker.socket

# 启动服务 cri-dockerd
sudo systemctl daemon-reload
sudo systemctl enable cri-docker
sudo systemctl restart cri-docker

# 安装 crictl 命令
curl -# https://vmcc.xyz:8443/k8s/crictl-v1.24.2-linux-amd64.tar.gz \
	-o crictl-v1.24.2-linux-amd64.tar.gz
tar -xf crictl-v1.24.2-linux-amd64.tar.gz
sudo cp crictl /usr/bin/
# crictl 配置文件
sudo tee /etc/crictl.yaml >/dev/null <<EOF
runtime-endpoint: unix:///var/run/cri-dockerd.sock
image-endpoint: unix:///var/run/cri-dockerd.sock
timeout: 10
debug: false
pull-image-on-create: true
EOF

[kiosk@k8s-worker2]$

安装运行时

# 安装 containerd
sudo apt install -y containerd

# 创建目录
sudo mkdir /etc/containerd

# 生成默认配置文件
containerd config default | \
sudo tee /etc/containerd/config.toml >/dev/null

# 修改配置文件
sudo sed -i \
-e '/sandbox_image/s?k8s.gcr.io?registry.aliyuncs.com/google_containers?' \
-e '/SystemdCgroup/s?false?true?' \
-e '/registry.mirrors/a\        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]' \
-e '/registry.mirrors/a\          endpoint = ["https://docker.nju.edu.cn/"]' /etc/containerd/config.toml

# 服务重启
sudo systemctl restart containerd

# 安装 crictl 命令
curl -# https://vmcc.xyz:8443/k8s/crictl-v1.24.2-linux-amd64.tar.gz \
	-o crictl-v1.24.2-linux-amd64.tar.gz
tar -xf crictl-v1.24.2-linux-amd64.tar.gz
sudo cp crictl /usr/bin/
# crictl 配置文件
sudo tee /etc/crictl.yaml >/dev/null <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
pull-image-on-create: true
EOF

K. 安装 k8s

[kiosk@k8s-master|k8s-worker1|k8s-worker2]$

安装 kubeadm、kubelet 和 kubectl

# 更新 apt 包索引并安装使用 Kubernetes apt 仓库所需要的包
sudo apt -y install apt-transport-https ca-certificates curl

# 下载 Google Cloud 公开签名秘钥
curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -

# 添加 Kubernetes apt 仓库
MIRROR_URL=https://mirror.nju.edu.cn/kubernetes/apt/
sudo tee /etc/apt/sources.list.d/kubernetes.list >/dev/null <<EOF
deb $MIRROR_URL kubernetes-xenial main
EOF

# 更新 apt 包索引
sudo cp /etc/apt/trusted.gpg /etc/apt/trusted.gpg.d
sudo apt update -y
sudo apt-cache madison kubelet | grep 1.24

# 安装 kubelet、kubeadm 和 kubectl 考试版本
sudo apt install -y kubelet=1.24.1-00 kubeadm=1.24.1-00 kubectl=1.24.1-00

# 锁定版本
sudo apt-mark hold kubelet kubeadm kubectl

[kiosk@k8s-worker2]$

# 增加 k8s 支持
 sudo sed -i '/ExecStart=\//s|$| --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --cgroup-driver=systemd|' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# 重启 kubelet 服务
sudo systemctl daemon-reload
sudo systemctl restart kubelet

[kiosk@k8s-master]$

初始化

# 生成初始文件
sudo kubeadm config print init-defaults > kubeadm-config.yaml

# 修改文件
sudo sed -i \
  -e "/advertiseAddress/s?:.*?: 192.168.147.128?" \
  -e "/name/s?:.*?: k8s-master?" \
  -e "/clusterName/s?:.*?: ck8s?" \
  -e "/imageRepository/s?:.*?: registry.aliyuncs.com/google_containers?" \
  -e "/criSocket/s+containerd/containerd+cri-dockerd+" kubeadm-config.yaml

# 使用初始文件，初始化集群
sudo kubeadm init --config kubeadm-config.yaml

…输出省略…
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.147.128:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:c4781194de65ebb47984fc5e7e64d4897875410825ce4d18df81da1a298afa1f

配置文件

# 创建目录
mkdir -p $HOME/.kube

# user 复制配置文件
sudo \cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# root 变量
sudo tee -a ~root/.bashrc >/dev/null <<EOF
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF

创建网络

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

命令补全

# 立即生效
source <(kubectl completion bash)
    
# 永久生效
mkdir ~/.kube
kubectl completion bash > ~/.kube/completion.bash.inc
printf "
# Kubectl shell completion
source '$HOME/.kube/completion.bash.inc'
" >> $HOME/.bash_profile
source $HOME/.bash_profile

命令别名

# 永久生效
tee -a $HOME/.bashrc >/dev/null <<EOF
alias k='kubectl'
complete -F __start_kubectl k
EOF

# 立即生效
source $HOME/.bashrc

[kiosk@k8s-worker1]$

加入集群

sudo kubeadm join 192.168.147.128:6443 \
  --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:c4781194de65ebb47984fc5e7e64d4897875410825ce4d18df81da1a298afa1f \
  --cri-socket unix://var/run/cri-dockerd.sock

[kiosk@k8s-worker2]$

加入集群

sudo kubeadm join 192.168.147.128:6443 \
  --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:c4781194de65ebb47984fc5e7e64d4897875410825ce4d18df81da1a298afa1f

C. 确认环境正常

[kiosk@k8s-master]

$ kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
scheduler           `Healthy`  ok
controller-manager  `Healthy`  ok
etcd-0              `Healthy`  {"health":"true","reason":""}

$ kubectl get nodes
NAME          STATUS   ROLES                  AGE     VERSION
k8s-worker1  `Ready`  <none>                 4m4s    `v1.24.1`
k8s-worker2  `Ready`  <none>                 4m44s   `v1.24.1`
k8s-master   `Ready`  control-plane,master   13m     `v1.24.1`

$ kubectl -n kube-system get pod -w
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-555bc4b957-8ccgh   1/1     Running   0          27m
calico-node-5qqcq                          1/1     Running   0          9m29s
calico-node-7qclz                          1/1     Running   0          27m
calico-node-kcvt5                          1/1     Running   0          9m29s
coredns-74586cf9b6-69fn7                   1/1     Running   0          156m
coredns-74586cf9b6-8mgl9                   1/1     Running   0          156m
etcd-k8s-master                            1/1     Running   0          156m
kube-apiserver-k8s-master                  1/1     Running   0          156m
kube-controller-manager-k8s-master         1/1     Running   0          156m
kube-proxy-8j248                           1/1     Running   0          9m29s
kube-proxy-g7r55                           1/1     Running   0          9m29s
kube-proxy-rbdcp                           1/1     Running   0          156m
kube-scheduler-k8s-master                  1/1     Running   0          156m
<Ctrl-C>