k8s集成containerd，集成crictl工具，集成kata

场景记录：需要使用kata安全容器，与runc共存，同时替换docker为containerd，使用crictl工具作为镜像管理的客户端工具，记录实操及问题解决的过程基础环境：（kvm虚拟机）centos-7.9 + k8s-v1.23.0，k8s集群环境安装可参见这里https://blog.csdn.net/sinat_38453878/article/details/123224546?s

o！xiaoxuan

2855人浏览 · 2022-03-03 18:12:11

o！xiaoxuan · 2022-03-03 18:12:11 发布

场景记录：需要使用kata安全容器，与runc共存，同时替换docker为containerd，使用crictl工具作为镜像管理的客户端工具，记录实操及问题解决的过程
基础环境：（kvm虚拟机）centos-7.9 + k8s-v1.23.0

安装kata

参见这里 https://blog.csdn.net/sinat_38453878/article/details/123207170?spm=1001.2014.3001.5502

配置containerd

containerd包含在docker里面，无需单独安装

生成默认的配置文件：

containerd config default > /etc/containerd/config.toml

contianerd配置镜像源地址

安装crictl工具

curl -SLO https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.12.0/crictl-v1.12.0-linux-amd64.tar.gz

tar -xvf crictl-v1.12.0-linux-amd64.tar.gz -C /usr/local/bin/

vi /etc/crictl.yaml

crictl 文件内容

#runtime-endpoint: unix:///run/dockershim.sock
#image-endpoint: unix:///run/dockershim.sock
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false

此处可能报错
[root@localhost install-k8s]# crictl images
FATA[0000] listing images failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService
解决：
执行 containerd config default > /etc/containerd/config.toml
如果依然报错
在这里插入图片描述
检查 xfs文件系统，执行xfs_info / ，如果ftype=0，如上图所示，文件系统不合适，那么很不幸，要麻烦了，实际就是从头开始（必须将文件系统修正，不然后续集成containerd,kata等都没法继续）
参见这里
https://www.pimwiddershoven.nl/entry/docker-on-centos-7-machine-with-xfs-filesystem-can-cause-trouble-when-d-type-is-not-supported
如果当前使用环境是kvm虚拟机，再参见这里
https://blog.csdn.net/sinat_38453878/article/details/123198394?spm=1001.2014.3001.5502

准备好containerd的相关镜像

更改k8s配置之前预先准备好ctr i list的containerd的镜像
脚本

#!/bin/bash
# pull images

ver=v1.23.0
registry=registry.cn-hangzhou.aliyuncs.com/google_containers
images=`kubeadm config images list --kubernetes-version=$ver |awk -F '/' '{print $2}'`

for image in $images
do
if [ $image != coredns ];then
    ctr -n k8s.io image pull ${registry}/$image
    if [ $? -eq 0 ];then
        ctr -n k8s.io image tag ${registry}/$image k8s.gcr.io/$image
        ctr -n k8s.io image rm ${registry}/$image
    else
        echo "ERROR: 下载镜像报错，$image"
    fi
else
    ctr -n k8s.io image pull ${registry}/coredns:1.8.6
    ctr -n k8s.io image tag ${registry}/coredns:1.8.6  k8s.gcr.io/coredns/coredns:v1.8.6
    ctr -n k8s.io image rm ${registry}/coredns:1.8.6
fi
done

备注：ctr和crictl工具使用过程中会出现二者列出的信息不一致的情况，参见这里

修改kubelet配置文件，使用containerd作为运行时

位置：可以通过kubectl status kubelet -l查看
本次操作：vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
相关的文件：vi /etc/sysconfig/kubelet，vi /var/lib/kubelet/kubeadm-flags.env
/etc/kubernetes/bootstrap-kubelet.conf
/etc/kubernetes/kubelet.conf
在这里插入图片描述
设置相关的启动参数

设置启动引擎
–cgroup-driver=systemd

设置containerd为运行时
–container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock

重启kubelet

[root@localhost install-k8s]# systemctl daemon-reload
[root@localhost install-k8s]# systemctl restart kubelet
[root@localhost install-k8s]# systemctl status kubelet-l

查看日志

journalctl -xefu kubelet

master节点查看运行时修改是否成功 kubectl get nodes -o wide，看到运行时为containerd即修改成功
在这里插入图片描述
问题记录：更换运行时后pod起不来，可能是containerd拉取不到镜像，systemctl status kubelet -l可看到相关日志

更多更换containerd后的出现的问题请看这里
https://blog.csdn.net/sinat_38453878/article/details/123255489?spm=1001.2014.3001.5502

配置containerd动态使用kata（默认runc）

完整的配置好kata的containerd配置文件：

version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
plugin_dir = ""
disabled_plugins = []
required_plugins = []
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[ttrpc]
  address = ""
  uid = 0
  gid = 0

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[plugins]
  [plugins."io.containerd.gc.v1.scheduler"]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"
  [plugins."io.containerd.grpc.v1.cri"]
    disable_tcp_service = true
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    stream_idle_timeout = "4h0m0s"
    enable_selinux = false
    selinux_category_range = 1024
    sandbox_image = "k8s.gcr.io/pause:3.2"
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    disable_cgroup = false
    disable_apparmor = false
    restrict_oom_score_adj = false
    max_concurrent_downloads = 3
    disable_proc_mount = false
    unset_seccomp_profile = ""
    tolerate_missing_hugetlb_controller = true
    disable_hugetlb_controller = true
    ignore_image_defined_volumes = false
    [plugins."io.containerd.grpc.v1.cri".containerd]
      snapshotter = "overlayfs"
      default_runtime_name = "runc"
      no_pivot = false
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
        privileged_without_host_devices = false
        base_runtime_spec = ""
      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = "/usr/bin/kata-runtime"
        runtime_root = ""
        privileged_without_host_devices = false
        base_runtime_spec = ""
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          runtime_engine = ""
          runtime_root = ""
          privileged_without_host_devices = false
          base_runtime_spec = ""
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
         [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
         runtime_type = "io.containerd.kata.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.katacli]
         runtime_type = "io.containerd.runc.v1"
         [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.katacli.options]
           NoPivotRoot = false
           NoNewKeyring = false
           ShimCgroup = ""
           IoUid = 0
           IoGid = 0
           BinaryName = "/usr/bin/kata-runtime"
           Root = ""
           CriuPath = ""
           SystemdCgroup = false
    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      max_conf_num = 1
      conf_template = ""
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = ""
    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""
  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"
  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"
  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"
  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false
  [plugins."io.containerd.runtime.v1.linux"]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]
  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]
  [plugins."io.containerd.snapshotter.v1.devmapper"]
    root_path = ""
    pool_name = ""
    base_image_size = ""
    async_remove = false

修改好/etc/containerd/config.toml之后重启containerd

[root@localhost containerd]# systemctl daemon-reload
[root@localhost containerd]# systemctl restart containerd
[root@localhost containerd]# systemctl status containerd

使用

一切都配置妥当之后就可以开始使用了

RuntimeClass方式

创建RuntimeClass

[root@localhost critool]# vi kata-runtime.yaml
[root@localhost critool]# kubectl apply -f kata-runtime.yaml 
Warning: node.k8s.io/v1beta1 RuntimeClass is deprecated in v1.22+, unavailable in v1.25+
runtimeclass.node.k8s.io/kata-containers created
[root@localhost critool]# kubectl get runtimeclass
NAME              HANDLER   AGE
kata-containers   kata      27s

kata-runtime.yaml内容

kind: RuntimeClass
apiVersion: node.k8s.io/v1beta1
metadata:
  name: kata-containers
handler: kata

创建基于kata的pod

[root@localhost critool]# vi kata-pod.yaml
[root@localhost critool]# kubectl apply -f kata-pod.yaml 
pod/kata-nginx created
[root@localhost critool]# kubectl get pods
NAME         READY   STATUS    RESTARTS   AGE
kata-nginx   1/1     Running   0          3s

kata-pod文件内容

apiVersion: v1
kind: Pod
metadata:
  name: kata-nginx
spec:
  runtimeClassName: kata-containers
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80

验证

[root@localhost critool]# kubectl get pods -o wide
NAME         READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
kata-nginx   1/1     Running   0          12s   10.244.85.194   k8s-node01   <none>           <none>

节点上查看kata运行列表

[root@localhost containerd]# kata-runtime list
ID                                                                 PID         STATUS      BUNDLE                                                                                                                  CREATED                          OWNER
2a26331f251a9aa2d5872b04d7bc39fba2fef77746e94b1603ff4c38d4c1fa4d   -1          running     /run/containerd/io.containerd.runtime.v2.task/k8s.io/2a26331f251a9aa2d5872b04d7bc39fba2fef77746e94b1603ff4c38d4c1fa4d   2022-03-03T09:47:58.334721786Z   #0
d67741f50c19ee4790db2e819dd4ad32bf9c97214307cb746189c13a66137402   -1          running     /run/containerd/io.containerd.runtime.v2.task/k8s.io/d67741f50c19ee4790db2e819dd4ad32bf9c97214307cb746189c13a66137402   2022-03-03T09:47:57.929736794Z   #0

untrusted_workload_runtime 的方式

kata-pod-untrusted.yaml文件内容

apiVersion: v1
kind: Pod
metadata:
  name: kata-nginx-untrusted
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  containers:
  - name: nginx
    image: nginx:1.21.4
    ports:
    - containerPort: 80

启动kata的pod

[root@localhost critool]# kubectl apply -f kata-pod-untrusted.yaml
pod/kata-nginx-untrusted created
[root@localhost critool]# kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
kata-nginx             1/1     Running   0          12m
kata-nginx-untrusted   1/1     Running   0          7s