OpenStack虚拟机部署K8S集群，使用Cinder提供PV报错：mount: special device does not exist
1、出错信息kubelet挂载pv卷失败，从出错信息来看失败原因是设备不存在# kubectl describe pods testpvcpod-cinderMounting command: systemd-runMounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/7...
·
1、出错信息
kubelet挂载pv卷失败，从出错信息来看失败原因是设备不存在
# kubectl describe pods testpvcpod-cinder
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/7d0ca177-2624-4b4e-9353-222fb491f8cc /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
Output: Running scope as unit run-18372.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/7d0ca177-2624-4b4e-9353-222fb491f8cc does not exist
Warning FailedMount 6m6s kubelet, k8s-n1 Unable to attach or mount volumes: unmounted volumes=[testpv-cinder], unattached volumes=[default-token-lrww8 testpv-cinder]: timed out waiting for the condition
Warning FailedMount 116s (x4 over 6m1s) kubelet, k8s-n1 (combined from similar events): MountVolume.SetUp failed for volume "pvc-203a9760-85e9-4f58-91cd-081e9be96946" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/7d0ca177-2624-4b4e-9353-222fb491f8cc /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
Output: Running scope as unit run-21619.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/7d0ca177-2624-4b4e-9353-222fb491f8cc does not exist
Warning FailedMount 93s kubelet, k8s-n1 Unable to attach or mount volumes: unmounted volumes=[testpv-cinder], unattached volumes=[testpv-cinder default-token-lrww8]: timed out waiting for the condition
2、带Cinder PV pod部署流程
2.1、K8S集群信息
在OpenStack集群中部署了4个虚拟机，分别用于部署K8S Master和Node节点
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-m1 Ready master 112s v1.17.0
k8s-m2 Ready master 87s v1.17.0
k8s-m3 Ready master 21s v1.17.0
k8s-n1 Ready <none> 13s v1.17.0
2.2、创建Cinder Storage Class
# cat cinder-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: standard
provisioner: kubernetes.io/cinder
reclaimPolicy: Delete
volumeBindingMode: Immediate
# kubectl apply -f cinder-sc.yaml
storageclass.storage.k8s.io/standard created
# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
standard kubernetes.io/cinder Delete Immediate false 5s
2.3、创建带动态Cinder PV的POD
# cat busybox-cinder-dym.yaml
apiVersion: v1
kind: Pod
metadata:
name: testpvcpod-cinder
spec:
containers:
- name: busybox-pvc-cinder
image: busybox
command:
- sh
- -c
- "while true; do date > /home/loop; sleep 10m; done"
imagePullPolicy: IfNotPresent
volumeMounts:
- name: testpv-cinder
mountPath: /home
volumes:
- name: testpv-cinder
persistentVolumeClaim:
claimName: cinder-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cinder-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: standard
resources:
requests:
storage: 1Gi
# kubectl apply -f busybox-cinder-dym.yaml
pod/testpvcpod-cinder created
persistentvolumeclaim/cinder-pvc created
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
nfs-pv 1Gi RWX Recycle Bound default/nfs-pvc slow 26h
pvc-ce961d70-ea7f-4f23-9cf7-7de782951136 1Gi RWO Delete Bound default/cinder-pvc standard 7m46s
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cinder-pvc Bound pvc-ce961d70-ea7f-4f23-9cf7-7de782951136 1Gi RWO standard 7m49s
nfs-pvc Bound nfs-pv 1Gi RWX slow 26h
PV、PVC和POD都已经创建好，但是POD处于ContainerCreating状态
# kubectl get pods
NAME READY STATUS RESTARTS AGE
testpvcpod 1/1 Running 0 25h
testpvcpod-cinder 0/1 ContainerCreating 0 7m45s
没有running的原因就是因为本文开头的问题
3、解决问题
3.1、查看Cinder PV卷是否挂载到节点上
k8s-n1节点上的/dev/vdb就是这个PV
[root@k8s-n1 ~]# fdisk -l
Disk /dev/vda: 54.8 GB, 54760833024 bytes, 106954752 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00072170
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 2099199 1048576 83 Linux
/dev/vda2 2099200 104857599 51379200 8e Linux LVM
Disk /dev/mapper/centos-root: 47.2 GB, 47240445952 bytes, 92266496 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-swap: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/vdb: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
3.2、PVC对应的目录不存在
[root@k8s-n1 ~]# ls -l /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
ls: cannot access /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946: No such file or directory
3.3、手工创建目录
[root@k8s-n1 ~]# mkdir -p /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
[root@k8s-n1 ~]# ls -l /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
total 0
3.4、手工挂载卷
[root@k8s-n1 ~]# mkfs.xfs /dev/vdb
meta-data=/dev/vdb isize=512 agcount=4, agsize=65536 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=262144, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@k8s-n1 ~]# mount /dev/vdb /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
[root@k8s-n1 ~]# df
/dev/vdb 1038336 32992 1005344 4% /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
3.5、POD处于running状态
# kubectl get pods
NAME READY STATUS RESTARTS AGE
testpvcpod 1/1 Running 0 42h
testpvcpod-cinder 1/1 Running 0 9m54s
查看卷已经挂载到pod上
# kubectl exec -it testpvcpod-cinder df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 46110724 5801528 40309196 13% /
tmpfs 65536 0 65536 0% /dev
tmpfs 4004356 0 4004356 0% /sys/fs/cgroup
/dev/vdb 1038336 32996 1005340 3% /home
4、问题分析
因为本环境用的是K8S原生的Cinder驱动，没有使用CSI，而原生的Cinder驱动在1.17.0版本及后续版本将逐渐废弃，转为CSI。所以这块代码可能存在bug。即k8s master虽然调用OpenStack接口为pod创建并挂载了卷，但是kubelet不知道卷信息，无法做自动mount操作。
后续有空分析下这块代码，并验证下CSI Cinder的使用
华为开发者空间，是为全球开发者打造的专属开发空间，汇聚了华为优质开发资源及工具，致力于让每一位开发者拥有一台云主机，基于华为根生态开发、创新。
更多推荐
0
0- 0
已为社区贡献14条内容
所有评论(0)