istio init CrashLoopBackOff解决方法

学习istio途中，按照官网提供的例子，创建一个bookinfo项目，当执行完成kubectl apply -f bookinfo.yaml之后发现启动失败[root@k8s-master kube]# kubectl get podNAMEREADYSTATUSRESTARTSAGEdetails-v1-79f774bdb9-sgfk50/...

闹玩儿扣眼珠子

4089人浏览 · 2021-12-27 17:27:58

闹玩儿扣眼珠子 · 2021-12-27 17:27:58 发布

学习istio途中，按照官网提供的例子，创建一个bookinfo项目，当执行完成kubectl apply -f bookinfo.yaml之后发现启动失败

[root@k8s-master kube]# kubectl get pod
NAME                                  READY   STATUS                  RESTARTS        AGE
details-v1-79f774bdb9-sgfk5           0/2     Init:CrashLoopBackOff   7 (4m31s ago)   15m
nginx                                 0/2     Init:CrashLoopBackOff   5 (2m44s ago)   5m49s
nginx2                                0/2     Init:CrashLoopBackOff   1 (4s ago)      5s
productpage-v1-6b746f74dc-4hdpp       0/2     Init:CrashLoopBackOff   7 (4m13s ago)   15m
prometheus-operator-7ccf6dfc8-vrzmr   1/1     Running                 3 (4d1h ago)    32d
ratings-v1-b6994bb9-84vsx             0/2     Init:CrashLoopBackOff   7 (4m17s ago)   15m
reviews-v1-545db77b95-w4ntv           0/2     Init:CrashLoopBackOff   7 (4m21s ago)   15m
reviews-v2-7bf8c9648f-qh56t           0/2     Init:CrashLoopBackOff   7 (4m19s ago)   15m
reviews-v3-84779c7bbc-hvskz           0/2     Init:CrashLoopBackOff   7 (4m43s ago)   15m

所有的pod的状态都为Init:CrashLoopBackOff，查看pod init container日志如下：

[root@k8s-master kube]# kubectl logs productpage-v1-6b746f74dc-4hdpp istio-init
2021-12-27T09:06:18.691915Z     info    Istio iptables environment:
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_EXCLUDE_INTERFACES=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
ISTIO_META_DNS_CAPTURE=
2021-12-27T09:06:18.691971Z     info    Istio iptables variables:
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15021,15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBE_VIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false
DNS_CAPTURE=false
CAPTURE_ALL_DNS=false
DNS_SERVERS=[],[]
OUTPUT_PATH=
NETWORK_NAMESPACE=
CNI_MODE=false
EXCLUDE_INTERFACES=

2021-12-27T09:06:18.692095Z     info    Writing following contents to rules file: /tmp/iptables-rules-1640595978692007050.txt1847614466
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
2021-12-27T09:06:18.692140Z     info    Running command: iptables-restore --noflush /tmp/iptables-rules-1640595978692007050.txt1847614466
2021-12-27T09:06:18.694937Z     error   Command error output: xtables parameter problem: iptables-restore: unable to initialize table 'nat'

Error occurred at line: 1
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
2021-12-27T09:06:18.694953Z     error   Failed to execute: iptables-restore --noflush /tmp/iptables-rules-1640595978692007050.txt1847614466, exit status 2

最后两行有明显错误，istio-init在执行iptables命令的时候报错，中文翻译是：命令错误输出：xtables参数问题：iptables还原：无法初始化表“nat”

问题原因：iptables模块未被加载，所以我们可以尝试在所有k8s集群节点上加载iptables模块

直接加载模块

[root@k8s-master kube]# modprobe ip_tables
[root@k8s-master kube]# modprobe iptable_filter

执行完成之后可以通过命令查看一下模块加载情况

[root@k8s-master ~]# lsmod |grep -E "ip_tables|iptable_filter"
iptable_filter         16384  0
ip_tables              28672  2 iptable_filter,iptable_nat

先验证一下

[root@k8s-master kube]# kubectl get pod -o wide
NAME                                  READY   STATUS                  RESTARTS        AGE     IP                NODE         NOMINATED NODE   READINESS GATES
details-v1-79f774bdb9-sgfk5           0/2     Init:CrashLoopBackOff   8 (2m55s ago)   18m     100.97.125.21     k8s-node01   <none>           <none>
nginx                                 0/2     Init:CrashLoopBackOff   6 (3m17s ago)   9m17s   100.97.125.17     k8s-node01   <none>           <none>
nginx2                                0/2     Init:CrashLoopBackOff   5 (34s ago)     3m33s   100.97.125.38     k8s-node01   <none>           <none>
productpage-v1-6b746f74dc-4hdpp       0/2     Init:CrashLoopBackOff   8 (2m38s ago)   18m     100.97.125.12     k8s-node01   <none>           <none>
prometheus-operator-7ccf6dfc8-vrzmr   1/1     Running                 3 (4d1h ago)    32d     100.116.59.107    k8s-master   <none>           <none>
ratings-v1-b6994bb9-84vsx             0/2     Init:CrashLoopBackOff   8 (2m37s ago)   18m     100.107.114.157   k8s-node03   <none>           <none>
reviews-v1-545db77b95-w4ntv           0/2     Init:CrashLoopBackOff   8 (2m38s ago)   18m     100.107.114.159   k8s-node03   <none>           <none>
reviews-v2-7bf8c9648f-qh56t           0/2     Init:CrashLoopBackOff   8 (2m35s ago)   18m     100.97.125.20     k8s-node01   <none>           <none>
reviews-v3-84779c7bbc-hvskz           2/2     Running                 0               18m     100.116.59.124    k8s-master   <none>           <none>

可以看到master节点已经running，接下来在所有节点都执行，可以手动每个节点执行，也可以使用ansible批量执行（需提前定义分组）

[root@k8s-master kube]# ansible node -m shell -a "modprobe ip_tables;modprobe iptable_filter"
192.168.3.52 | CHANGED | rc=0 >>

192.168.3.53 | CHANGED | rc=0 >>

192.168.3.51 | CHANGED | rc=0 >>

执行之后需要稍等会，再次查看pod状态，也可以删除pod重新拉起

[root@k8s-master kube]# kubectl get pod
NAME                                  READY   STATUS    RESTARTS       AGE
details-v1-79f774bdb9-sgfk5           2/2     Running   0              42m
nginx                                 2/2     Running   0              32m
nginx2                                2/2     Running   0              26m
productpage-v1-6b746f74dc-4pqpc       2/2     Running   0              2m8s
prometheus-operator-7ccf6dfc8-vrzmr   1/1     Running   3 (4d1h ago)   32d
ratings-v1-b6994bb9-84vsx             2/2     Running   0              42m
reviews-v1-545db77b95-w4ntv           2/2     Running   0              42m
reviews-v2-7bf8c9648f-qh56t           1/2     Running   0              42m
reviews-v3-84779c7bbc-hvskz           2/2     Running   0              42m

刚刚执行的modprobe命令为手动加载，当重启操作系统后还需要再次加载，为了永久生效，我们可以在

/etc/sysconfig/modules/

创建一个新文件然后写入如下内容，并设置权限

[root@k8s-master ~]# cat /etc/sysconfig/modules/iptables.modules
modprobe -- ip_tables
modprobe -- iptable_filter
[root@k8s-master ~]# chmod 755 /etc/sysconfig/modules/iptables.modules   #设置权限
[root@k8s-master ~]# sh /etc/sysconfig/modules/iptables.modules            #临时生效

重启后再次查看依然生效，centos 8亲测有效

Last login: Wed Dec 29 12:16:01 2021 from 192.168.3.202
[root@k8s-master ~]# lsmod |grep -E "ip_tables|iptable_filter"
iptable_filter         16384  0
ip_tables              28672  2 iptable_filter,iptable_nat