• 安装部署
  • v3.2.1安装不了,需要手工拷贝master的/run/systemd/resolv.conf到worker节点,才可以安装成功

创建部署问题时,请参考下面模板,你提供的信息越多,越容易及时获得解答。如果未按模板创建问题,管理员有权关闭问题。
确保帖子格式清晰易读,用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题,不能指望别人花上半个小时给你解答。

操作系统信息
例如:虚拟机,Centos7.9,32C/64G

Kubernetes版本信息
Client Version: version.Info{Major:“1”, Minor:“21”, GitVersion:“v1.21.5”, GitCommit:“aea7bbadd2fc0cd689de94a54e5b7b758869d691”, GitTreeState:“clean”, BuildDate:“2021-09-15T21:10:45Z”, GoVersion:“go1.16.8”, Compiler:“gc”, Platform:“linux/amd64”}

Server Version: version.Info{Major:“1”, Minor:“21”, GitVersion:“v1.21.5”, GitCommit:“aea7bbadd2fc0cd689de94a54e5b7b758869d691”, GitTreeState:“clean”, BuildDate:“2021-09-15T21:04:16Z”, GoVersion:“go1.16.8”, Compiler:“gc”, Platform:“linux/amd64”}

容器运行时
docker version / crictl version / nerdctl version 结果贴在下方

Client: Docker Engine - Community

Version: 20.10.12

API version: 1.41

Go version: go1.16.12

Git commit: e91ed57

Built: Mon Dec 13 11:45:41 2021

OS/Arch: linux/amd64

Context: default

Experimental: true

Server: Docker Engine - Community

Engine:

Version: 20.10.12

API version: 1.41 (minimum version 1.12)

Go version: go1.16.12

Git commit: 459d0df

Built: Mon Dec 13 11:44:05 2021

OS/Arch: linux/amd64

Experimental: false

containerd:

Version: 1.4.12

GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d

runc:

Version: 1.0.2

GitCommit: v1.0.2-0-g52b36a2

docker-init:

Version: 0.19.0

GitCommit: de40ad0

KubeSphere版本信息
例如:v3.3.1 全新安装

问题是什么
v3.2.1安装出错,需要手工拷贝master的/run/systemd/resolv.conf到worker节点,才可以安装成功

报错摘要如下:

Jan 06 10:40:52 worker1 kubelet[25457]: E0106 10:40:52.139408 25457 kubelet.go:2211] “Container runtime network not ready” networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni coJan 06 10:40:51 worker1 kubelet[25457]: E0106 10:40:51.156442 25457 pod_workers.go:190] “Error syncing pod, skipping” err="failed to \“CreatePodSandbox\” for \“cilium-9wdrw_kube-system(b30c66a8-2b55-4eae-b46f-7d3ca1c0fae1)\” with CreateJan 06 10:40:51 worker1 kubelet[25457]: E0106 10:40:51.156256 25457 pod_workers.go:190] “Error syncing pod, skipping” err="failed to \“CreatePodSandbox\” for \“kube-proxy-mc7dw_kube-system(71bf2f35-61b8-4bb1-a01a-dc407e4875ea)\” with CrJan 06 10:40:51 worker1 kubelet[25457]: E0106 10:40:51.156243 25457 kuberuntime_manager.go:790] “CreatePodSandbox for pod failed” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod=“kube-system/cilium-9wdrw”

Jan 06 10:40:51 worker1 kubelet[25457]: E0106 10:40:51.156068 25457 kuberuntime_sandbox.go:41] “Failed to generate sandbox config for pod” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/cilium-9wJan 06 10:40:51 worker1 kubelet[25457]: E0106 10:40:51.156138 25457 kuberuntime_manager.go:790] “CreatePodSandbox for pod failed” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod=“kube-system/kube-proxy-mc7dw” Jan 06 10:40:51 worker1 kubelet[25457]: E0106 10:40:51.156041 25457 kuberuntime_sandbox.go:41] “Failed to generate sandbox config for pod” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/kube-proxJan 06 10:40:50 worker1 kubelet[25457]: E0106 10:40:50.155436 25457 pod_workers.go:190] “Error syncing pod, skipping” err="failed to \“CreatePodSandbox\” for \“nodelocaldns-dt664_kube-system(f4841504-5cc7-4842-8b66-41bc43015dd6)\” with Jan 06 10:40:50 worker1 kubelet[25457]: E0106 10:40:50.155274 25457 kuberuntime_manager.go:790] “CreatePodSandbox for pod failed” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/nodelocaldns-dt664Jan 06 10:40:50 worker1 kubelet[25457]: E0106 10:40:50.155190 25457 kuberuntime_sandbox.go:41] “Failed to generate sandbox config for pod” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/nodelocalJan 06 10:40:49 worker1 kubelet[25457]: I0106 10:40:49.227049 25457 cni.go:239] “Unable to update cni config” err=“no networks found in /etc/cni/net.d”

Jan 06 10:40:47 worker1 kubelet[25457]: E0106 10:40:47.114289 25457 kubelet.go:2211] “Container runtime network not ready” networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni coJan 06 10:40:44 worker1 kubelet[25457]: I0106 10:40:44.226215 25457 cni.go:239] “Unable to update cni config” err=“no networks found in /etc/cni/net.d”

Jan 06 10:40:43 worker1 kubelet[25457]: E0106 10:40:43.156915 25457 pod_workers.go:190] “Error syncing pod, skipping” err="failed to \“CreatePodSandbox\” for \"cilium-operator-8567457f7b-qkpbq_kube-system(3db957e2-32c3-4c4e-8346-b4426b6Jan 06 10:40:43 worker1 kubelet[25457]: E0106 10:40:43.156634 25457 kuberuntime_manager.go:790] “CreatePodSandbox for pod failed” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/cilium-operator-85Jan 06 10:40:43 worker1 kubelet[25457]: E0106 10:40:43.156517 25457 kuberuntime_sandbox.go:41] “Failed to generate sandbox config for pod” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/cilium-opJan 06 10:40:42 worker1 kubelet[25457]: E0106 10:40:42.088992 25457 kubelet.go:2211] “Container runtime network not ready” networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni coJan 06 10:40:40 worker1 kubelet[25457]: E0106 10:40:40.155721 25457 pod_workers.go:190] “Error syncing pod, skipping” err="failed to \“CreatePodSandbox\” for \“cilium-9wdrw_kube-system(b30c66a8-2b55-4eae-b46f-7d3ca1c0fae1)\” with CreateJan 06 10:40:40 worker1 kubelet[25457]: E0106 10:40:40.155523 25457 kuberuntime_manager.go:790] “CreatePodSandbox for pod failed” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod=“kube-system/cilium-9wdrw”

Jan 06 10:40:40 worker1 kubelet[25457]: E0106 10:40:40.155411 25457 kuberuntime_sandbox.go:41] “Failed to generate sandbox config for pod” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/cilium-9wJan 06 10:40:39 worker1 kubelet[25457]: I0106 10:40:39.225606 25457 cni.go:239] “Unable to update cni config” err=“no networks found in /etc/cni/net.d”

Jan 06 10:40:37 worker1 kubelet[25457]: E0106 10:40:37.155836 25457 pod_workers.go:190] “Error syncing pod, skipping” err="failed to \“CreatePodSandbox\” for \“kube-proxy-mc7dw_kube-system(71bf2f35-61b8-4bb1-a01a-dc407e4875ea)\” with CrJan 06 10:40:37 worker1 kubelet[25457]: E0106 10:40:37.155592 25457 kuberuntime_manager.go:790] “CreatePodSandbox for pod failed” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod=“kube-system/kube-proxy-mc7dw” Jan 06 10:40:37 worker1 kubelet[25457]: E0106 10:40:37.155474 25457 kuberuntime_sandbox.go:41] “Failed to generate sandbox config for pod” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/kube-proxJan 06 10:40:37 worker1 kubelet[25457]: E0106 10:40:37.064371 25457 kubelet.go:2211] “Container runtime network not ready” networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni coJan 06 10:40:36 worker1 kubelet[25457]: E0106 10:40:36.154461 25457 pod_workers.go:190] “Error syncing pod, skipping” err="failed to \“CreatePodSandbox\” for \“nodelocaldns-dt664_kube-system(f4841504-5cc7-4842-8b66-41bc43015dd6)\” with Jan 06 10:40:36 worker1 kubelet[25457]: E0106 10:40:36.154376 25457 kuberuntime_manager.go:790] “CreatePodSandbox for pod failed” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/nodelocaldns-dt664Jan 06 10:40:36 worker1 kubelet[25457]: E0106 10:40:36.154329 25457 kuberuntime_sandbox.go:41] “Failed to generate sandbox config for pod” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/nodelocalJan 06 10:40:34 worker1 kubelet[25457]: I0106 10:40:34.225280 25457 cni.go:239] “Unable to update cni config” err=“no networks found in /etc/cni/net.d”

上面错误是worker节点上的出错信息, worker节点错误信息里还有说no networks found in /etc/cni/net.d,查看woker节点这个目录下也没有文件

Jan 06 10:48:09 worker1 kubelet[25457]: E0106 10:48:09.364874 25457 kubelet.go:2211] “Container runtime network not ready” networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni coJan 06 10:48:09 worker1 kubelet[25457]: I0106 10:48:09.309215 25457 cni.go:239] “Unable to update cni config” err=“no networks found in /etc/cni/net.d”

Jan 06 10:48:04 worker1 kubelet[25457]: E0106 10:48:04.340778 25457 kubelet.go:2211] “Container runtime network not ready” networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni coJan 06 10:48:04 worker1 kubelet[25457]: I0106 10:48:04.308599 25457 cni.go:239] “Unable to update cni config” err=“no networks found in /etc/cni/net.d”

Jan 06 10:48:01 worker1 kubelet[25457]: E0106 10:48:01.157751 25457 pod_workers.go:190] “Error syncing pod, skipping” err="failed to \“CreatePodSandbox\” for \“nodelocaldns-dt664_kube-system(f4841504-5cc7-4842-8b66-41bc43015dd6)\” with Jan 06 10:48:01 worker1 kubelet[25457]: E0106 10:48:01.157459 25457 kuberuntime_manager.go:790] “CreatePodSandbox for pod failed” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/nodelocaldns-dt664Jan 06 10:48:01 worker1 kubelet[25457]: E0106 10:48:01.157349 25457 kuberuntime_sandbox.go:41] “Failed to generate sandbox config for pod” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/nodelocalJan 06 10:48:01 worker1 kubelet[25457]: E0106 10:48:01.155508 25457 pod_workers.go:190] “Error syncing pod, skipping” err="failed to \“CreatePodSandbox\” for \“kube-proxy-mc7dw_kube-system(71bf2f35-61b8-4bb1-a01a-dc407e4875ea)\” with CrJan 06 10:48:01 worker1 kubelet[25457]: E0106 10:48:01.155323 25457 kuberuntime_manager.go:790] “CreatePodSandbox for pod failed” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod=“kube-system/kube-proxy-mc7dw” Jan 06 10:48:01 worker1 kubelet[25457]: E0106 10:48:01.155237 25457 kuberuntime_sandbox.go:41] “Failed to generate sandbox config for pod” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/kube-proxJan 06 10:47:59 worker1 kubelet[25457]: E0106 10:47:59.313212 25457 kubelet.go:2211] “Container runtime network not ready” networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni coJan 06 10:47:59 worker1 kubelet[25457]: I0106 10:47:59.307873 25457 cni.go:239] “Unable to update cni config” err=“no networks found in /etc/cni/net.d”

Jan 06 10:47:59 worker1 kubelet[25457]: E0106 10:47:59.156740 25457 pod_workers.go:190] “Error syncing pod, skipping” err="failed to \“CreatePodSandbox\” for \“cilium-9wdrw_kube-system(b30c66a8-2b55-4eae-b46f-7d3ca1c0fae1)\” with CreateJan 06 10:47:59 worker1 kubelet[25457]: E0106 10:47:59.156492 25457 kuberuntime_manager.go:790] “CreatePodSandbox for pod failed” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod=“kube-system/cilium-9wdrw”

Jan 06 10:47:59 worker1 kubelet[25457]: E0106 10:47:59.156351 25457 kuberuntime_sandbox.go:41] “Failed to generate sandbox config for pod” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/cilium-9wJan 06 10:47:55 worker1 kubelet[25457]: E0106 10:47:55.156259 25457 pod_workers.go:190] “Error syncing pod, skipping” err="failed to \“CreatePodSandbox\” for \"cilium-operator-8567457f7b-qkpbq_kube-system(3db957e2-32c3-4c4e-8346-b4426b6Jan 06 10:47:55 worker1 kubelet[25457]: E0106 10:47:55.155918 25457 kuberuntime_manager.go:790] “CreatePodSandbox for pod failed” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/cilium-operator-85Jan 06 10:47:55 worker1 kubelet[25457]: E0106 10:47:55.155793 25457 kuberuntime_sandbox.go:41] “Failed to generate sandbox config for pod” err=“open /run/systemd/resolve/resolv.conf: no such file or directory” pod="kube-system/cilium-opJan 06 10:47:54 worker1 kubelet[25457]: I0106 10:47:54.306851 25457 cni.go:239] “Unable to update cni config” err=“no networks found in /etc/cni/net.d”

在kubelet启动一个pod前,需要为该pod创建一个合适的运行环境,该运行环境被称为sanbox。

在sandbox的多项配置中,包含了需要从宿主机节点继承的一部分dns配置,如search domain。

此时,kubelet需要读取宿主机的DNS配置文件,将其中的配置写入pod sandbox中。

至于宿主机的DNS配置文件位于何处,是由kubelet的配置文件声明的,该文件默认位于/var/lib/kubelet/config.yaml下,配置项名为resolvConf,可通过

cat /var/lib/kubelet/config.yaml | grep resolvConf

查看该配置项的值。 从你贴出的报错信息看,该路径应该为 /run/systemd/resolve/resolv.conf

此路径表明你的系统使用了systemd-resolved,而当kubeadm检测到该程序的存在时,会将resolvConf指向此路径。

正常情况下,/run/systemd/resolve/resolv.conf应该由systemd-resolved创建并维护,因此kubelet能正常读取。但在你的worker节点中,很显然,从报错可知,该文件的缺失导致了kubelet创建pod sandbox的失败。

手动创建该文件能临时绕过该问题,但并不是建议的方案。你的系统中,systemd-resolved很可能没有正常运行,你可通过

systemctl status systemd-resolved

查看该服务的状态,并进一步解决问题。 在最简单的情况下,它可能只是停止运行了,此时执行

systemctl start systemd-resolved

重新启动即可。

至于报错

Unable to update cni config” err=“no networks found in /etc/cni/net.d

是因为CNI pod无法启动,因此节点的CNI未初始化,在解决上述问题后应该也能自行恢复。

商业产品与合作咨询