创建部署问题时，请参考下面模板，你提供的信息越多，越容易及时获得解答。如果未按模板创建问题，管理员有权关闭问题。
确保帖子格式清晰易读，用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题，不能指望别人花上半个小时给你解答。
操作系统信息
例如：虚拟机/物理机，Centos7.5/Ubuntu18.04，4C/8G
物理机，物理机，Ubuntu 20.04.5 LTS，56C/256G
Kubernetes版本信息
Client Version: version.Info{Major:“1”, Minor:“23”, GitVersion:“v1.23.10”, GitCommit:“7e54d50d3012cf3389e43b096ba35300f36e0817”, GitTreeState:“clean”, BuildDate:“2022-08-17T18:32:54Z”, GoVersion:“go1.17.13”, Compiler:“gc”, Platform:“linux/amd64”}
容器运行时
将 docker version / crictl version / nerdctl version 结果贴在下方
Client: Docker Engine - Community
Version: 24.0.3
API version: 1.43
Go version: go1.20.5
Git commit: 3713ee1
Built: Wed Jul 5 20:44:47 2023
OS/Arch: linux/amd64
Context: default
KubeSphere版本信息
例如：v2.1.1/v3.0.0。离线安装还是在线安装。在已有K8s上安装还是使用kk安装。
使用kk在线安装，版本v3.3
问题是什么
过程简述：原有3个node，角色全是master和worker，添加一个worker节点。
./kk add nodes -f sample.yaml
执行结果是集群3节点OK，新加节点失败，原因
查找etcd失败原因，如下
etcd其余2节点OK，剩余1个起不来，然后对所有节点的/var/lib/etcd目录备份，执行命令摘掉坏的etcd节点
etcdctl –endpoints=https://192.168.90.217:2379 member remove f186c58a5a1cec75
查看etcd集群状态，healthy
修改sample.yaml，去掉etcd坏的节点
执行./kk add nodes -f sample.yaml，集群丢失，k8s server退出，重启宿主机不恢复！
错误如下
20:46:46 CST [InstallKubeBinariesModule] Enable kubelet service
20:46:47 CST success: [shak8s06]
20:46:47 CST success: [shak8s04]
20:46:47 CST success: [shak8s05]
20:46:47 CST [InstallKubeBinariesModule] Generate kubelet env
20:46:48 CST [ERRO] check remote file exist err: strconv.Atoi: parsing “[sudo] password for CN\\tao.wang: \r\n0”: invalid syntax
20:46:48 CST [ERRO] check remote file exist err: strconv.Atoi: parsing “[sudo] password for CN\\tao.wang: \r\n0”: invalid syntax
20:46:48 CST [ERRO] check remote file exist err: strconv.Atoi: parsing “[sudo] password for CN\\tao.wang: \r\n0”: invalid syntax
20:46:49 CST success: [shak8s06]
20:46:49 CST success: [shak8s05]
20:46:49 CST success: [shak8s04]
20:46:49 CST [JoinNodesModule] Generate kubeadm config
20:46:51 CST [ERRO] check remote file exist err: strconv.Atoi: parsing “[sudo] password for CN\\tao.wang: \r\n0”: invalid syntax
20:46:51 CST [ERRO] check remote file exist err: strconv.Atoi: parsing “[sudo] password for CN\\tao.wang: \r\n0”: invalid syntax
20:46:51 CST [ERRO] check remote file exist err: strconv.Atoi: parsing “[sudo] password for CN\\tao.wang: \r\n0”: invalid syntax
20:46:53 CST success: [shak8s06]
20:46:53 CST success: [shak8s05]
20:46:53 CST success: [shak8s04]
20:46:53 CST [JoinNodesModule] Join control-plane node
20:46:53 CST stdout: [shak8s06]
[sudo] password for CN\tao.wang:
[discovery.bootstrapToken.token: Invalid value: "“: the bootstrap token is invalid, discovery.tlsBootstrapToken: Invalid value: ”": the bootstrap token is invalid]
To see the stack trace of this error execute with –v=5 or higher
20:46:53 CST stdout: [shak8s05]
[sudo] password for CN\tao.wang:
[discovery.bootstrapToken.token: Invalid value: "“: the bootstrap token is invalid, discovery.tlsBootstrapToken: Invalid value: ”": the bootstrap token is invalid]
To see the stack trace of this error execute with –v=5 or higher
20:46:53 CST stdout: [shak8s04]
[sudo] password for CN\tao.wang:
[discovery.bootstrapToken.token: Invalid value: "“: the bootstrap token is invalid, discovery.tlsBootstrapToken: Invalid value: ”": the bootstrap token is invalid]
To see the stack trace of this error execute with –v=5 or higher
20:46:54 CST stdout: [shak8s05]
[sudo] password for CN\tao.wang:
[preflight] Running pre-flight checks
W1212 20:46:54.264331 11027 removeetcdmember.go:80] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in “/var/lib/kubelet”
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]
The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d
The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the “iptables” command.
If your cluster was setup to utilize IPVS, run ipvsadm –clear (or similar)
to reset your system’s IPVS tables.
The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
20:46:54 CST message: [shak8s05]
join node failed: Failed to exec command: sudo -E /bin/bash -c “/usr/local/bin/kubeadm join –config=/etc/kubernetes/kubeadm-config.yaml –ignore-preflight-errors=FileExisting-crictl,ImagePull”
[sudo] password for CN\tao.wang:
[discovery.bootstrapToken.token: Invalid value: "“: the bootstrap token is invalid, discovery.tlsBootstrapToken: Invalid value: ”": the bootstrap token is invalid]
To see the stack trace of this error execute with –v=5 or higher: Process exited with status 3
20:46:54 CST retry: [shak8s05]
20:46:54 CST stdout: [shak8s06]
[sudo] password for CN\tao.wang:
[preflight] Running pre-flight checks
W1212 20:46:54.185690 14014 removeetcdmember.go:80] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in “/var/lib/kubelet”
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]
The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d
The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the “iptables” command.
If your cluster was setup to utilize IPVS, run ipvsadm –clear (or similar)
to reset your system’s IPVS tables.
The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
20:46:54 CST message: [shak8s06]
join node failed: Failed to exec command: sudo -E /bin/bash -c “/usr/local/bin/kubeadm join –config=/etc/kubernetes/kubeadm-config.yaml –ignore-preflight-errors=FileExisting-crictl,ImagePull”
[sudo] password for CN\tao.wang:
[discovery.bootstrapToken.token: Invalid value: "“: the bootstrap token is invalid, discovery.tlsBootstrapToken: Invalid value: ”": the bootstrap token is invalid]
To see the stack trace of this error execute with –v=5 or higher: Process exited with status 3
20:46:54 CST retry: [shak8s06]
20:46:54 CST stdout: [shak8s04]
[sudo] password for CN\tao.wang:
[preflight] Running pre-flight checks
W1212 20:46:54.275162 10763 removeetcdmember.go:80] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in “/var/lib/kubelet”
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]
The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d
The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the “iptables” command.
If your cluster was setup to utilize IPVS, run ipvsadm –clear (or similar)
to reset your system’s IPVS tables.
The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
20:46:54 CST message: [shak8s04]
join node failed: Failed to exec command: sudo -E /bin/bash -c “/usr/local/bin/kubeadm join –config=/etc/kubernetes/kubeadm-config.yaml –ignore-preflight-errors=FileExisting-crictl,ImagePull”
[sudo] password for CN\tao.wang:
[discovery.bootstrapToken.token: Invalid value: "“: the bootstrap token is invalid, discovery.tlsBootstrapToken: Invalid value: ”": the bootstrap token is invalid]
To see the stack trace of this error execute with –v=5 or higher: Process exited with status 3
20:46:54 CST retry: [shak8s04]
