阅读量:1
K8s The connection to the server 192.168.56.120:6443 was refused报错解决
背景说明
自己本地虚拟机有一套使用kuebadm部署的3节点k8s集群，因为很长时间不使用了，今天登陆后在使用kubectl命令时一直报The connection to the server 192.168.56.120:6443 was refused - did you specify the right host or port?错误，接下来就将解决的整个过程分享给大家。
环境说明
操作系统信息以及k8s集群版本信息请看下面
[root@k8s-master][/usr/local] $cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core) [root@k8s-master][/usr/local] $kubectl version Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"} 部署信息
| 序号 | ip | 角色 |
|---|---|---|
| 1 | 192.168.56.120 | k8s-master |
| 2 | 192.168.56.121 | k8s-node1 |
| 3 | 192.168.56.122 | k8s-node2 |
场景复现
[root@k8s-master][/usr/local] $kubectl get node The connection to the server 192.168.56.120:6443 was refused - did you specify the right host or port? 问题排查
基础环境排查
#1 基础环境排查 #1.0 端口排查（端口已经不存在） [root@k8s-master][~] $ss -antpl |grep 6443 #1.1 查看内存（都够用） [root@k8s-master][~] $free -h total used free shared buff/cache available Mem: 4.7G 1.2G 366M 58M 3.1G 3.1G Swap: 0B 0B 0B #1.2 查看存储（也没问题） [root@k8s-master][~] $df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 2.4G 0 2.4G 0% /dev tmpfs 2.4G 0 2.4G 0% /dev/shm tmpfs 2.4G 58M 2.3G 3% /run tmpfs 2.4G 0 2.4G 0% /sys/fs/cgroup /dev/mapper/centos-root 37G 22G 16G 59% / #1.3 查看防火墙（防火墙是关闭的） [root@k8s-master][~] $systemctl status firewalld ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:firewalld(1) 组件服务排查
#2 排查组件 #2.1 查看docker信息（没有报错信息） [root@k8s-master][~] $systemctl status docker #2.2 查看kubelet（发现master节点丢失的信息，但是内容不详细） [root@k8s-master][/usr/local] $systemctl status kubelet Jul 17 10:45:45 k8s-master kubelet[14485]: E0717 10:45:45.603977 14485 kubelet.go:2422] "Error getting node" err="node \"k8s-master\" not found" Jul 17 10:45:45 k8s-master kubelet[14485]: E0717 10:45:45.634017 14485 controller.go:144] failed to ensure lease exists, will retry in 7s, error: Get "https://192.168....ion refused #3 查看etcd情况 [root@k8s-master][~] $docker ps |grep k8s_etcd_etcd #从以下的信息可以看出证书已经过期了，在确认一遍 [root@k8s-master][~] $docker logs dc2c0e42972e {"level":"warn","ts":"2024-07-17T02:42:22.385Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34074","server-name":"","error":"remote error: tls: bad certificate"} {"level":"warn","ts":"2024-07-17T02:43:33.346Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34428","server-name":"","error":"tls: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2024-07-17T02:43:33Z is after 2024-03-14T07:34:55Z"} #4 查看证书过期情况（发现已经过期了，到此找到报错根因） [root@k8s-master][~] $kubeadm certs check-expiration 根因定位
#从以下的信息可以看出证书过期，导致api-server服务未启动 [root@k8s-master][~] $docker logs dc2c0e42972e {"level":"warn","ts":"2024-07-17T02:42:22.385Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34074","server-name":"","error":"remote error: tls: bad certificate"} {"level":"warn","ts":"2024-07-17T02:43:33.346Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34428","server-name":"","error":"tls: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2024-07-17T02:43:33Z is after 2024-03-14T07:34:55Z"} 问题解决
既然已经定位到是证书过期的问题，那接下来就针对证书过期进行解决。
安装go环境
# 如果 k8s 版本较高，则需要 go 语言版本 1.19 以上(尽量使用和k8s版本一致的go版本，要不会报很多错误) kubectl version [root@k8s-master][~] $kubectl version Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"} #1 下载 go 语言软件包 mkdir -p /root/pac cd /root/pac wget https://studygolang.com/dl/golang/go1.17.3.linux-amd64.tar.gz #2 解压并配置环境变量 tar xf go1.19.1.linux-amd64.tar.gz -C /usr/local vim /etc/profile export GOROOT=/usr/local/go export PATH=$PATH:/usr/local/go/bin export GOPATH=/go source /etc/profile #3 验证 [root@k8s-master][~] $go version go version go1.17.3 linux/amd64 下载k8s源码
#1 查看当前环境 k8s 版本 kubectl version #2 下载源码 cd /root/pac wget https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.23.0.tar.gz #2 解压 tar xf v1.23.0.tar.gz 修改CA有效期
#1 进入到指定目录 [root@k8s-master][~/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert] # cd /root/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert #2 备份cert.go [root@k8s-master][~/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert] # cp cert.go{,.bak} #3 修改cert.go [root@k8s-master][~/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert] # vim cert.go // NewSelfSignedCACert creates a CA certificate func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) { now := time.Now() tmpl := x509.Certificate{ SerialNumber: new(big.Int).SetInt64(0), Subject: pkix.Name{ CommonName: cfg.CommonName, Organization: cfg.Organization, }, DNSNames: []string{cfg.CommonName}, NotBefore: now.UTC(), //NotAfter: now.Add(duration365d * 10).UTC(), // 修改为 100 年，默认为 10 年 NotAfter: now.Add(duration365d * 100).UTC(), KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign, BasicConstraintsValid: true, IsCA: true, } certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key) if err != nil { return nil, err } return x509.ParseCertificate(certDERBytes) } 修改证书有效期
[root@k8s-master][~/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert] # cd /root/xxx/go/kubernetes-1.23.0/cmd/kubeadm/app/constants #2 备份constants.go [root@k8s-master][~/xxx/go/kubernetes-1.23.0/cmd/kubeadm/app/constants] # cp constants.go{,.bak} #3 修改文件 [root@k8s-master][~/xxx/go/kubernetes-1.23.0/cmd/kubeadm/app/constants] # vim constants.go const ( // KubernetesDir is the directory Kubernetes owns for storing various configuration files KubernetesDir = "/etc/kubernetes" // ManifestsSubDirName defines directory name to store manifests ManifestsSubDirName = "manifests" // TempDirForKubeadm defines temporary directory for kubeadm // should be joined with KubernetesDir. TempDirForKubeadm = "tmp" // CertificateValidity defines the validity for all the signed certificates generated by kubeadm //CertificateValidity = time.Hour * 24 * 365 //修改为 100 年，默认为 1 年 CertificateValidity = time.Hour * 24 * 365 * 100 // DefaultCertificateDir defines default certificate directory DefaultCertificateDir = "pki" // CACertAndKeyBaseName defines certificate authority base name CACertAndKeyBaseName = "ca" // CACertName defines certificate name CACertName = "ca.crt" // CAKeyName defines certificate name 编译 kubeadm
# 提前装rsync yum install rsync -y #1 重新编译kubeadm [root@k8s-master][~/xxx/go/kubernetes-1.23.0] # pwd /root/xxx/go/kubernetes-1.23.0 [root@k8s-master][~/xxx/go/kubernetes-1.23.0] # make all WHAT=cmd/kubeadm GOFLAGS=-v #2 替换原来的 kubeadm [root@k8s-master][~] # mv /usr/bin/kubeadm{,.bak} [root@k8s-master][~/xxx/go/kubernetes-1.23.0] # pwd /root/xxx/go/kubernetes-1.23.0 [root@k8s-master][~/xxx/go/kubernetes-1.23.0] # ll _output/local/bin/linux/amd64/kubeadm -rwxr-xr-x. 1 root root 43667456 Mar 6 11:26 _output/local/bin/linux/amd64/kubeadm [root@k8s-master][~/xxx/go/kubernetes-1.23.0] # cp _output/local/bin/linux/amd64/kubeadm /usr/bin/kubeadm [root@k8s-master][~/xxx/go/kubernetes-1.23.0] # chmod +x /usr/bin/kubeadm 更新证书
#1 备份原文件 [root@k8s-master][~] # cp -a /etc/kubernetes{,.bak_20240717} #2 更新证书 [root@k8s-master][/etc/kubernetes] # kubeadm certs renew all [renew] Reading configuration from the cluster... [renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed certificate embedded in the kubeconfig file for the controller manager to use renewed certificate for liveness probes to healthcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed certificate embedded in the kubeconfig file for the scheduler manager to use renewed Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates. 检查结果
[root@k8s-master][~] $kubeadm certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Jun 23, 2124 02:49 UTC 99y no apiserver Jun 23, 2124 02:49 UTC 99y ca no apiserver-etcd-client Jun 23, 2124 02:49 UTC 99y etcd-ca no apiserver-kubelet-client Jun 23, 2124 02:49 UTC 99y ca no controller-manager.conf Jun 23, 2124 02:49 UTC 99y no etcd-healthcheck-client Jun 23, 2124 02:49 UTC 99y etcd-ca no etcd-peer Jun 23, 2124 02:49 UTC 99y etcd-ca no etcd-server Jun 23, 2124 02:49 UTC 99y etcd-ca no front-proxy-client Jun 23, 2124 02:49 UTC 99y front-proxy-ca no scheduler.conf Jun 23, 2124 02:49 UTC 99y no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Mar 12, 2033 07:34 UTC 8y no etcd-ca Mar 12, 2033 07:34 UTC 8y no front-proxy-ca Mar 12, 2033 07:34 UTC 8y no 再次使用kubectl命令，看看问题是否解决
[root@k8s-master][~] $kubectl get node NAME STATUS ROLES AGE VERSION k8s-master Ready control-plane,master 489d v1.23.0 k8s-node1 Ready <none> 489d v1.23.0 k8s-node2 Ready <none> 489d v1.23.0 至此，问题得到解决。
