手动修复v1.14-v1.15版本k8s的issue:76956

该issue见 https://github.com/kubernetes/kubernetes/issues/76956 , apiserver和controller-manager每次start的时候会刷下面信息。不影响使用,但是有强迫症可以按照本文思路手动修改

E0423 17:35:58.491576       1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0423 17:35:58.491625       1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0423 17:35:58.491901       1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0423 17:35:58.492191       1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0423 17:35:58.492367       1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0423 17:35:58.492507       1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted

从这个issue我们可以找到对应的修复pr https://github.com/kubernetes/kubernetes/pull/77553 , 我们看到是metrics信息被注册了两次的错误输出。

但是该pr只被merge到了 v1.16 以上 https://github.com/kubernetes/kubernetes/commit/f101466d2e4d96854c80f58203de2cc6b5aaeb6a#diff-60187c78fbca5a995fd7d2ac913eb6c8 , 意味着1.14,1.15都存在这个。

master (#77553)  v1.17.0-alpha.2  v1.17.0-alpha.1 v1.17.0-alpha.0 v1.16.3-beta.0 v1.16.2 v1.16.2-beta.0 v1.16.1 v1.16.1-beta.0 v1.16.0 v1.16.0-rc.2 v1.16.0-rc.1 v1.16.0-beta.2 v1.16.0-beta.1 v1.16.0-beta.0 v1.16.0-alpha.3 v1.16.0-alpha.2

实战

下载源码

手动下载源码编译,这里我使用 v1.15.5 来做示范。我们使用容器编译,这里我使用 4c8gcentos7.6 + docker-ce-18.09 。想办法把源码拉下来

git clone https://github.com/kubernetes/kubernetes -b v1.15.5
cd kubernetes
git checkout -b v1.15.5

编译前准备

因为我们拉取后修改了源码,编译出的version信息会是下面带 -dirty

$ kube-controller-manager --version
Kubernetes v1.15.5-dirty

可以执行下面命令去掉 -dirty

sed -ri 's#KUBE_GIT_TREE_STATE="dirty"#KUBE_GIT_TREE_STATE="clean"#g' hack/lib/version.sh

另外如果有需求也编译docker镜像的话可以先提前准备本地镜像,修改 kubernetes/build/lib/release.sh

"${DOCKER[@]}" build --pull -q -t "${docker_image_tag}" "${docker_build_path}" >/dev/null
修改为下面
"${DOCKER[@]}" build -q -t "${docker_image_tag}" "${docker_build_path}" >/dev/null

因为我们使用docker镜像编译,需要提前准备镜像

[root@k8s-m1 kubernetes]# grep FROM  _output/images/kube-build\:build-7db96ab759-5-v1.12.10-1/Dockerfile 
FROM k8s.gcr.io/kube-cross:v1.12.10-1

拉取该镜像

curl -s https://zhangguanzhang.github.io/bash/pull.sh | bash -s k8s.gcr.io/kube-cross:v1.12.10-1

把merge打patch

merge的url为 https://github.com/kubernetes/kubernetes/commit/f101466d2e4d96854c80f58203de2cc6b5aaeb6a#diff-60187c78fbca5a995fd7d2ac913eb6c8

# 以及后面的改为patch执行下面命令打patch

[root@k8s-m1 kubernetes]# wget https://github.com/kubernetes/kubernetes/commit/f101466d2e4d96854c80f58203de2cc6b5aaeb6a.patch
[root@k8s-m1 kubernetes]# patch -p1  < f101466d2e4d96854c80f58203de2cc6b5aaeb6a.patch 
patching file pkg/util/workqueue/prometheus/prometheus.go
patching file pkg/util/workqueue/prometheus/prometheus_test.go
patching file pkg/util/workqueue/prometheus/BUILD
patching file pkg/util/workqueue/prometheus/prometheus.go
patching file pkg/util/workqueue/prometheus/prometheus_test.go
patching file staging/src/k8s.io/client-go/util/workqueue/delaying_queue.go
Hunk #1 succeeded at 44 (offset 1 line).
Hunk #2 succeeded at 76 (offset 3 lines).
Hunk #3 succeeded at 152 (offset 6 lines).
patching file staging/src/k8s.io/client-go/util/workqueue/metrics.go
patching file staging/src/k8s.io/client-go/util/workqueue/metrics_test.go
patching file staging/src/k8s.io/client-go/util/workqueue/rate_limiting_queue_test.go

编译

为了和官方尽量保持一致,这里使用docker容器编译,先说下几个环境变量:

KUBE_BUILD_CONFORMANCE=nKUBE_BUILD_HYPERKUBE=n 参数决定是否构建 hyperkube-amd64conformance-amd64 镜像,默认是 y 构建,这里设置为n表示不构建

[root@k8s-m1 kubernetes]# KUBE_BUILD_PLATFORMS=linux/amd64 KUBE_BUILD_CONFORMANCE=n KUBE_BUILD_HYPERKUBE=n make quick-release
+++ [1021 21:30:50] Verifying Prerequisites....
+++ [1021 21:30:51] Building Docker image kube-build:build-7db96ab759-5-v1.12.10-1
+++ [1021 21:30:54] Syncing sources to container
+++ [1021 21:30:58] Running build command...
+++ [1021 21:31:43] Building go targets for linux/amd64:
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/cloud-controller-manager
    cmd/kubelet
    cmd/kubeadm
    cmd/hyperkube
    cmd/kube-scheduler
    vendor/k8s.io/apiextensions-apiserver
    cluster/gce/gci/mounter
+++ [1021 21:34:05] Building go targets for linux/amd64:
    cmd/kube-proxy
    cmd/kubeadm
    cmd/kubelet
+++ [1021 21:34:49] Building go targets for linux/amd64:
    cmd/kubectl
+++ [1021 21:35:11] Building go targets for linux/amd64:
    cmd/gendocs
    cmd/genkubedocs
    cmd/genman
    cmd/genyaml
    cmd/genswaggertypedocs
    cmd/linkcheck
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e/e2e.test
+++ [1021 21:37:01] Building go targets for linux/amd64:
    cmd/kubemark
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e_node/e2e_node.test
+++ [1021 21:38:03] Syncing out of container
+++ [1021 21:38:33] Building tarball: manifests
+++ [1021 21:38:33] Building tarball: src
+++ [1021 21:38:33] Starting tarball: client linux-amd64
+++ [1021 21:38:33] Waiting on tarballs
+++ [1021 21:38:49] Building images: linux-amd64
+++ [1021 21:38:49] Building tarball: node linux-amd64
+++ [1021 21:38:49] Starting docker build for image: cloud-controller-manager-amd64
+++ [1021 21:38:49] Starting docker build for image: kube-apiserver-amd64
+++ [1021 21:38:49] Starting docker build for image: kube-controller-manager-amd64
+++ [1021 21:38:49] Starting docker build for image: kube-scheduler-amd64
+++ [1021 21:38:49] Starting docker build for image: kube-proxy-amd64
+++ [1021 21:38:49] Building conformance image for arch: amd64
+++ [1021 21:38:49] Building hyperkube image for arch: amd64
Sending build context to Docker daemon  52.65MB
Step 1/2 : FROM k8s.gcr.io/debian-base-amd64:v1.0.0
Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
!!! [1021 21:39:07] Call tree:
!!! [1021 21:39:07]  1: /root/kubernetes/build/lib/release.sh:231 kube::release::create_docker_images_for_server(...)
!!! [1021 21:39:07]  2: /root/kubernetes/build/lib/release.sh:237 kube::release::build_server_images(...)
!!! [1021 21:39:07]  3: /root/kubernetes/build/lib/release.sh:93 kube::release::package_server_tarballs(...)
!!! [1021 21:39:07]  4: build/release.sh:45 kube::release::package_tarballs(...)
Sending build context to Docker daemon  50.07MB
Step 1/2 : FROM k8s.gcr.io/debian-iptables-amd64:v11.0.2
Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
!!! [1021 21:39:08] Call tree:
!!! [1021 21:39:08]  1: /root/kubernetes/build/lib/release.sh:231 kube::release::create_docker_images_for_server(...)
!!! [1021 21:39:08]  2: /root/kubernetes/build/lib/release.sh:237 kube::release::build_server_images(...)
!!! [1021 21:39:08]  3: /root/kubernetes/build/lib/release.sh:93 kube::release::package_server_tarballs(...)
!!! [1021 21:39:08]  4: build/release.sh:45 kube::release::package_tarballs(...)
Sending build context to Docker daemon    156MB
Step 1/2 : FROM k8s.gcr.io/debian-base-amd64:v1.0.0
Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
!!! [1021 21:39:12] Call tree:
!!! [1021 21:39:12]  1: /root/kubernetes/build/lib/release.sh:231 kube::release::create_docker_images_for_server(...)
!!! [1021 21:39:12]  2: /root/kubernetes/build/lib/release.sh:237 kube::release::build_server_images(...)
!!! [1021 21:39:12]  3: /root/kubernetes/build/lib/release.sh:93 kube::release::package_server_tarballs(...)
!!! [1021 21:39:12]  4: build/release.sh:45 kube::release::package_tarballs(...)
Sending build context to Docker daemon  134.3MB
Step 1/2 : FROM k8s.gcr.io/debian-base-amd64:v1.0.0
Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
!!! [1021 21:39:13] Call tree:
!!! [1021 21:39:13]  1: /root/kubernetes/build/lib/release.sh:231 kube::release::create_docker_images_for_server(...)
!!! [1021 21:39:13]  2: /root/kubernetes/build/lib/release.sh:237 kube::release::build_server_images(...)
!!! [1021 21:39:13]  3: /root/kubernetes/build/lib/release.sh:93 kube::release::package_server_tarballs(...)
!!! [1021 21:39:13]  4: build/release.sh:45 kube::release::package_tarballs(...)
Sending build context to Docker daemon  205.2MB
Step 1/2 : FROM k8s.gcr.io/debian-base-amd64:v1.0.0
Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
!!! [1021 21:39:13] Call tree:
!!! [1021 21:39:13]  1: /root/kubernetes/build/lib/release.sh:231 kube::release::create_docker_images_for_server(...)
!!! [1021 21:39:13]  2: /root/kubernetes/build/lib/release.sh:237 kube::release::build_server_images(...)
!!! [1021 21:39:13]  3: /root/kubernetes/build/lib/release.sh:93 kube::release::package_server_tarballs(...)
!!! [1021 21:39:13]  4: build/release.sh:45 kube::release::package_tarballs(...)
Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
make[1]: *** [build] Error 1
!!! [1021 21:39:14] Call tree:
!!! [1021 21:39:14]  1: /root/kubernetes/build/lib/release.sh:420 kube::release::build_hyperkube_image(...)
!!! [1021 21:39:14]  2: /root/kubernetes/build/lib/release.sh:231 kube::release::create_docker_images_for_server(...)
!!! [1021 21:39:14]  3: /root/kubernetes/build/lib/release.sh:237 kube::release::build_server_images(...)
!!! [1021 21:39:14]  4: /root/kubernetes/build/lib/release.sh:93 kube::release::package_server_tarballs(...)
!!! [1021 21:39:14]  5: build/release.sh:45 kube::release::package_tarballs(...)
Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
make[1]: *** [build] Error 1
!!! [1021 21:39:14] Call tree:
!!! [1021 21:39:14]  1: /root/kubernetes/build/lib/release.sh:424 kube::release::build_conformance_image(...)
!!! [1021 21:39:14]  2: /root/kubernetes/build/lib/release.sh:231 kube::release::create_docker_images_for_server(...)
!!! [1021 21:39:14]  3: /root/kubernetes/build/lib/release.sh:237 kube::release::build_server_images(...)
!!! [1021 21:39:14]  4: /root/kubernetes/build/lib/release.sh:93 kube::release::package_server_tarballs(...)
!!! [1021 21:39:14]  5: build/release.sh:45 kube::release::package_tarballs(...)
!!! [1021 21:39:14] previous Docker build failed
!!! [1021 21:39:14] Call tree:
!!! [1021 21:39:14]  1: /root/kubernetes/build/lib/release.sh:231 kube::release::create_docker_images_for_server(...)
!!! [1021 21:39:14]  2: /root/kubernetes/build/lib/release.sh:237 kube::release::build_server_images(...)
!!! [1021 21:39:14]  3: /root/kubernetes/build/lib/release.sh:93 kube::release::package_server_tarballs(...)
!!! [1021 21:39:14]  4: build/release.sh:45 kube::release::package_tarballs(...)
!!! [1021 21:39:14] previous tarball phase failed
make: *** [quick-release] Error 1

查看结果

构建出错,不用管,出错是要构建k8s的docker镜像,二进制已经被构建出来的,我们可以查看 _output/release-stage 目录

[root@k8s-m1 kubernetes]# ll _output/release-stage/server/linux-amd64/kubernetes/server/bin/
total 828572
-rwxr-xr-x 2 root root 134324933 Oct 21 21:38 cloud-controller-manager
drwxr-xr-x 2 root root      4096 Oct 21 21:38 cloud-controller-manager.dockerbuild
-rwxr-xr-x 1 root root 250209280 Oct 21 21:38 hyperkube
-rwxr-xr-x 2 root root 205195189 Oct 21 21:38 kube-apiserver
drwxr-xr-x 2 root root      4096 Oct 21 21:38 kube-apiserver.dockerbuild
-rwxr-xr-x 2 root root 155984780 Oct 21 21:38 kube-controller-manager
drwxr-xr-x 2 root root      4096 Oct 21 21:38 kube-controller-manager.dockerbuild
-rwxr-xr-x 2 root root  50064955 Oct 21 21:38 kube-proxy
drwxr-xr-x 2 root root      4096 Oct 21 21:38 kube-proxy.dockerbuild
-rwxr-xr-x 2 root root  52643510 Oct 21 21:38 kube-scheduler
drwxr-xr-x 2 root root      4096 Oct 21 21:38 kube-scheduler.dockerbuild

替换后查看输出信息

cp _output/release-stage/server/linux-amd64/kubernetes/server/bin/kube-controller-manager /usr/local/bin/
cp: overwrite ‘/usr/local/bin/kube-controller-manager’? y
[root@k8s-m1 kubernetes]# systemctl status kube-controller-manager.service 
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Mon 2019-10-21 20:47:21 CST; 55min ago
     Docs: https://github.com/kubernetes/kubernetes
  Process: 114302 ExecStart=/usr/local/bin/kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/controller-manager.kubeconfig --authorization-kubeconfig=/etc/kubernetes/controller-manager.kubeconfig --bind-address=0.0.0.0 --client-ca-file=/etc/kubernetes/pki/ca.crt --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig --leader-elect=true --cluster-cidr=10.244.0.0/16 --service-cluster-ip-range=10.96.0.0/12 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --root-ca-file=/etc/kubernetes/pki/ca.crt --use-service-account-credentials=true --controllers=*,bootstrapsigner,tokencleaner --experimental-cluster-signing-duration=86700h --feature-gates=RotateKubeletClientCertificate=true --node-monitor-period=5s --node-monitor-grace-period=2m --pod-eviction-timeout=1m --logtostderr=false --log-dir=/var/log/kubernetes/kube-controller-manager --v=2 (code=killed, signal=TERM)
 Main PID: 114302 (code=killed, signal=TERM)

Oct 21 20:45:10 k8s-m1 systemd[1]: Started Kubernetes Controller Manager.
Oct 21 20:47:21 k8s-m1 systemd[1]: Stopping Kubernetes Controller Manager...
Oct 21 20:47:21 k8s-m1 systemd[1]: Stopped Kubernetes Controller Manager.
[root@k8s-m1 kubernetes]# systemctl start kube-controller-manager.service 
[root@k8s-m1 kubernetes]# systemctl status kube-controller-manager.service 
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-10-21 21:43:25 CST; 1s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 11996 (kube-controller)
    Tasks: 8
   Memory: 12.2M
   CGroup: /system.slice/kube-controller-manager.service
           └─11996 /usr/local/bin/kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/controller-manager.kubeconfig --authorization-kubeconfig=/etc/kubernetes/controller-manager.kubeconfig -...

Oct 21 21:43:25 k8s-m1 systemd[1]: Started Kubernetes Controller Manager.

已经没有相关错误了

参考

我来评几句
登录后评论

已发表评论数()

相关站点

+订阅
热门文章