Kubernetes集群Aggregation证书配置

2025-03-02 约 1000 字预计阅读 2 分钟

概述

除标准的 Kubernetes apiserver 外，Aggregation API 还涉及另一个服务器: 扩展 apiserver。Kubernetes apiserver 将需要与你的扩展 apiserver 通信，并且你的扩展 apiserver 也需要与 Kubernetes apiserver 通信。为了确保此通信的安全，Kubernetes apiserver 使用 x509 证书向扩展 apiserver 认证。

/kubernetes%E9%9B%86%E7%BE%A4aggregation%E8%AF%81%E4%B9%A6%E9%85%8D%E7%BD%AE/img.png

操作

大致流程如下:

Kubernetes apiserver: 对发出请求的用户身份认证，并对请求的API路径执行鉴权
Kubernetes apiserver: 将请求转发到扩展apiserver
扩展apiserver: 认证来自Kubernetes apiserver的请求
扩展apiserver: 对来自原始用户的请求鉴权
扩展apiserver: 执行

下面是 kubeadm 部署的集群测试的过程。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" --v 10
I0304 09:42:51.378787   10851 loader.go:373] Config loaded from file:  /Users/oscar01.liu/.kube/config
I0304 09:42:51.379769   10851 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.27.4 (darwin/amd64) kubernetes/fa3d799" 'https://10.189.110.45:6443/apis/external.metrics.k8s.io/v1beta1'
I0304 09:42:51.397867   10851 round_trippers.go:510] HTTP Trace: Dial to tcp:10.189.110.45:6443 succeed
I0304 09:42:51.443699   10851 round_trippers.go:553] GET https://10.189.110.45:6443/apis/external.metrics.k8s.io/v1beta1 200 OK in 63 milliseconds
I0304 09:42:51.443722   10851 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 17 ms TLSHandshake 32 ms ServerProcessing 12 ms Duration 63 ms
I0304 09:42:51.443728   10851 round_trippers.go:577] Response Headers:
I0304 09:42:51.443735   10851 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 09b5f820-4989-4a0f-b960-ba42a3d23dcc
I0304 09:42:51.443741   10851 round_trippers.go:580]     Audit-Id: 28bb3fb9-05c2-47fb-8ead-8fcdeadbaa02
I0304 09:42:51.443746   10851 round_trippers.go:580]     Audit-Id: 28bb3fb9-05c2-47fb-8ead-8fcdeadbaa02
I0304 09:42:51.443752   10851 round_trippers.go:580]     Cache-Control: no-cache, private
I0304 09:42:51.443758   10851 round_trippers.go:580]     Cache-Control: no-cache, private
I0304 09:42:51.443763   10851 round_trippers.go:580]     Content-Type: application/json
I0304 09:42:51.443769   10851 round_trippers.go:580]     Date: Tue, 04 Mar 2025 01:42:51 GMT
I0304 09:42:51.443790   10851 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 2e34c289-755e-4f96-9198-4c4cf8381758
I0304 09:42:51.443800   10851 round_trippers.go:580]     Content-Length: 220
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"external.metrics.k8s.io/v1beta1","resources":[{"name":"externalmetrics","singularName":"","namespaced":true,"kind":"ExternalMetricValueList","verbs":["get"]}]

参数对比

下面是我司当前的 Kubernetes 集群的参数配置，涉及一些敏感信息，部分参数已经隐藏了。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


kube-apiserver \
--advertise-address=10.xx.xx.xx \
--secure-port=443 \
--service-cluster-ip-range=10.x.x.x/22 \
--allow-privileged=true \
--authorization-mode=Node,RBAC \
--max-requests-inflight=2000 \
--min-request-timeout=3600 \
--cors-allowed-origins=.* \
--client-ca-file=/apps/conf/ssl/kubernetes/k8s.ca.crt \
--tls-cert-file=/apps/conf/ssl/kubernetes/k8s.server.crt \
--tls-private-key-file=/apps/conf/ssl/kubernetes/k8s.server.key \
--enable-admission-plugins=NamespaceLifecycle,NamespaceExists,LimitRanger,ServiceAccount,ResourceQuota,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook \
--enable-bootstrap-token-auth=false \
--etcd-cafile=/apps/conf/kubernetes/ssl/etcd_ca.pem \
--etcd-certfile=/apps/conf/kubernetes/ssl/etcd_client.pem \
--etcd-keyfile=/apps/conf/kubernetes/ssl/etcd_client-key.pem \
--etcd-servers=xxx \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--runtime-config=extensions/v1beta1/thirdpartyresources=false \
--kubelet-client-certificate=/apps/conf/ssl/kubernetes/k8s.client.crt \
--kubelet-client-key=/apps/conf/ssl/kubernetes/k8s.client.key \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--service-account-key-file=/apps/conf/ssl/kubernetes/service-account.pub \
--service-account-signing-key-file=/apps/conf/ssl/kubernetes/service-account.key

下面是 kubeadm 部署的集群的参数配置，涉及一些敏感信息，部分参数已经隐藏了。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


kube-apiserver \
--advertise-address=10.xx.xx.xx \
--allow-privileged=true \
--authorization-mode=Node,RBAC \
--client-ca-file=/etc/kubernetes/pki/ca.crt \
--enable-admission-plugins=NodeRestriction \
--enable-bootstrap-token-auth=true \
--feature-gates=InPlacePodVerticalScaling=true \
--etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt \
--etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt \
--etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key \
--etcd-servers=https://127.0.0.1:2379 \
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \
--kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt \
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \
--requestheader-allowed-names=front-proxy-client \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--secure-port=6443 \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--service-account-key-file=/etc/kubernetes/pki/sa.pub \
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
--service-cluster-ip-range=10.96.0.0/12 \
--tls-cert-file=/etc/kubernetes/pki/apiserver.crt \
--tls-private-key-file=/etc/kubernetes/pki/apiserver.key \
--token-auth-file=/etc/kubernetes/pki/static-token-file.csv \
--v=5

两部分的参数中，主要是以下几个参数的差异，这些参数都是需要补充到生产集群的。

1
2
3
4
5
6
7
8
9


# 下面的参数需要配置CA和创建的相关证书
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt \
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key \
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \
# 下面的参数保持即可，其中requestheader-allowed-names必须包含在front-proxy-ca.crt的CN中
--requestheader-allowed-names=front-proxy-client \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \

自签过程

可以参考下面的脚本签署相关的证书。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


# 生成 CA 的私钥
openssl genrsa -out /etc/kubernetes/pki/front-proxy-ca.key 2048

# 使用 CA 私钥生成自签名的 CA 证书
openssl req -x509 -new -nodes -key /etc/kubernetes/pki/front-proxy-ca.key \
  -subj "/CN=front-proxy-ca" \
  -days 3650 -out /etc/kubernetes/pki/front-proxy-ca.crt
  
# 生成客户端证书的私钥  
openssl genrsa -out /etc/kubernetes/pki/front-proxy-client.key 2048 

# 使用客户端私钥生成证书签名请求（CSR）
openssl req -new -key /etc/kubernetes/pki/front-proxy-client.key \
  -subj "/CN=front-proxy-client" \
  -out /etc/kubernetes/pki/front-proxy-client.csr
  
# 使用 front-proxy-ca 签署客户端证书
openssl x509 -req -in /etc/kubernetes/pki/front-proxy-client.csr \
  -CA /etc/kubernetes/pki/front-proxy-ca.crt \
  -CAkey /etc/kubernetes/pki/front-proxy-ca.key \
  -CAcreateserial \
  -out /etc/kubernetes/pki/front-proxy-client.crt \
  -days 3650 
  
# 验证生成的证书
openssl x509 -in /etc/kubernetes/pki/front-proxy-client.crt -text -noout

参考资料

Kubernetes Aggregation Layer

💡赞赏支持

微信打赏

支付宝打赏

目录

Kubernetes集群Aggregation证书配置

概述

操作

参数对比

自签过程

参考资料