cilium-chain和bandwidth-manager

2024-03-29 约 975 字预计阅读 2 分钟

概述

基于 Cilium 官网关于 Bandwidth Manage，主要是描述 Cilium 作为唯一 CNI 的情况下，没有太多介绍 bandwidth-manager 是否能够使用在 Chain 模式下，本文主要记录一下 Cilium Chain 模式下的 bandwidth-manager 的部署测试和使用。

部署和测试

在 Kubernetes + Flannel 的集群部署 Cilium，注意是采用 Chain 的方式，这才是我们测试的场景。

1

helm install cilium . --version 1.14.4 --namespace=kube-system --set cni.chainingMode=generic-veth --set cni.customConf=true --set cni.configMap=cni-configuration --set routingMode=native --set enableIPv4Masquerade=false --set bandwidthManager.enabled=true

部署的情况如下，可以看到 Flannel 和 Cilium 是共存的。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


[root@master ~]# k get pods -o wide
NAME                               READY   STATUS      RESTARTS        AGE   IP              NODE     
cilium-4rtwr                       1/1     Running     35 (23h ago)    46h   192.168.1.201   node1    
cilium-8946r                       1/1     Running     8 (23h ago)     46h   192.168.1.200   master   
cilium-operator-85c7767d4c-wzhxg   1/1     Running     35 (23h ago)    46h   192.168.1.201   node1    
cilium-wcmsl                       1/1     Running     41 (23h ago)    46h   192.168.1.202   node2    
coredns-6d4b75cb6d-c58px           0/1     Running     213 (20m ago)   45h   10.244.2.2      node2    
coredns-6d4b75cb6d-gsjcq           1/1     Running     202 (36m ago)   45h   10.244.1.2      node1    
etcd-master                        1/1     Running     3 (23h ago)     30d   192.168.1.200   master   
kube-apiserver-master              1/1     Running     6 (23h ago)     30d   192.168.1.200   master   
kube-controller-manager-master     1/1     Running     27 (23h ago)    30d   192.168.1.200   master   
kube-flannel-ds-l95kb               1/1     Running     0               46h   192.168.1.201   node1    
kube-flannel-ds-p6kcs               1/1     Running     0               46h   192.168.1.202   node2    
kube-flannel-ds-vljth               1/1     Running     0               4s    192.168.1.200   master   
kube-scheduler-master              1/1     Running     24 (23h ago)    30d   192.168.1.200   master   

然后再进入 Cilium 的 Pod 里查看一下 bandwidth-manager 的情况，可以看到 BandwidthManager 字段显示是开启的状态的。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


[root@master ~]# k exec -it cilium-4rtwr -- cilium status --verbose
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.24 (v1.24.8) [linux/amd64]
Kubernetes APIs:        ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   Partial   [eth0 , flannel.1 ]
Host firewall:          Disabled
CNI Chaining:           generic-veth
Cilium:                 Ok   1.14.4 (v1.14.4-87dd2b64)
NodeMonitor:            Listening for events on 4 CPUs with 64x4096 of shared memory
Cilium health daemon:   Ok
IPAM:                   IPv4: 1/254 allocated from 10.0.2.0/24,
Allocated addresses:
  10.0.2.74 (router)
IPv4 BIG TCP:           Disabled
IPv6 BIG TCP:           Disabled
BandwidthManager:       EDT with BPF [CUBIC] [eth0, flannel.1]
Host Routing:           Legacy
Masquerading:           Disabled
Clock Source for BPF:   ktime
Controller Status:      25/25 healthy
...
...

最后，来测试一下 BandwidthManager 的作用，部署下面的 netperf 客户端和服务端。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Pod
metadata:
  annotations:
    # Limits egress bandwidth to 10Mbit/s.
    kubernetes.io/egress-bandwidth: "10M"
  labels:
    # This pod will act as server.
    app.kubernetes.io/name: netperf-server
  name: netperf-server
spec:
  containers:
  - name: netperf
    image: cilium/netperf
    ports:
    - containerPort: 12865
---
apiVersion: v1
kind: Pod
metadata:
  # This Pod will act as client.
  name: netperf-client
spec:
  affinity:
    # Prevents the client from being scheduled to the
    # same node as the server.
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app.kubernetes.io/name
            operator: In
            values:
            - netperf-server
        topologyKey: kubernetes.io/hostname
  containers:
  - name: netperf
    args:
    - sleep
    - infinity
    image: cilium/netperf
EOF

下面做具体的带宽压测，结果正如我们的配置一样，限制在了 9.51 Mbit/s。

1
2
3
4
5
6
7
8
9


[root@master ~]# NETPERF_SERVER_IP=$(kubectl get pod netperf-server -o jsonpath='{.status.podIP}')
[root@master ~]# kubectl exec netperf-client -- netperf -t TCP_MAERTS -H "${NETPERF_SERVER_IP}"
MIGRATED TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.244.2.4 (10.244.) port 0 AF_INET
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

131072  16384  16384    10.00       9.51

另外需要找 server 端的 agent 才能通过 Cilium 的客户端查找相关的信息。

1
2
3
4


[root@master ~]# kubectl exec -it -n kube-system cilium-wcmsl -- cilium bpf bandwidth list
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
IDENTITY   EGRESS BANDWIDTH (BitsPerSec)
1287       10M

总结

在 bandwidth-manager 的使用下，实际上是会有更多的类似的应用场景，比如 CFP: QoS for network #24194，这个也是我们在业务上希望探索的方向，就是利用 Cilium Chain 的 BandwidthManager，给容器网络带来更多如 QoS 的管控。

参考资料

警告

本文最后更新于 2024年3月29日，文中内容可能已过时，请谨慎参考。

💡赞赏支持

微信打赏

支付宝打赏

目录

cilium-chain和bandwidth-manager

概述

部署和测试

总结

参考资料