目录

Kubeadm系列-01-preflight

概述

我们知道 kubeadm init 的过程中会进行很多 preflight 的检查,这些主要是指内核参数、模块、CRI 等环境的检查,如果有哪些配置不符合 Kubernetes 的要求,就会抛出 Warning 或者 Error 的信息。

代码的实现

下面就是 preflight 检查的主要接口。

1
2
3
4
5
6
// Checker validates the state of the system to ensure kubeadm will be
// successful as often as possible.
type Checker interface {
	Check() (warnings, errorList []error)
	Name() string
}

如果有 diy 的 check 需求,可以在代码里继承这个接口进行扩展,下面举个 check 的例子,很明显 ContainerRuntimeCheck 是对 CRI 也就是容器运行时进行的检查。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
// ContainerRuntimeCheck verifies the container runtime.
type ContainerRuntimeCheck struct {
	runtime utilruntime.ContainerRuntime
}

// Name returns label for RuntimeCheck.
func (ContainerRuntimeCheck) Name() string {
	return "CRI"
}

// Check validates the container runtime
func (crc ContainerRuntimeCheck) Check() (warnings, errorList []error) {
	klog.V(1).Infoln("validating the container runtime")
	if err := crc.runtime.IsRunning(); err != nil {
		errorList = append(errorList, err)
	}
	return warnings, errorList
}

而真正起到检查的作用是下面这个函数,其实就是宿主机执行一下 crictl info,并且接收其返回,老铁们不妨在宿主机上直接运行一下看看结果。

1
2
3
4
5
6
7
// IsRunning checks if runtime is running
func (runtime *CRIRuntime) IsRunning() error {
	if out, err := runtime.crictl("info").CombinedOutput(); err != nil {
		return errors.Wrapf(err, "container runtime is not running: output: %s, error", string(out))
	}
	return nil
}

所有的 Check 里面是会有小部分交错的部分,比如说检查防火墙的问题,先会对 Firewall 这个服务做 service check,然后才会对具体的端口进行检查。

下面是所有 check 的统计

  1. CRI: 检查容器运行时是否有在运行
  2. Service: 检查是否enable和active
  3. Firewall: 检查防火墙是否有关闭
  4. Port: 检查某些端口是否有放开
  5. Privileged: 检查一些权限的问题
  6. Dir Available: 检查目录是否有效
  7. File Available: 检查文件是否有效
  8. File Existing: 检查文件是否存在
  9. File Content: 检查文件中是否有指定的内容
  10. In Path: 检查某些可执行文件是否在指定的目录
  11. Hostname: 检查主机名的格式
  12. HTTP Proxy: 检查本机是否有Proxy设置
  13. HTTP Proxy CIDR: 检查本机有哪些地址会走Proxy
  14. System Verification: 检查系统版本
  15. Kubernetes Version: 检查Kubernetes的版本
  16. Kubelet Version: 检查Kubelet的版本
  17. SwapCheck: 检查Swap是否关闭
  18. External Etcd Version: 检查外部etcd的版本
  19. Image Pull: 检查镜像仓库是否连通
  20. Num CPU: 检查本机CPU数量是否符合kubeadm的最低要求
  21. Mem: 检查本机内存是否符合kubeadm的最低要求

真正在做检查的时候,还会区分是 controlplane 还是普通的 worker 节点,不同角色具体要做的检查是不尽相同的。

我们看一下 In Path 这个检查,也就是检查一些必要的二进制文件或者命令是否已经安装,另外还要看 mandatory 如果是 true 的话,那就是必须要符合的,否则就是可有可无,不过如果没有就会提示出来,会建议用户去安装的。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
InPathCheck{executable: "crictl", mandatory: true, exec: execer},
InPathCheck{executable: "conntrack", mandatory: true, exec: execer},
InPathCheck{executable: "ip", mandatory: true, exec: execer},
InPathCheck{executable: "iptables", mandatory: true, exec: execer},
InPathCheck{executable: "mount", mandatory: true, exec: execer},
InPathCheck{executable: "nsenter", mandatory: true, exec: execer},
InPathCheck{executable: "ebtables", mandatory: false, exec: execer},
InPathCheck{executable: "ethtool", mandatory: false, exec: execer},
InPathCheck{executable: "socat", mandatory: false, exec: execer},
InPathCheck{executable: "tc", mandatory: false, exec: execer},
InPathCheck{executable: "touch", mandatory: false, exec: execer})

最后我们看一下 System Verification,主要是针对主机的系统来进行一些模块的检测,我们主要看一下 Linux 下的检查,内核很多模块有以及没有,还是有较大的差别的,所以不要轻视这部分的检查,以为主要是 Linux 系统就没啥问题了,有时候恰恰是这部分的内容更难排查。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
// DefaultSysSpec is the default SysSpec for Linux
var DefaultSysSpec = SysSpec{
	OS: "Linux",
	KernelSpec: KernelSpec{
		Versions: []string{`^3\.[1-9][0-9].*$`, `^([4-9]|[1-9][0-9]+)\.([0-9]+)\.([0-9]+).*$`}, // Requires 3.10+, or newer
		// TODO(random-liu): Add more config
		// TODO(random-liu): Add description for each kernel configuration:
		Required: []KernelConfig{
			{Name: "NAMESPACES"},
			{Name: "NET_NS"},
			{Name: "PID_NS"},
			{Name: "IPC_NS"},
			{Name: "UTS_NS"},
			{Name: "CGROUPS"},
			{Name: "CGROUP_CPUACCT"},
			{Name: "CGROUP_DEVICE"},
			{Name: "CGROUP_FREEZER"},
			{Name: "CGROUP_PIDS"},
			{Name: "CGROUP_SCHED"},
			{Name: "CPUSETS"},
			{Name: "MEMCG"},
			{Name: "INET"},
			{Name: "EXT4_FS"},
			{Name: "PROC_FS"},
			{Name: "NETFILTER_XT_TARGET_REDIRECT", Aliases: []string{"IP_NF_TARGET_REDIRECT"}},
			{Name: "NETFILTER_XT_MATCH_COMMENT"},
			{Name: "FAIR_GROUP_SCHED"},
		},
		Optional: []KernelConfig{
			{Name: "OVERLAY_FS", Aliases: []string{"OVERLAYFS_FS"}, Description: "Required for overlayfs."},
			{Name: "AUFS_FS", Description: "Required for aufs."},
			{Name: "BLK_DEV_DM", Description: "Required for devicemapper."},
			{Name: "CFS_BANDWIDTH", Description: "Required for CPU quota."},
			{Name: "CGROUP_HUGETLB", Description: "Required for hugetlb cgroup."},
			{Name: "SECCOMP", Description: "Required for seccomp."},
			{Name: "SECCOMP_FILTER", Description: "Required for seccomp mode 2."},
		},
		Forbidden: []KernelConfig{},
	},
	Cgroups: []string{"cpu", "cpuacct", "cpuset", "devices", "freezer", "memory", "pids"},
	CgroupsOptional: []string{
		// The hugetlb cgroup is optional since some kernels are compiled without support for huge pages
		// and therefore lacks corresponding hugetlb cgroup
		"hugetlb",
		// The blkio cgroup is optional since some kernels are compiled without support for block I/O throttling.
		// Containerd and cri-o will use blkio to track disk I/O and throttling in both cgroup v1 and v2.
		"blkio",
	},
	CgroupsV2:         []string{"cpu", "cpuset", "devices", "freezer", "memory", "pids"},
	CgroupsV2Optional: []string{"hugetlb", "blkio"},
	RuntimeSpec: RuntimeSpec{
		DockerSpec: &DockerSpec{
			Version:     []string{`1\.1[1-3]\..*`, `17\.0[3,6,9]\..*`, `18\.0[6,9]\..*`, `19\.03\..*`, `20\.10\..*`},
			GraphDriver: []string{"aufs", "btrfs", "overlay", "overlay2", "devicemapper", "zfs"},
		},
	},
}
警告
本文最后更新于 2022年3月20日,文中内容可能已过时,请谨慎参考。