概述
open-local 是阿里开源的一个 CSI 插件，用于将本地磁盘挂载到容器中。在挂载的过程中，根据选择的模式不同的，open-local 可以自动创建 LVM 卷，并且会对磁盘进行格式化，这个格式化的过程是通过 mkfs 命令完成的，但是目前的实现看，open-local 还没有提供关于格式化参数的配置，而我们当前的场景会需要做不同类型数据库的格式化的标准，比如说是用 ES 和 Kafka，在生产业务中，这两种组件虽然都使用 xfs 的文件系统，但是在格式化的时候，可能会有不同参数。
为了让 open-local 可以支持不同类型的 mkfs 的参数，我们需要对 open-local 进行修改，当前的想法是通过 StorageClass 的 parameters 字段来传递参数，然后在 open-local 中解析这些参数，然后在格式化的时候使用这些参数。
修改点
以下是 open-local 默认的配置下，使用 xfs 格式化 LVM 卷的日志。
1
2
3
4
5
6
7
8
|
# k logs open-local-agent-ddds9 -c csi-plugin
I0315 06:05:07.924059 45312 nodeserver.go:117] NodePublishVolume: start to mount volume local-e7e28290-319e-49ba-9de8-655ccd578434 to target path /var/lib/kubelet/pods/0c593eb9-134d-45bf-af5b-fb93afafc1ae/volumes/kubernetes.io~csi/local-e7e28290-319e-49ba-9de8-655ccd578434/mount
I0315 06:05:07.930594 45312 nodeutils.go:52] createLV: vg open-local-pool-0, volume local-e7e28290-319e-49ba-9de8-655ccd578434, LVM Type linear
I0315 06:05:07.974556 45312 mount_linux.go:516] Disk "/dev/open-local-pool-0/local-e7e28290-319e-49ba-9de8-655ccd578434" appears to be unformatted, attempting to format as type: "xfs" with options: [-f /dev/open-local-pool-0/local-e7e28290-319e-49ba-9de8-655ccd578434]
I0315 06:05:08.341667 45312 mount_linux.go:526] Disk successfully formatted (mkfs): xfs - /dev/open-local-pool-0/local-e7e28290-319e-49ba-9de8-655ccd578434 /var/lib/kubelet/pods/0c593eb9-134d-45bf-af5b-fb93afafc1ae/volumes/kubernetes.io~csi/local-e7e28290-319e-49ba-9de8-655ccd578434/mount
I0315 06:05:08.440898 45312 nodeutils.go:195] mountLvmFS: mount devicePath /dev/open-local-pool-0/local-e7e28290-319e-49ba-9de8-655ccd578434 to targetPath /var/lib/kubelet/pods/0c593eb9-134d-45bf-af5b-fb93afafc1ae/volumes/kubernetes.io~csi/local-e7e28290-319e-49ba-9de8-655ccd578434/mount successfully, options: [rw rw nouuid]
I0315 06:05:08.440939 45312 nodeserver.go:632] no need to set throttle for volume local-e7e28290-319e-49ba-9de8-655ccd578434
I0315 06:05:08.440965 45312 nodeserver.go:217] NodePublishVolume: mount local volume local-e7e28290-319e-49ba-9de8-655ccd578434 to /var/lib/kubelet/pods/0c593eb9-134d-45bf-af5b-fb93afafc1ae/volumes/kubernetes.io~csi/local-e7e28290-319e-49ba-9de8-655ccd578434/mount successfully
|
从日志可以看到，mkfs 在执行的时候，只用了 -f 这个参数，且从源码上看，open-local 并没有提供配置参数的接口，所以需要找到源码修改的地方，另外可以参考大部分的 CSI 传参的一些方式，这里通过 req.VolumeContext 这个结构，从 StorageClass 里获取 parameters.xfsMkOptions 最终修改可以参考下面的 diff。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
|
diff --git a/pkg/csi/nodeutils.go b/pkg/csi/nodeutils.go
index 49839b38..74313b38 100644
--- a/pkg/csi/nodeutils.go
+++ b/pkg/csi/nodeutils.go
@@ -5,6 +5,7 @@ import (
"os"
"path/filepath"
"strconv"
+ "strings"
"time"
"github.com/alibaba/open-local/pkg"
@@ -126,8 +127,19 @@ func (ns *nodeServer) mountLvmFS(ctx context.Context, req *csi.NodePublishVolume
mountFlags := req.GetVolumeCapability().GetMount().GetMountFlags()
options = append(options, mountFlags...)
options = append(options, collectMountOptions(fsType, options)...)
-
- if err := ns.k8smounter.FormatAndMount(devicePath, targetPath, fsType, options); err != nil {
+ var mkOptions []string
+ if fsType == localtype.VolumeFSTypeXFS {
+ // 检查 VolumeContext 中是否包含 localtype.XFSMkOptions
+ if mkOptionStr, found := req.VolumeContext[localtype.XFSMkOptions]; found && mkOptionStr != "" {
+ // 如果存在该 key 并且值不为空，分割选项
+ mkOptions = strings.Split(mkOptionStr, " ")
+ log.Infof("mkOptions: %v", mkOptions)
+ } else {
+ // 如果没有设置 mkfs 选项，可以提供默认值或保持 mkOptions 为空
+ mkOptions = nil
+ }
+ }
+ if err := ns.k8smounter.FormatAndMountSensitiveWithFormatOptions(devicePath, targetPath, fsType, options, nil, mkOptions); err != nil {
return fmt.Errorf("mountLvmFS: fail to format and mount volume(volume id:%s, device path: %s): %s", req.VolumeId, devicePath, err.Error())
}
diff --git a/pkg/types.go b/pkg/types.go
index bcec4f74..d472c88b 100644
--- a/pkg/types.go
+++ b/pkg/types.go
@@ -53,6 +53,7 @@ const (
VolumeFSTypeExt4 = "ext4"
VolumeFSTypeExt3 = "ext3"
VolumeFSTypeXFS = "xfs"
+ XFSMkOptions = "xfsMkOptions"
VolumeIOPS = "iops"
VolumeBPS = "bps"
|
测试和验证
下面我们定义一个新的 StorageClass，专门给需要定制化 xfsMkOptions 的 PVC 使用。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
meta.helm.sh/release-name: open-local
meta.helm.sh/release-namespace: kube-system
labels:
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: open-local
app.kubernetes.io/version: v0.8.0-alpha
helm.sh/chart: open-local-v0.8.0-alpha
name: open-local-lvm-test-xfs
parameters:
csi.storage.k8s.io/fstype: xfs
volumeType: LVM
xfsMkOptions: "-i attr=2 -l lazy-count=1,sectsize=4096 -b size=4096 -d sectsize=4096 -L runzhliu"
provisioner: local.csi.aliyun.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
|
然后基于官方的例子，简单修改一下 storageClassName 为 open-local-lvm-test-xfs。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
|
apiVersion: v1
kind: Service
metadata:
name: minio-service
spec:
clusterIP: None
ports:
- port: 9000
protocol: TCP
targetPort: 9000
name: http-metrics
selector:
app: minio
type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
name: minio-external-service
labels:
app: minio
spec:
ports:
- port: 9000
protocol: TCP
targetPort: 9000
name: http-metrics
selector:
app: minio
type: NodePort
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
app: minio
name: minio
spec:
podManagementPolicy: Parallel
replicas: 1
selector:
matchLabels:
app: minio
serviceName: minio-service
template:
metadata:
labels:
app: minio
name: minio
spec:
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
- args:
- server
- /data
env:
- name: MINIO_PROMETHEUS_AUTH_TYPE
value: public
- name: MINIO_ROOT_USER
value: minio
- name: MINIO_ROOT_PASSWORD
value: miniostorage
image: minio/minio
imagePullPolicy: Always
livenessProbe:
httpGet:
path: /minio/health/live
port: 9000
scheme: HTTP
name: minio
ports:
- containerPort: 9000
protocol: TCP
volumeMounts:
- mountPath: /data/
name: minio-data
volumeClaimTemplates:
- metadata:
name: minio-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10Gi"
storageClassName: open-local-lvm-test-xfs
volumeMode: Filesystem
|
然后创建资源，查看日志，看看相关的 xfsMkOptions 是否已经生效。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
k logs open-local-agent-gzznc -c csi-plugin -f
I0315 12:57:27.397158 143277 main.go:46] Version: v0.8.0, Commit: d088ea3de1e0adb5828299211f02c7ea0a7a2cf3
I0315 12:57:27.397362 143277 csi.go:53] CSI Driver Name: local.csi.aliyun.com, nodeID: node5, endPoints unix://var/lib/kubelet/plugins/local.csi.aliyun.com/csi.sock
W0315 12:57:27.397381 143277 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0315 12:57:27.397804 143277 driver.go:96] driver mode: node
I0315 12:57:27.397815 143277 nodeserver.go:73] NewNodeServer: MAX_VOLUMES_PERNODE is set to(not default): 256
I0315 12:57:27.410303 143277 server.go:110] Lvmd Starting with socket: :1736 ...
I0315 12:57:33.100204 143277 nodeserver.go:117] NodePublishVolume: start to mount volume local-408c4088-f534-4a93-81db-6290836f1cb3 to target path /var/lib/kubelet/pods/8a2ea67c-9867-4ee4-9713-9eb8a944685a/volumes/kubernetes.io~csi/local-408c4088-f534-4a93-81db-6290836f1cb3/mount
I0315 12:57:33.104883 143277 nodeutils.go:53] createLV: vg open-local-pool-0, volume local-408c4088-f534-4a93-81db-6290836f1cb3, LVM Type linear
I0315 12:57:33.104922 143277 nodeutils.go:136] mkOptions: [-i attr=2 -l lazy-count=1,sectsize=4096 -b size=4096 -d sectsize=4096 -L runzhliu]
I0315 12:57:33.148503 143277 mount_linux.go:516] Disk "/dev/open-local-pool-0/local-408c4088-f534-4a93-81db-6290836f1cb3" appears to be unformatted, attempting to format as type: "xfs" with options: [-i attr=2 -l lazy-count=1,sectsize=4096 -b size=4096 -d sectsize=4096 -L runzhliu -f /dev/open-local-pool-0/local-408c4088-f534-4a93-81db-6290836f1cb3]
I0315 12:57:33.517415 143277 mount_linux.go:526] Disk successfully formatted (mkfs): xfs - /dev/open-local-pool-0/local-408c4088-f534-4a93-81db-6290836f1cb3 /var/lib/kubelet/pods/8a2ea67c-9867-4ee4-9713-9eb8a944685a/volumes/kubernetes.io~csi/local-408c4088-f534-4a93-81db-6290836f1cb3/mount
I0315 12:57:33.636316 143277 nodeutils.go:207] mountLvmFS: mount devicePath /dev/open-local-pool-0/local-408c4088-f534-4a93-81db-6290836f1cb3 to targetPath /var/lib/kubelet/pods/8a2ea67c-9867-4ee4-9713-9eb8a944685a/volumes/kubernetes.io~csi/local-408c4088-f534-4a93-81db-6290836f1cb3/mount successfully, options: [rw rw nouuid]
I0315 12:57:33.636347 143277 nodeserver.go:632] no need to set throttle for volume local-408c4088-f534-4a93-81db-6290836f1cb3
I0315 12:57:33.636363 143277 nodeserver.go:217] NodePublishVolume: mount local volume local-408c4088-f534-4a93-81db-6290836f1cb3 to /var/lib/kubelet/pods/8a2ea67c-9867-4ee4-9713-9eb8a944685a/volumes/kubernetes.io~csi/local-408c4088-f534-4a93-81db-6290836f1cb3/mount successfully
|
最后再宿主机上，通过 lsblk -f 和 xfs_info 查看挂载的文件系统的信息。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
sda LVM2_member YuL0Ko-NxjF-TcPd-LDxl-1XO3-Qw3O-VbE2LS
├─open--local--pool--0-local--408c4088--f534--4a93--81db--6290836f1cb3 xfs runzhliu 45e6c107-17a1-4da3-9421-0b5c79b53272 /var/lib/kubelet/pods/8a2ea67c-9867-4ee4-9713-9eb8a944685a/volumes/kubernetes
# xfs_info /dev/mapper/open--local--pool--0-local--408c4088--f534--4a93--81db--6290836f1cb3
meta-data=/dev/mapper/open--local--pool--0-local--408c4088--f534--4a93--81db--6290836f1cb3 isize=512 agcount=4, agsize=655360 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
|
从上面的结果看，创建出来的 LV 以及格式化出来的 xfs 系统都是满足设置的 xfsMkOptions。
总结
本文介绍了如何修改 open-local 的源码，使其支持通过 StorageClass 的 parameters 字段传递 xfsMkOptions 的参数，以便在格式化磁盘的时候可以定制化参数。这样可以满足不同场景下对磁盘格式化的需求。