8. kubebuilder 进阶: webhook

注:本文已发布超过一年,请注意您所使用工具的相关版本是否适用

注:本文所有示例代码都可以在 blog-code 仓库中找到

在前面的文章当中我们已经完成了 NodePool Operator 的基本功能开发与测试,但是有时候我们会有这种需求,例如创建或者删除资源的时候需要对资源进行一些检查的操作,如果校验不成功就不通过。或者是需要在完成实际的创建之前做一些其他操作,例如我创建一个 pod 之前对 pod 的资源做一些调整等。这些都可以通过准入控制的WebHook来实现。

准入控制存在两种 WebHook,变更准入控制 MutatingAdmissionWebhook,和验证准入控制 ValidatingAdmissionWebhook,执行的顺序是先执行 MutatingAdmissionWebhook 再执行 ValidatingAdmissionWebhook。

创建 webhook

我们通过命令创建相关的脚手架代码和 api

1
kubebuilder create webhook --group nodes --version v1 --kind NodePool --defaulting --programmatic-validation

执行之后可以看到多了一些 webhook 相关的文件和配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
  ├── api
│   └── v1
│   ├── groupversion_info.go
│   ├── nodepool_types.go
+ │   ├── nodepool_webhook.go # 在这里实现 webhook 的相关接口
+ │   ├── webhook_suite_test.go # webhook 测试
│   └── zz_generated.deepcopy.go
├── bin
├── config
+ │   ├── certmanager # 用于部署
│   ├── crd
│   │   ├── bases
│   │   │   └── nodes.lailin.xyz_nodepools.yaml
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │   ├── cainjection_in_nodepools.yaml
+ │   │   └── webhook_in_nodepools.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   ├── manager_config_patch.yaml
+ │   │   ├── manager_webhook_patch.yaml
+ │   │   └── webhookcainjection_patch.yaml
│   ├── manager
│   ├── prometheus
│   ├── rbac
│   ├── samples
│   │   └── nodes_v1_nodepool.yaml
+ │   └── webhook # webhook 部署配置
├── controllers
├── main.go

实现逻辑

实现 MutatingAdmissionWebhook 接口

这个只需要实现 Default 方法就行

1
2
3
4
5
6
7
8
9
// Default implements webhook.Defaulter so a webhook will be registered for the type
func (r *NodePool) Default() {
nodepoollog.Info("default", "name", r.Name)

// 如果 labels 为空,我们就给 labels 加一个默认值
if len(r.Labels) == 0 {
r.Labels["node-pool.lailin.xyz"] = r.Name
}
}

实现 ValidatingAdmissionWebhook 接口

实现 ValidatingAdmissionWebhook也是一样只需要实现对应的方法就行了,默认是注册了 Create 和 Update 事件的校验,我们这里主要是限制 Labels 和 Taints 的 key 只能是满足正则 ^node-pool.lailin.xyz/*[a-zA-z0-9]*$ 的固定格式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

// TODO(user): change verbs to "verbs=create;update;delete" if you want to enable deletion validation.
//+kubebuilder:webhook:path=/validate-nodes-lailin-xyz-v1-nodepool,mutating=false,failurePolicy=fail,sideEffects=None,groups=nodes.lailin.xyz,resources=nodepools,verbs=create;update,versions=v1,name=vnodepool.kb.io,admissionReviewVersions={v1,v1beta1}

var _ webhook.Validator = &NodePool{}

// ValidateCreate implements webhook.Validator so a webhook will be registered for the type
func (r *NodePool) ValidateCreate() error {
nodePoolLog.Info("validate create", "name", r.Name)

return r.validate()
}

// ValidateUpdate implements webhook.Validator so a webhook will be registered for the type
func (r *NodePool) ValidateUpdate(old runtime.Object) error {
nodePoolLog.Info("validate update", "name", r.Name)

return r.validate()
}

// ValidateDelete implements webhook.Validator so a webhook will be registered for the type
func (r *NodePool) ValidateDelete() error {
nodePoolLog.Info("validate delete", "name", r.Name)

// TODO(user): fill in your validation logic upon object deletion.
return nil
}

// validate 验证
func (r *NodePool) validate() error {
err := errors.Errorf("taint or label key must validatedy by %s", keyReg.String())

for k := range r.Spec.Labels {
if !keyReg.MatchString(k) {
return errors.WithMessagef(err, "label key: %s", k)
}
}

for _, taint := range r.Spec.Taints {
if !keyReg.MatchString(taint.Key) {
return errors.WithMessagef(err, "taint key: %s", taint.Key)
}
}

return nil
}

部署

实现了之后直接在 make run 是跑不起来的,因为 webhook 注册的地址不对,我们这里先看一下如何进行部署运行,然后再来看如何对 WebHook 进行本地调试。

WebHook 的运行需要校验证书,kubebuilder 官方建议我们使用 cert-manager 简化对证书的管理,所以我们先部署一下 cert-manager 的服务

1
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml

然后我们 build 镜像并且将镜像 load 到集群中

1
2
3
make docker-build

kind load docker-image --name kind --nodes kind-worker controller:latest

然后查看一下 config/default/kustomization.yaml文件,确认 webhook 相关的配置没有被注释掉

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
# Adds namespace to all resources.
namespace: node-pool-operator-system

# Value of this field is prepended to the
# names of all resources, e.g. a deployment named
# "wordpress" becomes "alices-wordpress".
# Note that it should also match with the prefix (text before '-') of the namespace
# field above.
namePrefix: node-pool-operator-

# Labels to add to all resources and selectors.
#commonLabels:
# someName: someValue

bases:
- ../crd
- ../rbac
- ../manager
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
# crd/kustomization.yaml
- ../webhook
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required.
- ../certmanager
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
#- ../prometheus

patchesStrategicMerge:
# Protect the /metrics endpoint by putting it behind auth.
# If you want your controller-manager to expose the /metrics
# endpoint w/o any authn/z, please comment the following line.
- manager_auth_proxy_patch.yaml

# Mount the controller config file for loading manager configurations
# through a ComponentConfig type
#- manager_config_patch.yaml

# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
# crd/kustomization.yaml
- manager_webhook_patch.yaml

# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'.
# Uncomment 'CERTMANAGER' sections in crd/kustomization.yaml to enable the CA injection in the admission webhooks.
# 'CERTMANAGER' needs to be enabled to use ca injection
- webhookcainjection_patch.yaml

# the following config is for teaching kustomize how to do var substitution
vars:
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.
- name: CERTIFICATE_NAMESPACE # namespace of the certificate CR
objref:
kind: Certificate
group: cert-manager.io
version: v1
name: serving-cert # this name should match the one in certificate.yaml
fieldref:
fieldpath: metadata.namespace
- name: CERTIFICATE_NAME
objref:
kind: Certificate
group: cert-manager.io
version: v1
name: serving-cert # this name should match the one in certificate.yaml
- name: SERVICE_NAMESPACE # namespace of the service
objref:
kind: Service
version: v1
name: webhook-service
fieldref:
fieldpath: metadata.namespace
- name: SERVICE_NAME
objref:
kind: Service
version: v1
name: webhook-service

检查一下 manager/manager.yaml 是否存在 imagePullPolicy: IfNotPresent不存在要加上

然后执行部署命令即可

1
2
3
4
5
6
make deploy

# 检查 pod 是否正常启动
▶ kubectl -n node-pool-operator-system get pods
NAME READY STATUS RESTARTS AGE
node-pool-operator-controller-manager-66bd747899-lf7xb 0/2 ContainerCreating 0 7s

使用 yaml 文件测试一下

1
2
3
4
5
6
7
8
apiVersion: nodes.lailin.xyz/v1
kind: NodePool
metadata:
name: worker
spec:
labels:
"xxx": "10"
handler: runc

提交之后可以发现报错,因为 label key 不满足我们的要求

1
2
3
4
5
6
7
▶ kubectl apply -f config/samples/                                          
Error from server (label key: xxx: taint or label key must validatedy by ^node-pool.lailin.xyz/*[a-zA-z0-9]*$): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"nodes.lailin.xyz/v1\",\"kind\":\"NodePool\",\"metadata\":{\"annotations\":{},\"name\":\"worker\"},\"spec\":{\"handler\":\"runc\",\"labels\":{\"xxx\":\"10\"}}}\n"}},"spec":{"labels":{"node-pool.lailin.xyz/worker":null,"xxx":"10"},"taints":null}}
to:
Resource: "nodes.lailin.xyz/v1, Resource=nodepools", GroupVersionKind: "nodes.lailin.xyz/v1, Kind=NodePool"
Name: "worker", Namespace: ""
for: "config/samples/nodes_v1_nodepool.yaml": admission webhook "vnodepool.kb.io" denied the request: label key: xxx: taint or label key must validatedy by ^node-pool.lailin.xyz/*[a-zA-z0-9]*$

再用一个正常的 yaml 测试

1
2
3
4
5
6
7
8
apiVersion: nodes.lailin.xyz/v1
kind: NodePool
metadata:
name: worker
spec:
labels:
"node-pool.lailin.xyz/xxx": "10"
handler: runc

可以正常提交

1
2
 kubectl apply -f config/samples/                     
nodepool.nodes.lailin.xyz/worker configured

本地调试

虽然 kubebuilder 已经为我们做了很多事情将服务部署运行基本傻瓜化了,但是每次做一点点修改就需要重新编译部署还是非常的麻烦,所以我们来看看如何在本地进行联调。

PS: 这里会用到之前 4. kustomize 简明教程 讲到的 kustomize 的特性构建开发环境,如果忘记了可以先看看之前的文章哦

我们先看看 config/webhook/manifests.yaml这里面包含了两个准入控制的信息,不过他们的配置类似,我们看一个就行了,这里以 MutatingWebhookConfiguration 为例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
creationTimestamp: null
name: mutating-webhook-configuration
webhooks:
- admissionReviewVersions:
- v1
- v1beta1
clientConfig:
service:
name: webhook-service
namespace: system
path: /mutate-nodes-lailin-xyz-v1-nodepool
failurePolicy: Fail
name: mnodepool.kb.io
rules:
- apiGroups:
- nodes.lailin.xyz
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- nodepools
sideEffects: None

主要是 clientConfig 的配置,如果想要本地联调,我们需要将 clientConfig.service 删掉,替换成

1
2
clientConfig:
url: https://host.docker.internal:9443/mutate-nodes-lailin-xyz-v1-nodepool

注意: host.docker.internal是 docker desktop 的默认域名,通过这个可以调用到宿主机上的服务,url path mutate-nodes-lailin-xyz-v1-nodepool需要和 service 中的 path 保持一致

然后再加上 caBundle

1
2
clientConfig:
caBundle: CA证书 base64 后的字符串

证书

想要本地联调需要先生成证书,我们使用 openssl 来生成,先创建一个 config/cert 文件夹,我们把证书都放到这里

首先创建一个 csr.conf文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
C = CN
ST = Guangzhou
L = Shenzhen
CN = host.docker.internal

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = host.docker.internal # 这里由于我们直接访问的是域名所以用 DNS

[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names

然后生成 CA 证书并且签发本地证书

1
2
3
4
5
6
7
8
9
# 生成 CA 证书
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -subj "/CN=host.docker.internal" -days 10000 -out ca.crt

# 签发本地证书
openssl genrsa -out tls.key 2048
openssl req -new -SHA256 -newkey rsa:2048 -nodes -keyout tls.key -out tls.csr -subj "/C=CN/ST=Shanghai/L=Shanghai/O=/OU=/CN=host.docker.internal"
openssl req -new -key tls.key -out tls.csr -config csr.conf
openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt -days 10000 -extensions v3_ext -extfile csr.conf

配置变更

我们为了和原本的开发体验保持一致,所以利用 kustomize 的特性新建一个 config/dev 文件夹,包含两个文件修改我们想要的配置

1
2
3
4
▶ tree config/dev
config/dev
├── kustomization.yaml
└── webhook_patch.yaml

先看一下 kustomization.yaml,从 default 文件夹中继承配置,然后使用 patches 修改一些配置,主要是分别给两种准入控制 WebHook 添加 url 字段,然后使用 webhook_patch.yaml 对两个文件做些统一的配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
resources:
- ../default

patches:
- patch: |
- op: "add"
path: "/webhooks/0/clientConfig/url"
value: "https://host.docker.internal:9443/mutate-nodes-lailin-xyz-v1-nodepool"
target:
kind: MutatingWebhookConfiguration
- patch: |
- op: "add"
path: "/webhooks/0/clientConfig/url"
value: "https://host.docker.internal:9443/validate-nodes-lailin-xyz-v1-nodepool"
target:
kind: ValidatingWebhookConfiguration
- path: webhook_patch.yaml
target:
group: admissionregistration.k8s.io

webhook_patch.yaml 这个主要是移除 cert-manager.io 的 annotation,本地调试不需要使用它进行证书注入,然后移除掉 service 并且添加 CA 证书

1
2
3
4
5
6
7
- op: "remove"
path: "/metadata/annotations/cert-manager.io~1inject-ca-from"
- op: "remove"
path: "/webhooks/0/clientConfig/service"
- op: "add"
path: "/webhooks/0/clientConfig/caBundle"
value: CA 证书 base64 后的值

CA 证书的值可以通过以下命令获取

1
cat config/cert/ca.crt | base64 | tr -d '\n'

然后修改一下 main.go将证书文件夹指定到我们刚刚生成好的文件目录

1
2
3
4
5
6
7
8
9
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
Port: 9443,
HealthProbeBindAddress: probeAddr,
LeaderElection: enableLeaderElection,
LeaderElectionID: "97acaccf.lailin.xyz",
+ CertDir: "config/cert/", // 手动指定证书位置用于测试
})

为了方便调试,在 makefile 中添加

1
2
3
dev: manifests kustomize
cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
$(KUSTOMIZE) build config/dev | kubectl apply -f -

最后执行一下 make dev 然后再执行 make run 就行了

总结

今天完成了准入控制 WebHook 的实现,虽然这个例子可能不太好,如果只需要校验正则,直接配置一下//+kubebuilder:validation:Pattern=string就行了,但是学习了这个之后其实可以做很多事情,例如给 pod 增加 sidecar 根据应用类型的不同注入不同的一些 agent 等等

kubebuilder 的功能也使用的差不多了,知其然也要知其所以然,我们下一篇来看看源码

关注我获取更新

wechat
知乎
github

猜你喜欢