CCE基于nginx-ingress实现灰度发布
更新时间：2024-09-20
本文介绍如何利用百度云容器服务的Ingress功能，实现蓝绿发布。
背景信息：
灰度及蓝绿发布是为新版本创建一个与老版本完全一致的生产环境，在不影响老版本的前提下，按照一定的规则把部分流量切换到新版本，当新版本试运行一段时间没有问题后，将用户的全量流量从老版本迁移至新版本。
其中A/B测试就是一种灰度发布方式，一部分用户继续使用老版本的服务，将一部分用户的流量切换到新版本，如果新版本运行稳定，则逐步将所有用户迁移到新版本。
Ingress-Nginx Annotation 简介
CCE 基于 Nginx Ingress Controller 实现了项目的网关，作为项目对外的流量入口和项目中各个服务的反向代理。而 Ingress-Nginx 支持配置 Ingress Annotations 来实现不同场景下的灰度发布和测试，可以满足金丝雀发布、蓝绿部署与 A/B 测试等业务场景。
Nginx Annotations 支持以下 4 种 Canary 规则：
- nginx.ingress.kubernetes.io/canary-by-header：基于 Request Header 的流量切分，适用于灰度发布以及 A/B 测试。当 Request Header 设置为 always时，请求将会被一直发送到 Canary 版本；当 Request Header 设置为 never时，请求不会被发送到 Canary 入口；对于任何其他 Header 值，将忽略 Header，并通过优先级将请求与其他金丝雀规则进行优先级的比较。
- nginx.ingress.kubernetes.io/canary-by-header-value：要匹配的 Request Header 的值，用于通知 Ingress 将请求路由到 Canary Ingress 中指定的服务。当 Request Header 设置为此值时，它将被路由到 Canary 入口。该规则允许用户自定义 Request Header 的值，必须与上一个 annotation (即：canary-by-header）一起使用。
- nginx.ingress.kubernetes.io/canary-weight：基于服务权重的流量切分，适用于蓝绿部署，权重范围 0 - 100 按百分比将请求路由到 Canary Ingress 中指定的服务。权重为 0 意味着该金丝雀规则不会向 Canary 入口的服务发送任何请求。权重为 100 意味着所有请求都将被发送到 Canary 入口。
- nginx.ingress.kubernetes.io/canary-by-cookie：基于 Cookie 的流量切分，适用于灰度发布与 A/B 测试。用于通知 Ingress 将请求路由到 Canary Ingress 中指定的服务的cookie。当 cookie 值设置为 always时，它将被路由到 Canary 入口；当 cookie 值设置为 never时，请求不会被发送到 Canary 入口；对于任何其他值，将忽略 cookie 并将请求与其他金丝雀规则进行优先级的比较。
canary规则优先级：canary-by-header - > canary-by-cookie - > canary-weight
安装nginx-ingress-controller
通过组件安装
进入“容器引擎”-“集群详情”，点击左侧导航栏“组件管理”，找到CCE Ingress Nginx Controller，点击安装；
配置Ingress Nginx Controller：填写IngressClass名称，选择命名空间，选择节点组，如果没有节点组点击“新建节点组”，配置污点设置、容器配额、容忍设置、服务访问；
点击确认，完成创建。
通过YAML安装
Plain Text
1# yaml文件内容见附录
2kubectl apply -f ingress-nginx.yaml
3kubectl apply -f ingress-nginx-service.yaml部署production任务
1. 创建production应用资源
Plain Text
1kubectl apply -f production.yaml
Plain Text
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: production
5spec:
6 replicas: 1
7 selector:
8 matchLabels:
9 app: production
10 template:
11 metadata:
12 labels:
13 app: production
14 spec:
15 containers:
16 - name: production
17 image: registry.baidubce.com/jpaas-public/echoserver:1.10
18 ports:
19 - containerPort: 8080
20 env:
21 - name: NODE_NAME
22 valueFrom:
23 fieldRef:
24 fieldPath: spec.nodeName
25 - name: POD_NAME
26 valueFrom:
27 fieldRef:
28 fieldPath: metadata.name
29 - name: POD_NAMESPACE
30 valueFrom:
31 fieldRef:
32 fieldPath: metadata.namespace
33 - name: POD_IP
34 valueFrom:
35 fieldRef:
36 fieldPath: status.podIP
37
38---
39
40apiVersion: v1
41kind: Service
42metadata:
43 name: production
44 labels:
45 app: production
46spec:
47 ports:
48 - port: 80
49 targetPort: 8080
50 protocol: TCP
51 name: http
52 selector:
53 app: production2. 创建 Production 版本的应用路由 (Ingress)
通过控制台创建
点击左侧导航栏“流量接入”后点击“Ingress”,界面点击“新建Ingress”；
如图完成Ingress的配置：
通过YAML创建
Plain Text
1kubectl apply -f production.ingress.yaml
Plain Text
1apiVersion: extensions/v1beta1
2kind: Ingress
3metadata:
4 name: production
5 annotations:
6 kubernetes.io/ingress.class: nginx
7spec:
8 rules:
9 - host: cce.canary.io
10 http:
11 paths:
12 - backend:
13 serviceName: production
14 servicePort: 803、本机访问应用：
绑定hosts，集群内访问选择 clusterIP，集群外访问选择 externalIP （即 BLB IP）：
Plain Text
1vi /etc/hosts 增加一行，如“106.12.7.210 cce.canary.io”
Plain Text
1{ip} cce.canary.io
curl cce.canary.io 如下访问成功
创建canary版本任务
1. 创建canary版本应用资源
Plain Text
1kubectl apply -f canary.yaml
Plain Text
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: canary
5spec:
6 replicas: 1
7 selector:
8 matchLabels:
9 app: canary
10 template:
11 metadata:
12 labels:
13 app: canary
14 spec:
15 containers:
16 - name: canary
17 image: registry.baidubce.com/jpaas-public/echoserver:1.10
18 ports:
19 - containerPort: 8080
20 env:
21 - name: NODE_NAME
22 valueFrom:
23 fieldRef:
24 fieldPath: spec.nodeName
25 - name: POD_NAME
26 valueFrom:
27 fieldRef:
28 fieldPath: metadata.name
29 - name: POD_NAMESPACE
30 valueFrom:
31 fieldRef:
32 fieldPath: metadata.namespace
33 - name: POD_IP
34 valueFrom:
35 fieldRef:
36 fieldPath: status.podIP
37
38---
39
40apiVersion: v1
41kind: Service
42metadata:
43 name: canary
44 labels:
45 app: canary
46spec:
47 ports:
48 - port: 80
49 targetPort: 8080
50 protocol: TCP
51 name: http
52 selector:
53 app: canary2. 基于权重创建 canary 版本的应用路由 (Ingress)
通过控制台创建
点击左侧导航栏“流量接入”后点击“Ingress”,界面点击“新建Ingress”；
在高级设置-注释中点击“+增加注释”, 增加以下标签键：
Plain Text
1#高级配置中的标签设置如下：
2nginx.ingress.kubernetes.io/canary: "true"
3nginx.ingress.kubernetes.io/canary-weight: "30"
通过YAML创建
Plain Text
1kubectl apply -f canary.ingress.yaml -n canary-demo
Plain Text
1apiVersion: extensions/v1beta1
2kind: Ingress
3metadata:
4 name: canary
5 annotations:
6 kubernetes.io/ingress.class: nginx
7 nginx.ingress.kubernetes.io/canary: "true"
8 nginx.ingress.kubernetes.io/canary-weight: "30"
9spec:
10 rules:
11 - host: cce.canary.io
12 http:
13 paths:
14 - backend:
15 serviceName: canary
16 servicePort: 803. 访问应用域名验证
Plain Text
1for i in $(seq 1 10); do curl cce.canary.io | grep Hostname ; done如下图，流量部分打入canary
应用的 Canary 版本基于权重 (30%) 进行流量切分后，访问到 Canary 版本的概率接近 30%，流量比例可能会有小范围的浮动，属正常现象
附录：ingress-nginx相关yaml文件
- ingress-nginx.yaml
Plain Text
1apiVersion: v1
2kind: Namespace
3metadata:
4 name: ingress-nginx
5 labels:
6 app.kubernetes.io/name: ingress-nginx
7 app.kubernetes.io/part-of: ingress-nginx
8
9---
10
11kind: ConfigMap
12apiVersion: v1
13metadata:
14 name: nginx-configuration
15 namespace: ingress-nginx
16 labels:
17 app.kubernetes.io/name: ingress-nginx
18 app.kubernetes.io/part-of: ingress-nginx
19
20---
21kind: ConfigMap
22apiVersion: v1
23metadata:
24 name: tcp-services
25 namespace: ingress-nginx
26 labels:
27 app.kubernetes.io/name: ingress-nginx
28 app.kubernetes.io/part-of: ingress-nginx
29
30---
31kind: ConfigMap
32apiVersion: v1
33metadata:
34 name: udp-services
35 namespace: ingress-nginx
36 labels:
37 app.kubernetes.io/name: ingress-nginx
38 app.kubernetes.io/part-of: ingress-nginx
39
40---
41apiVersion: v1
42kind: ServiceAccount
43metadata:
44 name: nginx-ingress-serviceaccount
45 namespace: ingress-nginx
46 labels:
47 app.kubernetes.io/name: ingress-nginx
48 app.kubernetes.io/part-of: ingress-nginx
49
50---
51apiVersion: rbac.authorization.k8s.io/v1beta1
52kind: ClusterRole
53metadata:
54 name: nginx-ingress-clusterrole
55 labels:
56 app.kubernetes.io/name: ingress-nginx
57 app.kubernetes.io/part-of: ingress-nginx
58rules:
59 - apiGroups:
60 - ""
61 resources:
62 - configmaps
63 - endpoints
64 - nodes
65 - pods
66 - secrets
67 verbs:
68 - list
69 - watch
70 - apiGroups:
71 - ""
72 resources:
73 - nodes
74 verbs:
75 - get
76 - apiGroups:
77 - ""
78 resources:
79 - services
80 verbs:
81 - get
82 - list
83 - watch
84 - apiGroups:
85 - ""
86 resources:
87 - events
88 verbs:
89 - create
90 - patch
91 - apiGroups:
92 - "extensions"
93 - "networking.k8s.io"
94 resources:
95 - ingresses
96 verbs:
97 - get
98 - list
99 - watch
100 - apiGroups:
101 - "extensions"
102 - "networking.k8s.io"
103 resources:
104 - ingresses/status
105 verbs:
106 - update
107
108---
109apiVersion: rbac.authorization.k8s.io/v1beta1
110kind: Role
111metadata:
112 name: nginx-ingress-role
113 namespace: ingress-nginx
114 labels:
115 app.kubernetes.io/name: ingress-nginx
116 app.kubernetes.io/part-of: ingress-nginx
117rules:
118 - apiGroups:
119 - ""
120 resources:
121 - configmaps
122 - pods
123 - secrets
124 - namespaces
125 verbs:
126 - get
127 - apiGroups:
128 - ""
129 resources:
130 - configmaps
131 resourceNames:
132 # Defaults to "<election-id>-<ingress-class>"
133 # Here: "<ingress-controller-leader>-<nginx>"
134 # This has to be adapted if you change either parameter
135 # when launching the nginx-ingress-controller.
136 - "ingress-controller-leader-nginx"
137 verbs:
138 - get
139 - update
140 - apiGroups:
141 - ""
142 resources:
143 - configmaps
144 verbs:
145 - create
146 - apiGroups:
147 - ""
148 resources:
149 - endpoints
150 verbs:
151 - get
152
153---
154apiVersion: rbac.authorization.k8s.io/v1beta1
155kind: RoleBinding
156metadata:
157 name: nginx-ingress-role-nisa-binding
158 namespace: ingress-nginx
159 labels:
160 app.kubernetes.io/name: ingress-nginx
161 app.kubernetes.io/part-of: ingress-nginx
162roleRef:
163 apiGroup: rbac.authorization.k8s.io
164 kind: Role
165 name: nginx-ingress-role
166subjects:
167 - kind: ServiceAccount
168 name: nginx-ingress-serviceaccount
169 namespace: ingress-nginx
170
171---
172apiVersion: rbac.authorization.k8s.io/v1beta1
173kind: ClusterRoleBinding
174metadata:
175 name: nginx-ingress-clusterrole-nisa-binding
176 labels:
177 app.kubernetes.io/name: ingress-nginx
178 app.kubernetes.io/part-of: ingress-nginx
179roleRef:
180 apiGroup: rbac.authorization.k8s.io
181 kind: ClusterRole
182 name: nginx-ingress-clusterrole
183subjects:
184 - kind: ServiceAccount
185 name: nginx-ingress-serviceaccount
186 namespace: ingress-nginx
187
188---
189
190apiVersion: apps/v1
191kind: Deployment
192metadata:
193 name: nginx-ingress-controller
194 namespace: ingress-nginx
195 labels:
196 app.kubernetes.io/name: ingress-nginx
197 app.kubernetes.io/part-of: ingress-nginx
198spec:
199 replicas: 1
200 selector:
201 matchLabels:
202 app.kubernetes.io/name: ingress-nginx
203 app.kubernetes.io/part-of: ingress-nginx
204 template:
205 metadata:
206 labels:
207 app.kubernetes.io/name: ingress-nginx
208 app.kubernetes.io/part-of: ingress-nginx
209 annotations:
210 prometheus.io/port: "10254"
211 prometheus.io/scrape: "true"
212 spec:
213 # wait up to five minutes for the drain of connections
214 terminationGracePeriodSeconds: 300
215 serviceAccountName: nginx-ingress-serviceaccount
216 nodeSelector:
217 kubernetes.io/os: linux
218 containers:
219 - name: nginx-ingress-controller
220 image: registry.baidubce.com/jpaas-public/nginx-ingress-controller:0.30.0
221 args:
222 - /nginx-ingress-controller
223 - --configmap=$(POD_NAMESPACE)/nginx-configuration
224 - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
225 - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
226 - --publish-service=$(POD_NAMESPACE)/ingress-nginx
227 - --annotations-prefix=nginx.ingress.kubernetes.io
228 securityContext:
229 allowPrivilegeEscalation: true
230 capabilities:
231 drop:
232 - ALL
233 add:
234 - NET_BIND_SERVICE
235 # www-data -> 101
236 runAsUser: 101
237 env:
238 - name: POD_NAME
239 valueFrom:
240 fieldRef:
241 fieldPath: metadata.name
242 - name: POD_NAMESPACE
243 valueFrom:
244 fieldRef:
245 fieldPath: metadata.namespace
246 ports:
247 - name: http
248 containerPort: 80
249 protocol: TCP
250 - name: https
251 containerPort: 443
252 protocol: TCP
253 livenessProbe:
254 failureThreshold: 3
255 httpGet:
256 path: /healthz
257 port: 10254
258 scheme: HTTP
259 initialDelaySeconds: 10
260 periodSeconds: 10
261 successThreshold: 1
262 timeoutSeconds: 10
263 readinessProbe:
264 failureThreshold: 3
265 httpGet:
266 path: /healthz
267 port: 10254
268 scheme: HTTP
269 periodSeconds: 10
270 successThreshold: 1
271 timeoutSeconds: 10
272 lifecycle:
273 preStop:
274 exec:
275 command:
276 - /wait-shutdown
277
278---
279
280apiVersion: v1
281kind: LimitRange
282metadata:
283 name: ingress-nginx
284 namespace: ingress-nginx
285 labels:
286 app.kubernetes.io/name: ingress-nginx
287 app.kubernetes.io/part-of: ingress-nginx
288spec:
289 limits:
290 - min:
291 memory: 90Mi
292 cpu: 100m
293 type: Container- ingress-nginx-service.yaml
Plain Text
1kind: Service
2apiVersion: v1
3metadata:
4 name: ingress-nginx
5 namespace: ingress-nginx
6 labels:
7 app.kubernetes.io/name: ingress-nginx
8 app.kubernetes.io/part-of: ingress-nginx
9spec:
10 externalTrafficPolicy: Cluster
11 type: LoadBalancer
12 selector:
13 app.kubernetes.io/name: ingress-nginx
14 app.kubernetes.io/part-of: ingress-nginx
15 ports:
16 - name: http
17 port: 80
18 protocol: TCP
19 targetPort: http
20 - name: https
21 port: 443
22 protocol: TCP
23 targetPort: https