EvenChan's Ops.

k8s之纵向扩缩容Vpa

字数统计: 2.4k阅读时长: 11 min
2021/01/12

vpa

纵向容器自动缩放器(VPA)使用户无需设置最新的资源限制和对容器中容器的要求。配置后,它将根据使用情况自动设置请求,从而允许在节点上进行适当的调度,以便为每个Pod提供适当的资源量。它还将保持限制和初始容器配置中指定的请求之间的比率。

它既可以根据资源的使用情况来缩减对资源过度使用的Pod,也可以对资源需求不足的向上扩展Pod。

Kubernetes VPA 包含以下组件:

  • Recommender:用于根据监控指标结合内置机制给出资源建议值
  • Updater:用于实时更新 pod resource requests
  • History Storage:用于采集和存储监控数据
  • Admission Controller: 用于在 pod 创建时修改 resource requests

架构图

VPA体系结构图

主要流程是:Recommender在启动时从History Storage获取历史数据,根据内置机制修改VPA API object资源建议值。Updater监听VPA API object,依据建议值动态修改 pod resource requests。VPA Admission Controller则是用于 pod 创建时修改 pod resource requests。History Storage则是通过Kubernetes Metrics API采集和存储监控数据。

CPU 和内存的建议值均是依据历史数据+固定机制计算而成。

在 Kubernetes VPA 中缺少资源回收的机制,但Recommender却可以配合Updater动态修改 pod resource requests 的值。也就是说 pod resource requests - 推荐值 = 资源回收值。这间接实现了资源回收的功能。

VPA工作流程

空值

  1. VPA连续检查您在设置过程中配置的指标值,默认间隔为10秒
  2. 如果达到阈值,VPA会尝试更改分配的内存和/或CPU
  3. VPA主要更新部署或复制控制器规范中的资源
  4. 重新启动Pod后,新资源将全部应用于创建的实例。

使用VPA时需要考虑以下几点:

  • 如果不重新启动Pod,将无法更改资源。到目前为止,主要的合理性在于,这样的变化可能会导致很多不稳定。因此,考虑重新启动Pod并根据新分配的资源对其进行调度。
  • VPA和HPA尚不兼容,不能在同一pod上使用。如果要在同一群集中同时使用它们,请确保在设置中分开它们的作用域。
  • VPA仅根据观察到的过去和当前资源使用情况来调整容器的资源请求。它没有设置资源限制。对于行为不当的应用程序可能会出现问题,这些应用程序开始使用越来越多的资源,导致pod被Kubernetes杀死。

安装

当前默认版本是Vertical Pod Autoscaler 0.8.0

1
2
3
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler
./hack/vpa-up.sh

注意:该脚本当前读取环境变量:$REGISTRY$TAG。除非要使用VPA的非默认版本,否则请确保未设置它们。

如果出现如下报错,我们可以升级openssl到支持-addext选项的版本,或在下载via-release-0.8这个分支的代码并使用./hack/vpa-up.sh

1
2
ERROR: Failed to create CA certificate for self-signing. If the error is "unknown option -addext", update your openssl version or deploy VPA from the vpa-release-0.8 branch.
deployment.apps/vpa-admission-controller created

安装好之后会创建三个pod,这三个pod使用的镜像需要翻墙下载

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[root@master01 vertical-pod-autoscaler]# kubectl -n kube-system  get pod 
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-688c5dc8c7-rcn2t 1/1 Running 0 59d
calico-node-n7sj8 1/1 Running 7 89d
calico-node-vk9c9 1/1 Running 1 89d
coredns-6955765f44-mgcc4 1/1 Running 0 59d
coredns-6955765f44-vwzxt 1/1 Running 0 59d
csi-disk-provisioner-0 2/2 Running 0 17d
csi-diskplugin-jmw74 2/2 Running 0 17d
csi-diskplugin-mv99m 2/2 Running 0 17d
dex-1591193803-6844b4846c-4wwzg 1/1 Running 1 19d
dex-k8s-authenticator-1591193977-54f66bd94f-tv2rl 1/1 Running 1 59d
etcd-master01 1/1 Running 8 89d
kube-apiserver-master01 1/1 Running 0 19d
kube-controller-manager-master01 1/1 Running 0 13d
kube-proxy-s98fh 1/1 Running 0 59d
kube-proxy-zn2l6 1/1 Running 0 59d
kube-scheduler-master01 1/1 Running 0 13d
kube-sealyun-lvscare-node01 1/1 Running 0 89d
kuboard-85d499bd9b-xmvtp 1/1 Running 0 15d
metrics-server-6b4d999dcb-xzx4p 1/1 Running 0 13d
snapshot-controller-0 1/1 Running 0 14d
traefik-0 1/1 Running 0 20d
vpa-admission-controller-6b88678dcb-d5vb6 1/1 Running 0 7s
vpa-recommender-5fc9b4c4cb-d5j8l 1/1 Running 0 32s
vpa-updater-865849df48-rcl94 1/1 Running 0 40s
[root@master01 vertical-pod-autoscaler]# kubectl -n kube-system get deployments.apps -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES
....
vpa-admission-controller 1/1 1 1 62m admission-controller us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-admission-controller:0.8.0 app=vpa-admission-controller
vpa-recommender 1/1 1 1 62m recommender us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-recommender:0.8.0 app=vpa-recommender
vpa-updater 1/1 1 1 62m updater us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-updater:0.8.0 app=vpa-updater

这里可以使用 misterli/vpa-recommender:0.8.0,misterli/vpa-updater:0.8.0,misterli/vpa-admission-controller:0.8.0这三个镜像进行替换

使用

  • VPA 有以下四种更新策略:

    •Initial:仅在 Pod 创建时修改资源请求,以后都不再修改。

    •Auto:默认策略,在 Pod 创建时修改资源请求,并且在 Pod 更新时也会修改。

    •Recreate:类似 Auto,在 Pod 的创建和更新时都会修改资源请求,不同的是,只要Pod 中的请求值与新的推荐值不同,VPA 都会驱逐该 Pod,然后使用新的推荐值重新启一个。因此,一般不使用该策略,而是使用 Auto,除非你真的需要保证请求值是最新的推荐值。

    •Off:不改变 Pod 的资源请求,不过仍然会在 VPA 中设置资源的推荐值。

我们部署一个使用vpa的示例:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
---
apiVersion: "autoscaling.k8s.io/v1beta2"
kind: VerticalPodAutoscaler
metadata:
name: hamster-vpa
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: hamster
updatePolicy:
updateMode: "Auto"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hamster
spec:
selector:
matchLabels:
app: hamster
replicas: 2
template:
metadata:
labels:
app: hamster
spec:
containers:
- name: hamster
image: ubuntu:16.04
resources:
requests:
cpu: 100m
memory: 50Mi
command: ["/bin/sh"]
args:
- "-c"
- "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"

我们发现稍等片刻vpa会对pod的资源进行动态更新

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[root@master01 vertical-pod-autoscaler]# kubectl apply -f test.yaml 
verticalpodautoscaler.autoscaling.k8s.io/hamster-vpa created
deployment.apps/hamster created
[root@master01 vertical-pod-autoscaler]# kubectl get pod -w
NAME READY STATUS RESTARTS AGE
event-exporter-56b4dbc8fb-gsmx4 1/1 Running 0 40d
hamster-bb5bb8754-krljt 1/1 Running 0 21s
hamster-bb5bb8754-nvhf2 1/1 Running 0 21s
hamster-bb5bb8754-krljt 1/1 Terminating 0 92s
hamster-bb5bb8754-qs8dl 0/1 Pending 0 0s
hamster-bb5bb8754-qs8dl 0/1 Pending 0 0s
hamster-bb5bb8754-qs8dl 0/1 ContainerCreating 0 0s
hamster-bb5bb8754-qs8dl 0/1 ContainerCreating 0 0s
hamster-bb5bb8754-qs8dl 1/1 Running 0 1s
hamster-bb5bb8754-krljt 0/1 Terminating 0 2m3s
hamster-bb5bb8754-krljt 0/1 Terminating 0 2m4s
hamster-bb5bb8754-krljt 0/1 Terminating 0 2m4s
hamster-bb5bb8754-nvhf2 1/1 Terminating 0 2m32s
hamster-bb5bb8754-d2rnh 0/1 Pending 0 0s
hamster-bb5bb8754-d2rnh 0/1 Pending 0 0s
hamster-bb5bb8754-d2rnh 0/1 ContainerCreating 0 0s
hamster-bb5bb8754-d2rnh 0/1 ContainerCreating 0 1s
hamster-bb5bb8754-d2rnh 1/1 Running 0 2s

查看一下vpa的详情可以看到vpa建议的值

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@master01 vertical-pod-autoscaler]# kubectl describe vpa hamster-vpa 
Name: hamster-vpa
Namespace: default
.....
Recommendation:
Container Recommendations:
Container Name: hamster
Lower Bound:
Cpu: 579m
Memory: 262144k
Target:
Cpu: 587m
Memory: 262144k
Uncapped Target:
Cpu: 587m
Memory: 262144k
Upper Bound:
Cpu: 4554m
Memory: 262144k
Events: <none>

查看当前pod的资源限制,pod的requests已经被动态调整了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@master01 vertical-pod-autoscaler]# kubectl get pod hamster-bb5bb8754-kp9jk  -o yaml 
apiVersion: v1
kind: Pod
....
name: hamster
resources:
requests:
cpu: 587m
memory: 262144k
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-jqvcc
readOnly: true
.....

还可以限制动态调整的范围

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
---
apiVersion: "autoscaling.k8s.io/v1beta2"
kind: VerticalPodAutoscaler
metadata:
name: hamster-vpa
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: hamster
resourcePolicy:
containerPolicies:
- containerName: '*'
minAllowed:
cpu: 100m
memory: 50Mi
maxAllowed:
cpu: 1
memory: 500Mi
controlledResources: ["cpu", "memory"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hamster
spec:
selector:
matchLabels:
app: hamster
replicas: 2
template:
metadata:
labels:
app: hamster
spec:
containers:
- name: hamster
image: ubuntu:16.04
resources:
requests:
cpu: 100m
memory: 50Mi
command: ["/bin/sh"]
args:
- "-c"
- "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"

上面的yaml创建了一个包含2个Pod的deployment,每个Pod运行一个请求100m cpu 的容器,并尝试使用略高于500m cpu的容器。该命令还会创建一个指向deployment的VPA配置。VPA将观察Pod的行为,大约5分钟后,它们应使用更高的CPU请求进行更新(请注意,VPA不会在deployment中修改模板,但Pod的实际请求会被更新)。要查看VPA配置和当前推荐的资源请求:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[root@master01 vertical-pod-autoscaler]# kubectl describe vpa hamster-vpa 
Name: hamster-vpa
Namespace: default
...
Status:
Conditions:
Last Transition Time: 2020-09-01T03:55:47Z
Status: True
Type: RecommendationProvided
Recommendation:
Container Recommendations:
Container Name: hamster
Lower Bound:
Cpu: 552m
Memory: 262144k
Target:
Cpu: 587m
Memory: 262144k
Uncapped Target:
Cpu: 587m
Memory: 262144k
Upper Bound:
Cpu: 1
Memory: 370459537
Events: <none>

查看pod 详情,发现此时 resources字段已经被更改

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[root@master01 vertical-pod-autoscaler]# kubectl get pod hamster-bb5bb8754-gcrbc  -o yaml 
apiVersion: v1
kind: Pod
....
image: ubuntu:16.04
imagePullPolicy: IfNotPresent
name: hamster
resources:
requests:
cpu: 587m
memory: 262144k
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
...
[root@master01 vertical-pod-autoscaler]# kubectl get pod -w
NAME READY STATUS RESTARTS AGE
event-exporter-56b4dbc8fb-gsmx4 1/1 Running 0 40d
hamster-bb5bb8754-ghggr 1/1 Running 0 5s
hamster-bb5bb8754-xnndg 1/1 Running 0 5s
hamster-bb5bb8754-xnndg 1/1 Terminating 0 62s
hamster-bb5bb8754-gcrbc 0/1 Pending 0 0s
hamster-bb5bb8754-gcrbc 0/1 Pending 0 0s
hamster-bb5bb8754-gcrbc 0/1 ContainerCreating 0 0s
hamster-bb5bb8754-gcrbc 0/1 ContainerCreating 0 1s
hamster-bb5bb8754-gcrbc 1/1 Running 0 1s
hamster-bb5bb8754-xnndg 0/1 Terminating 0 93s
hamster-bb5bb8754-xnndg 0/1 Terminating 0 94s
hamster-bb5bb8754-xnndg 0/1 Terminating 0 94s
hamster-bb5bb8754-ghggr 1/1 Terminating 0 2m2s
hamster-bb5bb8754-gwxgw 0/1 Pending 0 0s
hamster-bb5bb8754-gwxgw 0/1 Pending 0 0s
hamster-bb5bb8754-gwxgw 0/1 ContainerCreating 0 0s
hamster-bb5bb8754-gwxgw 0/1 ContainerCreating 0 1s
hamster-bb5bb8754-gwxgw 1/1 Running 0 2s

event事件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
0s          Normal    EvictedByVPA        pod/hamster-bb5bb8754-xnndg     Pod was evicted by VPA Updater to apply resource recommendation.
0s Normal SuccessfulCreate replicaset/hamster-bb5bb8754 Created pod: hamster-bb5bb8754-gcrbc
0s Normal Scheduled pod/hamster-bb5bb8754-gcrbc Successfully assigned default/hamster-bb5bb8754-gcrbc to node01
0s Normal Killing pod/hamster-bb5bb8754-xnndg Stopping container hamster
0s Normal Pulled pod/hamster-bb5bb8754-gcrbc Container image "ubuntu:16.04" already present on machine
0s Normal Created pod/hamster-bb5bb8754-gcrbc Created container hamster
0s Normal Started pod/hamster-bb5bb8754-gcrbc Started container hamster
0s Normal EvictedByVPA pod/hamster-bb5bb8754-ghggr Pod was evicted by VPA Updater to apply resource recommendation.
0s Normal Killing pod/hamster-bb5bb8754-ghggr Stopping container hamster
0s Normal SuccessfulCreate replicaset/hamster-bb5bb8754 Created pod: hamster-bb5bb8754-gwxgw
0s Normal Scheduled pod/hamster-bb5bb8754-gwxgw Successfully assigned default/hamster-bb5bb8754-gwxgw to node01
0s Normal Pulled pod/hamster-bb5bb8754-gwxgw Container image "ubuntu:16.04" already present on machine
0s Normal Created pod/hamster-bb5bb8754-gwxgw Created container hamster
0s Normal Started pod/hamster-bb5bb8754-gwxgw Started container hamster
CATALOG
  1. 1. vpa
    1. 1.1. 架构图
    2. 1.2. VPA工作流程
    3. 1.3. 安装
    4. 1.4. 使用