Month: April 2022

Reserve resources for kubelet

During the operation of kubernetes cluster, kubelet needs to regularly send heartbeat to API server and report the operation status

In fact, there are great potential safety hazards here

Because kubelet and pods running on the node need to share CPU and ram resources, kubelet may not get enough resources to run itself in extreme cases

In other words, pods is likely to consume all the physical resources of the node, resulting in the node’s heartbeat stopping or untimely reporting. The API server may mistakenly believe that the node is offline
In an idealistic state, all pods deployed to this node should not consume 100% of physical resources
for example

100m CPU * 9, so in theory, 100m will remain
However, due to the extremely complex cluster environment, there is no guarantee that the scheduler will reserve resources for each node, and there is no guarantee that every authorized operator will notice this problem
Therefore, there is often resource overload, resulting in kubelet’s heartbeat stopping
The solution is very simple and crude

Deploy a “placeholder” application and allocate its requests CPU resources to 100m
Because there is no specific application for this placeholder application (such as nginx), it will not really consume CPU

Here, requests are easy to control
1000m single core can support
Normal application of 100m * 9, space occupying nginx of 100m * 1,
However, requests only affect container scheduling and CPU competition, and can not guarantee that CPU resources will not be exhausted
Set limits to
100m * 9 application, 100m * 1 space occupying nginx, this time is good, but limits can be oversold, so there is no guarantee that all administrators will set aside that 100m for you
Conclusion,
It’s best to use only limits, which can ensure that requests are consistent with limits by default
Because requests cannot be oversold, when nginx occupies 100m, the nine normal applications that only write limits cannot exceed 900mhttps://googleads.g.doubleclick.net/pagead/ads?client=ca-pub-7712570106118575&output=html&h=280&adk=2825196814&adf=4093481066&pi=t.aa~a.2603011395~i.46~rp.1&w=706&fwrn=4&fwrnh=100&lmt=1650205817&num_ads=1&rafmt=1&armr=3&sem=mc&pwprc=1923896069&psa=1&ad_type=text_image&format=706×280&url=https%3A%2F%2Fstudyk8s.com%2F2022%2F04%2F17%2F3766%2F&fwr=0&pra=3&rh=177&rw=706&rpe=1&resp_fmts=3&wgl=1&fa=27&uach=WyJXaW5kb3dzIiwiMTAuMC4wIiwieDg2IiwiIiwiMTAwLjAuNDg5Ni4xMjciLFtdLG51bGwsbnVsbCwiNjQiLFtbIiBOb3QgQTtCcmFuZCIsIjk5LjAuMC4wIl0sWyJDaHJvbWl1bSIsIjEwMC4wLjQ4OTYuMTI3Il0sWyJHb29nbGUgQ2hyb21lIiwiMTAwLjAuNDg5Ni4xMjciXV0sZmFsc2Vd&tt_state=W3siaXNzdWVyT3JpZ2luIjoiaHR0cHM6Ly9wYWdlYWQyLmdvb2dsZXN5bmRpY2F0aW9uLmNvbSIsInN0YXRlIjoyMCwiaGFzUmVkZW1wdGlvblJlY29yZCI6dHJ1ZX1d&dt=1650205817490&bpp=1&bdt=365&idt=1&shv=r20220413&mjsv=m202204110101&ptt=9&saldr=aa&abxe=1&cookie=ID%3De1e9e75d484277dc-22e2d1cebbd10034%3AT%3D1649940877%3ART%3D1649940877%3AS%3DALNI_MYLFH7MHMVP9tSJTFU30yFvzIVVCA&prev_fmts=0x0%2C294x245&nras=2&correlator=3409815582494&frm=20&pv=1&ga_vid=510179359.1616343974&ga_sid=1650205817&ga_hid=211755682&ga_fc=1&u_tz=480&u_his=9&u_h=1280&u_w=3072&u_ah=1240&u_aw=3072&u_cd=30&u_sd=1.25&dmc=8&adx=200&ady=1246&biw=1518&bih=1160&scr_x=0&scr_y=0&eid=44759875%2C44759926%2C44759837%2C44761793%2C31067083%2C31061828%2C21067496&oid=2&pvsid=2181360227943243&pem=820&tmod=767505671&uas=0&nvt=1&ref=https%3A%2F%2Fstudyk8s.com%2Fwp-admin%2Fpost.php%3Fpost%3D3766%26action%3Dedit&eae=0&fc=1408&brdim=1529%2C0%2C1529%2C0%2C3072%2C0%2C1550%2C1247%2C1535%2C1160&vis=1&rsz=%7C%7Cs%7C&abl=NS&fu=128&bc=31&ifi=3&uci=a!3&btvi=1&fsb=1&xpc=8KLUnNbKwo&p=https%3A//studyk8s.com&dtd=30
The above is a folk prescription to deal with this problem. Does the kubernetes native application deal with this problem
The answer is yes, but trouble

• –Kube reserved is used to configure the amount of resources reserved for Kube components (kubelet, Kube proxy, dockerd, etc.), such as — Kube reserved = CPU = 1000m, memory = 8gi, ephemeral storage = 16gi.
• –Kube reserved CGroup. If you set — Kube reserved, you must set the corresponding CGroup, and the CGroup directory must be created in advance, otherwise kubelet will not be created automatically, resulting in kubelet startup failure. For example, set it to Kube reserved CGroup = / kubelet service 。 If this item is not set, the above — Kube reserved will not take effect.
• –System reserved is used to configure the amount of resources reserved for the system process, such as — system reserved = CPU = 500m, memory = 4Gi, ephemeral storage = 4Gi.
• –System reserved CGroup. If — system reserved is set, the corresponding CGroup must be set, and the CGroup directory must be created in advance, otherwise kubelet will not be created automatically, resulting in kubelet startup failure. For example, set it to system reserved CGroup = / system slice。 If this item is not set, the above — system reserved will not take effect.

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/cpuset/system.slice/kubelet.service /sys/fs/cgroup/hugetlb/system.slice/kubelet.service
ExecStart=/opt/kubernetes/bin/kubelet \
  --eviction-hard=memory.available<1024Mi,nodefs.available<10%,nodefs.inodesFree<5% \
  --system-reserved=cpu=0.5,memory=1G \
  --kube-reserved=cpu=0.5,memory=1G \
  --kube-reserved-cgroup=/system.slice/kubelet.service \
  --system-reserved-cgroup=/system.slice \
  --enforce-node-allocatable=pods,kube-reserved,system-reserved \
  --address=192.168.0.101 \
  --hostname-override=192.168.0.101 \
  --cgroup-driver=cgroupfs \
  --pod-infra-container-image=hub.breezey.top/kubernetes/pause-amd64:3.0 \
  --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
  --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
  --cert-dir=/opt/kubernetes/ssl \
  --cluster-dns=192.168.0.200 \
  --cluster-domain=k8s.breezey.top. \
  --hairpin-mode=promiscuous-bridge \
  --allow-privileged=true \
  --fail-swap-on=false \
  --serialize-image-pulls=false \
  --max-pods=60 \
  --logtostderr=true \
  --v=2 
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

And unfortunately, in the kubernetes cluster installed by RKE
Not in this way

The default is no kubelet.service

Need to add manually

Then specify through parameters

RKE kubernetes installed by default
This is the CGroup used by the kubelet process
6:cpu,cpuacct:/docker/4b28805380958bf3efb5c0d0a9c93ba42fab4d36e1be9036f3b9c4b01ef3bd62

it is below

How to make pod monopolize CPU kernel

When using docker, you can bind the container to a CPU kernel through the parameter cpuset

In this case, the number of context switches between CPUs by the operating system is greatly reduced, so the performance of applications in the container will be greatly improved.
In fact, cpuset is a common way to deploy online application type pods in production environment

To use functions similar to docker cpuset on kubernetes, the following two conditions need to be met
The QoS of the pod is guaranteed

Set requests and limits to the same value, such as 2

spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      limits:
        memory: "200Mi"
        cpu: "2"
      requests:
        memory: "200Mi"
        cpu: "2"

In this way, the above pod will be bound to two exclusive CPU cores

Kubernetes CPU resource control

To control CPU resources, you need to use the requests and limits attributes

In short
Requests shows the CPU of the process in cgroups on the CFS scheduler Shares, and limit is the bandwidth limit on CFS
Requests and limits actually use different algorithms to control CPU resources

Let’s start with requests
Use – CPU shares in docker run to allocate CPU resources in proportion
For example, two containers

docker run --name c1 -itd --rm -c 1024 progrium/stress --cpu 6
docker run --name c2 -itd --rm -c 2048 progrium/stress --cpu 6

–cpu-shares shares can only provide a certain proportion of competition, so when there is only one container, the container can still obtain all CPU resources

docker run --name c1 -itd --rm -c 1024 progrium/stress --cpu 6

On kubernetes
The agileek / cpuset test image is used for pressure test. There are six cores with 16g of memory on node slave2
When there is only one S1 with 100m requests and only one replica, it can still eat 100% of the CPU because there is no competition

    name: s1
    resources:
      requests:
        cpu: 100m

Unfortunately, this image does not seem to support a special number for multi-core CPUs. I have six cores on this node. Here, I need to adjust replica to 6 to eat up all six cores on this node

We join a competitor, clone S1 to S2, and adjust requests to 50m on S2
Finally, on the slave2 node
s1 （100m）* 6
s2 （50m） * 6
It can be seen that the CPU occupancy ratio of S1 and S2 is gradually approaching 4:2

Limits shows the CPU cfs_ period_ Us and CPU cfs_ quota_ Us attribute
cpu. cfs_ period_ Us: set the time period (in microseconds)（ μ s) ) must be the same as CFS_ quota_ Use with us.
cpu. cfs_ quota_ Us: set the maximum time (in microseconds) that can be used in the cycle（ μ s））。 The configuration here refers to the maximum usage of a single CPU by a task.
For example, set the limit to 50%
Set CPU cfs_ quota_ Us is set to 50000, relative to CPU cfs_ period_ 100000 US is 50%
set up
s1 requests 100m limits 100m
s2 requests 50m limits 50m
Will get

In this case, the total CPU utilization directly drops to 6%, and the kernel application curve almost drops to 0

Kubernetes Pod QoS

The basic concept and use of QoS will not be repeated here

https://kubernetes.io/zh/docs/tasks/configure-pod-container/quality-service-pod/

This is mainly to explain a problem

QoS acts on the pod, but the resources that determine QoS act on the container. Then, when there is only one container on a pod, this problem becomes very simple

apiVersion: v1
kind: Pod
metadata:
  name: qos-demo
  namespace: lizhe
spec:
  containers:
  - name: qos-demo-ctr
    image: nginx
    resources:
      limits:
        memory: "200Mi"
        cpu: "700m"
      requests:
        memory: "200Mi"
        cpu: "700m"

apiVersion: v1
kind: Pod
metadata:
  name: qos-demo
  namespace: lizhe
spec:
  containers:
  - name: qos-demo-ctr1
    image: busybox
    command:
    - sleep
    - infinity
    resources:
      limits:
        memory: "200Mi"
        cpu: "700m"
      requests:
        memory: "200Mi"
        cpu: "700m"
  - name: qos-demo-ctr2
    image: busybox
    command:
    - sleep
    - infinity
    resources:
      limits:
        memory: "300Mi"
        cpu: "800m"
      requests:
        memory: "300Mi"
        cpu: "800m"

If the requests and limits of container2 are inconsistent, the QoS should be burstable for container2 and guaranteed for container1

container1

requests 200,700 limits 200,700

container2

requests 200,700 limits 300,800

apiVersion: v1
kind: Pod
metadata:
  name: qos-demo
  namespace: lizhe
spec:
  containers:
  - name: qos-demo-ctr1
    image: busybox
    command:
    - sleep
    - infinity
    resources:
      limits:
        memory: "200Mi"
        cpu: "700m"
      requests:
        memory: "200Mi"
        cpu: "700m"
  - name: qos-demo-ctr2
    image: busybox
    command:
    - sleep
    - infinity
    resources:
      limits:
        memory: "300Mi"
        cpu: "800m"
      requests:
        memory: "200Mi"
        cpu: "700m"

conclusion
QoS really works on pod
The QoS is determined by the resources of the container

All container requests and limits are consistent == guaranteed
There are inconsistencies == burstable
No configuration == best effort

Introduction to kubernetes

The word kubernetes itself means helmsman and aviator in Greek.
It was opened source by Google in 2014

Now that you have seen this article, I believe you must understand how valuable it is to master and learn it.
There is a saying that software is eating the world and containers are eating software. This is why I have written a lot of content about docker at the same time.

Kubernetes is becoming an operating system for container and cloud environments.
Before 2017, kubernetes was in competition with mesosphere and docker swarm. It was not until 2017 that kubernetes gradually became the de facto standard of container arrangement.
At present, mainstream cloud manufacturers provide kubernetes services.