en

Kubernetes scheduling process

The core of kubernetes scheduler is two independent control loops

• Informer Path
• Scheduling Path

Informer is responsible for monitoring the changes of API objects related to scheduling such as pod, node and service in etcd.

Scheduling is the main cycle responsible for scheduling. Its main function is to continuously extract pod from the scheduling queue, and then score the node 0-10 according to the node information obtained from the scheduler cache. The node with the highest score is the result of this scheduling

After the scheduling algorithm is completed, the scheduler modifies the NodeName field value of the Pod object to the name of the above node, called bind.

Predictions before priorities

There are four default scheduling policies in kubernetes:

GeneralPredicates

The most basic scheduling strategy, such as calculating whether CPU and memory resources are enough

2. Volume related filtering rules

Check whether PV conflicts, etc

3. Host related rules

Check node stains, etc

4. Pod related filtering rules

Check pod affinity, etc

The above four strategies are used as filters to determine whether a node can run the pod to be scheduled

In order not to remotely access the API server in the critical scheduling path, kubernetes will only update the pod and node information in the scheduler cache in the bind phase of the default scheduler. This API object update method based on optimistic assumptions is called assume.

After Assume is completed, the scheduler will create a Goroutine to initiate an update Pod request to API Server asynchronously to complete the Bind operation. If the asynchronous request fails, everything will return to normal after the scheduler cache is synchronized.

After the end of the asynchronous bind and before the new pod is actually scheduled, kubelet Haihui on the node verifies again whether the pod can run on the node through an operation called admit, that is, execute the basic scheduling algorithm of general predictions (whether the resources are available and whether the end ports conflict) again as kubelet’s secondary confirmation.

The job of the optimization strategy is to score these nodes

The most commonly used scoring rule for preference strategy is leadrequestedpriorityThe longest used scoring rule for the preference strategy is LeastRequestedPriority

score = (cpu((capacity-sum(requested))10/capacity) + memory((capacity-sum(requested))10/capacity

Working with LeastRequestedPriority is also the BalancedResourceAllocation

score = 10 - variance(cpuFraction,memoryFraction,volumeFraction)*10

The definition of the fraction of each resource is the resource requested by the pod / the available resources on the node. The function of the variance algorithm is to calculate the “distance” between each two resource fractions, and finally select the node with the smallest resource fraction gap.

BalancedResourceAllocation selects the node with the most balanced resource allocation among all nodes, so as to avoid the situation that a large amount of CPU is allocated and a large amount of memory is left on a node

In addition, there are three preferred strategies: nodeaffinitypriority, tainttolerationpriority and interpodaffinitypriority. As a preferred strategy, the more fields a node satisfies the above rules, the higher its score will be

In the default preference policy, there is also a policy called imagelocalpriority. It’s in kubernetes v1 The scheduling rule provided in 12, that is, if the pod to be scheduled needs to use a large image and already exists on some nodes, the scores of these nodes will be relatively high

Of course, in order to avoid scheduling stack, the scheduler will optimize the score according to the distribution of images, that is, if the number of nodes distributed in large images is small, the weight of these nodes will be reduced, so as to hedge the risk of scheduling stack.

API server authentication mechanism

In the section of coredns, although my attempt to install coredns failed, we can still see how DNS components are installed

There was an error during installation

Failed with no API token found for service account “coredns”

We chose to disable serviceaccount to avoid this error simply and rudely

To: Kube_ ADMISSION_ CONTROL=”–admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota”
Change to: Kube_ ADMISSION_ CONTROL=”–admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota”

So in this section, I’m going to study serviceaccount

When you access apiserver from outside the cluster, you use the kubectl command line. At this time, service account is not required. However, when we access it from inside a pod, there is often no kubectl available. Therefore, if you want to access API server from inside a pod, you need service account

If you look at all the services in the current cluster, you can see that there must be a service called kubernetes. The type is clusterip. In fact, this service is API server

To test the above, I’ll create a Ubuntu pod and use it to try to connect to the API server

apiVersion: apps/v1

kind: Deployment

metadata:

  name: ub2

  labels:

    app: ub2

spec:

  replicas: 1

  selector:

    matchLabels:

      app: ub2

  template:

    metadata:

      labels:

        app: ub2

    spec:

      containers:

      - name: ub2

        image: libaibai/ub2:latest

        command: ["/bin/bash","-c","sleep 3600"]

        ports:

        - containerPort: 80

Then we use this pod to try to access apiserver

The cert of API server is stored in secret

This file will be mapped to / var / run / secrets / kubernetes. For each pod io/serviceaccount/ca.crt

However, you cannot directly use this cert to pass the verification. The 401 below is the same as the 401 above. It is the result of the client ignoring or trusting the server

But the server does not authorize the client

if the client does not trust the server

You need to be authorized by API server to use
/var/run/secrets/kubernetes. io/serviceaccount/token

curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ikxsc0ZlMElUY3FLanNONGRoYzMtTGJDWXZjZWE2WXVkR2pOajdtNXNHTTAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tZzZzdzYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjQ1NzY1NzdkLTNkNGYtNDUyMS1iNDdkLWY5MmJhOGUxNjg3MyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.SAdvtBm1v4-YzD7qTC4KOdt6PDY4vAyptGzQkGNP17BqAqDNi9-dbrfJdYEFgof7nyMZH_TCMhi1FXMZJwQ5zOKXeS_YKruJuvJ1iXLwW2MoVy-conJmkoMU-SRcsy8tjv5KkLEZa9adIRG_8jHGc4zSRvs-5YkApsIYYOMFcUbhIVjyMt_rXIeb900hj1e5AhcaSQ19gy2paQkcCbaFnxq7dsgSbpsy011QDe-FzaWv7yY8xI3PJj-JIRgrDxV4WbtalJ5V5IrNcFhHyB7hxawZKvexIJ_ndPewacA42vIUoemAYCQYXm3Bio_ynd-gTQJQzo0YBGxSxfrHnIaHsQ"  https://10.43.0.1:443 

The return status code becomes 403 because our cluster uses RBAC
Kubernetes divides the clients connected to API server into two types

Real people
Application in pod (service account)
Both normal users and service accounts can belong to one or more groups

Common groups:
All plug-ins that request authentication for the client will not be authenticated:
System: the authenticated group will be automatically assigned to a user who has successfully passed the authentication
System: serviceaccounts contains all serviceaccounts in the system
System: serviceaccounts: < namespace > group contains all serviceaccounts in a specific namespace

Let’s do an experiment
The first is the failed experiment
We create a Ubuntu instance under Lizhe namespace,

Then try to access the API server without permission

Sure enough, I got 403
The following is the correct experiment
Given permission

kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --group=system:serviceaccounts:lizhe

how to remove：

kubectl delete clusterrolebindings permissive-binding

call curl again

Serviceaccount, like pod, secret and configmap, is a resource. A resource must exist in a namespace

Each pod is associated with a serviceaccount, but multiple pods can use the same serviceaccount
Each pod will be assigned a single service account in this pod namespace
In the description file of the pod, you can assign a service account to a pod. If the service account is not specified explicitly, the pod will use the default service account of the namespace
Create a new namespace lizhe10

Create Ubuntu 10 under it

Experiment 1
When Ubuntu 10 uses the default service account (provided by lizhe10), it gets 403 and cannot access the API server

Experiment 2
Failed to assign service account – > under other namespace to pod
Pod Ubuntu 10 under namespace lizhe10
SA lizhesa under namespace Lizhe
Failed to assign lizhesa to Ubuntu 10

Experiment 3
Create an SA called lizhesa10 under the namespace lizhe10

Assign the newly created SA lizhesa10 to Ubuntu 10

At this point, if you try to use the SA lizhe10 to access the API server, you will get 403

然后我们为 lizhesa10 赋权限

语法是这样的，之前我们用的是 group 

 $ clusterrolebinding NAME --clusterrole=NAME [--user=username] [--group=groupname] [--serviceaccount=namespace:serviceaccountname] [--dry-run] 

This time we use service account

kubectl create clusterrolebinding permissive-binding-lizhesa10 --clusterrole=cluster-admin --serviceaccount=lizhe10:lizhesa10 

After it takes effect, you can get the correct response

Here are some digressions
If you use the describe command and look at the SA, you can see that it carries a token

kubectl describe sa lizhesa10 -n lizhe10 

This token is actually the one you get in the pod

RBAC authorization plug-in takes user role as the key factor to determine whether users can perform operations.

The principal (person, SA or group) is associated with one or more roles, each of which is allowed to perform specific actions on a specific resource

If a user has multiple roles, they can do any action allowed by the role.

RBAC does this by creating four specific kubernetes resources

Role and clusterrole

They specify what actions can be performed on the resource

Rolebinding and clusterrolebinding

They bind these roles to specific users, groups, or SAS

We used it before

kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --group=system:serviceaccounts:lizhe 

It means to create a binding to clusterrole with the type of permission binding, which is clusterrole binding. The bound object is a group whose scope is Lizhe namespace

RoleBinding and ClusterRoleBinding

RoleBinding

Rolebinding can bind a subject (user) in the same namespace to a role with specific permissions, then the subject has the permissions defined by the role.

ClusterRoleBinding

Clusterrolebinding binds specific subjects to clusterrole at the whole cluster level and all namespaces, and grants permissions.

Role and ClusterRole

Role

The role object can only be used to grant access to resources in a namespace.

ClusterRole

The clusterrole object can grant access rights to resources in the whole cluster, and can also grant access rights to the following resources:

Cluster wide resources (such as nodes)

Non resource type endpoint (for example “/ healthz”)

Resources across the range of all namespaces (for example, pod, you need to run the command kubectl get pods — all namespaces to query all pods in the cluster)

When scheduler is not required

The main task of scheduler is to schedule pod
For the specific process, please refer to the kubernetes pod creation process
In several cases, kubernetes can bypass the scheduler and create a pod directly

• NodeName is specified when creating a pod
• Pod created by daemon set daemonset (daemonset specifies nodeName, so it is the same as the first one)
• Static pod

Further explanation

The pod created by the daemon set can run before the scheduler starts

The pod created by the daemon set can not comply with the unscheduled of the node

The pod created by the daemon set has a higher priority, the demodulator will avoid expelling it, and the autoscaler will manage them separately

Static pod is similar to daemon set. Kubelet can not only communicate with kubernetes API server and obtain pod description file, but also obtain resource definition from local directory. The pod defined in this way can only be managed by kubelet and can only run on one node.

API services do not monitor these pods, and they do not have controllers or perform health checks on them. Kubelet monitors these pods and restarts them when they crash.

Kubelet also periodically scans the configuration directory to find changes to pod definitions, and adds or removes pods accordingly

Kubernetes pod creation process

We know that kubernetes consists of many different components

apiVersion: v1
kind: Pod
metadata:
  name: demo
  namespace: lizhe
spec:
  containers:
  - name: demo-ctr
    image: nginx

• Kubectl sends a request to create a pod to the API server
• API server generates pod creation information
• API server stores creation information to etcd
• The scheduler queries the pod Spec.node = = null confirm that this pod is not allocated
• The scheduler allocates node pod Spec.node = node1, save to etcd
• Kubelet queries etcd and operates on the node

In the second case, create a pod through deploy (replicaset)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx

• Kubectl sends a request to create a dep to the API server
• API server generates dep creation information
• API server stores creation information to etcd
• Replicaset in controller manager the controller obtains replicaset information from etcd
• The controller manager sends a request to create three pods to the API server
• API server generates pod creation information
• API server stores creation information to etcd
• The scheduler queries the pod Spec.node = = null confirm that this pod is not allocated
• The scheduler allocates node pod Spec.node = node1, save to etcd
• Kubelet queries etcd and operates on the node

In fact, only the above steps are extra

initContainers

By initializing the container, you can also easily do something similar to poststart

However, it should be clearly distinguished that the target of initializing the container is pod itself, while the target of poststart is container

In most cases, the two methods can be interchanged

apiVersion: v1
kind: Pod
metadata:
  name: init-demo
  labels:
    app: init
spec:
  initContainers:
    - name: init
      image: busybox
      command:
      - wget
      - "-O"
      - "/tmp/index.html"
      - https://studyk8s.com
      volumeMounts:
        - name: initdir
          mountPath: "/tmp"

  containers:
    - name: nginx
      image: nginx
      ports:
      - containerPort: 80
      volumeMounts:
        - name: initdir
          mountPath: /usr/share/nginx/html
  volumes:
    - name: initdir
      emptyDir: {}

The init container is very similar to an ordinary container, except for the following two points:

• Init container always runs to completion (for example, execute WGet, curl, Ping and other commands)
• If there are multiple init containers, the operation must be ended before the next startup

If the init container in the pod fails to start, kubernetes will continue to restart the pod until the init container succeeds. If the restart policy value corresponding to the pod is never (no restart as long as you exit), it will not restart.

Prestop hook

The semantics of Prestop are also very simple, and call before stopping.

But before what stops? Before the container stops? Before pod destruction?

The exact time of Prestop is called before Kubernetes sends top note to SIGTERM.
It should be noted that the SIGTERM signal is sent to the pod and the prestop hook is the container

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        lifecycle:
          preStop:
            exec:
              command:
              - sh
              - -c
              - sleep 30
      - name: u1
        command:
        - sleep
        - infinity
        image: ubuntu
        lifecycle:
          preStop:
            exec:
              command:
              - sh
              - -c
              - date "+%Y-%m-%d %H:%M:%S" >> /datetmp.txt && sleep 5
      - name: u2
        command:
        - sleep
        - infinity
        image: ubuntu
        lifecycle:
          preStop:
            exec:
              command:
              - sh
              - -c
              - date "+%Y-%m-%d %H:%M:%S" >> /datetmp.txt && sleep 10

In the script, nginx is blocked for 30 seconds, and finally it should be killed
There is a difference of 5 seconds between the blocking time of the two Ubuntu containers,
The observed result is that they all end together in the last 30 seconds, while the prestop script starts asynchronously at the same time

Poststart hook

As the name suggests, this hook allows you to execute some logic after the container is started
There are three key points to explain

• Poststart and the main thread of the container execute asynchronously. They are not started at the same time after the main thread is started
• The container will be in the state of postpending before it runs
• The semantics of the hook contains the meaning of “execute at least once”. Therefore, repeated execution of the hook will affect the system, which needs careful consideration

For the first question

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        lifecycle:
          postStart:
            exec:
              command:
              - sh
              - -c
              - date "+%Y-%m-%d %H:%M:%S" > /datetmp.txt && sleep 30
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 30
        readinessProbe:
          httpGet:
            path: /
            port: 80
            scheme: HTTP
          initialDelaySeconds: 30

You can see that the start time of poststart is the same as that of the container
Because my current system time is UTC + 8, it is 9:23 in datetmp and 17:23 in nginx console log, but they do start at the same time instead of waiting until after the main thread

For the second question

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        lifecycle:
          postStart:
            exec:
              command:
              - sh
              - -c
              - echo "postStart postStart postStart" && sleep infinity
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 30
        readinessProbe:
          httpGet:
            path: /
            port: 80
            scheme: HTTP
          initialDelaySeconds: 30

The difference between the waiting state of the poststart hook and the ready probe is
It does not acknowledge that the container has started
Note that the start status is false, while the start status of the ready probe is true

When a single pod is installed in three containers, the poststart hooks of the three containers and the three mainlines and six scripts of the container will be started at the same time

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        lifecycle:
          postStart:
            exec:
              command:
              - sh
              - -c
              - echo "postStart postStart postStart" && sleep 30
      - name: u1
        command:
        - sleep
        - infinity
        image: ubuntu
        lifecycle:
          postStart:
            exec:
              command:
              - sh
              - -c
              - date "+%Y-%m-%d %H:%M:%S" >> /datetmp.txt
      - name: u2
        command:
        - sleep
        - infinity
        image: ubuntu
        lifecycle:
          postStart:
            exec:
              command:
              - sh
              - -c
              - date "+%Y-%m-%d %H:%M:%S" >> /datetmp.txt

Here, the Ubuntu container is blocked for 30 seconds, but the nginx is blocked forever. As a result, the three containers on this pod cannot be started successfully at the same time

Therefore, although the poststart hook acts on the container, it will affect other containers on the pod
Change nginx blocking to 30 seconds and Ubuntu to 5 seconds
You can see that after 5 seconds and about 10 seconds, the three containers are still in the starting state

You can also see that on the two Ubuntu, the poststart hook is asynchronous and started at the same time

Health probe

Kubernetes comes with two types of probes
livenessProbe and readinessProbe

The livenessProbe checks the health of the container in the following three ways

• Through the HTTP get request, the expected response code is 200-399
• TCP socket connection succeeded
• Exec calls the command with exit code 0

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 30

Containers that fail to pass the livenessProbe check will be restarted and will only be restarted
The check methods of the ready probe are the same as HTTP, command and TCP, but the readinessProbe will not restart the container, but prevent the container from receiving requests

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 30
        readinessProbe:
          httpGet:
            path: /
            port: 801
            scheme: HTTP
          initialDelaySeconds: 30

Limits – may cause OOM

At first glance, this is nonsense. You limit the maximum memory.
Of course, you should oom when you exceed it

In fact, the maximum memory used by many common applications cannot be simply limited by limits

In the docker container, the container cannot perceive that it is in a restricted and resource isolated environment.
It will think that it can use all host resources like normal non-container applications
In Linux Environment

Even if -c allocates 1000 microcores and 200m memory, the container still thinks it is using 6 cores and 8GB memory of the host

docker run -i -t --name ubuntu -c 1000 -m 200M ubuntu /bin/bash

This situation will cause some trouble for traditional programs, such as Java
Memory aspect
If – XMS is not set, Java will use (it thinks) 1 / 64 of the physical memory as the minimum heap capacity by default.
If – Xmx is not set, Java will use (it thinks) 1 / 4 of the physical memory as the maximum heap capacity by default.

 initial heap size
 Larger of 1/64th of the machine's physical memory on the machine or some reasonable minimum. Before Java SE 5.0, the default initial heap size was a reasonable minimum, which varies by platform. You can override this default using the -Xms command-line option.
 maximum heap size
 Smaller of 1/4th of the physical memory or 1GB. Before Java SE 5.0, the default maximum heap size was 64MB. You can override this default using the -Xmx command-line option.
 Note: The boundaries and fractions given for the heap size are correct for Java SE 5.0. They are likely to be different in subsequent releases as computers get more powerful.

Because the wrong resource information is obtained, the application in the container may allocate more resources than it can use, triggering an oom exception
So, can this problem be avoided by simply setting – Xmx to 200m? Not at all
The container – M parameter will limit the maximum memory usage of the entire container, but – XMS and – Xmx only limit the size of the heap, which will actually be used by a complete Java program

• The maximum heap memory of the Java program is the value set by – Xmx
• Garbage collection memory used in garbage collection
• Memory used by JIT optimization
• Memory used by off-heap of Java program
• Memory used by Metaspace of Java program
• The memory occupied by JNI code
• The memory is occupied when the JVM starts.

This does not include some resources that the Ubuntu container itself may consume (although it is negligible), so setting – Xmx to 80% to 90% of – M may be a safe choice
The specific value needs to be tested according to the actual project
In terms of CPU, because CPU can be oversold, there will be no oom,
However, the program in the container thinks it can use all six cores of the host. Is there really no problem?

docker run --name cputest -it --rm -c 512 progrium/stress --cpu 1

Since – C only indicates the CPU Competitive Allocation of the container, with a maximum of 512 micro cores, it still uses 100% of the CPU here. – the CPU forcibly locks it on one core, so it uses one core of all cores

docker run --name cputest -it --rm -c 512 progrium/stress --cpu 6

All host CPUs are used this time
In fact, the – C parameter does not affect the program to use of enough CPU resources. Even the – C 512 will not affect the scheduling of threads on multiple cores (not limited to 0.5 of one kernel),
Docker – C only restricts competition
Requests on Kubernetes limit contention and limits limit the CPU fragment time obtained by the CPU, so they will not affect CPU scheduling
There will be no case where 6 cores and 120 threads are scheduled too frequently

Reserve resources for kubelet

During the operation of kubernetes cluster, kubelet needs to regularly send heartbeat to API server and report the operation status

In fact, there are great potential safety hazards here

Because kubelet and pods running on the node need to share CPU and ram resources, kubelet may not get enough resources to run itself in extreme cases

In other words, pods is likely to consume all the physical resources of the node, resulting in the node’s heartbeat stopping or untimely reporting. The API server may mistakenly believe that the node is offline
In an idealistic state, all pods deployed to this node should not consume 100% of physical resources
for example

100m CPU * 9, so in theory, 100m will remain
However, due to the extremely complex cluster environment, there is no guarantee that the scheduler will reserve resources for each node, and there is no guarantee that every authorized operator will notice this problem
Therefore, there is often resource overload, resulting in kubelet’s heartbeat stopping
The solution is very simple and crude

Deploy a “placeholder” application and allocate its requests CPU resources to 100m
Because there is no specific application for this placeholder application (such as nginx), it will not really consume CPU

Here, requests are easy to control
1000m single core can support
Normal application of 100m * 9, space occupying nginx of 100m * 1,
However, requests only affect container scheduling and CPU competition, and can not guarantee that CPU resources will not be exhausted
Set limits to
100m * 9 application, 100m * 1 space occupying nginx, this time is good, but limits can be oversold, so there is no guarantee that all administrators will set aside that 100m for you
Conclusion,
It’s best to use only limits, which can ensure that requests are consistent with limits by default
Because requests cannot be oversold, when nginx occupies 100m, the nine normal applications that only write limits cannot exceed 900mhttps://googleads.g.doubleclick.net/pagead/ads?client=ca-pub-7712570106118575&output=html&h=280&adk=2825196814&adf=4093481066&pi=t.aa~a.2603011395~i.46~rp.1&w=706&fwrn=4&fwrnh=100&lmt=1650205817&num_ads=1&rafmt=1&armr=3&sem=mc&pwprc=1923896069&psa=1&ad_type=text_image&format=706×280&url=https%3A%2F%2Fstudyk8s.com%2F2022%2F04%2F17%2F3766%2F&fwr=0&pra=3&rh=177&rw=706&rpe=1&resp_fmts=3&wgl=1&fa=27&uach=WyJXaW5kb3dzIiwiMTAuMC4wIiwieDg2IiwiIiwiMTAwLjAuNDg5Ni4xMjciLFtdLG51bGwsbnVsbCwiNjQiLFtbIiBOb3QgQTtCcmFuZCIsIjk5LjAuMC4wIl0sWyJDaHJvbWl1bSIsIjEwMC4wLjQ4OTYuMTI3Il0sWyJHb29nbGUgQ2hyb21lIiwiMTAwLjAuNDg5Ni4xMjciXV0sZmFsc2Vd&tt_state=W3siaXNzdWVyT3JpZ2luIjoiaHR0cHM6Ly9wYWdlYWQyLmdvb2dsZXN5bmRpY2F0aW9uLmNvbSIsInN0YXRlIjoyMCwiaGFzUmVkZW1wdGlvblJlY29yZCI6dHJ1ZX1d&dt=1650205817490&bpp=1&bdt=365&idt=1&shv=r20220413&mjsv=m202204110101&ptt=9&saldr=aa&abxe=1&cookie=ID%3De1e9e75d484277dc-22e2d1cebbd10034%3AT%3D1649940877%3ART%3D1649940877%3AS%3DALNI_MYLFH7MHMVP9tSJTFU30yFvzIVVCA&prev_fmts=0x0%2C294x245&nras=2&correlator=3409815582494&frm=20&pv=1&ga_vid=510179359.1616343974&ga_sid=1650205817&ga_hid=211755682&ga_fc=1&u_tz=480&u_his=9&u_h=1280&u_w=3072&u_ah=1240&u_aw=3072&u_cd=30&u_sd=1.25&dmc=8&adx=200&ady=1246&biw=1518&bih=1160&scr_x=0&scr_y=0&eid=44759875%2C44759926%2C44759837%2C44761793%2C31067083%2C31061828%2C21067496&oid=2&pvsid=2181360227943243&pem=820&tmod=767505671&uas=0&nvt=1&ref=https%3A%2F%2Fstudyk8s.com%2Fwp-admin%2Fpost.php%3Fpost%3D3766%26action%3Dedit&eae=0&fc=1408&brdim=1529%2C0%2C1529%2C0%2C3072%2C0%2C1550%2C1247%2C1535%2C1160&vis=1&rsz=%7C%7Cs%7C&abl=NS&fu=128&bc=31&ifi=3&uci=a!3&btvi=1&fsb=1&xpc=8KLUnNbKwo&p=https%3A//studyk8s.com&dtd=30
The above is a folk prescription to deal with this problem. Does the kubernetes native application deal with this problem
The answer is yes, but trouble

• –Kube reserved is used to configure the amount of resources reserved for Kube components (kubelet, Kube proxy, dockerd, etc.), such as — Kube reserved = CPU = 1000m, memory = 8gi, ephemeral storage = 16gi.
• –Kube reserved CGroup. If you set — Kube reserved, you must set the corresponding CGroup, and the CGroup directory must be created in advance, otherwise kubelet will not be created automatically, resulting in kubelet startup failure. For example, set it to Kube reserved CGroup = / kubelet service 。 If this item is not set, the above — Kube reserved will not take effect.
• –System reserved is used to configure the amount of resources reserved for the system process, such as — system reserved = CPU = 500m, memory = 4Gi, ephemeral storage = 4Gi.
• –System reserved CGroup. If — system reserved is set, the corresponding CGroup must be set, and the CGroup directory must be created in advance, otherwise kubelet will not be created automatically, resulting in kubelet startup failure. For example, set it to system reserved CGroup = / system slice。 If this item is not set, the above — system reserved will not take effect.

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/cpuset/system.slice/kubelet.service /sys/fs/cgroup/hugetlb/system.slice/kubelet.service
ExecStart=/opt/kubernetes/bin/kubelet \
  --eviction-hard=memory.available<1024Mi,nodefs.available<10%,nodefs.inodesFree<5% \
  --system-reserved=cpu=0.5,memory=1G \
  --kube-reserved=cpu=0.5,memory=1G \
  --kube-reserved-cgroup=/system.slice/kubelet.service \
  --system-reserved-cgroup=/system.slice \
  --enforce-node-allocatable=pods,kube-reserved,system-reserved \
  --address=192.168.0.101 \
  --hostname-override=192.168.0.101 \
  --cgroup-driver=cgroupfs \
  --pod-infra-container-image=hub.breezey.top/kubernetes/pause-amd64:3.0 \
  --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
  --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
  --cert-dir=/opt/kubernetes/ssl \
  --cluster-dns=192.168.0.200 \
  --cluster-domain=k8s.breezey.top. \
  --hairpin-mode=promiscuous-bridge \
  --allow-privileged=true \
  --fail-swap-on=false \
  --serialize-image-pulls=false \
  --max-pods=60 \
  --logtostderr=true \
  --v=2 
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

And unfortunately, in the kubernetes cluster installed by RKE
Not in this way

The default is no kubelet.service

Need to add manually

Then specify through parameters

RKE kubernetes installed by default
This is the CGroup used by the kubelet process
6:cpu,cpuacct:/docker/4b28805380958bf3efb5c0d0a9c93ba42fab4d36e1be9036f3b9c4b01ef3bd62

it is below