en

Kubernetes node taints

kubectl taint nodes 192.168.204.129 key1=value1:NoExecute
node/192.168.204.129 tainted

If you do not specify a key, you can use

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-hello-deployment3
  namespace: lizhe
  labels:
    app: nginx-hello3
spec:
  replicas: 6
  selector:
    matchLabels:
      app: nginx-hello3
  template:
    metadata:
      labels:
        app: nginx-hello3
    spec:
      tolerations:
      - effect: NoExecute
        operator: Exists
      containers:
      - name: nginx-hello3
        image: nginx
        ports:
        - containerPort: 80

All keys will be ignored

You can see nginx3 running on 129

If a specific key needs to be specified, for example, key = key2 is specified here. In fact, this key2 does not exist, so there is no tolerance

    spec:
      tolerations:
      - effect: NoExecute
        operator: Exists
        key: key2

You can also add value, but you can’t use the operator

The Deployment "nginx-hello-deployment3" is invalid: spec.template.spec.tolerations[0].operator: Invalid value: core.Toleration{Key:"key1", Operator:"Exists", Value:"value2", Effect:"NoExecute", TolerationSeconds:(*int64)(nil)}: value must be empty when `operator` is 'Exists'

    spec:
      tolerations:
      - effect: NoExecute
        key: key1
        value: value2

If you want to tolerate it

    spec:
      tolerations:
      - effect: NoExecute
        key: key1
        value: value1

Kubernetes pod affinity

Similar to node affinity, pod affinity is mainly used to restrict the work of the scheduler. As the name suggests, pod affinity is scheduled with reference to the operation of pod

Instead of controlling scheduling according to nodes like node affinity

In kubernetes node affinity, we created the deploy of nginx

You can see that nginx is all running on 129

Here we try to make nginx2 follow nginx scheduling on the same node

use

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-hello-deployment2
  namespace: lizhe
  labels:
    app: nginx-hello2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-hello2
  template:
    metadata:
      labels:
        app: nginx-hello2
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - nginx-hello
            topologyKey: numberCores
      containers:
      - name: nginx-hello2
        image: nginx
        ports:
        - containerPort: 80

It is found that nginx2 is not only scheduled on 129, but also scheduled on 139

We have to talk about topologykey, because our three nodes specify numbercores = 6

in other words,

On the node running pod app=nginx-hello with numbercores and a value of 6

As long as it is a node with numbercores of 6, nginx2 can be run

We change numbercores to 7 on 129, so that only numbercores of 129 is 7 and other nodes are 6

In this way, nginx2 can only be scheduled on 129 in order to follow nginx 7

Because the affinity configuration cannot take effect again on the created pod (because it has been scheduled)

So here we delete the original dep and recreate it

This time, you can see that because only 129 has 7, nginx is all scheduled on 129

If you want pod a to be scheduled only on the node that has run pod B, you just need to

Topologykey is written as kubernetes io/hostname

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-hello-deployment2
  namespace: lizhe
  labels:
    app: nginx-hello2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-hello2
  template:
    metadata:
      labels:
        app: nginx-hello2
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - nginx-hello
            topologyKey: kubernetes.io/hostname
      containers:
      - name: nginx-hello2
        image: nginx
        ports:
        - containerPort: 80

Kubernetes node affinity

Node affinity is a generalization of the node selector method, which allows the necessary rules and preference rules to be specified

The following example is an nginx, which is required by affinity rules

Only nodes with more than 3 cores (excluding 3) are considered

Try to select nodes that are not master nodes

There is also a weight with a value of 1. If this node matches the name and is not master, it will get a weight score of 1

Finally, the scheduler will select a node with the highest weight score, and the weight is a list

Look at the following example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-hello-deployment
  namespace: lizhe
  labels:
    app: nginx-hello
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-hello
  template:
    metadata:
      labels:
        app: nginx-hello
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: numberCores
                operator: Gt
                values: ["3"]
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchFields:
              - key: metadata.name
                operator: NotIn
                values: ["master"]
      containers:
      - name: nginx-hello
        image: nginx
        ports:
        - containerPort: 80

In this example, numbercores is used. Please don’t mistakenly think that this is a native field. In fact, this is a label on node, which we need to create manually

If it is not created, the scheduling will not succeed

add label

now it can be scheduled

Quick installation of NFS

Sometimes we will use kubernetes in the local environment. At this time, it is unlikely to want some convenient storage classes similar to those on AWS and other public clouds. At this time, we will generally choose to use Linux NFS to replace these storage volumes of shared clouds

If you don’t need a storage class, you can directly use docker run, but here, because I’m lazy, I hope to automatically get a storage class after NFS is created. In fact, the most convenient is the helm provided by rancher. Note here

NFS client provisioner and NFS server provisioner have different driving modes and the same functions

The external NFS driver of k8s can be divided into two types according to its working mode (as NFS server or NFS client):

NFS client mounts the remote NFS server to the local directory through the built-in NFS driver of k8s; Then use itself as a storage provider and associate it with storage class. When the user creates a corresponding PVC to apply for PV, the provider compares the requirements of PVC with its own attributes. Once satisfied, it creates a subdirectory to which the PV belongs in the locally mounted NFS directory to provide dynamic storage services for pod

NFS server is different from NFS client. This driver does not use k8s NFS driver to mount remote NFS for local redistribution, but directly maps local files into the container, and then uses Ganesha NFSD to provide NFS services; Each time you create a PV, you can directly create the corresponding folder in the local NFS root directory and export the subdirectory. Dynamically providing kubernetes back-end storage volumes with NFS

Unfortunately, I use NFS provisioner here https://github.com/kubernetes-retired/external-storage

There’s nothing to say about clicking directly to install. Note that the default storage path on the node is /srv. As for which node to store, it’s random here, or you can specify it through node name

As you can see, my workload is assigned to 139,

The most convenient point of this method is that you can get the storage class directly

Then we try to install an nginx to use the NFS provider

Then I will create a test file on the / test path using the shell on the pod of nginx

You can see that this file can be found on NFS node

Kubernetes use private images

Private images cannot be used directly by default

The general practice is to create a secret of docker registry type first

kubectl create secret docker-registry regsecret --docker-username=dockername --docker-password=password --docker-email=helloworld@lizhe.name

Modify in deployment

    spec:
      imagePullSecrets:
      - name: regsecret
      containers:
      - image: lizhepac/privateimage
        imagePullPolicy: Always
        name: private
        resources: {}

kubernetes-downward-api

In the kubernetes environment, because the container is dynamically assigned to the node, that is, the specific information of the running environment cannot be known before the container is started

How to get the pod IP of the current container in the program? Or is it the memory limit of the current container?

Kubernetes uses the downward API to inject metadata about pod into containers and clusters through environment variables and files.

apiVersion: v1
kind: Pod
metadata:
  name: getpodip
spec:
  containers:
  - image: nginx
    name: getpodip
    env: 
    - name: POD_IP
      valueFrom:
        fieldRef: 
          fieldPath: status.podIP
    - name: MEM_LIMITS
      valueFrom:
        resourceFieldRef: 
          containerName: getpodip
          resource: limits.memory

However, since the environment variables cannot be changed after the container is started, you can also mount the information to the disk volume, so that after each modification, the program can update the information by re reading the contents of the file

apiVersion: v1
kind: Pod
metadata:
  name: getpodip
spec:
  containers:
  - image: nginx
    name: getpodip
    volumeMounts:
      - name: pod-info
        mountPath: /pod-info
        readOnly: true
      - name: pod-resource
        mountPath: /pod-resource
        readOnly: true
  volumes:
    - name: pod-info
      downwardAPI:
        items:
          - path: POD_IP
            fieldRef: 
              fieldPath: metadata.name
    - name: pod-resource
      downwardAPI:
        items:
          - path: "cpu_limit"
            resourceFieldRef:
              containerName: getpodip
              resource: limits.cpu
              divisor: 1m
          - path: "cpu_request"
            resourceFieldRef:
              containerName: getpodip
              resource: requests.cpu
              divisor: 1m
          - path: "mem_limit"
            resourceFieldRef:
              containerName: getpodip
              resource: limits.memory
              divisor: 1Mi
          - path: "mem_request"
            resourceFieldRef:
              containerName: getpodip
              resource: requests.memory
              divisor: 1Mi

But I don’t know why here is metadata Name cannot be replaced with status podIP

The Pod "getpodip" is invalid: 
* spec.volumes[0].downwardAPI.fieldRef.fieldPath: Unsupported value: "status.podIP": supported values: "metadata.annotations", "metadata.labels", "metadata.name", "metadata.namespace", "metadata.uid"
* spec.containers[0].volumeMounts[0].name: Not found: "pod-info"

• Env vars using valueFrom.fieldRef:
  •  spec.nodeName – the node’s name
  •  status.hostIP – the node’s IP
  •  metadata.name – the pod’s name
  •  metadata.namespace – the pod’s namespace
  •  status.podIP – the pod’s IP address
  •  spec.serviceAccountName – the pod’s service account name
  •  metadata.uid – the pod’s UID
  •  metadata.labels[”] – the value of the pod’s label (for example, metadata.labels[‘mylabel’])
  •  metadata.annotations[”] – the value of the pod’s annotation (for example, metadata.annotations[‘myannotation’])
• DownwardAPI volumes via items of fieldRef:
  •  spec.nodeName – the node’s name
  •  status.hostIP – the node’s IP
  •  metadata.name – the pod’s name
  •  metadata.namespace – the pod’s namespace
  •  status.podIP – the pod’s IP address
  •  spec.serviceAccountName – the pod’s service account name
  •  metadata.uid – the pod’s UID
  •  metadata.labels- the value of the pod’s label (for example, metadata.labels[‘mylabel’])
  •  metadata.annotations – the value of the pod’s annotation (for example, metadata.annotations[‘myannotation’])

Kubernetes secret is it safe

If you want to save your password in kubernetes, you must think of using secret

  docker-registry Create a secret for use with a Docker registry
  generic         Create a secret from a local file, directory or literal value
  tls             Create a TLS secret

There are three types of secret. Let’s create a generic secret to save a password

kubectl创建机密的通用secretname——from literal=“mykey=myvalue”

You can simply view this secret with the following command

kubectl get secret secretname -o jsonpath='{.data}'

Secret saves Base64 encoded data and provides it to pod as environment variable or mount volume after decoding

Therefore, some people often mistakenly believe that secret is secure. However, Base64 coding is not an encryption method. From the perspective of security, it is no different from plain text.

echo 'bXl2YWx1ZQ==' | base64 --decode

In fact, the security of secret lies in

• Only the node running the pod that needs to access secret can have secret
• On the node, secret is stored in TMPFS memory, never written to physical memory, and will be deleted along with pod

tmpfs has the following advantages:

1。 The size of the dynamic file system, / dev / SHM / one thing to pay attention to is the capacity. Under Linux, the maximum size is half of the memory by default, which can be seen by using the DF – H command. But it will not really occupy this memory. If there is no file under / dev / SHM /, the memory it occupies is actually 0 bytes; If it has a maximum of 1G and 100m files in it, the remaining 900m can still be used by other applications, but the 100m memory it occupies will never be recycled and re-divided by the system

2。 Another major benefit of tmpfs is its lightning speed. Because a typical tmpfs file system will completely reside in RAM, reading and writing can be almost instantaneous.

3。 TMPFS data will not be retained after restart because virtual memory is volatile in nature. Therefore, it is necessary to do some scripts to do operations such as loading and binding.

Here I create an Nginx deployment and mount the secret I just created

This pod is assigned to node 2

The size of secret cannot exceed 1MB. It cannot be used to store too large data, nor is it suitable for non configured data.

Because of Base64 encoding, the stored data can only be about 700kb

Crashloopbackoff

Crashloopbackoff status indicates that kubelet has not given up, which means that kubelet will increase the interval between next restart after crash

The interval will be 10 seconds, 20 seconds, 40 seconds, 80 seconds and 160 seconds, and finally converge to 300 seconds

Once the interval reaches 300 seconds, kubelet will restart the container indefinitely at a fixed interval of 300 seconds

node-capacity

View the capacity of the current node

Capacity:
  cpu:                4
  ephemeral-storage:  50824704Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             8123340Ki
  pods:               110

If you want to add a new capacity in the status field above, you must use the patch API

Start proxy

kubectl proxy

Add new capacity

curl --header "Content-Type: application/json-patch+json" \
--request PATCH \
--data '[{"op": "add", "path": "/status/capacity/studyk8s.com~1lizhe", "value": "4"}]' \
http://localhost:8001/api/v1/nodes/k1/status

curl --header "Content-Type: application/json-patch+json" \
--request PATCH \
--data '[{"op": "remove", "path": "/status/capacity/studyk8s.com~1lizhe"}]' \
http://localhost:8001/api/v1/nodes/k1/status

The above code will generate studyk8s com/lizhe: 4

Then it can be used in Dep

        resources:
          limits:
            studyk8s.com/lizhe: 4

So studyk8s COM / Lizhe has become a resource

Because only node1 has such resources and has been exhausted, it is impossible to deploy multiple nodes

Kubernetes – priority and preemption mechanism of default scheduler

The priority and preemption mechanism solves the problem of what to do after pod scheduling fails

By default, when a pod fails to be scheduled, it will be shelved first. The scheduler will not reschedule the pod until the pod is updated or the cluster state changes.

But sometimes we want a pod with higher priority to crowd out a pod with lower priority

First, you need to submit a priorityclass definition in kubernetes

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: high-priority-10000
value: 10000
globalDefault: false
description: "my desc 10000"

If priorityclass is not identified, its priority is 0

If you set globaldefault: false, the value of this PC will become the default value of the system

Our cluster state is 3 nodes, each with 4 cores and 8GB of memory

I deployed three nginx tags, each requiring three kernels

    resources:
      limits:
        cpu: 3
        memory: 2Gi
      requests:
        cpu: 3
        memory: 2Gi

In this way, we will eat all the CPU resources

Then deploy an nginx to compete

With the same requests and the same limits, the scheduler cannot complete the scheduling without using the priority and preemption mechanism

After adding priority

Finally, nginx1 squeezed out all nginx0