en

kube-scheduler

Scheduler operates in a way that multiple people are responsible for the same work and then activate a main work node based on voting.
The leader election mechanism can ensure high availability, but there is only one scheduler node working normally, and other nodes are blocked as candidate nodes.
After the leader node quits for some reason, other candidate nodes run for election through the election mechanism. The node that becomes the new leader will take over the work of the original node.
The election mechanism is implemented through distributed locks,

• The distributed lock depends on a key on etcd. The operations of the key are atomic operations. Taking the key as a distributed lock, it has two states – existence and nonexistence.
• When the key (distributed lock) does not exist: a node in multiple nodes successfully creates the key (obtains the lock) and writes the information of its own node. The node that obtains the lock is called the leader node. The leader node will regularly update (renew) the information of the key.
• When a key (distributed lock) exists: other nodes are blocked and acquire locks regularly. These nodes are called candidate nodes. The process of the candidate node obtaining the lock regularly is as follows: obtain the data of the key regularly and verify whether the leader lease in the data expires. If it does not expire, it cannot be preempted. If it has expired, update the key and write the information of its own node. If the update is successful, it will become the leader node.

As we all know, the main function of the scheduler is to find the most suitable node node in the cluster for the newly created pod (in fact, if the pod needs to be migrated), and schedule the pod to this node.

1. Select available nodes first
2. Reselect the optimal node
3. If pod is used Spec.nodename enforces the constraint to schedule the pod to the specified node. Bypassing the scheduler will not check the human resources

The scheduling core consists of two independent control loops

One is the list watch mechanism and cache mechanism on the informer path loop

One is the pre selection mechanism and binding callback mechanism on the scheduler path loop

First control loop informer path

The main purpose is to start a series of informers to monitor the changes of pod, node, Sservice and other API objects related to scheduling in etcd.

Example: when a pod to be scheduled (nodeName field is empty) is created, the scheduler will add the pod to be scheduled to the scheduling queue through the handler of pod informer.

By default, the kubernetes scheduling queue is a priority queue, and when some cluster information changes, the scheduler will perform special operations on the contents of the scheduling queue. It is mainly due to the consideration of scheduling priority and preemption.

Responsible for updating the scheduler cache. Kubernetes scheduling is the most fundamental principle for performance optimization. It caches the cluster information as much as possible to improve the execution efficiency of predict and priority scheduling algorithms.

In the second control loop,

the scheduler is responsible for the scheduling path of the main loop of pod scheduling

Main logic: constantly leave the pod from the scheduling queue and call the predictions algorithm for “filtering”. “Filter” to get a group of nodes, that is, a list of all hosts that can run this pod. The node information required by the predictions algorithm is directly obtained from the scheduler cache to ensure the execution efficiency of the algorithm.

The scheduler will call the priorities algorithm to score the nodes in the above list from 0 to 10. The node with the highest score is the result of scheduling.

After the execution of the scheduling algorithm, the scheduler needs to fill the value of the nodeName field of the Pod object into the Node name. = = = = > Bind phase.

The specific location is

/home/lizhe/k8s/kubernetes/pkg/scheduler/scheduler. go

Here, I remove the nodeName in the previous example and apply nginx again

It can be seen that without starting Kube scheduler, pod cannot be correctly assigned to node and will always be in pending state

start kube-scheduler

./kube-scheduler --kubeconfig=kubeconfig.yaml

Re create nginx pod and you will find that it is still pending

By checking the pod information, it is found that the node holds the unallocated taint by default

Next, I try to delete this taint

./kubectl taint nodes ubuntu node.kubernetes.io/not-ready:PreferNoSchedule-
./kubectl taint nodes ubuntu node.kubernetes.io/not-ready:NoSchedule-

After deleting taint, you can see that the scheduler does allocate node to pod

However, the pod status is still pending

We have seen that the scheduler successfully allocated node

However, kubelet does not seem to create the instance of pod correctly, and the status of pod is still pending. Why?

kube-apiserver、etcd、kubelet and docker

Here we will try to build a cluster with minimal startup using the previously compiled source code

To run a minimum level of kubernetes, there must be at least three basic components:

• Kubelet: the agent running on each node in the cluster, which is responsible for the core components of the container
• Kube apiserver: a component of the kubernetes control plane that provides the only access to resource operations
• Container runtime (docker)

We created a minik8s folder for the experiment

Copy kubectl and kubelet executable binaries into this folder, and then create a pods folder

Start kubelet to try

sudo ./kubelet --pod-manifest-path=pods --fail-swap-on=false 

Then create a HelloWorld pod

apiVersion: v1

kind: Pod

metadata:

  name: hello

spec:

  containers:

  - image: busybox

    name: hello

    command: ["echo", "hello world!"]

Kubelet process will automatically load this file and try to start the corresponding pod, but an error is reported here

This is because kubernetes’ pod will give priority to starting a {k8s. By default gcr. IO / pause: pause image of 3.2, which cannot be obtained for some reasons. We can re specify an accessible image with the — pod infra container image parameter:

sudo ./kubelet --pod-manifest-path=pods --fail-swap-on=false --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2

To start apiserver, start an etcd service first

apiVersion: v1

kind: Pod

metadata:

  name: etcd

  namespace: kube-system

spec:

  containers:

  - name: etcd

    command:

    - etcd

    - --data-dir=/var/lib/etcd

    image: registry.aliyuncs.com/google_containers/etcd:3.4.3-0

    volumeMounts:

    - mountPath: /var/lib/etcd

      name: etcd-data

  hostNetwork: true

  volumes:

  - hostPath:

      path: /var/lib/etcd

      type: DirectoryOrCreate

    name: etcd-data

Then start apiserver

apiVersion: v1

kind: Pod

metadata:

  name: kube-apiserver

  namespace: kube-system

spec:

  containers:

  - name: kube-apiserver

    command:

    - kube-apiserver

    - --etcd-servers=http://127.0.0.1:2379

    image: cnych/kube-apiserver:v1.18.5

  hostNetwork: true

Try the kubectl command line

The reason why the pod information is not obtained here is the lack of kubeconfig configuration

Create kubeconfig yaml

apiVersion: v1

kind: Config

clusters:

- cluster:

    server: http://127.0.0.1:8080

  name: mink8s

contexts:

- context:

    cluster: mink8s

  name: mink8s

current-context: mink8s

Restart the kubelet process and add the kubeconfig parameter

sudo ./kubelet --pod-manifest-path=pods --fail-swap-on=false --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2 --kubeconfig=kubeconfig.yaml

Next, try to deploy an nginx in this environment

create nginx yaml

apiVersion: v1

kind: Pod

metadata:

  name: nginx

spec:

  containers:

  - image: nginx

    name: nginx

Because there is no default service account, an error will be reported directly

Create a default service account

apiVersion: v1

kind: ServiceAccount

metadata:

  name: default

  namespace: default

Then try again and find that the token is missing

Modify the service account configuration and close the token

apiVersion: v1

kind: ServiceAccount

metadata:

  name: default

  namespace: default

automountServiceAccountToken: false

You can see here that neither nginx pod nor HelloWorld started normally

This is because there is no scheduler, and the scheduler is responsible for scheduling. Here, we directly use nodeName to fix the pod to the node and edit nginx yaml

apiVersion: v1

kind: Pod

metadata:

  name: nginx

spec:

  containers:

  - image: nginx

    name: nginx

  nodeName: ubuntu

Here is a point to note. If you apply directly, an error will be reported. You need to delete the original pod before creating it

Call curl

./kubectl get pods -owide
./kubectl logs curl

apiVersion: v1

kind: Pod

metadata:

  name: curl

spec:

  containers:

  - image: curlimages/curl

    name: curl

    command: ["curl", "172.17.0.3"]

  nodeName: ubuntu

If you don’t use docker, you can directly use binary files to start apiserver

sudo ./kube-apiserver –etcd-servers=http://127.0.0.1:2379 –service-cluster-ip-range=10.0.0.0/24

How to compile kubernetes source code

Kubernetes source code is located in

The code structure looks very complex, but actually there are traces to follow. It conforms to the code specification of golang language. For these two specifications, please refer to

https://github.com/golang-standards/project-layout

https://www.jianshu.com/p/4726b9ac5fb1

Cloc results are as follows

Then let’s try to build

Then you can get_ output

In addition, there is another way to quickly get the docker image of each component of kubernetes

make quick-release

The compilation results are almost the same, but this will generate images

kube bench

kube-bench is tool that checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark.

Installing using docker

docker run --rm -v `pwd`:/host aquasec/kube-bench install

call

./kube-bench

use kubectl in a pod

reference resources
https://kubernetes.io/zh/docs/tasks/run-application/access-api-from-pod/

In some cases, we would like to use kubectl to operate the cluster in a pod, for example

In Argo workflow, a task is required, and kubectl can be called to modify some resources

Method 1

The stupidest way is to mount kubeconfig yaml file. In this case, you use the user in config file, which will not be discussed here, because this way is too stupid

Method 2

Use kubectl image, and then assign RBAC permission to pod, so that pod can operate the cluster through the service account bound by RBAC

The image used is bitnami/kubectl
The entrypoint used is /bin/bash

Try to get cluster information through kubectl get nodes

Directly add a cluster level admin role to the default Sa of the namespace

kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --group=system:serviceaccounts:lizhe
clusterrolebinding.rbac.authorization.k8s.io/permissive-binding created

Then let’s try

kube shell

Install

$ pip install kube-shell

Here, I first copy my Kube config file to the default path, and then start the Kube shell

cp /etc/rancher/rke2/rke2.yaml ~/.kube/config
kube-shell

Kubernetes lens

Kubernetes endpoints

Using endpoints, you can easily use services (traefik, nodeport, loadbalancer, etc.) to connect external resources

The following is an example of traefik connecting PostgreSQL

endpoint.yaml

apiVersion: v1
kind: Endpoints
metadata:
  name: postgre-endpoint
  namespace: postgre
subsets:
  - addresses:
      - ip: 10.10.20.135
    ports:
      - port: 5432

create service

apiVersion: v1
kind: Service
metadata:
  name: postgre-endpoint
  namespace: postgre
spec:
  ports:
    - port: 5432
      protocol: TCP
      targetPort: 5432

IngressRoute

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: postgre-endpoint
  namespace: postgre
spec:
  entryPoints:
    - postgre
  routes:
  - match: HostSNI(`*`)
    services:
    - name: postgre-endpoint
      port: 5432

Cannot delete kubernetes namespace

add parameter

--grace-period=0 --force

if still cannot be deleted

echo $(kubectl config view --raw -oyaml | grep client-cert  |cut -d ' ' -f 6) |base64 -d > /tmp/client.pem
echo $(kubectl config view --raw -oyaml | grep client-key-data  |cut -d ' ' -f 6 ) |base64 -d > /tmp/client-key.pem
echo $(kubectl config view --raw -oyaml | grep certificate-authority-data  |cut -d ' ' -f 6  ) |base64 -d > /tmp/ca.pem

curl --cert /tmp/client.pem --key /tmp/client-key.pem --cacert /tmp/ca.pem -H "Content-Type: application/json" -X PUT --data-binary @/tmp/temp.json https://xxx.xxx.xxx.xxx:6443/api/v1/namespaces/<namespaces>/finalize

juju charmed kubernetes

download

https://raw.githubusercontent.com/charmed-kubernetes/bundle/master/overlays/calico-overlay.yaml

wget https://raw.githubusercontent.com/charmed-kubernetes/bundle/master/overlays/calico-overlay.yaml --no-check-certificate

The content of the file is like this

description: Charmed Kubernetes overlay to add Calico CNI.
applications:
  calico:
    annotations:
      gui-x: '480'
      gui-y: '750'
    charm: cs:~containers/calico
  flannel:
relations:
- - calico:etcd
  - etcd:db
- - calico:cni
  - kubernetes-master:cni
- - calico:cni
  - kubernetes-worker:cni

juju bootstrap

juju add-model k8s
juju deploy charmed-kubernetes

sudo snap install kubectl --classic
juju scp kubernetes-master/0:config ~/.kube/config