Kubernetes

Control signal

Before we start discussing how kubernetes uses control signals to control containers,

Maybe you need to review the start and stop of the container in the control signal part of docker

When kubernetes decides to close a container,

Whether the pod needs to be closed or the container is restarted due to the failure of the survival probe,

Kubernetes sends a SIGTERM signal to the container.

The container is obliged to respond to the SIGTERM signal as soon as possible and terminate it quickly, such as closing the open database connection, cleaning up temporary files, etc

Kubernetes will provide a default grace period of 30 seconds. If the container does not close after 30 seconds, it will send a sigkill signal to kill the container

The default 30 seconds can be modified through the following properties of pod

.spec.terminationGracePeriodSeconds 

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
spec:
  containers:
  - name: nginx
    image: nginx
  terminationGracePeriodSeconds: 30

Note that this attribute is used on POD, not container

Node Allocatable

Capacity:
cpu: 4
ephemeral-storage: 50824704Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 8123340Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 49442272013
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 8123340Ki
pods: 110

• Kubelet Node Allocatable 用来为 Kube 组件和 System 进程预留资源，从而保证当节点出现满负荷时也能保证 Kube 和 System 进程有足够的资源。
• 目前支持 cpu, memory, ephemeral-storage 三种资源预留。
• Node Capacity 是节点的所有硬件资源，kube-reserved 是给 kube 组件预留的资源，system-reserved 是给系统进程预留的资源，eviction-threshold 是 kubelet 驱逐的阈值设定，allocatable 才是真正调度器调度 Pod 时的参考值（保证节点上所有 Pods 的 request 资源不超过 Allocatable）。

Node Allocatable Resource = Node Capacity - Kube-reserved - system-reserved - eviction-threshold

使用 CLI参数

--enforce-node-allocatable=pods
--kube-reserved=memory=...
--system-reserved=memory=...
--eviction-hard=...

使用配置文件

/var/lib/kubelet/config.yaml

apiVersion: kubelet.config.k8s.io/v1beta1
......
enforceNodeAllocatable:
- pods
kubeReserved:  # 配置 kube 资源预留
  cpu: 500m
  memory: 1Gi
  ephemeral-storage: 1Gi
systemReserved:  # 配置系统资源预留
  memory: 1Gi
evictionHard:  # 配置硬驱逐阈值
  memory.available: "300Mi"
  nodefs.available: "10%"

rke1的情况

nodes:
  - address: 10.10.10.1
    user: rke
    role: [controlplane,etcd]
  - address: 10.10.10.2
    user: rke
    role: [controlplane,etcd,worker]
  - address: 10.10.10.3
    user: rke
    role: [worker]
system_images:
    kubernetes: rancher/hyperkube:v1.13.5-rancher1
services:
  etcd:
    backup_config:
      enabled: true 
      interval_hours: 6
      retention: 12
    extra_args:
      quota-backend-bytes: '4294967296'
  kubelet:
    cluster_domain: cluster.local
    cluster_dns_server: 10.43.0.10
    fail_swap_on: true
    extra_args:
      max-pods: 20
      enforce-node-allocatable: 'pods'
      system-reserved: 'cpu=1,memory=2000Mi'
      kube-reserved: 'cpu=1.5,memory=3000Mi'
  kube-controller:
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
    extra_args:
      node-cidr-mask-size: '24'
      node-monitor-period: '5s'
      node-monitor-grace-period: '10s'
      node-startup-grace-period: '20s'
      pod-eviction-timeout: '10s'
  kube-api:
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: 30000-32767
    pod_security_policy: false
    always_pull_images: false
dns:
  provider: coredns

rke2的情况

microk8s的情况

它的kubelet 启动程序是

/snap/microk8s/2407/kubelite

它的参数文件

/var/snap/microk8s/2407/args/kubelet

修改/var/snap/microk8s/2407/args/kubelet,追加以下内容

--enforce-node-allocatable=pods
--kube-reserved=cpu=500m
--kube-reserved=memory=1Gi
--system-reserved=cpu=500m
--system-reserved=memory=1Gi
--eviction-hard=10%

Git 常用命令

获取commitid
git rev-parse HEAD

通过commit获取 邮箱
git log --no-merges --pretty=format:\"%ae\" -1 " + commitid

通过commit获取 注释
git log --pretty=format:\"%s\" -1 " + commitid

通过commit获取 name
git log --pretty=format:\"%cn\" -1 " + commitid

通过commit获取 时间
git log --pretty=format:\"%ad\" -1 " + commitid

Argocd Notifications annotations

slack

notifications.argoproj.io/subscribe.on-sync-succeeded.slack:<my-channel>

webhook

notifications.argoproj.io/subscribe.on-sync-succeeded.<webhook-name>: ""

gmail

notifications.argoproj.io/subscribe.on-sync-succeeded.gmail: $EMAIL_USER

Kube-bench

kube-bench is tool that checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark.

使用docker安装

docker run --rm -v `pwd`:/host aquasec/kube-bench install

调用

./kube-bench

Kustomize patchesStrategicMerge

在使用补丁机制的时候，要尽量保证每一个补丁只做一件事

原始 nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    workload.user.cattle.io/workloadselector: deployment-lizhe-nginx
  name: nginx
  namespace: lizhe
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      workload.user.cattle.io/workloadselector: deployment-lizhe-nginx
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        workload.user.cattle.io/workloadselector: deployment-lizhe-nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        resources: {}
        stdin: true
        tty: true
      restartPolicy: Always
      terminationGracePeriodSeconds: 30

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- nginx.yaml
namePrefix: prod-
patchesStrategicMerge:
- increase_replicas.yaml
- set_memory.yaml

increase_replicas.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: lizhe
spec:
  replicas: 10

set_memory.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: lizhe
spec:
  template:
    spec:
      containers:
      - name: my-nginx
        resources:
          limits:
            memory: 512Mi

Use Kubectl in a Pod

参考 https://kubernetes.io/zh/docs/tasks/run-application/access-api-from-pod/

某些情况下，我们会希望在一个 pod 中能够使用 kubectl 对本集群进行操作，例如

argo workflow 中，需要一个 task 可以调用 kubectl 来修改某些资源

方法一、

最笨的方式是，mount kubeconfig yaml 文件，这种情况下你使用的是 config 文件中的 user ，这里就不讨论了，因为这个方式实在是太笨了

方法二、

使用 kubectl 镜像，然后赋给 pod 对应的 RBAC 权限，这样 pod 就可以通过 RBAC 绑定的 service account 对集群进行操作

使用的镜像是 bitnami/kubectl
使用的 entrypoint 是 /bin/bash

通过 kubectl get nodes 尝试获取集群信息

直接给 Namespace 的 default sa添加一个 cluster级的 admin role

kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --group=system:serviceaccounts:lizhe
clusterrolebinding.rbac.authorization.k8s.io/permissive-binding created

然后我们来尝试一下

Kustomize install

参考 https://kubectl.docs.kubernetes.io/installation/kustomize/

最容易的应该是直接使用 二进制 文件

https://github.com/kubernetes-sigs/kustomize/releases

sudo chmod a+x kustomize /usr/local/sbin
sudo cp kustomize /usr/local/sbin

我们来尝试一个helloworld

先编写一个 nginx 的 deploy

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    workload.user.cattle.io/workloadselector: deployment-lizhe-nginx
  name: nginx
  namespace: lizhe
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      workload.user.cattle.io/workloadselector: deployment-lizhe-nginx
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        workload.user.cattle.io/workloadselector: deployment-lizhe-nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        resources: {}
        stdin: true
        tty: true
      restartPolicy: Always
      terminationGracePeriodSeconds: 30

创建一个空文件

kustomization.yaml

然后使用 kustomize cli 来对其添加内容

kustomize edit add resource nginx.yaml

下面我们尝试使用 kustomize edit 给它加一个前缀

需要先修改 kustomization.yaml

kustomize edit set nameprefix 'prod-'

执行

kustomize build ./

你也可以直接应用到集群

kustomize build ./  | kubectl apply -f -

Kube-shell

安装

$ pip install kube-shell

这里我先把我的 kube config 文件拷贝到默认路径下，然后再启动kube-shell

cp /etc/rancher/rke2/rke2.yaml ~/.kube/config
kube-shell

Kubeless

安装非常简单，只是别用 microk8s 有一个服务启动不起来，rke2 实测没有问题

export RELEASE=$(curl -s https://api.github.com/repos/kubeless/kubeless/releases/latest | grep tag_name | cut -d '"' -f 4)
kubectl create ns kubeless
kubectl create -f https://github.com/kubeless/kubeless/releases/download/$RELEASE/kubeless-$RELEASE.yaml

kubectl get pods -n kubeless

export OS=$(uname -s| tr '[:upper:]' '[:lower:]')
curl -OL https://github.com/kubeless/kubeless/releases/download/$RELEASE/kubeless_$OS-amd64.zip && \
  unzip kubeless_$OS-amd64.zip && \
  sudo mv bundles/kubeless_$OS-amd64/kubeless /usr/local/bin/

查看支持的 runtime

root@ubuntu:~# kubeless get-server-config
INFO[0000] Current Server Config:                       
INFO[0000] Supported Runtimes are: ballerina0.981.0, dotnetcore2.0, dotnetcore2.1, dotnetcore2.2, dotnetcore3.1, go1.13, go1.14, java1.8, java11, nodejs10, nodejs12, nodejs14, php7.2, php7.3, php7.4, python3.6, python3.7, python3.8, ruby2.3, ruby2.4, ruby2.5, ruby2.6, jvm1.8, nodejs_distroless8, nodejsCE8, vertx1.8 
root@ubuntu:~# 

我们创建一个 test.py

def hello(event, context):
  print(event)
  return event['data']

然后通过这个文件创建一个 function

kubeless function deploy hello --runtime python3.7 \
                                --from-file test.py \
                                --handler test.hello

这里需要下载一个镜像，所以需要等待，不要直接启动

可以添加 -n Namespace

kubeless function deploy hello --runtime python3.7 \
                                --from-file test.py \
                                --handler test.hello \
                                -n lizhe

安装 UI

kubectl create -f https://raw.githubusercontent.com/kubeless/kubeless-ui/master/k8s.yaml

kubectl proxy -p 8080
curl -L --data '{"Another": "Echo"}' \
  --header "Content-Type:application/json" \
  localhost:8080/api/v1/namespaces/default/services/hello:http-function-port/proxy/