en

Kong Certificate installation

First, open port 8443 of Kong

Complete docker compose The YML file is as follows

version: "3"
 
networks:
 kong-net:
  driver: bridge
 
services:
 
  #######################################
  # Postgres: The database used by Kong
  #######################################
  kong-database:
    image: postgres:9.6
    restart: always
    networks:
      - kong-net
    environment:
      POSTGRES_USER: kong
      POSTGRES_DB: kong
      POSTGRES_PASSWORD: kong
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "kong"]
      interval: 5s
      timeout: 5s
      retries: 5
 
  #######################################
  # Kong database migration
  #######################################
  kong-migration:
    image: kong:latest
    command: "kong migrations bootstrap"
    networks:
      - kong-net
    restart: on-failure
    environment:
      - KONG_DATABASE=postgres
      - KONG_PG_HOST=kong-database
      - KONG_PG_DATABASE=kong
      - KONG_PG_PASSWORD=kong
    links:
      - kong-database
    depends_on:
      - kong-database
 
  #######################################
  # Kong: The API Gateway
  #######################################
  kong:
    image: kong:latest
    restart: always
    networks:
      - kong-net
    environment:
      KONG_DATABASE: postgres
      KONG_PG_HOST: kong-database
      KONG_PG_PASSWORD: kong
      KONG_PROXY_LISTEN: 0.0.0.0:8000,0.0.0.0:8443 ssl
      KONG_ADMIN_LISTEN: 0.0.0.0:8001
    depends_on:
      - kong-migration
    links:
      - kong-database
    healthcheck:
      test: ["CMD", "curl", "-f", "http://kong:8001"]
      interval: 5s
      timeout: 2s
      retries: 15
    ports:
      - "8001:8001"
      - "8000:8000"
      - "8443:8443"
 
 
  #######################################
  # Konga database prepare
  #######################################
  konga-prepare:
    image: pantsel/konga:latest
    command: "-c prepare -a postgres -u postgresql://kong:kong@kong-database:5432/konga"
    networks:
      - kong-net
    restart: on-failure
    environment:
      - KONG_DATABASE=postgres
      - KONG_PG_HOST=kong-database
      - KONG_PG_DATABASE=konga
      - KONG_PG_PASSWORD=kong
    links:
      - kong-database
    depends_on:
      - kong-database
 
  #######################################
  # Konga: Kong GUI
  #######################################
  konga:
    image: pantsel/konga:latest
    restart: always
    networks:
     - kong-net
    environment:
      DB_ADAPTER: postgres
      DB_URI: postgresql://kong:kong@kong-database:5432/konga
      NODE_ENV: production
    links:
      - kong-database
    depends_on:
      - kong
      - konga-prepare
    ports:
      - "1337:1337"

Then you need to inject a cert

curl -k -X POST \
  http://diynocap.com:8001/certificates \
  -H 'Content-Type: multipart/form-data' \
  -F cert=@./cert.crt \
  -F key=@./private.key \
  -F snis[]=studyk8s.com

Although it will prompt Zsh: no matches found: snis [] = studyk8s But the certificate was created successfully

It can also be added through Konga’s UI interface

The back-end service can normally use HTTP port 80, and then expose the front-end Kong to 8443

Ingress certificate configuration

The certificate configuration of ingress is relatively simple

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: feiutest.cn
    http:
      paths:
      - path:
        backend:
          serviceName: test-ingress
          servicePort: 80
  tls:
  - hosts:
    - feiutest.cn
    secretName: nginx-test

Here is how to create a certificate

kubectl create secret tls who-tls --cert=tls.crt --key=tls.key

ingress sticky session

Deployment for testing

Ingress for testing

The image used is contained / whoamI, and the effect is as follows

After changing the number of pods to 3, you can easily get the following information. The request will rotate between the three pods

Just add annotation

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "ingresscoookie"
    nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"

nginx ingress https backend

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-nginx-ingress
  namespace: study
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/ssl-services: "nginx-service"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
  - host: www.bestofgit.com
    http:
      paths:
      - backend:
          serviceName: nginx-service
          servicePort: 443
  tls:
  - hosts:
    - www.bestofgit.com

nginxingress letsencrypt

Here I use kubernetes + an independent domain name of my own. It runs in AWS environment. The IP of EC2 host is 54.95.179.97

Except that cert manager is installed in the system namespace, all resources (deployment, service, ingress, issuer, Cert) are installed in the study namespace

It should be noted that the namespace needs to be created in advance

install cert-manager

kubectl create namespace cattle-system
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.0.4/cert-manager.crds.yaml
kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.0.4

lizhedeMacBook-Pro:lz_study lizhe$ cat issuer.yaml 
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: study
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: marshal_li_b@163.com
    privateKeySecretRef:
      name: letsencrypt-prod
    http01: {}
lizhedeMacBook-Pro:lz_study lizhe$ 

• metadata. Name is the name of the issuing authority we created, which will be referenced later when we create the certificate
• spec.acme. Email is your own email. There will be an email reminder when the certificate is about to expire, but cert manager will automatically reissue the certificate to us for renewal by using acme protocol
• spec.acme. Server is the server of acme protocol. Let’s use let’s encrypt here, and the address will be written like this
• spec.acme. Privatekeysecretref indicates which secret object this issuing authority’s private key will be stored in. The name is not important
• spec.acme. Http01 here instructs the issuing authority to use http-01 for acme protocol (DNS can also be used. The purpose of acme protocol is to prove that the machine and domain name belong to you, and then allow you to issue certificates)

Create certificate

It should be noted that the namespace needs to be created in advance

lizhedeMacBook-Pro:lz_study lizhe$ cat cert.yaml 
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: study-pactera-deg-com
  namespace: study
spec:
  secretName: best-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - www.bestofgit.com
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - www.bestofgit.com
lizhedeMacBook-Pro:lz_study lizhe$ 

• Spec.secretname indicates the secret in which the certificate is finally saved
• spec.issuerRef. The kind value is clusterissuer, indicating that the issuing authority is not in this namespace, but in the global
• spec.issuerRef. Name the name of the issuing authority we created (clusterissuer. Metadata. Name)
• Spec.dnsnames indicates which domain names the certificate can be used for
• spec.acme. config. http01. When the ingress class uses http-01 to verify the domain name and machine, cert manager will try to create an ingress object to realize the verification. If this value is specified, kubernets will be added to the created ingress io/ingress. Class this annotation. If our ingress controller is nginx ingress controller, specify this field to allow the created ingress to be processed by nginx ingress controller.
• spec.acme. config. http01. Domains indicates which domain names the certificate can be used for

Then install nginx

deployment

lizhedeMacBook-Pro:lz_study lizhe$ cat nginx.yaml 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: study
spec:
  replicas: 2 # tells deployment to run 2 pods matching the template
  template: # create pods using pod definition in this template
    metadata:
      # unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is
      # generated from the deployment name
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
lizhedeMacBook-Pro:lz_study lizhe$ 

svc.yaml

lizhedeMacBook-Pro:lz_study lizhe$ cat svc.yaml 
apiVersion: v1
kind: Service
metadata: 
  name: nginx-service
  namespace: study
spec:
  type: ClusterIP
  ports:
    - port: 80
  selector: 
      app: nginx
lizhedeMacBook-Pro:lz_study lizhe$ 

ingress.yaml

lizhedeMacBook-Pro:lz_study lizhe$ cat ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-nginx-ingress
  namespace: study
  annotations:
    kubernetes.io/ingress.class: "nginx"
    certmanager.k8s.io/issuer: "letsencrypt-prod"
    certmanager.k8s.io/acme-challenge-type: http01
spec:
  rules:
  - host: www.bestofgit.com
    http:
      paths:
      - backend:
          serviceName: nginx-service
          servicePort: 80
  tls:
  - hosts:
    - www.bestofgit.com
    secretName: best-tls
lizhedeMacBook-Pro:lz_study lizhe$ 

kubernetes ingress

Kubernetes ingress is usually used to expose services in the form of L7 loadbalance, which will not be repeated here

Here is just to describe some neglected problems of ingress itself as clearly as possible

Ingress is one of the standard resource types of kubernetes API. It is actually a set of rules that forward requests to specified service resources based on DNS name (host) or URL path

The implementation of ingress is divided into two parts: ingress controller and ingress

Ingress controller is the entrance of traffic. It is an entity software, generally nginx and haproxy.

Ingress describes specific routing rules.

Because the content is messy, it is divided into several specific real cases to discuss

K3s

Recently, I tried k3s. The default implementation of k3s (no need to install rancher, bare k3s) is not nginx ingress, but traefik

After installing the cluster, I encountered a very strange problem. In the past, when using kubernetes, as long as an ingress rule is added, DNS can access this deployment as long as it points to any node, but only one node can access it normally in k3s, for example

• 192.168.204.165
• 192.168.204.166
• 192.168.204.167
• 192.168.204.168

Only 192.168.204.168 can access ingress

Checking the ingress nginx controller, it is found that rke2 does not use the daemon set to deploy the controller like rke1, but inexplicably uses ordinary deployment, which leads to that only 168 deployed with ingress controller can receive requests normally

Rke1 is much better, because it is a daemonset, so every node can work normally

Here, we manually change the deployment quantity to 4, and the controller is successfully distributed on 4 nodes

In this case, any node can receive the request normally

Address shows all addresses

Kubernetes externalip Introduction

How to expose services is a very important and confusing problem

A basic general rule here is

If you need to communicate on the TCP layer (L4), use nodeport (typical application is MySQL)

If you only need to communicate over HTTP or HTTPS (L7), use ingress (the typical application is nginx)

Most HelloWorld programs will simply and brutally use nodeport. It is easy to use and can be used in almost any case, but it is not elegant. Ingress gracefully solves some problems with the help of reverse proxy service, but it can not support L4 layer network communication

However, most documents and books rarely mention external IP, and even the official documents are very vague.

If there are external IPs that route to one or more cluster nodes, Kubernetes Services can be exposed on those externalIPs. Traffic that ingresses into the cluster with the external IP (as destination IP), on the Service port, will be routed to one of the Service endpoints. externalIPs are not managed by Kubernetes and are the responsibility of the cluster administrator.

deploy a MySQL

create a nodeport service

You can see that 30686 has been opened and can connect to the database normally

To create a service of externalip, you need to create a service of type clusterip, and then add externalip

spec:
  clusterIP: 10.43.245.251
  clusterIPs:
  - 10.43.245.251
  externalIPs:
  - 192.168.194.149
  ports:
  - name: default
    port: 3306
    protocol: TCP
    targetPort: 3306
  selector:
    workloadID_mysqle: "true"
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Does it seem strange that it is an externalip, but the type is clusterip

Externalip allows you to directly use IP plus 3306 to connect to MySQL without using the weird looking high port (30686)

The advantage of externalip is

You can directly use IP + original port to access the service

The disadvantages are also obvious

No high availability and no load balancing

External IP is not managed by a cluster. IP also needs manual control. You have to do a lot of manual work

Here, load balancing and high availability can be handled by a portal or Robin DNS. Of course, complex health check and circuit breaker mechanisms may be required to reach the commercial level

Etcd watch mechanism

First, start etcd with docker

docker run -d\
  -p 2379:2379 \
  -p 2380:2380 \
  --name etcd quay.io/coreos/etcd:latest \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name node1 --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://0.0.0.0:2379 --listen-peer-urls=http://0.0.0.0:2380 --initial-advertise-peer-urls=http://0.0.0.0:2380

We add a key value pair to etcd

curl http://127.0.0.1:2379/v2/keys/key1 -XPUT -d value="Hello world"

Etcd’s watch monitoring can be divided into two types: one-time monitoring and continuous monitoring

One time monitoring

If an event is listened to, JSON data will be returned. After returning, the listener exits, and the listener needs to be started again later. Recursive is to listen for changes of all child nodes.

curl http://127.0.0.1:2379/v2/keys/key1?wait=true&recursive=true

Then let’s change the value of key1

curl http://127.0.0.1:2379/v2/keys/key1 -XPUT -d value="changed value"

When the value of key1 changes, you can see that the previous wait process is over and listen for the return

Permanent monitoring. If an event is monitored, it will not exit and continue to return data. It is more reliable than one-time monitoring

curl http://127.0.0.1:2379/v2/keys/key1\?wait\=true\&recursive\=true\&stream\=true

Kubernetes informer mechanism

With the exception of CRI, most kubernetes components are designed with declarative interfaces

There are two main reasons why CRI adopts command interface

All runtimes need to re implement the same logic and support many pod level functions and mechanisms;

The definition of pod evolves very quickly in CRI design, and functions such as initializing containers need the cooperation of runtime;

Although the community finally chose the command interface for CRI, kubelet will still ensure that the state of pod will continue to migrate to the desired state.

After receiving the request, apiserver will save the relevant state of the current pod in etcd, and then other components will query the state and migrate the state. You can refer to the kubernetes pod creation process. This mechanism will lead to excessive load and low efficiency of apiserver

Informer is mainly used to act as the middle layer between apiserver and the controller. Based on the event mechanism, on the one hand, it constructs the local cache, on the other hand, it triggers the event method, so that the controller can respond quickly and obtain data quickly. It mainly uses the list & watch mechanism

The controller does not get the expected state of the object by constantly polling the API and calling list and get

Instead, informer uses the watch mechanism that relies on etcd to know the change state of objects through events and establish a local cache

Etcd’s watch mechanism is a kind of subscription mode mechanism. It listens to one or a group of keys. Any change of keys will send a message

List & watch will only be executed once, that is, first execute list, pull the data to the queue, and then enter the watch stage. In order to enable different apiservers to load these watch requests evenly, the client will actively disconnect from the apiserver with a timeout of 60 seconds, and then reissue the watch request

Why did kubernetes remove docker

Let’s take a look at the actual relationship between kubernetes and docker

In the article “what is a container”, we mentioned that a container is not just a docker, but actually a container

A set of specifications defined by OCI organization, including runtime spec and image format spec

Kubernetes introduced CRI in version 1.5 (2016), so the relationship between docker and kubernetes becomes as follows

Dockershim is maintained by the community and allows kubernetes to use docker gasket of docker normally

CRI-O

When the container runtime standard was proposed, some people in red hat began to think that they could build a simpler runtime, which was only used by kubernetes. In this way, there is the Skunkworks project, finally named cri-o, which implements a minimum CRI interface

Finally, this structure evolved into this

In 2021 when this paper was written, that is, six years after the concept of CRI interface was put forward, kubernetes has become the de facto standard of container layout environment, and naturally began to be higher than the native docker in terms of discourse power. On the other hand, docker seems to have never seriously considered supporting CRI interface, and kubernetes community still needs to maintain dockership in the warehouse

It’s not hard to understand why kubernetes plans to start gradually removing support for docker

Not to mention that for kubernetes, docker should not support a series of interfaces of other modules

The relationship between docker and kubernetes CRI is more like the following

Docker provides more functions far beyond the needs of CRI and provides a full set of tools from operation to construction. It is precisely because of this responsibility and versatility that kubernetes has to pay more costs when docking with docker.

There are many new CRI features that have been repeatedly delayed due to their inability to implement on dockership, resulting in kubernetes finally choosing to try to remove dockership from the project.