Docker

kubernetes-kompose tool

Kompose can convert the yaml file of docker compose into the yaml file used by kubernetes

Installation reference https://kompose.io/installation/

curl -L https://github.com/kubernetes/kompose/releases/download/v1.22.0/kompose-linux-amd64 -o kompose
chmod +x kompose
sudo mv ./kompose /usr/local/bin/kompose

Prepare a docker-compose yml

version: "3.9"
    
services:
  db:
    image: mysql:5.7
    volumes:
      - db_data:/var/lib/mysql
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: somewordpress
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: wordpress
    
  wordpress:
    depends_on:
      - db
    image: wordpress:latest
    volumes:
      - wordpress_data:/var/www/html
    ports:
      - "8000:80"
    restart: always
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: wordpress
      WORDPRESS_DB_NAME: wordpress
volumes:
  db_data: {}
  wordpress_data: {}

try convert

lizhe@ubuntu:~/works$ kompose convert
FATA Version 3.9 of Docker Compose is not supported. Please use version 1, 2 or 3 
lizhe@ubuntu:~/works$ 

Modify the version of the compose file from 3.9 to 3

Then we create the kubernetes resource

check the application by svc

kubectl port-forward --namespace default service/wordpress 8000:8000

It is easy to find that the DB does not generate a service here, because docker compose There is no port in dB in YML file
Modify it

You can see that the DB service is generated this time

Buildkit tool

In fact, buildkit does not need to be installed. You can directly download the corresponding binary executable file

https://github.com/moby/buildkit/releases

After downloading, you will get a folder similar to the following

FROM ubuntu
COPY buildctl /

Then start the daemon

sudo ./buildkitd

Direct use

buildctl build \
    --frontend gateway.v0 \
    --opt source=docker/dockerfile \
    --local context=. \
    --local dockerfile=.

What you get is an internal cache

By default, the build result and intermediate cache will only remain internally in BuildKit. An output needs to be specified to retrieve the result.

In other words, if you need to use this image, you need to use it with output
Build directly to local, where local is an unprocessed raw file, which is output to the specified folder, rather than the traditional docker local image

buildctl build \
    --frontend gateway.v0 \
    --opt source=docker/dockerfile \
    --local context=. \
    --local dockerfile=. \
    --output type=local,dest=./images

Build to local tar file

buildctl build \
    --frontend gateway.v0 \
    --opt source=docker/dockerfile \
    --local context=. \
    --local dockerfile=. \
    --output type=tar,dest=out.tar

Build to local image of traditional docker

buildctl build \
    --frontend gateway.v0 \
    --opt source=docker/dockerfile \
    --local context=. \
    --local dockerfile=. \
    --output type=docker,name=myimage | docker load

build to oci image

buildctl build \
    --frontend gateway.v0 \
    --opt source=docker/dockerfile \
    --local context=. \
    --local dockerfile=. \
    --output type=oci,dest=oci_output.tar

buildctl build \
    --frontend gateway.v0 \
    --opt source=docker/dockerfile \
    --local context=. \
    --local dockerfile=. \
    --output type=image,name=docker.io/libaibai/testbuildkit,push=true

CPU resource limit

The docker container will think that it can use all the CPU resources of the host. After starting the container, try to occupy a completed CPU core

docker run -it --name ubuntu ubuntu /bin/bash

while : ; do : ; done &

You can see that one of the four cores is 100US

We know that the docker – CPU shares parameter can control the percentage of CPU the container can occupy
For example, when a container is started separately, because there is no competition, even if – CPU shares = 500, the container can occupy all cores

To compete, we lock two containers on the first kernel and set the percentage

docker run -it --name test1 --cpu-shares=500 --cpuset-cpus 0 ubuntu /bin/bash

Mainly, there are two bash processes on the top at the bottom right, one is 66.4 and the other is 33.2

However – CPU shares can only control the percentage of usage, which is the only parameter provided by the early docker that can allocate CPU resources

This principle is the same as requests in kubernetes resources

If you want to implement limits in kubernetes resources, you need to use

• cpu.cfs_quota_us
• cpu.cfs_period_us

The default period is 100ms, or 100000 US. The following statement means that 100000 units of resources can be used, 20000, or 20%

docker run -it --name test1 --cpu-period=100000 --cpu-quota=20000 --cpuset-cpus 0 ubuntu /bin/bash

The bash process in the figure below is locked at 19.9%

Docker TLS remote connection-2376

If you want to use TLS to safely access the dockerd 2376 port on the remote server, you need to generate a certificate on the remote server, copy it to the client PC, and then use the certificate to access 2376. You can use the following script to generate a certificate

gen_docker_ca.sh

#!/bin/bash
#related configs
SERVER="192.168.194.142"
PASSWORD="password"
COUNTRY="CN"
STATE="LN"
CITY="DL"
ORGANIZATION="lz"
ORGANIZATIONAL_UNIT="Dev"
EMAIL="test@lizhe.name"

###start create file###
echo "start create file"

#go to the path
cd /opt/docker_ca 
#use aes256 gen the key
openssl genrsa -aes256 -passout pass:$PASSWORD  -out ca-key.pem 4096
#gen ca cert
openssl req -new -x509 -passin "pass:$PASSWORD" -days 365 -key ca-key.pem -sha256 -out ca.pem -subj "/C=$COUNTRY/ST=$STATE/L=$CITY/O=$ORGANIZATION/OU=$ORGANIZATIONAL_UNIT/CN=$SERVER/emailAddress=$EMAIL"

#gen server used private key
openssl genrsa -out server-key.pem 4096
#gen csr
openssl req -subj "/CN=$SERVER" -sha256 -new -key server-key.pem -out server.csr
#white list
sh -c 'echo subjectAltName = IP:'$SERVER',IP:0.0.0.0 >> extfile.cnf'
#extendedKeyUsage = serverAuth put into extfile.cnf
sh -c 'echo extendedKeyUsage = serverAuth >> extfile.cnf'
#Use the CA certificate, CA key and the above server certificate request file to generate the server self signed certificate
openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem -passin "pass:$PASSWORD" -\CAcreateserial -out server-cert.pem -extfile extfile.cnf

#Generate client certificate RSA private key file
openssl genrsa -out key.pem 4096
#Generate client certificate request file
openssl req -subj '/CN=client' -new -key key.pem -out client.csr
#Continue setting certificate extension properties
sh -c 'echo extendedKeyUsage = clientAuth >> extfile.cnf'
#Generate client self signed certificate (generated according to the above client private key file and client certificate request file)
openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem -passin "pass:$PASSWORD" -\CAcreateserial -out cert.pem -extfile extfile.cnf

#Change key permissions
chmod 0400 ca-key.pem key.pem server-key.pem
#Change key permissions
chmod 0444 ca.pem server-cert.pem cert.pem
#Delete useless files
rm client.csr server.csr

echo "Generate file complete"
###End###

vim /lib/systemd/system/docker.service

systemctl daemon-reload
systemctl restart docker

Copy ca.pem cert.pem key from docker server PEM these three files to MacOS

The generated certificate file is as follows, and the modification permission is 400

Use the correct certificate to access the remote dockerd

docker --tlsverify --tlscacert /Users/lizhe/docker-ca/ca.pem --tlscert /Users/lizhe/docker-ca/cert.pem --tlskey /Users/lizhe/docker-ca/key.pem -H 192.168.194.142:2376 version

Docker remote call

Remote call here refers to remote call through containerd. Please refer to the introduction of docker component for dockerd
Modify docker service

sudo vim /usr/lib/systemd/system/docker.service

在ExecStart末尾追加 -H tcp://0.0.0.0:2375

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock -H tcp://0.0.0.0:2375

Reload profile

systemctl daemon-reload
sudo systemctl restart docker

Call docker command through other computers

docker -H 192.168.194.142 ps -a

Docker privileged

The –privileged flag maps the user ID 0 we saw earlier directly to the user ID 0 of the host and gives it unrestricted access to any system call it chooses

docker run -i -t --rm --privileged ubuntu /bin/bash

Docker component introduction

Docker Cli
This is a command line tool for users

Dockerd
Listen to docker API requests. Dockerd receives API requests through UNIX, TCP and FD.

UNIX socket is / var / run / docker For sock, you need root or docker group permission to start dockerd

Dockerd will pull up containerd and keep communication with containerd when starting

Containerd

• The main responsibility is to manage the life cycle of the container
• Pull, push image
• Storage management
• Run the runc of the container
• Management network

Containerd contains a daemon service that exposes grpc APIs. These APIs are relatively low-level. Dockerd manages the life cycle of containers through container, and container runs containers through runc

Runc
/Usr / bin / docker runc can be regarded as part of containerd, which is a binary tool for running OCI compliant containers.
The container image is packaged in OCI standard format and generally includes config. Config JSON file and system root directory

docker save -o nginx.tar nginx

Save the container as an image tar file, and then decompress it to see the internal structure of the image

containerd-shim
The existence of containerd ship enables the container to run independently from containerd. (by default, dockerd is stopped and the container is stopped, but it can be realized through daemon.json configuration. After dockerd is stopped, the container runs as usual.)

As the parent process of container, containerd shim is mainly responsible for the following responsibilities:

• So that runc can exit after running the container, without starting a runtime resident process for a container
• Keep the stdio of the container open, so that the container will not exit after receiving SIGPIPE after writing stdio
• Report the exit status of the container to containerd

The calling order is dockerd — > containerd — > containerd shim — > runc API — > “CMD”

Docker swarm

Swarm engine has been integrated in the new version of docker and does not need additional installation

docker swarm init --advertise-addr 192.168.194.137

Use the command prompted above to join to the cluster on node 2

docker swarm join --token SWMTKN-1-42oor9ruw9jv0bqw7vqmm6c29jwo5mnf34xvfn9iwojpor8c4q-bbxltikj5m5r1fnkhezlhjkts 192.168.194.137:2377

Now you can see that there are two nodes in the cluster

Create an overlay network

docker network create --driver overlay --opt encrypted --subnet 10.10.1.0/24 ol_net --attachable

Start nginx service

Finally, I start a Ubuntu container. You can see that this container on node 1 can ping nginx on node 1 and node 2 at the same time

Build docker image from a docker container

For ease of understanding, I will try to use manual operation instead of dockerfile
Start a Ubuntu container first

docker run -i -t --name base ubuntu /bin/bash

If you install docker in this container and try to execute docker PS, you will get the following error

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

The reason is actually very simple, because there is docker inside the container In fact, the simplest way to build an image in a container is to mount the sock, and then the container will share the same docker with the host sock

docker run -i -t --name base2 -v /var/run/docker.sock:/var/run/docker.sock ubuntu /bin/bash

However, the side effect is that because the host’s sock is used, all containers running on the host can be seen here, and actually use the same docker environment as the host

Create a dockerfile

FROM nginx
RUN echo 'hello dockerfile' > /usr/share/nginx/html/index.html

Build again

error checking context: 'no permission to read from '/proc/1/mem''.

This is because the current container does not have the root permission to add the host when it is created
Add –privileged

docker run -i -t --name base3 --privileged -v /var/run/docker.sock:/var/run/docker.sock ubuntu /bin/bash

Here is an episode. If you use dockerfile directly under / path, you will get an error

error checking context: ‘file (‘/proc/4427/fd/5′) not found or excluded by . dockerignore’.

The solution is not to build under the / root path

Version before 1.11

Docker version 1.11 is a big change,
First, the first mock exam is divided into 4 independent modules from single module, engine, containerd, runc and containerd-shim.
Containerd shim replaces the original process 1 and is used to recycle orphan processes
Containerd is used to manage the life cycle of containers,
Runc is used to run the container

This change made it more compliant and standardized in 2017
In order to install the version before 1.11, I tried for almost 4 hours
The main reason is that almost all apt installation packages have been offline
Students who want to try can follow the following steps
Download the DEB offline package and install it with Ubuntu 16. 18 and 20 are not compatible

http://archive.ubuntu.com/ubuntu/pool/universe/d/docker.io/docker.io_1.10.3-0ubuntu6_amd64.deb

Start a Ubuntu image, block its No. 1 process, and generate a zombie process by reference

Kill process 42, so that the parent process of process 45 will become process 1 of the container

Then kill process 45. At this time, process 1 of the container should not be able to recycle it
Then we get a zombie process that can’t be recycled