Month: May 2021

AKS证书过期时间

参考内容

https://docs.microsoft.com/zh-cn/azure/aks/certificate-rotation

kubectl config view --raw -o jsonpath="{.clusters[?(@.name == 'cluster_name')].cluster.certificate-authority-data}" | base64 -d | openssl x509 -text | grep -A2 Validity

使用 az aks rotate-certs 轮换证书可能会导致 AKS 群集最多停机 30 分钟。

az aks rotate-certs -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME

NginxIngress Letsencrypt

这里我使用的是 Kubernetes + 我自己的一个独立域名, 运行在 aws 环境上 , ec2 主机的 ip 是 54.95.179.97

除了 cert-manager 安装在 system 命名空间下之外, 所有的资源 ( deployment, service, ingress, issuer, cert ) 全部都安装在 study 命名空间下

需要注意的是命名空间需要提前创建

安装cert-manager

kubectl create namespace cattle-system
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.0.4/cert-manager.crds.yaml
kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.0.4

cert-manager 提供了 Issuer 和 ClusterIssuer 这两种用于创建签发机构的自定义资源对象，

Issuer 只能用来签发自己所在 namespace 下的证书

ClusterIssuer 可以签发任意 namespace 下的证书

创建 issuer

lizhedeMacBook-Pro:lz_study lizhe$ cat issuer.yaml 
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: study
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: marshal_li_b@163.com
    privateKeySecretRef:
      name: letsencrypt-prod
    http01: {}
lizhedeMacBook-Pro:lz_study lizhe$ 

• metadata.name 是我们创建的签发机构的名称，后面我们创建证书的时候会引用它
• spec.acme.email 是你自己的邮箱，证书快过期的时候会有邮件提醒，不过 cert-manager 会利用 acme 协议自动给我们重新颁发证书来续期
• spec.acme.server 是 acme 协议的服务端，我们这里用 Let’s Encrypt，这个地址就写死成这样就行
• spec.acme.privateKeySecretRef 指示此签发机构的私钥将要存储到哪个 Secret 对象中，名称不重要
• spec.acme.http01 这里指示签发机构使用 HTTP-01 的方式进行 acme 协议 (还可以用 DNS 方式，acme 协议的目的是证明这台机器和域名都是属于你的，然后才准许给你颁发证书)

创建证书

需要注意的是命名空间需要提前创建

lizhedeMacBook-Pro:lz_study lizhe$ cat cert.yaml 
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: study-pactera-deg-com
  namespace: study
spec:
  secretName: best-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - www.bestofgit.com
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - www.bestofgit.com
lizhedeMacBook-Pro:lz_study lizhe$ 

• spec.secretName 指示证书最终存到哪个 Secret 中
• spec.issuerRef.kind 值为 ClusterIssuer 说明签发机构不在本 namespace 下，而是在全局
• spec.issuerRef.name 我们创建的签发机构的名称 (ClusterIssuer.metadata.name)
• spec.dnsNames 指示该证书的可以用于哪些域名
• spec.acme.config.http01.ingressClass 使用 HTTP-01 方式校验该域名和机器时，cert-manager 会尝试创建Ingress 对象来实现该校验，如果指定该值，会给创建的 Ingress 加上 kubernetes.io/ingress.class 这个 annotation，如果我们的 Ingress Controller 是 Nginx Ingress Controller，指定这个字段可以让创建的 Ingress 被 Nginx Ingress Controller 处理。
• spec.acme.config.http01.domains 指示该证书的可以用于哪些域名

然后安装nginx

deployment

lizhedeMacBook-Pro:lz_study lizhe$ cat nginx.yaml 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: study
spec:
  replicas: 2 # tells deployment to run 2 pods matching the template
  template: # create pods using pod definition in this template
    metadata:
      # unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is
      # generated from the deployment name
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
lizhedeMacBook-Pro:lz_study lizhe$ 

svc.yaml

lizhedeMacBook-Pro:lz_study lizhe$ cat svc.yaml 
apiVersion: v1
kind: Service
metadata: 
  name: nginx-service
  namespace: study
spec:
  type: ClusterIP
  ports:
    - port: 80
  selector: 
      app: nginx
lizhedeMacBook-Pro:lz_study lizhe$ 

ingress.yaml

lizhedeMacBook-Pro:lz_study lizhe$ cat ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-nginx-ingress
  namespace: study
  annotations:
    kubernetes.io/ingress.class: "nginx"
    certmanager.k8s.io/issuer: "letsencrypt-prod"
    certmanager.k8s.io/acme-challenge-type: http01
spec:
  rules:
  - host: www.bestofgit.com
    http:
      paths:
      - backend:
          serviceName: nginx-service
          servicePort: 80
  tls:
  - hosts:
    - www.bestofgit.com
    secretName: best-tls
lizhedeMacBook-Pro:lz_study lizhe$ 

Traefik Letsencrypt

这个例子展示如何在 Traefik 上使用 letsencrypt 自动证书

首先需要在Traefik的cli上添加

- --providers.kubernetesIngress.ingressClass=traefik-cert-manager

创建一个后端应用 nginx，打开它的 80 端口，注意这里后端服务是 http 的

创建一个 clusterip 的 service

准备妥当之后我们来先生成证书

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: YOUR
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: issuer-account-key
    # Add a single challenge solver, HTTP01
    solvers:
      - http01:
          ingress:
            class: traefik-cert-manager
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: golanghelloworld-cert
  namespace: lizhe
spec:
  commonName: nginx.golanghelloworld.com
  secretName: golanghelloworld-cert
  dnsNames:
    - nginx.golanghelloworld.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

确认证书成功创建了

kubectl describe certificate -n lizhe golanghelloworld-cert

Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Issuing    84s   cert-manager  Issuing certificate as Secret does not exist
  Normal  Generated  84s   cert-manager  Stored new private key in temporary Secret resource "golanghelloworld-cert-dr48r"
  Normal  Requested  84s   cert-manager  Created new CertificateRequest resource "golanghelloworld-cert-rq7cc"
  Normal  Issuing    57s   cert-manager  The certificate has been successfully issued

将证书绑定到对应的 IngressRoute

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: webappingress
  namespace: lizhe
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`nginx.golanghelloworld.com`) && PathPrefix(`/golang`)
      kind: Rule
      services:
        - name: ngsvc
          port: 80
      middlewares:
        - name: golang-stripprefix
  tls:
    secretName: golanghelloworld-cert
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: golang-stripprefix
  namespace: lizhe
spec:
  stripPrefix:
    prefixes:
      - /golang

这样 TLS 就生效了

使用sealer快速安装Kubernetes

sealer项目的主要目的是让用户可以像 分发docker镜像 一样，分发Kubernetes集群

https://github.com/alibaba/sealer

#install Sealer binaries
wget https://github.com/alibaba/sealer/releases/download/v0.1.4/sealer-0.1.4-linux-amd64.tar.gz && \
tar zxvf sealer-0.1.4-linux-amd64.tar.gz && mv sealer /usr/bin
#run a kubernetes cluster 
sealer run kubernetes:v1.19.9 --masters 192.168.194.186 --passwd 123456

Sonarqube Jenkins pipeline

下面是一个使用 docker slave 的 sonarqube 例子

pipeline {
    agent none
    
    environment {
        SONAR_LOGIN='xxxxxxxxxxxx'
    }


    stages {
        stage ('Checkout') {
            agent any
            steps {
                git branch: 'master', credentialsId: 'sonarpoc', url: 'https://github.com/zl86790/sonarhelloworld.git'
                sh "ls /var/jenkins_home/workspace/${env.JOB_NAME}/"
            }
        }
        stage('Scan') {
            agent {
                docker {
                    image 'docker:20.10.6'
                    args "-v /var/run/docker.sock:/var/run/docker.sock -v /var/jenkins_home/workspace/${env.JOB_NAME}/:/usr/src"
                }
            }
            steps {
                dir("/var/jenkins_home/workspace/${env.JOB_NAME}/") {
                    sh "cp -r ./ /usr/src"
                    sh "ls -sail /usr/src"
                    sh """
                    docker run --rm -e SONAR_HOST_URL="http://xxx.xxx.xxx.xxx:9000/" -e SONAR_LOGIN='${env.SONAR_LOGIN}' -v /usr/src:/usr/src sonarsource/sonar-scanner-cli \
                    -Dsonar.projectKey=test -Dsonar.sources=/usr/src
                    """
                }
            }
        }
    }
}

例子2

pipeline {
    agent none
    
    environment {
        SONAR_LOGIN='310f5acc8da8d088640b7da3b1123f2d626e0d3d'
    }


    stages {
        stage ('Checkout') {
            agent any
            steps {
                sh "rm -rf /var/jenkins_home/workspace/${env.JOB_NAME}/*"
                git branch: 'development', credentialsId: '574ec31b-8c2f-4454-83f0-0708b49289f8', url: 'https://github.com/xxx/xxxx.git'
            }
        }
        stage('Scan') {
            agent {
                docker {
                    image 'docker:20.10.6'
                    args "-v /var/run/docker.sock:/var/run/docker.sock -v /var/jenkins_home/workspace/${env.JOB_NAME}/:/usr/src"
                }
            }
            steps {
                dir("/var/jenkins_home/workspace/${env.JOB_NAME}/") {
                    sh """
                    ls /usr/src
                    rm -rf /usr/src/*
                    ls /usr/src
                    ls /var/jenkins_home/workspace/${env.JOB_NAME}
                    cp -r /var/jenkins_home/workspace/${env.JOB_NAME}/* /usr/src/
                    ls /usr/src
                    docker run --rm -e SONAR_HOST_URL="http://xxx.xxx.xxx.xxx:9000/" -e SONAR_LOGIN='${env.SONAR_LOGIN}' -v /var/jenkins_home/workspace/${env.JOB_NAME}/:/usr/src sonarsource/sonar-scanner-cli \
                    -Dsonar.projectKey=test -Dsonar.sources=./
                    """
                }
            }
        }
    }
}

Traefik rewrite path and prefixes

在dns资源有限，但是仍然要区分后端服务时，就需要使用 subdirectory 而不是 subdomain

常规来说，如果是 nginx ingress 或者 traefik 1，需要添加 annotation

参考以下官方文档，这里是 traefik2 的情况

https://doc.traefik.io/traefik/migration/v1-to-v2/#strip-and-rewrite-path-prefixes

这里使用的 namespace 是 my

先创建一个 nginx服务

再创建一个 clusterip 的 Service

使用下面的yaml

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: myir
  namespace: my
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`mynginx.lizhe.com`) && PathPrefix(`/mytest`)
      kind: Rule
      services:
        - name: mysvc
          port: 80
      middlewares:
        - name: my-stripprefix
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: my-stripprefix
  namespace: my
spec:
  stripPrefix:
    prefixes:
      - /mytest

Redis keys

keys 命令的参数是一个正则，指令非常简单，只要提供简单的字符串就可以得到结果

但是缺点也很明显

Keys 不提供 offset 和 limit 参数，会一次性返回所有符合条件的key
Keys 是遍历算法，可能会造成 redis服务卡顿 （redis是单线程）

Prometheus Alerting

https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md

这里我在自己的dashboard中创建了两个 panel

添加一个 notification channel 这里绑定到了 teams

rate(node_network_transmit_bytes_total{job="node-exporter", instance="10.0.2.7:9100", device!="lo"}[$__interval])
1- sum(increase(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)/sum(increase(node_cpu_seconds_total[5m])) by (instance)

配置alert

用 wrk 触发之后可以得到