Author: studyk8s

kube-prometheus 安装(yaml方式）

之前我们在 kube-prometheus 安装 中提到过，仓库又过时了

https://github.com/prometheus-community/community/issues/28#issuecomment-670406329

最新的安装请参考

https://github.com/prometheus-operator/kube-prometheus

https://github.com/prometheus-operator/kube-prometheus#installing

下载代码

git clone https://github.com/prometheus-operator/kube-prometheus.git

安装

# Create the namespace and CRDs, and then wait for them to be available before creating the remaining resources
kubectl create -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl create -f manifests/

这个脚本似乎和我的 rke2 有一些冲突，很多资源在 rke2 中已经存在了

不过这一些 already exists 的错误似乎并不影响使用

Prometheus

$ kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

Grafana

$ kubectl --namespace monitoring port-forward svc/grafana 3000

默认密码是
username: admin
password: admin
第一次登录之后会提示你修改密码

自带的模板

Alert Manager

$ kubectl --namespace monitoring port-forward svc/alertmanager-main 9093

Argo Workflows workflow sensor

对应的sensor不会持有具体内容，而是引用template中的内容

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: backendgithub
  namespace: argo
spec:
  template:
    spec:
      containers:
        - name: "sensor"
          image: "argoproj/sensor:v1.3.1"
          imagePullPolicy: IfNotPresent
      # serviceAccountName: operate-workflow-sa
  dependencies:
    - name: be-dep
      eventSourceName: backendgithub
      eventName: backendgitwebhook
  triggers:
    - template:
        name: build-be-workflow-trigger
        argoWorkflow:
          group: argoproj.io
          version: v1alpha1
          resource: workflows
          operation: submit
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: backendbuildkit-
                namespace: argo
              spec:
                # serviceAccountName: operate-workflow-sa
                entrypoint: main 
                onExit: exit-handler  
                imagePullSecrets:
                - name: regcred
                arguments:
                  parameters:
                    - name: repo
                      value: https://github.com/lizhe/lizhe-web-be.git
                    - name: branch
                      value: develop
                    - name: image
                      value: xxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/lizhewebbe
                    - name: dockercontext
                      value: /work/lizheweb-server
                    - name: dockerfile
                      value: dev.Dockerfile
                    - name: depfileyaml
                      value: lizhewebbe/lizhewebbe_dep.yaml
                    - name: projectname
                      value: lizhe-web-be
                templates:
                  - name: main
                    dag:
                      tasks:
                        - name: clean
                          templateRef:
                            name: buildkit
                            template: clean
                        - name: clone
                          templateRef:
                            name: buildkit
                            template: clone
                          arguments:
                            parameters:
                              - name: repo
                                value: '{{workflow.parameters.repo}}'
                              - name: branch
                                value: '{{workflow.parameters.branch}}'
                          depends: clean
                        - name: getcommitid
                          templateRef:
                            name: buildkit
                            template: getcommitid
                          arguments: {}
                          depends: clone
                        - name: mountcommitid
                          templateRef:
                            name: buildkit
                            template: mountcommitid
                          arguments: {}
                          depends: getcommitid
                        - name: build
                          templateRef:
                            name: buildkit
                            template: backend
                          arguments: {}
                          depends: mountcommitid
                        - name: image
                          templateRef:
                            name: buildkit
                            template: image
                          arguments:
                            parameters:
                              - name: image
                                value: '{{workflow.parameters.image}}'
                              - name: dockercontext
                                value: '{{workflow.parameters.dockercontext}}'
                              - name: dockerfile
                                value: '{{workflow.parameters.dockerfile}}'
                              - name: commitid
                                value: '{{tasks.mountcommitid.outputs.parameters.commitid}}'
                          depends: build
                        - name: deploy
                          templateRef:
                            name: buildkit
                            template: deploy
                          arguments:
                            parameters:
                              - name: depfileyaml
                                value: '{{workflow.parameters.depfileyaml}}'
                          depends: image
                          onExit: deploy-exit-handler  
                  - name: exit-handler
                    steps:
                    - - name: email
                        templateRef:
                          name: buildkit
                          template: exit-handler
                        arguments:
                          parameters:
                            - name: projectname
                              value: '{{workflow.parameters.projectname}}'
                  - name: deploy-exit-handler
                    steps:
                    - - name: clean
                        templateRef:
                          name: buildkit
                          template: deploy-exit-handler
                volumeClaimTemplates:
                  - metadata:
                      name: work
                      creationTimestamp: null
                    spec:
                      accessModes:
                        - ReadWriteOnce
                      resources:
                        requests:
                          storage: 2Gi
                      storageClassName: gp2
                    status: {}
                ttlStrategy:
                  secondsAfterCompletion: 1800
                  secondsAfterSuccess: 1800
                  secondsAfterFailure: 1800
                        

Argo Workflows workflow event source

Event source

# Info on GitHub Webhook: https://developer.github.com/v3/repos/hooks/#create-a-hook
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: backendgithub
  namespace: argo
spec:
  service:
    ports:
      - port: 12000
        targetPort: 12000
  github:
    backendgitwebhook:
      repositories:
        - owner: lizhe
          names:
            - lizhe-web-be
      # Github will send events to following port and endpoint
      webhook:
        # endpoint to listen to events on
        endpoint: /push
        # port to run internal HTTP server on
        port: "12000"
        # HTTP request method to allow. In this case, only POST requests are accepted
        method: POST
        # url the event-source will use to register at Github.
        # This url must be reachable from outside the cluster.
        # The name for the service is in `<event-source-name>-eventsource-svc` format.
        # You will need to create an Ingress or Openshift Route for the event-source service so that it can be reached from GitHub.
        url: http://xxxxxxxxxxxxxxxxxxxxxxxxxx.ap-northeast-1.elb.amazonaws.com:12000
      # type of events to listen to.
      # following listens to everything, hence *
      # You can find more info on https://developer.github.com/v3/activity/events/types/
      events:
        - "push"

      # apiToken refers to K8s secret that stores the github api token
      # if apiToken is provided controller will create webhook on GitHub repo
      # +optional
      apiToken:
        # Name of the K8s secret that contains the access token
        name: github-access
        # Key within the K8s secret whose corresponding value (must be base64 encoded) is access token
        key: token

#      # webhookSecret refers to K8s secret that stores the github hook secret
#      # +optional
#      webhookSecret:
#        # Name of the K8s secret that contains the hook secret
#        name: github-access
#        # Key within the K8s secret whose corresponding value (must be base64 encoded) is hook secret
#        key: secret

      # type of the connection between event-source and Github.
      # You should set it to false to avoid man-in-the-middle and other attacks.
      insecure: true
      # Determines if notifications are sent when the webhook is triggered
      active: true
      # The media type used to serialize the payloads
      contentType: json
      
    # begitwebhook-without-api-credentials:
    #   owner: "argoproj"
    #   repository: "argo"
    #   webhook:
    #     endpoint: "/push"
    #     port: "13000"
    #     method: "POST"
    #   events:
    #     - "*"
    #   webhookSecret:
    #     name: github-access
    #     key: secret
    #   insecure: true
    #   active: true
    #   contentType: "json"

#    example-with-secure-connection:
#      owner: "argoproj"
#      repository: "argo"
#      webhook:
#        endpoint: "/push"
#        port: "13000"
#        method: "POST"
#        url: "http://myargofakeurl.fake"
#        # k8s secret that contains the cert
#        serverCertSecret:
#          name: my-secret
#          key: cert-key
#        # k8s secret that contains the private key
#        serverKeySecret:
#          name: my-secret
#          key: pk-key
#      events:
#        - "push"
#        - "delete"
#      apiToken:
#        name: github-access
#        key: token
#      webhookSecret:
#        name: github-access
#        key: secret
#      insecure: true
#      active: true
#      contentType: "json"

Argo Workflows workflow template

在之前的例子中，我们是把 workflow 的 描述内容直接写进了 sensor

本例中将从一个 template 中引用

下面是 template 配置文件

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: buildkit
  namespace: argo
spec:
  templates:
  - name: clean
    container:
      name: ''
      image: k8s.gcr.io/busybox
      imagePullPolicy: IfNotPresent
      command:
        - /bin/sh
      args:
        - '-c'
        - 'rm -rf /work/*'
      workingDir: /work
      volumeMounts:
        - name: work
          mountPath: /work
  - name: clone
    inputs:
      artifacts:
      - name: argo-source
        path: /src
        git:
          repo: '{{workflow.parameters.repo}}'
          revision: '{{workflow.parameters.branch}}'
          usernameSecret:
            name: github-creds
            key: username
          passwordSecret:
            name: github-creds
            key: password
    container:
      image: k8s.gcr.io/busybox
      command:
        - /bin/sh
      args:
        - '-c'
        - cp -rf /src/. /work/ && ls /work
      workingDir: /src
      volumeMounts:
        - name: work
          mountPath: /work
  - name: getcommitid
    inputs: {}
    outputs: {}
    metadata: {}
    container:
      name: ''
      image: xxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/getcommitid
      command:
        - /bin/bash
      args:
        - '-c'
        - /getcommitid.sh
      workingDir: /work
      volumeMounts:
        - name: work
          mountPath: /work
  - name: mountcommitid
    outputs:
      parameters:
        - name: commitid
          valueFrom:
            path: /work/commitid.txt
            default: mockup_commitid
    metadata: {}
    container:
      name: ''
      image: k8s.gcr.io/busybox
      command:
        - /bin/sh
      args:
        - '-c'
        - ls /work
      workingDir: /work
      resources: {}
      volumeMounts:
        - name: work
          mountPath: /work
  - name: build
    container:
      name: ''
      image: 'xxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/node-alpine-3-13'
      command:
        - sh
      args:
        - -c
        - 'yarn install && yarn build'
      workingDir: /work
      env:
        - name: ENVTEST
          value: helloworld
      volumeMounts:
        - name: work
          mountPath: /work
  - name: buildbe
    container:
      name: ''
      image: 'xxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/lizhewebbemvnbuilder'
      command:
        - mvn
      args:
        - 'clean'
        - 'install'
        - 'package'
      workingDir: /work/lizheweb-server
      env:
        - name: ENVTEST
          value: helloworld
      volumeMounts:
        - name: work
          mountPath: /work
  - name: build-storybook
    container:
      name: ''
      image: 'xxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/node-alpine-3-13'
      command:
        - sh
      args:
        - -c
        - 'yarn install && yarn build-storybook'
      workingDir: /work/packages/components
      env:
        - name: ENVTEST
          value: helloworld
      volumeMounts:
        - name: work
          mountPath: /work
  - name: image
    inputs:
      parameters:
        - name: image
        - name: commitid
        - name: dockercontext
        - name: dockerfile
    container:
      name: ''
      image: gcr.io/kaniko-project/executor@sha256:f652f28537fa76e8f4f9393de13a064f0206003c451ce2ad6e4359fd5a21acbc
      args:
        - '-f'
        - '{{inputs.parameters.dockercontext}}/{{inputs.parameters.dockerfile}}'
        - '-c'
        - '{{inputs.parameters.dockercontext}}'
        - --destination={{inputs.parameters.image}}:{{inputs.parameters.commitid}}
      workingDir: /work
      env:
        - name: envkey
          value: envvalue
      resources: {}
      volumeMounts:
      - name: docker-config
        mountPath: /kaniko/.docker/
      - name: aws-secret
        mountPath: /root/.aws/
      - name: work
        mountPath: /work
    volumes:
    - name: docker-config
      configMap:
        name: docker-config
    - name: aws-secret
      secret:
        secretName: aws-secret
  - name: deploy
    inputs:
      parameters:
        - name: depfileyaml
    container:
      name: ''
      image: 'xxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/sedcommitid:latest'
      command:
        - /main.sh
      args:
        - '{{inputs.parameters.depfileyaml}}'
      workingDir: /
      env:
        - name: GIT_URL
          valueFrom:
            secretKeyRef:
              key: fecom
              name: giturl
        - name: REPO
          value: lizhe-devops
        - name: BRANCH
          value: main
        - name: USER_NAME
          value: lizhe
        - name: USER_EMAIL
          value: YOUR@EMAIL.COM
      volumeMounts:
        - name: work
          mountPath: /work
  - name: deploy-exit-handler
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["task cleanup"]
  - name: exit-handler
    inputs:
      parameters:
        - name: projectname
    container:
      image: xxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/lizhewfmailsender
      command:
        - /bin/bash
      args: 
      - -c
      - "/main {{workflow.name}} {{workflow.status}} {{workflow.failures}} /work/commitid.txt"
      workingDir: /work
      env:
        - name: projectname
          value: '{{workflow.parameters.projectname}}'
        - name: smtppass
          valueFrom:
            secretKeyRef:
              key: password
              name: smtpaccount
        - name: smtpuser
          valueFrom:
            secretKeyRef:
              key: username
              name: smtpaccount
        - name: smtphost
          value: smtp.gmail.com
        - name: smtpport
          value: "587"
        - name: sender
          value: SENDER@GMAIL.COM
        # - name: receivers
        #   value: XXX@EMAIL.COM,XXX@EMAIL.COM
        - name: smtpCc
          value: XXX@EMAIL.COM,XXX@EMAIL.COM
        - name: subject
          value: mockup_subject
        - name: body
          value: mockup_body
      volumeMounts:
        - name: work
          mountPath: /work

Argo Workflows ServiceAccount

官方文档例子中一直使用的 sa 是

serviceAccountName: operate-workflow-sa

不过如果你觉得太麻烦也可以像我一样，直接把 role 附加到 Namespace 上

下面的例子是 把 权限 role 直接绑定到了 名为 argo 的 Namespace 上

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argo-operate-workflow-role
  namespace: argo
rules:
  - apiGroups:
      - argoproj.io
    verbs:
      - "*"
    resources:
      - workflows
      - workflowtemplates
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argo-operate-workflow-role-binding
  namespace: argo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argo-operate-workflow-role
subjects:
  - kind: ServiceAccount
    name: default
    namespace: argo

Namespace 无法删除的情况

加参数

--grace-period=0 --force

如果还不行

echo $(kubectl config view --raw -oyaml | grep client-cert  |cut -d ' ' -f 6) |base64 -d > /tmp/client.pem
echo $(kubectl config view --raw -oyaml | grep client-key-data  |cut -d ' ' -f 6 ) |base64 -d > /tmp/client-key.pem
echo $(kubectl config view --raw -oyaml | grep certificate-authority-data  |cut -d ' ' -f 6  ) |base64 -d > /tmp/ca.pem

curl --cert /tmp/client.pem --key /tmp/client-key.pem --cacert /tmp/ca.pem -H "Content-Type: application/json" -X PUT --data-binary @/tmp/temp.json https://xxx.xxx.xxx.xxx:6443/api/v1/namespaces/<namespaces>/finalize

ArgoCD 添加cluster

ArgoCD 默认可以使用internal cluster，但是有的时候我们会希望通过ArgoCD来同步其他的Kubernetes集群，这个时候就需要把其他集群添加到argo中

这里使用3台机器

main 控制机，我会在这台机器上操作
argocd 是部署了argocd的机器
target 是我要同步的目标机器

在main上安装 argocd 命令行

参考 https://argoproj.github.io/argo-cd/cli_installation/

wget https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64

sudo cp argocd-linux-amd64 /usr/local/bin/argocd

sudo chmod 777 /usr/local/bin/argocd

然后通过cli 登录到 argo，这里有的博客上写的是需要连接 8083端口，实际测试只需要连接 8000的 web 端口就可以，使用nodeport 或者 port forward 都可以连接

kubectl port-forward --address 0.0.0.0 svc/argocd-server -n argocd 8000:443
argocd login localhost:8000 --grpc-web

然后通过 target 目标机上的 kubeconfig 文件来添加

argocd cluster add microk8s --kubeconfig target.yaml

添加完成之后

Juju Charmed Kubernetes

下载

https://raw.githubusercontent.com/charmed-kubernetes/bundle/master/overlays/calico-overlay.yaml

wget https://raw.githubusercontent.com/charmed-kubernetes/bundle/master/overlays/calico-overlay.yaml --no-check-certificate

文件内容是这样的

description: Charmed Kubernetes overlay to add Calico CNI.
applications:
  calico:
    annotations:
      gui-x: '480'
      gui-y: '750'
    charm: cs:~containers/calico
  flannel:
relations:
- - calico:etcd
  - etcd:db
- - calico:cni
  - kubernetes-master:cni
- - calico:cni
  - kubernetes-worker:cni

juju bootstrap

juju add-model k8s
juju deploy charmed-kubernetes

sudo snap install kubectl --classic
juju scp kubernetes-master/0:config ~/.kube/config

Kubernetes CMD的常见写法

command: ['/bin/sh']

command:  ["rm", "-fr", "/var/lib/dbs/lost+found"]

command:
- 'sh'
- '-c'
- 'pwd'

command: ["sh", "-c", "pwd; sleep 2; pwd;"]

command: ["sh"]
args: ["-c","pwd; sleep 2; pwd;"]

如果脚本很长，又想写进 yaml文件

command: ['sh']
        args:
        - "-c"
        - |
          set -ex
          if [ "$a" -eq "$b" ];then
            touch /tmp/test
          else
            echo "not eq"
          fi

Kubernetes job initcontainer

这个模型主要用来做编译部署、编译导入

init 容器可以使用 git 下载代码到 emptyDir，然后由主容器来 build

apiVersion: batch/v1
kind: Job
metadata:
  generateName: test-
  namespace: default
spec:
  backoffLimit: 6
  completions: 1
  parallelism: 1
  template:
    spec:
      initContainers:
        - name: init
          image: ubuntu
          imagePullPolicy: IfNotPresent
          command: ["touch","/tmp/test.txt"]
          volumeMounts:
            - name: mytmp
              mountPath: "/tmp"
      containers: 
        - name: test
          image: ubuntu
          command: ["ls", "/tmp"]
          volumeMounts:
            - name: mytmp
              mountPath: "/tmp"
          imagePullPolicy: IfNotPresent
      dnsPolicy: ClusterFirst
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: mytmp
        emptyDir: {}