refer
https://rancher.com/docs/rancher/v2.x/en/installation/install-rancher-on-k8s/amazon-eks/
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --install \
ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--set controller.service.type=LoadBalancer \
--version 3.12.0 \
--create-namespacekubectl get service ingress-nginx-controller --namespace=ingress-nginxEFK 是大而全，Promtail + Loki + Grafana 是小而精
Loki 采用对日志进行打标签的方式而非全文索引的方式，从而大大降低了对服务器资源的依赖
下面是两者的区别
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
kubectl create namespace loki
helm upgrade --install loki --namespace=loki grafana/loki-stack --set grafana.enabled=true如果需要持久化
参考https://grafana.com/docs/loki/next/installation/helm/
helm upgrade --install loki --namespace=loki grafana/loki-stack --set grafana.enabled=true,loki.persistence.enabled=true,loki.persistence.storageClassName=standard,loki.persistence.size=200Gi
helm upgrade --install loki --namespace=loki grafana/loki-stack --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false,loki.persistence.enabled=true,loki.persistence.storageClassName=gp2,loki.persistence.size=200Gi
grafana.enabled=true
prometheus.enabled=true
prometheus.alertmanager.persistentVolume.enabled=false
prometheus.server.persistentVolume.enabled=false
loki.persistence.enabled=true
loki.persistence.storageClassName=gp2
loki.persistence.size=200Gi
$ kubectl get secret --namespace loki loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
kubectl port-forward --namespace loki service/loki-grafana 3000:80
如果使用的是 rancher
kubectl port-forward --namespace loki-stack service/loki-stack-grafana 3000:80
可以看到已经自动添加了 data source loki
curl https://get.datree.io | /bin/bash
我们来使用datree来测试下面的yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
lifecycle:
postStart:
exec:
command:
- sh
- -c
- date "+%Y-%m-%d %H:%M:%S" > /datetmp.txt && sleep 30
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 30
readinessProbe:
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 30datree test demo.yaml
❌ Ensure Deployment has more than one replica configured [1 occurrences]
💡 Incorrect value for key `replicas` - running 2 or more replicas will increase the availability of the service
我们的应用没有使用 HA
❌ Ensure each container image has a pinned (tag) version [1 occurrences]
💡 Incorrect value for key `image` - specify an image version to avoid unpleasant "version surprises" in the future
我们使用了 latest 作为 tag
❌ Ensure each container has a configured CPU limit [1 occurrences]
💡 Missing property object `limits.cpu` - value should be within the accepted boundaries recommended by the organization
我们没有设定 cpu limit
❌ Ensure each container has a configured CPU request [1 occurrences]
💡 Missing property object `requests.cpu` - value should be within the accepted boundaries recommended by the organization
没有设定 cpu request
❌ Ensure each container has a configured memory request [1 occurrences]
💡 Missing property object `requests.memory` - value should be within the accepted boundaries recommended by the organization
没有设定 内存 request
❌ Ensure each container has a configured memory limit [1 occurrences]
💡 Missing property object `limits.memory` - value should be within the accepted boundaries recommended by the organization
没有设定内存 limit如果需要一次性检查多个文件，可以直接使用文件夹
默认情况下，它只有21条（?) rule
Each CLI invocation is running a default policy that includes 21 built-in rules.
可以通过 cliId 登录来重新配置
参考 https://argoproj.github.io/argo-rollouts/getting-started/
安装
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
安装dashboard
参考 https://argoproj.github.io/argo-rollouts/dashboard/
argo rollouts 提供了一个 kubectl plugin，先安装这个plugin
brew install argoproj/tap/kubectl-argo-rollouts
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-darwin-amd64插件是一个独立的可执行文件，名称以 kubectl- 开头。 要安装插件，将其可执行文件移动到 PATH 中的任何位置。
lizhe@ubuntu:~$ curl -LO https://github.com/argoproj/argo-rollouts/releases/download/v1.0.2/kubectl-argo-rollouts-linux-amd64
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 641 100 641 0 0 679 0 --:--:-- --:--:-- --:--:-- 679
100 61.3M 100 61.3M 0 0 115k 0 0:09:05 0:09:05 --:--:-- 87576
lizhe@ubuntu:~$ chmod +x ./kubectl-argo-rollouts-linux-amd64
lizhe@ubuntu:~$ sudo mv ./kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts
lizhe@ubuntu:~$ kubectl argo rollouts version
kubectl-argo-rollouts: v1.0.2+7a23fe5
BuildDate: 2021-06-15T19:36:00Z
GitCommit: 7a23fe5dbf78181248c48af8e5224246434e7f99
GitTreeState: clean
GoVersion: go1.16.3
Compiler: gc
Platform: linux/amd64
lizhe@ubuntu:~$
使用plugin启动dashboard
下面是两个官方例子
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/basic/rollout.yaml
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/basic/service.yamlrollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: rollouts-demo
spec:
replicas: 5
strategy:
canary:
steps:
- setWeight: 20
- pause: {}
- setWeight: 40
- pause: {duration: 10}
- setWeight: 60
- pause: {duration: 10}
- setWeight: 80
- pause: {duration: 10}
revisionHistoryLimit: 2
selector:
matchLabels:
app: rollouts-demo
template:
metadata:
labels:
app: rollouts-demo
spec:
containers:
- name: rollouts-demo
image: argoproj/rollouts-demo:blue
ports:
- name: http
containerPort: 8080
protocol: TCP
resources:
requests:
memory: 32Mi
cpu: 5mservice.yaml
apiVersion: v1
kind: Service
metadata:
name: rollouts-demo
spec:
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: rollouts-demo
根据官方文档的说法
Initial creations of any Rollout will immediately scale up the replicas to 100% (skipping any canary upgrade steps, analysis, etc…) since there was no upgrade that occurred.
在初次安装时，所有部署都会直接 100%，因为不需要 upgrade
进行更新 blue -> yellow
kubectl argo rollouts set image rollouts-demo \
rollouts-demo=argoproj/rollouts-demo:yellow在更新过程中，根据规则
更新20% 之后，它会暂停，直到用户进行unpause/promote
以下命令可以让你切换到其他 namespace
kubectl argo rollouts dashboard -n lizhe
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/basic/rollout.yaml -n lizhe
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/basic/service.yaml -n lizhe
不过上面的过程中会留下 replica = 0 的一些垃圾 ReplicaSet
如果你手动删除它们，会造成 argo rollouts 丢失历史记录
完整的配置文件请参考
https://argoproj.github.io/argo-rollouts/features/specification/
这里提供了 revisionHistoryLimit: 3 ， 我们的配置文件里这里是 revisionHistoryLimit: 3
也就是说，1 正在运行的记录，2 replica = 0 的历史记录一共3个，我们试着增加几次部署
可以看到最早创建的 purple 版本被移除了
参考官方
https://kubernetes.io/zh/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/
这个例子是一个最简单的helloworld 例子，并不涉及编码内容，但是可以用来理解CRD是如何工作的
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
# 名字必需与下面的 spec 字段匹配，并且格式为 '<名称的复数形式>.<组名>'
name: crontabs.stable.example.com
spec:
# 组名称，用于 REST API: /apis/<组>/<版本>
group: stable.example.com
# 列举此 CustomResourceDefinition 所支持的版本
versions:
- name: v1
# 每个版本都可以通过 served 标志来独立启用或禁止
served: true
# 其中一个且只有一个版本必需被标记为存储版本
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
image:
type: string
replicas:
type: integer
# 可以是 Namespaced 或 Cluster
scope: Namespaced
names:
# 名称的复数形式，用于 URL：/apis/<组>/<版本>/<名称的复数形式>
plural: crontabs
# 名称的单数形式，作为命令行使用时和显示时的别名
singular: crontab
# kind 通常是单数形式的驼峰编码（CamelCased）形式。你的资源清单会使用这一形式。
kind: CronTab
# shortNames 允许你在命令行使用较短的字符串来匹配资源
shortNames:
- ct
经过上面的步骤我们会得到一个新的服务端点在
/apis/stable.example.com/v1/namespaces/*/crontabs/...创建CronTab定制对象
my-crontab.yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
cronSpec: "*/60 * * * *"
image: my-awesome-cron-image
创建一个Argo cd 应用，每当yaml有新提交时，更新kubernetes集群
这一步实际上是比较容易的
创建app
开启自动同步
提交更新
注意这里的 commitid
然后会看到启动了一个新的build
build 结束之后可以看到新的 镜像 已经提交了
同时可以看到 yaml 文件的仓库也被更新了
构建镜像之后，更新yaml repository触发 argo cd
核心脚本只有一个
cat /work/commitid.txt | xargs -i sed 's/:[0-9A-Za-z]\{40,40\}$/:{}/' /argogithubsource_cd/argoapp.yaml我们创建一个 main.sh
#!/bin/bash
git config --global user.email $USER_EMAIL
git config --global user.name $USER_NAME
echo $GIT_URL
echo $REPO
echo $BRANCH
rm -rf /work/$REPO
git clone -b $BRANCH $GIT_URL
pwd
echo "ls /"
ls /
echo "ls REPO"
ls $REPO
echo "============"/$REPO/$1"================"
cat /$REPO/$1
echo "============/work/commitid.txt================"
cat /work/commitid.txt
cat /work/commitid.txt | xargs -i sed -i 's/:[0-9A-Za-z]\{40,40\}$/:{}/' /$REPO/$1
cat /work/commitid.txt | xargs -i sed -i 's/:mockup_tag_name/:{}/' /$REPO/$1
echo "============"/$REPO/$1"================"
cat /$REPO/$1
cd /$REPO && git add ./*
cd /$REPO && cat /work/commitid.txt | xargs -i git commit -m {} ./*
cd /$REPO && git push其中主要的配置
5个环境变量
$GIT_URL = https://USERNAME:PASSWORD@github.com/zl86790/argogithubsource_cd.git
$REPO = argogithubsource_cd
$BRANCH = main
$USER_EMAIL = YOUREMAIL@google.com
$USER_NAME = YOURNAME
一个参数
$1 = argoapp.yaml在提交了新代码之后，这个容器会使用上面的脚本根据前面步骤生成的 commitid，自动更新 deply的git仓库，修改 应用的 tag name，并且提交到部署代码仓库
最后通过 deploy 步骤进行部署
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
name: github
namespace: argo
spec:
template:
serviceAccountName: operate-workflow-sa
dependencies:
- name: test-dep
eventSourceName: github
eventName: gitwebhook
triggers:
- template:
name: argo-workflow-trigger
argoWorkflow:
group: argoproj.io
version: v1alpha1
resource: workflows
operation: submit
source:
resource:
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
generateName: buildkit-
namespace: argo
spec:
templates:
- name: main
dag:
tasks:
- name: clone
template: clone
arguments:
parameters:
- name: repo
value: '{{workflow.parameters.repo}}'
- name: branch
value: '{{workflow.parameters.branch}}'
- name: getcommitid
template: getcommitid
arguments: {}
depends: clone
- name: mountcommitid
template: mountcommitid
arguments: {}
depends: getcommitid
- name: build
template: build
arguments: {}
depends: mountcommitid
- name: image
template: image
arguments:
parameters:
- name: image
value: '{{workflow.parameters.image}}'
- name: commitid
value: '{{tasks.mountcommitid.outputs.parameters.commitid}}'
depends: build
- name: deploy
template: deploy
arguments: {}
depends: image
- name: clone
inputs:
parameters:
- name: repo
- name: branch
container:
name: ''
image: 'alpine/git:v2.26.2'
args:
- clone
- '--depth'
- '1'
- '--branch'
- '{{inputs.parameters.branch}}'
- '--single-branch'
- '{{inputs.parameters.repo}}'
- .
workingDir: /work
volumeMounts:
- name: work
mountPath: /work
- name: getcommitid
inputs: {}
outputs: {}
metadata: {}
container:
name: ''
image: libaibai/getcommitid
command:
- /bin/bash
args:
- '-c'
- /getcommitid.sh
workingDir: /work
volumeMounts:
- name: work
mountPath: /work
- name: mountcommitid
outputs:
parameters:
- name: commitid
valueFrom:
path: /work/commitid.txt
default: mockup_commitid
metadata: {}
container:
name: ''
image: busybox
command:
- /bin/sh
args:
- '-c'
- ls /work
workingDir: /work
resources: {}
volumeMounts:
- name: work
mountPath: /work
- name: build
container:
name: ''
image: 'golang:alpine3.13'
command:
- go
args:
- build
- main.go
workingDir: /work
env:
- name: ENVTEST
value: helloworld
volumeMounts:
- name: work
mountPath: /work
- name: image
inputs:
parameters:
- name: image
- name: commitid
container:
name: ''
image: >-
gcr.io/kaniko-project/executor@sha256:f652f28537fa76e8f4f9393de13a064f0206003c451ce2ad6e4359fd5a21acbc
command:
- /kaniko/executor
args:
- '-f'
- /work/Dockerfile
- '-c'
- /work/
- >-
--destination={{inputs.parameters.image}}:{{inputs.parameters.commitid}}
workingDir: /work
env:
- name: envkey
value: envvalue
resources: {}
volumeMounts:
- name: work
mountPath: /work
- name: jenkins-docker-cfg
mountPath: /root
volumes:
- name: jenkins-docker-cfg
projected:
sources:
- secret:
name: regcred
items:
- key: .dockerconfigjson
path: .docker/config.json
- name: deploy
container:
name: ''
image: 'libaibai/git_sed'
command:
- /main.sh
args:
- argoapp.yaml
workingDir: /
env:
- name: GIT_URL
value: https://zl86790:75755618a@github.com/zl86790/argogithubsource_cd.git
- name: REPO
value: argogithubsource_cd
volumeMounts:
- name: work
mountPath: /work
entrypoint: main
arguments:
parameters:
- name: repo
value: 'https://username:password@github.com/zl86790/argogithubsource.git'
- name: branch
value: master
- name: image
value: libaibai/argogolanghelloworld
volumeClaimTemplates:
- metadata:
name: work
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: local-path
status: {}
ttlStrategy:
secondsAfterCompletion: 1800
secondsAfterSuccess: 1800
secondsAfterFailure: 1800构建用的argo workflow下载代码并且进行build image
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
name: github
namespace: argo
spec:
template:
serviceAccountName: operate-workflow-sa
dependencies:
- name: test-dep
eventSourceName: github
eventName: gitwebhook
triggers:
- template:
name: argo-workflow-trigger
argoWorkflow:
group: argoproj.io
version: v1alpha1
resource: workflows
operation: submit
source:
resource:
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
generateName: buildkit-
namespace: argo
spec:
templates:
- name: main
dag:
tasks:
- name: clone
template: clone
arguments:
parameters:
- name: repo
value: '{{workflow.parameters.repo}}'
- name: branch
value: '{{workflow.parameters.branch}}'
- name: getcommitid
template: getcommitid
arguments: {}
depends: clone
- name: mountcommitid
template: mountcommitid
arguments: {}
depends: getcommitid
- name: build
template: build
arguments: {}
depends: mountcommitid
- name: image
template: image
arguments:
parameters:
- name: image
value: '{{workflow.parameters.image}}'
- name: commitid
value: '{{tasks.mountcommitid.outputs.parameters.commitid}}'
depends: build
- name: clone
inputs:
parameters:
- name: repo
- name: branch
container:
name: ''
image: 'alpine/git:v2.26.2'
args:
- clone
- '--depth'
- '1'
- '--branch'
- '{{inputs.parameters.branch}}'
- '--single-branch'
- '{{inputs.parameters.repo}}'
- .
workingDir: /work
volumeMounts:
- name: work
mountPath: /work
- name: getcommitid
inputs: {}
outputs: {}
metadata: {}
container:
name: ''
image: libaibai/getcommitid
command:
- /bin/bash
args:
- '-c'
- /getcommitid.sh
workingDir: /work
volumeMounts:
- name: work
mountPath: /work
- name: mountcommitid
outputs:
parameters:
- name: commitid
valueFrom:
path: /work/commitid.txt
default: mockup_commitid
metadata: {}
container:
name: ''
image: busybox
command:
- /bin/sh
args:
- '-c'
- ls /work
workingDir: /work
resources: {}
volumeMounts:
- name: work
mountPath: /work
- name: build
container:
name: ''
image: 'golang:alpine3.13'
command:
- go
args:
- build
- main.go
workingDir: /work
env:
- name: ENVTEST
value: helloworld
volumeMounts:
- name: work
mountPath: /work
- name: image
inputs:
parameters:
- name: image
- name: commitid
container:
name: ''
image: >-
gcr.io/kaniko-project/executor@sha256:f652f28537fa76e8f4f9393de13a064f0206003c451ce2ad6e4359fd5a21acbc
command:
- /kaniko/executor
args:
- '-f'
- /work/Dockerfile
- '-c'
- /work/
- >-
--destination={{inputs.parameters.image}}:{{inputs.parameters.commitid}}
workingDir: /work
env:
- name: envkey
value: envvalue
resources: {}
volumeMounts:
- name: work
mountPath: /work
- name: jenkins-docker-cfg
mountPath: /root
volumes:
- name: jenkins-docker-cfg
projected:
sources:
- secret:
name: regcred
items:
- key: .dockerconfigjson
path: .docker/config.json
entrypoint: main
arguments:
parameters:
- name: repo
value: 'https://USERNAME:PASSWORD@github.com/zl86790/argogithubsource.git'
- name: branch
value: master
- name: image
value: libaibai/argogolanghelloworld
volumeClaimTemplates:
- metadata:
name: work
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: local-path
status: {}
ttlStrategy:
secondsAfterCompletion: 1800
secondsAfterSuccess: 1800
secondsAfterFailure: 18002. 检查sensor
这是错误的情况
下面是正确的
3. Push 新 commit
Argo event source github 使用webhook监听github提交，触发构建workflow
2. github token https://github.com/settings/tokens
3. 把token写入 secret
4. 创建Event source
# Info on GitHub Webhook: https://developer.github.com/v3/repos/hooks/#create-a-hook
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: github
namespace: argo
spec:
service:
ports:
- port: 12000
targetPort: 12000
github:
gitwebhook:
repositories:
- owner: zl86790
names:
- argogithubsource
# Github will send events to following port and endpoint
webhook:
# endpoint to listen to events on
endpoint: /push
# port to run internal HTTP server on
port: "12000"
# HTTP request method to allow. In this case, only POST requests are accepted
method: POST
# url the event-source will use to register at Github.
# This url must be reachable from outside the cluster.
# The name for the service is in `<event-source-name>-eventsource-svc` format.
# You will need to create an Ingress or Openshift Route for the event-source service so that it can be reached from GitHub.
url: http://xxx.xxx.xxx.xxx:30001
# type of events to listen to.
# following listens to everything, hence *
# You can find more info on https://developer.github.com/v3/activity/events/types/
events:
- "*"
# apiToken refers to K8s secret that stores the github api token
# if apiToken is provided controller will create webhook on GitHub repo
# +optional
apiToken:
# Name of the K8s secret that contains the access token
name: github-access
# Key within the K8s secret whose corresponding value (must be base64 encoded) is access token
key: token
# # webhookSecret refers to K8s secret that stores the github hook secret
# # +optional
# webhookSecret:
# # Name of the K8s secret that contains the hook secret
# name: github-access
# # Key within the K8s secret whose corresponding value (must be base64 encoded) is hook secret
# key: secret
# type of the connection between event-source and Github.
# You should set it to false to avoid man-in-the-middle and other attacks.
insecure: true
# Determines if notifications are sent when the webhook is triggered
active: true
# The media type used to serialize the payloads
contentType: json
gitwebhook-without-api-credentials:
owner: "argoproj"
repository: "argo"
webhook:
endpoint: "/push"
port: "13000"
method: "POST"
events:
- "*"
webhookSecret:
name: github-access
key: secret
insecure: true
active: true
contentType: "json"
# gitwebhook-with-secure-connection:
# owner: "argoproj"
# repository: "argo"
# webhook:
# endpoint: "/push"
# port: "13000"
# method: "POST"
# url: "http://myargofakeurl.fake"
# # k8s secret that contains the cert
# serverCertSecret:
# name: my-secret
# key: cert-key
# # k8s secret that contains the private key
# serverKeySecret:
# name: my-secret
# key: pk-key
# events:
# - "push"
# - "delete"
# apiToken:
# name: github-access
# key: token
# webhookSecret:
# name: github-access
# key: secret
# insecure: true
# active: true
# contentType: "json"5. 检查
