Kaniko on Kubernetes Jenkins
kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.
kaniko doesn’t depend on a Docker daemon and executes each command within a Dockerfile completely in userspace. This enables building container images in environments that can’t easily or securely run a Docker daemon, such as a standard Kubernetes cluster.
kaniko is meant to be run as an image: gcr.io/kaniko-project/executor. We do not recommend running the kaniko executor binary in another image, as it might not work.
https://github.com/GoogleContainerTools/kaniko
在之前安装的jenkins上配置 cloud，添加kubernetes，测试连接会得到以下错误
提示没有足够的权限来操作pods
这里我们需要添加RBAC权限配置，可以参考 Api server 认证机制
kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --group=system:serviceaccounts:jenkins
创建RBAC权限之后，jenkins有了集群的管理权限
创建jenkins从 git 上 clone 代码用的账户
因为 kaniko 会直接将build好的 image push 到明明的目标仓库，所以这里需要创建 dockerhub 使用的regcred
kubectl create secret docker-registry regcred --docker-server=https://index.docker.io/v1/ --docker-username=YOURNAME --docker-password=YOURPASSWORD --docker-email=YOUREMAIL@ADDRESS -n jenkins创建一个叫 helloworld 的 pipeline job
def label = "mypod-${UUID.randomUUID().toString()}"
podTemplate(label: label, yaml: """
kind: Pod
metadata:
name: kaniko
spec:
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:debug
imagePullPolicy: Always
command:
- /busybox/cat
tty: true
volumeMounts:
- name: jenkins-docker-cfg
mountPath: /root
volumes:
- name: jenkins-docker-cfg
projected:
sources:
- secret:
name: regcred
items:
- key: .dockerconfigjson
path: .docker/config.json
"""
) {
node(label) {
stage('Build with Kaniko') {
sh label: '', script: 'free -m'
timeout(time: 2, unit: 'HOURS'){
dir('sourcecodes') {
git credentialsId: '0559259b-d672-457f-8d2e-d3925c412d70', url: 'https://github.com/zl86790/kanikotest.git'
}
}
container(name: 'kaniko', shell: '/busybox/sh') {
withEnv(['PATH+EXTRA=/busybox:/kaniko']) {
sh '''#!/busybox/sh
pwd
ls /home/jenkins/agent/workspace/helloworld/sourcecodes/
/kaniko/executor -f /home/jenkins/agent/workspace/helloworld/sourcecodes/Dockerfile -c `pwd` --destination=libaibai/kanikotest:test0.1
'''
}
}
}
}
}尝试启动一个job
出现无法reach到服务器的错误
tcpSlaveAgentListener/: Host is unreachable (Host unreachable)这里主要是我的 k8s2.me 是一个 /etc/hosts 填写的本地DNS，所以解析不到，换成clusterip
再次尝试启动 job
提示 tcpSlaveAgentListener 404 找不到
java.io.IOException: http://10.43.113.251:8080/tcpSlaveAgentListener/ is invalid: 404 Not Found
然后我们去打开这个 agent
再次启动job
还是错误，端口50000 连不上
SEVERE: http://10.43.113.251:8080/ provided port:50000 is not reachable
java.io.IOException: http://10.43.113.251:8080/ provided port:50000 is not reachable
这个错误就比较容易处理了，在 clusterip 的 service 上，把50000端口暴露出来
再次启动job，终于成功了
在job执行结束后，可以看到 pod 被remove掉了
可以看到image已经上传成功了
