Kaniko on Kubernetes Jenkins

kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.

kaniko doesn’t depend on a Docker daemon and executes each command within a Dockerfile completely in userspace. This enables building container images in environments that can’t easily or securely run a Docker daemon, such as a standard Kubernetes cluster.

kaniko is meant to be run as an image: gcr.io/kaniko-project/executor. We do not recommend running the kaniko executor binary in another image, as it might not work.

https://github.com/GoogleContainerTools/kaniko

在之前安装的jenkins上配置 cloud，添加kubernetes，测试连接会得到以下错误

提示没有足够的权限来操作pods

这里我们需要添加RBAC权限配置，可以参考 Api server 认证机制

kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --group=system:serviceaccounts:jenkins

创建RBAC权限之后，jenkins有了集群的管理权限

创建jenkins从 git 上 clone 代码用的账户

因为 kaniko 会直接将build好的 image push 到明明的目标仓库，所以这里需要创建 dockerhub 使用的regcred

kubectl create secret docker-registry regcred --docker-server=https://index.docker.io/v1/ --docker-username=YOURNAME --docker-password=YOURPASSWORD --docker-email=YOUREMAIL@ADDRESS -n jenkins

创建一个叫 helloworld 的 pipeline job

def label = "mypod-${UUID.randomUUID().toString()}"
podTemplate(label: label, yaml: """
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    command:
    - /busybox/cat
    tty: true
    volumeMounts:
      - name: jenkins-docker-cfg
        mountPath: /root
  volumes:
  - name: jenkins-docker-cfg
    projected:
      sources:
      - secret:
          name: regcred
          items:
            - key: .dockerconfigjson
              path: .docker/config.json
"""
) {
    node(label) {
    stage('Build with Kaniko') {
      sh label: '', script: 'free -m'
      timeout(time: 2, unit: 'HOURS'){
        dir('sourcecodes') {
          git credentialsId: '0559259b-d672-457f-8d2e-d3925c412d70', url: 'https://github.com/zl86790/kanikotest.git'
        }
      }


      container(name: 'kaniko', shell: '/busybox/sh') {
        withEnv(['PATH+EXTRA=/busybox:/kaniko']) {
          sh '''#!/busybox/sh
          pwd
          ls /home/jenkins/agent/workspace/helloworld/sourcecodes/
          /kaniko/executor -f /home/jenkins/agent/workspace/helloworld/sourcecodes/Dockerfile -c `pwd` --destination=libaibai/kanikotest:test0.1
          '''
        }
      }
    }
  }
}

尝试启动一个job

出现无法reach到服务器的错误

tcpSlaveAgentListener/: Host is unreachable (Host unreachable)

这里主要是我的 k8s2.me 是一个 /etc/hosts 填写的本地DNS，所以解析不到，换成clusterip

再次尝试启动 job

提示 tcpSlaveAgentListener 404 找不到

java.io.IOException: http://10.43.113.251:8080/tcpSlaveAgentListener/ is invalid: 404 Not Found

然后我们去打开这个 agent

再次启动job

还是错误，端口50000 连不上

SEVERE: http://10.43.113.251:8080/ provided port:50000 is not reachable

java.io.IOException: http://10.43.113.251:8080/ provided port:50000 is not reachable

这个错误就比较容易处理了，在 clusterip 的 service 上，把50000端口暴露出来

再次启动job，终于成功了

在job执行结束后，可以看到 pod 被remove掉了

可以看到image已经上传成功了