Month: April 2021

Kubernetes Role & ClusterRole

Kubernetes 的 Role 是一组权限的集合，表明了拥有这个集合的Subject（Kubernetes User Account、Kubernetes Service Account、Group）可以做什么

仅仅是 允许，Role不包含 不允许，也就是绑定了这个role的subject可以做什么

例如 允许在特定的namespace （pod-role namespace）下，列出pod的对应权限集合

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

与普通的Role不同，还有一种 可以在任何 namespace 下使用的权限集合

以下这种表示 任意namespace下，可以列出的pod的权限集合

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

Kubernetes 拥有大量内置的集群role

calico-node                                                            2021-10-09T13:06:48Z
cattle-fleet-system-fleet-agent-role                                   2021-10-09T13:21:08Z
cattle-globalrole-admin                                                2021-10-09T13:19:47Z
cattle-globalrole-authn-manage                                         2021-10-09T13:19:47Z
cattle-globalrole-catalogs-manage                                      2021-10-09T13:19:47Z
cattle-globalrole-catalogs-use                                         2021-10-09T13:19:47Z
cattle-globalrole-clusters-create                                      2021-10-09T13:19:47Z
cattle-globalrole-clustertemplaterevisions-create                      2021-10-09T13:19:47Z
cattle-globalrole-clustertemplates-create                              2021-10-09T13:19:47Z
cattle-globalrole-features-manage                                      2021-10-09T13:19:47Z
cattle-globalrole-kontainerdrivers-manage                              2021-10-09T13:19:47Z
cattle-globalrole-nodedrivers-manage                                   2021-10-09T13:19:47Z
cattle-globalrole-podsecuritypolicytemplates-manage                    2021-10-09T13:19:47Z
cattle-globalrole-restricted-admin                                     2021-10-09T13:19:47Z
cattle-globalrole-roles-manage                                         2021-10-09T13:19:47Z
cattle-globalrole-settings-manage                                      2021-10-09T13:19:47Z
cattle-globalrole-user                                                 2021-10-09T13:19:47Z
cattle-globalrole-user-base                                            2021-10-09T13:19:47Z
cattle-globalrole-users-manage                                         2021-10-09T13:19:47Z
cattle-globalrole-view-rancher-metrics                                 2021-10-09T13:19:47Z
cattle-impersonation-u-at6ks67sqf                                      2021-10-09T13:20:18Z
cattle-impersonation-u-b4qkhsnliz                                      2021-10-09T13:20:16Z
cattle-impersonation-u-mo773yttt4                                      2021-10-09T13:22:27Z
cattle-impersonation-u-oz75ayhmkg                                      2021-10-09T13:20:19Z
cattle-unauthenticated                                                 2021-10-09T13:19:35Z
cert-manager-cainjector                                                2021-10-09T13:14:13Z
cert-manager-controller-approve:cert-manager-io                        2021-10-09T13:14:13Z
cert-manager-controller-certificates                                   2021-10-09T13:14:13Z
cert-manager-controller-certificatesigningrequests                     2021-10-09T13:14:13Z
cert-manager-controller-challenges                                     2021-10-09T13:14:13Z
cert-manager-controller-clusterissuers                                 2021-10-09T13:14:13Z
cert-manager-controller-ingress-shim                                   2021-10-09T13:14:13Z
cert-manager-controller-issuers                                        2021-10-09T13:14:13Z
cert-manager-controller-orders                                         2021-10-09T13:14:13Z
cert-manager-edit                                                      2021-10-09T13:14:13Z
cert-manager-view                                                      2021-10-09T13:14:13Z
cert-manager-webhook:subjectaccessreviews                              2021-10-09T13:14:13Z
cluster-admin                                                          2021-10-09T13:05:55Z
cluster-crd-clusterRole                                                2021-10-09T13:19:37Z
cluster-owner                                                          2021-10-09T13:20:16Z
create-ns                                                              2021-10-09T13:20:20Z
edit                                                                   2021-10-09T13:05:55Z
flannel                                                                2021-10-09T13:06:48Z
fleet-bundle-deployment                                                2021-10-09T13:20:45Z
fleet-content                                                          2021-10-09T13:20:45Z
fleet-controller                                                       2021-10-09T13:20:31Z
fleet-controller-bootstrap                                             2021-10-09T13:20:31Z
fleetworkspace-admin                                                   2021-10-09T13:19:35Z
fleetworkspace-member                                                  2021-10-09T13:19:35Z
fleetworkspace-readonly                                                2021-10-09T13:19:35Z
gitjob                                                                 2021-10-09T13:20:31Z
global-unrestricted-psp-clusterrole                                    2021-10-09T13:05:56Z
local-clustermember                                                    2021-10-09T13:20:16Z
local-clusterowner                                                     2021-10-09T13:20:13Z
p-b5lm7-namespaces-edit                                                2021-10-09T13:20:09Z
p-b5lm7-namespaces-readonly                                            2021-10-09T13:20:09Z
p-qpk2x-namespaces-edit                                                2021-10-09T13:20:09Z
p-qpk2x-namespaces-readonly                                            2021-10-09T13:20:09Z
project-crd-clusterRole                                                2021-10-09T13:19:37Z
project-member                                                         2021-10-09T13:20:15Z
project-member-promoted                                                2021-10-09T13:20:19Z
rke2-cloud-controller-manager                                          2021-10-09T13:05:57Z
rke2-coredns-rke2-coredns                                              2021-10-09T13:06:48Z
rke2-coredns-rke2-coredns-autoscaler                                   2021-10-09T13:06:48Z
rke2-ingress-nginx                                                     2021-10-09T13:07:47Z
system-unrestricted-psp-role                                           2021-10-09T13:05:56Z
system:aggregate-to-admin                                              2021-10-09T13:05:55Z
system:aggregate-to-edit                                               2021-10-09T13:05:55Z
system:aggregate-to-view                                               2021-10-09T13:05:55Z
system:auth-delegator                                                  2021-10-09T13:05:55Z
system:basic-user                                                      2021-10-09T13:05:55Z
system:certificates.k8s.io:certificatesigningrequests:nodeclient       2021-10-09T13:05:55Z
system:certificates.k8s.io:certificatesigningrequests:selfnodeclient   2021-10-09T13:05:55Z
system:certificates.k8s.io:kube-apiserver-client-approver              2021-10-09T13:05:55Z
system:certificates.k8s.io:kube-apiserver-client-kubelet-approver      2021-10-09T13:05:55Z
system:certificates.k8s.io:kubelet-serving-approver                    2021-10-09T13:05:55Z
system:certificates.k8s.io:legacy-unknown-approver                     2021-10-09T13:05:55Z
system:controller:attachdetach-controller                              2021-10-09T13:05:55Z
system:controller:certificate-controller                               2021-10-09T13:05:56Z
system:controller:clusterrole-aggregation-controller                   2021-10-09T13:05:55Z
system:controller:cronjob-controller                                   2021-10-09T13:05:55Z
system:controller:daemon-set-controller                                2021-10-09T13:05:55Z
system:controller:deployment-controller                                2021-10-09T13:05:55Z
system:controller:disruption-controller                                2021-10-09T13:05:55Z
system:controller:endpoint-controller                                  2021-10-09T13:05:55Z
system:controller:endpointslice-controller                             2021-10-09T13:05:55Z
system:controller:endpointslicemirroring-controller                    2021-10-09T13:05:55Z
system:controller:ephemeral-volume-controller                          2021-10-09T13:05:56Z
system:controller:expand-controller                                    2021-10-09T13:05:55Z
system:controller:generic-garbage-collector                            2021-10-09T13:05:56Z
system:controller:horizontal-pod-autoscaler                            2021-10-09T13:05:56Z
system:controller:job-controller                                       2021-10-09T13:05:56Z
system:controller:namespace-controller                                 2021-10-09T13:05:56Z
system:controller:node-controller                                      2021-10-09T13:05:56Z
system:controller:persistent-volume-binder                             2021-10-09T13:05:56Z
system:controller:pod-garbage-collector                                2021-10-09T13:05:56Z
system:controller:pv-protection-controller                             2021-10-09T13:05:56Z
system:controller:pvc-protection-controller                            2021-10-09T13:05:56Z
system:controller:replicaset-controller                                2021-10-09T13:05:56Z
system:controller:replication-controller                               2021-10-09T13:05:56Z
system:controller:resourcequota-controller                             2021-10-09T13:05:56Z
system:controller:root-ca-cert-publisher                               2021-10-09T13:05:56Z
system:controller:route-controller                                     2021-10-09T13:05:56Z
system:controller:service-account-controller                           2021-10-09T13:05:56Z
system:controller:service-controller                                   2021-10-09T13:05:56Z
system:controller:statefulset-controller                               2021-10-09T13:05:56Z
system:controller:ttl-after-finished-controller                        2021-10-09T13:05:56Z
system:controller:ttl-controller                                       2021-10-09T13:05:56Z
system:discovery                                                       2021-10-09T13:05:55Z
system:heapster                                                        2021-10-09T13:05:55Z
system:kube-aggregator                                                 2021-10-09T13:05:55Z
system:kube-controller-manager                                         2021-10-09T13:05:55Z
system:kube-dns                                                        2021-10-09T13:05:55Z
system:kube-proxy                                                      2021-10-09T13:05:56Z
system:kube-scheduler                                                  2021-10-09T13:05:55Z
system:kubelet-api-admin                                               2021-10-09T13:05:55Z
system:monitoring                                                      2021-10-09T13:05:55Z
system:node                                                            2021-10-09T13:05:55Z
system:node-bootstrapper                                               2021-10-09T13:05:55Z
system:node-problem-detector                                           2021-10-09T13:05:55Z
system:node-proxier                                                    2021-10-09T13:05:55Z
system:persistent-volume-provisioner                                   2021-10-09T13:05:55Z
system:public-info-viewer                                              2021-10-09T13:05:55Z
system:rke2-controller                                                 2021-10-09T13:05:56Z
system:rke2-metrics-server                                             2021-10-09T13:07:39Z
system:rke2-metrics-server-aggregated-reader                           2021-10-09T13:07:39Z
system:service-account-issuer-discovery                                2021-10-09T13:05:55Z
system:volume-scheduler                                                2021-10-09T13:05:55Z
u-at6ks67sqf-view                                                      2021-10-09T13:20:14Z
u-b4qkhsnliz-view                                                      2021-10-09T13:20:11Z
u-mo773yttt4-view                                                      2021-10-09T13:22:27Z
u-oz75ayhmkg-view                                                      2021-10-09T13:20:14Z
user-whnpx-view                                                        2021-10-09T13:19:48Z
view                                                                   2021-10-09T13:05:55Z

这些 role 大多数面向的是系统实用程序， 但其中有四个是针对一般用户的

cluster-admin: 提供完整的集群访问权限
admin: 提供整个命名空间完整的访问权限
edit: 允许用户修改命名空间中的一切
view: 提供命名空间只读权限

Kubernetes Service Account

在 Kubernetes User Account 中我们介绍了如何创建一个用户，用户就是 User Account，是给外部用户使用的

这里的 Service Account ，是给 Pod 使用的

基于 Service Account ， 运行在Pod中的进程可以获取对应的username和token，从而用来访问 Api Server

对于 Service Account 最简单的使用是 base 在 namespace 上的

每一个namespace都携带一个名叫 default 的默认的 service account

我们在helloworld namespace 上部署一个nginx，然后我通过这个nginx尝试访问apiserver

这里需要分为2个步骤

1 先对应服务器端 CA

1.1 可以直接选择忽略

curl -k https://192.168.204.149:6443/api

1.2 找到并使用服务器端CA

看到 401 之后，此时并不是 RBAC 权限阻止了你，而是 api server 阻止了你

2. 需要对应 apiserver 的 token

/var/run/secrets/kubernetes.io/serviceaccount/token

找到token之后在命令行中加入token

curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjN6UW1TTTR2VlI2M3MxVklnRjFPZmtIa1JiVU92bVk1UENtMlBRQVFWQTAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJoZWxsb3dvcmxkIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tbHZjY2QiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6Ijg2YTQ3MjE1LTRiODctNDRhYy1hMTc2LWZkN2E0OTI4MjU1YiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpoZWxsb3dvcmxkOmRlZmF1bHQifQ.RnvDLG86wveeOmbY1jwJyu2YYX84KFnZkWP0hMo5gbSnMNfy8MDW6l00NiTqueTS5Efvzl9IQNo7cYJWLmwZ43hRSj9UaLnLHoWl3wgb0eT16AlSVL7WX5f05jdxH55gKqv-gOsF1u1btzltiI5a5iz5bvr02lOa23JD6xWpPKt5beBfcW6BSvbBboMYtwh_4fue6nOCBd96-nhOcd3ytdhQUu6Ta2o3BW7Ro1bMyJuwTWD_YxCKQj3Ab47n13W2MiztvXOF9dqGCRACBy_ZqvXNB3QsCyjDgnSWcc4jlkxKYsgmiQgOgFkdDzIrC-qGAdLpTF941QtYYZXqH91keQ"  https://192.168.204.149:6443/api 

此时你已经可以访问 api 端点了，但是仍然不能访问pod

curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjN6UW1TTTR2VlI2M3MxVklnRjFPZmtIa1JiVU92bVk1UENtMlBRQVFWQTAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJoZWxsb3dvcmxkIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tbHZjY2QiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6Ijg2YTQ3MjE1LTRiODctNDRhYy1hMTc2LWZkN2E0OTI4MjU1YiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpoZWxsb3dvcmxkOmRlZmF1bHQifQ.RnvDLG86wveeOmbY1jwJyu2YYX84KFnZkWP0hMo5gbSnMNfy8MDW6l00NiTqueTS5Efvzl9IQNo7cYJWLmwZ43hRSj9UaLnLHoWl3wgb0eT16AlSVL7WX5f05jdxH55gKqv-gOsF1u1btzltiI5a5iz5bvr02lOa23JD6xWpPKt5beBfcW6BSvbBboMYtwh_4fue6nOCBd96-nhOcd3ytdhQUu6Ta2o3BW7Ro1bMyJuwTWD_YxCKQj3Ab47n13W2MiztvXOF9dqGCRACBy_ZqvXNB3QsCyjDgnSWcc4jlkxKYsgmiQgOgFkdDzIrC-qGAdLpTF941QtYYZXqH91keQ"  https://192.168.204.149:6443/api/v1/namespaces/default/pods

看见403之后，这里可以肯定已经是被 RBAC 阻止了

我们来给这个 serviceaccount 一个 最大管理员权限

kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --group=system:serviceaccounts:helloworld

由于赋给了最大管理员权限，所以此时，不仅能看到 default namespace 下的 pod 信息，还能看到 helloworld namespace下的pod信息

Service Account 使用一种特殊的secret对象，叫做 ServiceAccountToken，它会被自动mount到各个pod中

使用Golang为kubernetes创建用户

在 Kubernetes User Account 中我们使用OPENSSL创建了CSR和私钥，并且为kubernetes创建了一个新用户，这里提供一个相对简单的方法，使用golang和shell

创建 main.go

package main

import (
	"crypto/rand"
	"crypto/rsa"
	"crypto/x509"
	"crypto/x509/pkix"
	"encoding/asn1"
	"encoding/pem"
	"os"
)

func main() {
	name := os.Args[1]
	user := os.Args[2]

	key, err := rsa.GenerateKey(rand.Reader, 2048)
	if err != nil {
		panic(err)
	}

	keyDer := x509.MarshalPKCS1PrivateKey(key)
	keyBlock := pem.Block{
		Type:  "RSA PRIVATE KEY",
		Bytes: keyDer,
	}
	keyFile, err := os.Create(name + "-key.pem")
	if err != nil {
		panic(err)
	}

	pem.Encode(keyFile, &keyBlock)
	keyFile.Close()
	commonName := user

	emailAddress := "hello@lizhe.name"

	org := "lizhe"
	orgUnit := "lz"
	city := "DL"
	state := "LN"
	country := "CN"

	subject := pkix.Name{
		CommonName:         commonName,
		Country:            []string{country},
		Locality:           []string{city},
		Organization:       []string{org},
		OrganizationalUnit: []string{orgUnit},
		Province:           []string{state},
	}

	asn1, err := asn1.Marshal(subject.ToRDNSequence())
	if err != nil {
		panic(err)
	}

	csr := x509.CertificateRequest{
		RawSubject:         asn1,
		EmailAddresses:     []string{emailAddress},
		SignatureAlgorithm: x509.SHA256WithRSA,
	}

	bytes, err := x509.CreateCertificateRequest(rand.Reader, &csr, key)
	if err != nil {
		panic(err)
	}

	csrFile, err := os.Create(name + ".csr")
	if err != nil {
		panic(err)
	}

	pem.Encode(csrFile, &pem.Block{Type: "CERTIFICATE REQUEST", Bytes: bytes})
	csrFile.Close()
}

调用脚本，此脚本需要两个参数 client 和 lizhe

client 会得到一个 client.csr 而 lizhe 将会是这个 csr的 subject

go run main.go client lizhe

创建一个 main.sh 文件，填入以下内容

#!/bin/sh

csr_name="my-client-csr"
name="${1}"
csr="${2}"

cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${csr_name}
spec:
  groups:
  - system:authenticated
  request: $(cat ${csr} | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - client auth
EOF

echo
echo "Approving signing request."
kubectl certificate approve ${csr_name}

echo 
echo "Downloading certificate."
kubectl get csr ${csr_name} -o jsonpath='{.status.certificate}' \
        | base64 --decode > $(basename ${csr} .csr).crt

echo
echo "Cleaning up"
kubectl delete csr ${csr_name}

echo
echo "Add the following to the 'users' list in your kubeconfig file:"
echo "- name: ${name}"
echo "  user:"
echo "    client-certificate: ${PWD}/$(basename ${csr} .csr).crt"
echo "    client-key: ${PWD}/$(basename ${csr} .csr)-key.pem"

echo
echo "Next you may want to add a role-binding for this user."

调用这个shell，注意因为没有太多的异常处理，请确认可以成功运行

./main.sh lizhe client.csr

正常的配置文件是这样的

我们需要使用以下内容，所以我copy一份上面的config文件，并且进行一些改动

- name: lizhe
  user:
    client-certificate: /home/lizhe/works/createuser/client.crt
    client-key: /home/lizhe/works/createuser/client-key.pem

拷贝文件到工作目录

sudo cp /etc/rancher/k3s/k3s.yaml /home/lizhe/works/createuser/k3s_lizhe.yaml

修改 k3s_lizhe.yaml

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTVRnd01UazNNREF3SGhjTk1qRXdOREV3TURFMU5UQXdXaGNOTXpFd05EQTRNREUxTlRBdwpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTVRnd01UazNNREF3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFSL1NnS2NVeGM4T01ZeHhMcDZWeDVyak9JSUJnNUpGS3lBV3d5WXRpWG8KM3NxdHVMQXVnUldXVVpwQklKbXRsQ1htcjg0ZFR4ejJmbXpuV3ZSa3ZKOC9vMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVTZsemFvSjdVKy9qUnNHYVZqOWxqCjRiam1aMGt3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnTUxQcEF2ejFuTzFKTWR6QnhKMmY3UnUzWTlVd0VkYkkKbXphOXBLZTlOdlFDSVFEQWh1WFp4SlJuMGxwVDQvdVFUUEtIck9DYjZNMExncXNDYXgyMjVIVFNOdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://127.0.0.1:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: lizhe
  user:
    client-certificate: /home/lizhe/works/createuser/client.crt
    client-key: /home/lizhe/works/createuser/client-key.pem

我们来使用curl检查一下生成的 x509 文件是不是能正常工作

curl --key ./client-key.pem --cert ./client.crt --insecure https://localhost:6443/api

下面我们去创建 rolebinding

cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: default
  name: podrole
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
EOF

kubectl create rolebinding podrolebinding  -n default --clusterrole podrole --user lizhe

尝试使用lizhe这个用户来查询 default namespace 下的 pods

curl --cert ./client.crt --key ./client-key.pem --insecure -s https://localhost:6443/api/v1/namespaces/default/pods

什么都没有得到，我们创建一个pod再试

配置对应的 kubeconfig文件

sudo kubectl config --kubeconfig=./k3s.yaml set-credentials lizhe --client-certificate=/home/lizhe/works/createuser/client.crt --client-key=/home/lizhe/works/createuser/client-key.pem
sudo kubectl config --kubeconfig=./k3s.yaml set-context lizhe_context --cluster=default --namespace=default --user=lizhe
sudo kubectl config --kubeconfig=./k3s.yaml use-context lizhe_context

也可以直接创建一个新的 kubeconfig文件，这样这个文件就只包含 lizhe 这一个用户

kubectl config --kubeconfig=./k3s_lizhe.yaml set-credentials lizhe --client-certificate=/home/lizhe/works/createuser/client.crt --client-key=/home/lizhe/works/createuser/client-key.pem
kubectl config --kubeconfig=./k3s_lizhe.yaml set-context lizhe_context --cluster=default --namespace=default --user=lizhe

这样创建的文件没有cluster信息，需要手动拷贝

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTVRnd01UazNNREF3SGhjTk1qRXdOREV3TURFMU5UQXdXaGNOTXpFd05EQTRNREUxTlRBdwpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTVRnd01UazNNREF3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFSL1NnS2NVeGM4T01ZeHhMcDZWeDVyak9JSUJnNUpGS3lBV3d5WXRpWG8KM3NxdHVMQXVnUldXVVpwQklKbXRsQ1htcjg0ZFR4ejJmbXpuV3ZSa3ZKOC9vMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVTZsemFvSjdVKy9qUnNHYVZqOWxqCjRiam1aMGt3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnTUxQcEF2ejFuTzFKTWR6QnhKMmY3UnUzWTlVd0VkYkkKbXphOXBLZTlOdlFDSVFEQWh1WFp4SlJuMGxwVDQvdVFUUEtIck9DYjZNMExncXNDYXgyMjVIVFNOdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://127.0.0.1:6443
  name: default
contexts:
- context:
    cluster: default
    namespace: default
    user: lizhe
  name: lizhe_context
current-context: ""
kind: Config
preferences: {}
users:
- name: lizhe
  user:
    client-certificate: client.crt
    client-key: client-key.pem

每次打开 shell，通过直接使用这个 KUBECONFIG，都能切换lizhe这个用户了

RBAC 授权

API Server使用以下几种授权策略，通过API Server的启动参数 --authorization-mode 设置

• AlwaysDeny：表示拒绝所有请求。仅用于测试
• AlwaysAllow：表示允许所有请求。如果有集群不需要授权流程，则可以采用该策略
• Node：节点授权是一种特殊用途的授权模式，专门授权由 kubelet 发出的 API 请求
• Webhook：是一种 HTTP 回调模式，允许使用远程 REST 端点管理授权
• ABAC：基于属性的访问控制，表示使用用户配置的授权规则对用户请求进行匹配和控制
• RBAC：基于角色的访问控制，默认使用该规则

RBAC（Role-Based Access Control）基于角色的访问控制，在Kubernetes 1.5 中引入，现为默认标准。
相对其他访问控制方式，拥有如下优势：

1. 对集群中的资源和非资源均拥有完整的覆盖
2. 整个RBAC完全由几个API对象完成，同其他API对象一样，可以用kubectl或API进行操作
3. 可以在运行时进行操作，无需重启API Server

Kubernetes 的 RBAC 流程需要定义三个主要组件

• Subject
• Rule
• Rolebinding

Subject 即 实际需要被检查的对象，包含 user （Kubernetes User Account）、service account(Kubernetes Service Account) 和 group

Rule 是一组包含了类似 CRUD 动作的列表，还包含一些kubernetes附加的功能，watch、list 和 exec。kubernetes使用 Role（Kubernetes Role & ClusterRole）来定义 Rule 的适用范围。

X509

配置KUBECONFIG

在Kubernetes User Account中我们添加了一个用户，下面语句可以将这个用户直接配置进kube config 配置文件

sudo kubectl config --kubeconfig=./kubeconfig.yaml set-credentials lizhe --client-certificate=/home/lizhe/works/client.crt --client-key=/home/lizhe/works/client.key

创建一个新的context

kubectl config --kubeconfig=./kubeconfig.yaml set-context lizhe_context --cluster=local --namespace=default --user=lizhe

切换 context

kubectl config --kubeconfig=./kubeconfig.yaml use-context lizhe_context

这个用户只能访问 default namespace 下的 list pod

Kubernetes User Account

Kubernetes的User Account用于从 集群外部 访问 api-server 时 使用

kubectl命令就是作为kubernetes-admin用户来执行的，默认使用x509客户端证书方式，x509 是双向认证的

参考 https://kubernetes.io/docs/reference/access-authn-authz/authentication/

查看当前集群中的用户信息

kubectl config view

下面我们在集群中添加新的 user account

现在3台机器，k8s1、k8s2 和 k8s_user ，

k8s_user 上生成私钥

openssl genrsa -out client.key 2048

生成csr

openssl req -new -key client.key -out client.csr -subj "/CN=lizhe"

确认服务器上的 Kubernetes 根证书

我这里是 /etc/kubernetes/ssl

在Kubernetes节点 k8s1 上找到下面两个文件，转存到 k8s_user

kube-ca-key.pem 和 kube-ca.pem

k8s_user上

生成 client.crt

openssl x509 -req -in client.csr -CA /home/lizhe/works/ssl/kube-ca.pem -CAkey /home/lizhe/works/ssl/kube-ca-key.pem -CAcreateserial -out client.crt -days 365

这是错误的情况

如果一切顺利的话

不校验server端证书

curl --cert ./client.crt --key ./client.key --insecure -s https://k8s1.me:6443/api

校验server端证书

curl --cert ./client.crt --key ./client.key --cacert /home/lizhe/works/ssl/kube-ca.pem -s https://k8s1.me:6443/api

上面的用户 lizhe 会被分配到 group system:authenticated

这个 group 默认携带 以下 3 个 clusterrole

system:public-info-viewer
system:discovery
system:basic-user

访问 api 还是可以的，但是要访问 pod，基本没戏

curl --cert ./client.crt --key ./client.key --cacert /home/lizhe/works/ssl/kube-ca.pem -s https://k8s1.me:6443/api/v1/namespaces/default/pods

创建一个可以 list pod 的 clusterrole，然后绑定 用户 lizhe 到 podrole

cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: default
  name: podrole
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
EOF

kubectl create rolebinding podrolebinding  -n default --clusterrole podrole --user lizhe

curl --cert ./client.crt --key ./client.key --cacert /home/lizhe/works/ssl/kube-ca.pem -s https://k8s1.me:6443/api/v1/namespaces/default/pods

Kubernetes 使用私有镜像进阶

在 Kubernetes 使用私有镜像 中我们讨论了如何使用私有镜像

通过创建 secret 然后指定对应的 imagePullSecrets

如果不在deployment中指定 imagePullSecrets 还是会得到以下错误

需要在所有的deployment都指定使用的 secret 就成为了一个问题

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
        - name: nginx
          image: lizhepac/privateimage
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80

可以通过 patch serviceaccount 的方式来对namespace默认提供类似注入的功能

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regsecret"}]}'

然后重新部署 nginx

可以看到，集群自动加入了 imagePullSecrets

正常开始下载了

Kubernetes 使用私有镜像

对于私有镜像，默认肯定是无法直接使用的

常规做法是先创建 docker-registry 类型的 secret

kubectl create secret docker-registry regsecret --docker-username=dockername --docker-password=password --docker-email=helloworld@lizhe.name

在 deployment 中修改

    spec:
      imagePullSecrets:
      - name: regsecret
      containers:
      - image: lizhepac/privateimage
        imagePullPolicy: Always
        name: private
        resources: {}

Kubernetes Ingress

Kubernetes Ingress 通常用来以 L7 LoadBalance 的形式对外暴露服务，这里就不再赘述了

这里只是为了尽量清晰的描述一下 Ingress本身一些让人忽略的问题

Ingress是Kubernetes API的标准资源类型之一，它其实就是一组基于DNS名称（host）或URL路径把请求转发到指定的Service资源的规则

Ingress 的实现分为两个部分 Ingress Controller 和 Ingress .

1. Ingress Controller 是流量的入口，是一个实体软件， 一般是Nginx 和 Haproxy 。
2. Ingress 描述具体的路由规则。

因为内容比较乱，所以这里分成几个具体的真实Case来讨论

K3s的情况

最近尝试了 K3s，K3s 的默认实现（不需要安装rancher，裸K3s）竟然不是 Nginx Ingress 而是 traefik

安装集群之后，我遇到了一个很诡异的问题，以往使用Kubernetes的时候，只要加入了一个Ingress规则，那么DNS只要指向任意节点都可以访问到这个 deployment，可是在 K3s 下竟然只有一个节点可以正常访问，例如

• 192.168.204.165
• 192.168.204.166
• 192.168.204.167
• 192.168.204.168

只有192.168.204.168才能访问到Ingress

检查ingress-nginx-controller发现RKE2 并不像RKE1 一样，使用DaemonSet来部署这个controller，而是莫名其妙的使用了普通的 Deployment， 这就导致只有部署了 ingress controller 的 168 能正常接收请求

RKE1 的情况就好很多，因为是 DaemonSet 所以每个节点都可以正常工作

这里我们手动把部署数量改成4，很顺利的 controller 被分布在了4个节点上

在这种情况下任意节点都可以正常接收请求

ADDRESS 显示了全部地址