cn

Vault Kubernetes HA安装

HA 安装主要是需要 consul

添加仓库 https://helm.releases.hashicorp.com

设置values

server.replicas = 1
ui.enabled = true （port 8500）

配置一个ingress

consul 的安装就结束了

使用以下 values 安装vault

server.ha.enabled = true
server.ha.replicas = 1
server.ui.enabled = true ( port 8200 )

可以看到storage是

storage "consul" {
  path = "vault"
  address = "HOST_IP:8500"
}

初始化

/ $ vault operator init
Unseal Key 1: rUrvo0abV/NT0K4x5ZaCCkbyVbsf88rEUAr1qn00fNrw
Unseal Key 2: vZgywK0ZBh+RI+lKYb4oqzRt7P4BWuRacLTHYEZdtRL6
Unseal Key 3: 8yH0vbYloMEVzaLxPQgCalzUS3uG823UEfzqcLrlLQiL
Unseal Key 4: 9dVwFNK5agYi4fyZW1x0RayhG2YG6eZSp4wLVeLsXPjE
Unseal Key 5: w+KUxcV48Q1ZzyvtipzRcl0yLeYAD0g1ihQBfvZ1NZix

Initial Root Token: s.YIhdm8s9jMRqCqgUP00YyJjI

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
/ $ 

unseal

export VAULT_TOKEN=s.YIhdm8s9jMRqCqgUP00YyJjI
vault operator unseal rUrvo0abV/NT0K4x5ZaCCkbyVbsf88rEUAr1qn00fNrw
vault operator unseal vZgywK0ZBh+RI+lKYb4oqzRt7P4BWuRacLTHYEZdtRL6
vault operator unseal 8yH0vbYloMEVzaLxPQgCalzUS3uG823UEfzqcLrlLQiL
vault operator unseal 9dVwFNK5agYi4fyZW1x0RayhG2YG6eZSp4wLVeLsXPjE
vault operator unseal w+KUxcV48Q1ZzyvtipzRcl0yLeYAD0g1ihQBfvZ1NZix
vault status

Vault Kubernetes 安装

Vault可以用来管理 秘钥、加密 数据 和 基于身份的权限验证 等

先安装longhorn作为持久化storage 提供 PV

添加仓库 https://helm.releases.hashicorp.com

设置values

dataStorage.size = 1Gi

vault 需要一个初始化动作

vault operator init


/ $ vault operator init
Unseal Key 1: 0knf13iA1JvSgIixWC5VMQKtcyHbrfBWoqxSWGKAbuCY
Unseal Key 2: UDcCwhlGcYpdRvYvfLD51hdFTh6QXIjxP3NCMATF6hys
Unseal Key 3: fpO7dILIfIMB6pfOy/IRb5U1zMpELWSWbjUXalwlhYkP
Unseal Key 4: oy4BRHPogCxaF231MfIQjwdF0GNZGcVJRDjwrNvW+yki
Unseal Key 5: sOwl2LUwW0ZZYlgh/T0H9Ajw4VHVwsKuRIMhdvwfGfKw

Initial Root Token: s.sJSDGrPe7rwnb5NDUs5VzPon

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
/ $ 

然后需要对秘钥 unseal

export VAULT_TOKEN=s.sJSDGrPe7rwnb5NDUs5VzPon
vault operator unseal 0knf13iA1JvSgIixWC5VMQKtcyHbrfBWoqxSWGKAbuCY
vault operator unseal UDcCwhlGcYpdRvYvfLD51hdFTh6QXIjxP3NCMATF6hys
vault operator unseal fpO7dILIfIMB6pfOy/IRb5U1zMpELWSWbjUXalwlhYkP
vault operator unseal oy4BRHPogCxaF231MfIQjwdF0GNZGcVJRDjwrNvW+yki
vault operator unseal sOwl2LUwW0ZZYlgh/T0H9Ajw4VHVwsKuRIMhdvwfGfKw
vault status

这样就完成了

可以看到默认使用了 file 作为 storage

Minio SSE-C 加密

SSE-C 加密是在客户端进行的加密，使用的是GPG

GPG（FNU Privary Fuard）是最流行的数据加密，数据签名工具软件

支持的算法：

公钥：RSA，ELG，DSA

对称加密：3DES，CAST5，BLOWFISH，AES，AES256

散列：MD5，SHA1，.. ..,SHA256，SHA512

基本用法：

加密操作： –symmetric 或 -c

解密操作：–decrypt 或 -d

在加密大文件的时候，性能损耗还是不小的

下面是配置文件

lizhe@ubuntu:~$ cat ~/.s3cfg 
# Setup endpoint
host_base = minio.lizhe.com
host_bucket = minio.lizhe.com
bucket_location = us-east-1
use_https = False

# Setup access keys
access_key = iZq4uq5dupOzCgNkThzU 
secret_key = qn8DeztD1Qe7tAOf2jr9qwpybxcZrklWsv58OhAt 

# Enable S3 v4 signature APIs
signature_v2 = False

gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = `123456`

lizhe@ubuntu:~$ 

创建桶

lizhe@ubuntu:~$ s3cmd mb s3://my-bucket-name
Bucket 's3://my-bucket-name/' created
lizhe@ubuntu:~$ s3cmd ls
2021-06-19 15:22  s3://my-bucket-name

上传文件

lizhe@ubuntu:~$ echo "helloworld" > hello.txt
lizhe@ubuntu:~$ s3cmd -e put hello.txt s3://my-bucket-name
upload: '/tmp/tmpfile-v1wNUCw8NqGPE6096L5X' -> 's3://my-bucket-name/hello.txt'  [1 of 1]
 87 of 87   100% in    0s     4.91 kB/s  done
lizhe@ubuntu:~$ s3cmd -e get s3://my-bucket-name/hello.txt /tmp -f
download: 's3://my-bucket-name/hello.txt' -> '/tmp/hello.txt'  [1 of 1]
 87 of 87   100% in    0s     4.03 kB/s  done
lizhe@ubuntu:~$ 

我们来看一下加密结果

可以看到

上传的文件 和 下载的文件 内容都是正常的，只是存储在服务器上的文件内容已经被加密了

我们尝试使用 gpg 命令来解密

注意我这里的密码是 `123456` 而不是 123456

`123456`

使用root用户同样可以解密

可以看到 这里默认使用的是 AES256 算法

Minio SSE-S3加密

MinIO使用密钥管理系统（KMS）支持SSE-S3（使用S3托管密钥的服务器端加密）

如果客户端请求SSE-S3，或启用了自动加密，则MinIO服务器会使用唯一的对象密钥对每个对象进行加密，该对象密钥受KMS管理的主密钥保护。由于开销极低，因此可以为每个应用程序和实例打开自动加密。

MinIO通过KES访问KMS，而不是直接访问KMS，每个MinIO群集均具有“自己的” KES实例。KES是一个用于高性能应用程序的无状态分布式密钥管理系统，它处理”其” MinIO群集发出的加密/解密请求，从而使KMS不必处理大量流量，KMS仅作为KES安全密钥的后台存储。

这里需要

1. vault服务器作为中央秘钥仓库
2. kes服务器作为连接minio和vault的中间件
3. minio服务器

在Kubernetes安装 vault

https://github.com/hashicorp/vault-helm

HA模式
server.ha.enabled = true
server.ha.replicas = 1
server.ui.enabled = true ( port 8200 )

单机节点
server.standalone.enabled = true

HA模式需要 consul 和 存储
https://helm.releases.hashicorp.com

server.replicas = 1

vault operator init

/ $ vault operator init
Unseal Key 1: 0mQk974OWKaSVTXGQ0Uzyj2bte1QM3AaOpSpjgazf9QZ
Unseal Key 2: 8/WoBHinaV0gMK6GsRuhnZWgolHpnBj2jkfAAW2khxTY
Unseal Key 3: gTYfW8gB9B3mI9c8oBYzWqks2Lkfu5cjC+MzWgdKU7sr
Unseal Key 4: FqF8oLlXh3zgnt5+4itdf83ecfmp9fSDC2hDSl4YG6kS
Unseal Key 5: ACAwtnxeD9Wgiq4VeQmQkhQg5IEXo6AYUgVV1u+Bq2g0

Initial Root Token: s.SkECG9cBaD259MLYh1Zvx9jv

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
/ $ 

对key解封

export VAULT_TOKEN=s.SkECG9cBaD259MLYh1Zvx9jv
vault operator unseal 0mQk974OWKaSVTXGQ0Uzyj2bte1QM3AaOpSpjgazf9QZ
vault operator unseal 8/WoBHinaV0gMK6GsRuhnZWgolHpnBj2jkfAAW2khxTY
vault operator unseal gTYfW8gB9B3mI9c8oBYzWqks2Lkfu5cjC+MzWgdKU7sr
vault status

vault 启动完成

export VAULT_TOKEN=s.SkECG9cBaD259MLYh1Zvx9jv

启用保险柜的K / V后端

vault secrets enable kv

启用AppRole身份验证

vault auth enable approle

创建文件
minio-kes-policy.hcl
path "kv/minio/*" {
  capabilities = [ "create", "read", "delete" ]
}

------------
cat > minio-kes-policy.hcl <<EOF
path "kv/minio/*" {
  capabilities = [ "create", "read", "delete" ]
}
EOF
# vault policy write minio-key-policy ./minio-kes-policy.hcl
# vault policy list
# vault policy read minio-key-policy

创建一个新的AppRole ID并将其绑定到策略

现在，我们需要创建一个新的AppRole ID并授予该ID特定的权限。 该应用程序（即KES服务器）将通过AppRole角色ID 和机密ID 向Vault进行身份验证，并且仅允许执行特定策略授予的操作。

因此，我们首先为KES服务器创建一个新角色：

vault write auth/approle/role/kes-role token_num_uses=0  secret_id_num_uses=0  period=5m

然后，我们将策略绑定到角色：

vault write auth/approle/role/kes-role policies=minio-key-policy

最后，我们从Vault请求AppRole角色ID和秘密ID。 一，角色ID：

vault read auth/approle/role/kes-role/role-id

然后是秘密ID：

vault write -f auth/approle/role/kes-role/secret-id

Longhorn backup

使用docker启动一个nfs服务器, 特别注意需要 nfsv4 版本，v3不可用

客户端安装 nfs-common （理论上不需要，如果有问题请安装）

sudo apt-get install nfs-common

使用 docker-compose.yml 启动一个 nfsv4 服务

mkdir -p /data/docker-volumes

version: "2.1"
services:
  # https://hub.docker.com/r/itsthenetwork/nfs-server-alpine
  nfs:
    image: itsthenetwork/nfs-server-alpine:12
    container_name: nfs
    restart: unless-stopped
    privileged: true
    environment:
      - SHARED_DIRECTORY=/data
    volumes:
      - /data/docker-volumes:/data
    ports:
      - 2049:2049

设置 nfs 位置

nfs://192.168.194.190:/

创建一个备份

可以看到在对应文件夹下出现了 pvc 的备份

Minio 安装

下面两个地址都可以获取 helm3的安装包

sudo helm repo add bitnami https://charts.bitnami.com/bitnami

另一个是

https://github.com/minio/charts

这里用的是第二种 官方地址

安装之前确认自己有可用的 storage class，这里用的是 longhorn

默认的存储空间大小是 500GB，所以如果你只是装来玩玩注意修改一下 values

persistence.size = 5Gi

找到对应的 秘钥

我这里的是

accesskey iZq4uq5dupOzCgNkThzU
secretkey qn8DeztD1Qe7tAOf2jr9qwpybxcZrklWsv58OhAt

如果只能使用命令行来查看

kubectl get secret --namespace minio minio -o jsonpath="{.data.accesskey}" | base64 -d
kubectl get secret --namespace minio minio -o jsonpath="{.data.secretkey}" | base64 -d

我们创建一个 ingress 端口 9000 就可以打开dashboard 了

Traefik basic auth

1. 这里我要添加3个密码

user1 lizhe1
user2 lizhe2
user3 lizhe3

2. 使用 htpasswd

Last login: Fri Jun 18 18:30:05 on console
lizhe@lizhedeMacBook-Pro ~ % htpasswd -nb user1 lizhe1 
user1:$apr1$yL48ADZo$k1fYmQU5.f8po/RJWzPWL/

lizhe@lizhedeMacBook-Pro ~ % htpasswd -nb user2 lizhe2
user2:$apr1$IYposYR0$nebFgm5z13tHtTUFRKYyP1

lizhe@lizhedeMacBook-Pro ~ % htpasswd -nb user3 lizhe3
user3:$apr1$VKL5nrB3$obUGnGAWI610SOPvYnS6W.

lizhe@lizhedeMacBook-Pro ~ % 

3. 把它们放进一个文件里

4. 用base64加密

lizhe@lizhedeMacBook-Pro tmp % cat policy | openssl base64
dXNlcjE6JGFwcjEkeUw0OEFEWm8kazFmWW1RVTUuZjhwby9SSld6UFdMLwp1c2Vy
MjokYXByMSRJWXBvc1lSMCRuZWJGZ201ejEzdEh0VFVGUktZeVAxCnVzZXIzOiRh
cHIxJFZLTDVuckIzJG9iVUduR0FXSTYxMFNPUHZZblM2Vy4K
lizhe@lizhedeMacBook-Pro tmp %

5. middleware yaml

# Declaring the user list
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: dev-auth
  namespace: lizhe
spec:
  basicAuth:
    secret: authsecret

---
# Note: in a kubernetes secret the string (e.g. generated by htpasswd) must be base64-encoded first.
# To create an encoded user:password pair, the following command can be used:
# htpasswd -nb user password | openssl base64

apiVersion: v1
kind: Secret
metadata:
  name: authsecret
  namespace: lizhe

data:
  users: |2
    dXNlcjE6JGFwcjEkeUw0OEFEWm8kazFmWW1RVTUuZjhwby9SSld6UFdMLwp1c2Vy
    MjokYXByMSRJWXBvc1lSMCRuZWJGZ201ejEzdEh0VFVGUktZeVAxCnVzZXIzOiRh
    cHIxJFZLTDVuckIzJG9iVUduR0FXSTYxMFNPUHZZblM2Vy4K

6 创建middleware

7 创建 ingress route

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: webappingress
  namespace: lizhe
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`nginx.lizhe.com`)
      kind: Rule
      services:
        - name: nginxsvc
          port: 80
      middlewares:
        - name: dev-auth

8 验证

Wrk脚本

本例主要用来演示如何使用 wrk script 来进行压测

这里使用的还是我们的 whoami

下面这个脚本可以将返回的 response 写入到文件里

-- example script that demonstrates response handling and
-- retrieving an authentication token to set on all future
-- requests

-- wrk -c 1 -t 1 -d 5 -s ./test.lua http://whoami.lizhe.com

token = nil
path  = "/"
logfile = io.open("wrk.log", "w");

request = function()
    wrk.headers["test_token"] = "helloworld"
    return wrk.format("GET", path)
end

response = function(status, headers, body)
    logfile:write("status:" .. status .. "\n");
    logfile:write("body:" .. body .. "\n");
end

ulimit -n 2048

Bitnami chat

https://charts.bitnami.com/bitnami

Kong 证书安装

首先要打开 Kong的 8443 端口

完整的 docker-compose.yml 文件如下

version: "3"
 
networks:
 kong-net:
  driver: bridge
 
services:
 
  #######################################
  # Postgres: The database used by Kong
  #######################################
  kong-database:
    image: postgres:9.6
    restart: always
    networks:
      - kong-net
    environment:
      POSTGRES_USER: kong
      POSTGRES_DB: kong
      POSTGRES_PASSWORD: kong
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "kong"]
      interval: 5s
      timeout: 5s
      retries: 5
 
  #######################################
  # Kong database migration
  #######################################
  kong-migration:
    image: kong:latest
    command: "kong migrations bootstrap"
    networks:
      - kong-net
    restart: on-failure
    environment:
      - KONG_DATABASE=postgres
      - KONG_PG_HOST=kong-database
      - KONG_PG_DATABASE=kong
      - KONG_PG_PASSWORD=kong
    links:
      - kong-database
    depends_on:
      - kong-database
 
  #######################################
  # Kong: The API Gateway
  #######################################
  kong:
    image: kong:latest
    restart: always
    networks:
      - kong-net
    environment:
      KONG_DATABASE: postgres
      KONG_PG_HOST: kong-database
      KONG_PG_PASSWORD: kong
      KONG_PROXY_LISTEN: 0.0.0.0:8000,0.0.0.0:8443 ssl
      KONG_ADMIN_LISTEN: 0.0.0.0:8001
    depends_on:
      - kong-migration
    links:
      - kong-database
    healthcheck:
      test: ["CMD", "curl", "-f", "http://kong:8001"]
      interval: 5s
      timeout: 2s
      retries: 15
    ports:
      - "8001:8001"
      - "8000:8000"
      - "8443:8443"
 
 
  #######################################
  # Konga database prepare
  #######################################
  konga-prepare:
    image: pantsel/konga:latest
    command: "-c prepare -a postgres -u postgresql://kong:kong@kong-database:5432/konga"
    networks:
      - kong-net
    restart: on-failure
    environment:
      - KONG_DATABASE=postgres
      - KONG_PG_HOST=kong-database
      - KONG_PG_DATABASE=konga
      - KONG_PG_PASSWORD=kong
    links:
      - kong-database
    depends_on:
      - kong-database
 
  #######################################
  # Konga: Kong GUI
  #######################################
  konga:
    image: pantsel/konga:latest
    restart: always
    networks:
     - kong-net
    environment:
      DB_ADAPTER: postgres
      DB_URI: postgresql://kong:kong@kong-database:5432/konga
      NODE_ENV: production
    links:
      - kong-database
    depends_on:
      - kong
      - konga-prepare
    ports:
      - "1337:1337"

然后需要注入一个 cert

curl -k -X POST \
  http://diynocap.com:8001/certificates \
  -H 'Content-Type: multipart/form-data' \
  -F cert=@./cert.crt \
  -F key=@./private.key \
  -F snis[]=studyk8s.com

虽然会提示 zsh: no matches found: snis[]=studyk8s.com 但是证书仍然被创建成功了

也可以通过 konga的 UI 界面来添加

后端服务可以正常使用 http 80 端口，然后把前端的 kong暴露到 8443