Month: June 2021

Elasticsearch helm安装cluster

这里使用的是 https://helm.elastic.co/ 仓库中的官方包

因为我是单机节点而且资源有限，所以添加了 values

antiAffinity = soft
volumeClaimTemplate.resources.requests.storage = 3Gi
resources.requests.cpu = 500m
replicas = 5

可以看到5个节点都是健康的

curl localhost:9200/_nodes/process?pretty

Vagrant One-time ssh password

# To view in Web UI
path "sys/mounts" {
  capabilities = [ "read", "update" ]
}

# To configure the SSH secrets engine
path "ssh/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# To enable secrets engines
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

/ $ vault secrets enable ssh
Success! Enabled the ssh secrets engine at: ssh/
/ $ 
/ $ vault write ssh/roles/otp_key_role \
>     key_type=otp \
>     default_user=ubuntu \
>     cidr_list=0.0.0.0/0
Success! Data written to: ssh/roles/otp_key_role
/ $ 
/ $ 

/ $ vault secrets enable ssh
Success! Enabled the ssh secrets engine at: ssh/
/ $ 
/ $ vault write ssh/roles/otp_key_role \
>     key_type=otp \
>     default_user=ubuntu \
>     cidr_list=0.0.0.0/0
Success! Data written to: ssh/roles/otp_key_role
/ $ 
/ $ vault auth enable userpass
Success! Enabled userpass auth method at: userpass/
/ $ 
/ $ vault write auth/userpass/users/bob password="training" policies="ssh-otp"
Success! Data written to: auth/userpass/users/bob
/ $ 

Vault dynamic secrets MongoDB

dynamic secret是在访问时生成的，也就是说 dynamic secret 在被读取之前是不存在的，因此没有人窃取它们或其他客户使用相同机密的风险。同时因为Vault具有内置的吊销机制，所以dynamic secret可以在使用后立即吊销，从而将机密存在的时间减至最少。

官方入门文档中提供了一个 aws engine 的说明

https://learn.hashicorp.com/tutorials/vault/getting-started-dynamic-secrets?in=vault/getting-started

同时提供了

postgres https://learn.hashicorp.com/tutorials/vault/database-secrets?in=vault/secrets-management

mongodb https://learn.hashicorp.com/tutorials/vault/database-mongodb?in=vault/secrets-management

下面是一个 MongoDB的例子

启动一个数据库

docker run -d \
    -p 0.0.0.0:27017:27017 -p 0.0.0.0:28017:28017 \
    --name=mongodb \
    -e MONGO_INITDB_ROOT_USERNAME="mdbadmin" \
    -e MONGO_INITDB_ROOT_PASSWORD="hQ97T9JJKZoqnFn2NXE" \
    mongo

Configure MongoDB secrets engine

vault write mongodb/config/mongo-test \
      plugin_name=mongodb-database-plugin \
      allowed_roles="tester" \
      connection_url="mongodb://mdbadmin:hQ97T9JJKZoqnFn2NXE@192.168.194.193:27017/admin?tls=false" \
      username="mdbadmin" \
      password="hQ97T9JJKZoqnFn2NXE"

vault write mongodb/roles/tester \
    db_name=mongo-test \
    creation_statements='{ "db": "admin", "roles": [{ "role": "readWrite" }, {"role": "read", "db": "foo"}] }' \
    default_ttl="1h" \
    max_ttl="24h"

vault read mongodb/creds/tester

Key                Value
---                -----
lease_id           mongodb/creds/tester/7GPR6sRrPloRvAn5yGzoYPjI
lease_duration     1h
lease_renewable    true
password           rKzEPbGPw-6mG19-4hKL
username           v-root-tester-BNUUEQrC8iyHGRDeK7Bu-1624377059

登录到mongodb查看新用户

docker exec -it mongodb mongo \
    --username v-root-tester-BNUUEQrC8iyHGRDeK7Bu-1624377059 \
    --password rKzEPbGPw-6mG19-4hKL

show dbs

UI 上的内容

Vault KV Secrets Engine – Version 2

以下两种方式都可以打开 kv engine

vault secrets enable -version=2 kv
vault secrets enable kv-v2

直接put 会出现403权限不足

/tmp $ vault kv put secret/hello foo=world
Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/internal/ui/mounts/secret/hello
Code: 403. Errors:

* preflight capability check returned 403, please ensure client's policies grant access to path "secret/hello/"
/tmp $ 

启用AppRole身份验证

vault auth enable approle

创建 policy.hcl

/tmp $ cat policy.hcl 
path "kv/helloworld/*" {
  capabilities = [ "create", "read", "delete" ]
}
/tmp $ 

使用 hcl文件创建权限

vault policy write helloworld-policy ./policy.hcl

/tmp $ export VAULT_TOKEN=s.YIhdm8s9jMRqCqgUP00YyJjI
/tmp $ cat policy.hcl 
path "kv/helloworld/*" {
  capabilities = [ "create", "read", "delete" ]
}
/tmp $ vault policy write helloworld-policy ./policy.hcl
Success! Uploaded policy: helloworld-policy
/tmp $ 
/tmp $ vault policy list
default
helloworld-policy
root
/tmp $ 
/tmp $ vault policy read helloworld-policy
path "kv/helloworld/*" {
  capabilities = [ "create", "read", "delete" ]
}
/tmp $ 

测试写入

/tmp $ vault kv put kv/helloworld foo=world
Key              Value
---              -----
created_time     2021-06-22T06:59:38.416440707Z
deletion_time    n/a
destroyed        false
version          1
/tmp $ 
/tmp $ vault kv put kv/helloworld/test test=testvalue
Key              Value
---              -----
created_time     2021-06-22T07:00:04.372497077Z
deletion_time    n/a
destroyed        false
version          1
/tmp $ 

测试 get

vault kv put kv/helloworld foo=world

测试删除

vault kv delete kv/helloworld/test

Vault Kubernetes HA安装

HA 安装主要是需要 consul

添加仓库 https://helm.releases.hashicorp.com

设置values

server.replicas = 1
ui.enabled = true （port 8500）

配置一个ingress

consul 的安装就结束了

使用以下 values 安装vault

server.ha.enabled = true
server.ha.replicas = 1
server.ui.enabled = true ( port 8200 )

可以看到storage是

storage "consul" {
  path = "vault"
  address = "HOST_IP:8500"
}

初始化

/ $ vault operator init
Unseal Key 1: rUrvo0abV/NT0K4x5ZaCCkbyVbsf88rEUAr1qn00fNrw
Unseal Key 2: vZgywK0ZBh+RI+lKYb4oqzRt7P4BWuRacLTHYEZdtRL6
Unseal Key 3: 8yH0vbYloMEVzaLxPQgCalzUS3uG823UEfzqcLrlLQiL
Unseal Key 4: 9dVwFNK5agYi4fyZW1x0RayhG2YG6eZSp4wLVeLsXPjE
Unseal Key 5: w+KUxcV48Q1ZzyvtipzRcl0yLeYAD0g1ihQBfvZ1NZix

Initial Root Token: s.YIhdm8s9jMRqCqgUP00YyJjI

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
/ $ 

unseal

export VAULT_TOKEN=s.YIhdm8s9jMRqCqgUP00YyJjI
vault operator unseal rUrvo0abV/NT0K4x5ZaCCkbyVbsf88rEUAr1qn00fNrw
vault operator unseal vZgywK0ZBh+RI+lKYb4oqzRt7P4BWuRacLTHYEZdtRL6
vault operator unseal 8yH0vbYloMEVzaLxPQgCalzUS3uG823UEfzqcLrlLQiL
vault operator unseal 9dVwFNK5agYi4fyZW1x0RayhG2YG6eZSp4wLVeLsXPjE
vault operator unseal w+KUxcV48Q1ZzyvtipzRcl0yLeYAD0g1ihQBfvZ1NZix
vault status

Vault Kubernetes 安装

Vault可以用来管理 秘钥、加密 数据 和 基于身份的权限验证 等

先安装longhorn作为持久化storage 提供 PV

添加仓库 https://helm.releases.hashicorp.com

设置values

dataStorage.size = 1Gi

vault 需要一个初始化动作

vault operator init


/ $ vault operator init
Unseal Key 1: 0knf13iA1JvSgIixWC5VMQKtcyHbrfBWoqxSWGKAbuCY
Unseal Key 2: UDcCwhlGcYpdRvYvfLD51hdFTh6QXIjxP3NCMATF6hys
Unseal Key 3: fpO7dILIfIMB6pfOy/IRb5U1zMpELWSWbjUXalwlhYkP
Unseal Key 4: oy4BRHPogCxaF231MfIQjwdF0GNZGcVJRDjwrNvW+yki
Unseal Key 5: sOwl2LUwW0ZZYlgh/T0H9Ajw4VHVwsKuRIMhdvwfGfKw

Initial Root Token: s.sJSDGrPe7rwnb5NDUs5VzPon

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
/ $ 

然后需要对秘钥 unseal

export VAULT_TOKEN=s.sJSDGrPe7rwnb5NDUs5VzPon
vault operator unseal 0knf13iA1JvSgIixWC5VMQKtcyHbrfBWoqxSWGKAbuCY
vault operator unseal UDcCwhlGcYpdRvYvfLD51hdFTh6QXIjxP3NCMATF6hys
vault operator unseal fpO7dILIfIMB6pfOy/IRb5U1zMpELWSWbjUXalwlhYkP
vault operator unseal oy4BRHPogCxaF231MfIQjwdF0GNZGcVJRDjwrNvW+yki
vault operator unseal sOwl2LUwW0ZZYlgh/T0H9Ajw4VHVwsKuRIMhdvwfGfKw
vault status

这样就完成了

可以看到默认使用了 file 作为 storage

Minio SSE-C 加密

SSE-C 加密是在客户端进行的加密，使用的是GPG

GPG（FNU Privary Fuard）是最流行的数据加密，数据签名工具软件

支持的算法：

公钥：RSA，ELG，DSA

对称加密：3DES，CAST5，BLOWFISH，AES，AES256

散列：MD5，SHA1，.. ..,SHA256，SHA512

基本用法：

加密操作： –symmetric 或 -c

解密操作：–decrypt 或 -d

在加密大文件的时候，性能损耗还是不小的

下面是配置文件

lizhe@ubuntu:~$ cat ~/.s3cfg 
# Setup endpoint
host_base = minio.lizhe.com
host_bucket = minio.lizhe.com
bucket_location = us-east-1
use_https = False

# Setup access keys
access_key = iZq4uq5dupOzCgNkThzU 
secret_key = qn8DeztD1Qe7tAOf2jr9qwpybxcZrklWsv58OhAt 

# Enable S3 v4 signature APIs
signature_v2 = False

gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = `123456`

lizhe@ubuntu:~$ 

创建桶

lizhe@ubuntu:~$ s3cmd mb s3://my-bucket-name
Bucket 's3://my-bucket-name/' created
lizhe@ubuntu:~$ s3cmd ls
2021-06-19 15:22  s3://my-bucket-name

上传文件

lizhe@ubuntu:~$ echo "helloworld" > hello.txt
lizhe@ubuntu:~$ s3cmd -e put hello.txt s3://my-bucket-name
upload: '/tmp/tmpfile-v1wNUCw8NqGPE6096L5X' -> 's3://my-bucket-name/hello.txt'  [1 of 1]
 87 of 87   100% in    0s     4.91 kB/s  done
lizhe@ubuntu:~$ s3cmd -e get s3://my-bucket-name/hello.txt /tmp -f
download: 's3://my-bucket-name/hello.txt' -> '/tmp/hello.txt'  [1 of 1]
 87 of 87   100% in    0s     4.03 kB/s  done
lizhe@ubuntu:~$ 

我们来看一下加密结果

可以看到

上传的文件 和 下载的文件 内容都是正常的，只是存储在服务器上的文件内容已经被加密了

我们尝试使用 gpg 命令来解密

注意我这里的密码是 `123456` 而不是 123456

`123456`

使用root用户同样可以解密

可以看到 这里默认使用的是 AES256 算法

Minio SSE-S3加密

MinIO使用密钥管理系统（KMS）支持SSE-S3（使用S3托管密钥的服务器端加密）

如果客户端请求SSE-S3，或启用了自动加密，则MinIO服务器会使用唯一的对象密钥对每个对象进行加密，该对象密钥受KMS管理的主密钥保护。由于开销极低，因此可以为每个应用程序和实例打开自动加密。

MinIO通过KES访问KMS，而不是直接访问KMS，每个MinIO群集均具有“自己的” KES实例。KES是一个用于高性能应用程序的无状态分布式密钥管理系统，它处理”其” MinIO群集发出的加密/解密请求，从而使KMS不必处理大量流量，KMS仅作为KES安全密钥的后台存储。

这里需要

1. vault服务器作为中央秘钥仓库
2. kes服务器作为连接minio和vault的中间件
3. minio服务器

在Kubernetes安装 vault

https://github.com/hashicorp/vault-helm

HA模式
server.ha.enabled = true
server.ha.replicas = 1
server.ui.enabled = true ( port 8200 )

单机节点
server.standalone.enabled = true

HA模式需要 consul 和 存储
https://helm.releases.hashicorp.com

server.replicas = 1

vault operator init

/ $ vault operator init
Unseal Key 1: 0mQk974OWKaSVTXGQ0Uzyj2bte1QM3AaOpSpjgazf9QZ
Unseal Key 2: 8/WoBHinaV0gMK6GsRuhnZWgolHpnBj2jkfAAW2khxTY
Unseal Key 3: gTYfW8gB9B3mI9c8oBYzWqks2Lkfu5cjC+MzWgdKU7sr
Unseal Key 4: FqF8oLlXh3zgnt5+4itdf83ecfmp9fSDC2hDSl4YG6kS
Unseal Key 5: ACAwtnxeD9Wgiq4VeQmQkhQg5IEXo6AYUgVV1u+Bq2g0

Initial Root Token: s.SkECG9cBaD259MLYh1Zvx9jv

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
/ $ 

对key解封

export VAULT_TOKEN=s.SkECG9cBaD259MLYh1Zvx9jv
vault operator unseal 0mQk974OWKaSVTXGQ0Uzyj2bte1QM3AaOpSpjgazf9QZ
vault operator unseal 8/WoBHinaV0gMK6GsRuhnZWgolHpnBj2jkfAAW2khxTY
vault operator unseal gTYfW8gB9B3mI9c8oBYzWqks2Lkfu5cjC+MzWgdKU7sr
vault status

vault 启动完成

export VAULT_TOKEN=s.SkECG9cBaD259MLYh1Zvx9jv

启用保险柜的K / V后端

vault secrets enable kv

启用AppRole身份验证

vault auth enable approle

创建文件
minio-kes-policy.hcl
path "kv/minio/*" {
  capabilities = [ "create", "read", "delete" ]
}

------------
cat > minio-kes-policy.hcl <<EOF
path "kv/minio/*" {
  capabilities = [ "create", "read", "delete" ]
}
EOF
# vault policy write minio-key-policy ./minio-kes-policy.hcl
# vault policy list
# vault policy read minio-key-policy

创建一个新的AppRole ID并将其绑定到策略

现在，我们需要创建一个新的AppRole ID并授予该ID特定的权限。 该应用程序（即KES服务器）将通过AppRole角色ID 和机密ID 向Vault进行身份验证，并且仅允许执行特定策略授予的操作。

因此，我们首先为KES服务器创建一个新角色：

vault write auth/approle/role/kes-role token_num_uses=0  secret_id_num_uses=0  period=5m

然后，我们将策略绑定到角色：

vault write auth/approle/role/kes-role policies=minio-key-policy

最后，我们从Vault请求AppRole角色ID和秘密ID。 一，角色ID：

vault read auth/approle/role/kes-role/role-id

然后是秘密ID：

vault write -f auth/approle/role/kes-role/secret-id

Longhorn backup

使用docker启动一个nfs服务器, 特别注意需要 nfsv4 版本，v3不可用

客户端安装 nfs-common （理论上不需要，如果有问题请安装）

sudo apt-get install nfs-common

使用 docker-compose.yml 启动一个 nfsv4 服务

mkdir -p /data/docker-volumes

version: "2.1"
services:
  # https://hub.docker.com/r/itsthenetwork/nfs-server-alpine
  nfs:
    image: itsthenetwork/nfs-server-alpine:12
    container_name: nfs
    restart: unless-stopped
    privileged: true
    environment:
      - SHARED_DIRECTORY=/data
    volumes:
      - /data/docker-volumes:/data
    ports:
      - 2049:2049

设置 nfs 位置

nfs://192.168.194.190:/

创建一个备份

可以看到在对应文件夹下出现了 pvc 的备份

Minio 安装

下面两个地址都可以获取 helm3的安装包

sudo helm repo add bitnami https://charts.bitnami.com/bitnami

另一个是

https://github.com/minio/charts

这里用的是第二种 官方地址

安装之前确认自己有可用的 storage class，这里用的是 longhorn

默认的存储空间大小是 500GB，所以如果你只是装来玩玩注意修改一下 values

persistence.size = 5Gi

找到对应的 秘钥

我这里的是

accesskey iZq4uq5dupOzCgNkThzU
secretkey qn8DeztD1Qe7tAOf2jr9qwpybxcZrklWsv58OhAt

如果只能使用命令行来查看

kubectl get secret --namespace minio minio -o jsonpath="{.data.accesskey}" | base64 -d
kubectl get secret --namespace minio minio -o jsonpath="{.data.secretkey}" | base64 -d

我们创建一个 ingress 端口 9000 就可以打开dashboard 了