cn

注入中断

可以通过向服务调用过程注入中断的方式，测试服务通信中断的后果

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx-default-v2
  namespace: version
spec:
  hosts:
  - "hello.lizhe.com"
  gateways:
  - nginx-gateway
  http:
  - route:
    - destination:
        host: nginx-service.version.svc.cluster.local
        subset: v2
    fault:
      delay:
        fixedDelay: 1s
        percentage:
          value: 100
      abort:
        httpStatus: 500
        percentage:
          value: 50

注入延迟

通过向服务调用注入延迟，可以测试服务通信不畅的后果。

1. 注入延迟

virtualservice.networking.istio.io/nginx-default-v2 configured
root@istio1:~/ver# cat vs.yaml 
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx-default-v2
  namespace: version
spec:
  hosts:
  - "hello.lizhe.com"
  gateways:
  - nginx-gateway
  http:
  - route:
    - destination:
        host: nginx-service.version.svc.cluster.local
        subset: v2
    fault:
      delay:
        fixedDelay: 7s
        percentage:
          value: 100

服务熔断

没有使用 熔断 的 DestinationRule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: nginx-desrule 
  namespace: version
spec: 
  host: nginx-service.version.svc.cluster.local
  subsets:
  - name: v1 
    labels: 
      version: v1
  - name: v2
    labels: 
      version: v2
  - name: v3
    labels: 
      version: v3

可以看到 无论是 1 连接并发，还是 10 连接并发 都没有错误返回 （ 非 200 ）

服务熔断是一种保护措施，在服务实例无法正常提供服务的情况下，将其从负载均衡器中移除，不再分配任务

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: nginx-desrule 
  namespace: version
spec: 
  host: nginx-service.version.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutiveErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100
  subsets:
  - name: v1 
    labels: 
      version: v1
  - name: v2
    labels: 
      version: v2
  - name: v3
    labels: 
      version: v3

wrk -c 1 -t 1 http://hello.lizhe.com:31380/helloworld.html

加入了熔断之后，1 连接时候仍然是正常的，10连接并发的时候 75640 次请求中，出现了 71660 次错误

核心配置中

  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutiveErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100

HTTP 和 TCP 连接池大小都是 1

maxConnections: 1
http1MaxPendingRequests: 1

每连接最大请求数

maxRequestsPerConnection: 1

只允许出错 1 次 （此方法已经过时）

consecutiveErrors: 1

没1秒做一次请求计数

interval: 1s

发生故障的的 pod 最少在被移除 3 分钟之后才能再次加入负载均衡池

baseEjectionTime: 3m

可以从负载均衡池中移除 100% 的 pod

maxEjectionPercent: 100

maxRequestsPerConnection 过时后 新的 配置

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: nginx-desrule 
  namespace: version
spec: 
  host: nginx-service.version.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutive5xxErrors: 1
      consecutiveGatewayErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100
  subsets:
  - name: v1 
    labels: 
      version: v1
  - name: v2
    labels: 
      version: v2
  - name: v3
    labels: 
      version: v3

outlier detection consecutive errors is deprecated, use consecutiveGatewayErrors or consecutive5xxErrors instead
destinationrule.networking.istio.io/nginx-desrule configured

Istio配置文件中的host

不使用gateway的情况

如果没有gw，那么只需要 DestinationRule 和 VirtualService 就可以了

在DestinationRule中，host 需要对应的是 Kubernetes中的service

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: nginx-desrule 
  namespace: lizhe
spec: 
  host: nginx-service
  subsets:
  - name: v1 
    labels: 
      version: v1
  - name: v2
    labels: 
      version: v2
  - name: v3
    labels: 
      version: v3

对应的VirtualService

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx-default
  namespace: version
spec: 
  hosts:
  - nginx-service
  http:
  - route: 
    - destination: 
        host: nginx-service
        subset: v2

此时使用以下3种内部dns都可以访问

curl nginx-service/helloworld.html
curl nginx-service.version/helloworld.html
curl nginx-service.version.svc/helloworld.html

使用Gateway的情况

创建一个 Gateway

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: nginx-gateway
  namespace: version
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*.lizhe.com"

再创建DestinationRule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: nginx-desrule 
  namespace: version
spec: 
  host: nginx-service.version.svc.cluster.local
  subsets:
  - name: v1 
    labels: 
      version: v1
  - name: v2
    labels: 
      version: v2
  - name: v3
    labels: 
      version: v3

最后用一个VirtualService把它们两个粘在一起

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx-default-v2
  namespace: version
spec:
  hosts:
  - "hello.lizhe.com"
  gateways:
  - nginx-gateway
  http:
  - route:
    - destination:
        host: nginx-service.version.svc.cluster.local
        subset: v2

对于service可以使用全量名

下面是一个错误例子

不过可以使用 通配符 *

root@istio1:~/yamls# cat dr.yaml 
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: nginx-desrule 
  namespace: version
spec: 
  host: nginx-service.version.svc.cluster.local
  subsets:
  - name: v2
    labels: 
      version: v2

root@istio1:~/yamls# cat vs.yaml 
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx-default
  namespace: version
spec: 
  hosts:
  - "*.version.svc.cluster.local"
  http:
  - route: 
    - destination: 
        host: nginx-service.version.svc.cluster.local
        subset: v2

root@istio1:~/yamls# kubectl apply -f ./
destinationrule.networking.istio.io/nginx-desrule unchanged
virtualservice.networking.istio.io/nginx-default configured
root@istio1:~/yamls# 

gw和vs匹配

使用2个 VirtualService ，一个持有 Gateway，一个不持有Gateway

Traefik sticky session

Traefik的模型是

route -> traefik service -> k8s service -> pod

这里构建了3个pod，分别对应3个 k8s service

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoingressroute
  namespace: who
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`who.lizhe.com`)
    kind: Rule
    services:
    - name: whots
      kind: TraefikService

apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:
  name: whots
  namespace: who
spec:
  weighted:
    services:
      - name: whoami1
        weight: 3  # 定义权重
        port: 80
        kind: Service  # 可选，默认就是 Service
      - name: whoami2
        weight: 3  # 定义权重
        port: 80
        kind: Service  # 可选，默认就是 Service
      - name: whoami3
        weight: 3  # 定义权重
        port: 80
        kind: Service  # 可选，默认就是 Service

修改ts

apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:
  name: whots
  namespace: who
spec:
  weighted:
    sticky:
      cookie:
        name: traefiktoken
    services:
      - name: whoami1
        weight: 3  # 定义权重
        port: 80
        kind: Service  # 可选，默认就是 Service
      - name: whoami2
        weight: 3  # 定义权重
        port: 80
        kind: Service  # 可选，默认就是 Service
      - name: whoami3
        weight: 3  # 定义权重
        port: 80
        kind: Service  # 可选，默认就是 Service

这种模型上，很难受的一个点是，它的负载均衡不是落在 pod 上的，而是落在 service 上，这样pod在伸缩之后，你还需要修改对应的 service，实际应用没有 ingress 那么方便

更新

不使用 traefik service 的话，可以直接将 sticky 写在 ingress route 上

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: backendingress
  namespace: backend
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`nginx.lizhe.com`) && PathPrefix(`/backend`)
      kind: Rule
      services:
        - name: myservice
          port: 8080
          sticky:
            cookie:
              httpOnly: true
              name: mytickytoken
      middlewares:
        - name: backend-stripprefix

上面的例子也可以直接使 IngressRoute 指向 kubernetes的service

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoingressroute
  namespace: lizhe
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`who.lizhe.com`)
    kind: Rule
    services:
    - name: whoami
      port: 80
      sticky:
        cookie:
          httpOnly: true
          name: lizhetoken
          

Ingress sticky session

测试用的 deployment

测试用的 ingress

使用的镜像是 containous/whoami ， 显示出来的效果是这样的

把 pod 数改成 3 之后，很容易就可以得到下面的情况，请求会在3个pod之间轮训

添加annotation就可以了

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "ingresscoookie"
    nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"

auth服务dns

在 ForwardAuth Middleware 中我用来做 auth 鉴权服务的 golang 应用是启动在 vscode 上的，是一个单机服务

在使用过程中有一个小问题，本地dns无法正确生效

在 /etc/hosts中配置的dns无法让 Kubernetes 中的 traefik forward middleware 连接到 golang auth 服务，替换成IP之后故障消除

192.168.194.181	authgo.lizhe.com

然后修改 middleware

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: lizhe-auth
  namespace: lizhe
spec:
  forwardAuth:
    address: http://authgo.lizhe.com:3080/auth

curl http://authgo.lizhe.com:3080/auth 可以工作，可惜nginx无法工作

添加一个外部的公网 dns
auth.diynocap.com

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: lizhe-auth
  namespace: lizhe
spec:
  forwardAuth:
    address: http://authgo.diynocap.com:3080/auth

可惜这样还是会得到一个 500

auth服务水平伸缩

本节中我们的目的是把 golang auth 应用安装进 Kubernetes，以提供水平伸缩和编排

之前已经打过docker image

libaibai/lizhe-auth:v0.1

先测试一下clusterip

然后我们来替换成集群内部 dns

curl -I authservice.auth.svc:3080/auth

也可以正常工作

ForwardAuth Middleware

https://doc.traefik.io/traefik/middlewares/forwardauth/

创建中间件

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: lizhe-auth
spec:
  forwardAuth:
    address: http://192.168.194.181:3080/auth

我会使用 下面的nginx作为测试用的 真实服务

分别创建 traefik service 和 ingressroute
这里为了测试我们没有添加任何中间件

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: authingressroute
  namespace: lizhe
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`auth.lizhe.com`)
    kind: Rule
    services:
    - name: authts
      kind: TraefikService

kind: TraefikService
metadata:
  name: authts
  namespace: lizhe
spec:
  weighted:
    services:
      - name: nginxsvc
        port: 80
        kind: Service

然后我们加入中间件

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: authingressroute
  namespace: lizhe
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`auth.lizhe.com`)
    kind: Rule
    services:
    - name: authts
      kind: TraefikService
    middlewares:
      - name: lizhe-auth

再次访问 测试用 nginx

这里可以看到header中存在很多信息

我们用curl命令测试在header 中添加一些 token

直接关闭 auth 服务器，会得到 500 错误

这里为了观察被访问真实服务（刚才的nginx）是否能正确获得 request header 的token，我把nginx替换成了我们用来显示 request header 的 golang 程序

可以看到 request header 中的token被正确的在整个调用链上传递，至此，POC结束

Golang Auth服务

以下golang程序会输出接收到的rquest header

package main

import (
	"log"
	"net/http"

	handler "lizhe-auth/internal/handler"
)

func main() {
	http.HandleFunc("/auth", handler.WithArgHandler)
	log.Println("server started on 3080")
	http.ListenAndServe(":"+"3080", nil)
}

package handler

import (
	"log"
	"net/http"
)

func WithArgHandler(writer http.ResponseWriter, request *http.Request) {
	log.Println(request.Header)
}

构建Dockerfile

FROM golang:alpine as golang   

RUN mkdir -p /root/auth
RUN mkdir -p /root/auth/output
COPY ./ /root/auth

WORKDIR /root/auth/output
RUN go build /root/auth/cmd/main.go


FROM alpine as alpine

RUN mkdir -p /root/auth
COPY --from=golang --chown=root:root /root/auth/output /root/auth

CMD /root/auth/main 

然后build镜像

docker build -t libaibai/lizhe-auth:v0.1 .
docker run -i -t --name test -p 3080:3080 libaibai/lizhe-auth:v0.1