之前写过一篇《记录session认证信息，并由logstash进行压缩处理》的笔记，其中篇尾也说明了该方案的缺点：日志占用空间大，logstash压力高，且因MD5值在Apisix外部计算得到，无法以其为key，进行接口限流操作。
后来需要以认证信息的MD5值为key，使用limit_rate插件进行接口访问限流，计算MD5值的操作只能在ingress（APISIX）层完成。
经过调研，结合运维技术栈，选择使用Go进行apisix-go-plugin-runner插件开发，官方开发教程
下图表左侧展示了 APISIX 的工作流程，而右侧的插件运行器负责运行用不同语言编写的外部插件。apisix-go-plugin-runner 就是这样一个支持 Go 语言的运行器。
结合APISIX的工作流程，我们需要在ext-plugin-pre-req进行rpc调用go-runner插件，并将计算后的MD5值赋值给新的header，传递给后面的 limit rate 等插件。
开发环境
- GO
# go version go version go1.22.6 linux/amd64 - APISIX
# docker images|grep apisix apache/apisix 2.15.3-centos 34a2aed251a5 17 months ago 439MB - 准备实验环境
# 1. 下载官方 docker compose 项目 $ git clone https://github.com/apache/apisix-docker.git $ cd apisix-docker/example # 2. 修改apisix配置，使其默认端口为80/443 $ cat docker-compose.yml ... apisix: image: apache/apisix:2.15.3-centos restart: always volumes: - ./apisix_log:/usr/local/apisix/logs - ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro depends_on: - etcd ##network_mode: host ports: - "9180:9180/tcp" - "9091:9091/tcp" - "9092:9092/tcp" - "80:80/tcp" # 修改这两项端口映射 - "443:443/tcp" networks: apisix: ... $ cd apisix_conf/ $ cat config.yaml ... apisix: node_listen: 80 # 修改该配置 enable_ipv6: false ssl: # 此段需要添加 enable: true listen: - port: 443 enable_http2: true ssl_protocols: "TLSv1.2 TLSv1.3" ssl_ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA" ... $ # 3. run docker compose $ docker-compose -p docker-apisix up -d # 4. check docker ps $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 895e8a3e8425 apache/apisix:2.15.3-centos "/docker-entrypoint.…" 6 hours ago Up 4 hours 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:9091-9092->9091-9092/tcp, :::9091-9092->9091-9092/tcp, 0.0.0.0:9180->9180/tcp, :::9180->9180/tcp docker-apisix_apisix_1 cdc4c422fcfb nginx:1.19.0-alpine "/docker-entrypoint.…" 6 hours ago Up 6 hours 0.0.0.0:9081->80/tcp, :::9081->80/tcp docker-apisix_web1_1 04f8874bf46a grafana/grafana:7.3.7 "/run.sh" 6 hours ago Up 6 hours 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp docker-apisix_grafana_1 add5b20273c1 bitnami/etcd:3.4.15 "/opt/bitnami/script…" 6 hours ago Up 6 hours 0.0.0.0:2379->2379/tcp, :::2379->2379/tcp, 2380/tcp docker-apisix_etcd_1 d9123f062765 prom/prometheus:v2.25.0 "/bin/prometheus --c…" 6 hours ago Up 6 hours 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp docker-apisix_prometheus_1 d9d159373372 apache/apisix-dashboard:2.13-alpine "/usr/local/apisix-d…" 6 hours ago Up 6 hours 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp docker-apisix_apisix-dashboard_1 2e835683b790 nginx:1.19.0-alpine "/docker-entrypoint.…" 6 hours ago Up 6 hours 0.0.0.0:9082->80/tcp, :::9082->80/tcp docker-apisix_web2_1
部署完成，可以通过 localhost:9000 访问 dashboard
默认配置文件在 apisix-docker/example/apisix_conf 目录下。
代码开发
- 代码结构
$ cd apisix-go-plugin-runner $ tree cmd cmd └── go-runner ├── main.go ├── main_test.go ├── plugins # 主要在plugins目录下开发 │ ├── md5_header.go │ ├── md5_header_test.go │ └── mock_utils.go └── version.go md5_header.go
主要代码文件，其中重点实现init()、Name()、ParseConf()、RequestFilter()方法package plugins import ( "crypto/md5" "encoding/hex" "encoding/json" "net/http" pkgHTTP "github.com/apache/apisix-go-plugin-runner/pkg/http" "github.com/apache/apisix-go-plugin-runner/pkg/log" "github.com/apache/apisix-go-plugin-runner/pkg/plugin" ) const md5HeaderName = "md5-header" // 注册插件 func init() { if err := plugin.RegisterPlugin(&Md5Header{}); err != nil { log.Fatalf("failed to register plugin %s: %s", md5HeaderName, err.Error()) } } // 定义Md5Header结构体，继承 plugin.DefaultPlugin type Md5Header struct { plugin.DefaultPlugin } // 定义Md5HeaderConfig结构体，为了从配置中读取 源header名称 ，和MD5计算后需赋值的 目标header名称 type Md5HeaderConfig struct { SrcHeader string `json:"src_header"` DstHeader string `json:"dst_header"` } func (*Md5Header) Name() string { return md5HeaderName } // 读取JSON配置，并反序列化为Md5HeaderConfig实例 func (p *Md5Header) ParseConf(in []byte) (interface{}, error) { conf := Md5HeaderConfig{} err := json.Unmarshal(in, &conf) if err != nil { log.Errorf("failed to parse config for plugin %s: %s", p.Name(), err.Error()) } return conf, err } // 处理Request的主要方法，MD5值计算在此实现 func (*Md5Header) RequestFilter(conf interface{}, _ http.ResponseWriter, r pkgHTTP.Request) { cfg := conf.(Md5HeaderConfig) srcHeaderValue := r.Header().Get(cfg.SrcHeader) if srcHeaderValue != "" { // 计算MD5 hashStr := md5.Sum([]byte(srcHeaderValue)) md5Str := hex.EncodeToString(hashStr[:]) // 输出其中的16位 r.Header().Set(cfg.DstHeader, md5Str[8:24]) } }- 单元测试，伪造request，对RequestFilter()进行测试
md5_header_test.go
package plugins import ( "github.com/stretchr/testify/require" "testing" ) func TestRequestFilter(t *testing.T) { in := []byte(`{"src_header": "Authorization", "dst_header": "Logged-Id"}`) p := &Md5Header{} conf, err := p.ParseConf(in) require.NoError(t, err) req := &mockHTTPRequest{body: []byte("hello apisix")} req.setHeader() req.Header().Set("Authorization", "eyJhbGciOini982xMiJ9.eyJzdWIiOiJzZWxsZXJfMTA1NzE2NDc3MTUxMjQwX3BjXzE3MjA0Mjc2OTQ2MzkiLCJvcyI6InBjIiwic2NvcGVzIjpbXSwiaW5kdX89aclUeXBlIjoyLCJsb2dpblR5cGUiOjEsImp0aSI6ImVjMWQwODhkLWI1Y2Etabcd0987YmFlLTU4Y2MzZjc0NWJiNSIsImlhdCI6MTcyMDQyNzY5NCwiZXhwIjabcdzMDE5Njk0fQ.xzVh7UgKlnm85sigabcdYooUDtMeRSmGuR15abcdefghoKNrcAOD0cuXSx8KQ2lWjz3ztWO3Upw2_3Y98J4-Dw") p.RequestFilter(conf, nil, req) require.Equal(t, "b951e9d3d754215c", req.Header().Get("Logged-Id")) }mock_utils.go
package plugins import ( "context" internalHTTP "github.com/apache/apisix-go-plugin-runner/internal/http" pkgHTTP "github.com/apache/apisix-go-plugin-runner/pkg/http" "net" "net/http" "net/url" ) // mockHTTPRequest implements pkgHTTP.Request type mockHTTPRequest struct { body []byte header *internalHTTP.Header } func (r *mockHTTPRequest) SetBody(body []byte) { r.body = body } func (r *mockHTTPRequest) setHeader() { r.header = &internalHTTP.Header{ Header: http.Header{}, } } func (*mockHTTPRequest) Args() url.Values { panic("unimplemented") } func (r *mockHTTPRequest) Body() ([]byte, error) { return r.body, nil } func (*mockHTTPRequest) Context() context.Context { panic("unimplemented") } func (r *mockHTTPRequest) Header() pkgHTTP.Header { return r.header } func (*mockHTTPRequest) ID() uint32 { panic("unimplemented") } func (*mockHTTPRequest) Method() string { panic("unimplemented") } func (*mockHTTPRequest) Path() []byte { panic("unimplemented") } func (*mockHTTPRequest) RespHeader() http.Header { panic("unimplemented") } func (*mockHTTPRequest) SetPath([]byte) { panic("unimplemented") } func (*mockHTTPRequest) SrcIP() net.IP { panic("unimplemented") } func (*mockHTTPRequest) Var(string) ([]byte, error) { panic("unimplemented") }
代码构建
借助Makefile能很方便进行测试和构建，但要注意此文件中没有定义交叉平台编译选项，为了避免不必要的问题，选择和实际生产环境同平台进行构建。
此处选用 CentOS7（Linux/Amd64）平台编译构建
- 提前安装go
https://go.dev/doc/install - 提前安装gcc
yum install -y gcc - 测试
$ CGO_ENABLED=1 make test - 构建
$ CGO_ENABLED=1 make build $ ls -l go-runner -rwxr-xr-x 1 root root 9424126 8月 8 13:11 go-runner
部署上线
- 首先需要将go-runner可执行文件放到apisix容器里
$ docker cp go-runner docker-apisix_apisix_1:/opt - 定义apisix配置，启用 go-runner
$ cat config.yaml #配置文件添加 ... ext-plugin: cmd: ["/opt/go-runner", "run"] $ docker-compose -p docker-apisix restart apisix #重启apisix - 在route中定义plugins配置，启用 md5-header 插件，并定义插件配置以传递给
Md5HeaderConfig{ "plugins": { "ext-plugin-pre-req": { // 通过ext-plugin-pre-req在apisix内置插件被调用前rpc调用 md5-header "_meta": { "disable": false }, "conf": [ { "name": "md5-header", "value": "{\"src_header\": \"Authorization\", \"dst_header\": \"Logged-Id\"}" // 定义md5-header的配置 } ] } } } - 可以通过 response-rewrite 插件将处理后的结果返回到响应头中，便于观察
{ "plugins": { "response-rewrite": { "disable": false, "headers": { "Authorization": "$http_authorization", "Logged-Id": "$http_logged_id" } } } } - 通过postman调用
日志记录
- log_format定义记录 http_logged_id 字段
access_log_format: "$http_x_forwarded_for $remote_addr \"-\" $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$upstream_addr\" $upstream_response_time $request_time \"$http_x_request_tag\" \"$http_Traceparent\" \"$http_logged_id\"" - logstash定义grok表达式，转储成 user_agent.logged_id 字段
filter { grok { patterns_dir => ["/usr/share/logstash/config/patterns"] match => [ "message", "(?:%{NGINX_ADDRESS_LIST:nginx.access.remote_ip_list}) \"-\" (-|%{DATA:user.name}) \[%{HTTPDATE:nginx.access.time}\] (%{NGINX_HOST})? \"%{DATA:nginx.access.info}\" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} \"(-|%{DATA:http.request.referrer})\" \"(-|%{DATA:user_agent.original})\" \"(-|%{HOSTPORT:nginx.upstream.address})\" (-|%{NUMBER:nginx.upstream.response.time:float}) (-|%{NUMBER:nginx.request.time:float}) \"(-|%{DATA:http.header.x_request_tag})\"( \"00-%{DATA:Trace.Id}-%{DATA:Trace.SpanId}-01\")( \"(-|%{DATA:user_agent.logged_id})\")?" ] } }
此方案在ingress层进行MD5值计算，并由ext-plugin-pre-req控制在apisix内置插件前调用md5-header，可以保证内置插件可获取 $http_logged_id 变量，在apisix的很多traffic类型插件中，需要指定一个key进行规则限制。此变量可以在未进行consumer定义时，较简单直接的以用户粒度定位已登录的用户，而不仅仅是使用client ip进行定位。
评论区