bug: limit-count rejections are not captured in the metrics

[indrekj]

Current Behavior

When the limit-count plugin rejects a request, there will be a log message, e.g.

2025/02/25 08:16:10 [warn] 49#49: *42708330 [lua] plugin.lua:1153: run_plugin(): limit-count exits with http status code 429, client: [redacted], server: _, request: "POST /some-address HTTP/1.1", host: "[redacted]", referrer: "[redacted]"

However, these "429" response status codes aren't in the metrics. I've tried it with both the datadog plugin and the prometheus plugin.

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

1. Enable limit-count plugin
2. Set up a route with strict rate limiting (set rejected_code to 429)
3. Enable prometheus plugin
4. Send a bunch of requests to the route so that it gets limited
5. Query the response metrics (/apisix/prometheus/metrics). Expected: 429 responses, actual: no 429 responses

Environment

• Installed through the Apache APISIX Helm Chart
• APISIX version (run apisix version): 3.7.0
• Operating system (run uname -a): Linux apisix-76f84f94f7-blqrb 5.10.228-219.884.amzn2.x86_64 #1 SMP Wed Oct 23 17:17:00 UTC 2024 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V):

nginx version: openresty/1.21.4.2
built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
built with OpenSSL 1.1.1s  1 Nov 2022 (running with OpenSSL 1.1.1w  11 Sep 2023)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_RUNTIME_VER=1.0.1 -DNGX_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_HTTP_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.2 --add-module=../echo-nginx-module-0.63 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.33 --add-module=../ngx_lua-0.10.25 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.34 --add-module=../array-var-nginx-module-0.06 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.13 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../ngx_multi_upstream_module-1.1.1 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../apisix-nginx-module-1.15.0 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../apisix-nginx-module-1.15.0/src/stream --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../apisix-nginx-module-1.15.0/src/meta --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../wasm-nginx-module-0.6.5 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../lua-var-nginx-module-v0.5.3 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../grpc-client-nginx-module-v0.4.4 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module

[indrekj]

This is not a bug actually, finally figured it out. We were overriding the error page handler:

      httpSrv: |
        access_log /dev/stdout main buffer=16384 flush=3 if=$loggable;

        location @error {
          default_type 'text/html';
          echo '<html><head><title>$status $status_text</title></head><body><center><h1>$status $status_text</h1></center></html>';
        }
        error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 421 422 423 424 425 426 428 429 431 451 500 501 502 503 504 505 506 507 508 510 511 @error;