go-pdk plugins break Kong 3.9.0 #14260

dpiddock · 2025-02-11T16:53:20Z

Is there an existing issue for this?

I have searched the existing issues

Kong version (`$ kong version`)

3.9.0

Current Behavior

Kong 3.9.0 DBless returns errors on the root admin page when a go plugin is loaded, if there is no existing configuration. This is terminal to Kong on Kubernetes as it causes KIC to fail to configure Kong instances.

It does not happen every request with the MVP but it does in our larger live Kong plugin setup in Kubernetes.
Error in Kong output:

2025/02/11 16:25:51 [error] 1411#0: *1292 [lua] api_helpers.lua:558: handle_error(): /usr/local/share/lua/5.1/lapis/application.lua:440: /usr/local/share/lua/5.1/kong/api/api_helpers.lua:329: body encoding failed while flushing response: Cannot serialise cdata: type not supported
stack traceback:
	[C]: in function 'error'
	/usr/local/share/lua/5.1/kong/pdk/response.lua:732: in function 'fn'
	/usr/local/share/lua/5.1/kong/api/api_helpers.lua:329: in function </usr/local/share/lua/5.1/kong/api/api_helpers.lua:307>

stack traceback:
	[C]: in function 'error'
	/usr/local/share/lua/5.1/lapis/application.lua:440: in function 'handler'
	/usr/local/share/lua/5.1/lapis/application.lua:186: in function 'resolve'
	/usr/local/share/lua/5.1/lapis/application.lua:217: in function </usr/local/share/lua/5.1/lapis/application.lua:215>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/lapis/application.lua:215: in function 'dispatch'
	/usr/local/share/lua/5.1/lapis/nginx.lua:231: in function 'serve'
	/usr/local/share/lua/5.1/kong/init.lua:1878: in function 'admin_content'
	content_by_lua(nginx-kong.conf:355):2: in main chunk, client: 172.22.0.1, server: kong_admin, request: "GET / HTTP/1.1", host: "localhost:8001"
2025/02/11 16:25:51 [debug] 1411#0: *1292 [lua] init.lua:24: poll(): worker-events: emulate poll method
172.22.0.1 - - [11/Feb/2025:16:25:51 +0000] "GET / HTTP/1.1" 500 42 "-" "curl/8.12.0

Client receives a 500 and:

{"message":"An unexpected error occurred"}

KIC logs:

Error: unable to build kong api client(s): Kong Gateway version is not supported: getting Kong version: failed fetching Kong client root: HTTP status 500 (message: "An unexpected error occurred")

Expected Behavior

Kong returns valid json document when queried on admin interface /, as it does for version 3.8.0.

Steps To Reproduce

Download MVP: https://gist.github.com/dpiddock/d2bfe437f48341bd33a5f616077cba8a
docker compose up -d --build
curl localhost:8001/ (may not error every time)

Anything else?

The exact same plugins and configuration work correctly with Kong 3.8.0.

The text was updated successfully, but these errors were encountered:

jocadu · 2025-02-27T21:09:47Z

Hi,
I am observing the same issue. The KIC logs show:

2025-02-27T20:30:12Z info Diagnostics server disabled {"v": 0}
2025-02-27T20:30:12Z info setup Starting controller manager {"v": 0, "release": "3.4.1", "repo": "https://github.com/Kong/kubernetes-ingress-controller.git", "commit": "9e84f95865ce430cae38f0524 ef0e4bfaf61c55e"}
2025-02-27T20:30:12Z info setup The ingress class name has been set {"v": 0, "value": "kong"}
2025-02-27T20:30:12Z info setup Getting enabled options and features {"v": 0}
2025-02-27T20:30:12Z info setup Getting the kubernetes client configuration {"v": 0}
W0227 20:30:12.265236 1 client_config.go:667] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2025-02-27T20:30:12Z info setup Starting standalone health check server {"v": 0}
2025-02-27T20:30:12Z info setup Getting the kong admin api client configuration {"v": 0}
W0227 20:30:12.265435 1 client_config.go:667] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
Error: unable to build kong api client(s): Kong Gateway version is not supported: getting Kong version: failed fetching Kong client root: HTTP status 500 (message: "An unexpected error occurred")

The issue is NOT showing in another deployment of Kong 3.9.0 + KIC 3.4.1 where I do not have any Go plugins

kbajy · 2025-02-27T22:37:00Z

Hi,

I'm experiencing the same issue when upgrading from 3.8.0 to 3.9.0.

ricfeatherstone-sky · 2025-03-10T14:06:38Z

I have the same issue, identical error message.

Additionally, to get this far I had to update the values to add runAsUser: 1001 to the containerSecurityContext block.

docker run kong/kong:3.8 getent passwd | grep kong returns kong:x:1000:1000:Kong default user:/home/kong:/bin/sh

docker run kong/kong:3.9 getent passwd | grep kong returns kong:x:1001:1001:Kong default user:/home/kong:/bin/sh

Checking another issue with plugins, I found this, fix going out in 3.10

BhautikChudasama · 2025-03-15T06:31:47Z

I am facing similar issue.
+1

ricardoalcantara · 2025-04-03T13:23:28Z

I am having a similar issue too with this 3.9.0 version. But I am trying to use JS Custom plugin in my case.

kong  | 2025/04/03 13:07:01 [error] 1405#0: *9034 [lua] api_helpers.lua:558: handle_error(): /usr/local/share/lua/5.1/lapis/application.lua:440: /usr/local/share/lua/5.1/kong/api/api_helpers.lua:329: body encoding failed while flushing response: Cannot serialise cdata: type not supported
kong  | stack traceback:
kong  |         [C]: in function 'error'
kong  |         /usr/local/share/lua/5.1/kong/pdk/response.lua:732: in function 'fn'
kong  |         /usr/local/share/lua/5.1/kong/api/api_helpers.lua:329: in function </usr/local/share/lua/5.1/kong/api/api_helpers.lua:307>
kong  | 
kong  | stack traceback:
kong  |         [C]: in function 'error'
kong  |         /usr/local/share/lua/5.1/lapis/application.lua:440: in function 'handler'
kong  |         /usr/local/share/lua/5.1/lapis/application.lua:186: in function 'resolve'
kong  |         /usr/local/share/lua/5.1/lapis/application.lua:217: in function </usr/local/share/lua/5.1/lapis/application.lua:215>
kong  |         [C]: in function 'xpcall'
kong  |         /usr/local/share/lua/5.1/lapis/application.lua:215: in function 'dispatch'
kong  |         /usr/local/share/lua/5.1/lapis/nginx.lua:231: in function 'serve'
kong  |         /usr/local/share/lua/5.1/kong/init.lua:1878: in function 'admin_content'
kong  |         content_by_lua(nginx-kong.conf:356):2: in main chunk, client: 172.24.0.1, server: kong_admin, request: "GET / HTTP/1.1", host: "localhost:8001", referrer: "http://localhost:8002/"
kong  | 172.24.0.1 - - [03/Apr/2025:13:07:01 +0000] "GET / HTTP/1.1" 500 42 "http://localhost:8002/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0"
kong  | 2025/04/03 13:07:01 [error] 1405#0: *9034 [lua] api_helpers.lua:558: handle_error(): /usr/local/share/lua/5.1/lapis/application.lua:440: /usr/local/share/lua/5.1/kong/api/api_helpers.lua:329: body encoding failed while flushing response: Cannot serialise cdata: type not supported
kong  | stack traceback:
kong  |         [C]: in function 'error'
kong  |         /usr/local/share/lua/5.1/kong/pdk/response.lua:732: in function 'fn'
kong  |         /usr/local/share/lua/5.1/kong/api/api_helpers.lua:329: in function </usr/local/share/lua/5.1/kong/api/api_helpers.lua:307>
kong  | 
kong  | stack traceback:
kong  |         [C]: in function 'error'
kong  |         /usr/local/share/lua/5.1/lapis/application.lua:440: in function 'handler'
kong  |         /usr/local/share/lua/5.1/lapis/application.lua:186: in function 'resolve'
kong  |         /usr/local/share/lua/5.1/lapis/application.lua:217: in function </usr/local/share/lua/5.1/lapis/application.lua:215>
kong  |         [C]: in function 'xpcall'
kong  |         /usr/local/share/lua/5.1/lapis/application.lua:215: in function 'dispatch'
kong  |         /usr/local/share/lua/5.1/lapis/nginx.lua:231: in function 'serve'
kong  |         /usr/local/share/lua/5.1/kong/init.lua:1878: in function 'admin_content'
kong  |         content_by_lua(nginx-kong.conf:356):2: in main chunk, client: 172.24.0.1, server: kong_admin, request: "GET / HTTP/1.1", host: "localhost:8001", referrer: "http://localhost:8002/"
kong  | 172.24.0.1 - - [03/Apr/2025:13:07:01 +0000] "GET / HTTP/1.1" 500 42 "http://localhost:8002/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0"

I am running it using docker

Dockerfile

FROM kong:3.9.0

USER root
RUN apt update
RUN apt install -y nodejs npm python3 make g++
RUN npm install -g kong-pdk

And the docker compose configuration is

volumes:
  db: {}

services:
  kong-database:
    image: postgres:14
    container_name: kong-database
    environment:
      POSTGRES_USER: kong
      POSTGRES_DB: kong
      POSTGRES_PASSWORD: kongpass
    ports:
      - "5432:5432"
    networks:
      - kong-net
    restart: unless-stopped
    volumes:
      - db:/var/lib/postgresql/data

  kong:
    build: .
    container_name: kong
    volumes:
      - ./js-plugins:/usr/local/kong/js-plugins
    environment:
      KONG_DATABASE: postgres
      KONG_PG_HOST: kong-database
      KONG_PG_USER: kong
      KONG_PG_PASSWORD: kongpass
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_ADMIN_LISTEN: 0.0.0.0:8001
      KONG_ADMIN_GUI_URL: http://localhost:8002
      KONG_PLUGINSERVER_NAMES: js
      KONG_PLUGINSERVER_JS_SOCKET: /usr/local/kong/js_pluginserver.sock
      KONG_PLUGINSERVER_JS_START_CMD: /usr/local/bin/kong-js-pluginserver -v --plugins-directory /usr/local/kong/js-plugins
      KONG_PLUGINSERVER_JS_QUERY_CMD: /usr/local/bin/kong-js-pluginserver --plugins-directory /usr/local/kong/js-plugins --dump-all-plugins
    ports:
      - "8000:8000"
      - "8443:8443"
      - "8001:8001"
      - "8444:8444"
      - "8002:8002"
      - "8445:8445"
      - "8003:8003"
      - "8004:8004"
    networks:
      - kong-net
    depends_on:
      - kong-database
    restart: unless-stopped

  redis:
    image: redis:7.0
    container_name: kong-redis
    ports:
      - "6380:6379"
    networks:
      - kong-net
    restart: unless-stopped

networks:
  kong-net:
    driver: bridge

Thank you.

sgrzemski · 2025-04-14T07:34:09Z

Same goes with my 3.8.0 to 3.9.0. I'm running Kong on K8s with DB (PostgreSQL) and custom plugins, but this issue looks like a pure PDK stuff. Bumping for visibility.
@windmgc, @liverpool8056, @ADD-SP, @Oyami-Srk, @lhanjian

vtty0 · 2025-04-14T07:45:11Z

+1

windmgc · 2025-04-14T08:39:03Z

Hi, please check if this fixes your issue in kong gateway 3.10: Kong/go-pdk#226 (comment)

xianghai2 added area/kubernetes Issues where Kong is running on top of Kubernetes area/ingress-controller Issues where Kong is running as a Kubernetes Ingress Controller labels Feb 12, 2025

Tchekda mentioned this issue Apr 22, 2025

fix(core): separate pluginserver runtime data from kong configuration #14430

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

go-pdk plugins break Kong 3.9.0 #14260

go-pdk plugins break Kong 3.9.0 #14260

dpiddock commented Feb 11, 2025

jocadu commented Feb 27, 2025

kbajy commented Feb 27, 2025

ricfeatherstone-sky commented Mar 10, 2025 •

edited

Loading

BhautikChudasama commented Mar 15, 2025

ricardoalcantara commented Apr 3, 2025

sgrzemski commented Apr 14, 2025

vtty0 commented Apr 14, 2025

windmgc commented Apr 14, 2025

go-pdk plugins break Kong 3.9.0 #14260

go-pdk plugins break Kong 3.9.0 #14260

Comments

dpiddock commented Feb 11, 2025

Is there an existing issue for this?

Kong version ($ kong version)

Current Behavior

Expected Behavior

Steps To Reproduce

Anything else?

jocadu commented Feb 27, 2025

kbajy commented Feb 27, 2025

ricfeatherstone-sky commented Mar 10, 2025 • edited Loading

BhautikChudasama commented Mar 15, 2025

ricardoalcantara commented Apr 3, 2025

sgrzemski commented Apr 14, 2025

vtty0 commented Apr 14, 2025

windmgc commented Apr 14, 2025

Kong version (`$ kong version`)

ricfeatherstone-sky commented Mar 10, 2025 •

edited

Loading