HTTPRoute weighted backends with filters attached sends traffic to the wrong backend

[dprotaso]

Description:
When I add an weighted backend to an HTTPRoute I expect traffic to go to that backend with the correct filters applied

Instead I'm seeing traffic being sent to the wrong backend.

Repro steps:

1. I have a simple go server that prints request headers.

Go HTTP Sever

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"
)

func main() {
	target := os.Getenv("TARGET")
	if target == "" {
		target = "World"
	}

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		// Log all request headers
		fmt.Println("Request Headers:")
		for name, values := range r.Header {
			for _, value := range values {
				fmt.Printf("%s: %s\n", name, value)
			}
		}

		// Send response
		fmt.Fprintf(w, "Hello %s!\n", target)
	})

	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}

	log.Printf("Starting server on port %s...\n", port)
	if err := http.ListenAndServe(":"+port, nil); err != nil {
		log.Fatal(err)
	}
}

Next deploy two workloads using this go application

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: second
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: second
  template:
    metadata:
      labels:
        app: second
    spec:
      containers:
      - name: helloworld
        image: test-image
        env:
        - name: TARGET
          value: "second"
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: second
  namespace: default
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: second
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: first
    namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: first
  template:
    metadata:
      labels:
        app: first
    spec:
      containers:
      - name: helloworld
        image: test-image
        env:
        - name: TARGET
          value: "first"
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: first
  namespace: default
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: first

Next we configure two routes with equal weights. For each backend we add request headers to correctly identify the split that should occur.

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: sample
  namespace: default
  labels:
    app: sample
spec:
  parentRefs:
  - name: external-gateway
     namespace: default
  hostnames:
  - "sample.com"
  rules:
  - backendRefs:
    - name: first
      port: 80
      weight: 10
      filters:
      - type: RequestHeaderModifier
        requestHeaderModifier:
          set:
          - name: first
            value: first
    - name: second
      port: 80
      weight: 10
      filters:
      - type: RequestHeaderModifier
        requestHeaderModifier:
          set:
          - name: second
            value: second

After making requests to sample.com - you'll see the wrong headers end up in the wrong backend

See: First: first in the below logs

second-764d49d88f-bgmtc helloworld Request Headers:
second-764d49d88f-bgmtc helloworld User-Agent: curl/8.7.1
second-764d49d88f-bgmtc helloworld Accept: */*
second-764d49d88f-bgmtc helloworld X-Forwarded-For: 172.18.0.3
second-764d49d88f-bgmtc helloworld X-Forwarded-Proto: http
second-764d49d88f-bgmtc helloworld X-Envoy-External-Address: 172.18.0.3
second-764d49d88f-bgmtc helloworld X-Request-Id: 19f2119b-d88f-4be2-9806-b043528d3f84
second-764d49d88f-bgmtc helloworld First: first

Environment:
Using Envoy Gateway v1.3.0 testing on kind using cloud-provider-kind

Logs:

[dprotaso]

The issue here is that the route configuration after being weighted goes to the same cluster

          "route": {
           "weighted_clusters": {
            "clusters": [
             {
              "name": "httproute/default/sample/rule/0",
              "weight": 10,
              "request_headers_to_add": [
               {
                "header": {
                 "key": "first",
                 "value": "first"
                },
                "append_action": "OVERWRITE_IF_EXISTS_OR_ADD"
               }
              ]
             },
             {
              "name": "httproute/default/sample/rule/0",
              "weight": 10,
              "request_headers_to_add": [
               {
                "header": {
                 "key": "second",
                 "value": "second"
                },
                "append_action": "OVERWRITE_IF_EXISTS_OR_ADD"
               }
              ]
             }
            ]
           },
           "cluster_not_found_response_code": "INTERNAL_SERVER_ERROR",
           "upgrade_configs": [
            {
             "upgrade_type": "websocket"
            }
           ]
          },

Then the ClusterLoadAssignment does another weighted selection across endpoints.

     "endpoint_config": {
      "@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
      "cluster_name": "httproute/default/sample/rule/0",
      "endpoints": [
       {
        "locality": {
         "region": "httproute/default/sample/rule/0/backend/0"
        },
        "lb_endpoints": [
         {
          "endpoint": {
           "address": {
            "socket_address": {
             "address": "10.244.0.22",
             "port_value": 8080
            }
           },
           "health_check_config": {}
          },
          "health_status": "HEALTHY",
          "load_balancing_weight": 1
         }
        ],
        "load_balancing_weight": 10
       },
       {
        "locality": {
         "region": "httproute/default/sample/rule/0/backend/1"
        },
        "lb_endpoints": [
         {
          "endpoint": {
           "address": {
            "socket_address": {
             "address": "10.244.0.21",
             "port_value": 8080
            }
           },
           "health_check_config": {}
          },
          "health_status": "HEALTHY",
          "load_balancing_weight": 1
         }
        ],
        "load_balancing_weight": 10
       }
      ],
      "policy": {
       "overprovisioning_factor": 140
      }
     }

[dprotaso]

I think the right thing to do is to setup each Kubernetes Service as it's own 'cluster' and not a locality.region within a single cluster

[arkodg]

thanks for flagging this @dprotaso, looks like traffic splitting doesnt work correctly when backendRefs.filters is set because internally we use weighted clusters for this case

there are 2 issues

• naming the weighted clusters based off backendRef and not just rule
• making sure CLAs/endpoints map to each

[arkodg]

this is the relevant code that needs to be updated

gateway/internal/xds/translator/route.go

Line 238 in bc190aa

func buildXdsWeightedRouteAction(backendWeights *ir.BackendWeights, settings []*ir.DestinationSetting) *routev3.RouteAction {

[dprotaso]

Upstream k8s conformance test that exercises this issue - kubernetes-sigs/gateway-api#3604