Session invalidation when scaling Kratos replicas in Kubernetes

[treylade]

Session invalidation when scaling Kratos replicas in Kubernetes

Environment

• Ory Kratos version: v1.3.1
• Kubernetes deployment using official Helm chart (version 0.50.7)
• Database: PostgreSQL
• Service Mesh: Linkerd
• CNI: Flannel
• External Access: Cloudflare Tunnels (no K8s Ingress)
• Deployment Scale: 3 replicas

Description

When scaling Ory Kratos from 1 to multiple replicas in a Kubernetes environment, sessions become invalid intermittently. While the root cause is not yet determined, one hypothesis is that it might be related to session cache inconsistency between replicas, though this is purely speculative and needs validation.

Steps to Reproduce

1. Deploy Ory Kratos using the official Helm chart
2. Configure custom NGINX proxy for HTTPS termination
3. Scale deployment to multiple replicas
4. Authenticate user through frontend application
5. Make subsequent API calls using the obtained session

Current Behavior

• With 1 replica: Sessions work perfectly
• With >1 replicas: Random "No valid session found" errors occur during:
  • Direct Frontend-to-Kratos via Cloudflare Tunnel and NGINX Proxy obtained ory_kratos_session (no issue)
  • Subsequent internal Service-to-Service communication via internal DSN of Kratos Public / Admin Service API forwarding the initially obtained ory_kratos_session (no valid session found error comes up)

Expected Behavior

Sessions should remain valid regardless of which replica handles the request.

Configuration

# Relevant parts of values.yaml
replicaCount: 3

deployment:
  annotations:
    linkerd.io/inject: enabled

kratos:
  config:
    serve:
      public:
        base_url: https://auth.example.org/
        cors:
          enabled: true
          allowed_origins:
            - https://app.example.org
          allowed_methods:
            - POST
            - GET
            - PUT
            - PATCH
            - DELETE
          allowed_headers:
            - Authorization
            - Cookie
            - Content-Type
          allow_credentials: true

    session:
      cookie:
        persistent: true
      lifespan: 24h

Service architecture:

Frontend -> Cloudflare Tunnel -> NGINX Proxy -> Kratos Public API (internal port 80)
Quart Microservices -> Internal cluster DNS -> Kratos Public & Admin APIs
- Public API: For session validation, user data retrieval
- Admin API: For privileged operations (internal only)

Additional Context

1. All services (including NGINX proxies) are replicated 3 times
2. Session is stored in PostgreSQL database
3. While all data is properly stored in PostgreSQL, the exact cause of session invalidation is unclear. The in-memory session caching hypothesis is just one possible explanation that needs investigation
4. No Kubernetes Ingress is used and shall not be used; all external traffic routes through Cloudflare Tunnels (thus configuring Ingress via Ory Kratos values.yaml is not an option)

Attempted Solutions

1. Implemented session affinity at the Service level:

sessionAffinity: ClientIP
sessionAffinityConfig:
  clientIP:
    timeoutSeconds: 10800  # 3 hours

2. Ensured proper pod distribution using topology spread constraints and anti-affinity rules

Questions

Has anyone successfully run Ory Kratos with multiple replicas in a Kubernetes environment? We're experiencing session invalidation issues when scaling beyond a single replica, and would greatly appreciate any insights or recommendations from the community regarding horizontal scaling of Kratos.

Relevant Links

• Similar issue discussion
• Official K8s deployment documentation

Config files / Logs

Please advise what further config files and logs would be helpful, and I'll provide them.

[aeneasr]

Sounds like you’re not using the database but instead SQLite, or have some secrets misconfigured. But definitely a config issue on your end!

[treylade]

I am using a Postgres Cluster via Crunchy Postgres Operator which allows me to configure HA and offsite backups.
I configured the DSN to point to the Primary Database Instance.

Do you see any issues with that setup? What would be the suggested way?

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: ory-db
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  shutdown: false
  postgresVersion: 16
  databaseInitSQL:
    key: init.sql
    name: ory-db-init-sql
  instances:
    - name: postgres
      replicas: 3
      resources:
        requests:
          memory: "2Gi"
          cpu: "1"
        limits:
          memory: "3Gi"
          cpu: "2"
      dataVolumeClaimSpec:
        storageClassName: longhorn
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: "5Gi"
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  postgres-operator.crunchydata.com/cluster: ory-db
                  postgres-operator.crunchydata.com/data: postgres
  users:
    - name: pldr-system
      password:
        type: AlphaNumeric
      databases:
        - kratos
        - keto
    - name: readonly
      password:
        type: AlphaNumeric
      databases:
        - kratos
        - keto
  patroni:
    switchover:
      enabled: false
    dynamicConfiguration:
      postgresql:
        parameters:
          # https://pgtune.leopard.in.ua/
          max_connections: 500
          shared_buffers: 1GB
          max_prepared_transactions: 100
          statement_timeout: '60s'
          idle_in_transaction_session_timeout: '60s'
          synchronous_commit: 'remote_write'
          effective_cache_size: 2304MB
          maintenance_work_mem: 192MB
          checkpoint_completion_target: 0.9
          wal_buffers: 16MB
          default_statistics_target: 100
          random_page_cost: 1.1
          effective_io_concurrency: 200
          work_mem: 1966kB
          huge_pages: off
          min_wal_size: 1GB
          max_wal_size: 4GB
        pg_hba: ["host all all all md5"] # Allow non-SSL connections for CLI tools
  monitoring:
    pgmonitor:
      exporter:
        image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres-exporter:latest
  backups:
    pgbackrest:
      configuration:
      - secret:
          name: pgo-s3-credentials
      global:
        repo1-retention-full: "28"
        repo1-path: "/staging/pgbackrest/postgres-operator/ory-db/repo1"
      repos:
        - name: repo1
          s3:
            bucket: "ory-db-backup"
            endpoint: "https://s3url.de"
            region: "eu-central-2"
          schedules:
            full: "0 3 * * *" # Full backup daily at 3:00 AM
      manual:
        repoName: repo1
        options:
         - --type=full

[aeneasr]

I fear that’s way too specific to your environment to fall under common OSS community help, as I can’t spot immediate issues and it requires too much time to reproduce your environment. Sorry! If you can’t figure it out there’s a couple of support options which you can find on our website to get more involved help from us :)

[treylade]

I understand. Thank you for having a look. If I find a solution, I will post an update here.