Configure Tempo with reserved IP scheme

[ahsan-n]

Hi so we unknowingly configured our AWS-VPC with reservered subnet scheme: 183.x.x.x.
By default this is not allowed. To overcome this we are trying to utilize bind-addr with following configurations but getting the error:

Ingester logs

ts=2022-07-14T18:53:25.49344234Z caller=memberlist_logger.go:74 level=warn msg="Got ping for unexpected node 'tempo-tempo-distributed-distributor-ccdd74c9-gjpxs-2cbfb8a9' from=[::]:7946"
4
ts=2022-07-14T18:53:27.494257534Z caller=memberlist_logger.go:74 level=warn msg="Got ping for unexpected node tempo-tempo-distributed-distributor-ccdd74c9-gjpxs-2cbfb8a9 from=127.0.0.1:42338"
3
ts=2022-07-14T18:53:27.49431992Z caller=memberlist_logger.go:74 level=error msg="Failed fallback ping: EOF"
2
ts=2022-07-14T18:53:35.49318193Z caller=memberlist_logger.go:74 level=info msg="Suspect tempo-tempo-distributed-distributor-ccdd74c9-gjpxs-2cbfb8a9 has failed, no acks received"
1
ts=2022-07-14T18:53:35.493612266Z caller=memberlist_logger.go:74 level=warn msg="Got ping for unexpected node 'tempo-tempo-distributed-querier-648749c87-876sr-1e7a6ade' from=[::]:7946"

With these logs our assumption is distributor and querier not getting configured with that reservered CIDR. Not sure what we are missing here:

global:
  extraEnv:
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
tempo:
  structuredConfig:
    memberlist:
      bind_addr:
        - ${MY_POD_IP}

complete config

global:
  extraEnv:
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP

querier:
  extraEnv:
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP

gateway:
  enabled: true
  ingress:
    enabled: true
    className: ""
    annotations:
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/load-balancer-name: observability-tempo-alb
      alb.ingress.kubernetes.io/group.name: observability-tempo-alb
      alb.ingress.kubernetes.io/target-type: ip
    hosts:
      - host: "*.tracing.domain-engineering.com"
        paths:
          - path: /
            pathType: Prefix

compactor:
  enabled: true
  working_directory: /data/retention
  shared_store: s3
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  nodeSelector:
    domain: observability
serviceAccount:
  create: false
  name: tempo
search:
  enabled: false
traces:
  jaeger:
    # -- Enable Tempo to ingest Jaeger GRPC traces
    grpc:
      enabled: true
    # -- Enable Tempo to ingest Jaeger Thrift Binary traces
    thriftBinary:
      enabled: true
    # -- Enable Tempo to ingest Jaeger Thrift Compact traces
    thriftCompact:
      enabled: true
    # -- Enable Tempo to ingest Jaeger Thrift HTTP traces
    thriftHttp:
      enabled: true
  # -- Enable Tempo to ingest Zipkin traces
  zipkin:
    enabled: true
  otlp:
    # -- Enable Tempo to ingest Open Telemetry HTTP traces
    http:
      enabled: true
    # -- Enable Tempo to ingest Open Telemetry GRPC traces
    grpc:
      enabled: true
  # -- Enable Tempo to ingest Open Census traces
  opencensus:
    enabled: true
ingester:
  replicas: 1
tempo:
  structuredConfig:
    memberlist:
      bind_addr:
        - ${MY_POD_IP}
    server:
      http_listen_port: 3100
    storage:
      trace:
        backend: s3
        s3:
          bucket: OUR-BUCKET
          endpoint: s3.ap-southeast-1.amazonaws.com
          insecure: true

[mdisibio]

Hi, can you provide more information about what isn't working? The error messages Got ping for unexpected node... are not necessarily a problem, we get them too.

tag: tempo1.3.2 Is it possible to try running the latest Tempo 1.4.1? There were some memberlist fixes included which may help.

[ahsan-n]

Hi I have updated the config can you re-look, that old one was added by mistake. About whats not working ingester not appearing as healthy
I assume the problem is 127.0.0.1:42338 isnt it suppose to assign cluster IP if I configured this:

global:
  extraEnv:
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
tempo:
  structuredConfig:
    memberlist:
      bind_addr:
        - ${MY_POD_IP}```

![image](https://user-images.githubusercontent.com/74932557/179194325-a203223f-f6eb-4e5f-b3a6-9d7a025e1c59.png)