Vector Sidecar data-dir config

[ShahroZafar]

We are planning to run vector as a sidecar for some critical application whose load is very high as compared to others. We are doing this to avoid CPU split meaning some pods consuming very large cpu as compared to others.
In the sidecar pattern we can have more than 1 pod running on the same node but we also want to limit that sidecar reads only its own logs so we have this in includes
"/var/log/pods/{{ $namespace }}_${VECTOR_SELF_POD_NAME}_*/**/*"

However we want to know how should we configure data-dir. Can we have same data-dir shared between 2 vector pods or we should separate them out.
Also since the include path has pod_name in it, if pod restarts, the path will change. So even if there is 1 pod on 1 node the files which are read will be different. How should we configure data-dir i.e. checkpointing in this case

Configs:

data_dir: /vector-data-dir
expire_metrics_secs: 300
acknowledgements:
  enabled: true
api:
  enabled: true
  address: 0.0.0.0:8686
  playground: false
sources:
  kubernetes_logs:
    type: kubernetes_logs
    glob_minimum_cooldown_ms: 5000
    ingestion_timestamp_field: "ingest_timestamp"
    include_paths_glob_patterns:
    - "/var/log/pods/{{ $namespace }}_${VECTOR_SELF_POD_NAME}_*/**/*"
  internal_metrics:
    type: internal_metrics
transforms:
  dedot_keys:
    type: remap
    inputs:
      - kubernetes_logs
    source: |
      . = map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }
sinks:
  prometheus_exporter:
    type: prometheus_exporter
    flush_period_secs: 60
    inputs:
      - internal_metrics
    address: 0.0.0.0:9090
    buffer:
      type: memory
      when_full: block
      max_events: 500
  kafka:
    type: kafka
    bootstrap_servers: kafka:9092
    topic: vector
    batch:
      timeout_secs: 1
      max_bytes: 1000000
      max_events: 4000
    librdkafka_options:
      client.id: "vector"
      request.required.acks: "1"
    message_timeout_ms: 0
    buffer:
      type: memory
      when_full: block
      max_events: 1000
    inputs:
      - dedot_keys
    encoding:
      codec: json
    compression: zstd

[jszwedko]

Hi @ShahroZafar ,

Each data directory should be separate: Vector assumes it has exclusive access to it.

Also since the include path has pod_name in it, if pod restarts, the path will change. So even if there is 1 pod on 1 node the files which are read will be different. How should we configure data-dir i.e. checkpointing in this case

I'm not sure I understand this part 🤔 You would want to ensure that the Vector process reading file X is using the data directory that has checkpoints for file X.

[ShahroZafar]

Hi @jszwedko ,

What I meant to say is since its a sidecar, vector kubernetes_log property include_paths_glob_patterns is set to "/var/log/pods/{{ $namespace }}_${VECTOR_SELF_POD_NAME}_*/**/*" so that sidecar reads logs of the containers in the same pod. The datadir currently is /vector-data-dir

Now imagine that pod gets re-created on the same node. In this case the include_paths_glob_patterns will change since its referring the pod name but the data-dir would be the same. Do you think we should include ${VECTOR_SELF_POD_NAME} in data-dir as well? If yes, how can we ensure we don't miss logs across restarts? Maybe a prestop hook?

[jszwedko]

Oh I see. Yes, I think including ${VECTOR_SELF_POD_NAME} in the mounted data directory path would be a good way to ensure that the process has access to the correct checkpoints.

[kirmadaa]

Hi @ShahroZafar i have a question "I need to send the logs from permit PDP to vector aggregator using websocket like i want to add a websocket on pdp side which sends the logs to vector ip and add a sorce in vector which basically recieves the log" is it possible if it is how ?