Patch upgradeLog on the upgrade path

[w13915984028]

IMPORTANT: Please do not create a Pull Request without creating an issue first.

Problem:

The upstream logging has some enhancements, the upgradeLog needs to adapt to them.

Solution:

This PR and #7653 enhance the upgradeLog in v1.5.0

Related Issue:
#7652
#7654

Test plan:

Upgrade v1.4.*->v1.5.0, ugpradeLog should work properly, no matter rancher-logging addon is enabled or not

local test log:

run the same test script several times

...
patch clusteroutput hvst-upgrade-b2f59-upgradelog-clusteroutput
clusteroutput.logging.banzaicloud.io/hvst-upgrade-b2f59-upgradelog-clusteroutput patched (no change)
patch clusterflow hvst-upgrade-b2f59-upgradelog-clusterflow
clusterflow.logging.banzaicloud.io/hvst-upgrade-b2f59-upgradelog-clusterflow patched (no change)
patch logging hvst-upgrade-b2f59-upgradelog-infra
logging.logging.banzaicloud.io/hvst-upgrade-b2f59-upgradelog-infra patched (no change)
patch logging hvst-upgrade-b2f59-upgradelog-operator-root
logging.logging.banzaicloud.io/hvst-upgrade-b2f59-upgradelog-operator-root patched


NAME                                          LOGGINGREF                           CONTROLNAMESPACE        WATCHNAMESPACES   PROBLEMS
hvst-upgrade-b2f59-upgradelog-infra           harvester-upgradelog                 harvester-system                          
hvst-upgrade-b2f59-upgradelog-operator-root   harvester-upgradelog-operator-root   cattle-logging-system            

v141->v150 upgrade test, with addon rancher-logging disabled, the upgradeLog works well

[m-ildefons]

Instead of writing a file to disk, use the following to pipe a heredoc directly into kubectl:

cat <<EOF | kubectl [...] -f -
spec:
  loggingRef: "$loggingref"
EOF

This avoids having to clean up with rm later.

[w13915984028]

sometimes we cat the file conent for better debugging, it looks to be a bit lenthy, but does no big harm

[m-ildefons]

Variables should be qouted with ", unless you want to risk expansion and globbing.

I can only recommend to install shellcheck as a linter for shellscripts in your IDE. It will really help your write more robust shellscripts.

[starbops]

The execution logic LGTM. But I'm a bit confused about the field loggingRef. What does it reference to? Does the value have to be some real objects, or is it just used as a key to show the uniqueness of the Logging CR? It is even weird that the field also exists in the Logging CRD. What does it mean for a Logging CR to reference to another Logging CR? Could you kindly elaborate on that? Also, I found that the field is optional on their documentation.
https://kube-logging.dev/docs/configuration/crds/v1beta1/logging_types/#loggingspec-loggingref

Thank you.

[w13915984028]

@starbops

The general design is:

(1). In a system, user may deploy several groups of fluentbit & fluentd, the fluentbit needs a method to look it's paired fluentd; it also applies to clusterflow/flow and clusteroutput/output, even in a single namespace, multi groups of fluentbit & fluentd are still allowed.

(2). The logging object is for logging-operator, each logging will rollout a pair of fluentbit & fluentd

NAME                                  LOGGINGREF                     CONTROLNAMESPACE        WATCHNAMESPACES   PROBLEMS
hvst-upgrade-d6kpr-upgradelog-infra                                  harvester-system                          
rancher-logging-kube-audit            harvester-kube-audit-log-ref   cattle-logging-system                     
rancher-logging-root                                                 cattle-logging-system        

(3). The loggingRef is required by 1, it has stronger effect than using label/annotation, hence it also propagates to (2), this is an unique string to identify the pair in one cluster.

technically, in the upgraded rancher-logging version, when a logging is deployed, it will generate related secret for bluentbit

$ kubectl get secret -n cattle-logging-system
NAME                                                             TYPE                 DATA   AGE
...
rancher-logging-kube-audit-fluentbit                             Opaque               1      22m


$kubectl get secret -n cattle-logging-system rancher-logging-kube-audit-fluentbit -oyaml
apiVersion: v1
data:
  fluent-bit.conf: CltTRVJWSUNFXQogI...ZQo=

The fluent-bit.conf is base64 encoded, the decoded value is like:

[SERVICE]
    Flush        1
    Grace        5
    Daemon       Off
    Log_Level    info
    Parsers_File /fluent-bit/etc/parsers.conf
    Coro_Stack_Size    24576
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020
    storage.path  /buffers
[INPUT]
    Name         tail
    DB  /tail-db/tail-containers-state.db
    DB.locking  true
    Mem_Buf_Limit  5MB
    Parser  json
    Path  /kube-audit-logs/audit.log
    Refresh_Interval  5
    Skip_Long_Lines  On
    Tag  kube-audit
[OUTPUT]
    Name          forward
    Match         *
    Host          rancher-logging-kube-audit-fluentd.cattle-logging-system.svc.cluster.local.
    Port          24240
    Retry_Limit  False

In the [OUTPUT] Host field, you will find the trick, without a proper loggingRef, it just points to rancher-logging-root. And the original expected fluentd won't get any log.

When testing harvester/addons#47, I first noticed that the audit log does not work, and finnally found that the above is Host rancher-logging-root.cattle-logging-system.svc.cluster.local.

This rule applies to upgradeLog as well.