Skip to content

worker-0 is getting SIGKILLed instead of nicely reloading upon issuing gracefulReload #3341

@alex-vmw

Description

@alex-vmw

Describe the bug
We were performing tests that were issuing multiple /api/config.gracefulReload calls to fluentd. Sometimes after issuing a /api/config.gracefulReload we see that worker-0 is getting SIGKILLed instead of nicely reloading.

When the issue happens we see the below message in the logs:

2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: Worker 0 finished unexpectedly with signal SIGKILL

When the issue does NOT happen we see the below message in the logs instead:

2021-04-21 02:29:03 +0000 [info]: #0 fluent/log.rb:329:info: restart fluentd worker worker=0

To Reproduce
Reload fluentd config multiple times via /api/config.gracefulReload call and look for below message in the log:

2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: Worker 0 finished unexpectedly with signal SIGKILL

Expected behavior
Fluentd worker-0 should always nicely reload when /api/config.gracefulReload is issued.

Your Environment
Fluentd or td-agent version: fluentd 1.12.2 we also tested with 1.9.1 and 1.11.2 and saw same issue in those as well.
Operating system: Ubuntu 20.04.1 LTS
Kernel version: 5.4.0-42-generic

Your Configuration

<ROOT>
  <system>
    log_level info
    rpc_endpoint "127.0.0.1:24444"
  </system>
  <source>
    @type systemd
    @id in_systemd_docker
    path "/var/log/journal"
    tag "systemd.unit"
    read_from_head false
    <storage>
      @type "local"
      persistent true
      path "/var/log/-kube-fluentd-operator--fluentd-journald-cursor.json"
    </storage>
    <entry>
      field_map {"_SYSTEMD_UNIT":"unit","MESSAGE":"log","_PID":["pid"],"_PRIORITY":"priority","_COMM":"cmd","_HOSTNAME":"hostname"}
      field_map_strict true
      fields_lowercase true
    </entry>
  </source>
  <match systemd.unit>
    @type rewrite_tag_filter
    <rule>
      key "unit"
      pattern ^(.+)$
      tag "systemd.$1"
    </rule>
  </match>
  <filter systemd.kubelet.service>
    @type parser
    reserve_data true
    key_name "log"
    emit_invalid_record_to_error false
    <parse>
      @type "kubernetes"
      time_format "%m%d %H:%M:%S.%N"
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
    </parse>
  </filter>
  <source>
    @type tail
    @id in_tail_container_logs
    path "/var/log/containers/*.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-containers.log.pos"
    tag "kubernetes.*"
    read_from_head true
    <parse>
      @type "multiline"
      format1 /^(?<partials>([^\n]+ (stdout|stderr) P [^\n]+\n)*)/
      format2 /(?<time>[^\n]+) (?<stream>stdout|stderr) F (?<log>[^\n]*)/
      format3 /|(?<json>{.*})/
      time_format "%Y-%m-%dT%H:%M:%S.%N%:z"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type record_transformer
    @id filter_crio_container_logs
    enable_ruby true
    remove_keys partials
    <record>
      log ${record["partials"]&.gsub(/.+ (stdout|stderr) P (.+)\n/, '\\2')}${record["log"]}
    </record>
  </filter>
  <filter kubernetes.**>
    @type parser
    @id filter_docker_container_logs
    key_name "json"
    remove_key_name_field true
    reserve_data true
    emit_invalid_record_to_error false
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      time_type string
    </parse>
  </filter>
  <source>
    @type tail
    @id in_tail_startupscript
    path "/var/log/startupscript.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-startupscript.log.pos"
    tag "startupscript"
    <parse>
      @type "syslog"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_docker
    path "/var/log/docker.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-docker.log.pos"
    tag "docker"
    <parse>
      @type "regexp"
      expression /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_etcd
    path "/var/log/etcd.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-etcd.log.pos"
    tag "k8s.etcd"
    <parse>
      @type "none"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kubelet
    multiline_flush_interval 5s
    path "/var/log/kubelet.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kubelet.log.pos"
    tag "k8s.kubelet"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_proxy
    multiline_flush_interval 5s
    path "/var/log/kube-proxy.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-proxy.log.pos"
    tag "k8s.kube-proxy"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver
    multiline_flush_interval 5s
    path "/var/log/kube-apiserver.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-apiserver.log.pos"
    tag "k8s.kube-apiserver"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_controller_manager
    multiline_flush_interval 5s
    path "/var/log/kube-controller-manager.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-controller-manager.log.pos"
    tag "k8s.kube-controller-manager"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_scheduler
    multiline_flush_interval 5s
    path "/var/log/kube-scheduler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-scheduler.log.pos"
    tag "k8s.kube-scheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_rescheduler
    multiline_flush_interval 5s
    path "/var/log/rescheduler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-rescheduler.log.pos"
    tag "k8s.rescheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_glbc
    multiline_flush_interval 5s
    path "/var/log/glbc.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-glbc.log.pos"
    tag "k8s.glbc"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_cluster_autoscaler
    multiline_flush_interval 5s
    path "/var/log/cluster-autoscaler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-cluster-autoscaler.log.pos"
    tag "k8s.cluster-autoscaler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver_audit
    multiline_flush_interval 5s
    path "/var/log/kubernetes/kube-apiserver-audit.log"
    pos_file "/var/log/-kube-fluentd-operator--kube-apiserver-audit.log.pos"
    tag "k8s.kube-apiserver-audit"
    <parse>
      @type "multiline"
      format_firstline "/^\\S+\\s+AUDIT:/"
      format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
      time_format "%Y-%m-%dT%T.%L%Z"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
    @id filter_kube_metadata
  </filter>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <filter kubernetes.**>
    @type record_transformer
    enable_ruby true
    <record>
      kubernetes_namespace_container_name ${record["kubernetes"]["namespace_name"]}.${record["kubernetes"]["pod_name"]}.${record["kubernetes"]["container_name"]}
      container_info ${record["docker"]["container_id"]}-${record["stream"]}
    </record>
  </filter>
  <match kubernetes.**>
    @type rewrite_tag_filter
    <rule>
      key "kubernetes_namespace_container_name"
      pattern ^(.+)$
      tag "kube.$1"
    </rule>
  </match>
  <filter kube.*.*.*>
    @type record_modifier
    remove_keys "dummy_"
    <record>
      dummy_ ${record["kubernetes"]&.delete("master_url"); record["kubernetes"]&.delete("namespace_id"); if record["kubernetes"]&.has_key?("labels"); record["kubernetes"]["labels"].delete("pod-template-generation"); record["kubernetes"]["labels"].delete("controller-revision-hash");  record["kubernetes"]["labels"].delete("pod-template-hash"); end; nil}
    </record>
  </filter>
  <filter kube.*.*.*>
    @type record_transformer
    remove_keys kubernetes_namespace_container_name
  </filter>
  <filter kube.kube-system.**>
    @type parser
    reserve_data true
    key_name "log"
    emit_invalid_record_to_error false
    <parse>
      @type "kubernetes"
      time_format "%m%d %H:%M:%S.%N"
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
    </parse>
  </filter>
  <source>
    @type prometheus
  </source>
  <source>
    @type prometheus_monitor
  </source>
  <source>
    @type prometheus_output_monitor
  </source>
  <match systemd.** kube.kube-system.**>
    @type copy
    <store>
      @id X
      @type "vmware_loginsight"
      host "X.X.X.X"
      include_tag_key true
      log_text_keys ["log","msg","message"]
      port 9000
      scheme "http"
      serializer "json"
      ssl_verify false
      tag_key "tag"
    </store>
  </match>
  <match **>
    @type null
  </match>
</ROOT>

Your Error Log

2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: fluentd RPC got /api/config.gracefulReload request
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: fluentd supervisor process got SIGUSR2
2021-04-21 02:09:57 +0000 [info]: fluent/log.rb:329:info: Reloading new config
2021-04-21 02:09:57 +0000 [info]: fluent/log.rb:329:info: adding rewrite_tag_filter rule: unit [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x0000000004474560 @keys="unit">, /^(.+)$/, "", "systemd.$1"]
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: Kubernetes URL is not set - inspecting environ
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: Kubernetes URL is now 'https://172.21.0.1:443/api'
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: Found directory with secrets: /var/run/secrets/kubernetes.io/serviceaccount
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: Found CA certificate: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: Found pod token: /var/run/secrets/kubernetes.io/serviceaccount/token
2021-04-21 02:09:57 +0000 [debug]: fluent/log.rb:308:debug: Creating K8S client
2021-04-21 02:09:57 +0000 [info]: fluent/log.rb:329:info: adding rewrite_tag_filter rule: kubernetes_namespace_container_name [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00007f4ef49266a8 @keys="kubernetes_namespace_container_name">, /^(.+)$/, "", "kube.$1"]
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: adding store type="vmware_loginsight"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: using configuration file: <ROOT>
  <system>
    log_level info
    rpc_endpoint 127.0.0.1:24444
  </system>
  <source>
    @type systemd
    @id in_systemd_docker
    path "/var/log/journal"
    tag "systemd.unit"
    read_from_head false
    <storage>
      @type "local"
      persistent true
      path "/var/log/-kube-fluentd-operator--fluentd-journald-cursor.json"
    </storage>
    <entry>
      field_map {"_SYSTEMD_UNIT":"unit","MESSAGE":"log","_PID":["pid"],"_PRIORITY":"priority","_COMM":"cmd","_HOSTNAME":"hostname"}
      field_map_strict true
      fields_lowercase true
    </entry>
  </source>
  <match systemd.unit>
    @type rewrite_tag_filter
    <rule>
      key "unit"
      pattern ^(.+)$
      tag "systemd.$1"
    </rule>
  </match>
  <filter systemd.kubelet.service>
    @type parser
    reserve_data true
    key_name "log"
    emit_invalid_record_to_error false
    <parse>
      @type "kubernetes"
      time_format "%m%d %H:%M:%S.%N"
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
    </parse>
  </filter>
  <source>
    @type tail
    @id in_tail_container_logs
    path "/var/log/containers/*.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-containers.log.pos"
    tag "kubernetes.*"
    read_from_head true
    <parse>
      @type "multiline"
      format1 /^(?<partials>([^\n]+ (stdout|stderr) P [^\n]+\n)*)/
      format2 /(?<time>[^\n]+) (?<stream>stdout|stderr) F (?<log>[^\n]*)/
      format3 /|(?<json>{.*})/
      time_format "%Y-%m-%dT%H:%M:%S.%N%:z"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type record_transformer
    @id filter_crio_container_logs
    enable_ruby true
    remove_keys partials
    <record>
      log ${record["partials"]&.gsub(/.+ (stdout|stderr) P (.+)\n/, '\\2')}${record["log"]}
    </record>
  </filter>
  <filter kubernetes.**>
    @type parser
    @id filter_docker_container_logs
    key_name "json"
    remove_key_name_field true
    reserve_data true
    emit_invalid_record_to_error false
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      time_type string
    </parse>
  </filter>
  <source>
    @type tail
    @id in_tail_startupscript
    path "/var/log/startupscript.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-startupscript.log.pos"
    tag "startupscript"
    <parse>
      @type "syslog"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_docker
    path "/var/log/docker.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-docker.log.pos"
    tag "docker"
    <parse>
      @type "regexp"
      expression /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_etcd
    path "/var/log/etcd.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-etcd.log.pos"
    tag "k8s.etcd"
    <parse>
      @type "none"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kubelet
    multiline_flush_interval 5s
    path "/var/log/kubelet.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kubelet.log.pos"
    tag "k8s.kubelet"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_proxy
    multiline_flush_interval 5s
    path "/var/log/kube-proxy.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-proxy.log.pos"
    tag "k8s.kube-proxy"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver
    multiline_flush_interval 5s
    path "/var/log/kube-apiserver.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-apiserver.log.pos"
    tag "k8s.kube-apiserver"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_controller_manager
    multiline_flush_interval 5s
    path "/var/log/kube-controller-manager.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-controller-manager.log.pos"
    tag "k8s.kube-controller-manager"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_scheduler
    multiline_flush_interval 5s
    path "/var/log/kube-scheduler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-scheduler.log.pos"
    tag "k8s.kube-scheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_rescheduler
    multiline_flush_interval 5s
    path "/var/log/rescheduler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-rescheduler.log.pos"
    tag "k8s.rescheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_glbc
    multiline_flush_interval 5s
    path "/var/log/glbc.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-glbc.log.pos"
    tag "k8s.glbc"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_cluster_autoscaler
    multiline_flush_interval 5s
    path "/var/log/cluster-autoscaler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-cluster-autoscaler.log.pos"
    tag "k8s.cluster-autoscaler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver_audit
    multiline_flush_interval 5s
    path "/var/log/kubernetes/kube-apiserver-audit.log"
    pos_file "/var/log/-kube-fluentd-operator--kube-apiserver-audit.log.pos"
    tag "k8s.kube-apiserver-audit"
    <parse>
      @type "multiline"
      format_firstline "/^\\S+\\s+AUDIT:/"
      format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
      time_format "%Y-%m-%dT%T.%L%Z"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <filter kubernetes.**>
    @type record_transformer
    enable_ruby true
    <record>
      kubernetes_namespace_container_name ${record["kubernetes"]["namespace_name"]}.${record["kubernetes"]["pod_name"]}.${record["kubernetes"]["container_name"]}
      container_info ${record["docker"]["container_id"]}-${record["stream"]}
    </record>
  </filter>
  <match kubernetes.**>
    @type rewrite_tag_filter
    <rule>
      key "kubernetes_namespace_container_name"
      pattern ^(.+)$
      tag "kube.$1"
    </rule>
  </match>
  <filter kube.*.*.*>
    @type record_modifier
    remove_keys "dummy_"
    <record>
      dummy_ ${record["kubernetes"]&.delete("master_url"); record["kubernetes"]&.delete("namespace_id"); if record["kubernetes"]&.has_key?("labels"); record["kubernetes"]["labels"].delete("pod-template-generation"); record["kubernetes"]["labels"].delete("controller-revision-hash");  record["kubernetes"]["labels"].delete("pod-template-hash"); end; nil}
    </record>
  </filter>
  <filter kube.*.*.*>
    @type record_transformer
    remove_keys kubernetes_namespace_container_name
  </filter>
  <filter kube.kube-system.**>
    @type parser
    reserve_data true
    key_name "log"
    emit_invalid_record_to_error false
    <parse>
      @type "kubernetes"
      time_format "%m%d %H:%M:%S.%N"
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
    </parse>
  </filter>
  <source>
    @type prometheus
  </source>
  <source>
    @type prometheus_monitor
  </source>
  <source>
    @type prometheus_output_monitor
  </source>
  <match systemd.** kube.kube-system.** kube.avi-system.** kube.cluster-services.** kube.decc-tests.** kube.default.** kube.gulel-core.** kube.ingress.**>
    @type copy
    <store>
      @id out_vmware_loginsight_kubecluster_logs
      @type "vmware_loginsight"
      host "10.182.107.52"
      include_tag_key true
      log_text_keys ["log","msg","message"]
      port 9000
      scheme "http"
      serializer "json"
      ssl_verify false
      tag_key "tag"
    </store>
  </match>
  <match **>
    @type null
  </match>
</ROOT>
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:systemd plugin_id="in_systemd_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_startupscript"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_etcd"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_kubelet"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_kube_proxy"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_kube_apiserver"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_kube_controller_manager"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_kube_scheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_rescheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_glbc"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_cluster_autoscaler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:tail plugin_id="in_tail_kube_apiserver_audit"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:prometheus plugin_id="object:e704"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:prometheus_monitor plugin_id="object:e718"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on input plugin type=:prometheus_output_monitor plugin_id="object:e72c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on output plugin type=:rewrite_tag_filter plugin_id="object:e740"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on output plugin type=:rewrite_tag_filter plugin_id="object:e754"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on output plugin type=:copy plugin_id="object:e768"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:parser plugin_id="object:cbd4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:record_transformer plugin_id="filter_crio_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:parser plugin_id="filter_docker_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:kubernetes_metadata plugin_id="object:e77c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:record_transformer plugin_id="object:e790"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:record_modifier plugin_id="object:e7a4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:record_transformer plugin_id="object:e7b8"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on filter plugin type=:parser plugin_id="object:cd28"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on output plugin type=:vmware_loginsight plugin_id="out_vmware_loginsight_kubecluster_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling stop on output plugin type=:null plugin_id="object:cd50"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:prometheus_output_monitor plugin_id="object:e72c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:systemd plugin_id="in_systemd_docker"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:systemd plugin_id="in_systemd_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_container_logs"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_startupscript"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_docker"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_etcd"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_kubelet"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_kube_proxy"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_kube_proxy"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_kube_apiserver"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_kube_apiserver"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_kube_controller_manager"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_kube_controller_manager"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_kube_scheduler"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_kube_scheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_rescheduler"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_rescheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_glbc"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_glbc"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_cluster_autoscaler"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_cluster_autoscaler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:tail plugin_id="in_tail_kube_apiserver_audit"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_kube_apiserver_audit"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:prometheus plugin_id="object:e704"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:prometheus plugin_id="object:e704"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown input plugin type=:prometheus_monitor plugin_id="object:e718"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:prometheus_monitor plugin_id="object:e718"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:prometheus_output_monitor plugin_id="object:e72c"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_startupscript"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_etcd"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down input plugin type=:tail plugin_id="in_tail_kubelet"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown output plugin type=:rewrite_tag_filter plugin_id="object:e754"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown output plugin type=:rewrite_tag_filter plugin_id="object:e740"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown output plugin type=:copy plugin_id="object:e768"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down output plugin type=:copy plugin_id="object:e768"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down output plugin type=:rewrite_tag_filter plugin_id="object:e754"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down output plugin type=:rewrite_tag_filter plugin_id="object:e740"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:parser plugin_id="object:cd28"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:parser plugin_id="object:cbd4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:parser plugin_id="filter_docker_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:record_transformer plugin_id="filter_crio_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:record_transformer plugin_id="object:e790"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:record_transformer plugin_id="object:e790"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:kubernetes_metadata plugin_id="object:e77c"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:kubernetes_metadata plugin_id="object:e77c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:record_transformer plugin_id="object:e7b8"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:parser plugin_id="object:cd28"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown filter plugin type=:record_modifier plugin_id="object:e7a4"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:record_modifier plugin_id="object:e7a4"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:parser plugin_id="object:cbd4"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:parser plugin_id="filter_docker_container_logs"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:record_transformer plugin_id="filter_crio_container_logs"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down filter plugin type=:record_transformer plugin_id="object:e7b8"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown output plugin type=:vmware_loginsight plugin_id="out_vmware_loginsight_kubecluster_logs"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down output plugin type=:vmware_loginsight plugin_id="out_vmware_loginsight_kubecluster_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: preparing shutdown output plugin type=:null plugin_id="object:cd50"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: shutting down output plugin type=:null plugin_id="object:cd50"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:systemd plugin_id="in_systemd_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_startupscript"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_etcd"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_kubelet"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_kube_proxy"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_kube_apiserver"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_kube_controller_manager"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_kube_scheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_rescheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_glbc"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_cluster_autoscaler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:tail plugin_id="in_tail_kube_apiserver_audit"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:prometheus plugin_id="object:e704"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:prometheus_monitor plugin_id="object:e718"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on input plugin type=:prometheus_output_monitor plugin_id="object:e72c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on output plugin type=:rewrite_tag_filter plugin_id="object:e740"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on output plugin type=:rewrite_tag_filter plugin_id="object:e754"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on output plugin type=:copy plugin_id="object:e768"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:parser plugin_id="object:cbd4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:record_transformer plugin_id="filter_crio_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:parser plugin_id="filter_docker_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:kubernetes_metadata plugin_id="object:e77c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:record_transformer plugin_id="object:e790"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:record_modifier plugin_id="object:e7a4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:record_transformer plugin_id="object:e7b8"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on filter plugin type=:parser plugin_id="object:cd28"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on output plugin type=:vmware_loginsight plugin_id="out_vmware_loginsight_kubecluster_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling after_shutdown on output plugin type=:null plugin_id="object:cd50"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:prometheus_output_monitor plugin_id="object:e72c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:prometheus_monitor plugin_id="object:e718"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:systemd plugin_id="in_systemd_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_startupscript"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_kube_proxy"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_kubelet"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_etcd"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_kube_controller_manager"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_kube_apiserver"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_glbc"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_rescheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_cluster_autoscaler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_kube_apiserver_audit"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:prometheus plugin_id="object:e704"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing input plugin type=:tail plugin_id="in_tail_kube_scheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing output plugin type=:rewrite_tag_filter plugin_id="object:e740"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing output plugin type=:rewrite_tag_filter plugin_id="object:e754"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing output plugin type=:copy plugin_id="object:e768"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:record_modifier plugin_id="object:e7a4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:parser plugin_id="filter_docker_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:record_transformer plugin_id="filter_crio_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:kubernetes_metadata plugin_id="object:e77c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:parser plugin_id="object:cbd4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:record_transformer plugin_id="object:e7b8"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:record_transformer plugin_id="object:e790"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing filter plugin type=:parser plugin_id="object:cd28"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing output plugin type=:vmware_loginsight plugin_id="out_vmware_loginsight_kubecluster_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: closing output plugin type=:null plugin_id="object:cd50"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:systemd plugin_id="in_systemd_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_startupscript"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_docker"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_etcd"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_kubelet"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_kube_proxy"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_kube_apiserver"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_kube_controller_manager"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_kube_scheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_rescheduler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_glbc"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_cluster_autoscaler"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:tail plugin_id="in_tail_kube_apiserver_audit"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:prometheus plugin_id="object:e704"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:prometheus_monitor plugin_id="object:e718"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on input plugin type=:prometheus_output_monitor plugin_id="object:e72c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on output plugin type=:rewrite_tag_filter plugin_id="object:e740"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on output plugin type=:rewrite_tag_filter plugin_id="object:e754"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on output plugin type=:copy plugin_id="object:e768"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:parser plugin_id="object:cbd4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:record_transformer plugin_id="filter_crio_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:parser plugin_id="filter_docker_container_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:kubernetes_metadata plugin_id="object:e77c"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:record_transformer plugin_id="object:e790"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:record_modifier plugin_id="object:e7a4"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:record_transformer plugin_id="object:e7b8"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on filter plugin type=:parser plugin_id="object:cd28"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on output plugin type=:vmware_loginsight plugin_id="out_vmware_loginsight_kubecluster_logs"
2021-04-21 02:09:58 +0000 [debug]: fluent/log.rb:308:debug: calling terminate on output plugin type=:null plugin_id="object:cd50"
2021-04-21 02:09:58 +0000 [info]: fluent/log.rb:329:info: Worker 0 finished unexpectedly with signal SIGKILL
2021-04-21 02:09:59 +0000 [info]: fluent/log.rb:329:info: adding match pattern="systemd.unit" type="rewrite_tag_filter"
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered output plugin 'rewrite_tag_filter'
2021-04-21 02:09:59 +0000 [info]: #0 fluent/log.rb:329:info: adding rewrite_tag_filter rule: unit [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x0000000002915468 @keys="unit">, /^(.+)$/, "", "systemd.$1"]
2021-04-21 02:09:59 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="systemd.kubelet.service" type="parser"
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered filter plugin 'parser'
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'regexp'
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'kubernetes'
2021-04-21 02:09:59 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kubernetes.**" type="record_transformer"
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered filter plugin 'record_transformer'
2021-04-21 02:09:59 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kubernetes.**" type="parser"
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'json'
2021-04-21 02:09:59 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2021-04-21 02:09:59 +0000 [trace]: #0 fluent/log.rb:286:trace: registered filter plugin 'kubernetes_metadata'
2021-04-21 02:09:59 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2021-04-21 02:09:59 +0000 [debug]: #0 fluent/log.rb:308:debug: Kubernetes URL is not set - inspecting environ
2021-04-21 02:09:59 +0000 [debug]: #0 fluent/log.rb:308:debug: Kubernetes URL is now 'https://172.21.0.1:443/api'
2021-04-21 02:09:59 +0000 [debug]: #0 fluent/log.rb:308:debug: Found directory with secrets: /var/run/secrets/kubernetes.io/serviceaccount
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: Found CA certificate: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: Found pod token: /var/run/secrets/kubernetes.io/serviceaccount/token
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: Creating K8S client
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kubernetes.**" type="record_transformer"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding match pattern="kubernetes.**" type="rewrite_tag_filter"
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: adding rewrite_tag_filter rule: kubernetes_namespace_container_name [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00000000032771a0 @keys="kubernetes_namespace_container_name">, /^(.+)$/, "", "kube.$1"]
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kube.*.*.*" type="record_modifier"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered filter plugin 'record_modifier'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kube.*.*.*" type="record_transformer"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding filter pattern="kube.kube-system.**" type="parser"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding match pattern="systemd.** kube.kube-system.** kube.avi-system.** kube.cluster-services.** kube.decc-tests.** kube.default.** kube.gulel-core.** kube.ingress.**" type="copy"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered output plugin 'copy'
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: adding store type="vmware_loginsight"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered output plugin 'vmware_loginsight'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding match pattern="**" type="null"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered output plugin 'null'
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered buffer plugin 'memory'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="systemd"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'systemd'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'multiline'
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'tail'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'syslog'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'none'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="tail"
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="prometheus"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'prometheus'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="prometheus_monitor"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'monitor_agent'
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'prometheus_monitor'
2021-04-21 02:10:00 +0000 [info]: fluent/log.rb:329:info: adding source type="prometheus_output_monitor"
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'prometheus_output_monitor'
2021-04-21 02:10:00 +0000 [warn]: #0 fluent/log.rb:350:warn: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: starting fluentd worker pid=21827 ppid=1 worker=0
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: listening prometheus http server on http:://0.0.0.0:24231//metrics for worker0
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: Start async HTTP server listening http://0.0.0.0:24231
2021-04-21 02:10:00 +0000 [debug]: #0 fluent/log.rb:308:debug: Async::IO::SocketBinding to #<Addrinfo: 0.0.0.0:24231 TCP>
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-x4n6n_gulel-nimbus_gulel-container-716c1123-c4d94247f088e7c728dde61b09fd97b2614529931d81441fb1c7122150539c15.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-3bbf17ba-p58cb_gulel-worker_gulel-container-3bbf17ba-ea2da9f9fccb45a90613321fd2b9b827acec313f07437e7e25e043e5d8a4f2e9.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-cb51080e-dldz8_gulel-worker_gulel-container-cb51080e-1822bea7688a3b0867749954b5aa678c336c814ba2f0c6ece24bad5023b55771.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-bxbsd_gulel-nimbus_gulel-container-aa5df3e5-fc443f8ae7db9a102e77ed305d42e6e6e415b6f83ab4bfd479c6714ec48b8643.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/node-exporter-fjzmp_cluster-services_node-exporter-2ba2ebd7e22ec4b9bb65ec97fe3f9428367cb7ea9e6d84cb073fe56e74b7aeea.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/node-exporter-fjzmp_cluster-services_kube-rbac-proxy-54672a541018e7b8ac5cfc48ce0f5341f1ed2d04fd358c31718498d5cd1cc76c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-9c8cb82a-9p5b6_gulel-worker_gulel-container-9c8cb82a-1bb2d9f7bea86b48bdcd1f48a010f81618e7790429d575381ffba75aed35fd31.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-2nd4v_gulel-nimbus_gulel-container-da3d64e9-819568e95c2ad0bece1fc089b2a01f96a31c8c9f124962cd78457ddf9fdb86a8.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/redis-master-0_gulel-core_redis-d3ab86f045d201a8d1f403151109d78760800069dd32ee44bb8e050b8fbdb579.log
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: gulel-core/redis-master-0
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: gulel-core
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-df090692-x6nxp_gulel-worker_gulel-container-df090692-1371a6b4317343c90c6f7f65c3f3f882f46c8c517e7d7090e9e8a21f3905a530.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/vsphere-csi-node-7xc6j_kube-system_node-driver-registrar-c934e748c655e1edbafaf56ca502a9b431aa85132602bd50526bb98689745354.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-6c10db90-hwgmm_gulel-worker_gulel-container-6c10db90-35dc4109de3c71f2865f1aafb602211e8d681fbea2674681e2ac512355d06ccd.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-0aa5d677-2lnz7_gulel-worker_gulel-container-0aa5d677-c0e73223a6471e29c56390017ee21b208a326e8b8887d85b95e7ed102716a615.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-080de4f4-tqcd2_gulel-worker_gulel-container-080de4f4-10fd99598c9e9a924459086e8b1b98cc1ff1b3886f5131e811673e9acec611bd.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-v6x2f_gulel-nimbus_gulel-container-3dc10de7-8f794664e6665e59169a854d5fdcfb062fba122c2fac907fc92c8a8afde7c41d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-8cjfc_gulel-nimbus_gulel-container-0f97a1bf-bfa0abdddba2086312f12f15cb97f67a88fdf0854b22d40b54fa4ec2dc4c0c92.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-fdhbt_gulel-nimbus_gulel-container-ec76f9d7-7cacdd4b2f764e6c38238af51b68d9e7b39ab2e01a4efb87480ca3c7d5eb32cc.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-lpmr5_gulel-nimbus_gulel-container-8a91aaeb-6bdcacbaf527308bfe4dea87a51530ea2ecde1c6a0375f9c8a7ca8eb858d547c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/dns-analyser-226c2_decc-tests_dns-analyser-68a16c77ccc01e1d0b5a47399150ff986db84b946f68b485d97eec6ea182eac3.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-launcher-64b8bf6c87-b2qhr_gulel-core_launcher-382073d7c99978eae2dd708d8b91cbbbc6c4f3b5e22c8f5cc43e6b66809fce94.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-n8g4n_gulel-nimbus_gulel-container-7090d280-97465abf4251ba9386a62c0f24efdd5094bb6c800642e51615f31412c9fa05cc.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-cd72696c-gn44q_gulel-worker_gulel-container-cd72696c-bbcee55395c3d1095092580eefa11d787aa74773802cacaf0407bbfe4eda8a5c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-a1e6b5e2-tpbjm_gulel-worker_gulel-container-a1e6b5e2-8425322a761b74bcb27ecebee962513f86d00646374aefe4543e862ec3743029.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/amko-0_avi-system_amko-5136ddda74ea4fa6828578552820803eca47346c495f73b254ffc146d16ffbb5.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-tdpg9_gulel-nimbus_gulel-container-d4b5a145-3245c17dbf33b0e30f2365515a3071ef1bb31e1e95fff08c6a07824bb135c8e0.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-b8be93c2-mxx57_gulel-worker_gulel-container-b8be93c2-acd729f3c849474875e90253cf8b2da7112ed88f22e71ef9b4887c793a36b11d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-frblc_gulel-nimbus_gulel-container-efdbd678-372d99f5759d04d793085be83b08364f6ccf8bc1c963bb136c0b194e003f6629.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-dffe3e2e-kklhq_gulel-worker_gulel-container-dffe3e2e-bf5c05c3ce1472d91c41fdda5b5180937fa37e01d4b847abb0df20a2d11e47fe.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-797f2472-mjlnq_gulel-worker_gulel-container-797f2472-bd07de4657d88b81e634a8d62c5c488909ad49431cc0226436c7a82de8503ed8.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-rbr49_gulel-nimbus_gulel-container-b941b2db-dbb6dc75696ff7ca878bcf77157e7104df230028d91ceda90d3d763b3624454b.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-56850107-gtx5t_gulel-worker_gulel-container-56850107-067020efc500cc1dce418c597c1527e07e5d8169b004bc72b2a3754b3cf633cb.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-fqhh8_gulel-nimbus_gulel-container-e2f13708-fbd5023b4f490d89013798758cb5a4913954f25d5fd97e966d55fe529b781d73.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-f3f24cec-v92fx_gulel-worker_gulel-container-f3f24cec-0ef41dac511e7aee97dba18ab2114ad416c9696b452e8ad15bd4e95db8474061.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/wavefront-proxy-64bb97d567-pbn7k_cluster-services_wavefront-proxy-7b7b4aab699225347581254e075bef52193377b5acf53e29c1eed614b70ced2d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-dm7kd_gulel-nimbus_gulel-container-7769ba08-a5431afae7c3cad26c983d19736078adc052935b71bd5cbd8da98d59b5c176d3.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-lxnmt_gulel-nimbus_gulel-container-dfd642a9-9f2ed9479ef1fc72bbec1597b3a20ac1e05935fd05961a882ed47b9d3d4f181c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-2a45c77d-8mshq_gulel-worker_gulel-container-2a45c77d-41cdf801493d868ae40dbab8906564f996fbd71c12520e7d36ba411a9482a85b.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-254e0c58-7sdkp_gulel-worker_gulel-container-254e0c58-98b8026683720bc84ebe510feeb58c1053865edb52213b417d108735de63a59c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-mk7wf_gulel-nimbus_gulel-container-7a5f9eb2-7a7ade713c7f4a195e776430cbcec98171243fbae4ad848be40ad1bc66dabe03.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-72c55315-jzmpz_gulel-worker_gulel-container-72c55315-c7c61657a8579226dd5196afb6121add96315e0308efef0a6668c0c0c2066aa8.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-brhxm_gulel-nimbus_gulel-container-5f016956-2305be0ed34c6ae3d95c7f348329c17a8c9be9c78f3187b4b8809e167c13567c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-ed48df94-9dmmn_gulel-worker_gulel-container-ed48df94-182f7f39b774ac1c3735ff1b4c157638ca157872d09d24b675302d26e28464e5.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-828b1ce3-mxmgr_gulel-worker_gulel-container-828b1ce3-615bbb0f48bb746e2b89c749d5b88cc3d9396c5dc4302d569690f12ae0692c3d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-0a4b4b51-fvrj5_gulel-worker_gulel-container-0a4b4b51-bd885bd0885d87de41b0884a95a912f57d78f7c153e85fd8c9a440c6a88f1ed7.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-690680ba-blmb8_gulel-worker_gulel-container-690680ba-5420cbac19b39ae9621434880b3d33202afc6ce8bbbeccbc6ce6d4b01025ab3f.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-gd8kx_gulel-nimbus_gulel-container-518a193f-5e9e305e69d9697cab96ce50d758fd222024770cd0e77c27c3a320c5c123a981.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/simple-flask-app-1-5cf57d9858-hx4tz_default_avi-ingress-simple-flask-app-1-581425470ec14950a4c9bcabdf62509c4e7150e2a8a4861183a4c6c36cd4ba53.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/devtools-test9-579d46559c-czsmq_default_devtools-test9-e3da3c06a7ff7b55c9dac0b5ce8eaf65b38f9b512a12366099aded1fef62bd5e.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-ndkds_gulel-nimbus_gulel-container-8638022a-d775090cf5c60d2eeabd9b5771a52c402ae153e83f24c74a331b84087d4ec035.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-f7v4p_gulel-nimbus_gulel-container-d299f2ff-f5c7a95b68b0db8bee73ff9d5345dc25532b905491ed46b4bec378b3c8742e5a.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-4sd49_gulel-nimbus_gulel-container-a37d80d3-601b8b7d39401db489d383b07c469e1357e6359491b1b8ee285b8bba0fbbaafa.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-f3d6cfef-f29fh_gulel-worker_gulel-container-f3d6cfef-b3ca02cdff0d97d0794465846efb3ba60188fcf0fd69b7b446308a137142b03c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-knq4b_gulel-nimbus_gulel-container-4306ed6b-d6c7b314b89fd1b7f986987dfb55040ae04c5cb0d0c3359ec9e2f905809db639.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/ako-0_avi-system_ako-4ac2dd331841699e850a18a6d08fa4c54a534e9076d3e5e8f9b3217595933636.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/antrea-agent-ng6zk_kube-system_install-cni-30e2407edd47e568de992bf495d75c90e11281caec8d9b31dce29e4805554f5d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/redis-master-0_gulel-core_redis-bcf428fce103c52a6f1cbbe5f67d4bff6bcd5b4361afd6209832d08012bd4d1e.log
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: gulel-core/redis-master-0
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: gulel-core
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-f5e95df9-9s72x_gulel-worker_gulel-container-f5e95df9-2224550a49008851d50a0a3c1a84fdf88d59a98f399e1594dabeb30d1ec8b16d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-znzvv_gulel-nimbus_gulel-container-92ed7b7a-9a0593622b4572e04bf1b68797201a41972c4a40502e161088a86a1f12744a47.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/antrea-agent-ng6zk_kube-system_antrea-ovs-a3c32e86d0dcca43a28caff94732a518038d568ffd48bf14f589d10a910f38f1.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/vsphere-csi-node-7xc6j_kube-system_liveness-probe-5b358341424c5dbfbb1a431e6bb0bdba43f07f9f76e224e9546b2f172b81055a.log
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: kube-system/vsphere-csi-node-7xc6j
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: kube-system
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-l42k8_gulel-nimbus_gulel-container-e940ce85-d49cce8d1c7610fa26af60939c82dd151d9d5991adf11c1142655046534c0431.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-9ddf16a2-xfdsn_gulel-worker_gulel-container-9ddf16a2-076ebdf890a2cd87c57fff349212214108e962d4f89b27ffd3edf0c75cb45170.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-cx4sg_gulel-nimbus_gulel-container-53637411-efef1eed3628b9d3d1e2fbe7e760ee4c6cf56c5a9749207ac71d146ac911664c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kube-fluentd-operator-bk4bz_cluster-services_reloader-b6389c82549a504b3801926ecd63a2161cad81eeda7c1fcad91232b6acfc501e.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-238f72e4-6z45s_gulel-worker_gulel-container-238f72e4-e1223fe0e897eec6d1ce2dfaaa36348baae90d529f6f74efd18afe448045b6dc.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-m6x7g_gulel-nimbus_gulel-container-7405325f-b517943c19fd639f59c2d5a9ddf0b37af81c08409134538c7eb7dfc10bf8ee20.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-0debfe48-xpgvh_gulel-worker_gulel-container-0debfe48-f3ac8b3fce1ce260f3a224bbd3faa8b3316b7d597ba32ca417a2103e3cc34824.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/avi-ingress-test-vmohite5-78ddc68d75-7jx4j_default_avi-ingress-test-vmohite5-e9363ad309ae4154370397de6602d474b54517e3c7d677f372bd6b14b5088985.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-4st2v_gulel-nimbus_gulel-container-3706cf80-96844bedc92478b277633cf63f2e8e7e534f911e453a47a4f7839f5d1b0907c9.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-b5d392d7-5fd7b_gulel-worker_gulel-container-b5d392d7-d451c086e89963d46a6804e6198d7991880a6fdd4d3c60f1adb11ca2841b4b89.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-6349c60a-4rtd7_gulel-worker_gulel-container-6349c60a-f902f37a7400af7ae6172f3a7ed34e9f65ddf7f147c66e36bae7daad96d1056e.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/antrea-agent-ng6zk_kube-system_antrea-agent-2302c55456870a3c6f4f8efa6231037e12fc71c31254f62a184d4ffe45ea1cb0.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/avi-ingress-test-vmohite1-698474f8f9-9qzzh_default_avi-ingress-test-vmohite1-da9c901f762723a9c3297ed46a849a46d2a050c818f06756ea636a16efed7138.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-5dc1d02f-8lqbs_gulel-worker_gulel-container-5dc1d02f-423b4e99f08fbbafa25434da42be31187618bf4f4593c16499f0f099087592ab.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-f7c68c84-wr5zh_gulel-worker_gulel-container-f7c68c84-baf968a68f4e1e89bf29a213e43e8390c2e64522e5d3b5436a518d9d554b1713.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-gzb67_gulel-nimbus_gulel-container-10b4b02e-0049df16d6ddc1fb4b8673dbe541e7bf822c8e5d636f03710d0f0ea652172de8.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-0f96f412-jmxrb_gulel-worker_gulel-container-0f96f412-fec4182b02e2724381459248dd01c4b42f56a9fb6f2f3f9eea2985850fde7958.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-4fbece99-6s6mn_gulel-worker_gulel-container-4fbece99-fccea3a1f19377ebb397a7ee44dfe1a690b4dd9a5f37b00fe79f6fca75d81d30.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/avi-ingress-test-vmohite4-596f6bc5f7-c4k7n_default_avi-ingress-test-vmohite4-497e177ee338171a2a2df3e1fd42c029435d902eedcfef182b27879126ef94f0.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-1153a8f6-qnq5k_gulel-worker_gulel-container-1153a8f6-ed75e93583aae4535ed24dbea566368f0bc6d81a37da1877b01ed04a1c7e3003.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-fjn5n_gulel-nimbus_gulel-container-03739a5c-e54d8d413f45f486c0fbf03d139c7b01cbcf5e4954c3229284f61cf9e99299c5.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-xxm8j_gulel-nimbus_gulel-container-0bbae7cf-b9a460fd1a53fdad4bc81301338148e02e42b63f0552997966ed89c4213fd5ad.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-69545c86-n8hhl_gulel-worker_gulel-container-69545c86-feb9649c2afe1ee7042d2a32a11a597fdb7d45a6e47039f189794ebc0fa1f887.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-409c7a3d-2tfcj_gulel-worker_gulel-container-409c7a3d-ced352bb8aee2dd36953f06187ee3c45fff333a4c51e273beb885eebb3bc63d5.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-2srt6_gulel-nimbus_gulel-container-f7aa5330-b275b62f0c3a3351c44d67971aa56a7b69edf29de4e391e05367c98ffa15d4ca.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-fvn8d_gulel-nimbus_gulel-container-3661369e-e0cfd787602cb87c865287bc4ec07403ea515ffa882fd538e30799f5ae9fcc13.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/avi-ingress-test-vmohite6-5ddb66cc69-rx2f2_default_avi-ingress-test-vmohite6-6b62779368cba4a248b9b2ae0c52c918d3f20cb59d1c5012c4000813314ef654.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-99eb2b1a-fp2jw_gulel-worker_gulel-container-99eb2b1a-de76edb6ba47731784cd71810471b281cf6da2f5f830b9a89b319f08139d4ae8.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-86d8g_gulel-nimbus_gulel-container-cfe4f336-b14d30b88ff6006b26d3f3f22c82c05eaedad92baa4aed7cfc2f494be49d8710.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-wdxwh_gulel-nimbus_gulel-container-5f6912db-094819918a37776c8f920379b4f22d83a41aa922afb18abaf03aede46eb3aafc.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-l87hq_gulel-nimbus_gulel-container-221f2d18-918ce4c82b27afdee4bf3c516a6d75d62ef92487f41a72b595667c70b823d09e.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-g66z6_gulel-nimbus_gulel-container-156dd45a-cc89ce70c567b409260378bd88d1006a7dc79ebedd9e5d7d82df16c09a6d1ac7.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/vsphere-csi-node-7xc6j_kube-system_vsphere-csi-node-8d57d6c883155b50a1eed4ec8c32d1d80ffd26b5bdf3f9e865eceb6d915715ac.log
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: kube-system/vsphere-csi-node-7xc6j
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: kube-system
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/avi-ingress-test-vmohite-776758d87c-rfp85_default_avi-ingress-test-vmohite-1d8c5a555ececa09e5e32efd51687f302f07cb25a4521a2ef63521781ea09e05.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/coredns-78dfc56b6f-dbg6w_kube-system_coredns-f0e56eb26fce56c32ea5168326852f62737ad5f7f5ece77c7a66398aeadc402c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-frslx_gulel-nimbus_gulel-container-84cc7b41-e9832af3b8da3ec3457b16514beb81cb9de304eaa6ec542bef23dc5feea8825e.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-9ef05fd1-nmf57_gulel-worker_gulel-container-9ef05fd1-d92ee5e7f57ee15da77faa85dc4e2468d5f2e18762165d91ae20857fbdd2aaf2.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-j2gqn_gulel-nimbus_gulel-container-41ea5a71-093009cc896beb4274a2e0fb1261e24599ea42fe62014c6b06c19ff8e4ea4ae9.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-lmvrk_gulel-nimbus_gulel-container-84c884f6-5b59112cfb66bc897083db45bff5cd3c23df3ef29821b122ec1f38f7c780ac1f.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-jrdzr_gulel-nimbus_gulel-container-d56fadeb-869cefed088ddf77affa59620244bbc07f51761229a538e83c6941a18ca58f21.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-7a1886c6-2pwl7_gulel-worker_gulel-container-7a1886c6-7f8695a354ec3c0e938ee058a9579f33247d8e51193d6f9359449eeafcce8350.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-vsmvc_gulel-nimbus_gulel-container-faeb472d-2a3541f8af38d2c11837bb353fd8ed9f2d5f186925ca80aa60f1c98eb0e42b09.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-d24dedfe-mlg48_gulel-worker_gulel-container-d24dedfe-ee319ae2282cedfb5e1320f94a8b08ac4ca263b348c898bc3f546079897f69fe.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-18b1f346-b45qt_gulel-worker_gulel-container-18b1f346-616251a0583ed3b2913582086320bb6224525d184ee9cc52cb2442caa02f616f.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-4xzvn_gulel-nimbus_gulel-container-74d686e8-953f477249f430caef0c2965a6121a39c311a1ff7b18fd789015160e483cef67.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-6bwm2_gulel-nimbus_gulel-container-56475349-f43b4a5882a9351093a45fbbec7afe1c5c66b05aa5c815b78a11246959d8f9fa.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-9cjq2_gulel-nimbus_gulel-container-1e39095f-80e991b0cea02475274391afb2c081de5439b5d8bdaacb681426897dcf613a8d.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-f4thl_gulel-nimbus_gulel-container-1ffce882-d6d6728621d280fdc6f92f0d21b0ed7ada6707acf78aa9171af6870dab6d1b08.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/metrics-server-67d7bb8965-l95gt_cluster-services_metrics-server-e9c9b95d8809a9e9eccea15c58ea3ba0a47ce6ef62dfc5a8dac93faa443b4598.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/kube-proxy-27jt4_kube-system_kube-proxy-deaa9fa41007a33167c1044c296731b0e0f6249d23190857521933abfb6a03fa.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/gulel-nimbus-53b9a040-qrh6n_gulel-worker_gulel-container-53b9a040-36c4ca15bd2a0085095274744cf498662f2e38971b1e43b6051edba455b5625c.log
2021-04-21 02:10:00 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/nfs-proxy-54bb9c4458-qm25f_cluster-services_nfs-proxy-063928113ca392cd3f06708cca1ce5b9394a84677fa221f2d7c0daa53e912e37.log
2021-04-21 02:10:00 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: cluster-services/nfs-proxy-54bb9c4458-qm25f
2021-04-21 02:10:00 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: cluster-services
2021-04-21 02:10:01 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:08 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:08 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: cluster-services/wavefront-proxy-64bb97d567-pbn7k
2021-04-21 02:10:08 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: cluster-services
2021-04-21 02:10:08 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:21 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:21 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: cluster-services/metrics-server-67d7bb8965-l95gt
2021-04-21 02:10:21 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: cluster-services
2021-04-21 02:10:21 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:10:29 +0000 [info]: #0 fluent/log.rb:329:info: stats - namespace_cache_size: 13, pod_cache_size: 131, namespace_cache_api_updates: 7, pod_cache_api_updates: 7, id_cache_miss: 7, pod_cache_host_updates: 131, namespace_cache_host_updates: 13
2021-04-21 02:10:42 +0000 [debug]: #0 fluent/log.rb:308:debug: #<Async::HTTP::Server:0x00000000042e21c0>Incoming connnection from #<Addrinfo: 172.24.12.100:58052 TCP> to Async::HTTP::Protocol::HTTP1
2021-04-21 02:10:59 +0000 [info]: #0 fluent/log.rb:329:info: stats - namespace_cache_size: 13, pod_cache_size: 131, namespace_cache_api_updates: 7, pod_cache_api_updates: 7, id_cache_miss: 7, pod_cache_host_updates: 131, namespace_cache_host_updates: 13
2021-04-21 02:11:01 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:11:01 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching pod metadata: gulel-core/gulel-launcher-64b8bf6c87-b2qhr
2021-04-21 02:11:01 +0000 [trace]: #0 fluent/log.rb:286:trace: fetching namespace metadata: gulel-core
2021-04-21 02:11:01 +0000 [info]: #0 fluent/log.rb:329:info: disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-04-21 02:11:29 +0000 [info]: #0 fluent/log.rb:329:info: stats - namespace_cache_size: 13, pod_cache_size: 131, namespace_cache_api_updates: 8, pod_cache_api_updates: 8, id_cache_miss: 8, pod_cache_host_updates: 131, namespace_cache_host_updates: 13

Additional context

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions