Chapter 3. Editing kubelet log level verbosity and gathering logs

Procedure

• Use the following MachineConfig object for persistent kubelet log level configuration:

  Copy to Clipboard Toggle word wrap

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 99-master-kubelet-loglevel
   spec:
     config:
       ignition:
         version: 3.2.0
       systemd:
         units:
           - name: kubelet.service
             enabled: true
             dropins:
               - name: 30-logging.conf
                 contents: |
                   [Service]
                   Environment="KUBELET_LOG_LEVEL=2"

  Generally, it is recommended to apply 0-4 as debug-level logs and 5-8 as trace-level logs.

Procedure

• After the kubelet’s log level verbosity is configured properly, you can gather logs by running the following commands:

  Copy to Clipboard Toggle word wrap

  $ oc adm node-logs --role master -u kubelet

  Copy to Clipboard Toggle word wrap

  $ oc adm node-logs --role worker -u kubelet

  Alternatively, inside the node, run the following command:

  Copy to Clipboard Toggle word wrap

  $ journalctl -b -f -u kubelet.service

• To collect master container logs, run the following command:

  Copy to Clipboard Toggle word wrap

  $ sudo tail -f /var/log/containers/*

• To directly gather the logs of all nodes, run the following command:

  Copy to Clipboard Toggle word wrap

  - for n in $(oc get node --no-headers | awk '{print $1}'); do oc adm node-logs $n | gzip > $n.log.gz; done