Configure all containers in a Kubernetes cluster to regularly and reliably synchronize with a Network Time Protocol (NTP) server to ensure consistency among all containers in the cluster. When times are out of sync, this can lead to unexpected behavior. This then results in poor performance, among other things.
It can also lead to errors that are not necessarily caused by the timing problem. For example, if you want to make a system update or update Kubernetes. Just there are then also problems with certificates.
Problems or confusion can also arise when reading log files if a different time is suddenly displayed in the log.
Problems can also arise with the so-called cronjobs during execution.
Install NTP on each machine in your cluster:
Here for CentOS / RHEL:
sudo yum install ntp ntpdate
sudo systemctl start ntpd
sudo systemctl enable ntpd
sudo systemctl status ntpd
sudo ntpdate -u -s 0.centos.pool.ntp.org
sudo systemctl restart ntpd
If you then check with “ timedatectl “, it should look like this:
First, we should find out what time zone our pods use.
Containers do not inherit time zones from hosts and only have access to the clock through the kernel. The default time zone for most images is UTC, but it is not guaranteed and can vary from container to container.
Imagine we have the following pod description:
apiVersion: v1 kind: Pod metadata: name: busybox-sleep spec: containers: - name: busybox image: busybox args: - sleep - "1000000"
In the above mentioned file the command "sleep" is executed 1000000 times within the OS Busybox.
Now save this file as timecheck.yaml. After that please create this pod and we will check which timezone is used by the pod.
$ kubectl apply -f timecheck.yaml
You should now get a response that the pod was successfully created.
Now we use this command to check which time zone the Pod is using:
$ kubectl exec busybox-sleep date
The Pod uses the UTC time zone.
So, now we change this with additional information (red box) in the yaml file:
apiVersion: v1 kind: Pod metadata: name: busybox-sleep spec: containers: - name: busybox image: busybox args: - sleep - "1000000" volumeMounts: - name: tz-config mountPath: /etc/localtime volumes: - name: tz-config hostPath: path: /usr/share/zoneinfo/Europe/Berlin
Save this file as timecheck-new.yaml and then start it with
$ kubectl apply -f timecheck-new.yaml
Now we check again with the command which time zone the Pod uses:
$ kubectl exec busybox-sleep -- date
As we can see now, we have the desired CEST time zone in our pod.
If you want to change the time zone for all pods, you need to include this extension with the volumes and volumeMounts in their pod description. This ensures that each pod will then use the desired time zone in the future.
Automated setting of time zones:
The k8tz tool is a Kubernetes Admission Controller and a "CLI" tool to inject time zones into pods and CronJobs.
The tool and documentation can be found here.
Author: Ralf Menti
Please feel free to contact us for any question that is not answered yet.
We are looking forward to get in contact with you!
Hinter Stöck 17
+49 7433 93724 00